Merge pull request #7 from databrickslabs/feature/add-docs

Feature/add docs

Author: nkvuong

.github/workflows/docs.yml

@@ -0,0 +1,26 @@
+name: docs
+on:
+  push:
+    branches:
+    - main
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+    - name: install pandoc
+      run: sudo apt-get install pandoc
+    - uses: actions/setup-python@v2
+    - uses: actions/checkout@v2
+      with:
+        ref: main
+        fetch-depth: 0 # otherwise, you will failed to push refs to dest repo
+    - name: Build and Commit
+      uses: sphinx-notes/pages@v2
+      with:
+        documentation_path: docs/source
+        requirements_path: docs/docs-requirements.txt
+    - name: Push changes
+      uses: ad-m/github-push-action@master
+      with:
+        github_token: ${{ secrets.GITHUB_TOKEN }}
+        branch: gh-pages

.gitignore

@@ -200,4 +200,7 @@ Icon
 .AppleDesktop
 Network Trash Folder
 Temporary Items
-.apdisk
+.apdisk
+
+# Ignore docs/_build
+docs/_build

README.md

@@ -1,8 +1,26 @@
-# PROJECT NAME
-Standard Project Template for Databricks Labs Projects
+# Delta Sharing Java Connector
+A java connector for [delta-sharing](https://delta.io/sharing/) that allows you to easily ingest data on any JVM.
+
 
 ## Project Description
-Short description of project's purpose
+This project brings delta-sharing capabilities to java.
+The Java connector follows the Delta Sharing protocol to read shared tables from a Delta Sharing Server. To further reduce and limit egress costs on the Data Provider side, we implemented a persistent cache to reduce and limit the egress costs on the Data Provider side by removing any unnecessary reads.
+
+- The data is served to the connector via persisted cache to limit the egress costs whenever possible.
+  - Instead of keeping all table data in memory, we will use file stream readers to serve larger datasets even when there isn't enough memory available.
+  - Each table will have a dedicated file stream reader per part file that is held in the persistent cache. File stream readers allow us to read the data in blocks of records and we can process data with more flexibility.
+  - Data records are provided as a set of Avro GenericRecords that provide a good balance between the flexibility of representation and integrational capabilities. GenericRecords can easily be exported to JSON and/or other formats using EncoderFactory in Avro.
+
+- Every time the data access is requested the connector will check for the metadata updates and refresh the table data in case of any metadata changes.
+   - The connector requests the metadata for the table based on its coordinate from the provider. The table coordinate is the profile file path following with `#` and the fully qualified name of a table (<share-name>.<schema-name>.<table-name>)
+   - A lookup of table to metadata is maintained inside the JVM. The connector then compares the received metadata with the last metadata snapshot. If there is no change, then the existing table data is served from cache. Otherwise, the connector will refresh the table data in the cache.
+  
+- When the metadata changes are detected both the data and the metadata will be updated.
+  - The connector will request the pre-signed urls for the table defined by the fully qualified table name. The connector will only download the file whose metadata has changed and will store these files into the persisted cache location.
+
+
+In the current implementation, the persistent cache is located in dedicated temporary locations that are destroyed when the JVM is shutdown. This is an important consideration since it avoids persisting orphaned data locally.
+
 
 ## Project Support
 Please note that all projects in the /databrickslabs github account are provided for your exploration only, and are not formally supported by Databricks with Service Level Agreements (SLAs).  They are provided AS-IS and we do not make any guarantees of any kind.  Please do not submit a support ticket relating to any issues arising from the use of these projects.
@@ -11,13 +29,11 @@ Any issues discovered through the use of this project should be filed as GitHub
 
 
 ## Building the Project
-Instructions for how to build the project
-
-## Deploying / Installing the Project
-Instructions for how to deploy the project, or install it
-
-## Releasing the Project
-Instructions for how to release a version of the project
+The project is implemented on top of maven.
+To build the project locally:
+- Make sure you are in the root directory of the project
+- Run `mvn clean install`
+- The jars will be available in /target directory
 
 ## Using the Project
-Simple examples on how to use the project
+To use the connector in your projects use maven coordinates with desired version.

codecov.yml

@@ -0,0 +1,17 @@
+ignore:
+  - "vendor/**/*"
+  - "dist/**/*"
+codecov:
+  token: 8b1ac57c-84af-41c8-b7d2-f00fff24ef4c
+coverage:
+  status:
+    project: yes
+    patch: true
+    changes: true
+comment:
+  layout: "reach, diff, flags, files"
+  behavior: default
+  require_changes: false   # if true: only post the comment if coverage changes
+  require_base: true       # [true :: must have a base report to post]
+  require_head: true       # [true :: must have a head report to post]
+  branches: []             # branch names that can post comment

docs/Makefile

@@ -0,0 +1,21 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = source
+BUILDDIR      = _build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+

docs/docs-requirements.txt

@@ -0,0 +1,9 @@
+Sphinx==4.4.0
+sphinx-material==0.0.35
+nbsphinx==0.8.8
+pandoc==2.0.1
+ipython==8.0.1
+sphinxcontrib-fulltoc==1.2.0
+livereload==2.6.3
+autodocsumm==0.2.7
+sphinx-tabs==3.2.0

docs/source/conf.py

@@ -0,0 +1,102 @@
+# Configuration file for the Sphinx documentation builder.
+#
+# This file only contains a selection of the most common options. For a full
+# list see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+import os
+import sys
+sys.path.insert(0, os.path.abspath('../../python'))
+
+
+# -- Project information -----------------------------------------------------
+
+project = 'delta-sharing-java-connector'
+copyright = '2022, Databricks Inc'
+author = 'Milos Colic, Vuong Nguyen'
+
+# The full version, including alpha/beta/rc tags
+release = 'v0.1-alpha'
+
+
+# -- General configuration ---------------------------------------------------
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    "sphinx_material",
+    "nbsphinx",
+    "sphinx_tabs.tabs",
+    "sphinx.ext.githubpages",
+    "sphinx.ext.autosectionlabel",
+    "sphinx.ext.todo"
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store', ".env"]
+source_suffix = [".rst", ".md"]
+
+pygments_style = 'sphinx'
+nbsphinx_execute = 'never'
+napoleon_use_admonition_for_notes = True
+sphinx_tabs_disable_tab_closing = True
+todo_include_todos = True
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = 'sphinx_material'
+
+# Material theme options (see theme.conf for more information)
+html_theme_options = {
+
+    # Set the name of the project to appear in the navigation.
+    'nav_title': f'delta-sharing-java-connector {release}',
+
+    # Specify a base_url used to generate sitemap.xml. If not
+    # specified, then no sitemap will be built.
+    # 'base_url': 'https://project.github.io/project',
+
+    # Set the color and the accent color
+    'color_primary': 'green',
+    'color_accent': 'green',
+
+    # Set the repo location to get a badge with stats
+    'repo_url': 'https://github.com/databrickslabs/delta-sharing-java-connector/',
+    'repo_name': 'delta-sharing-java-connector',
+
+    'master_doc': False,
+
+    'globaltoc_depth': 1,
+    'globaltoc_collapse': True,
+    'globaltoc_includehidden': True,    
+    'heroes': {'index': 'Simple and easy to use java connector for delta sharing.'},
+    "version_dropdown": True,
+    # "version_json": "../versions-v2.json",
+
+}
+html_title = project
+html_short_title = project
+html_logo = 'images/delta-sharing-java-logo.png'
+html_sidebars = {
+    "**": ["logo-text.html", "globaltoc.html", "localtoc.html", "searchbox.html"]
+}
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']

docs/source/images/delta-sharing-java-logo.png

@@ -0,0 +1,57 @@
+.. delta-sharing-java-connector documentation master file, created by
+   sphinx-quickstart on Wed Feb  2 11:01:42 2022.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+.. image:: images/delta-sharing-java-logo.png
+   :width: 10%
+   :alt: delta-sharing-java-connector
+   :align: left
+
+delta-sharing-java-connector is an extension to the `delta sharing <https://delta.io/sharing/>`__ that allows easy and simple ingestion of delta sharing tables in java applications.
+
+.. image:: images/high-level-design.png
+   :width: 100%
+   :alt: High level design of the connector
+   :align: center
+
+delta-sharing-java-connector provides:
+   - simple to use APIs;
+   - local caching of remote files to limit egress/ingress costs;
+   - readers based on a round robin streams to limit runtime memory requirements;
+
+Since this is a java connector, all APIs can be used from either java or scala.
+The intended usage of the connector is to provide connectivity to remote data hosted on delta sharing
+from JVM based applications that may run only on a single node.
+
+Note: the code does depend on spark binaries but does not require spark service running at serving time.
+
+
+Documentation
+=============
+
+.. toctree::
+   :maxdepth: 1
+   :caption: Contents:
+
+   provider/json
+   usage/DeltaSharing
+   usage/TableReader
+
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`search`
+
+
+.. * :ref:`modindex`
+
+
+Project Support
+===============
+
+Please note that all projects in the ``databrickslabs`` github space are provided for your exploration only, and are not formally supported by Databricks with Service Level Agreements (SLAs). They are provided AS-IS and we do not make any guarantees of any kind. Please do not submit a support ticket relating to any issues arising from the use of these projects.
+
+Any issues discovered through the use of this project should be filed as GitHub Issues on the Repo. They will be reviewed as time permits, but there are no formal SLAs for support.