- Jupyter Notebook 81.1%
- Python 16.9%
- Lua 0.8%
- Dockerfile 0.7%
- HCL 0.3%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
7c42b83f66* Make the Backblaze B2 notebook safe to re-run and correct a few B2 notes Running the notebook a second time failed at commit and merge with no changes, which left main_commit_id unset and broke both dataset cells. It now skips images already on the branch, only commits and merges when there is something new, and skips the initial dataset publish if the dataset already exists. Also reword a few descriptions of B2 to match what B2 actually does, fix the setup step numbering, and drop the Docker Hub token mention. * Be precise about what lakeFS keeps in Postgres Commit records live in the lakeFS metadata store alongside branches, so say that, and stop describing everything as being in B2. Also tidy the conditional write paragraph. |
||
| .github/workflows | ||
| 00_notebooks | ||
| 01_standalone_examples | ||
| 02_lakefs_enterprise | ||
| data | ||
| images | ||
| jupyter | ||
| lakeFS-hooks@01a70e297b | ||
| .gitignore | ||
| .gitmodules | ||
| docker-compose.yml | ||
| LICENSE.txt | ||
| README.md | ||
lakefs-samples
Incorporating the Docker Compose formally known as Everything Bagel.
This sample repository captures a collection of notebooks, dockerized applications and code snippets that demonstrate how to use lakeFS.
lakeFS is a popular community solution for managing data. It provides a consistent and scalable data management layer on top of cloud storage, such as Amazon S3, Azure Blob Storage, or Google Cloud Storage. It allows users to create and manage data in a version-controlled and immutable manner, and offers features such as data governance, data lineage, and data access controls. lakeFS is compatible with a wide range of data processing frameworks and tools.
Go to lakefs_enterprise folder if you want to use lakeFS Enterprise instead of lakeFS Community Edition
Let's Get Started 👩🏻💻
Clone this repository
git clone https://github.com/treeverse/lakeFS-samples.git
cd lakeFS-samples
You now have two options:
Run a Notebook server with your existing lakeFS Server
If you have already installed lakeFS or are utilizing lakeFS cloud, all you need to run is the Jupyter notebook server:
docker compose up
Once the stack's up and running, open the Jupyter Notebook (http://localhost:8888) and check out the catalog of sample notebooks to explore lakeFS.
Once you've finished, run the following to remove all the containers:
docker compose down
Don't have a lakeFS Server or Object Store?
If you want to provision a lakeFS server as well as MinIO for your object store, plus Jupyter then bring up the full stack:
docker compose --profile local-lakefs up
As above, open the Jupyter Notebook (http://localhost:8888) peruse the catalog of sample notebooks to explore lakeFS.
Environment Details
- Jupyter Notebook is based on the Jupyter PySpark notebook and provides an interactive environment in which to explore lakeFS using Python and PySpark.
- lakeFS can be provisioned as part of this environment, or provided by lakeFS cloud or your own installation.
- If you run lakeFS as part of this environment, MinIO is provided as an S3-compatible object store. If you run lakeFS yourself you can use other S3-compatible object stores include S3, GCS, as well as MinIO
Containers
URLs and login details
- Jupyter http://localhost:8888/
If you've brought up the full stack you'll also have:
- LakeFS http://localhost:8000/ (
AKIAIOSFOLKFSSAMPLES/wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY) - MinIO http://localhost:9001/ (
minioadmin/minioadmin) - Spark UI http://localhost:4040/
Other Examples
Under the standalone_examples folder are a set of examples that need to be run on their own. Some use the repository's Docker Compose file and extend it, and others are self-contained and use their own Dockerfile.
- Airflow (1) - Four examples of using lakeFS with Airflow:
- Versioning DAGs and running pipeline from hooks using a configurable version of DAGs
- Isolating Airflow job run and atomic promotion to production
- Integration of lakeFS with Airflow via Hooks
- Troubleshooting production issues
- Integration of lakeFS with Airflow and Databricks
- Integration of lakeFS with Airflow and Iceberg
- Airflow (2) - lakeFS + Airflow
- Azure Databricks
- AWS Databricks
- Databricks CI/CD
- AWS Glue and Athena
- AWS Glue and Trino
- lakeFS Iceberg REST Catalog with Trino client
- lakeFS + Dagster
- lakeFS + Prefect
- lakeFS Mount Demo: Fast Data Loading and Reproducibility for Deep Learning Workloads with lakeFS Mount
- Reproducibility and Building an AI Agent by using lakeFS, LangChain and LLM/OpenAI Models
See also the accompanying blog - Image Segmentation Demo: ML Data Version Control and Reproducibility at Scale
- Multimodal Data Demo: ML Data Version Control and Reproducibility of Multimodal Data
- Labelbox integration
- Kafka integration
- Flink integration
- Red Hat OpenShift AI integration
- Backblaze B2 integration - Run lakeFS Enterprise with Backblaze B2 as its storage, and publish a versioned, shareable Dataset from data stored there
- How to backup, migrate or clone a repo
- Running lakeFS with PostgreSQL as K/V store
- Running lakeFS with NetApp ONTAP storage
- Running lakeFS with OneBucket (Attimis) as an S3-compatible blockstore
Got Questions or Want to Chat?
👉🏻 Join the lakeFS Slack group - https://lakefs.io/slack