mirror of
https://github.com/apache/iceberg-rust.git
synced 2026-11-06 10:18:04 +00:00
Apache Iceberg
https://rust.iceberg.apache.org/
- Rust 98.5%
- Python 0.7%
- Shell 0.6%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
59946648ef* fix(arrow): release a positional delete file whose load fails try_start_pos_del_load marks a positional delete file Loading, and only the success path marked it Loaded. If opening or parsing the file failed, or the load was dropped because another delete file failed, the entry stayed Loading: a retry on the same loader, a clone of the same ArrowReader, or a concurrent load of the same file waited for a notification that never came. The Load action now carries a PosDelLoadGuard. finish() marks the file loaded; dropping the guard without finishing removes the Loading marker and wakes the waiters. A woken waiter checks the state again and loads the file itself if it isn't loaded, instead of assuming it is. Closes #3434. * refactor(arrow): assert the Loading invariant when abandoning a load Only try_start_pos_del_load inserts into positional_deletes, and only the guard that it returns changes the entry afterwards, so the entry is always the guard's Loading marker when it is abandoned. * test(arrow): cover waiters that load after an abandoned positional load Hold the Load guard, poll load_file_for_task for the same file until it waits, then drop the guard: the waiter loads the file itself, or returns the open error when the file is gone. Also load a positional delete file that opens but fails to parse, then rewrite it and load it again on the same loader, which hangs without the guard. * fix(arrow): return a failed positional delete load's error to its waiters Waiters on a failed load used to retry it one after another, so a missing file shared by many tasks was opened once per task. The load now records its error and the tasks waiting on that attempt return it. A load dropped without an error, when a sibling's error stops load_deletes, still lets its waiters retry, and later calls start a new attempt either way. Abandoning a load also tolerates a poisoned lock, since it runs from Drop. * test(arrow): keep the sibling's load in flight when another delete file fails On the current-thread test runtime the local filesystem never yields, so the valid file was only loaded on the second call and its guard was never dropped mid-load. A test storage now yields before reads and holds one file until the test opens a gate. * fix(arrow): pass the whole load error to waiters |
||
| .cargo | ||
| .devcontainer | ||
| .github | ||
| .idea | ||
| bindings/python | ||
| crates | ||
| dev | ||
| docs/rfcs | ||
| website | ||
| .asf.yaml | ||
| .gitattributes | ||
| .gitignore | ||
| .licenserc.yaml | ||
| .taplo.toml | ||
| .typos.toml | ||
| AGENTS.md | ||
| Cargo.lock | ||
| Cargo.toml | ||
| CHANGELOG.md | ||
| CONTRIBUTING.md | ||
| deny.toml | ||
| LICENSE | ||
| Makefile | ||
| NOTICE | ||
| README.md | ||
| rust-toolchain.toml | ||
| rustfmt.toml | ||
| SECURITY-THREAT-MODEL.md | ||
Apache Iceberg™ Rust
Rust implementation of Apache Iceberg™.
Components
The Apache Iceberg Rust project is composed of the following components:
| Name | Release | Docs |
|---|---|---|
| iceberg |  |
  |
| iceberg-catalog-loader |  |
|
| iceberg-catalog-glue |  |
|
| iceberg-catalog-hms |  |
|
| iceberg-catalog-rest |  |
|
| iceberg-catalog-s3tables |  |
|
| iceberg-catalog-sql |  |
|
| iceberg-cache-moka |  |
|
| iceberg-storage-opendal |  |
|
Iceberg Rust Implementation Status
The features that Iceberg Rust currently supports can be found here.
Supported Rust Version
Iceberg Rust is built and tested with stable rust, and will keep a rolling MSRV (minimum supported rust version). At least three months from latest rust release is supported. MSRV is updated when we release iceberg-rust.
Check the current MSRV on crates.io.
Contribute
Apache Iceberg is an active open-source project, governed under the Apache Software Foundation (ASF). Apache Iceberg Rust is always open to people who want to use or contribute to it. Here are some ways to get involved.
- Start with Contributing Guide.
- Submit Issues for bug report or feature requests.
- Discuss at dev mailing list (subscribe / unsubscribe / archives)
- Talk to the community directly at Slack #rust channel.
The Apache Iceberg community is built on the principles described in the Apache Way and all who engage with the community are expected to be respectful, open, come with the best interests of the community in mind, and abide by the Apache Foundation Code of Conduct.
Users
- Databend: An open-source cloud data warehouse that serves as a cost-effective alternative to Snowflake.
- Lakekeeper: An Apache-licensed Iceberg REST Catalog with data access controls.
- Moonlink: A Rust library that enables sub-second mirroring (CDC) of Postgres tables into Iceberg.
- RisingWave: A Postgres-compatible SQL database designed for real-time event streaming data processing, analysis, and management.
- Wrappers: Postgres Foreign Data Wrapper development framework in Rust.
- ETL: Stream your Postgres data anywhere in real-time.
- Apache DataFusion Comet: High-performance accelerator for Apache Spark, built on top of the powerful Apache DataFusion query engine.
- Spice.ai: An open-source compute and AI runtime for data-grounded agents.
License
Licensed under the Apache License, Version 2.0