- Rust 99.7%
- Python 0.2%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
bb47f8091e# Which issue does this PR close? - Closes #12023 # Rationale for this change `BooleanBufferBuilder::append_slice` sets one bit at a time behind a data-dependent branch. This is the path the Parquet reader takes to build `BooleanArray` values (`BooleanBuffer::from(&[bool])`), and it dominates reading BOOLEAN columns. See the issue for details. # What changes are included in this PR? For slices of more than 8 values, `append_slice` now: 1. fills the first byte up to the next byte boundary (at most 7 values), 2. packs each following group of 8 bools into one byte with `u64::from_le_bytes(bytes).wrapping_mul(0x0102_0408_1020_4080) >> 56`, writing whole bytes, and 3. writes the remaining fewer than 8 values into the last byte. Steps 1 and 3, and slices of up to 8 values, build their bits in a register with `(v as u8) << i` and update each byte with a single OR, so no step branches on the values. Up to 8 values touch at most two bytes, which also covers 8 values at an unaligned offset, where no whole byte would be left to pack. The packing path lives in an `#[inline(never)]` helper so the inlined code for short appends stays small. The `unsafe` call to `set_bit_raw` is gone. The multiply works because each input byte is 0 or 1 (`bool as u8`): byte `i` lands on bit `56 + i`, and no two partial products overlap, so there are no carries. `from_le_bytes` makes the result the same on big-endian targets. Whole bytes are written only past the previous length, and those bytes were just zero-filled by `advance`, so overwriting them is safe, including after `truncate`. Also adds a `boolean_buffer_builder` criterion bench to arrow-buffer, covering 64Ki-value appends, repeated appends of 1 to 7 values, and repeated appends of 8 values at offsets 0 and 3. # Are these changes tested? Yes. Two new unit tests: - `test_append_slice_all_byte_patterns` checks all 256 patterns of 8 bools, through both the short path and the multiply. - `test_append_slice_offsets` checks every starting bit offset 0..16 against slice lengths 0..100, starting after a `truncate`, and followed by another `append`. Benchmarks, Apple M-series (aarch64), from back-to-back runs of `main` and this PR: `cargo bench -p arrow-buffer --bench boolean_buffer_builder` (offset 3 starts mid-byte; all appends read fresh values): | case | main | this PR | change | |---|---|---|---| | 64Ki random, offset 0 | 165.0 µs | 1.22 µs | −99.3% | | 64Ki random, offset 3 | 165.6 µs | 1.26 µs | −99.2% | | 64Ki 90% true, offset 0 | 76.0 µs | 1.21 µs | −98.4% | | 64Ki 90% true, offset 3 | 75.6 µs | 1.27 µs | −98.3% | | 64Ki all true, offset 0 | 94.5 µs | 1.21 µs | −98.7% | | 64Ki all true, offset 3 | 93.7 µs | 1.25 µs | −98.7% | | 7168 appends of 1–7 values, random | 77.5 µs | 18.7 µs | −75.9% | | 7168 appends of 1–7 values, all false | 21.8 µs | 17.6 µs | −19.5% | | 7168 appends of 8 values, offset 0, random | 168.3 µs | 17.4 µs | −89.7% | | 7168 appends of 8 values, offset 3, random | 169.0 µs | 17.2 µs | −89.8% | | 7168 appends of 8 values, offset 0, all false | 27.2 µs | 17.3 µs | −36.2% | | 7168 appends of 8 values, offset 3, all false | 27.2 µs | 16.9 µs | −37.8% | The existing `bench_bool` in `arrow/benches/builder.rs` (`BooleanBuilder::append_slice`, 64 batches of 8Ki random bools), measured at |
||
| .config | ||
| .github | ||
| arrow | ||
| arrow-arith | ||
| arrow-array | ||
| arrow-avro | ||
| arrow-buffer | ||
| arrow-cast | ||
| arrow-cmp | ||
| arrow-csv | ||
| arrow-data | ||
| arrow-flight | ||
| arrow-integration-test | ||
| arrow-integration-testing | ||
| arrow-ipc | ||
| arrow-json | ||
| arrow-ord | ||
| arrow-pyarrow | ||
| arrow-pyarrow-integration-testing | ||
| arrow-pyarrow-testing | ||
| arrow-row | ||
| arrow-schema | ||
| arrow-select | ||
| arrow-string | ||
| dev | ||
| format | ||
| parquet | ||
| parquet-geospatial | ||
| parquet-testing@3acde70509 | ||
| parquet-variant | ||
| parquet-variant-compute | ||
| parquet-variant-json | ||
| parquet_derive | ||
| parquet_derive_test | ||
| testing@735ae7128d | ||
| .asf.yaml | ||
| .gitattributes | ||
| .gitignore | ||
| .gitmodules | ||
| AGENTS.md | ||
| Cargo.lock | ||
| Cargo.toml | ||
| CHANGELOG-old.md | ||
| CHANGELOG.md | ||
| CLAUDE.md | ||
| cliff.toml | ||
| CODE_OF_CONDUCT.md | ||
| CONTRIBUTING.md | ||
| LICENSE.txt | ||
| licenserc.toml | ||
| NOTICE.txt | ||
| README.md | ||
| rust-toolchain.toml | ||
| rustfmt.toml | ||
| SECURITY.md | ||
| typos.toml | ||
Native Rust implementation of Apache Arrow and Apache Parquet
Welcome to the Rust implementation of Apache Arrow, a popular in-memory columnar format and Apache Parquet, a popular columnar file format.
Community
We welcome participation from everyone and encourage you to join us, ask questions, help others, and get involved. All participation in the Apache Arrow project is governed by the Apache Software Foundation's code of conduct.
We use GitHub issues and pull requests for all technical discussions, reviews, new features, bug fixes and release coordination. This ensures that all communication is public and archived for future reference.
The dev@arrow.apache.org mailing list is the communication channel for the overall Apache Arrow community.
Instructions for signing up and links to the archives can be found on the Arrow Community page.
Some community members also use the Arrow Rust Discord Server and the official ASF Slack server for informal discussions and coordination.
This is a great place to meet other contributors and get guidance on where to contribute.
However, all technical designs should also be recorded and formalized in GitHub issues, so that they are accessible to everyone.
In Slack, find us in the #arrow-rust channel and feel free to ask for an invite via Discord, GitHub issues, or other means.
There is more information in the contributing guide and the security policy.
Repository Structure
This repository contains the following crates:
| Crate | Description | Latest API Docs | README |
|---|---|---|---|
arrow |
Core functionality (memory layout, arrays, low level computations) | docs.rs | (README) |
arrow-flight |
Support for Arrow-Flight IPC protocol | docs.rs | (README) |
parquet |
Support for the Apache Parquet columnar file format | docs.rs | (README) |
parquet_derive |
A crate for deriving RecordWriter/RecordReader for arbitrary, simple structs | docs.rs | (README) |
The current development version the API documentation can be found here.
Note: previously the object_store crate was also part of this repository,
but it has been moved to the arrow-rs-object-store repository
Release Versioning and Schedule
The Arrow Rust project releases approximately monthly and follows Semantic Versioning.
Due to available maintainer and testing bandwidth, arrow crates (arrow,
arrow-flight, etc.) are released on the same schedule with the same versions
as the parquet and parquet_derive crates.
This crate releases every month. We release new major versions (with potentially breaking API changes) at most once a quarter, and release incremental minor versions in the intervening months. See ticket #5368 for more details.
To keep our maintenance burden down, we do regularly scheduled releases (major
and minor) from the main branch. How we handle PRs with breaking API changes
is described in the contributing guide.
Planned Release Schedule
| Approximate Date | Version | Notes |
|---|---|---|
| October 2026 | 60.1.0 |
Minor, NO breaking API changes |
| November 2026 | 60.2.0 |
Minor, NO breaking API changes |
| December 2026 | 61.0.0 |
Major, potentially breaking API changes |
| January 2027 | 61.1.0 |
Minor, NO breaking API changes |
| February 2027 | 61.2.0 |
Minor, NO breaking API changes |
| March 2027 | 62.0.0 |
Major, potentially breaking API changes |
Rust Version Compatibility Policy
arrow-rs and parquet are built and tested with stable Rust, and will keep a rolling MSRV (minimum supported Rust version) that can only be updated in major releases on an as needed basis (e.g. project dependencies bump their MSRV or a particular Rust feature is useful for us etc.). The new MSRV if selected will be at least 6 months old. The minor releases are guaranteed to have the same MSRV.
Note: If a Rust hotfix is released for the current MSRV, the MSRV will be updated to the specific minor version that includes all applicable hotfixes preceding other policies.
Platform Support
Only little-endian platforms are officially supported and tested in CI. Big-endian platforms are not tested in CI and may not work correctly. Fixes for big-endian platforms are welcome and handled on a best-effort basis, but compatibility is not guaranteed.
Guidelines for panic vs Result
In general, use panics for bad states that are unreachable, unrecoverable or harmful.
For those caused by invalid user input, however, we prefer to report that invalidity
gracefully as an error result instead of panicking. In general, invalid input should result
in an Error as soon as possible. It is ok for code paths after validation to assume
validation has already occurred and panic if not. See ticket #6737 for more nuances.
Deprecation Guidelines
Minor releases may deprecate, but not remove APIs. Deprecating APIs allows downstream Rust programs to still compile, but generate compiler warnings. This gives downstream crates time to migrate prior to API removal.
To deprecate an API:
- Mark the API as deprecated using
#[deprecated]and specify the exact arrow-rs version in which it was deprecated - Concisely describe the preferred API to help the user transition
The deprecated version is the next version which will be released (please
consult the list above). To mark the API as deprecated, use the
#[deprecated(since = "...", note = "...")] attribute.
For example
#[deprecated(since = "51.0.0", note = "Use `date_part` instead")]
In general, deprecated APIs will remain in the codebase for at least two major releases after
they were deprecated (typically between 6 - 9 months later). For example, an API
deprecated in 51.3.0 can be removed in 54.0.0 (or later). Deprecated APIs
may be removed earlier or later than these guidelines at the discretion of the
maintainers.
Related Projects
There are several related crates in different repositories
| Crate | Description | Documentation |
|---|---|---|
object_store |
Object Storage (aws, azure, gcp, local, in-memory) interface | (README) |
datafusion |
In-memory query engine with SQL support | (README) |
ballista |
Distributed query execution | (README) |
parquet_opendal |
Use opendal for parquet Arrow IO |
(README) |
Collectively, these crates support a wider array of functionality for analytic computations in Rust.
For example, you can write SQL queries or a DataFrame (using the
datafusion crate) to read a parquet file (using the parquet crate),
evaluate it in-memory using Arrow's columnar format (using the arrow crate),
and send to another process (using the arrow-flight crate).
Generally speaking, the arrow crate offers functionality for using Arrow
arrays, and datafusion offers most operations typically found in SQL,
including joins and window functions.
You can find more details about each crate in their respective READMEs.