Watch

mirror of https://github.com/apache/datafusion-comet.git synced 2026-11-06 13:32:57 +00:00

Apache DataFusion Comet Spark Accelerator https://datafusion.apache.org/comet

arrow datafusion rust spark

Scala 49.3%
Rust 44.3%
Java 3.1%
Python 2.2%
Shell 0.5%
Other 0.5%

Find a file

Repository files (latest commit first)
Filename	Latest commit message	Latest commit date
Abhishek Mohan 5dcf745a8b docs: add a docs-review skill (#7153 ) * docs: add a documentation style guide Add docs/source/contributor-guide/style_guide.md, which writes down the vocabulary from #4419 (pipeline membership, implementation language and data format), the rule for "native", how to name operators in prose, and the page conventions that the docs-review skill in #4420 cites. Each rule has a stable ID so that reviews can cite it. The guide follows the code where #4419 and the code differ: "Hybrid" keeps the meaning it already has in expressions.md, expressions are described separately from plan nodes, and "Arrow-native" is a claim about data format only. The Generated Content section names the generated text exactly, by the markers GenerateDocs fills in, including the config tables built from QueryPlanSerde and contrib config classes, and the cast tables built from CometCast. P18 covers attribution of third-party content, which #4420 lists among its checks. The guide's scope covers .doc strings in contrib config classes, and its vocabulary and accuracy rules also apply to doc comments, AGENTS.md and agent skills. Link the page from the contributor guide toctree and from AGENTS.md. Part of #4419. Generated-by: OpenCode * docs: add a docs-review skill Add .ai/skills/review-comet-docs/SKILL.md, a sibling of review-comet-pr for documentation. It has three modes: PR mode reviews the docs a pull request changes, branch mode reviews a branch before it becomes a PR, and audit mode reviews an existing file or directory, including files not yet committed. The skill reads the documentation style guide on every run and cites its rule IDs rather than restating the rules. It finds the Apache remote and the PR's real base, reads everything from the reviewed revision, checks whether main has moved since, and checks only the added lines in PR and branch mode. Mechanical checks come first: vague "native" (including in Markdown italics), config keys (including keys built from a name at run time), operator names, links, anchors, images, external links, structure, redirects for moved pages, and version claims. Then it runs judgment checks, the most important being whether each claim, and each condition in it, still matches the code. Besides docs pages, it checks changed .doc strings, doc comments, AGENTS.md and agent skills. Findings are must-fix, should-fix, or nit, with nits dropped when the skill runs as part of a review-comet-pr review. It never edits files or posts comments. A short table of fixed cases says how to check the skill after changing it. In review-comet-pr, route docs changes to the new skill, mention documentation in its description, and remove "docs only" from the list of PRs it covers on its own. Fix two things in it that no longer match: `gh pr checks --failed` no longer exists, and the opt-in key is spark.comet.expression.<Name>.allowIncompatible. In review-comet-pr and review-comet-expression-pr, point at the style guide for exactly which docs text is generated. Closes #4420. Generated-by: OpenCode * docs: make review-comet-docs runs reproducible and check links with Sphinx - Pin the reviewed commit (HEAD_SHA) after checking it against headRefOid, and read everything at it, including the style guide. - List changed files with git diff instead of gh pr view --json files, which stops at 100 files. - Fail on a missing merge base (shallow clones) instead of diffing against an empty revision. - Check drift against the PR's own target branch, not always main. - Name the state file per run and quote its values, so concurrent reviews and paths with spaces work. - Use git grep -E with perl instead of git grep -P, so Git builds without PCRE work, and set core.quotePath=false for non-ASCII names. - Replace the hand-written link, anchor, toctree, and heading checks with a Sphinx build of the base and the head, compared by warning. It resolves the generated per-version pages the way docs/build.sh does, catches links that a renamed heading breaks on unchanged pages, handles reStructuredText, and resolves toctree entries properly. H1s are counted with markdown-it, which handles nested code fences. - Route AGENTS.md and agent skills to the skill from review-comet-pr. Part of #4420. Generated-by: OpenCode * docs: retry the merge base after deepening, sanitize the state file name, keep the N5 test case stable The merge-base step now tries again after deepening and stops with an error if there is still no base. Branch names with a slash no longer break the state file path. The N5 case moves to the scratch page, so it holds before and after #7151 rewrites understanding-comet-plans.md. Generated-by: OpenCode * docs: make the docs build and heading check fail loudly and run in audit mode build() now reports a failed Sphinx run instead of comparing empty warning lists, builds the working tree in audit mode, and uses a named virtual environment, as does the H1 check. Audit mode gets a config-key command, the stop rule no longer treats an empty search as an error, the generated-content check says what to compare, and the #6786 case reads the guide from main. Generated-by: OpenCode * docs: fix what seven agents hit when running the docs review skill Dry runs on Claude, GPT, Gemini and Grok models in zsh on macOS found these gaps. The venv now picks Python 3.10 or later and cleans up on failure, since the macOS system Python cannot install the docs requirements. Each call reloads the state with a given line, and a failed docs build no longer reads as a reason to stop. The guide falls back to the target when the head predates it. Audit mode reads the working tree for the H1 check and code reads, and skips the rename check. Code files are filtered to doc comments, SVG text is reviewed, protobuf message names are not operators, and the build output no longer carries the macOS /private prefix. Generated-by: OpenCode * docs: make the native rules testable N1 now covers every form of the banned phrases, such as ran natively and Comet-native. N2 allows the native shuffle path without pairing it with JVM shuffle. N3 replaces and similar with a test, names user-guide/, and says pages outside the contributor guide allow only the N2 compounds. Agents running the guide could not apply the old wording consistently. Generated-by: OpenCode (cherry picked from commit `80d4af0446`) * docs: allow Arrow-native in N2 and close the native rules T7 lists Arrow-native, which N3's limit to N2 compounds outside the contributor guide would otherwise forbid. N3 also says any other use of native is a finding, so a phrase outside the lists has an outcome. Generated-by: OpenCode (cherry picked from commit `ad78e9b928`) * docs: save the file list, read the guide from the working tree as a last resort The changed files that step 3 selects are now written to a file saved in the state, so later calls pass them to Git instead of rebuilding the list by hand. When neither the head nor the target has the guide, as for a PR made before it existed, the skill reads the working tree's copy and says so, instead of stopping. Audit mode leaves the revision out of the code searches, because an empty quoted revision makes Git fail. Generated-by: OpenCode * docs: count audit files for the cap, skip test sources, name the guide fallback Audit mode has no ADDED list, so the 10-file cap now counts the files step 1 lists. Test sources no longer enter the file list, since their string literals passed the doc-comment filter. The stop rule names the step 2 guide lookup as a fallback, and the key notes cover the EXPRNAME placeholder pages use. Generated-by: OpenCode * docs: limit the N2 scan compounds to Rust-implemented scans A dry run read native cache scan as allowed by N2, but the in-memory cache scan runs on the JVM. N2 now says the scan and writer compounds apply only to Rust-implemented ones. Generated-by: OpenCode (cherry picked from commit `e8b8000c98`) * docs: point N1 at T1 for not-a-fallback and at N5 for Spark A dry run rewrote "falls back to its native execution" and "executes natively (not falling back to Spark)" wrongly, because N1 offered only T1 or T3. N1 now says not falling back is T1, since codegen dispatch is not a fallback, and that Spark's own behavior is N5. Generated-by: OpenCode (cherry picked from commit `147f3f462a`) * docs: save the audit file list as PATHS Audit mode used an undefined <files> placeholder. It now saves the step 1 list as PATHS, as the other modes do, and says the mechanical checks cover every file while only the judgment checks are capped. Generated-by: OpenCode * docs: apply the style guide to new and changed text Existing pages move over as they are edited, so a review does not ask a change to restyle text it leaves alone, unless the change makes that text wrong (P16). Requested in review. Generated-by: OpenCode (cherry picked from commit `fdff0b18aa`) * docs: keep paths with spaces whole, build each revision with its own requirements The file list was passed as $(cat "$PATHS"), which splits a name such as "My Page.md" into two pathspecs, so the page was silently skipped. Commands now pass it through tr and xargs -0 -r. The docs build reused one environment installed from the checkout's requirements; a new venv function installs each revision's own docs/requirements.txt into an environment named after its hash, so a change that updates a Sphinx extension is built with it. Requested in review. Generated-by: OpenCode * docs: list audit files unquoted, note generated expression anchors Audit mode listed files with Git's default path quoting, so a page such as résumé.md became an escaped string and was skipped. The docs build copies only the expression templates, so a link to a heading that GenerateDocs adds to a per-version page warns although the published page has it; the skill now says to check the serde's notes instead. Requested in review. Generated-by: OpenCode * docs: print audit search results with unquoted file names The audit searches now pass core.quotePath=false too, so a finding on a page such as résumé.md names the file as written. Generated-by: OpenCode * docs: match the category page and Spark version for generated anchors A heading GenerateDocs adds appears only on the category page whose serde map holds the expression, at the Spark versions that register it. The manual check now confirms both, so a link to the same heading on another category page is reported as broken. Requested in review. Generated-by: OpenCode		2026-11-05 21:55:05 +00:00
.ai/skills	docs: add a docs-review skill (#7153 )	2026-11-05 21:55:05 +00:00
.claude
.github
.mvn
benchmarks
common
conf
contrib
dev
docs
kube
native
spark
spark-integration
.asf.yaml
.dockerignore
.gitignore
.prettierignore
.scalafix-syntactic.conf
.scalafix.conf
AGENTS.md
CHANGELOG.md
CLAUDE.md
CONTRIBUTING.md
LICENSE.txt
Makefile
mvnw
mvnw.cmd
NOTICE.txt
pom.xml
README.md
rust-toolchain.toml
scalafmt.conf

README.md

Apache DataFusion Comet

Apache DataFusion Comet is a high-performance accelerator for Apache Spark. Comet keeps Spark queries Arrow-native end-to-end: operators, expressions, shuffle, and broadcast all stay in Apache Arrow columnar format, avoiding the per-row overhead of Spark's row-based engine. Within the Arrow-native pipeline, operators and expressions execute as Rust code (via the Apache DataFusion query engine) or as JVM code that operates directly on Arrow batches. Comet integrates with the Spark ecosystem without requiring any code changes.

Comet provides a ~2x speedup over Spark 4.2 for TPC-DS @ SF 1000 (1TB), resulting in ~50% cost savings.

That speedup gives you a choice: finish the same Spark workload in half the time on the cluster you already have, or match your current Spark performance on half the resources. Either way, the gain translates directly into lower cloud bills, reduced on-prem capacity, and lower energy usage, with no changes to your existing Spark SQL, DataFrame, or PySpark code. Comet runs on commodity hardware: no GPUs, FPGAs, or other specialized accelerators are required, so the savings come from better utilization of the infrastructure you already run on.

See the Comet Benchmarking Guide for more details.

What Comet Accelerates

Comet accelerates Spark workloads by replacing Spark operators and expressions with high-performance implementations that process Apache Arrow columnar data directly. Most operators are powered by native Rust execution built on Apache DataFusion, while others run efficiently in the JVM on Arrow batches. This unified columnar execution model keeps processing within the Comet engine end-to-end, reducing overhead and delivering faster, more efficient query execution without reverting to Spark's traditional row-based engine.

Parquet scans: native Parquet reader integrated with Spark's query planner
Apache Iceberg: accelerated Parquet scans when reading Iceberg tables from Spark (see the Iceberg guide)
Shuffle: Arrow-IPC columnar shuffle with support for hash and range partitioning, in a native Rust implementation paired with a JVM fallback for unsupported partition key types
Expressions: hundreds of supported Spark expressions across math, string, datetime, array, map, JSON, hash, and predicate categories
Aggregations: hash aggregate with support for FILTER (WHERE ...) clauses
Joins: hash join, sort-merge join, and broadcast join
Scala/Java UDFs: support for keeping Scala/Java scalar UDFs in the Comet pipeline via Spark's whole-stage codegen (see the Scala UDF guide)

For the authoritative lists, see the supported expressions and supported operators pages.

Drop-In Integration

Comet is designed as a drop-in accelerator for Apache Spark, allowing you to integrate Comet into your existing Spark deployments and workflows seamlessly. With no code changes required, you can immediately harness the benefits of Comet's acceleration capabilities without disrupting your Spark applications.

Getting Started

Comet supports Apache Spark 3.4, 3.5, 4.0, 4.1, and 4.2. See the installation guide for the detailed version, Java, and Scala compatibility matrix.

Install Comet by adding the jar for your Spark and Scala version to the Spark classpath and enabling the plugin. A typical configuration looks like:

export COMET_JAR=/path/to/comet-spark-spark3.5_2.12-<version>.jar

$SPARK_HOME/bin/spark-shell \
    --jars $COMET_JAR \
    --conf spark.driver.extraClassPath=$COMET_JAR \
    --conf spark.executor.extraClassPath=$COMET_JAR \
    --conf spark.plugins=org.apache.spark.CometPlugin \
    --conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \
    --conf spark.comet.explain.fallback.enabled=true \
    --conf spark.memory.offHeap.enabled=true \
    --conf spark.memory.offHeap.size=4g \
    --conf spark.executor.memoryOverhead=2g

For full installation instructions, published jar downloads, and configuration reference, see the installation guide and the configuration reference.

Community

Join the DataFusion Slack and Discord channels to connect with other users, ask questions, and share your experiences with Comet.

Contributing

We welcome contributions from the community to help improve and enhance Apache DataFusion Comet. Whether it's fixing bugs, adding new features, writing documentation, or optimizing performance, your contributions are invaluable in shaping the future of Comet. Check out our contributor guide to get started.

License

Apache DataFusion Comet is licensed under the Apache License 2.0. See the LICENSE.txt file for details.