0.91.0¶
Released 2026-11-04 — GitHub Release
Changes¶
⚠️ Breaks¶
Add shared conversion to canonical decimal arrays or Widen (#11005) @connortsui20
Reject nullable nested levels in list-based spatial types (#11183) @connortsui20
Preserve extension literals in DataFusion comparisons (#11067) @connortsui20
Require matching dtypes for extension comparisons (#11016) @connortsui20
Change extension dtype nullability without validating again (#11012) @connortsui20
Separate the read and write IDs of array plugins (#11057) @connortsui20
Vortex spark integration is a fileformat not a tableprovider (#9658) @robert3005
Return errors for invalid temporal types and values (#11015) @connortsui20
DictStrategyprobes a fixed row sample instead of the first chunk (#10927) @mhk197Layout plugins read and write more than one wire ID (#10704) @mhk197
Make uncompressed_size_in_bytes always exact and drop its options (#10938) @joseph-isaacs
Add a canonical decimal width and import decimals at it (#10897) @connortsui20
Make protobuf metadata fields crate-private (#10831) @connortsui20
Decode stored DecimalType and PType values strictly (#10755) @connortsui20
Return a borrowed scalar from as_constant (#10790) @robert3005
Simplify error types, remove kind from struct (#10080) @robert3005
perf(cuda): migrate consumers to vortex-cuda-driver (#10722) @0ax1
✨ Features¶
feat(array): migrate simple encodings to scalar probes (#10859) @jiengup
Explorer: draw it with the Vortex design system, and build it with Bun (#11193) @lwwmanning
Replace layout lowering with a VTable::new_plan method (#11200) @joseph-isaacs
Expose the existing footer segment end offset (#11196) @brancz
test: remove alp tests which are covered by other tests (#11155) @myrrc
test: remove vortex-duckdb tests which are covered by other tests (#11153) @myrrc
Support decimal values in Widen (#10946) @connortsui20
Add ScalarFnVTable::execute_step for iterative scalar fn execution (#11100) @joseph-isaacs
Read Vortex file column statistics through vortex-jni (#10863) @robert3005
Select editions from JNI and Python, and write files for readers before vortex.zoned (#10805) @robert3005
fix(python): read pyarrow files positionally in open_readable (#11074) @jackylee-ch
feat(spark): push nested struct pruning down to the native scan (#10957) @ChristopherWirt
Record the uncompressed size in file stats when editions allow it (#10923) @joseph-isaacs
Add uncompressed_size_in_bytes to the preview edition (#10922) @joseph-isaacs
Execute interleaved decimal arrays (#10940) @connortsui20
Add a scan for the range of valid decimal values (#10939) @connortsui20
Add an in-memory
Widenencoding for integers (#10864) @connortsui20Add a naive
Unaryencoding in a newvortex-elias-fanocrate (#10609) @rapourduckdb: support windows-x86 and windows-arm64 (#9570) @myrrc
string-bench: benchmark onpair-16 and add the OnPair paper datasets (#10920) @joseph-isaacs
Push DataFusion dynamic filters through Scan API expression trees (#10895) @gatesn
Skip file filters proven true until their dynamic producer changes (#10894) @gatesn
Derive live statistics proofs for dynamic expressions (#10869) @gatesn
Add generic dynamic expressions with generation tracking (#10865) @gatesn
feat(fsst): Compute uncompressed size without decoding (#9755) @lwwmanning
Rebuild lists over only the elements valid lists reference (#10743) @mhk197
Add an unstable edition that includes all unreleased and upcoming encodings (#10558) @robert3005
Drive V2 scans in the DuckDB extension with a custom driver (#10809) @joseph-isaacs
Pass VortexSession to plan scan and optimizer (#10794) @myrrc
duckdb: export DECIMAL and TIME_NS min/max statistics (#10635) @jackylee-ch
feat(cuda): track device memory accesses across streams (#10721) @0ax1
feat(cuda): add driver context and resource ownership (#10720) @0ax1
Add Eval(Eval()) and Eval(pack()) reduce rules (#10732) @myrrc
Add lambdas and variables for higher-order scalar functions (#10502) @mhk197
🚀 Performance¶
Keep same-scale decimal widening casts lazy (#11006) @connortsui20
perf(spark): split large files into row ranges (#11118) @gitRasheed
perf(fastlanes): push FoR ordering comparisons to the encoded values (#10147) @robert3005
Count each run’s selected rows with one pass over the RunEnd filter mask (#11221) @joseph-isaacs
Drop null patch elements when sparse lists share their elements (#11182) @mhk197
Faster Spark reads and smaller JNI and stats fixes (#11109) @robert3005
Optimize arrays top down as execution reaches them, and retry a parent when its children change (#11090) @robert3005
refactor(cuda): avoid access tracking for address-only lookups (#10884) @0ax1
Store plan children in fixed-position slots (#11178) @joseph-isaacs
Push decimal aggregates through Widen (#11004) @connortsui20
Reuse compact elements when canonicalizing sparse lists (#11173) @gatesn
perf(spark): append rows straight into the Arrow batch (#11152) @gitRasheed
Export decimal Widen arrays to Arrow with one copy (#10994) @connortsui20
Push down list element casts and bool/int/uint casts to Float64 in DataFusion (#11161) @mhk197
Push decimal comparisons, between, and fills through Widen (#11003) @connortsui20
perf(decimal-byte-parts): compare wide decimals through their parts (#10146) @robert3005
perf(alp): push nullable and patched comparisons to the encoded ints (#10141) @robert3005
perf(sequence): answer ordering and inequality comparisons directly (#10144) @robert3005
Record whether an array has been optimized (#11089) @robert3005
Default duckdb-bench to the V2 scan for TPC-DS and ClickBench (#11047) @joseph-isaacs
Speed up the decimal widening copy (#10989) @connortsui20
Speed up V2 scans and default datafusion-bench to V2 for TPC-DS and ClickBench (#10799) @joseph-isaacs
Separate RowFn batch inspection from execution planning (#10899) @connortsui20
Push slices through filtered arrays (#10959) @joseph-isaacs
Avoid a per-run
zip_eqcall in run-end bool decoding (#10962) @connortsui20Execute Delta iteratively (#10208) @joseph-isaacs
perf(bytebool): take through the fallible lane kernels (#10143) @robert3005
perf: transfer statistics to executed arrays without cloning the set (#10241) @joseph-isaacs
Run
Widenarithmetic, cast, fill null, zip and between on the values (#10891) @connortsui20Compute
Widenaggregates on the values (#10890) @connortsui20For decompression works on owned arrays (#10256) @robert3005
Reduce SequenceId universe lock contention (#10867) @robert3005
fix: Convert V2 scan batches to Arrow on the driver’s tasks (#10861) @joseph-isaacs
Use NEON packing for wide primitive array comparisons (#10843) @connortsui20
Dispatch structural kernels by physical width (#10841) @connortsui20
perf: skip parent kernel lookups for encodings that register none (#10239) @joseph-isaacs
Compose and drive a V2 scan in DataFusion (#10708) @joseph-isaacs
Widen dictionary codes in Dict::fill_null (#10806) @ump45nose
Attach RowFn validity directly to canonical output (#10742) @connortsui20
fix: bit-packed arrays cast match more often (#10803) @joseph-isaacs
Plan a V2 scan as a chain of planners from one prepared scan (#10707) @joseph-isaacs
Pack float32 comparison results into bitmap words (#10741) @connortsui20
perf(cuda): release closed mappings off the decode thread (#10718) @0ax1
perf(cuda): reuse pinned buffer completion events (#10713) @0ax1
perf(cuda): copy cached file pages directly on the GPU (#10716) @0ax1
perf(cuda): disable timing on pinned transfer events (#10714) @0ax1
perf(cuda): read pooled files in 8 MiB chunks (#10715) @0ax1
Compare timestamp parts without evaluating each comparison twice (#10684) @robert3005
Optimise take on dictionaries with constant codes (#10643) @robert3005
Add Filter/Split/Project planner and add Datafusion usage of this (#10706) @joseph-isaacs
🐛 Bug Fixes¶
62 changes
Peel Arrow dictionaries before Arrow importer plugins run (#11185) @connortsui20
Display dates beyond jiff’s range without panicking (#11187) @connortsui20
Return an error for decimal scalars outside the precision (#10767) @ump45nose
fix(array): accept timestamps beyond jiff’s range (#10953) @gitRasheed
fix: preserve sliced patch offsets during deserialization (#11168) @blaginin
Fix dictionary code width for nullable primitive values (#11165) @blaginin
Avoid rebuilding constant null and empty lists for Arrow export (#11172) @gatesn
Reduce constant projections through select and extension storage (#11056) @gatesn
fix(layout): rebase sliced list chunks before writing the list layout (#11082) @gitRasheed
fix(cuda): report device errors to stream callback waiters (#10876) @0ax1
fix(cuda): reject an empty stream pool when it is created (#10875) @0ax1
fix(cuda): make loading a module image unsafe (#10874) @0ax1
fix(cuda): report missing driver library as an error (#10873) @0ax1
fix(cuda): do not leak contexts when construction fails (#10872) @0ax1
fix(cuda): harden driver error handling, access tracking and host copies (#10871) @0ax1
fix(jni): retain Java Arrow buffers independently (#11069) @ChristopherWirt
duckdb: report distinct count based on min/max (#11130) @myrrc
fix(duckdb): handle table filters in aggregate pushdown (#11071) @jackylee-ch
fix(duckdb): escape LIKE metacharacters in pushed string filters (#11072) @jackylee-ch
fix(jni): attach native worker threads to the JVM as daemons (#11085) @gitRasheed
Validate extension storage and return scalar read errors (#11014) @connortsui20
fix(python): keep the scheme of object store paths in the Ray datasource (#11046) @jackylee-ch
fix(spark): resolve pruned nested fields against each file’s struct (#11083) @gitRasheed
fix(spark): let abort close every writer after a failed write (#11084) @gitRasheed
fix(jni): release the schema export in VortexWriter.build() (#11081) @gitRasheed
fix(python): report the scan’s row count and columns from the Ray datasource (#11070) @jackylee-ch
fix(spark): read every column as nullable (#11020) @ChristopherWirt
Clarify DCO sign-off requirements and agent attribution (#11001) @joseph-isaacs
fix(duckdb): don’t drop or reject filters on function arguments Vortex can’t convert (#10958) @jackylee-ch
Update libz-sys to 1.1.31 to fix RUSTSEC-2026-0418 (#10963) @robert3005
fix(alp): bounds-check ALP-RD stored metadata and left-parts codes (#10760) @devtechedge
fix: add explore’s CSP and fix docs search and previews (#10951) @lwwmanning
Build precision 19 decimal scalars at
i128(#10903) @connortsui20Fix exact/inexact mixing in uncompressed_size_in_bytes (#10925) @joseph-isaacs
fix: keep nullability when short-circuiting AND/OR against a constant (#10896) @robert3005
feat(python): fail soft on unsupported Polars predicates in to_polars (#10414) @robert3005
perf: collapse single-value partitions instead of a direct struct projection path (#10536) @robert3005
Expose DataFusion output partitions and proven ordering (#10901) @gatesn
Honour NaN options in min/max over float-backed extension types (#10851) @robert3005
fix: preserve strict sortedness for singleton null batches (#10889) @connortsui20
fix: keep empty encoded batches neutral in IsConstant (#10893) @connortsui20
Speed up file writes: parallel file stats, single min/max pass (#10856) @robert3005
perf(layout): prune multi-field boolean predicates in the struct reader (#10689) @MichaHoffmann
Compute legacy file stats from the aggregate stats accumulators (#10846) @robert3005
Preserve ordered Scan API partitions in the v2 DataFusion adapter (#10062) @vortex-claude[bot]
DataFusion v2 expression converter injection (#10836) @gatesn
Reject raw serialization of sliced PCO arrays (#10834) @danking
fix(array): between over a constant null array is null, not false (#10744) @gitRasheed
fix(spark): support Spark 4.2 in the current connector (#10792) @ChristopherWirt
Avoid truncating dictionary fill codes (#10745) @connortsui20
fix: preserve codes nullability when slicing dict with a constant code (#10772) @robert3005
FoRScheme: declare and refine its bit-packed child (#10757) @mhk197fix(cuda): retain pinned sources across transfer failures (#10712) @0ax1
Dispatch count_ones using fearless_simd (#10683) @robert3005
fix(cuda): bound concurrent stream-pool initialization (#10711) @0ax1
fix: write empty nested list streams with structural layouts (#10533) @danking
Propagate child writer errors from the struct writer (#10710) @robert3005
fix(spark): return the requested field when Spark prunes nested fields (#10724) @gitRasheed
📖 Documentation¶
8 changes
docs(arrow): explain why dictionary imports need special handling (#11233) @connortsui20
docs llms.txt links to the benchmark and homepage llms.txt files (#11177) @lwwmanning
docs: add robots.txt, a sitemap, llms.txt, and a favicon (#11166) @lwwmanning
Document temporal storage and binary comparison execution (#11099) @connortsui20
Document the nullability contract of cast rules (#10798) @connortsui20
Describe scheme variants in the versioning docs (#10747) @connortsui20
🧰 Maintenance¶
78 changes
Avoid allocating Vec of chunks just to copy it in ChunkedArray constructor (#9922) @robert3005
Execute filter scalar fns’ inputs iteratively with execute_step (#11154) @joseph-isaacs
test(cuda): encode ALP test inputs with the test’s own context (#10881) @0ax1
refactor(cuda): slice mutable views through one implementation (#10880) @0ax1
Benchmark sparse-list canonicalization with a null fill (#11175) @gatesn
Duplicate handling in
Mask::from_excluded_indices(#11164) @blagininDispatch zip kernels on either value branch (#11138) @blaginin
bench: add sparse-wide datasets to compression suite (#11150) @gatesn
refactor(cuda): keep inflight pinned buffers unwrapped (#10878) @0ax1
Lock file maintenance (#11080) @renovate[bot]
Lock file maintenance (#11079) @renovate[bot]
Lock file maintenance (#11078) @renovate[bot]
Run the remaining benchmarks on the V2 scan (#11061) @joseph-isaacs
Add execution path consistency assertions (#11013) @connortsui20
Run the FineWeb benchmark on the V2 scan (#11055) @joseph-isaacs
Fix benchmark runner labels exceeding 255 character limit (#11054) @joseph-isaacs
Optimize VCF parsing with concurrent range requests and parallel row group processing (#10984) @joseph-isaacs
Ingest string-bench results into the compression tables (#11044) @joseph-isaacs
Run CUDA Rust and Python tests in parallel steps (#11039) @joseph-isaacs
chore: let Renovate propose minor and patch updates to GitHub Actions (#11041) @lwwmanning
chore: let Renovate propose grouped major updates for vortex-web (#11042) @lwwmanning
fix(bench): retry the post-ingest database write across RDS restarts (#11018) @lwwmanning
Fix Arrow export of constant null variants (#10973) @blaginin
Lock file maintenance (#11011) @renovate[bot]
Write the list layout only for statpopgen and polarsignals benchmark data (#11035) @mhk197
Lock file maintenance (#11010) @renovate[bot]
Lock file maintenance (#11009) @renovate[bot]
Enable the list layout in benchmarks through a bench-only edition (#11030) @mhk197
ci: cargo hack runs only check and not build (#10845) @robert3005
chore: wait 7 days for new releases and fix Renovate’s lock file PRs (#10997) @lwwmanning
chore: refresh lockfiles to clear Dependabot alerts (#10999) @lwwmanning
Benchmark the copy of decimal values into the canonical width (#10990) @connortsui20
Resolve three small
TODO(connor)items (#10974) @connortsui20Add a toggle for writing nested file statistics (#10972) @robert3005
Extend the RunEnd filter benchmark (#10978) @robert3005
Update dependency filelock to v4 [SECURITY] (#10955) @renovate[bot]
vortex-bench: run ClickBench on its first N files (#10936) @mprammer
bench: name the data directory when capturing file sizes (#10935) @mprammer
chore: remove the Cloudflare Pages deploys and update bench paths (#10913) @lwwmanning
Remove repeated elements from IN lists (#10685) @robert3005
Add compute benchmarks for
Widen(#10888) @connortsui20Allocate more kernel outputs through the session allocator (#10622) @robert3005
string-bench: run the new inputs in CI (#10924) @joseph-isaacs
Enable preview and unstable editions in the fuzzer (#10840) @robert3005
Upgrade fsst-rs to 0.7.0 and onpair to 0.3.0 (#10909) @robert3005
Update dependency com.google.guava:guava to v33.7.2-android [SECURITY] (#10870) @renovate[bot]
Lock file maintenance (#10788) @renovate[bot]
Skip codegen of unreachable type dispatch arms (#10740) @robert3005
Drop direct arrow-select dependency (#10826) @robert3005
Remove unused dictionary layout metadata (#10823) @connortsui20
Pin cc to 1.8.0 to fix the Windows CI build (#10815) @robert3005
Widen list take piece bounds to u64 before dispatch (#10738) @robert3005
Share FastLanes compare kernels between operators and signedness (#10739) @robert3005
Build arrays through non-generic constructors (#10737) @robert3005
Dispatch filter take of fixed-width values on byte width (#10736) @robert3005
Skip codegen of unreachable int to decimal carrier arms (#10735) @robert3005
Update Cargo.lock to pick up security fixes (#10810) @robert3005
Add decimal benchmarks for DecimalByteParts, casts and exports (#10752) @connortsui20
Add compat fixtures for old decimal forms (#10753) @connortsui20
ci: disable CodSpeed process isolation for Rust simulation benchmarks (#10796) @robert3005
Consolidate python script, formatting and c++ lint tests into one job (#10729) @robert3005
chore(deps): update rui314/setup-mold digest to 339e96b (#10778) @renovate[bot]
chore(deps): update swatinem/rust-cache digest to d1527c8 (#10779) @renovate[bot]
chore(deps): update pypa/gh-action-pypi-publish digest to 7245906 (#10777) @renovate[bot]
chore(deps): update taiki-e/install-action digest to 1e29709 (#10780) @renovate[bot]
chore(deps): update dtolnay/rust-toolchain digest to b725eda (#10776) @renovate[bot]
chore(deps): update rust crate criterion to v6 (#10784) @renovate[bot]
chore(deps): update codspeedhq/action digest to 78102a4 (#10775) @renovate[bot]
chore(deps): update actions/setup-python digest to 6d4fd22 (#10774) @renovate[bot]
chore(deps): update actions/setup-java digest to b843ebf (#10773) @renovate[bot]
chore(deps): update rust crate divan to v6 (#10785) @renovate[bot]
Report one-row Sequence arrays as sorted for any multiplier (#10602) @ump45nose
Move expression analysis, rewrites and statistics pruning into vortex-expr (#10734) @robert3005
Remove array_session (#10681) @robert3005
ci: cache Python wheel compilation on prebuilt runners (#10727) @robert3005
ci: use the SQL fixture profile for Vortex conversion (#10728) @robert3005
Parallelize TPC-DS generation within each table (#10692) @robert3005