feat(format): support Vortex file format by zhouyc-ali · Pull Request #376 · apache/paimon-cpp

zhouyc-ali · 2026-09-21T02:37:27Z

Purpose

Linked issue: close #319

Adds optional read and write support for the Apache Paimon Vortex file format, integrated through
the upstream Vortex Rust FFI and gated by PAIMON_ENABLE_VORTEX.

Add the Vortex file reader, writer, format factory, stats extractor, and Arrow C Data Interface
conversion. Vortex exports strings and binaries as Arrow view types; the reader normalizes them
back to the standard types declared in the Paimon schema, recursing through structs, lists and
fixed-size lists.
Pin the vendored Vortex to tag 0.75.0, which interoperates bidirectionally with paimon-java's
vortex-jni 0.73.0 (each implementation reads the other's files, verified end-to-end). Newer Vortex
(>= 0.77) emits encodings that vortex-jni 0.73.0 cannot read, breaking the C++ -> Java direction.
Add a callback-based IO bridge so Vortex reads and writes directly through paimon's
InputStream/OutputStream (works with any FileSystem: local, OSS, S3, Jindo), instead of
buffering whole files in memory on read or staging a local temporary file on write. The stock
Vortex C API cannot route IO through paimon's filesystem, so the bridge is added to vortex-ffi
as an injected Rust module (crates/vortex_callback_io/) plus a one-line patch
(cmake_modules/vortex.diff).
Support projection (including nested fields) and configurable output batch sizes; reset the reader
to the beginning when SetReadSchema is called.
Read the file row count for FileInfo; Vortex exposes no per-column statistics to Paimon
(consistent with the Java implementation).
Validate Vortex-incompatible schema types (MAP, MULTISET, VARIANT, BLOB).
Register Vortex in the file-format factory and bypass the generic PrefetchFileBatchReader
(Vortex performs random-access reads through its own callbacks).
Scan in storage order (ordered=true, matching paimon-java) so the physical row positions the
read path assigns by batch order stay aligned for deletion vectors and primary-key merge; rebase
sliced batches to offset 0 before handing them to Vortex, and copy the Arrow stream error message
before releasing the stream.
Enable Vortex in CI: add -DPAIMON_ENABLE_VORTEX=ON to build_paimon.sh, install the stable
Rust toolchain (Vortex MSRV 1.95), and build flatc 25.12.19 (ci/scripts/setup_flatc.sh), which
Vortex needs to compile its FlatBuffers schemas at build time.

Predicate/projection pushdown into Vortex is not implemented; all rows and columns are scanned and
projected to the read schema.

Tests

Clang Debug build with PAIMON_ENABLE_VORTEX=ON
full CTest suite: 33/33 targets passed
Vortex unit tests (paimon-vortex-format-test): 12/12 passed, including round-trips for
list<utf8>, fixed_size_list<utf8> and list<struct> (view normalization), nested-struct
projection (r.b from r:ROW<a,b>), row order across batches (ordered scan), and sliced
top-level struct write/read
Rust cargo test -p vortex-ffi callback_io: 3/3 passed
Java 0.73.0 -> C++ fixture read (TestVortexJavaCompatibility)
C++ 0.75.0 -> Java 0.73.0 direct read (verified out-of-band; the two directions are reciprocal)
clang-format and git diff --check clean; cpplint/cmake-format/rustfmt/pre-commit and Apache RAT
are enforced by CI

API and Format

Adds the optional Vortex file-format factory and the PAIMON_ENABLE_VORTEX CMake option. It reads
and writes the existing Apache Paimon Vortex file format through the upstream Vortex Rust FFI (pinned
to 0.75.0) and does not change the Vortex binary format or any existing public API under include/.
Building Vortex requires a Rust toolchain (>= 1.95) and flatc 25.12.19; both are set up in CI.

Documentation

Updated docs/source/user_guide/format_table.rst to list Vortex among the formats a format table
does not reach, and added crates/vortex_callback_io/README.md describing the injected module and
its build wiring.

Generative AI tooling

Generated-by: Qoder

Pin the vendored Vortex from commit d82de0e to tag 0.75.0 so paimon-cpp interoperates bidirectionally with paimon-java's vortex-jni 0.73.0: 0.75 reads 0.73-written files and, verified end-to-end, 0.73 reads 0.75-written files. Newer pins (>=0.77) break the cpp->java direction with "Encoding not found in registry". Adapt the FFI call sites to the 0.75 signatures (error message accessor, the arrow/dtype converters drop their session argument, VortexFile::data_source) and fix a latent ownership bug uncovered by the downgrade: at 0.75 vx_data_source_dtype returns a BORROWED pointer (arc_wrapper new_ref, no refcount bump), so it must not be released with vx_dtype_free. Freeing it spuriously decremented the data source's DType Arc, a use-after-free that surfaced later as a segfault in the scan path. The pointer is now treated as borrowed; it stays valid for the schema conversion, which the data source outlives.

…Array NormalizeViewArray only recursed through STRING_VIEW/BINARY_VIEW/STRUCT/LIST, so a FIXED_SIZE_LIST (or LARGE_LIST) with view-typed elements fell into the default branch and was returned unchanged. NormalizeViewType already normalizes those element types, so the read schema declared e.g. fixed_size_list<utf8> while the data stayed fixed_size_list<utf8view> (Vortex hardcodes Utf8 -> Utf8View on export). The ArrowSchema mismatch made callers reinterpret the view buffers as standard strings, silently corrupting data on read. Add LARGE_LIST and FIXED_SIZE_LIST branches symmetric to the LIST branch, including the unchanged-values fast path, reusing the source null bitmap, null count, offset, length and (for LARGE_LIST) value offsets. Covered by WriteThenReadListOfString, WriteThenReadFixedSizeListOfString and WriteThenReadListOfStruct, whose data includes nulls, empty strings and strings long enough to exceed the 12-byte inline view threshold.

Address four review findings plus a related write-path abort: - Scan in storage order. The scan used Vortex's default ordered=false, which may emit chunks out of order via buffer_unordered; NextBatch assigns physical row positions by batch order, so unordered chunks would misalign deletion vectors and primary-key merge. Set ScanOptions.ordered(true), matching paimon-java's VortexRecordsReader. - Recurse into nested fields when projecting to the read schema. ProjectToReadSchema only selected top-level columns, so reading a pruned nested field (e.g. r.b from r:ROW<a,b>) kept the full child under a pruned parent type, mismatching the exported Arrow schema and silently corrupting the output. Project struct/list/fixed-size-list children recursively. - Fix a use-after-free on the stream error path. get_last_error returns a pointer owned by the ArrowArrayStream; it was read after ReleaseStream() invalidated it. Copy the message before releasing. - Rebase sliced batches to offset 0 before vx_array_from_arrow. arrow-rs cannot import a sliced (offset > 0) top-level struct (the parent offset is re-applied to already-offset children, tripping an arrow-data slice assertion that aborts). Offset-0 batches pass through untouched, so the hot path is unchanged. - Attribute the adapted sink entry points in callback_io.rs to vortex-ffi's sink.rs (Apache-2.0, Copyright the Vortex contributors) per Apache-2.0 section 4. Adds regression tests: nested-struct projection, row order across batches, and sliced top-level struct write/read.

…x TSAN races The Debug/TSAN matrix jobs failed the check-clang-tidy gate and the TSAN race gate on the Vortex files: - clang-tidy (modernize-use-using / modernize-use-auto): convert the hand-written C-ABI typedefs in vortex_ffi.h to `using` aliases / plain structs, and use `auto` for the cast-initialized locals in vortex_io_callbacks.cpp and vortex_stats_extractor.cpp. - TSAN: Vortex deserializes its footer/dtype and drives segment reads from internal runtime threads inside the un-instrumented Rust staticlib, so TSAN cannot see the happens-before edges and reports false races whose only non-paimon frames are the vortex_array/vortex_error/vortex_file/vortex_layout/vortex_mask Rust modules. Suppress those module-scoped patterns (they match Rust frames only, so races confined to paimon C++ are still reported), following the existing vortex_buffer/vortex_io/CallbackReadAt precedent.

…validation, CI deps) - Serialize host OutputStream access between the Vortex writer task and a caller-thread Flush(): VortexOutputContext gains a stream mutex and a FlushStream() entry, and VortexFormatWriter::Flush() plus the write/flush callbacks all take it, so Write and Flush can no longer interleave on a stream with no concurrent-access contract. The remaining internally buffered data is drained when the sink closes in Finish(). - Thread the caller's Arrow memory pool into VortexFormatWriter (Create now takes it and the builder passes GetArrowPool(pool_)), so sliced-batch rebasing allocates from the Paimon pool instead of Arrow's default pool. - Validate Vortex-incompatible schema types wherever Vortex can be selected (default file format, file.format-per-level, changelog-file.format), mirroring the Lance validation, instead of only the default format. - setup_flatc.sh: install libclang before the flatc early-exit, so an environment that already has a matching flatc but lacks libclang no longer silently skips the bindgen build dependency. - Tests: FlushBetweenBatchesThenRead (flush interleaved with background writes) and SchemaValidationTest.TestVortexDataTypes (per-level/changelog validation).

…Pool fwd decl The Debug build's check-clang-tidy gate analyses every changed translation unit. Any TU that includes paimon/type_fwd.h and also uses arrow::MemoryPool (as the Vortex writer now does for its rebasing pool) trips bugprone-forward-declaration-namespace on the paimon::MemoryPool forward declaration, even though paimon::MemoryPool is a real public type defined in paimon/memory/. Annotate the forward declaration with NOLINT plus a comment explaining that it is intentional.

lxy-9602 · 2026-09-22T09:07:22Z

    if (normalized == "csv" || normalized == "text" || normalized == "json" ||
-        normalized == "mosaic") {
+        normalized == "mosaic" || normalized == "vortex") {
        return Status::NotImplemented(


Can we double-check if Lance supports this? If it doesn't, we should add this NotImplemented here.

lxy-9602 · 2026-09-22T10:37:51Z

+// offset-normalized (offset 0) so the rebuilt parents and children stay consistent.
+Result<std::shared_ptr<arrow::Array>> NormalizeViewArray(const std::shared_ptr<arrow::Array>& array,
+                                                         arrow::MemoryPool* pool) {
+    switch (array->type_id()) {


Please add a TODO here to track this potential performance hotspot for future optimization.

lxy-9602 · 2026-09-22T10:45:15Z

+            return std::make_shared<arrow::LargeListArray>(
+                arrow::large_list(array->type()->field(0)->WithType(values->type())), list.length(),
+                list.value_offsets(), values, list.null_bitmap(), list.null_count(), list.offset());
+        }


It seems that LARGE_LIST is not supported?

lxy-9602 · 2026-09-22T12:54:35Z

+      total_rows_(total_rows),
+      pool_(pool),
+      arrow_pool_(arrow_pool),
+      metrics_(std::make_shared<MetricsImpl>()) {}


I'm curious about the reason for the Paimon pool here. Looks like this pool is unused?

lxy-9602 · 2026-09-22T13:16:10Z

+// already matches. Leaf types are expected to match after view normalization; an unexpected leaf
+// mismatch is returned as-is for the caller's schema check to surface.
+Result<std::shared_ptr<arrow::Array>> ProjectArrayToType(
+    const std::shared_ptr<arrow::Array>& array,


This function can be replaced with NestedProjectionUtils::AlignArrayToReadType. Also, does Vortex not support column pruning? Do we need to read everything back and then prune it?

If you plan to support this in the next PR, please add a TODO to indicate that this issue is known and will be addressed later. If not supoort, there’s no need to add this kind of fallback; we can simply avoid testing scenarios like nested subcolumn pushdown.

lxy-9602 · 2026-09-23T01:54:51Z

+    // paimon-cpp reads back a Vortex table produced by the Java implementation.
+    arrow::FieldVector fields = {
+        arrow::field("_VALUE_KIND", arrow::int8()),
+        arrow::field("id", arrow::int32()),


Please add complete compatibility tests for Vortex supporting all field types, similar to test_data/mosaic.

lxy-9602 · 2026-09-23T01:56:37Z

+(BatchWriteBuilder / BatchTableWrite / BatchTableCommit) on JDK 17 with
+--add-opens=java.base/java.nio=ALL-UNNAMED (required by Arrow Java memory).
+Read back and verified by paimon-cpp in
+ScanAndReadInteTest.TestVortexJavaCompatibility.


Please remove this. This test data may be used by other tests in the future, so it shouldn’t be coupled to TestVortexJavaCompatibility.

lxy-9602 · 2026-09-23T01:59:08Z

+    };
+    std::shared_ptr<arrow::DataType> data_type = arrow::struct_(fields);
+    std::shared_ptr<arrow::Array> expected_array =
+        arrow::ipc::internal::json::ArrayFromJSON(data_type, R"([


Also, I see write support has already been added. Are you planning to leave the combined read/write TEST_P cases from the other integration tests for a follow-up commit?

lxy-9602 · 2026-09-23T02:26:05Z

+            CHECK_HOOK_STATUS(batch.status(), i);
+            eof = BatchReader::IsEofBatch(batch.value());
+        }
+        run_complete = true;


If the read eventually succeeds, please verify that the final read result matches the expectation. You can use ReadResultCollector for this.

lxy-9602 · 2026-09-23T02:27:25Z

+        run_complete = true;
+        break;
+    }
+    ASSERT_TRUE(run_complete);


Similarly, when the write eventually succeeds, please also verify that the written data can be read back and matches the expectation.

zhouyc-ali force-pushed the feat/vortex-file-format branch 2 times, most recently from eecbd85 to ab83648 Compare September 22, 2026 08:03

zhouyc-ali added 8 commits September 22, 2026 09:56

feat(format): support Vortex file format

09bb73b

fix(ci): install libclang so Vortex bindgen builds succeed in CI

1f0a62c

fix(ci): suppress un-instrumented Vortex Rust artifacts under LSAN/TSAN

2c20c46

zhouyc-ali force-pushed the feat/vortex-file-format branch from 36b8af1 to 0d74468 Compare September 22, 2026 15:09

lxy-9602 reviewed Sep 23, 2026

View reviewed changes

Navigation Menu

feat(format): support Vortex file format - #376

feat(format): support Vortex file format#376
zhouyc-ali wants to merge 9 commits into
apache:mainfrom
zhouyc-ali:feat/vortex-file-format

zhouyc-ali commented Sep 21, 2026 •

edited

Loading

Uh oh!

lxy-9602 Sep 22, 2026

Uh oh!

lxy-9602 Sep 22, 2026

Uh oh!

lxy-9602 Sep 22, 2026

Uh oh!

lxy-9602 Sep 22, 2026

Uh oh!

lxy-9602 Sep 22, 2026

Uh oh!

lxy-9602 Sep 22, 2026

Uh oh!

lxy-9602 Sep 23, 2026

Uh oh!

lxy-9602 Sep 23, 2026

Uh oh!

lxy-9602 Sep 23, 2026

Uh oh!

lxy-9602 Sep 23, 2026

Uh oh!

lxy-9602 Sep 23, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

zhouyc-ali commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Tests

API and Format

Documentation

Generative AI tooling

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zhouyc-ali commented Sep 21, 2026 •

edited

Loading