Skip to content

[Feature] Support Vortex file format #319

Description

@zhouyc-ali

Search before asking

  • I searched in the issues and found nothing similar.

Motivation

Apache Paimon Java supports Vortex through the paimon-vortex module, a Rust-based columnar file format with extensible compression encodings and lazily loaded summary statistics. Paimon C++ cannot currently access tables using this format.

Solution

Add Vortex support to the Paimon C++ file-format SPI:

  • Register the vortex file-format factory.
  • Implement compatible schema, reader, and writer handling.
  • Support column projection, predicate pushdown, and precise row selection.
  • Validate the same supported and unsupported logical types as Java.
  • Add C++ round-trip tests and Java-produced file compatibility tests.

Anything else?

Vortex is implemented in Rust, so the integration can follow the Mosaic Rust FFI patterns (#238).
Java reference module: https://github.com/apache/paimon/tree/master/paimon-vortex

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions