Skip to content

fix(read): support Parquet TIME columns - #378

Merged
lxy-9602 merged 4 commits into
apache:mainfrom
QuakeWang:verify-time
Sep 22, 2026
Merged

lxy-9602 merged 4 commits into
apache:mainfrom
QuakeWang:verify-time

Conversation

@QuakeWang

Copy link
Copy Markdown
Member

Purpose

Closes #292.

TIME schemas were rejected, and field mapping required a logical cast type even for identical Arrow types. This prevented reading existing Parquet TIME columns.

Parse and validate TIME as time32[ms], matching Java Paimon, and skip cast lookup for identical types. Millisecond values and nulls are preserved, including PyPaimon's TIME(0).

Tests

  • Passed: 7 parser, 85 core, 69 Parquet reading tests, the new TIME integration test, and an actual PyPaimon 2.0.0 table scan/read.
  • Full-repository pre-commit and git diff --check passed.
  • macOS core/read tests required static relinking. Full common tests hit an existing compilation error; full Parquet tests aborted in a VECTOR test. Lance was not enabled.

API and Format

No public API or storage format changes. Scope is reading existing Parquet TIME columns.

Documentation

No documentation changes.

Generative AI tooling

Generated-by: OpenAI Codex (GPT-6)

Parse TIME schemas as millisecond Arrow values and validate time32[ms]. Skip cast lookup for identical Arrow types so existing Parquet TIME columns can be read without losing milliseconds.

Add parser, schema validation, field mapping and Parquet read regression tests.

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
@lxy-9602

Copy link
Copy Markdown
Member

Thank you for the contribution! Before reviewing this PR in detail, I’d like to confirm the intended scope of the implementation. Does this PR only support reading append tables? It seems primary key tables may require additional changes, for example around FieldsComparator.

Also, although the TIME schema can pass validation during creation, it currently cannot be serialized when create table— for example, calling ToJsonString() fails with an unknown type error.

@QuakeWang

Copy link
Copy Markdown
Member Author

Thank you for the contribution! Before reviewing this PR in detail, I’d like to confirm the intended scope of the implementation. Does this PR only support reading append tables? It seems primary key tables may require additional changes, for example around FieldsComparator.

Also, although the TIME schema can pass validation during creation, it currently cannot be serialized when create table— for example, calling ToJsonString() fails with an unknown type error.

@lxy-9602 Yes, this PR currently targets reading existing append-only Parquet tables. I should have made that clearer in the description. You’re right about PK tables.

I checked the merge path, and RowToArrowArrayConverter also needs TIME support, beyond the comparator changes.

I missed the serialization gap. I’d like to address that here and leave PK support for a separate PR. Does that scope work for you? For serialization, we’ll also need to decide how to handle precision, since the current mapping to time32[ms] loses the original TIME(p) precision.

@lxy-9602

lxy-9602 commented Sep 21, 2026

Copy link
Copy Markdown
Member

I missed the serialization gap. I’d like to address that here and leave PK support for a separate PR. Does that scope work for you? For serialization, we’ll also need to decide how to handle precision, since the current mapping to time32[ms] loses the original TIME(p) precision.

Thanks for your reply! Yeah, I agreed that PK support should be a separate pr. I think C++ can follow the paimon-rust design: preserve TIME(p) in the Paimon logical schema and treat Arrow time32[ms] only as the execution/storage representation. Serialization should use the preserved logical precision instead of inferring it from the Arrow type. If introducing a separate logical type model is out of scope for this PR, storing the original precision in Arrow field metadata could be a transitional solution. Does this look good to you?

Store declared TIME precision in Arrow field metadata and use it when serializing schemas. Validate precision metadata and default Arrow-only TIME fields to precision zero.

Cover schema persistence and nested precision round trips. Verify Parquet TIME values across Java, Python and Rust fixtures and update ORC and Avro unsupported-type expectations.

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
@QuakeWang

Copy link
Copy Markdown
Member Author

Updated in c23c267. TIME precision is now preserved in Arrow field metadata and used for serialization, with TIME(0) as the default when metadata is absent. Added round-trip tests and updated the compatibility tests. PK support will stay in a separate PR.

Comment thread src/paimon/common/types/data_type.h
Comment thread src/paimon/common/types/data_type.h Outdated
Comment thread src/paimon/core/schema/arrow_schema_validator.cpp Outdated
Comment thread test/inte/read_inte_test.cpp Outdated
Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
Accept Arrow fields in the TIME precision accessor and reject TIME partition keys during schema validation. Remove redundant read integration coverage and add partition validation tests.

Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>

@lxy-9602 lxy-9602 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@lxy-9602
lxy-9602 merged commit 09e7f75 into apache:main Sep 22, 2026
28 of 29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Support reading Paimon TIME columns

2 participants