fix(read): support Parquet TIME columns - #378
Conversation
Parse TIME schemas as millisecond Arrow values and validate time32[ms]. Skip cast lookup for identical Arrow types so existing Parquet TIME columns can be read without losing milliseconds. Add parser, schema validation, field mapping and Parquet read regression tests. Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
|
Thank you for the contribution! Before reviewing this PR in detail, I’d like to confirm the intended scope of the implementation. Does this PR only support reading append tables? It seems primary key tables may require additional changes, for example around Also, although the |
@lxy-9602 Yes, this PR currently targets reading existing append-only Parquet tables. I should have made that clearer in the description. You’re right about PK tables. I checked the merge path, and I missed the serialization gap. I’d like to address that here and leave PK support for a separate PR. Does that scope work for you? For serialization, we’ll also need to decide how to handle precision, since the current mapping to |
Thanks for your reply! Yeah, I agreed that PK support should be a separate pr. I think C++ can follow the paimon-rust design: preserve |
Store declared TIME precision in Arrow field metadata and use it when serializing schemas. Validate precision metadata and default Arrow-only TIME fields to precision zero. Cover schema persistence and nested precision round trips. Verify Parquet TIME values across Java, Python and Rust fixtures and update ORC and Avro unsupported-type expectations. Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
|
Updated in c23c267. TIME precision is now preserved in Arrow field metadata and used for serialization, with |
Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
Accept Arrow fields in the TIME precision accessor and reject TIME partition keys during schema validation. Remove redundant read integration coverage and add partition validation tests. Signed-off-by: QuakeWang <wangfuzheng0814@foxmail.com>
Purpose
Closes #292.
TIME schemas were rejected, and field mapping required a logical cast type even for identical Arrow types. This prevented reading existing Parquet TIME columns.
Parse and validate TIME as time32[ms], matching Java Paimon, and skip cast lookup for identical types. Millisecond values and nulls are preserved, including PyPaimon's TIME(0).
Tests
API and Format
No public API or storage format changes. Scope is reading existing Parquet TIME columns.
Documentation
No documentation changes.
Generative AI tooling
Generated-by: OpenAI Codex (GPT-6)