Skip to content

fix(serde): align DataFile JSON with Java core - #975

Open
kamcheungting-db wants to merge 5 commits into
apache:mainfrom
kamcheungting-db:datafile-serde-java
Open

kamcheungting-db wants to merge 5 commits into
apache:mainfrom
kamcheungting-db:datafile-serde-java

Conversation

@kamcheungting-db

Copy link
Copy Markdown
Contributor

Summary

Make core DataFile JSON compatible with Java Iceberg.

This PR only changes DataFile JSON behavior. Core FileScanTask serde is added separately in #939.

Changes with examples

Partition data

Continue accepting positional arrays:

"partition": [7]

Also accept Java field-ID objects, which do not depend on field order:

"partition": {"1000": 7}

Binary metadata

Write binary values as Java-compatible hexadecimal strings:

"key-metadata": "0A0B"
"lower-bounds": {"keys": [1], "values": ["01000000"]}

Previously these values were written as JSON byte arrays.

Content names

Accept both current lowercase values and older Java enum-style values:

"content": "position-deletes"
"content": "POSITION_DELETES"

Integers

Reject values that are not actual, in-range JSON integers. For example:

"record-count": 1.5

Previously generic JSON conversion could coerce this to 1.

The parser also accepts a missing partition, accepts null metric maps, rejects duplicate metric keys, and validates partition value types during serialization.

Stack

This is 2 of 4 and depends on #974:

  1. Move DataFile serde into core (refactor(serde): move DataFile JSON serde into core #974)
  2. This PR: Java-compatible DataFile JSON
  3. Add core FileScanTask serde (feat(serde): add core FileScanTask JSON support #939)
  4. Validate schema and partition IDs strictly (fix(serde): validate schema and partition IDs as JSON integers #940)

Each PR targets main; later PRs contain the commits from earlier PRs.

Testing

  • DataFile focused tests: 23 passed across core and REST.
  • Full stack: 142 JSON serde tests passed.
  • Full stack: 332 REST tests passed; 1 existing test skipped.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant