Skip to content

feat(serde): support standalone FileScanTask JSON in core - #939

Open
kamcheungting-db wants to merge 9 commits into
apache:mainfrom
kamcheungting-db:filescan-task-serde
Open

kamcheungting-db wants to merge 9 commits into
apache:mainfrom
kamcheungting-db:filescan-task-serde

Conversation

@kamcheungting-db

@kamcheungting-db kamcheungting-db commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add standalone core JSON serialization and deserialization for FileScanTask.

Unlike REST scan responses, core task JSON is self-contained: it embeds the schema, partition spec, data file, delete files, range, and residual filter.

Example

{
  "task-type": "file-scan-task",
  "schema": {"type": "struct", "schema-id": 0, "fields": []},
  "spec": {"spec-id": 0, "fields": []},
  "data-file": {
    "spec-id": 0,
    "content": "data",
    "file-path": "/data.parquet",
    "file-format": "parquet",
    "partition": [],
    "record-count": 10,
    "file-size-in-bytes": 100
  },
  "start": 0,
  "length": 100,
  "delete-files": [],
  "residual-filter": true
}

Core versus REST delete files

Core JSON inlines delete files:

"delete-files": [{"content": "position-deletes", "file-path": "/delete.parquet"}]

REST may instead refer to a sibling response array:

"delete-file-references": [0]

Those references only make sense while parsing the complete REST response, so core serde rejects them and leaves them in the REST-specific path.

Split handling

C++ does not currently support split FileScanTasks. This PR accepts only the whole file:

"start": 0,
"length": 100

For a 100-byte file, values such as "start": 10 or "length": 90 are rejected instead of silently becoming a whole-file task. The REST parser now performs the same check.

The implementation also validates data/delete file roles, null delete entries, task type, and required fields. A missing residual filter defaults to alwaysTrue.

Stack

This is 3 of 4 and depends on #975:

  1. Move DataFile serde into core (refactor(serde): move DataFile JSON handling from REST to core #974)
  2. Java-compatible DataFile JSON (fix(serde): make DataFile JSON compatible with Java Iceberg #975)
  3. This PR: core FileScanTask JSON serde
  4. Strict schema and partition ID parsing (fix(serde): reject invalid schema and partition IDs in JSON #940)

Each PR targets main; later PRs contain the commits from earlier PRs.

Testing

  • 21 core FileScanTask tests passed.
  • Focused core and REST tests: 51 tests run, 50 passed, 1 existing row-lineage test skipped.
  • Full stack: 142 JSON serde tests passed.
  • Full stack: 332 REST tests passed; 1 existing test skipped.

@kamcheungting-db kamcheungting-db changed the title fix(serde): align FileScanTask JSON with Java core feat(serde): add core FileScanTask JSON support Sep 30, 2026
@kamcheungting-db

Copy link
Copy Markdown
Contributor Author

I split the original change into a reviewable stack:

  1. refactor(serde): move DataFile JSON handling from REST to core #974 — mechanical DataFile serde move into core
  2. fix(serde): make DataFile JSON compatible with Java Iceberg #975 — Java-compatible DataFile JSON
  3. feat(serde): support standalone FileScanTask JSON in core #939 — core FileScanTask serde and split validation
  4. fix(serde): reject invalid schema and partition IDs in JSON #940 — strict schema and partition ID parsing

For the incremental diff that belongs only to this PR, use:
kamcheungting-db/iceberg-cpp@datafile-serde-java...filescan-task-serde

The key distinction in this layer is:

  • Core task JSON is self-contained and uses inline "delete-files": [...].
  • REST may use response-local "delete-file-references": [0], which remains REST-only.
  • For a 100-byte file, only start = 0 and length = 100 are accepted; a split such as start = 10 is rejected instead of silently becoming a whole-file task.

The PR description now includes a complete JSON example and the stack order.

@kamcheungting-db kamcheungting-db changed the title feat(serde): add core FileScanTask JSON support feat(serde): support standalone FileScanTask JSON in core Sep 30, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant