Skip to content

feat(storage): support Jindo for OSS scan planning - #696

Open
XiaoHongbo-Hope wants to merge 3 commits into
apache:mainfrom
XiaoHongbo-Hope:codex/jindo-storage
Open

XiaoHongbo-Hope wants to merge 3 commits into
apache:mainfrom
XiaoHongbo-Hope:codex/jindo-storage

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Purpose

Add an opt-in, read-only JindoSDK backend for OSS scans. OpenDAL remains the default. PyPaimon can select it for native planning through apache/paimon#9124.

Changes

  • add the optional storage-jindo feature, selected with fs.oss.impl=jindo
  • dynamically load an installed JindoSDK without bundling its binary
  • preserve HTTP/HTTPS endpoint protocol when adapting Jindo configuration
  • support stat, paginated list, and ranged/full reads
  • fetch list pages on demand and convert Jindo list exceptions into Rust errors
  • initialize lazily and run blocking SDK calls on Tokio blocking workers
  • reject unsupported write operations explicitly

Benchmark

Release-mode, partition-filtered planning on two fixed OSS-backed snapshots. Both backends ran on the same host with identical predicates, in reversed order and two independent processes per backend.

Workload Planned scale OpenDAL warm median Jindo warm median
A ~281k files 1.55s 1.25s
B ~37k files / ~27k splits 5.92s 2.19s

Both backends produced matching snapshot, split, file-count, and planned-byte results. This measures scan planning, not data throughput. Request count and peak OSS QPS were not measured.

Validation

  • 6 Jindo storage tests and Python SDK discovery test
  • default OSS and storage-jindo builds
  • Clippy, formatting, dependency report, and diff checks
  • read-only OSS REST-catalog test covering snapshot listing, planning, and row reading
  • JindoSDK 6.10.4 local OSS-protocol probes: HTTPS emits TLS, list exceptions return errors, three-page listing is complete, and consuming one entry does not request later pages

Notes

No JindoSDK binary is packaged. Selecting Jindo without the feature or a loadable SDK returns a configuration error. A FileIO configured with Jindo rejects writes, deletes, renames, and copies.

@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review August 9, 2026 04:01
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as draft August 9, 2026 04:44
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review August 9, 2026 06:20
@JingsongLi

Copy link
Copy Markdown
Contributor

Have you compared Jindo and OpenDAL? What are the differences? (benchmark)

@XiaoHongbo-Hope

Copy link
Copy Markdown
Contributor Author

Have you compared Jindo and OpenDAL? What are the differences? (benchmark)

yes, I did a test, Jindo performance is a little better, but there is a lot of qps issue during my testing, so may need another testing. And, I think this PR is best to have, not very urgent now. Because qps issue is addressed in other way.

@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as draft August 10, 2026 01:29
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review September 20, 2026 13:34
@XiaoHongbo-Hope

Copy link
Copy Markdown
Contributor Author

Have you compared Jindo and OpenDAL? What are the differences? (benchmark)

I updated the performance result in PR descrption.

@XiaoHongbo-Hope
XiaoHongbo-Hope requested review from JingsongLi and QuakeWang and removed request for JingsongLi September 20, 2026 13:45

@QuakeWang QuakeWang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants