feat(storage): support Jindo for OSS scan planning - #696
Open
XiaoHongbo-Hope wants to merge 3 commits into
Open
XiaoHongbo-Hope wants to merge 3 commits into
XiaoHongbo-Hope wants to merge 3 commits into
Conversation
XiaoHongbo-Hope
marked this pull request as ready for review
August 9, 2026 04:01
XiaoHongbo-Hope
marked this pull request as draft
August 9, 2026 04:44
XiaoHongbo-Hope
marked this pull request as ready for review
August 9, 2026 06:20
Contributor
|
Have you compared Jindo and OpenDAL? What are the differences? (benchmark) |
Contributor
Author
yes, I did a test, Jindo performance is a little better, but there is a lot of qps issue during my testing, so may need another testing. And, I think this PR is best to have, not very urgent now. Because qps issue is addressed in other way. |
XiaoHongbo-Hope
marked this pull request as draft
August 10, 2026 01:29
XiaoHongbo-Hope
force-pushed
the
codex/jindo-storage
branch
from
September 20, 2026 12:38
10f72fe to
258cf5f
Compare
XiaoHongbo-Hope
force-pushed
the
codex/jindo-storage
branch
from
September 20, 2026 12:54
258cf5f to
74ead67
Compare
XiaoHongbo-Hope
marked this pull request as ready for review
September 20, 2026 13:34
Contributor
Author
I updated the performance result in PR descrption. |
XiaoHongbo-Hope
requested review from
JingsongLi and
QuakeWang
and removed request for
JingsongLi
September 20, 2026 13:45
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Add an opt-in, read-only JindoSDK backend for OSS scans. OpenDAL remains the default. PyPaimon can select it for native planning through apache/paimon#9124.
Changes
storage-jindofeature, selected withfs.oss.impl=jindoBenchmark
Release-mode, partition-filtered planning on two fixed OSS-backed snapshots. Both backends ran on the same host with identical predicates, in reversed order and two independent processes per backend.
Both backends produced matching snapshot, split, file-count, and planned-byte results. This measures scan planning, not data throughput. Request count and peak OSS QPS were not measured.
Validation
storage-jindobuildsNotes
No JindoSDK binary is packaged. Selecting Jindo without the feature or a loadable SDK returns a configuration error. A FileIO configured with Jindo rejects writes, deletes, renames, and copies.