Skip to content

feat(file-index): support file-local search with PK+DV filtering - #372

Draft
zjw1111 wants to merge 4 commits into
apache:mainfrom
zjw1111:codex/file-index-vector-full-text-search
Draft

zjw1111 wants to merge 4 commits into
apache:mainfrom
zjw1111:codex/file-index-vector-full-text-search

Conversation

@zjw1111

@zjw1111 zjw1111 commented Sep 18, 2026

Copy link
Copy Markdown
Collaborator

Purpose

Linked issue: #361

  • Add file-local vector/full-text search interfaces and a Lumina File Index vector reader, including explicit _INDEX_SCORE projection.
  • Support File Index Search on primary-key tables with deletion vectors for raw-convertible splits. Apply DV and user pre-filters before each file-local TopK/limit, and reject unsupported splits instead of falling back to merge reads.
  • Tantivy File Index is intentionally deferred. The full-text reader path is prepared, but there is no file-local full-text backend in this PR. Predicate + Search and cross-file global TopK remain out of scope.

Tests

  • CCACHE_DISABLE=1 cmake --build build --target paimon-core-test paimon-lumina-index-test -j 8
  • ASAN_OPTIONS=detect_leaks=0 ./build/debug/paimon-core-test --gtest_filter='RawFileSplitReadTest.*:InternalReadContext.*:TableReadTest.*:FileIndexEvaluatorTest.*:MergeFileSplitReadTest.*' (43 passed)
  • ASAN_OPTIONS=detect_leaks=0 ./build/debug/paimon-lumina-index-test --gtest_filter='LuminaFileIndexTest.*' (2 passed)
  • pre-commit on the latest changed C++ files and git diff --check passed.

API and Format

Adds File Index search requests/readers and a scored file-local result type. Adjusts the File Index reader options interface. The persisted File Index format is unchanged.

Documentation

API comments describe the new search and score behavior. No standalone user guide is included in this PR.

Generative AI tooling

Generated-by: OpenAI Codex (GPT-5-based; desktop app version not exposed).

@zjw1111
zjw1111 marked this pull request as draft September 18, 2026 05:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant