Skip to content

perf(scan): skip manifest lists on exact snapshot cache hits - #374

Draft
wangyong9999 wants to merge 1 commit into
apache:mainfrom
wangyong9999:perf/manifest-cache-fast-hit-20260919
Draft

wangyong9999 wants to merge 1 commit into
apache:mainfrom
wangyong9999:perf/manifest-cache-fast-hit-20260919

Conversation

@wangyong9999

@wangyong9999 wangyong9999 commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

Purpose

Closes #375.

Repeated scans of the same snapshot and bucket decode the manifest lists before checking the snapshot live-entry cache. Check that cache after selecting the snapshot, and bypass manifest-list reads and decoding on an exact hit. Misses still rebuild the complete bucket; each scan still applies its request-specific filters.

Store the table file count with the cached entries to preserve skipped-file metrics. An exact hit now reports zero scanned manifests. This targets repeated point-read planning without introducing another cache or changing snapshot discovery.

Tests

  • Local C++17 Release build of the identical patch on the downstream main branch succeeded without compiler warnings.
  • 88 focused scan, snapshot, predicate, serialization, and bucket-pruning tests passed on that downstream build. Regression coverage checks no manifest-cache access on an exact hit, changed predicates, eviction/rebuild, and retained file counts; existing tests cover snapshot changes.
  • pre-commit run --all-files and git diff --check passed on this upstream checkout.
  • The downstream CI suite passed. SCM test packages built successfully for both x86_64 and aarch64.
  • Upstream workflows require maintainer approval (action_required); they have not run. No standalone end-to-end performance uplift is claimed for this individual planner optimization.

API and Format

No public API, table storage format, or protocol change. Only the transient serialized snapshot-cache payload changes; its magic is versioned so old cache entries fall back to rebuilding. Snapshot selection and cache eligibility remain unchanged.

Documentation

Updated the manifest-entry cache guide to describe bypassing manifest lists and exact-hit scan metrics.

Generative AI tooling

Generated-by: Codex based on GPT-6.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Bypass manifest lists on exact snapshot live-entry cache hits

1 participant