feat(scan): support file indexes in real-time and PK scans

[lxy-9602]

Purpose

Linked issue: #158

This change enables file-index pruning for real-time reads and aligns scan-time index handling across append-only and primary-key tables.

• Apply embedded and external file indexes in the real-time merge reader.
• Combine bitmap index selections with deletion vectors for PK tables.
• Add scan-time pruning with embedded file indexes for both append-only and PK tables.
• Keep external index evaluation in the reader stage, consistent with Java Paimon.

Tests

• append-only and PK+DV real-time reads using embedded and external bitmap indexes;
• scan pruning when min/max statistics retain a file but its embedded bitmap index filters it;
• reader-stage bitmap filtering when a split is retained;
• predicate reconstruction after a field rename through schema evolution.

API and Format

Documentation

Generative AI tooling

Generated-by: OpenAI Codex (GPT-5)

[zjw1111]

One performance concern in the merge-reader fallback.

[zjw1111]

+1

[SteNicholas]

LGTM.