Skip to content

GH-3826: Preserve component paths for Bloom filters - #3827

Open
costas-db wants to merge 2 commits into
apache:masterfrom
costas-db:gh-3826-column-path-bloom-repro
Open

costas-db wants to merge 2 commits into
apache:masterfrom
costas-db:gh-3826-column-path-bloom-repro

Conversation

@costas-db

@costas-db costas-db commented Sep 28, 2026 •

Copy link
Copy Markdown

Rationale for this change

Parquet column paths are component-based, but Bloom filters were stored under dot-string keys. A top-level field named a.b (["a.b"]) therefore collided with nested field a.b (["a", "b"]), allowing one column’s Bloom filter to overwrite the other.

What changes are included in this PR?

  • Keys internal Bloom-filter maps by ColumnPath rather than flattened strings.
  • Preserves the existing addBloomFilter(String, …) API and adds addBloomFilterForPath(ColumnPath, …) for structured callers.
  • Keeps component paths intact when copying Bloom filters through ParquetRewriter.
  • Adds a direct regression plus a readable expected-output JSON fixture.

Are these changes tested?

Before the fix, the focused reproduction reports:

"topLevelContainsOwnBloomValues" : false
"nestedContainsOwnBloomValues" : true

After the fix, both the direct and expected-output tests pass.

Also verified:

  • Full TestParquetWriter
  • TestParquetFileWriter
  • ParquetRewriterTest
  • spotless:check
  • apache-rat:check

Are there any user-facing changes?

No on-disk format change. The existing string-based API remains compatible; this adds a component-based API for callers that already have a ColumnPath.

Closes #3826.

@costas-db costas-db changed the title GH-3826: Add reproduction for Bloom filter path collisions GH-3826: Preserve component paths for Bloom filters Sep 28, 2026
@costas-db
costas-db marked this pull request as ready for review September 28, 2026 07:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bloom filters collide for distinct column paths with the same dot string

1 participant