Skip to content

Seed the jackknife k-means and drop the patch-centre file - #367

Draft
cailmdaley wants to merge 1 commit into
developfrom
feat/jackknife-patch-centres
Draft

cailmdaley wants to merge 1 commit into
developfrom
feat/jackknife-patch-centres

Conversation

@cailmdaley

Copy link
Copy Markdown
Collaborator

calculate_2pcf builds the jackknife patches for ξ± with TreeCorr's k-means, which picks a fresh random seed on every unseeded call. Develop hides this by drawing the centres once per output tree and reusing {ver}_patches_npatch=N.dat, but a fresh tree, a rerun elsewhere, or two jobs racing to write the file all split the sky differently, so the ξ± jackknife covariance changed from run to run. It is the same class of bug that moved ρ/τ statistics by up to ~3σ between reruns (fixed in shear_psf_leakage #43).

This PR seeds the k-means (rng=np.random.default_rng(0)). Seeded, the patches are a pure function of the catalogue's positions and weights, so the centre file has no job left and goes too.

What changes

  • calculate_2pcf: seeded k-means; the patch-centre file, its reuse and its write are removed.
  • calculate_aperture_mass_dispersion builds its own patched catalogue and gets the same seed.
  • The thread-count test becomes a reproducibility test: two fresh output trees at 4 and 16 threads give identical per-patch-pair counts, and matching ξ± and jackknife variance. With the seed removed, the test fails.

Effect on results

  • ξ± values: essentially unchanged. Patches enter the covariance, and otherwise only decide which cell pairs TreeCorr's bin_slop approximates; not measured against develop's draws.
  • ξ± jackknife covariance: reproducible across trees, reruns and thread counts. It differs from any earlier unseeded draw.
  • Existing _patches_npatch=*.dat files are no longer read. They can be deleted.
  • A variant that reweights the catalogue gets its own patches. One that keeps positions and weights shares them.

Cost

The k-means runs on every measurement. On 16 threads it takes about 1 s per 10⁶ objects (≈10–15 s at 10⁷ for 20–200 patches), which is small next to the pair counting.

Verified

  • Container suite: 306 passed, 1 xfailed.

— Claude (Opus) on behalf of Cail.

🤖 Generated with Claude Code

…tre file

TreeCorr seeds its k-means afresh on every unseeded call, so the jackknife
patches, and with them the ξ± jackknife covariance, differed between runs
that did not share a {ver}_patches_npatch=N.dat. The k-means is now seeded
with a fixed generator, which makes the patches a pure function of the
catalogue's positions and weights on any thread count; the centre file and
its reuse logic go. The aperture-mass ξ± takes the same seed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant