ngmix: noise-fill defects under both BLEND_HANDLING options, drop epochs with a defect near the object - #915
Closed
cailmdaley wants to merge 5 commits into
Closed
cailmdaley wants to merge 5 commits into
cailmdaley wants to merge 5 commits into
Conversation
…defects Defects (nonzero flag, zero weight, invalid RMS) must be zero-weighted and noise-filled under every BLEND_HANDLING, with neighbour pixels left raw under uberseg. The masked-fraction cut must count that same raw set, and an epoch with a defect closer than EPOCH_CENTRAL_DEFECT_RADIUS to the stamp centre must be dropped, at a strict boundary and at the configured radius. The runner must pass both cut options on, the HSM centroid must ignore raw defect values, each epoch must keep its own OFFSET, and every tile must log its epoch-cut tally. The science test recovers the full 2x2 response for columns, 3-px bleeds, single pixels and edge bands at the veto radius, on 0.3" and 0.5" galaxies through round and elliptical 0.7" PSFs, and requires |c1|, |c2| < 5e-4 and |m11|, |m22| < 1%. Flagged pixels in the simulation hold a hot value, so an unfilled defect shows up. A column two pixels inside the radius is the positive control. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QAc1ywyd9Sbbra6po2iuaA
…ral ones Metacal shears the whole stamp without reading the weights, so a defect left raw under uberseg reached the fit. Every defect (nonzero flag, zero weight, invalid RMS) is now zero-weighted and noise-filled whatever BLEND_HANDLING is; uberseg only zeroes the weight of neighbour-side pixels and keeps their light. The option that meant "no neighbour treatment" is renamed from noisefill to none, and the retired name is rejected. The defect set is not symmetrized: at the veto radius the one-sided fill leaves |c| <= 3e-4, while a four-fold mask would quadruple m (-2.7% against -0.64% for a 3-px bleed at 10 px on a 0.5" galaxy) and, through an elliptical PSF, still leave c1 = 7.3e-4. An epoch is dropped when a defect lies closer than EPOCH_CENTRAL_DEFECT_RADIUS (default 10 px) to the stamp centre, the smallest radius at which columns, 3-px bleeds, single pixels and edge bands give |m11|, |m22| < 1% and |c1|, |c2| < 5e-4 on 0.3" and 0.5" galaxies through a 0.7" PSF. The masked-fraction cut counts the same raw defect set; EPOCH_MASKED_FRACTION_CUT (default 1/3) makes it configurable. Each tile logs "epoch cuts: considered=N masked_fraction=A central_veto=B objects_emptied=C". The HSM centroid is measured on the filled image. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QAc1ywyd9Sbbra6po2iuaA
prepare_postage_stamps already holds the object's galaxy vignet dict, so the per-epoch OFFSET is read from it instead of a second sqlitedict lookup. The pixel scale sets only the centroid-prior width; the comments that also credited it with a noise window are corrected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QAc1ywyd9Sbbra6po2iuaA
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
this PR needs a bit more context/motivation/explanation of what was done. |
This was referenced Sep 28, 2026
Merged
Brings the decision record to what this branch does: - central_defect_veto: committed option fixed_radii (was disabled); the 10 px EPOCH_CENTRAL_DEFECT_RADIUS is pinned by a Values entry on its module constant, now tagged. - epoch_masked_fraction_cut: the cut counts the defect set against EPOCH_MASKED_FRACTION_CUT (no longer [HARDCODED] on flags). - defect_fill: defects are noise-filled under every BLEND_HANDLING; the docstring [LINT] this branch fixes is dropped. - blend_handling: the no-treatment option is BLEND_HANDLING = none. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
Superseded by #922, which carries this branch's commits into one masked-pixel PR. |
cailmdaley
added a commit
that referenced
this pull request
Sep 28, 2026
Brings the tile segmentation map for UberSeg into #922, so the masked-pixels work sits in one place. feat/uberseg-seg-map is built on #897 (feat/unions-catalogue-tile-detection), so this provisionally carries #897's content too (UNIONS catalogue detection, TILE_UNIQUE_ID, ngmix row chunks) until #897 lands on develop. No textual conflicts: ngmix.py's row-selection chunking (#897) and the epoch cuts, defect fill and constants (#915/#916) touch disjoint hunks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cailmdaley
added a commit
that referenced
this pull request
Sep 28, 2026
#915/#916 renamed ngmix's BLEND_HANDLING `noisefill` to `none` (and reject the old name); the workflow's blend_handling (completeness.BLEND_HANDLINGS, config.yaml, Snakefile default, README) follows. Tests from each side meet the other's Ngmix.process: develop's run-health test stubs prepare_postage_stamps with the epoch-cut kwargs and epoch_cuts tally and sets the epoch-cut attributes; the defect-fill process tests give their vignettes an OFFSET (required under centroid_source="wcs") and their logger an error/warning method for log_run_health. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The filled hole biases m: metacal shears it, the sky doesn't. So ngmix drops an epoch with a defect within
EPOCH_CENTRAL_DEFECT_RADIUS(default 10 px; 0 disables) of the centre. For one high-S/N epoch (galaxy half-light radius 0.3″ or 0.5″, PSF FWHM 0.7″), defects at 10 px give |m| ≤ 1%, |c| ≤ 3e-4. A 0.7″ galaxy with a 0.9″ PSF needs 14 px (m11 = −6.8% at 10 px).EPOCH_MASKED_FRACTION_CUT(still 1/3) now counts all defects, not just flags, so it can drop more epochs. Default runs change only through these two cuts. Real epoch loss is unmeasured; tiles log it.— Claude (Opus) on behalf of Cail
🤖 Generated with Claude Code