ngmix: opt-in DEFECT_FILL = interpolate for narrow defects, with a 7 px veto - #916
Closed
cailmdaley wants to merge 3 commits into
Closed
cailmdaley wants to merge 3 commits into
cailmdaley wants to merge 3 commits into
Conversation
The interpolation fills only short defect runs (at most 3 px along a row or column, with clean pixels at both ends); anything wider, and runs reaching the stamp border, stays noise-filled. The interpolant must reproduce linear planes without reading defect values, commute with quarter turns of the stamp, and apply one operator to the science and metacal noise images. The weight is zero on the defects and on the quarter-turn orbit of the interpolated pixels, whose light stays. Each defect is vetoed at the radius of its own fill: EPOCH_INTERPOLATED_DEFECT_RADIUS for interpolated pixels, EPOCH_CENTRAL_DEFECT_RADIUS for noise-filled ones. The runner and Ngmix pass DEFECT_FILL to the epoch cuts and to metacal; noise stays the default. The science test recovers the full response matrix for interpolated columns, full and finite 3-px bleeds and single pixels at the interpolated radius, on 0.3" and 0.5" galaxies through round and elliptical 0.7" PSFs and on a 0.7" galaxy through a 0.9" PSF, and for noise-filled wide defects at the noise radius. A bleed three pixels inside the radius is the positive control. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QAc1ywyd9Sbbra6po2iuaA
… veto DEFECT_FILL = interpolate fills short defect runs (at most 3 px along a row or column, with clean pixels at both ends: columns, 3-px bleeds, single pixels) with a Clough-Tocher interpolant of the clean pixels within 4 px. The interpolant is averaged over the four quarter turns of the stamp, and each turn's triangulation serves the science image and the metacal noise image alike. Wider holes and runs reaching the stamp border keep the noise fill. noise stays the default and is unchanged. Interpolation restores the object's light, so the remaining bias is the hole in the likelihood: a one-sided zero-weight column 8 px from a 0.5" galaxy still gives c1 = -1.3e-3. The weight is therefore also zero on the quarter-turn orbit of the interpolated pixels (their light stays), which brings that column to c1 = +2e-6. Symmetrizing the fill instead would trade true light for interpolated light over four times the area. Each defect pixel is vetoed at the radius of its own fill: EPOCH_INTERPOLATED_DEFECT_RADIUS (default 7 px) for interpolated pixels and EPOCH_CENTRAL_DEFECT_RADIUS (10 px) for noise-filled ones. 7 px is the smallest radius at which interpolated columns, 3-px bleeds and single pixels give |m11|, |m22| < 1% and |c1|, |c2| < 5e-4 on 0.3", 0.5" and 0.7" galaxies, round and elliptical PSFs; it does not scale with galaxy size. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01QAc1ywyd9Sbbra6po2iuaA
Contributor
Author
|
i think this and #915 can be grouped under a masking/filling epic, it's probably worth creating a high-level design document for what we want our strategy to be here. |
This was referenced Sep 28, 2026
Merged
Brings develop (#875's decision record) in through the updated base, and amends the record for this branch: - defect_fill: the interpolate option is implemented (DEFECT_FILL = interpolate); noise stays the committed fill. - central_defect_veto: EPOCH_INTERPOLATED_DEFECT_RADIUS = 7 joins the Values, on its now-tagged module constant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
5 tasks
Contributor
Author
|
Superseded by #922, which carries this branch's commits into one masked-pixel PR. |
cailmdaley
added a commit
that referenced
this pull request
Sep 28, 2026
Brings the tile segmentation map for UberSeg into #922, so the masked-pixels work sits in one place. feat/uberseg-seg-map is built on #897 (feat/unions-catalogue-tile-detection), so this provisionally carries #897's content too (UNIONS catalogue detection, TILE_UNIQUE_ID, ngmix row chunks) until #897 lands on develop. No textual conflicts: ngmix.py's row-selection chunking (#897) and the epoch cuts, defect fill and constants (#915/#916) touch disjoint hunks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cailmdaley
added a commit
that referenced
this pull request
Sep 28, 2026
#915/#916 renamed ngmix's BLEND_HANDLING `noisefill` to `none` (and reject the old name); the workflow's blend_handling (completeness.BLEND_HANDLINGS, config.yaml, Snakefile default, README) follows. Tests from each side meet the other's Ngmix.process: develop's run-health test stubs prepare_postage_stamps with the epoch-cut kwargs and epoch_cuts tally and sets the epoch-cut attributes; the defect-fill process tests give their vignettes an OFFSET (required under centroid_source="wcs") and their logger an error/warning method for log_run_health. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DEFECT_FILL = interpolatefills narrow defects (≤ 3 px wide, clean on both sides: columns, 3-px bleeds, single pixels) by Clough–Tocher interpolation from surrounding clean pixels, averaged over four 90° rotations. It fills the metacal noise image too. Wider holes and edge bands stay noise-filled.Interpolated pixels keep zero weight, which still pulls the model fit when one-sided. For one high-S/N epoch with PSF FWHM 0.7″, a column 8 px from a galaxy of half-light radius 0.5″ gives c1 = −1.3e-3. So ngmix also zero-weights those pixels rotated by 90°, 180° and 270°, keeping their light; c1 drops to 2e-6 (#815's weight symmetrization, for defects).
Interpolated defects are vetoed within 7 px (
EPOCH_INTERPOLATED_DEFECT_RADIUS), noise-filled ones within 10 px. At 7 px, 0.3″ and 0.5″ galaxies give |m| ≤ 0.21%, |c| ≤ 5e-5, and a 0.7″ galaxy with a 0.9″ PSF gives |c| ≤ 3.6e-4. Noise-filled defects at 10 px give that galaxy m11 = −6.8%. A size-dependent radius would select on measured size, which responds to shear. Interpolation adds 4–16 ms per affected epoch (metacal: ~63 ms).Draft until #915 merges and the next Nibi campaign measures how many epochs the defect cuts drop.
— Claude (Opus) on behalf of Cail
🤖 Generated with Claude Code