Skip to content

Use Stephen Gwyn's UNIONS tile catalogue as the tile galaxy sample - #897

Merged
cailmdaley merged 25 commits into
developfrom
feat/unions-catalogue-tile-detection
Sep 29, 2026
Merged

Use Stephen Gwyn's UNIONS tile catalogue as the tile galaxy sample#897
cailmdaley merged 25 commits into
developfrom
feat/unions-catalogue-tile-detection

Conversation

@cailmdaley

@cailmdaley cailmdaley commented Sep 18, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

For real data, we now measure shapes for the objects in Stephen Gwyn's UNIONS tile catalogue (DR6), instead of detecting our own on each tile with SExtractor. Measuring shapes on that catalogue lets shape, photometry and redshift catalogues share one object list, so we can match them by ID rather than by position (#832). Image simulations have no DR6 catalogue, so they keep running SExtractor on the tile. PSF stars come from our own exposure-level SExtractor run in both cases.

One object list, one ID

Every final catalogue carries TILE_UNIQUE_ID = tile_id·10⁶ + NUMBER. Here tile_id = RRR·1000 + DDD identifies the tile and NUMBER is the object's number in that tile's catalogue. Together they make an ID that is unique across the survey, so shape, photometry and redshift catalogues can be joined directly. The ID is defined in one place (cfis.get_tile_unique_id), which raises if NUMBER reaches 10⁶ and the ID would collide.

From catalogue to the stamps ngmix needs

The rest of the pipeline expects what a SExtractor run produces: a FITS-LDAC catalogue with a postage stamp (VIGNET) per object, cut from the tile image. We fetch the DR6 catalogue for each tile and convert it into exactly that form, keeping Stephen's NUMBER. From there on, both detection modes run the same chain.

A SExtractor run also marks neighbours. In each object's stamp it writes −1e30 on pixels belonging to other detections, and ngmix gives those pixels zero weight and fills them with noise. A catalogue alone carries no such information. Without it, neighbour light enters the shape fit: on our first 25-tile test run in catalogue mode, 6.1% of objects came out more than a magnitude brighter than their catalogue magnitude, against 2.7% for the same tiles in SExtractor mode. So we also fetch Stephen's segmentation map for the tile. The map has the tile's pixel grid, but its labels are not the catalogue's NUMBERs. We give each object the footprint under its centre and mark every other footprint in its stamp, as SExtractor would. On tile 202.301, all but one of the 36,065 objects find their own footprint, no two share one, and the marking covers every neighbour pixel and none of an object's own. The fraction of stamps dropped as more than a third masked is 0.58%, against 0.55% in SExtractor mode.

Technical notes

  • Settings. tile_detection is unions_catalogue for data and sextractor for image simulations. Both are defaults per input type in workflow/config.yaml, which a run config can override. The DR6 files come from vos:cfis/tiles_DR6.
  • ngmix chunks select catalogue rows rather than NUMBER ranges, so the chunks cover every object exactly once however NUMBER is ordered or spaced.
  • Merged catalogues. DR6 lacks nine SExtractor-only columns (DR6 catalogue mode drops 9 SExtractor-only columns from the final catalogue #924). In catalogue mode, the final-catalogue merge drops those nine and stays strict on all the others. It requires TILE_UNIQUE_ID, so final catalogues made with and without this PR can't be merged together.
  • Campaign boundary. The workflow's params change: tile_detect runs the converter, and tile_get_catalogue is a new rule. Resuming a campaign started before this PR reruns its tile units.
  • Decision record. astra.yaml records the choice as detection.tile_detection and the neighbour marking as detection.catalogue_neighbour_marking.
  • Tested on the 25-tile Nibi pilot run in catalogue mode. It exposed the missing columns and the missing neighbour masking, both fixed here. A rerun of those tiles, compared against the SExtractor-mode run, is the check on the neighbour masking.

Closes #832.

— Claude (Opus) on behalf of Cail

🤖 Generated with Claude Code

cailmdaley and others added 6 commits September 17, 2026 13:11
@cailmdaley @claude
Add the UNIONS tile-catalogue inis to the workflow config chain
598a87d 
Move the two v2.0-era configs (fetch Steven Gwyn's per-tile .cat, convert it
with read_ext_sexcat_runner) into workflow/config/cfis and adapt them to the
workflow's conventions: fixed run names, NUMBER_LIST from SP_UNIT_NUM,
explicit chained INPUT_DIRs, and the catalogue source and retrieve mode
taken from the environment the rules export. The converter writes into
run_sp_tile_Sx so that the downstream tile chain reads one path in either
detection mode. get_images_runner expands RETRIEVE so it can be set per run.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@cailmdaley @claude
Add tile_detection: unions_catalogue to the workflow
8702483 
A run config key, validated at parse time, that picks where the tile's galaxy
sample comes from. `sextractor` is the existing SExtractor rule, byte for byte.
`unions_catalogue` replaces it with two rules: tile_get_catalogue fetches the
UNIONS per-tile .cat from inputs.catalogues (a local mirror or a vos: URL),
and tile_detect converts it with read_ext_sexcat_runner into run_sp_tile_Sx,
linking the module output as sextractor_runner so that tile_vignets, ngmix
and make_cat read the one path they always have; the manifest is
tile_detect.json in both modes. completeness.py checks tile_detect per mode
through SP_TILE_DETECTION, exported on that rule alone so a SExtractor run's
prologue is unchanged; run_report lists the fetch stage only for catalogue
runs. The catalogue's per-tile unique object ID reaches the final catalogue
as TILE_UNIQUE_ID through make_cat's copy of every sexcat column.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@cailmdaley @claude
Test the UNIONS catalogue conversion and its workflow wiring
379be48 
read_ext_sexcat on a synthetic ASCII catalogue and image: LDAC layout, the
tile header in LDAC_IMHEAD, the SExtractor aliases, zero-padded VIGNET stamps
cut at the right pixels, TILE_UNIQUE_ID = tile_id * 1e6 + NUMBER, and that
column surviving make_cat.save_sextractor_data into the final catalogue. On
the workflow side: the two detection inis share the stage's run dir, the
fetch ini feeds the converter, the tile_detect count table is selected per
mode with sextractor as the unset default, and run_report lists the fetch
stage only for catalogue runs.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@cailmdaley @claude
Fix TILE_UNIQUE_ID merge, NUMBER contiguity, and FITS int64 dtype test
affcdc0 
final_cat.param lists TILE_UNIQUE_ID again; merge_final_cat.py and
create_final_cat.py now drop a requested column that is absent from a
given catalogue (logging once) instead of failing, so sextractor-mode
runs (without the column) and unions_catalogue-mode runs (with it)
both merge cleanly.

read_ext_sexcat's NUMBER passthrough is not guaranteed contiguous or
in order in the external catalogue, which ngmix_range requires;
TILE_UNIQUE_ID already preserves the original NUMBER, so the converter
now renumbers NUMBER to 1..n_obj in output row order and says so in
its docstring.

test_read_ext_sexcat's TILE_UNIQUE_ID dtype check compared against
np.int64 directly; a FITS 'K' column reads back big-endian, so the
check now compares dtype.kind and itemsize instead.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@cailmdaley @claude
Build the UberSeg segmentation map for the UNIONS catalogue path
930b5af 
UberSeg identifies a stamp's central object BY LABEL: uberseg_weight keeps
the pixels whose nearest segmentation footprint carries the object's own
catalogue NUMBER. The UNIONS per-tile catalogue carries no segmentation map,
and a SExtractor run's labels are its own detection numbers, so the two
numberings have to be bridged before ngmix can mask a neighbour.

`blend_handling:` (noisefill | uberseg) joins `tile_detection:` in the run
config. The pair unions_catalogue + uberseg adds tile_segmentation: one
SExtractor run on the tile image whose only product is the SEGMENTATION check
image (config_tile_Sg.ini), followed by seg_relabel.py, which gives each
catalogue object the footprint its position falls in, relabels it with that
object's NUMBER, marks every unclaimed footprint a neighbour, and paints a
small disc for an object that falls on sky or on a footprint already claimed.
The result lands beside the sexcat, where vignetmaker's segmentation run looks
whichever mode wrote the catalogue. Steven's catalogue stays the sample;
SExtractor runs once, for the one product the catalogue cannot give.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@cailmdaley @claude
Default tile_detection to the UNIONS catalogue
57b7525 
The per-tile catalogue is the sample; SExtractor detection is the explicit
alternative. `inputs.catalogues` travels with it -- the Snakefile refuses
unions_catalogue without one -- and is committed as TBD for the run to set.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@cailmdaley

cailmdaley commented Sep 26, 2026

Copy link
Copy Markdown
Contributor Author

can this be split into separate (perhaps sequential) PRs on ubserseg and per-tile catalogs?

@martinkilbinger
martinkilbinger self-requested a review September 28, 2026 13:12
@cailmdaley

cailmdaley commented Sep 28, 2026

Copy link
Copy Markdown
Contributor Author

let's split out uberseg from this by scoping to only noise fill, and tackle uberseg along with #915, #916, etc.

cailmdaley and others added 6 commits September 28, 2026 16:11
@cailmdaley @claude
Build TILE_UNIQUE_ID once, in make_cat, from one helper
36e96b3 
cfis.get_tile_unique_id defines the survey-wide object ID,
tile_id * 10**6 + NUMBER with tile_id = RRR * 1000 + DDD, and raises when
NUMBER (or tile_id) falls outside [0, 10**6). cfis.get_tile_id parses the
tile, split_tile_unique_id inverts the encoding, and get_tile_number now
refuses tile components with more than three digits.

make_cat writes TILE_UNIQUE_ID next to TILE_ID for every final catalogue,
whatever produced the detection catalogue. read_ext_sexcat is a plain
format converter again: it copies NUMBER unchanged and adds no ID.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uemfjv9ybCwtKtprksZbVY
@cailmdaley @claude
Select ngmix chunks by catalogue row, not NUMBER value
4e76ccc 
ID_OBJ_MIN/MAX are now 1-based closed row ranges (ngmix.chunk_rows), so the
partition ngmix_range.py writes covers every object once however NUMBER is
ordered or spaced. ngmix_range emits NGMIX_ROW_MIN/MAX, and checks that the
EPOCH extensions are row-aligned instead of requiring NUMBER = 1..N.
NUMBER stays the object's identity: vignet/PSF store keys, the uberseg
label check and position seeding are unchanged. fake_psf keys its PSF
store by NUMBER rather than row + 1.

For SExtractor catalogues (NUMBER = row) chunk contents are unchanged, but
ngmix_range.py's hash changes, so a resume across this commit re-measures.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uemfjv9ybCwtKtprksZbVY
@cailmdaley @claude
Merge develop into feat/unions-catalogue-tile-detection
ba332f4 
Conflicts resolved by intent:
- config.yaml: develop's machines table; blend_handling stays top level,
  tile_detection and inputs.catalogues move into the table for input_type
  data (tile_detection joins run_config.MACHINE_KEYS), so image sims fall
  back to sextractor.
- bin/sp: develop's launcher, plus SP_TILE_DETECTION/SP_BLEND_HANDLING for
  run_report.
- get_images_runner.py, README.md: develop's text, plus the tile_detection
  notes.
The Snakefile treats inputs.catalogues: TBD as unset.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uemfjv9ybCwtKtprksZbVY
@cailmdaley @claude
Merge refactor/tile-unique-id: TILE_UNIQUE_ID in make_cat, ngmix row …
7c91679 
…chunks

read_ext_sexcat keeps the catalogue's NUMBER and builds no ID (the
refactor side of the one conflict); make_cat builds TILE_UNIQUE_ID for both
detection modes and ngmix chunks select rows, so the renumbering is no
longer needed. test_read_ext_sexcat.py, seg_relabel's docstring, tile.smk
and the merge scripts' comments follow. TILE_UNIQUE_ID joins the cfis and
cfis_image_sims final_cat.param lists; the missing-column filter keeps
older final catalogues mergeable.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uemfjv9ybCwtKtprksZbVY
@cailmdaley @claude
Segmentation run uses the tile detection's MegaPipe filter
e34b275 
config_tile_Sg.ini takes DOT_CONV_FILE from config_tile_Sx.ini (#896), so
the uberseg footprints are the ones SExtractor mode would draw.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uemfjv9ybCwtKtprksZbVY
@cailmdaley @claude
Merge origin/develop into feat/unions-catalogue-tile-detection
bfd2f87 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley cailmdaley mentioned this pull request Sep 28, 2026
Draft
5 tasks
cailmdaley and others added 4 commits September 28, 2026 17:18
@cailmdaley @claude
Move UberSeg segmentation to its own branch
2e09027 
The tile_segmentation rule, seg_relabel.py, config_tile_Sg.ini and the
blend_handling setting move to feat/uberseg-seg-map, which feeds the
consolidated blend/defect PR. This branch keeps the Gwyn catalogue path,
TILE_UNIQUE_ID and row-based ngmix chunking; the catalogue path has no
segmentation map, so uberseg needs the SExtractor mode.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley @claude
Default tile_detection and psf_model per input type, not per machine
5511597 
Both follow from the kind of input (the UNIONS catalogue and PSFEx for
data, SExtractor and the true simulation PSF for image sims), so they move
out of the machine entries into a top-level input_types: table. run_config
layers config.yaml < input_types[input_type] < machines[machine][input_type]
< run config, and expands $variables in every resolved key.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley @claude
Record tile_detection in astra.yaml
9d1e023 
Real data now takes its tile sample from Gwyn's UNIONS catalogue. The
detection sub-analysis gains a tile_detection decision (committed:
unions_catalogue), pinned to the input_types table, and notes that its
SExtractor decisions govern only the sextractor path. The converter's
multi-epoch post-processing is tagged under epoch_membership_ccd_bounds,
which applies to both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley @claude
Point inputs.catalogues at vos:cfis/tiles_DR6 on both machines
21e7c4e 
Stephen Gwyn's per-tile catalogues live at vos:cfis/tiles_DR6 (the path
develop's config_tile_Git_cat_vos.ini already uses). tile_get_catalogue
fetches them with vcp inside the job, using ~/.ssl/cadcproxy.pem; that works
from a candide compute node in the runtime container.

The unset/placeholder check moves from the Snakefile into
run_config.catalogue_source, so the parse-time refusal is unit-tested.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley cailmdaley changed the title Use Steven Gwyn's UNIONS tile catalogue as the tile galaxy sample (default); segmentation map for UberSeg Use Stephen Gwyn's UNIONS tile catalogue as the tile galaxy sample Sep 28, 2026
cailmdaley added a commit that referenced this pull request Sep 28, 2026
@cailmdaley @claude
Merge origin/feat/uberseg-seg-map into feat/masked-pixels
23e823e 
Brings the tile segmentation map for UberSeg into #922, so the masked-pixels
work sits in one place. feat/uberseg-seg-map is built on #897
(feat/unions-catalogue-tile-detection), so this provisionally carries #897's
content too (UNIONS catalogue detection, TILE_UNIQUE_ID, ngmix row chunks)
until #897 lands on develop.

No textual conflicts: ngmix.py's row-selection chunking (#897) and the epoch
cuts, defect fill and constants (#915/#916) touch disjoint hunks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
cailmdaley added a commit that referenced this pull request Sep 28, 2026
@cailmdaley
Merge origin/feat/unions-catalogue-tile-detection into feat/masked-pi…
6bf7aa7 
…xels

Keeps #897's content here current (inputs.catalogues set to vos:cfis/tiles_DR6,
parse-time check in run_config.catalogue_source).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

# Conflicts:
#	workflow/config.yaml
@cailmdaley

cailmdaley commented Sep 28, 2026

Copy link
Copy Markdown
Contributor Author

@martinkilbinger this is converging and ready for review!

@cailmdaley
cailmdaley marked this pull request as ready for review September 28, 2026 16:48
cailmdaley and others added 2 commits September 28, 2026 20:38
@cailmdaley @claude
Merge origin/develop into feat/unions-catalogue-tile-detection
0a0396a 
Brings in #879 (campaign merges), #911 (spread model removed), #854,
#905 and #923.

- run_config: input_types:/machines: table resolver kept; develop's
  required `run:` and unresolved-$ check folded in (now over every
  non-table key).
- create_final_cat.read_data keeps develop's raise on a missing column;
  the branch's skip-missing path is dropped (final_cat_merge is the one
  producer of final_cat_<run>.hdf5).
- final_cat.param carries both NUMBER and TILE_UNIQUE_ID.
- DAG tests follow the committed tile_detection default (data plans
  tile_get_catalogue); params pin regenerated for the tile_detect switch.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DksuyF9YrZAwHQeAMXsxp4
@cailmdaley @claude
Leave the retired standalone merge_final_cat.py to develop
9f034f8 
The workflow merges final catalogues through final_cat_merge; the
standalone script is retired, so this branch no longer patches it or
tests filter_available_columns.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DksuyF9YrZAwHQeAMXsxp4
cailmdaley added a commit that referenced this pull request Sep 28, 2026
@cailmdaley
Merge #897 (feat/unions-catalogue-tile-detection) into integration/v2…
e57a1ce 
…-bigrun
martinkilbinger
martinkilbinger reviewed Sep 29, 2026
View reviewed changes
Comment thread workflow/config.yaml Outdated
martinkilbinger
martinkilbinger reviewed Sep 29, 2026
View reviewed changes

@martinkilbinger martinkilbinger left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks very good to me!

Just a small change to de-claudify the config file comment string.

Comment thread workflow/config.yaml Outdated
@cailmdaley @claude
final_cat_merge: request only the columns the tile's detection mode c…
2905a88 
…an carry

Under tile_detection: unions_catalogue the detection columns are the UNIONS
DR6 catalogue's, which lacks nine SExtractor columns final_cat.param names
(MAG_WIN, MAGERR_WIN, FLUX_AUTO, FLUXERR_AUTO, FLUX_APER, FLUXERR_APER,
SNR_WIN, FWHM_IMAGE, FWHM_WORLD), so every catalogue-mode merge raised.
merge_final_cat.py now takes --tile-detection and subtracts exactly
SEXTRACTOR_ONLY_COLUMNS in that mode; the merge stays strict on the rest.
tests/module/test_final_cat_columns.py runs the converter on the DR6 header
and asserts that list is exactly the requested SExtractor columns DR6 lacks.
The params pin moves for final_cat_merge only.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeQNTBcjTqu6TPktxysovQ
cailmdaley added a commit that referenced this pull request Sep 29, 2026
@cailmdaley @claude
tests: key the retired-key ordering check on either defaults call
2e12ccd 
#897 renames run_config.apply_machine_defaults to apply_defaults; the
ordering assertion now holds on both sides of that merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley cailmdaley mentioned this pull request Sep 29, 2026
Open
cailmdaley and others added 6 commits September 29, 2026 16:16
@cailmdaley @claude
Mask neighbours in catalogue-mode VIGNETs from the DR6 segmentation map
8da837b 
tile_detection: unions_catalogue cut VIGNET straight from the tile image, so
ngmix (which masks where the tile VIGNET is -1e30) masked no neighbours: in
smk-g10 NGMIX_MAG_NOSHEAR ran >1 mag bright of MAG_AUTO for 6.1% of objects,
against 2.7% with SExtractor.

tile_get_catalogue now also fetches CFIS.<tile>.r.seg.fits.fz. The converter
relabels it to the catalogue's NUMBER by the footprint under each object's
centre pixel (relabel_seg: unclaimed footprints -1, a small disc for objects
without a footprint), writes it beside the sexcat as seg-<tile>.fits for
UberSeg, and sets VIGNET pixels on other objects' footprints to -1e30.
Off-image VIGNET pixels are -1e30 too, as SExtractor writes them.

Records detection.catalogue_neighbour_marking.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BeQNTBcjTqu6TPktxysovQ
@cailmdaley @claude
completeness: expect the seg-stamp vignetmaker run under blend_handli…
a4da900 
…ng: uberseg

tile_vignets gains a third vignetmaker run, the segmentation stamps ngmix's
SEG_VIGNET_PATH reads, when SP_BLEND_HANDLING=uberseg. Noise-fill runs
export nothing and expect exactly what they did before. Landing this here
keeps SCRIPT_HASH(completeness.py) at one campaign boundary with #897.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley @claude
Revert "completeness: expect the seg-stamp vignetmaker run under blen…
97c11a7 
…d_handling: uberseg"

UberSeg's seg stamps will ride the tile catalogue as a SEG_VIGNET column, so
there is no extra vignetmaker run to expect. completeness.py is again
byte-identical to 8da837b, keeping SCRIPT_HASH at the value smk-g11 runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley @claude
config.yaml: plainer input_types comment (review suggestion)
a8facf5 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley @claude
Drop the relabelled seg-map file output
bced5a3 
Its only consumer was an UberSeg seg run that no longer exists; the relabelled map is still used in memory for VIGNET neighbour marking.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley @claude
completeness: the catalogue converter writes one file
0ce4060 
The relabelled seg map is no longer written, so expect=2 failed every catalogue-mode tile_detect; a runner-driven test now ties the count to the converter's output.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@cailmdaley
cailmdaley merged commit f477c41 into develop Sep 29, 2026
3 checks passed
@cailmdaley
cailmdaley deleted the feat/unions-catalogue-tile-detection branch September 29, 2026 23:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@martinkilbinger martinkilbinger martinkilbinger left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Adopt Steven-Gwyn universal object IDs for v2

2 participants

@cailmdaley @martinkilbinger