diff --git a/CLAUDE.md b/CLAUDE.md index 01deff1fc..4722a455b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -118,4 +118,40 @@ keep in their own stores outside it. A `.felt/` directory (a markdown "fiber" no store used with the `felt` CLI) is **not tracked here**: it's gitignored, and where it exists it's a machine-local symlink into a private, separately git-synced store, so a fresh clone won't have one. Record durable decisions in the PR, issue, or docs -where the change lives. +where the change lives — and *scientific* decisions in `astra.yaml`, below. + +## Scientific decisions live in `astra.yaml` + +`astra.yaml` at the repo root is the pipeline's decision record: every +consequential scientific choice embedded in the code and committed configs, +with its rationale, alternatives and selected default. `universes/committed.yaml` +pins the option the committed configuration selects for every decision. +`@sc [decision:]` tags live at implementing code/config sites; optional +`Values:` sentences assert committed values within those tagged sites. The +rationale stays in ASTRA, while local `@sc` contracts hold site-specific +constraints. `tests/unit/test_decisions.py` checks tag syntax, bidirectional +decision/site coverage, values, and test `decision` markers. To inspect tags at +an implementation site, run `python -m tests.helpers.decisions [:]`; +`--decision ` lists its sites. The format is ASTRA. + +`uvx astra-tools@0.2.17 guide` is the briefing and +`uvx astra-tools@0.2.17 spec` the field reference. The file's header states +its conventions, including `[HARDCODED]` and `[LINT]`. + +Membership test: a different defensible choice would change which objects enter +the shear catalogue, or the numbers attached to them. Detection thresholds, +masking, star-selection cuts, PSF model degree, ngmix priors and seeding, flag +semantics, completeness gates: in. Workflow policy (manifests, chunking, +allocation, failure reporting, provenance) is out; it lives in the PR and in the +PRD, CosmoStat/shapepipe#848. + +**A scientific change is not finished until the record is.** When a change moves +what the pipeline measures, amend `astra.yaml` in the same PR (add the decision, +or edit its rationale, options, Values and site tags), pin the selected option +in `universes/committed.yaml`, and say so in the PR description. +`tests/unit/test_decisions.py` runs in CI; a scientific change that breaks its +site/value checks or leaves the record stale is unfinished. Before committing: + +```bash +uvx astra-tools@0.2.17 validate +``` diff --git a/astra.yaml b/astra.yaml new file mode 100644 index 000000000..87c844909 --- /dev/null +++ b/astra.yaml @@ -0,0 +1,1989 @@ +# ASTRA record of ShapePipe's scientific decisions: the choices embedded in +# the code and the committed workflow configs (workflow/config/cfis/), why +# they stand, and the alternatives. CLAUDE.md says when to amend it. +# workflow/config/cfis_image_sims/ is not yet in scope: its symlinks into +# cfis/ are governed there; its untagged overlays change input naming, the PSF +# model and some governed values (config_tile_Fe.ini COLNUM/EXP_PREFIX, +# final_cat.param columns). +# +# Conventions (`tests/helpers/decisions.py` implements them and documents the +# full value grammar; `tests/unit/test_decisions.py` enforces them): +# * Put `@sc [decision:]` at every code/config site that implements a +# decision. Decision citations have no id or prose; the record is the source +# of rationale. IDs are bare for top-level decisions and dotted for +# sub-analysis decisions; repeat `decision:` metadata to cite several. +# A local constraint adds a stable id and prose, with optional decision +# metadata: `@sc [decision:,label:] `. +# * In Python, put tags in a def/class docstring or as a comment immediately +# above a module statement. In Snakemake, put a comment immediately above +# the rule/statement. In config, a tag governs the active settings in the +# following paragraph, which ends at the next blank line or `@sc` line. +# Every active key before that boundary is governed. A blank line between a +# tag and its key is an error. To govern a whole section, put the tag +# directly above its `[SECTION]` header; it governs active keys through the +# next section header. A description comment may sit above the tag, and +# `scope:file` governs the whole file. +# * A rationale may end with `Values: = ; ... .`. Every ref must +# resolve to exactly one site tagged for that decision and match its value. +# Use a bare key when it selects one site; qualify it with a path suffix and +# `#` or `::` when needed. `= absent` requires a same-decision tag somewhere +# in the file (in that section or file-wide for INI/SETools) and asserts that +# no active setting exists anywhere in the target scope. INI absence checks +# include keys inherited from `[DEFAULT]`. +# * The checker validates citations in both directions, unique local-contract +# ids, literal values and test `decision` markers. The value grammar is +# documented in the helper module's docstring. +# * A decision's `default` is the option the committed code and configs +# select; universes/committed.yaml pins it. +# * `excluded: true` means considered and rejected. An option the code +# does not implement says so in its description and is not excluded. +# * [HARDCODED]: the committed choice is fixed in code, with no config key. +# [LINT]: the code disagrees with itself or its own documentation. +# * A prior insight repeated inside a sub-analysis carries a `_local` +# suffix, because insight ids are scoped. + +version: "0.0.14" +name: ShapePipe scientific decisions +description: >- + Codebase-level decision record for the ShapePipe weak-lensing pipeline + (UNIONS/CFIS) as orchestrated by workflow/Snakefile. Membership test: a + different defensible choice would change which objects enter the shear + catalogue, or the numbers attached to them. +tags: [shapepipe, weak-lensing, unions, codebase-record] +container: ghcr.io/cosmostat/shapepipe:develop + +inputs: + - id: tile_images + type: data + source: CFIS/UNIONS r-band MegaPipe tile stacks and single-exposure image/weight/flag triplets (workflow/config.yaml) + description: >- + Tiles and the exposures that built them; each exposure carries its + instrument flag image. + - id: healsparse_masks + type: data + source: UNIONS healsparse mask products (e.g. mask_ugriz_nside131072_n4.hsp) + description: >- + Sky-fixed mask maps, built outside ShapePipe and queried at object + positions; how they are used is the masking sub-analysis. + +outputs: + - id: final_cat + type: data + format: fits + description: >- + Per-tile shear catalogue family, the terminal science product + (make_cat_runner). + inputs: [tile_images, healsparse_masks] + decisions: [per_unit_completeness, postage_stamp_size, photometric_zeropoint] + +decisions: + + # ── cross-cutting ──────────────────────────────────────────────────────── + + per_unit_completeness: + label: Per-unit completeness gate + rationale: >- + Every shapepipe_run rule checks its products against a nominal + per-runner count (per chunk for tile_ngmix). A runner short of its count + fails the unit (exposure or tile), so a partial unit never enters the + catalogue and a failed unit's objects are absent. The one tolerated + shortfall is the exposure-side psfex_interp VALIDATION output: a CCD + whose model fails the acceptance gate + (star_selection_psf.psf_acceptance_thresholds) produces nothing and the + unit only warns. The MCCD chain, never run in a campaign, warns on every + runner; its counts follow config_exp_mccd.ini (one mask_query catalogue + per CCD; preprocessing merges an exposure's stars into one train and one + test catalogue). Science-path PSF rejection bypasses this table: + psfex_interp drops the epoch per object inside the tile run. + Values: + COMPLETENESS[exp_psf.psfex.psfex_interp_runner.warn] = True; + COMPLETENESS[exp_psf.mccd.mask_query_runner.expect] = 40; + COMPLETENESS[exp_psf.mccd.mccd_preprocessing_runner.expect] = 2. + default: exact_counts + options: + exact_counts: + label: Nominal count per runner; psfex_interp validation shortfall warns + insights: [des_psf_blacklist, guinot22_star_floor_22] + count_floor: + label: Tolerate recorded attrition down to a per-runner floor + excluded: true + excluded_reason: >- + The floors had no basis: across a 127-exposure, 64-tile campaign + every non-warning runner produced exactly its nominal count, so a + floor below it only admits failed units unremarked. + no_gate: + label: Accept whatever is produced + excluded: true + excluded_reason: >- + A stage producing 2 of 40 CCDs would flow into the catalogue + unremarked. + + postage_stamp_size: + label: Postage-stamp size shared by vignets, ngmix stamps and PSF models + rationale: >- + One size pins three coupled apertures: the SExtractor VIGNET around each + detection, the vignetmaker stamps ngmix fits, and the PSFEx model + stamp. The stamp bounds the measurable galaxy size and truncates the + wings of large galaxies. No rationale for 51 px (about 9.5 arcsec) is + recorded. + Values: + default_noimaflags.param#VIGNET = 51; + default.param#VIGNET = 51; + VIGNETMAKER_RUNNER_RUN_1.STAMP_SIZE = 51; + VIGNETMAKER_RUNNER_RUN_2.STAMP_SIZE = 51; + PSF_SIZE = 51. + default: px_51 + options: + px_51: + label: 51x51 px everywhere + larger_adaptive: + label: Larger or size-adaptive stamps + description: >- + Not implemented; the vignet, stamp and PSF sizes must change + together. + + photometric_zeropoint: + label: Magnitude zero-point convention + rationale: >- + Tiles use a fixed zero-point of 30 (ZP_FROM_HEADER=False), repeated in + ngmix's MAG_ZP; exposures read the per-image header PHOTZP. The fixed + value assumes the MegaPipe stacks are calibrated to 30; nothing in the + repo checks it, but on a sampled exposure (2114045p) FSCALE equals + 10^(-0.4 (PHOTZP - 30)) to 0.02%. Exposure epochs are rescaled by FSCALE, which must agree + with header PHOTZP as well as tile MAG_ZEROPOINT and ngmix MAG_ZP. + Magnitude cuts (the star-selection window, downstream galaxy cuts) + inherit their stage's convention. + Values: + MAG_ZEROPOINT = 30.0; + config_tile_Sx.ini#SEXTRACTOR_RUNNER.ZP_FROM_HEADER = False; + NGMIX_RUNNER.MAG_ZP = 30.0; + config_exp_psfex.ini#SEXTRACTOR_RUNNER.ZP_FROM_HEADER = True; + SEXTRACTOR_RUNNER.ZP_KEY = PHOTZP. + default: fixed_30_tiles_header_exposures + options: + fixed_30_tiles_header_exposures: + label: Tiles fixed 30.0; exposures from header PHOTZP + header_everywhere: + label: Per-image header zero-points on tiles too + description: >- + ZP_FROM_HEADER=True on tiles; changes magnitudes only where a tile's + header zero-point differs from 30. + +prior_insights: + des_psf_blacklist: + claim: >- + DES blacklists a CCD's PSF model rather than failing the exposure: in + Y3, a CCD with fewer than 25 stars surviving outlier rejection is + excluded downstream (~2% of data removed) and processing proceeds. + created_at: "2026-07-16T00:00:00Z" + evidence: + - id: ev_jarvis_y3 + doi: "10.48550/arXiv.2011.03409" + quote: + exact: "we enter it into a" + suffix: " “blacklist” and exclude this CCD" + location: { page: 10 } + guinot22_star_floor_22: + claim: >- + The published ShapePipe/UNIONS analysis applies a per-CCD quality floor + rather than failing whole exposures: a CCD with fewer than 22 selected + stars is discarded for PSF estimation and contributes no epoch to the + shape measurement, while processing continues. + created_at: "2022-04-01T00:00:00Z" + evidence: + - id: ev_guinot22_star_floor + doi: "10.48550/arXiv.2204.04798" + quote: + exact: 'The dashed line represents the cut at 22 stars/CCD below which the CCD is discarded for the PSF estimation.' + location: { page: 4 } + +analyses: + + # ═════════════════════════════════════════════════════════════════════════ + masking: + description: >- + Which masks reach the measurement, and where. ShapePipe generates no + masks; the instrument flag image delivered with each exposure is the + only one that reaches pixels. Sky-fixed masks (star halos and bodies, + manual regions, missing bands) are healsparse maps built and designed + outside ShapePipe, which only queries them at object positions into + catalogue columns that no stage cuts on. + inputs: + - id: exposure_flags + type: data + source: per-CCD instrument flag images split from each exposure (exp_split family) + - id: sky_masks + type: data + source: UNIONS healsparse mask maps + outputs: + - id: masked_measurement_inputs + type: data + format: fits + description: >- + Exposure detection catalogues carrying IMAFLAGS_ISO (and MASK_EXT when + maps are configured), the flag stamps ngmix reads, and the final + catalogue's optional MASK_ columns. + inputs: [exposure_flags, sky_masks] + decisions: [pixel_mask_source, psf_star_mask_veto, sky_mask_application] + decisions: + pixel_mask_source: + label: Only the instrument flag image reaches pixels + rationale: >- + On exposures SExtractor reads the split flag image, producing + IMAFLAGS_ISO, which the PSF star selection requires to be zero. The + multi-epoch vignet run cuts flag stamps from the same image; ngmix + gives flagged pixels weight 0 and drops mostly-flagged epochs + (shape_measurement.defect_fill, + shape_measurement.epoch_masked_fraction_cut). Tiles have no flag + image, so tile detection runs unflagged + (detection.detection_source_mode). Sky-fixed masks never touch + pixels: an object inside a star halo is measured from the same + unmodified pixels as one outside it. + Values: + config_exp_psfex.ini#SEXTRACTOR_RUNNER.FLAG_IMAGE = True; + VIGNETMAKER_RUNNER_RUN_2.ME_IMAGE_PATTERN = flag, image, weight, + background, background_rms. + default: instrument_flags_only + options: + instrument_flags_only: + label: Instrument flags gate pixels; sky masks stay at catalogue level + rasterised_sky_masks: + label: Rasterise healsparse masks into the pixel flags + insights: [farrens22_pipeline_masks] + excluded: true + excluded_reason: >- + Sky-fixed masks say where an object sits, not that its pixels are + corrupted, so acting on them is an analysis decision. + Rasterising would bake one mask version into every shape. + psf_star_mask_veto: + label: PSF-star candidates rejected on instrument flags only + rationale: >- + Of the masks, star selection cuts only on IMAFLAGS_ISO. mask_query sits + between SExtractor and setools in the exposure chain; when + MASK_PATHS names maps it writes MASK_EXT (0 clean, nonzero flagged, + off-coverage clean) onto each CCD's catalogue. MASK_PATHS ships + commented out, so the module passes catalogues through without + MASK_EXT. The intended map is the UNIONS star-body product (bit 2); + halo bits 0 and 1 are left out because halos say nothing about + whether a star is a good PSF sample. + Values: + MASK:star_selection.IMAFLAGS_ISO = "== 0"; + MASK:preselect.IMAFLAGS_ISO = "== 0"; + MASK:flag.IMAFLAGS_ISO = "== 0"; + MASK:star_selection.MASK_EXT = absent; + MASK:preselect.MASK_EXT = absent; + MASK:flag.MASK_EXT = absent. + default: instrument_flags_only + options: + instrument_flags_only: + label: IMAFLAGS_ISO == 0; no sky map queried + star_body_veto: + label: Also reject candidates on the star-body map (MASK_EXT == 0) + description: >- + Set MASK_PATHS to the star-body map and add MASK_EXT == 0 beside + each IMAFLAGS_ISO cut in star_selection.setools (one line per + mask block). Querying without the cut records MASK_EXT and + changes no star. + star_body_and_halo_veto: + label: Also reject candidates inside star halos + excluded: true + excluded_reason: >- + Halos flag objects for the final catalogue; a star inside another + star's halo is not thereby a bad PSF sample. + sky_mask_application: + label: Object-level sky masking deferred downstream + rationale: >- + The final catalogue ships every detected object. When MASK_EXT_PATHS + lists band:path pairs, make_cat queries each healsparse map at the + object's windowed position and writes the map value verbatim into a + MASK_ column; an object off coverage gets the map's sentinel + (False for boolean maps, reading as unmasked; typically -1 for + integer maps). The committed config sets no MASK_EXT_PATHS, so no + mask column is written and all mask cuts happen downstream. + default: deferred_downstream + options: + deferred_downstream: + label: No mask columns; all objects shipped + catalogue_columns: + label: Per-band MASK_ columns from MASK_EXT_PATHS, no cut + pipeline_cut: + label: Drop masked objects inside the pipeline + excluded: true + excluded_reason: >- + Location flags are analysis decisions; a pipeline cut would fix + one mask version into the catalogue. + prior_insights: + farrens22_pipeline_masks: + claim: >- + The published ShapePipe pipeline generated its own masks, including + Messier objects and CCD borders, and applied them to the images; the + current pipeline generates none. + created_at: "2022-06-01T00:00:00Z" + evidence: + - id: ev_farrens22_masks + doi: "10.48550/arXiv.2206.14689" + quote: + exact: 'Messier objects, and border regions.' + location: { page: 2 } + + # ═════════════════════════════════════════════════════════════════════════ + detection: + description: >- + Object detection with SExtractor on r-band tiles (the galaxy sample) and + on single-exposure CCDs (PSF-star candidates). Tiles follow the MegaPipe + parameters of Gwyn's UNIONS tile catalogue; exposures keep ShapePipe's + stock values. + inputs: + - id: tile_stack + type: data + source: MegaPipe r-band tile stack + weight (uncompressed, merged headers) + outputs: + - id: tile_sexcat + type: data + format: fits + description: Per-tile SExtractor LDAC catalogue with per-epoch CCD membership. + inputs: [tile_stack] + decisions: + [detection_threshold_policy, deblending_policy, background_model, + weight_map_usage, zero_weight_interpolation, detection_source_mode, + epoch_membership_ccd_bounds, photometry_parameters, + spurious_detection_cleaning, blend_photometry_mask_type, + saturation_level] + decisions: + detection_threshold_policy: + label: Detection significance, minimum area, matched filter + rationale: >- + Tiles use the MegaPipe tile-catalogue threshold, minimum area and + filter, so ShapePipe's galaxy sample matches the catalogue UNIONS + adopts; the 7x7 Gaussian of FWHM 3 px is near the CFIS average seeing + of 0.65 arcsec (about 3.5 px at 0.187 arcsec/px). Exposures only feed + star selection and keep stock values with the 3x3 FWHM 2 px kernel. + The tiles differ from Guinot+22 in all three. Matching image + simulations must use the same prescription. + Values: + default_tile.sex#DETECT_THRESH = 1.0; + default_tile.sex#ANALYSIS_THRESH = 1.0; + default_tile.sex#DETECT_MINAREA = 3; + default_tile.sex#FILTER = Y; + config_tile_Sx.ini#SEXTRACTOR_RUNNER.DOT_CONV_FILE = + $SP_CONFIG/gauss_3.0_7x7.conv; + default_exp.sex#DETECT_THRESH = 1.5; + default_exp.sex#ANALYSIS_THRESH = 1.5; + default_exp.sex#DETECT_MINAREA = 5; + default_exp.sex#FILTER = Y; + config_exp_psfex.ini#SEXTRACTOR_RUNNER.DOT_CONV_FILE = + $SP_CONFIG/default.conv. + default: megapipe_tiles + options: + megapipe_tiles: + label: MegaPipe values on tiles; stock values on exposures + insights: [guinot22_cfis_seeing] + stock_tiles: + label: Stock ShapePipe values on tiles (1.5 sigma, minarea 5, FWHM 2 px) + insights: [guinot22_sextractor_params] + excluded: true + excluded_reason: >- + Triggers spuriously on about 10% of grid-placed Sersic galaxies + in image simulations, and does not match the MegaPipe tile + catalogue. + deblending_policy: + label: Deblending contrast + rationale: >- + Tiles use the MegaPipe DEBLEND_MINCONT; exposures use a lower one, + the value Guinot+22 lists. Both share DEBLEND_NTHRESH. Contrast sets + object count, centroids, and blend contamination in shapes; + matching image simulations must preserve the stage-specific settings. + Values: + default_tile.sex#DEBLEND_MINCONT = 0.002; + default_exp.sex#DEBLEND_MINCONT = 0.001; + default_tile.sex#DEBLEND_NTHRESH = 32; + default_exp.sex#DEBLEND_NTHRESH = 32. + default: megapipe_tiles + options: + megapipe_tiles: + label: MINCONT 0.002 tiles / 0.001 exposures + insights: [guinot22_sextractor_params] + mincont_5em4_tiles: + label: MINCONT 0.0005 on tiles + excluded: true + excluded_reason: >- + Part of the stock tile parameter set rejected in favour of the + MegaPipe values (see detection_threshold_policy). + background_model: + label: Background estimation and photometric background + rationale: >- + Both passes use SExtractor AUTO backgrounds: tiles with the MegaPipe + mesh and filter sizes and a LOCAL photometric background, exposures + with a finer mesh and a GLOBAL one. ngmix subtracts the exposure + BACKGROUND map from each epoch and weights its pixels by + BACKGROUND_RMS (shape_measurement.galaxy_pixel_weights). The header + background path is off. Residual sky offsets propagate into + thresholds, fluxes, completeness and shapes. + Values: + default_tile.sex#BACK_TYPE = AUTO; + default_tile.sex#BACK_SIZE = 512; + default_tile.sex#BACK_FILTERSIZE = 9; + default_tile.sex#BACKPHOTO_TYPE = LOCAL; + BACKPHOTO_THICK = 30; + default_exp.sex#BACK_TYPE = AUTO; + default_exp.sex#BACK_SIZE = 64; + default_exp.sex#BACK_FILTERSIZE = 3; + default_exp.sex#BACKPHOTO_TYPE = GLOBAL; + config_exp_psfex.ini#SEXTRACTOR_RUNNER.BKG_FROM_HEADER = False; + config_tile_Sx.ini#SEXTRACTOR_RUNNER.BKG_FROM_HEADER = False. + default: auto_megapipe_tiles + options: + auto_megapipe_tiles: + label: AUTO everywhere; MegaPipe mesh and LOCAL photometry on tiles + manual_zero_tiles: + label: Tile background fixed to 0, trusting the stack subtraction + excluded: true + excluded_reason: >- + Part of the stock tile parameter set rejected in favour of the + MegaPipe values (see detection_threshold_policy). + weight_map_usage: + label: Weight map as inverse variance for detection + rationale: >- + MAP_WEIGHT on both passes (SExtractor default NONE): the per-pixel + variance sets the effective SNR and so the detection set. + RESCALE_WEIGHTS and WEIGHT_GAIN keep their SExtractor defaults. + Guinot+22 keeps every non-tabulated parameter at its default, which + would mean no weight map. + Values: + default_tile.sex#WEIGHT_TYPE = MAP_WEIGHT; + default_exp.sex#WEIGHT_TYPE = MAP_WEIGHT; + default_tile.sex#RESCALE_WEIGHTS = Y; + default_exp.sex#RESCALE_WEIGHTS = Y; + default_tile.sex#WEIGHT_GAIN = Y; + default_exp.sex#WEIGHT_GAIN = Y; + config_tile_Sx.ini#SEXTRACTOR_RUNNER.WEIGHT_IMAGE = True; + config_exp_psfex.ini#SEXTRACTOR_RUNNER.WEIGHT_IMAGE = True. + default: map_weight + options: + map_weight: + label: MAP_WEIGHT + no_weight: + label: No weight map (SExtractor default) + insights: [guinot22_sextractor_params] + zero_weight_interpolation: + label: Interpolation across zero-weight pixels + rationale: >- + INTERP_TYPE ALL on both passes (SExtractor default NONE): SExtractor + invents flux across zero-weight pixels, changing detections and + photometry near masked regions. Matching image simulations must use + the same interpolation operator; this is not a calibrated pixel repair. + Values: + default_tile.sex#INTERP_TYPE = ALL; + default_exp.sex#INTERP_TYPE = ALL; + default_tile.sex#INTERP_MAXXLAG = 16; + default_tile.sex#INTERP_MAXYLAG = 16; + default_exp.sex#INTERP_MAXXLAG = 16; + default_exp.sex#INTERP_MAXYLAG = 16. + default: interp_all + options: + interp_all: + label: INTERP_TYPE ALL + no_interpolation: + label: INTERP_TYPE NONE + spurious_detection_cleaning: + label: Cleaning of spurious detections + rationale: >- + CLEAN on both passes deletes detections + consistent with being wings of a brighter neighbour, a post-deblend + change to the object list. Keep the same cleaning prescription in + matching image simulations. + Values: + default_tile.sex#CLEAN_PARAM = 1.0; + default_exp.sex#CLEAN_PARAM = 1.0; + default_tile.sex#CLEAN = Y; + default_exp.sex#CLEAN = Y. + default: clean_1 + options: + clean_1: + label: CLEAN Y, CLEAN_PARAM 1.0 + blend_photometry_mask_type: + label: Neighbour pixels in blend photometry + rationale: >- + MASK_TYPE CORRECT on both passes replaces neighbour pixels by their + mirror across the object centre during photometry, changing blend + fluxes and windowed moments. This SExtractor photometry choice is + separate from ngmix's neighbour-pixel weighting decision. + Values: + default_tile.sex#MASK_TYPE = CORRECT; + default_exp.sex#MASK_TYPE = CORRECT. + default: correct + options: + correct: + label: MASK_TYPE CORRECT + blank: + label: MASK_TYPE BLANK + saturation_level: + label: Saturation level read from each image's header + rationale: >- + Both passes take each image's saturation level from its SATURATE + header card and set FLAGS bit 4 on objects with saturated pixels. + Star selection requires FLAGS == 0 at every step, so the level decides + which bright stars leave the PSF sample; tile FLAGS reach the + catalogue for downstream cuts. Where the card is absent, SExtractor + falls back to its built-in SATUR_LEVEL (50000 ADU per its + documentation), since neither .sex file sets one. Delivered data + carry the card: a sampled tile (CFIS.186.307) has SATURATE 9558.7 + and every CCD of a sampled exposure (2114045p) 65535, and split_exp + keeps each CCD's header. Changing the card or pinning a fixed level requires + checking the resulting bright-star selection. + Values: + default_exp.sex#SATUR_KEY = SATURATE; + default_tile.sex#SATUR_KEY = SATURATE; + MASK:star_selection.FLAGS = "== 0"; + MASK:preselect.FLAGS = "== 0"; + MASK:flag.FLAGS = "== 0"; + default_tile.sex#SATUR_LEVEL = absent; + default_exp.sex#SATUR_LEVEL = absent. + default: header_saturate + options: + header_saturate: + label: SATURATE header card; SExtractor default level where absent + fixed_level: + label: Fixed SATUR_LEVEL set in the .sex files + photometry_parameters: + label: Kron and aperture photometry definitions + rationale: >- + Stock Kron, aperture and flux-fraction settings on both passes; no + rationale recorded. MAG_AUTO is the axis of the star-selection + magnitude window and the catalogue magnitude; FLUX_AUTO is PSFEx's + photometric normalisation. A different Kron factor shifts magnitudes + and so every magnitude-based cut. The reported apertures and flux- + fraction measurements must also match the simulated catalogue. + Values: + default_tile.sex#PHOT_AUTOPARAMS = 2.5,3.5; + default_exp.sex#PHOT_AUTOPARAMS = 2.5,3.5; + default_tile.sex#PHOT_APERTURES = 5; + default_exp.sex#PHOT_APERTURES = 5; + default_tile.sex#PHOT_FLUXFRAC = 0.5; + default_exp.sex#PHOT_FLUXFRAC = 0.5; + PHOTFLUX_KEY = FLUX_AUTO. + default: kron_25_35 + options: + kron_25_35: + label: Kron 2.5/3.5, aperture 5 px, FLUXFRAC 0.5 + detection_source_mode: + label: Single-image, unflagged detection on the r-band tile + rationale: >- + No detection image, no flag image, and the default_noimaflags.param + column list: tiles have no instrument flag image and no detection + coadd exists, so detection sees every tile pixel and the tile + catalogue carries no IMAFLAGS_ISO. [LINT] + final_cat.param, read by the post-processing merge, requests + IMAFLAGS_ISO, which the tile chain never produces (issue #912). + Values: + SEXTRACTOR_RUNNER.DETECTION_IMAGE = False; + SEXTRACTOR_RUNNER.FLAG_IMAGE = False. + default: sx_nomask_single_image + options: + sx_nomask_single_image: + label: Unflagged single-image r-band detection + insights: [guinot22_stacked_detection] + masked_tile: + label: Detection on a tile masked by rasterised sky masks + description: >- + Not implemented; no tile pixel mask exists (see + masking.pixel_mask_source). + dual_image_coadd: + label: Dual-image mode with a detection coadd + description: Not implemented; no UNIONS detection coadd exists. + epoch_membership_ccd_bounds: + label: Which exposure CCDs an object belongs to (N_EPOCH) + rationale: >- + `all_world2pix(..., 0)` returns 0-based pixels, and the strict test + accepts `33 < x < 2080`. In 1-based FITS coordinates this is x=35 + through x=2080 inclusive: it trims DATASEC columns 33 and 34 but + admits its upper endpoint, column 2080. A WCS inversion failure skips + that CCD, lowering N_EPOCH; this sets how many exposures enter each + galaxy's multi-epoch fit. + Values: + SEXTRACTOR_RUNNER.CCD_SIZE = 33,2080,1,4612; + SEXTRACTOR_RUNNER.MAKE_POST_PROCESS = True. + default: trimmed_bounds_33_2080 + options: + trimmed_bounds_33_2080: + label: "x in (33,2080), y in (1,4612), strict" + inclusive_bounds: + label: Same bounds, inclusive + prior_insights: + guinot22_sextractor_params: + claim: >- + The published tile detection uses DETECT_THRESH 1.5, DETECT_MINAREA + 10, the default 3x3 filter and DEBLEND_MINCONT 0.001, with every + other SExtractor parameter at its default. + created_at: "2022-04-01T00:00:00Z" + evidence: + - id: ev_guinot22_table2 + doi: "10.48550/arXiv.2204.04798" + quote: + exact: 'Table 2. SExtractor parametrisation. All other parameters are kept to their default values.' + location: { page: 5 } + - id: ev_guinot22_mincont + doi: "10.48550/arXiv.2204.04798" + quote: + exact: 'DEBLEND_MINCONT 0.001' + location: { page: 5 } + - id: ev_guinot22_minarea + doi: "10.48550/arXiv.2204.04798" + quote: + exact: 'DETECT_MINAREA 10' + location: { page: 5 } + guinot22_cfis_seeing: + claim: >- + CFIS r-band data have an average seeing of 0.65 arcsec. + created_at: "2022-04-01T00:00:00Z" + evidence: + - id: ev_guinot22_seeing + doi: "10.48550/arXiv.2204.04798" + quote: + exact: 'is taking r-band data with an average seeing of 0.65 arcsec' + location: { page: 1 } + guinot22_stacked_detection: + claim: >- + Source extraction in the published analysis is performed on the + stacked tile images, for signal-to-noise and because artefacts are + suppressed relative to single exposures. + created_at: "2022-04-01T00:00:00Z" + evidence: + - id: ev_guinot22_stacked_detection + doi: "10.48550/arXiv.2204.04798" + quote: + exact: 'We do the extraction on stacked images which provide a better signal-to-noise ratio, and most artifacts have a reduced amplitude with respect to single exposures' + location: { page: 5 } + + # ═════════════════════════════════════════════════════════════════════════ + preparation: + description: >- + How exposures, astrometry, epoch lists and stamps are prepared before + anything is measured. + inputs: + - id: exposure_files + type: data + source: delivered CFIS exposure triplets (image/weight/flag MEF) + tile stacks + outputs: + - id: epoch_stamps + type: data + format: sqlite + description: Per-object multi-epoch vignets + per-CCD WCS log feeding ngmix. + inputs: [exposure_files] + decisions: + [astrometric_solution_source, ccd_split_extent, + epoch_provenance_from_tile_history, object_position_columns, + stamp_positioning_and_padding] + decisions: + astrometric_solution_source: + label: Astrometry taken verbatim from delivered per-CCD headers + rationale: >- + split_exp builds WCS(header) from each raw CCD header and + merge_headers stores them; every downstream world-to-pixel transform + (stamp placement, epoch membership, position seeding) uses that + solution. [HARDCODED] no astrometric re-derivation or refinement + exists; a joint re-fit would move every stamp centre and position seed. + default: delivered_headers + options: + delivered_headers: + label: WCS(header) verbatim, stored at split time + insights: [guinot22_gaia_astrometry] + astrometric_refit: + label: Joint astrometric re-solution + description: Not implemented. + ccd_split_extent: + label: All 40 MegaCam HDUs split and carried as candidate epochs + rationale: >- + Any HDU count other than N_HDU raises; every CCD is a candidate + epoch wherever the WCS lands it. + Values: + SPLIT_EXP_RUNNER.N_HDU = 40. + default: all_40_hdus + options: + all_40_hdus: + label: 40 HDUs, hard-fail on any other count + insights: [guinot22_forty_chips] + exclude_ccd_subset: + label: Exclude a subset of CCDs as epochs + description: Not implemented. + epoch_provenance_from_tile_history: + label: Epoch sets parsed from tile FITS HISTORY cards + rationale: >- + The coadd's own provenance is the epoch list: the file names in + column COLNUM of each HISTORY line, stripped of their full extension + and deduplicated. A tile whose header cannot be read fails rather + than yielding an empty list. Names keep their trailing p (2243881p), + an epoch letter rather than a prefix, so EXP_PREFIX is blank; + downstream code drops the p when it needs the bare exposure ID. A + mis-parse changes N_EPOCH and which exposures are fit. + Values: + FIND_EXPOSURES_RUNNER.COLNUM = 3. + default: history_parse + options: + history_parse: + label: HISTORY column 3, no prefix stripped, deduplicated + object_position_columns: + label: Windowed centroids (XWIN/YWIN) define every position + rationale: >- + PSF interpolation sites, tile and multi-epoch stamp centres, and the + catalogue position all use SExtractor's windowed centroid: in pixels + for tile stamps and exposure-side PSF validation, in world + coordinates for tile-side PSF interpolation and multi-epoch stamps. + Windowed, isophotal and model centroids differ systematically for + blends and asymmetric galaxies, and the centroid feeds the position + seed and the centroid prior. + Values: + VIGNETMAKER_RUNNER_RUN_1.POSITION_PARAMS = XWIN_IMAGE,YWIN_IMAGE; + VIGNETMAKER_RUNNER_RUN_1.COORD = PIX; + config_tile_PiViVi_psfex.ini#PSFEX_INTERP_RUNNER.POSITION_PARAMS = + XWIN_WORLD,YWIN_WORLD; + VIGNETMAKER_RUNNER_RUN_2.POSITION_PARAMS = XWIN_WORLD,YWIN_WORLD; + VIGNETMAKER_RUNNER_RUN_2.COORD = SPHE; + config_exp_psfex.ini#PSFEX_INTERP_RUNNER.POSITION_PARAMS = + XWIN_IMAGE,YWIN_IMAGE. + default: xwin_windowed + options: + xwin_windowed: + label: Windowed centroids everywhere + stamp_positioning_and_padding: + label: Nearest-pixel stamp extraction with zero padding + rationale: >- + [HARDCODED] stamps are cut around the pixel nearest the object's + position, with no sub-pixel interpolation; the sub-pixel remainder + is stored as the stamp's OFFSET, which ngmix uses as the Jacobian + origin (shape_measurement.centroid_source), so extraction and + centroid prior share one rounding. Multi-epoch stamps map the tile + world coordinate through the stored per-CCD WCS. Stamps overrunning + an image edge are kept, zero-filled outside the image; a stamp + centre that rounds outside the image raises and fails the vignet + run for the whole tile. + default: round_and_zero_pad + options: + round_and_zero_pad: + label: Nearest-pixel extraction, zero padding, no edge rejection + prior_insights: + guinot22_gaia_astrometry: + claim: >- + The astrometric solution the analysis relies on is the upstream + MegaPipe/Gaia DR2 calibration, accurate to within 20 mas; no + astrometric re-fit inside the pipeline is described. + created_at: "2022-04-01T00:00:00Z" + evidence: + - id: ev_guinot22_astrometry + doi: "10.48550/arXiv.2204.04798" + quote: + exact: 'An astrometric calibration within 20 mas was achieved using the Gaia DR2 observations' + location: { page: 2 } + guinot22_forty_chips: + claim: >- + Star selection and PSF modelling are carried out independently on + each of the 40 MegaCam chips, with no chip excluded. + created_at: "2022-04-01T00:00:00Z" + evidence: + - id: ev_guinot22_forty_chips + doi: "10.48550/arXiv.2204.04798" + quote: + exact: 'is performed independently on each of the 40 chips that constitute the MegaCAM' + location: { page: 3 } + + # ═════════════════════════════════════════════════════════════════════════ + star_selection_psf: + description: >- + Which objects constrain the PSF, the PSF model itself, and which CCD + models are good enough to use. + inputs: + - id: exposure_sexcat + type: data + source: per-CCD exposure SExtractor catalogues (default_exp.sex run) + outputs: + - id: psf_model + type: data + format: psf + description: >- + Per-CCD PSF models and the PSFs interpolated at object positions, + with HSM shape diagnostics. + inputs: [exposure_sexcat] + decisions: + [star_selection_box, psf_train_validation_split, + psfex_candidate_vetting, psf_modelling_software, + psf_model_complexity, psf_acceptance_thresholds] + decisions: + star_selection_box: + label: Stellar-locus selection, magnitude window and FWHM window around the mode + rationale: >- + Stars are flag-free objects in a MAG_AUTO window whose FWHM lies + within 0.2 px of the mode of a looser, size-limited preselection + (FWHM 0.3-1.5 arcsec at 0.187 arcsec/px). The mode estimator is an + iterative histogram zoom that falls back to the median below 20 + objects, so small-N behaviour changes selection on sparse CCDs. + PSFEx's own selection is off; see psfex_candidate_vetting for what + PSFEx may still apply. + Values: + MASK:star_selection.FLAGS = "== 0"; + MASK:preselect.FLAGS = "== 0"; + MASK:star_selection.IMAFLAGS_ISO = "== 0"; + MASK:preselect.IMAFLAGS_ISO = "== 0"; + MASK:star_selection.MAG_AUTO = ["> 18.", "< 22."]; + MASK:star_selection.FWHM_IMAGE = ["<= mode(FWHM_IMAGE{preselect}) + + 0.2", ">= mode(FWHM_IMAGE{preselect}) - 0.2"]; + MASK:preselect.MAG_AUTO = ["> 0", "< 21"]; + MASK:preselect.FWHM_IMAGE = ["> 0.3 / 0.187", "< 1.5 / 0.187"]; + PLOT:fwhm_field.SCATTER = "FWHM_IMAGE{star_selection}*0.187"; + SAMPLE_AUTOSELECT = N. + default: mode_centred_box + options: + mode_centred_box: + label: FWHM-mode-centred box, +-0.2 px, mag 18-22 + insights: [guinot22_star_box] + size_mag_locus_fit: + label: Fitted size-magnitude stellar locus + description: Not implemented. + psfex_autoselect: + label: PSFEx SAMPLE_AUTOSELECT vetting on top + insights: [guinot22_psfex_preselection_off] + excluded: true + excluded_reason: >- + Disabled so that the pipeline's own star selection is the only + one. + psf_train_validation_split: + label: Seeded 80/20 star split, model fit vs held-out validation + rationale: >- + The 80% sample fits the PSFEx model and feeds the tile multi-epoch + interpolation; the 20% sample is the independent residual + diagnostic (psfex_interp VALIDATION mode). The split trades + training stars per CCD, which interacts with the acceptance gate, + against an independent residual test. The permutation is seeded + from the digits of the unit's file number, so a CCD gets the same + split on every run. + Values: + RAND_SPLIT:star_split.RATIO = 20; + PSFEX_RUNNER.FILE_PATTERN = star_split_ratio_80; + PSFEX_INTERP_RUNNER.FILE_PATTERN = + star_split_ratio_80,star_split_ratio_20,psfex_cat; + PSFEX_INTERP_RUNNER.ME_DOT_PSF_PATTERN = star_split_ratio_80. + default: split_80_20_seeded + options: + split_80_20_seeded: + label: 80% train / 20% validation, seeded from the file number + insights: [guinot22_star_split] + split_80_20_unseeded: + label: Same split from an unseeded random draw + excluded: true + excluded_reason: >- + Makes the PSF star sample, and every shape downstream of it, + irreproducible run-to-run. + no_holdout: + label: All stars in the model, no held-out diagnostic + excluded: true + excluded_reason: Loses the independent residual and rho-statistic input. + psfex_candidate_vetting: + label: PSFEx built-in candidate cuts, unpinned + rationale: >- + default.psfex sets SAMPLE_AUTOSELECT N (compiled default Y) and + omits every other SAMPLE_* key, so [HARDCODED] PSFEx's compiled + defaults govern them and can change with its version. psfex -dd + (PSFEx 3.21.1, develop-runtime image) gives SAMPLE_FWHMRANGE + 2.0,10.0, SAMPLE_VARIABILITY 0.2, SAMPLE_MINSN 20, SAMPLE_MAXELLIP + 0.3, SAMPLE_FLAGMASK 0x00fe, SAMPLE_WFLAGMASK 0x0000 and + SAMPLE_IMAFLAGMASK 0x0. Per the PSFEx source (not re-read here), + MINSN, MAXELLIP, FLAGMASK, FWHMRANGE and VARIABILITY still cut + candidates with autoselect off; SETools cuts neither S/N nor + ellipticity, so MINSN and MAXELLIP can reject stars it kept. + BADPIXEL_FILTER N and PSF_RECENTER N, also compiled defaults, leave + flagged star vignets unfiltered and candidates unrecentred. + Compiled-in values admit no `= value` assertion, so the anchors + assert only that each SAMPLE_* key is absent; open issue #919 would + pin them in default.psfex. + Values: + SAMPLE_AUTOSELECT = N; + BADPIXEL_FILTER = N; + PSF_RECENTER = N; + SAMPLE_FWHMRANGE = absent; + SAMPLE_VARIABILITY = absent; + SAMPLE_MINSN = absent; + SAMPLE_MAXELLIP = absent; + SAMPLE_FLAGMASK = absent; + SAMPLE_WFLAGMASK = absent; + SAMPLE_IMAFLAGMASK = absent. + default: builtin_defaults + options: + builtin_defaults: + label: PSFEx 3.21.1 built-in SAMPLE_* values, no bad-pixel filter + pinned_explicit: + label: Write the SAMPLE_* values explicitly into default.psfex + psf_modelling_software: + label: PSF model, PSFEx per CCD or MCCD over the focal plane + rationale: >- + psf_model in workflow/config.yaml's machines: table (per machine and + input type; image sims use fake) selects the exposure and tile + config pair; the committed value is psfex, which fits each CCD + independently. MCCD (Liaudat+2021) fits one hybrid local+global + model over the focal plane; the completeness table treats its + counts as warnings because no campaign has run it. Up to the model, + its exposure chain matches PSFEx's: SExtractor reads the split + image, weight and instrument flag directly, and mask_query sits + between SExtractor and setools. The selected model name also flows + through SP_PSF in downstream inputs, so producer and consumer choices + must move together. + Values: + INSTANCE.N_COMP_LOC = 8; + INSTANCE.D_COMP_GLOB = 8; + INSTANCE.FP_GEOMETRY = CFIS; + INSTANCE.RMSE_THRESH = 1.25; + INPUTS.MIN_N_STARS = 20; + FIT.LOC_MODEL = hybrid; + config_exp_mccd.ini#EXECUTION.MODULE = + sextractor_runner,mask_query_runner,setools_runner,mccd_preprocessing_runner,mccd_fit_val_runner,merge_starcat_runner,mccd_plots_runner; + SEXTRACTOR_RUNNER.INPUT_DIR = + $SP_RUN/output/run_sp_exp_Sp/split_exp_runner/output; + SEXTRACTOR_RUNNER.FILE_PATTERN = image,weight,flag; + SEXTRACTOR_RUNNER.FLAG_IMAGE = True. + default: psfex + options: + psfex: + label: PSFEx, independent per-CCD models + insights: [guinot22_psfex_software, farrens22_two_psf_methods] + mccd_focal_plane: + label: MCCD hybrid local+global focal-plane model + insights: [farrens22_two_psf_methods] + psf_model_complexity: + label: PSFEx pixel basis with degree-2 spatial variation per CCD + rationale: >- + A pixel basis with polynomial variation in XWIN/YWIN, fit per CCD + at native sampling. PSF_ACCURACY is the fractional accuracy PSFEx + assumes for PSF pixel values; per the PSFEx documentation it enters + the fit weights, and so how closely the model follows bright stars. + Model flexibility trades overfitting against PSF leakage, the + dominant additive systematic in cosmic shear. A more flexible model + must be supported by the surviving training stars and checked on + held-out residuals under the acceptance gate; successful optimisation + alone does not establish a usable PSF. + Stock values; no rationale recorded. + Values: + BASIS_TYPE = PIXEL; + PSF_ACCURACY = 0.01; + BASIS_NUMBER = 20; + PSFVAR_DEGREES = 2; + PSF_SAMPLING = 1; + PSFVAR_KEYS = XWIN_IMAGE,YWIN_IMAGE; + PSFVAR_GROUPS = 1,1; + MEF_TYPE = INDEPENDENT. + default: pixel_basis_deg2_per_ccd + options: + pixel_basis_deg2_per_ccd: + label: PIXEL basis, degree 2, per CCD + insights: [guinot22_psf_no_oversampling] + deg3: + label: Degree-3 spatial variation + description: Needs more stars per CCD than the acceptance gate guarantees. + psf_acceptance_thresholds: + label: Per-CCD PSF-model quality gate + rationale: >- + A CCD whose model has ACCEPTED < STAR_THRESH or CHI2 > CHI2_THRESH + is not interpolated on either the validation or multi-epoch pass. + ACCEPTED counts stars that survive PSFEx's own sample cuts; PSFEx + fits the 80% training split, so STAR_THRESH applies the published + 22-star floor to that accepted count. In the science path the CCD's + epoch is dropped for every object on it, and an object left with no + epoch has no shape. + The pipeline has no minimum-epoch floor: NGMIX_N_EPOCH records what + survived and epoch-count cuts happen downstream. Guinot+22 discards + the CCD from PSF estimation rather than gating at interpolation. + Values: + config_exp_psfex.ini#PSFEX_INTERP_RUNNER.STAR_THRESH = 22; + config_exp_psfex.ini#PSFEX_INTERP_RUNNER.CHI2_THRESH = 2; + config_tile_PiViVi_psfex.ini#PSFEX_INTERP_RUNNER.STAR_THRESH = 22; + config_tile_PiViVi_psfex.ini#PSFEX_INTERP_RUNNER.CHI2_THRESH = 2. + default: stars22_chi2_2 + options: + stars22_chi2_2: + label: ">= 22 stars and chi2 <= 2 on both passes" + insights: [des_psf_blacklist_local, guinot22_star_floor_22_local] + stars20_science_path: + label: ">= 20 stars on the science path" + excluded: true + excluded_reason: >- + A 20-star science-path threshold would pass models with 20 or 21 + PSFEx-accepted training stars, below the 22-star floor retained + on the validation path. + des_25: + label: DES Y3 threshold (25 stars) + excluded: true + excluded_reason: >- + CFIS CCDs are smaller than DECam's; the floor is survey-specific. + prior_insights: + des_psf_blacklist_local: + claim: >- + DES Y3 blacklists any CCD with fewer than 25 stars surviving + outlier rejection in the PSF fit. + created_at: "2026-07-16T00:00:00Z" + evidence: + - id: ev_jarvis_y3_local + doi: "10.48550/arXiv.2011.03409" + quote: + exact: "fewer than 25 stars survived the outlier rejection" + location: { page: 10 } + guinot22_star_floor_22_local: + claim: >- + The published ShapePipe/UNIONS analysis discards a CCD from the PSF + estimation when fewer than 22 stars are selected on it. + created_at: "2022-04-01T00:00:00Z" + evidence: + - id: ev_guinot22_star_floor_local + doi: "10.48550/arXiv.2204.04798" + quote: + exact: 'The dashed line represents the cut at 22 stars/CCD below which the CCD is discarded for the PSF estimation.' + location: { page: 4 } + guinot22_star_box: + claim: >- + The published star selection keeps objects whose FWHM lies within + 0.04 arcsec of the mode of a size preselection, restricted to the + magnitude range 18 < r < 22. + created_at: "2022-04-01T00:00:00Z" + evidence: + - id: ev_guinot22_star_box + doi: "10.48550/arXiv.2204.04798" + quote: + exact: 'From this pre-selection we keep objects for which the FWHM is within 0.04 arcsec of the mode. In addition to these size cuts, we only use star candidates in the magnitude range 18 < r < 22.' + location: { page: 4 } + guinot22_star_split: + claim: >- + The published analysis randomly splits the star sample in two, 80% + building the PSF model and 20% held out for the validation tests. + created_at: "2022-04-01T00:00:00Z" + evidence: + - id: ev_guinot22_star_split + doi: "10.48550/arXiv.2204.04798" + quote: + exact: 'To be able to perform these tests properly, our star sample has been randomly divided in two:' + location: { page: 7 } + guinot22_psfex_preselection_off: + claim: >- + PSFEx's internal pre-selection is disabled because the pipeline + does its own star selection; the PSF is fit on the entire star + sample. + created_at: "2022-04-01T00:00:00Z" + evidence: + - id: ev_guinot22_preselection_off + doi: "10.48550/arXiv.2204.04798" + quote: + exact: 'Since we carry out our own star selection (see Sect. 4.1), we disable the internal PSFEx pre-selection, and the PSF is thus obtained using the entire star sample.' + location: { page: 4 } + guinot22_psfex_software: + claim: >- + The published UNIONS shear catalogue uses PSFEx for PSF modelling, + with MCCD named as upcoming rather than current work. + created_at: "2022-04-01T00:00:00Z" + evidence: + - id: ev_guinot22_psfex_software + doi: "10.48550/arXiv.2204.04798" + quote: + exact: 'We make use of the PSFEx software package' + location: { page: 4 } + farrens22_two_psf_methods: + claim: >- + ShapePipe ships two PSF modelling methods, PSFEx and MCCD, either or + both of which may be run and used for the galaxy shape measurement. + created_at: "2022-06-01T00:00:00Z" + evidence: + - id: ev_farrens22_two_psf + doi: "10.48550/arXiv.2206.14689" + quote: + exact: 'ShapePipe allows either or both methods to be run and subsequently used for the galaxy shape measurement.' + location: { page: 3 } + guinot22_psf_no_oversampling: + claim: >- + The PSFEx parametrisation is tabulated in the paper (pixel basis, + degree-2 spatial variation), with the deliberate choice not to + over-sample the PSF models. + created_at: "2022-04-01T00:00:00Z" + evidence: + - id: ev_guinot22_psf_complexity + doi: "10.48550/arXiv.2204.04798" + quote: + exact: 'The PSFEx parameters we used are presented in Table 1. We have chosen not to over-sample the PSF models.' + location: { page: 5 } + + # ═════════════════════════════════════════════════════════════════════════ + shape_measurement: + description: >- + Joint multi-epoch ngmix Gaussian fits with metacalibration. Most + choices are fixed in code, so silent defaults concentrate here. + inputs: + - id: vignets + type: data + source: multi-epoch galaxy, weight, flag, background and background-RMS vignets + interpolated PSFs + outputs: + - id: ngmix_cat + type: data + format: fits + description: Per-tile metacal shear catalogue chunks (ngmix_runner family). + inputs: [vignets] + decisions: + [ngmix_seed_mode, galaxy_model, fit_initialisation, fit_priors, + metacal_scheme, centroid_source, epoch_flux_rescaling, + psf_epoch_averaging, galaxy_pixel_weights, psf_likelihood_noise, + megacam_ccd_flip, defect_fill, blend_handling, + epoch_masked_fraction_cut, central_defect_veto] + decisions: + ngmix_seed_mode: + label: Per-object RNG seeded from sky position + rationale: >- + Each object's RNG (noise realisations, guesses, priors) is seeded + from its position: [HARDCODED] 3-arcsec sky boxes offset by the + first epoch's CCD number, folded and Cantor-paired mod 2^32. Results + therefore do not depend on how a tile is chunked, and metacal's + fixnoise counter-noise cancels across image-simulation branches that + share an object's box. + default: position_seed + options: + position_seed: + label: Per-object seed from (ra, dec, ccd), 3-arcsec boxes + tile_seed: + label: One tile-wide RandomState consumed in object order + excluded: true + excluded_reason: >- + Makes every noise stream depend on chunk boundaries and object + order. + galaxy_model: + label: Single-Gaussian galaxy and PSF models + rationale: >- + [HARDCODED] ngmix Fitter(model='gauss') for galaxy and PSF. The + standard defence, that model bias largely cancels in the metacal + response, is not stated in the code. + default: gauss + options: + gauss: + label: Single Gaussian + insights: [guinot22_gaussian_model] + exp_or_bdf: + label: exp / bdf galaxy models + description: Not exposed; make_runners fixes the model. + fit_initialisation: + label: Fit guesses and retries + rationale: >- + [HARDCODED] the galaxy guesser (TPSFFluxAndPriorGuesser) starts + from T = 0.25 and a PSF-flux fit; the PSF guesser (TFluxGuesser) + from T = 0.25 and the catalogue flux. The galaxy runner tries 5 + times, the PSF runner twice. With a non-convex likelihood, guess and + retries decide which objects converge. A failed fit is flagged; an + exception during an object's fit drops it with no ngmix row, leaving + it to the catalogue sentinels (catalogue_assembly.failure_sentinels). + default: prior_guess_t025_ntry5_2 + options: + prior_guess_t025_ntry5_2: + label: T guess 0.25; PSF-flux (galaxy) / catalogue flux (PSF); ntry 5 / 2 + hsm_initialisation: + label: Guesses from HSM adaptive moments (Guinot+22) + insights: [guinot22_hsm_initialisation] + description: Not implemented. + fit_priors: + label: ngmix joint prior + rationale: >- + [HARDCODED] ellipticity GPriorBA with sigma 0.4; flat T and F + priors with negative support (the bounds decide which noisy fits + survive and which rail); a centroid prior one pixel scale wide + (PIXEL_SCALE). The PSF fitter is built with the same joint prior. + Prior width drives noise bias; no rationale recorded. Guinot+22 used + a wider flux prior and a flat prior on r50 rather than T. [LINT] star + selection uses 0.187 arcsec/px, while NGMIX_RUNNER.PIXEL_SCALE pins + 0.186. If the key is absent, pixel_scale_from_wcs raises + AttributeError: it calls `.values()` on the first merged-header + value, but merge_headers inserts TILE_ID as a string before the + exposure entries, which split_exp stores as object ndarrays. The + fallback expects nested mappings; its docstring says it reads the + tile WCS. The key cannot be dropped until that fallback is fixed. + The ngmix_runner comment's noise-window claim is stale: get_noise is + never called. + Values: + get_prior.T_range = -1,1e3; + get_prior.F_range = -100,1e9; + NGMIX_RUNNER.PIXEL_SCALE = 0.186. + default: gpriorba04_flat + options: + gpriorba04_flat: + label: GPriorBA 0.4 + flat T/F with negative support + insights: [guinot22_ngmix_priors] + nonneg_informative: + label: Non-negative or informative T/F priors + excluded: true + excluded_reason: >- + Truncating negative support biases the noshear ensemble mean; + metacal wants a symmetric noise response. + metacal_scheme: + label: Metacalibration protocol + rationale: >- + [HARDCODED] the five metacal types and step, fixnoise with the noise + image, and ignore_failed_psf (an epoch whose PSF fit fails is + dropped). The reconvolution kernel, METACAL_PSF, defaults to + fitgauss and is unset in the committed config; it moves the response + directly. No sheared-PSF types run, so the catalogue has no PSF + response term. + Values: + METACAL_TYPES = noshear,1p,1m,2p,2m; + do_ngmix_metacal.metacal_pars[types] = noshear,1p,1m,2p,2m; + do_ngmix_metacal.metacal_pars[step] = 0.01; + do_ngmix_metacal.metacal_pars[fixnoise] = True; + do_ngmix_metacal.metacal_pars[use_noise_image] = True. + default: five_types_step001_fitgauss + options: + five_types_step001_fitgauss: + label: noshear+1p/1m/2p/2m, step 0.01, fitgauss, fixnoise + insights: [guinot22_metacal_five_images] + with_psf_response: + label: Add sheared-PSF types for R_psf + description: Not implemented. + centroid_source: + label: Jacobian origin at the coadd centroid + rationale: >- + The Jacobian origin, where the centroid prior centres, is the + sub-pixel offset the stamp extractor stored when cutting the stamp + ("wcs", the default at every level; the committed config sets no + CENTROID_SOURCE). Extraction and prior share one projection and one + rounding, so they cannot disagree near a rounding tie. + "hsm" re-centres on adaptive moments measured from the stamp. + default: wcs + options: + wcs: + label: Coadd-centroid offset from the stamp extractor + hsm: + label: HSM adaptive-moment centroid + excluded: true + excluded_reason: >- + Noisy, notably for stars, and follows the light rather than the + astrometry that placed the stamp. + epoch_flux_rescaling: + label: Epochs put on a common flux scale by header FSCALE + rationale: >- + [HARDCODED] each epoch's image is multiplied by its header FSCALE and + its weight divided by FSCALE squared (background RMS scaled with the + image) before the joint fit, so the epochs share the tile's + zero-point. + default: fscale + options: + fscale: + label: Rescale by header FSCALE + psf_epoch_averaging: + label: Catalogue PSF quantities averaged over epochs by galaxy weight + rationale: >- + [HARDCODED] the catalogue PSF shape and size (original image PSF + and metacal reconvolution kernel) are epoch averages weighted by each + epoch's summed galaxy inverse variance; epochs whose PSF fit failed + are left out. These columns feed downstream PSF-leakage estimates. + default: galaxy_weight_sum + options: + galaxy_weight_sum: + label: Weighted by summed galaxy inverse variance + galaxy_pixel_weights: + label: Per-pixel inverse variance from background-RMS vignets + rationale: >- + With BKG_RMS_VIGNET_PATH set, each pixel's weight is 1/rms^2 from the + SExtractor background-RMS map (all or nothing; a missing file + raises), and the noise realisations use the same per-pixel RMS; the + fallback is a scalar 1/sigma_mad^2. Each epoch is + background-subtracted with the SExtractor background vignet + (BKG_SUB, on unless set False). + Values: + NGMIX_RUNNER.BKG_RMS_VIGNET_PATH = + $NGMIX_VIGNET_DIR/vignetmaker_runner_run_2/output/background_rms_vignet{file_number_string}.sqlite. + default: rms_vignet_weights + options: + rms_vignet_weights: + label: Per-pixel background-RMS weights + scalar_sigma_mad: + label: Scalar sigma_mad per epoch + excluded: true + excluded_reason: Mis-reports errors wherever the RMS map varies. + psf_likelihood_noise: + label: Flat PSF-observation weight + rationale: >- + [HARDCODED] the PSF observation carries a flat weight 1/PSF_NOISE^2; + without it the g-prior swamps the PSF likelihood. The recovered PSF + shape and size are flat for PSF_NOISE from 1e-4 to 1e-6 on the + digital twin. + Values: + PSF_NOISE = 1e-5. + default: psf_noise_1em5 + options: + psf_noise_1em5: + label: PSF_NOISE 1e-5 + megacam_ccd_flip: + label: 180-degree rotation of tile vignets for CCDs below 18 and 36-37 + rationale: >- + [HARDCODED] "MegaPipe has CCDs that are upside down": the tile vignet + and segmentation stamp are rotated to register with the epoch stamp. + A wrong flip mis-registers the tile coverage flag against the epoch, + changing flagged pixels and the masked-fraction cut. The docstring + warns it is wrong for THELI CCDs. + default: megapipe_flip + options: + megapipe_flip: + label: Flip CCDs < 18 and 36/37 (MegaPipe orientation) + defect_fill: + label: Image content of defect pixels before metacal + rationale: >- + A defect pixel (nonzero instrument flag, zero exposure weight or + invalid background RMS) gets weight 0 in prepare_ngmix_weights. Its + image value still matters: ngmix's metacal deconvolves, shears and + reconvolves an InterpolatedImage of the whole image and copies the + weights through, so a zero-weight pixel's content spreads into the + weighted pixels within about a PSF width. DES's ngmixer fills for + that reason: "it may be important for codes that take moments or use + FFTs". In the committed default (`BLEND_HANDLING = noisefill`), + masked pixels are replaced with independent noise at the per-pixel + background RMS when supplied, or the stamp's robust noise scale + otherwise. With `BLEND_HANDLING = uberseg`, the image is left + untouched and weights are zeroed on defects and neighbour-side + pixels. [LINT] the prepare_ngmix_weights docstring says noisefill + keeps the weight of filled pixels (the code zeroes it), and the + ngmix_runner comment says noisefill fills neighbour pixels (it fills + flagged pixels and leaves neighbours untouched). The committed fill + uses the unsymmetrized defect set: DES symmetrized its masks, but + four-fold symmetrization quadruples m and still leaves an additive c1 + (symmetrized_4fold_noise). The cost of not symmetrizing, a hole in + the galaxy light, is bounded by central_defect_veto. Noise stays the + default until a survey A/B against interpolation. + Values: + VIGNETMAKER_RUNNER_RUN_1.MASKING = False; + VIGNETMAKER_RUNNER_RUN_2.MASKING = False. + default: noise + options: + noise: + label: Independent noise on the unsymmetrized defect set + description: >- + Masked pixels get independent noise using the per-pixel + background RMS when supplied, or the stamp's robust noise scale + otherwise; their inverse-variance weights stay zero. Metacal's + fixnoise noise image covers every pixel. + insights: [mask_metacal_acts_on_whole_stamp, mask_bad_column_symmetrize] + interpolate: + label: Interpolate short bounded runs; noise-fill the rest + description: >- + Not implemented. PR #916 measures c1 = -1.3e-3 for a column + 8 px from a 0.5 arcsec galaxy and +2e-6 when the weight is also + zeroed on its quarter-turn orbit. For a 3-px bleed 6 px from that + galaxy, PR #916 measures m11 = +0.89% when the fill is + symmetrized, against +0.19% otherwise. + insights: [mask_interpolate_with_noise, mask_sharp_edges_ring] + symmetrized_4fold_noise: + label: Four-fold-symmetrized defect set (M | rot90 | rot180 | rot270), then noise fill + description: Not implemented. + excluded: true + excluded_reason: >- + PR #915 measures m11 = -2.7% with four-fold symmetrization + against -0.64% without it for a 3-px bleed 10 px from a galaxy + with 0.5 arcsec half-light radius and a 0.7 arcsec PSF. Through + a PSF with ellipticity (0.05, 0.02), it measures c1 = 7.3e-4 + with symmetrization against 0.7e-4 without it. + insights: [mask_bad_column_symmetrize, mask_des_defect_practice, mask_fixed_orientation] + raw: + label: "No fill: raw defect values (BLEND_HANDLING = uberseg)" + description: >- + With BLEND_HANDLING = uberseg, defect pixels keep weight 0 but + their image values stay untouched in the image that metacal + deconvolves, shears and reconvolves. + excluded: true + excluded_reason: >- + Metacal acts on every pixel regardless of weight, so raw defects + leak into the weighted pixels. No published metacal pipeline does + this: DES dropped, model-filled or interpolated defects. + insights: [mask_metacal_acts_on_whole_stamp, mask_des_defect_practice] + blend_handling: + label: Neighbour treatment before metacal + rationale: >- + Covers only pixels shared with a neighbour; defect_fill is coupled to + it through BLEND_HANDLING. noisefill (the default; the committed + config sets no key) leaves neighbours fully weighted and untouched. + uberseg zeroes the weight of pixels nearer a neighbour's coadd + segmentation footprint than the target's (DILATE_NEIGHBOUR, default 1) + and leaves the image untouched, as official uberseg does: + esheldon/meds get_uberseg returns a weight map (a + nearest-segment-pixel Voronoi split). DES Y1's fiducial metacal and + the last Y3 config ran on uberseg-weighted stamps with raw neighbour + light (an earlier Y3 config subtracted MOF neighbours first); Y6 masks + no neighbours before the shear step, using uberseg only as the fit + weight after metacal. No DES or Rubin metacal/metadetect pipeline + noise-fills the neighbour side. Raw neighbour light is consistent with + metacal: it is real sky, and the artificial shear shears it with the + target, as the real shear does; the reconvolution spreads it slightly + further across the Voronoi boundary than the PSF already had. Known + residuals: about +2% m for uberseg-only relative to MOF subtraction in DES Y1 + simulations, and a blending bias dominated by shear-dependent + detection, which no pixel treatment fixes and which DES Y3 calibrated + with simulations (mask_des_y1_uberseg_only, + mask_blend_bias_detection). Noise-filling the neighbour side would + instead cut the target's own light along an unsheared boundary, a + sharp edge that rings in the FFTs. The recommended comparison arm is + uberseg (weight-only), with defect_fill held equal across arms. + default: none + options: + none: + label: No neighbour treatment (BLEND_HANDLING = noisefill) + description: >- + Neighbour pixels keep their full weight and image values, so the + fit sees all neighbour light, which biases shapes toward + neighbours (Jarvis et al. 2016); this masks less than even their + plain segmentation map. A candidate cause of the FLAGS=2 B-modes + investigated in #814. + insights: [mask_uberseg_neighbour_bias] + uberseg: + label: UberSeg, weight-only + description: >- + As in DES Y1/Y3. Needs the coadd segmentation stamp + (SEG_VIGNET_PATH). DILATE_NEIGHBOUR absorbs the coadd-vs-epoch + overlay offset: ShapePipe reuses one coadd seg stamp for every + epoch where MEDS reprojects it. + insights: [mask_uberseg_weight_only, mask_uberseg_neighbour_bias, mask_des_y1_uberseg_only, mask_blend_bias_detection] + uberseg_fill: + label: UberSeg plus noise fill of neighbour-side pixels + description: >- + Not implemented. Also replace neighbour-side pixels with noise, + removing neighbour light from metacal at the cost of a sharp, + unsheared edge through the target's wings (FFT ringing). No + published pipeline does this; useful only as a diagnostic arm, + where a smooth (apodized) taper would do less damage. + insights: [mask_sharp_edges_ring, mask_blend_bias_detection] + mof_subtract: + label: Subtract neighbour models, then UberSeg (DES Y1 alternative) + description: >- + Not implemented. Subtract multi-object-fit models of the + neighbours before metacal and keep uberseg weights. This removed + the ~2% uberseg-only bias in DES Y1 simulations, although Sheldon + et al. 2020 found similar blend biases with and without MOF. + insights: [mask_des_y1_uberseg_only, mask_blend_bias_detection] + central_defect_veto: + label: Per-epoch veto on a defect near the stamp centre + rationale: >- + Not on develop; implemented on feat/defect-fill-veto (7777181b). + There an epoch is dropped when a defect pixel lies strictly closer + to the stamp centre than its fill's radius, beside the + masked-fraction cut in the epoch loop. The veto reads only the + defect mask, so it selects on nothing shear-responsive; for the same + reason the radii are fixed rather than scaled by galaxy size: 10 px + for noise-filled pixels (EPOCH_CENTRAL_DEFECT_RADIUS) and 7 px for + interpolated ones (EPOCH_INTERPOLATED_DEFECT_RADIUS, + feat/defect-interpolation). Calibrated on 51-px stamps at known RMS + and high S/N, through a round and a (0.05, 0.02) elliptical 0.7 + arcsec PSF, on galaxies of half-light radius 0.3 and 0.5 arcsec, and + for interpolation also 0.7 and 0.9 arcsec. Known limits: at 10 px, + wide defects through the elliptical PSF sit at the 1% bound (m11 = + -0.98%; -0.24% at 11 px), and noise fill needs 14 px for 0.7 and 0.9 + arcsec galaxies. Measured on feat/defect-fill-veto and + feat/defect-interpolation. + default: disabled + options: + disabled: + label: No central veto (committed code) + fixed_radii: + label: Fixed radii, 10 px for noise-filled and 7 px for interpolated defects + description: >- + Implemented on feat/defect-fill-veto and feat/defect-interpolation, + not on develop. + size_scaled_radius: + label: Veto radius scaled by galaxy size + excluded: true + excluded_reason: >- + Galaxy size responds to shear, so a size-scaled veto selects on + a shear-responsive quantity. + epoch_masked_fraction_cut: + label: Per-epoch masked-fraction cut + rationale: >- + [HARDCODED] on develop, an epoch whose stamp has more than 1/3 of its + pixels flagged (any nonzero flag bit, including the tile-coverage bit + 2**10 set where the tile vignet is off-image) is dropped from the + multi-epoch fit; an object with no surviving epoch has no shape. + Zero-weight and invalid-RMS pixels are not counted. On + feat/defect-fill-veto the cut counts the raw, unsymmetrized defect + set (flagged, zero-weight and invalid-RMS pixels) against + EPOCH_MASKED_FRACTION_CUT, default 1/3. Before the cut, an epoch is + dropped silently if its galaxy stamp is all zeros or its + background-subtracted sigma_mad is not positive. DES was stricter: + Y1 rejected any epoch with a masked or zero-weight pixel, or with + its central 4-pixel region masked; Y3 cut at 10% raw zero-weight + pixels; Y6 drops images more than 10% missing and keeps objects at + mfrac < 0.1 (mask_multi_epoch_drop). UNIONS has fewer epochs than + DES, so the cost in effective number density has to be measured, + not assumed. A defect near the centre is handled separately + (central_defect_veto). + default: one_third + options: + one_third: + label: 1/3 of the stamp in the defect set + description: >- + Drop an epoch only if more than 1/3 of the stamp pixels are + defects (on develop, flagged pixels). + ten_percent: + label: 10% (DES Y3 / Y6) + description: >- + Not a default; EPOCH_MASKED_FRACTION_CUT = 0.1 on + feat/defect-fill-veto. Matches DES Y3 max_zero_weight_frac and + Y6 max_masked_fraction. + insights: [mask_multi_epoch_drop] + any_masked: + label: Any masked pixel (DES Y1) + description: >- + Not implemented. Drop an epoch with any masked stamp pixel, so no + fill is ever needed. DES could afford this with about 10 epochs + per band; UNIONS likely cannot. + insights: [mask_multi_epoch_drop, mask_des_defect_practice] + prior_insights: + guinot22_ngmix_priors: + claim: >- + The published ngmix fit uses a Gaussian centroid prior of width one + pixel scale, a flat prior on the half-light radius r50 in [-10, 1e6] + arcsec and a flat flux prior in [-1e4, 1e9]. + created_at: "2022-04-01T00:00:00Z" + evidence: + - id: ev_guinot22_cen_prior + doi: "10.48550/arXiv.2204.04798" + quote: + exact: 'Gaussian distribution centred on 0 with σ = pixel scale' + location: { page: 7 } + - id: ev_guinot22_r50_prior + doi: "10.48550/arXiv.2204.04798" + quote: + exact: 'r50: flat distribution in' + location: { page: 7 } + - id: ev_guinot22_flux_prior + doi: "10.48550/arXiv.2204.04798" + quote: + exact: 'F: flat distribution in' + location: { page: 7 } + guinot22_hsm_initialisation: + claim: >- + The published fit is initialised from adaptive moments measured on + each sheared version of the object. + created_at: "2022-04-01T00:00:00Z" + evidence: + - id: ev_guinot22_hsm_init + doi: "10.48550/arXiv.2204.04798" + quote: + exact: 'We thus run first an adaptive moments algorithm to initialise the least-square operation.' + location: { page: 7 } + guinot22_gaussian_model: + claim: >- + The published shape measurement models galaxies with a single + Gaussian profile, arguing the resulting model bias is small and + largely absorbed by metacalibration. + created_at: "2022-04-01T00:00:00Z" + evidence: + - id: ev_guinot22_gaussian + doi: "10.48550/arXiv.2204.04798" + quote: + exact: 'Despite being very simple, the model bias (Kacprzak et al.' + suffix: ' 2014) is small.' + location: { page: 7 } + guinot22_metacal_five_images: + claim: >- + Metacalibration in the published analysis creates four sheared + images for calibration plus one for measurement, with a shear step + of 0.01 and a 90-degree-rotated noise image to cancel noise + correlations. + created_at: "2022-04-01T00:00:00Z" + evidence: + - id: ev_guinot22_metacal + doi: "10.48550/arXiv.2204.04798" + quote: + exact: 'This method creates four images used for the calibration, and one for the measurement.' + location: { page: 6 } + mask_uberseg_weight_only: + label: UberSeg is a weight-map operation (Jarvis 2016) + claim: >- + UberSeg (DES SV) zeroes the WEIGHT of pixels in another object's + coadd segmentation footprint or closer to another object than to the + target; MEDS weights are also zeroed wherever a mask flag is set. + The image is not modified: in the forward-model fits it was designed + for, a zero-weight pixel drops out of the likelihood exactly. + esheldon/meds get_uberseg, which returns a weight map, matches. + created_at: "2026-09-26T00:00:00Z" + derived: true + evidence: + - id: ev1 + doi: 10.48550/arXiv.1507.05603 + version: 3 + quote: + exact: We then set pixels in the weight map to zero if they were either associated with other objects in the segmentation map or were closer to any other object than to the object of interest. + - id: ev2 + doi: 10.48550/arXiv.1507.05603 + version: 3 + quote: + exact: are both of the model-fitting variety + mask_uberseg_neighbour_bias: + label: Neighbour light biases shapes toward neighbours (Jarvis 2016) + claim: >- + With a plain segmentation-map mask, light from a bright neighbour + just outside its footprint biased the shape toward the neighbour; + UberSeg made that bias undetectable in end-to-end simulations. + ShapePipe's default (no neighbour treatment) masks less than even + the plain segmentation map. + created_at: "2026-09-26T00:00:00Z" + derived: false + evidence: + - id: ev1 + doi: 10.48550/arXiv.1507.05603 + version: 3 + quote: + exact: when using ordinary segmentation maps we found a significant bias of the galaxy shape in the direction toward neighbors. + - id: ev2 + doi: 10.48550/arXiv.1507.05603 + version: 3 + quote: + exact: we no longer detected any systematic bias in the shape estimates due to unmasked flux from neighboring objects + mask_metacal_acts_on_whole_stamp: + label: Metacal transforms every stamp pixel, weights unseen + claim: >- + Metacal builds an interpolated image of the whole postage stamp, + then deconvolves, shears and reconvolves it; its Fourier transforms + cannot accommodate missing data, so zero-weight pixels' image values + feed the sheared images. The method papers leave masking + unaddressed. ngmix does exactly this: an InterpolatedImage of + obs.image, weights copied through. + created_at: "2026-09-26T00:00:00Z" + derived: true + evidence: + - id: ev1 + doi: 10.48550/arXiv.1702.02600 + version: 1 + quote: + exact: For each galaxy and PSF postage stamp, we first create an + - id: ev2 + doi: 10.48550/arXiv.1702.02600 + version: 1 + quote: + exact: We have made no attempt to deal with the effects of masked pixels or blending + - id: ev3 + doi: 10.48550/arXiv.1702.02601 + version: 2 + quote: + exact: convolutions cannot accommodate missing data + mask_bad_column_symmetrize: + label: Filled bad columns give additive e1; symmetrize the mask + claim: >- + Sheldon & Huff 2017 found that filling bad columns (with the + best-fit model, not even noise) gave a large additive e1 bias and a + few-per-mille multiplicative bias, both removed by adding a + compensating column rotated 90 degrees about the stamp centre. + Sheldon et al. 2020 recommend the same compensating mask; DES Y6, + like previous DES pipelines, OR-s each bad-pixel mask with its + 90-degree rotation to cancel additive biases. + created_at: "2026-09-26T00:00:00Z" + derived: true + evidence: + - id: ev1 + doi: 10.48550/arXiv.1702.02601 + version: 2 + quote: + exact: as well as a multiplicative bias of a few parts in a thousand + - id: ev2 + doi: 10.48550/arXiv.1702.02601 + version: 2 + quote: + exact: at 90 degree rotation about the center of the image, restoring symmetry to the image + - id: ev3 + doi: 10.48550/arXiv.1911.02505 + version: 2 + quote: + exact: An additional compensating mask, rotated at right angles to the real mask, can be used to restore symmetry to the image + - id: ev4 + doi: 10.48550/arXiv.2501.05665 + version: 2 + quote: + exact: we rotate the bad pixel mask by 90 degrees and apply it via a logical + - id: ev5 + doi: 10.48550/arXiv.2501.05665 + version: 2 + quote: + exact: This process helps to cancel additive biases in the final shape measurement. + mask_des_defect_practice: + label: DES never passed raw defects through metacal + claim: >- + Every DES metacal generation removed defects from the image before + metacal: Y1 dropped any epoch with a masked pixel, Y3 filled + symmetrized defects with the best-fit central model (ngmix-y1-config + / ngmix-y3-config), and Y6 + interpolated symmetrized defects following "previous DES shear + measurement pipelines". None left raw defect values in a zero-weight + pixel, as ShapePipe's uberseg path does. + created_at: "2026-09-26T00:00:00Z" + derived: true + evidence: + - id: ev1 + doi: 10.48550/arXiv.2501.05665 + version: 2 + quote: + exact: Following previous DES shear measurement pipelines + - id: ev2 + doi: 10.48550/arXiv.2501.05665 + version: 2 + quote: + exact: Then, we apply a two-dimensional Clough-Tocher interpolation + mask_interpolate_with_noise: + label: Interpolate defects, and the noise image identically + claim: >- + The metadetect papers interpolate masked regions (bad columns, + cosmic rays, saturation) before the shear step, taking care not to + create a spurious shear, and pass the noise image used for metacal's + correlated-noise correction through the same interpolation. + created_at: "2026-09-26T00:00:00Z" + derived: true + evidence: + - id: ev1 + doi: 10.48550/arXiv.1911.02505 + version: 2 + quote: + exact: The FFT does not permit missing data, so the masked regions must be interpolated in some way. + - id: ev2 + doi: 10.48550/arXiv.1911.02505 + version: 2 + quote: + exact: so the noise field used for correcting correlated noise effects must also be propagated through the same coadding and interpolation + - id: ev3 + doi: 10.48550/arXiv.2303.03947 + version: 2 + quote: + exact: Before warping, we interpolated the simulated artifacts in each image such as cosmic rays, bad columns, and saturated pixels. + - id: ev4 + doi: 10.48550/arXiv.2303.03947 + version: 2 + quote: + exact: we must also run the noise image through the same procedures + mask_sharp_edges_ring: + label: Sharp mask edges ring through metacal; apodize large masks + claim: >- + Sharp mask edges ring in the metacal FFTs, so bright-star regions + are zeroed with an apodized (smoothly tapered) edge, and the noise + image gets the same masking. The same reasoning argues against a + hard noise fill cut along a neighbour's Voronoi boundary. + created_at: "2026-09-26T00:00:00Z" + derived: true + evidence: + - id: ev1 + doi: 10.48550/arXiv.2303.03947 + version: 2 + quote: + exact: Masks with sharp features can cause ringing in the FFTs used by the + - id: ev2 + doi: 10.48550/arXiv.2501.05665 + version: 2 + quote: + exact: we further mask bright stars with apodization to avoid FFT artifacts during the deconvolution, shearing, and reconvolution processes + - id: ev3 + doi: 10.48550/arXiv.2501.05665 + version: 2 + quote: + exact: we also applied the same masking and apodization to the + mask_multi_epoch_drop: + label: Drop heavily masked epochs (10% in DES) + claim: >- + With many dithered epochs, problematic data can simply be dropped + (Sheldon & Huff 2017). DES Y6 drops input images more than 10% + missing and cuts objects at masked fraction mfrac < 0.1, which its + image simulations show avoids shear calibration bias. ShapePipe's + cut is 1/3. + created_at: "2026-09-26T00:00:00Z" + derived: true + evidence: + - id: ev1 + doi: 10.48550/arXiv.1702.02601 + version: 2 + quote: + exact: data deemed problematic can simply be left out of the fit + - id: ev2 + doi: 10.48550/arXiv.2501.05665 + version: 2 + quote: + exact: Images with a missing pixel fraction higher than 10 + - id: ev3 + doi: 10.48550/arXiv.2501.05665 + version: 2 + quote: + exact: is sufficient not to introduce shear calibration biases + mask_des_y1_uberseg_only: + label: 'DES Y1 metacal: uberseg-only fiducial, ~2% m residual' + claim: >- + DES Y1 metacal's fiducial catalogue handled neighbours with uberseg + only (raw neighbour light in the image). In dense deblending + simulations it carried m = 2.18 +/- 0.16 per cent at S/N > 10, + removed by subtracting MOF models of the neighbours; in data the + relative uberseg-vs-MOF m was 0.023 +/- 0.009. + created_at: "2026-09-26T00:00:00Z" + derived: false + evidence: + - id: ev1 + doi: 10.48550/arXiv.1708.01533 + version: 2 + quote: + exact: masks pixels close to neighbours rather than assigning a fraction of the light in each pixel to them + - id: ev2 + doi: 10.48550/arXiv.1708.01533 + version: 2 + quote: + exact: We detect no bias when subtracting the light from neighbours. + mask_blend_bias_detection: + label: Stamp-metacal blend bias is mostly shear-dependent detection + claim: >- + Sheldon et al. 2020 find that per-stamp metacal's few-percent + blending bias comes from shear-dependent detection, not blended + light: it is similar with and without neighbour subtraction, and + metacal shears the space between objects coherently. DES Y3 kept Y1's + approach (Gatti et al. list no change to neighbour or mask handling) + and calibrated the resulting 2-3% with image simulations. + created_at: "2026-09-26T00:00:00Z" + derived: true + evidence: + - id: ev1 + doi: 10.48550/arXiv.1911.02505 + version: 2 + quote: + exact: We show that this bias is not due to blending itself, but rather to shear-dependent object detection. + - id: ev2 + doi: 10.48550/arXiv.1911.02505 + version: 2 + quote: + exact: we see similar biases when no deblending is performed + - id: ev3 + doi: 10.48550/arXiv.1911.02505 + version: 2 + quote: + exact: the space between objects is sheared coherently + - id: ev4 + doi: 10.48550/arXiv.2011.03408 + version: 3 + quote: + exact: shape catalogue differs from DES Y1 in the following ways + - id: ev5 + doi: 10.48550/arXiv.2011.03408 + version: 3 + quote: + exact: which affects the shear estimates when objects are blended + mask_fixed_orientation: + label: Fixed-orientation cameras make mask anisotropy coherent + claim: >- + Masks have a preferred direction (columns, bleeds, spikes). On a + camera with a fixed sky orientation (DECam, and CFHT/MegaCam on its + equatorial mount), mask-induced shape errors add up coherently; + with Rubin's camera rotation, unmasked trails averaged away. DES Y3 has + an unexplained mean e1 of 3.5e-4 that its simulations, which include + the real bad-pixel masks, do not reproduce; whether masks cause it is + not established. + created_at: "2026-09-26T00:00:00Z" + derived: true + evidence: + - id: ev1 + doi: 10.48550/arXiv.1708.01533 + version: 2 + quote: + exact: The DES focal plane does not rotate, so these effects always correspond to the same orientation in sky coordinates. + - id: ev2 + doi: 10.48550/arXiv.2303.03947 + version: 2 + quote: + exact: We find that these unmasked trails do not cause a shear bias, which we attribute to the camera rotations that randomize the direction of the trail on the sky. + - id: ev3 + doi: 10.48550/arXiv.2011.03408 + version: 3 + quote: + exact: Our shear catalogue is characterized by a non-null mean shear in one of the two components, whose origin is unknown. + + # ═════════════════════════════════════════════════════════════════════════ + catalogue_assembly: + description: >- + The per-tile science catalogue: shape chunks merged and joined to the + detection catalogue and PSF quantities. + inputs: + - id: ngmix_chunks + type: data + source: per-tile ngmix catalogue chunks + tile sexcat + PSF diagnostics + outputs: + - id: tile_final_cat + type: data + format: fits + description: The assembled per-tile science catalogue (final_cat family). + inputs: [ngmix_chunks] + decisions: + [star_galaxy_classification, tile_overlap_handling, + failure_sentinels] + decisions: + star_galaxy_classification: + label: Star/galaxy separation deferred out of the pipeline + rationale: >- + Classification is off and no spread-model input is wired, so the + catalogue ships every object and separation happens downstream. The + dormant make_cat classifier computes class = sm + 2 sm_err, with + stars at |class| < SM_STAR_THRESH and galaxies at class > + SM_GAL_THRESH, both read from config (the function defaults, 0.003 + and 0.01, are bypassed); the committed config sets neither key, so + enabling classification alone raises. Unlike Guinot+22 + (guinot22_spread_model_cut), it applies no s > 0 or magnitude cut. + Values: + MAKE_CAT_RUNNER.SM_DO_CLASSIFICATION = False. + default: deferred_downstream + options: + deferred_downstream: + label: No in-pipeline classification; catalogue ships all objects + spread_model_inline: + label: spread_model classification in make_cat + insights: [guinot22_spread_model_cut] + description: >- + Not wired; needs spread_model_runner after the PSF + interpolation, its output added to make_cat's inputs, and both + thresholds set. + tile_overlap_handling: + label: Tile-overlap duplicates neither removed nor flagged + rationale: >- + Adjacent tiles overlap, and objects in the overlap are measured in + both. [HARDCODED] make_cat attaches only TILE_ID; with no + unique-object rule or overlap flag, duplicates are left to + downstream selection. NUMBER_LIST selects input tiles; it does not + carve disjoint sky regions, and no config key makes these catalogues + unique or flags their overlap rows. + default: no_dedup_in_pipeline + options: + no_dedup_in_pipeline: + label: Ship duplicates; TILE_ID is the only handle + overlap_flagging: + label: Flag overlap objects in the catalogue + description: Not implemented. + nearest_tile_centre: + label: Keep each object only in its nearest tile + excluded: true + excluded_reason: >- + Requires cross-tile coordination at assembly time, which the + per-tile DAG deliberately avoids. + failure_sentinels: + label: Objects without shape measurements kept, with sentinel values + rationale: >- + [HARDCODED] detections with no ngmix row (no surviving epoch, or an + exception during the fit) stay in the catalogue with sentinels: + sizes, fluxes, magnitudes and flags 0, flux and magnitude errors -1, + ellipticities and their errors -10, size errors 1e30, + NGMIX_N_EPOCH 0. A failed object's NGMIX_MCAL_FLAGS reads 0, the + success value, so a cut on flags alone keeps it; NGMIX_N_EPOCH > 0 + removes it. + default: sentinel_values + options: + sentinel_values: + label: Keep with sentinels (flags 0, e -10, T_ERR 1e30) + drop_unmatched: + label: Drop objects without shapes + excluded: true + excluded_reason: >- + Loses the photometry-only population and hides attrition. + prior_insights: + guinot22_spread_model_cut: + claim: >- + The published analysis selects galaxies inside the pipeline with the + spread model, at s + 2 sigma_s > 0.0003 together with s > 0 and + 20 < MAG_AUTO < 26. + created_at: "2022-04-01T00:00:00Z" + evidence: + - id: ev_guinot22_spread_model + doi: "10.48550/arXiv.2204.04798" + quote: + exact: 'We then use the spread model to make the selection presented below:' + location: { page: 5 } + - id: ev_guinot22_spread_model_cut + doi: "10.48550/arXiv.2204.04798" + quote: + exact: 's + 2σs > 0.0003' + location: { page: 5 } diff --git a/pyproject.toml b/pyproject.toml index baddaa93e..53f0dae61 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -147,4 +147,5 @@ markers = [ "slow: heavy compute (minutes); excluded from the fast inner loop.", "candide: needs the candide cluster and/or its real data; auto-skipped elsewhere.", "unions: uses UNIONS-specific survey layout or data.", + "decision(*ids): the astra.yaml decisions this test protects.", ] diff --git a/src/shapepipe/modules/find_exposures_package/find_exposures.py b/src/shapepipe/modules/find_exposures_package/find_exposures.py index fecb952ce..2ee704ada 100644 --- a/src/shapepipe/modules/find_exposures_package/find_exposures.py +++ b/src/shapepipe/modules/find_exposures_package/find_exposures.py @@ -64,6 +64,10 @@ def get_exposure_list(self): Return list of exposure file used for the tile in process, from tiles FITS header. + @sc [decision:preparation.epoch_provenance_from_tile_history,label:convention] epochs-from-tile-history + Apply ``EXP_PREFIX`` only at the start of the parsed name; do not strip + matching text from the middle or remove the trailing ``p`` here. + Returns ------- list diff --git a/src/shapepipe/modules/make_cat_package/__init__.py b/src/shapepipe/modules/make_cat_package/__init__.py index 70a85b81f..838a91c4d 100644 --- a/src/shapepipe/modules/make_cat_package/__init__.py +++ b/src/shapepipe/modules/make_cat_package/__init__.py @@ -50,6 +50,12 @@ retained as the extension point for a future estimator family) SAVE_PSF_DATA : bool, optional Save PSF information if ``True``; default value is ``False`` +N_EPOCH_SLOTS : int, optional + Number of slots written for each per-epoch PSF column family + (``HSM_*_PSF_n``, ``EXP_ID_n``, ``CCD_n``) when ``SAVE_PSF_DATA`` is + ``True``; unfilled slots hold the family's sentinel, and an object with + more epochs than slots raises an error. A fixed count gives every tile + the same schema. Default is the tile's maximum ``N_EPOCH`` plus one """ diff --git a/src/shapepipe/modules/make_cat_package/make_cat.py b/src/shapepipe/modules/make_cat_package/make_cat.py index 948709520..ee545626a 100644 --- a/src/shapepipe/modules/make_cat_package/make_cat.py +++ b/src/shapepipe/modules/make_cat_package/make_cat.py @@ -111,6 +111,7 @@ def save_sextractor_data(final_cat_file, sexcat_path, remove_vignet=True): int Number of objects saved + @sc [decision:catalogue_assembly.tile_overlap_handling] """ sexcat_file = file_io.FITSCatalogue(sexcat_path, SEx_catalogue=True) sexcat_file.open() @@ -175,6 +176,7 @@ def save_sm_data( ------- int Number of objects saved + @sc [decision:catalogue_assembly.star_galaxy_classification] """ final_cat_file.open() @@ -250,6 +252,10 @@ def save_mask_ext_data(final_cat_file, band_paths, w_log): The lookup itself is ``shapepipe.utilities.mask_query.query_map``, shared with the ``mask_query`` module: one primitive, two consumers. + @sc [decision:masking.sky_mask_application,label:scope] mask-columns-verbatim + Query ``XWIN_WORLD`` and ``YWIN_WORLD`` in catalogue order and pass the + ``query_map`` result to ``MASK_`` unchanged. + Parameters ---------- final_cat_file : file_io.FITSCatalogue @@ -299,6 +305,7 @@ def process( mode="", cat_path=None, moments=False, + n_epoch_slots=None, ): """Process Catalogue. @@ -310,6 +317,9 @@ def process( Path to input catalogue moments : bool Option to run ``ngmix`` mode with moments + n_epoch_slots : int, optional + Number of per-epoch slots in ``psf`` mode; if ``None``, the + tile's ``max(N_EPOCH) + 1`` Returns -------- @@ -326,7 +336,7 @@ def process( if mode == "ngmix": err_msg = self._save_ngmix_data(cat_path, moments) elif mode == "psf": - self._save_psf_data(cat_path) + self._save_psf_data(cat_path, n_epoch_slots) else: err_msg = ( f"Invalid process mode ({mode}) for " @@ -412,6 +422,13 @@ def _save_ngmix_data(self, ngmix_cat_path, moments=False): moments : bool, optional If True, write the parallel ``NGMIXm_*`` (moments-branch) columns. + @sc [decision:catalogue_assembly.failure_sentinels,label:coupling] failure-sentinel-cut-semantics + Missing-row ``T``, ``SNR``, flux, magnitude, PSF-size and flag values + initialize to 0, which is in range and cannot identify failure. Use + ``NGMIX_N_EPOCH == 0`` for that cut; the -10 ellipticity, -1 + flux/magnitude-error and 1e30 size-error sentinels are out of range and + can also identify missing fits. + """ self._key_ends = ["1M", "1P", "2M", "2P", "NOSHEAR"] @@ -614,10 +631,14 @@ def _save_ngmix_data(self, ngmix_cat_path, moments=False): return None - def _save_psf_data(self, galaxy_psf_path): + def _save_psf_data(self, galaxy_psf_path, n_epoch_slots=None): """Save PSF data. - Save the PSF catalogue into the final one. + Save the PSF catalogue into the final one, as per-epoch column + families ``HSM_*_PSF_n`` (``psf_shape_cols``), ``EXP_ID_n`` and + ``CCD_n``. Slot ``n`` of every + family refers to the same epoch; slots no epoch fills keep the + family's sentinel. @sc [label:schema] psf-epoch-slot-columns The per-epoch families are ``psf_shape_cols`` plus ``EXP_ID``/``CCD``, @@ -629,11 +650,23 @@ def _save_psf_data(self, galaxy_psf_path): ---------- galaxy_psf_path : str Path to the PSF catalogue to save + n_epoch_slots : int, optional + Number of slots written per family; if ``None``, the tile's + ``max(N_EPOCH) + 1`` + + Raises + ------ + ValueError + If an object has more epochs than ``n_epoch_slots`` """ galaxy_psf_cat = SqliteDict(galaxy_psf_path) - max_epoch = np.max(self._final_cat_file.get_data()["N_EPOCH"]) + 1 + n_epoch = self._final_cat_file.get_data()["N_EPOCH"] + if n_epoch_slots is None: + n_slots = np.max(n_epoch) + 1 + else: + n_slots = n_epoch_slots n_obj = len(self._obj_id) # Per-epoch PSF shape columns copied from the producer's SHAPES dict: @@ -657,17 +690,26 @@ def _save_psf_data(self, galaxy_psf_path): self._output_dict = { f"{name}_{idx + 1}": np.full(n_obj, fill, dtype=dtype) for name, fill, dtype in psf_shape_cols + epoch_id_cols - for idx in range(max_epoch) + for idx in range(n_slots) } for idx, id_tmp in enumerate(self._obj_id): - if galaxy_psf_cat[str(id_tmp)] == "empty": + obj_epochs = galaxy_psf_cat[str(id_tmp)] + if obj_epochs == "empty": continue - for epoch, key in enumerate(galaxy_psf_cat[str(id_tmp)].keys()): + if len(obj_epochs) > n_slots: + galaxy_psf_cat.close() + raise ValueError( + f"Object {id_tmp} has {len(obj_epochs)} PSF epochs" + + f" (N_EPOCH={n_epoch[idx]}), more than the {n_slots}" + + f" per-epoch slots (N_EPOCH_SLOTS={n_epoch_slots})" + ) + + for epoch, (key, gpc_data) in enumerate(obj_epochs.items()): - shapes = galaxy_psf_cat[str(id_tmp)][key]["SHAPES"] + shapes = gpc_data["SHAPES"] # `key` is "-"; reading it in the enumeration that # assigns `epoch` aligns EXP_ID_n/CCD_n with HSM_*_PSF_n by diff --git a/src/shapepipe/modules/make_cat_runner.py b/src/shapepipe/modules/make_cat_runner.py index 176341757..a7da5fd25 100644 --- a/src/shapepipe/modules/make_cat_runner.py +++ b/src/shapepipe/modules/make_cat_runner.py @@ -37,7 +37,12 @@ def make_cat_runner( module_config_sec, w_log, ): - """Define The Make Catalogue Runner.""" + """Define The Make Catalogue Runner. + + @sc [decision:catalogue_assembly.star_galaxy_classification] + + @sc [decision:masking.sky_mask_application] + """ # Set input file paths if len(input_file_list) == 3: # No spread model input @@ -82,6 +87,10 @@ def make_cat_runner( save_psf = config.getboolean(module_config_sec, "SAVE_PSF_DATA") else: save_psf = False + if config.has_option(module_config_sec, "N_EPOCH_SLOTS"): + n_epoch_slots = config.getint(module_config_sec, "N_EPOCH_SLOTS") + else: + n_epoch_slots = None # Set final output file final_cat_file = make_cat.prepare_final_cat_file( @@ -135,7 +144,9 @@ def make_cat_runner( w_log.info(err_msg) if save_psf: - err_msg = sc_inst.process("psf", galaxy_psf_path) + err_msg = sc_inst.process( + "psf", galaxy_psf_path, n_epoch_slots=n_epoch_slots + ) # Optional per-band external healsparse mask lookup (UNIONS-WL/spherex#38): # add one MASK_ column per band, queried at each object's world diff --git a/src/shapepipe/modules/mask_query_runner.py b/src/shapepipe/modules/mask_query_runner.py index a636560b2..f4bc97941 100644 --- a/src/shapepipe/modules/mask_query_runner.py +++ b/src/shapepipe/modules/mask_query_runner.py @@ -26,7 +26,15 @@ def mask_query_runner( module_config_sec, w_log, ): - """Define The Mask Query Runner.""" + """Define The Mask Query Runner. + + @sc [decision:masking.psf_star_mask_veto,label:scope] mask-query-carries-not-cuts + Without MASK_PATHS the catalogue passes through with no MASK_EXT column; + with it, MASK_EXT is carried for measurement and nothing here removes a + star. The PSF-star veto is a setools edit (``MASK_EXT == 0`` beside + IMAFLAGS_ISO), not a change here. + + """ sexcat_path = input_file_list[0] # Get file prefix (optional) diff --git a/src/shapepipe/modules/mccd_package/shapepipe_auxiliary_mccd.py b/src/shapepipe/modules/mccd_package/shapepipe_auxiliary_mccd.py index f0aea4f65..170e746b9 100644 --- a/src/shapepipe/modules/mccd_package/shapepipe_auxiliary_mccd.py +++ b/src/shapepipe/modules/mccd_package/shapepipe_auxiliary_mccd.py @@ -249,6 +249,8 @@ def mccd_fit_pipeline( Fit the MCCD model to the Observations. + @sc [decision:star_selection_psf.psf_modelling_software] + Parameters ---------- trainstar_path : str diff --git a/src/shapepipe/modules/merge_headers_package/merge_headers.py b/src/shapepipe/modules/merge_headers_package/merge_headers.py index 1b15a1a5f..37cc393e3 100644 --- a/src/shapepipe/modules/merge_headers_package/merge_headers.py +++ b/src/shapepipe/modules/merge_headers_package/merge_headers.py @@ -35,6 +35,7 @@ def merge_headers(input_file_list, output_dir, tile_number=None): TypeError For invalid ``output_dir`` type + @sc [decision:preparation.astrometric_solution_source] """ if not isinstance(output_dir, str): raise TypeError( diff --git a/src/shapepipe/modules/ngmix_package/ngmix.py b/src/shapepipe/modules/ngmix_package/ngmix.py index b2e092240..62502c4cd 100644 --- a/src/shapepipe/modules/ngmix_package/ngmix.py +++ b/src/shapepipe/modules/ngmix_package/ngmix.py @@ -27,6 +27,7 @@ # Neighbour treatments selectable with the BLEND_HANDLING option. BLEND_HANDLINGS = ("noisefill", "uberseg") +# @sc [decision:shape_measurement.metacal_scheme] METACAL_TYPES = ('noshear', '1p', '1m', '2p', '2m') # Noise budget for the PSF observation's flat weight map (psf_wt = @@ -36,6 +37,7 @@ # value is non-critical once it is finite (validated on the digital twin: the # recovered PSF shape/size are flat across 1e-4..1e-6). See # make_ngmix_observation. +# @sc [decision:shape_measurement.psf_likelihood_noise] PSF_NOISE = 1e-5 @@ -102,6 +104,8 @@ def get_prior(pixel_scale, rng, T_range=None, F_range=None): Returns ------- ngmix.joint_prior.PriorSimpleSep + + @sc [decision:shape_measurement.fit_priors] """ if T_range is None: T_range = [-1.0, 1.0e3] @@ -165,6 +169,8 @@ def position_seed(ra, dec, ccd): ------- int Seed in ``[0, 2**32)`` for ``numpy.random.RandomState``. + + @sc [decision:shape_measurement.ngmix_seed_mode] """ box_x = int(np.floor((ra * 3600) / 3) + (ccd + 1)) box_y = int(np.floor((dec * 3600) / 3) + (ccd + 2)) @@ -586,6 +592,7 @@ def MegaCamFlip(self, vign, ccd_nb): numpy.ndarray The flipped postage stamp + @sc [decision:shape_measurement.megacam_ccd_flip] """ if ccd_nb < 18 or ccd_nb in [36, 37]: # swap x axis so origin is on top-right @@ -971,6 +978,7 @@ def process(self): dict Dictionary containing the NGMIX metacal results + @sc [decision:shape_measurement.fit_initialisation,decision:shape_measurement.ngmix_seed_mode] """ tile_cat = Tile_cat(self._tile_cat_path, self._seg_cat_path) vignet_cat = self._vignet_cat @@ -1151,7 +1159,10 @@ def prepare_postage_stamps( psf_obj=None, gal_obj=None, ): - # define per-object lists of individual exposures to go into ngmix + """Prepare the per-object lists of exposures passed to ngmix. + + @sc [decision:shape_measurement.central_defect_veto,decision:shape_measurement.epoch_masked_fraction_cut] + """ stamp = Postage_stamp(bkg_sub=bkg_sub) # Read each store's per-object dict ONCE: every sqlitedict access # unpickles the object's whole all-epoch dict, so keeping these out of @@ -1300,6 +1311,7 @@ def background_subtract(gal,bkg): ------- numpy.ndarray background subtracted galaxy + @sc [decision:shape_measurement.galaxy_pixel_weights] """ # background subtraction @@ -1329,6 +1341,7 @@ def rescale_epoch_fluxes(gal, weight, header, bkg_rms=None): rescaled weight image numpy.ndarray or None rescaled background RMS image + @sc [decision:shape_measurement.epoch_flux_rescaling] """ Fscale = header['FSCALE'] @@ -1533,6 +1546,7 @@ def uberseg_weight(weight, seg, object_number, dilate_neighbour=0): ------- numpy.ndarray Copy of ``weight`` with neighbour-side pixels zeroed. + @sc [decision:shape_measurement.blend_handling] """ weight = np.copy(weight) @@ -1612,6 +1626,7 @@ def prepare_ngmix_weights( Variance map for NGMIX. numpy.ndarray Noise image. + @sc [decision:masking.pixel_mask_source,decision:shape_measurement.blend_handling,decision:shape_measurement.defect_fill,decision:shape_measurement.galaxy_pixel_weights] """ if blend_handling not in BLEND_HANDLINGS: raise ValueError( @@ -1741,6 +1756,7 @@ def make_ngmix_observation( Returns ------- ngmix.observation.Observation + @sc [decision:shape_measurement.centroid_source,decision:shape_measurement.psf_likelihood_noise] """ psf_jacob = ngmix.Jacobian( row=(psf.shape[0] - 1) / 2, @@ -1836,6 +1852,7 @@ def _average_psf_fits(results_and_weights): dict Keys ``g_psf``, ``g_psf_err``, ``T_psf``, ``T_psf_err`` (weighted averages over the surviving epochs) and ``n_epoch`` (their count). + @sc [decision:shape_measurement.psf_epoch_averaging] """ n_epoch_used = 0 wsum = 0 @@ -1887,6 +1904,8 @@ def average_multiepoch_psf(obsdict): Keys: 'g_psf', 'g_psf_err', 'T_psf', 'T_psf_err' (weighted averages over the epochs whose PSF fit succeeded) and 'n_epoch' (the number of those surviving epochs). + + @sc [decision:shape_measurement.psf_epoch_averaging] """ # ignore_failed_psf=True drops failed-PSF epochs from the galaxy fit but # keeps them in obsdict; _average_psf_fits skips them on flags != 0. @@ -1941,6 +1960,7 @@ def average_original_psf(gal_obs_list, psf_runner): ------- dict Same keys as :func:`average_multiepoch_psf`. + @sc [decision:shape_measurement.psf_epoch_averaging] """ def fit(gal_obs): # Fit a COPY so gal_obs.psf stays pristine for metacal — see docstring. @@ -1973,6 +1993,7 @@ def make_runners(prior, flux_guess, rng): ------- tuple (runner, psf_runner) : ngmix.runners.Runner, ngmix.runners.PSFRunner + @sc [decision:shape_measurement.fit_initialisation,decision:shape_measurement.fit_priors,decision:shape_measurement.galaxy_model] """ fitter = ngmix.fitting.Fitter(model='gauss', prior=prior) guesser = ngmix.guessers.TPSFFluxAndPriorGuesser(rng=rng, T=0.25, prior=prior) @@ -2046,6 +2067,7 @@ def do_ngmix_metacal( dict (:func:`average_original_psf`). The two PSF dicts share keys but describe different PSFs; the named fields guard against transposing them. Unpacks positionally as ``resdict, psf_res, psf_orig_res``. + @sc [decision:shape_measurement.defect_fill,decision:shape_measurement.metacal_scheme] """ n_epoch = len(stamp.gals) if n_epoch == 0: diff --git a/src/shapepipe/modules/ngmix_runner.py b/src/shapepipe/modules/ngmix_runner.py index 27648947f..63f03872b 100644 --- a/src/shapepipe/modules/ngmix_runner.py +++ b/src/shapepipe/modules/ngmix_runner.py @@ -42,7 +42,9 @@ def ngmix_runner( module_config_sec, w_log, ): - """Define The Ngmix Runner.""" + """Define The Ngmix Runner. + @sc [decision:shape_measurement.blend_handling,decision:shape_measurement.centroid_source,decision:shape_measurement.defect_fill,decision:shape_measurement.metacal_scheme] + """ # Read config file entries # Photometric zero point diff --git a/src/shapepipe/modules/psfex_interp_package/psfex_interp.py b/src/shapepipe/modules/psfex_interp_package/psfex_interp.py index 68019858c..8ebca71a5 100644 --- a/src/shapepipe/modules/psfex_interp_package/psfex_interp.py +++ b/src/shapepipe/modules/psfex_interp_package/psfex_interp.py @@ -413,6 +413,11 @@ def interpsfex(self, dotpsfpath, pos): Use PSFEx generated model to perform spatial PSF interpolation. + @sc [decision:star_selection_psf.psf_acceptance_thresholds,label:gate] psf-gate-path-specific-output + Validation logs a rejected gate and skips writing validation output; + the multi-epoch science path logs the same sentinel and skips that CCD + so affected objects lose its epoch. + Parameters ---------- dotpsfpath : str diff --git a/src/shapepipe/modules/setools_package/setools.py b/src/shapepipe/modules/setools_package/setools.py index 33d8efa8b..7c82c715f 100644 --- a/src/shapepipe/modules/setools_package/setools.py +++ b/src/shapepipe/modules/setools_package/setools.py @@ -618,6 +618,14 @@ def _make_rand_split(self): This function creates mask with random indices corresponding to the specfied ratio. + @sc [decision:star_selection_psf.psf_train_validation_split,label:reproducibility] split-seeded-by-file-number + The permutation is seeded from the digits of the unit's file number, so + a CCD gets the same PSF training and validation stars on every run; + never draw from a global or unseeded generator here. The + ``ratio_`` subset (20 % under RATIO = 20) is the validation + sample and its complement trains PSFEx (``star_split_ratio_80``); + swapping them starves the model of stars below the acceptance gate. + Raises ------ ValueError diff --git a/src/shapepipe/modules/sextractor_package/sextractor_script.py b/src/shapepipe/modules/sextractor_package/sextractor_script.py index cc61a24d7..8faf240ab 100644 --- a/src/shapepipe/modules/sextractor_package/sextractor_script.py +++ b/src/shapepipe/modules/sextractor_package/sextractor_script.py @@ -53,6 +53,12 @@ def ccd_candidate_mask(w, ra, dec, ccd_size, margin_frac=0.5): Flag catalogue positions that lie near a CCD's sky footprint. + @sc [decision:detection.epoch_membership_ccd_bounds,label:invariant] candidate-mask-is-superset + The mask only prefilters the strict CCD_SIZE test in + :func:`make_post_process`; it must keep every position that test could + accept. Tightening it (margin, radius) silently drops epochs and lowers + N_EPOCH instead of just skipping divergent WCS inversions. + The footprint is obtained by forward-projecting (pixel to world) the CCD centre and corners, which is always well defined. The returned mask selects positions within the corner radius plus a fractional @@ -110,6 +116,13 @@ def make_post_process(cat_path, f_wcs_path, pos_params, ccd_size, w_log=None): This function will add one HDU for each epoch to the SExtractor catalogue. Note that this only works for tiles. + @sc [decision:detection.epoch_membership_ccd_bounds,label:convention] epoch-bounds-strict-per-ccd + An object is on a CCD when its inverse-projected pixel lies strictly inside + CCD_SIZE (33, 2080, 1, 4612 committed), and each such CCD adds one to + N_EPOCH. A WCS inversion that fails drops that one CCD's epoch for the + objects near it, never the tile. CCD_N is the 0-based index that split_exp + gives the CCD file and its header entry. + The columns will be: - ``NUMBER``: same as SExtractor NUMBER @@ -363,6 +376,15 @@ def set_input_files( Set up all of the input image files. + @sc [decision:detection.weight_map_usage,label:convention] weight-map-on-both-images + With a weight file, the same map weights detection and measurement (a + separate detection weight only when DETECTION_WEIGHT is set); without + one, the command line forces ``-WEIGHT_TYPE None`` over the config's + MAP_WEIGHT, which changes the detection set. Extra inputs are + positional (image, weight, flag, psf, detection image, detection + weight): a reordering that keeps the count mis-assigns files without + raising. + Parameters ---------- use_weight: bool @@ -456,6 +478,7 @@ def get_zero_point(self, use_zp, zp_key=None): zp_key: str Header key corresponding to the zero point + @sc [decision:photometric_zeropoint] """ if use_zp and not isinstance(zp_key, type(None)): zp_value = get_header_value(self._meas_img_path, zp_key) @@ -473,6 +496,7 @@ def get_background(self, use_bkg, bkg_key=None): bkg_key: str Header key corresponding to the background value + @sc [decision:detection.background_model] """ if use_bkg and not isinstance(bkg_key, type(None)): bkg_value = get_header_value(self._meas_img_path, bkg_key) diff --git a/src/shapepipe/modules/split_exp_package/split_exp.py b/src/shapepipe/modules/split_exp_package/split_exp.py index dfab428f7..5c08b8163 100644 --- a/src/shapepipe/modules/split_exp_package/split_exp.py +++ b/src/shapepipe/modules/split_exp_package/split_exp.py @@ -70,6 +70,19 @@ def create_hdus(self, exp_path, output_suffix, transf_int, save_header): Split a single exposures CCDs into separate files. + @sc [decision:preparation.ccd_split_extent,label:convention] split-all-hdus-or-raise + Every one of the N_HDU (40) CCDs, the ear CCDs 36-39 included, is + written and becomes a candidate epoch; any other HDU count raises + rather than splitting a partial exposure. The file suffix ``-`` + and the header list index are the same 0-based CCD number that becomes + CCD_N downstream. + + @sc [decision:preparation.astrometric_solution_source,label:convention] wcs-from-delivered-header + The stored WCS is ``WCS(header)`` of the delivered CCD header, + unmodified. Every downstream world-to-pixel transform (epoch + membership, stamp placement, position seeding) uses it, so a refit or + header edit here moves every stamp centre and epoch assignment. + Parameters ---------- exp_path : str diff --git a/src/shapepipe/modules/vignetmaker_package/vignetmaker.py b/src/shapepipe/modules/vignetmaker_package/vignetmaker.py index 50f874254..586dae28e 100644 --- a/src/shapepipe/modules/vignetmaker_package/vignetmaker.py +++ b/src/shapepipe/modules/vignetmaker_package/vignetmaker.py @@ -21,6 +21,8 @@ def get_stamps(image, positions, rad): Extract postage stamps and record their sub-pixel centring. + @sc [decision:preparation.stamp_positioning_and_padding] + The image is zero-padded by ``rad`` on every side and a ``(2 * rad + 1, 2 * rad + 1)`` stamp is sliced around the integer pixel nearest each position (``numpy.round``). The stamp values are @@ -306,6 +308,7 @@ def _get_stamp_me(self, image_dirs, image_pattern): dict Dictionary containing object id and vignets for each epoch + @sc [decision:preparation.stamp_positioning_and_padding] """ cat = file_io.FITSCatalogue(self._galcat_path, SEx_catalogue=True) cat.open() diff --git a/src/shapepipe/pipeline/str_handler.py b/src/shapepipe/pipeline/str_handler.py index e73b54270..06d6e3cbb 100644 --- a/src/shapepipe/pipeline/str_handler.py +++ b/src/shapepipe/pipeline/str_handler.py @@ -258,6 +258,8 @@ def _mode(self, input, eps=0.001, iter_max=1000): Compute the mode, the most frequent value of a continuous distribution. + @sc [decision:star_selection_psf.star_selection_box] + Parameters ---------- input : numpy.ndarray diff --git a/src/shapepipe/utilities/CONTRACTS b/src/shapepipe/utilities/CONTRACTS new file mode 100644 index 000000000..c79c51122 --- /dev/null +++ b/src/shapepipe/utilities/CONTRACTS @@ -0,0 +1,6 @@ +@cc utilities-do-not-import-modules +forbid: shapepipe.utilities.* -> shapepipe.modules.* +Utilities are the primitives module runners share (mask_query is one lookup +with two consumers, the mask_query and make_cat modules). A utility that +imports a module inverts that layering and makes one module's internals a +dependency of every other caller. diff --git a/src/shapepipe/utilities/mask_query.py b/src/shapepipe/utilities/mask_query.py index 982bfd3d3..d0920f27e 100644 --- a/src/shapepipe/utilities/mask_query.py +++ b/src/shapepipe/utilities/mask_query.py @@ -210,6 +210,7 @@ def query_map(path, ra, dec): Map value at each position; positions outside the map's coverage carry the map's sentinel value + @sc [decision:masking.sky_mask_application] """ values, _ = query_map_coverage(path, ra, dec) @@ -221,6 +222,12 @@ def flag_positions(paths, ra, dec, bits=None, w_log=None): Combine one or more healsparse masks into a single per-object integer flag. + @sc [decision:masking.psf_star_mask_veto,label:convention] off-coverage-is-clean + A position outside a map's coverage contributes 0 for boolean and integer + maps alike, so a map that misses an exposure never flags its stars; the + all-off-coverage case is logged as a warning instead. The flag is the + bitwise OR of the per-map contributions, 0 meaning clean. + Each map contributes at each position: * boolean map: ``1`` where the map is ``True``, ``0`` elsewhere; diff --git a/tests/README.md b/tests/README.md index 0b0df1524..ea3e460e7 100644 --- a/tests/README.md +++ b/tests/README.md @@ -27,10 +27,15 @@ policy — it applies everywhere. ``` slow heavy compute (minutes); excluded from the fast inner loop candide needs the candide cluster and/or its real data; auto-skipped elsewhere +decision(*ids) ASTRA decisions protected by a test; ids checked against astra.yaml ``` `--strict-markers` is on, so a typo'd marker is an error, not a silent no-op. -Use `pytest -m "not unions"` to run the survey-generic tests. +Science guardrails use `decision(*ids)` at module or test level; AST parsing +checks every ID against `astra.yaml` without importing the tests. +`tests/unit/test_decisions.py` also checks that `@sc` site tags and rationale +`Values:` refs agree with the decision record, and that test markers name real +decisions. Use `pytest -m "not unions"` to run the survey-generic tests. A `candide`-marked test is **collected everywhere** (so `--collect-only` shows it exists) but **skipped off-cluster** with a clear reason. Candide is detected diff --git a/tests/helpers/decisions.py b/tests/helpers/decisions.py new file mode 100644 index 000000000..14c613291 --- /dev/null +++ b/tests/helpers/decisions.py @@ -0,0 +1,1485 @@ +"""Parse ShapePipe's decision tags, ASTRA Values and local contracts. + +The value grammar behind ``Values: ref = value`` is: + +* Values are numbers, boolean words, strings (quote expressions), or flat + comma lists, optionally bracketed. Semicolons separate refs. Decimal + equality is exact (1 = 1.0, 5e-4 = 0.0005), without rounding. +* Boolean words follow the file's reader, case-insensitively: INI as + ConfigParser.getboolean (yes/true/on/1, no/false/off/0; Y/N are text), + .sex/.psfex also Y/N, Python True/False; elsewhere words are text and + 1/0 are numbers. Outer whitespace is trimmed; other strings are + case-sensitive. Lists preserve order and length. Only .param VIGNET and + .psfex PSF_SIZE accept square-size shorthand: 51 = 51,51. +* ``= absent`` asserts that a config key has no active line anywhere in the + file or named section. The same decision must tag at least one site in that + file (and, for INI/SETools, in the named section or at file scope). INI + absence checks include values inherited from ``[DEFAULT]``. Quote it + ("absent") to mean the text. +* .setools refs use ``SECTION.KEY``. Predicates keep their operators as + quoted text; repeated cuts on one key are an ordered list, e.g. + ``MAG_AUTO = ["> 18.", "< 22."]``. Expressions compare as text. +* Python selectors may append ``[key.subkey]`` to a named assignment + (identifier-like string dict keys); only the selected literal is read, + including ``dict(key=value)`` syntax. Ambiguous bindings fail, including + ``NAME[...] =`` or ``NAME.attr =`` in the same scope; ``.update()`` calls + and mutation from other scopes are not seen. No imports, calls, + arithmetic, argument defaults, environment expansion or implicit tool + defaults are evaluated: the check stays static rather than becoming a + second pipeline runtime. +* Option ids are stable references and are never parsed for values. +""" + +import argparse +import ast +import configparser +import fnmatch +import re +import sys +import tokenize +import warnings +from dataclasses import dataclass +from decimal import Decimal +from functools import lru_cache +from io import StringIO +from pathlib import Path + +import yaml + + +@dataclass(frozen=True) +class Site: + """One code declaration, statement, config paragraph, or config scope.""" + + path: str + start: int + end: int + kind: str + symbol: str = "" + section: str = "" + scope: str = "" + + @property + def identity(self): + return (self.path, self.start, self.end, self.kind, self.symbol, self.section) + + +@dataclass(frozen=True) +class Tag: + """One parsed ``@sc`` tag and the site it governs.""" + + path: str + line: int + decisions: tuple[str, ...] + ident: str | None + meta: dict + prose: str + site: Site | None + + +@dataclass(frozen=True) +class DecisionMarker: + """One pytest marker linking a test to an ASTRA decision.""" + + decision: str + path: str + line: int + + +def load_yaml(path): + """Load YAML with PyYAML's safe loader.""" + + return yaml.safe_load(Path(path).read_text(encoding="utf-8")) + + +ABSENT = "absent" +_NO_SETTING = object() +_TAG_LINE = re.compile(r"^\s*@sc(?:\s+\[([^\]]*)\])?(?:\s+(\S+))?\s*$") +_META_KEY = re.compile(r"[A-Za-z_][\w.-]*:[^,\s]+\Z") +_ID = re.compile(r"[A-Za-z_][\w.-]*\Z") +_CONFIG_SUFFIXES = { + ".ini", ".sex", ".param", ".conv", ".psfex", ".setools", + ".ww", ".conf", ".yaml", ".yml", +} +_SKIP = {".git", ".venv", "venv", "__pycache__", "node_modules", ".felt"} + + +def _tag_line(text): + """Parse one ``@sc`` line into metadata and an optional local id.""" + + match = _TAG_LINE.fullmatch(text.strip()) + if not match: + return None, None, f"malformed @sc tag: {text.strip()}" + meta_text, ident = match.groups() + meta = {} + if meta_text is not None: + pairs = [part.strip() for part in meta_text.split(",")] + if not pairs or any(not _META_KEY.fullmatch(part) for part in pairs): + return None, None, f"malformed @sc metadata: {meta_text}" + for pair in pairs: + key, value = pair.split(":", 1) + if key != "decision" and key in meta: + return None, None, f"duplicate @sc metadata key: {key}" + meta.setdefault(key, []).append(value) + if ident is not None and not _ID.fullmatch(ident): + return None, None, f"malformed local-contract id {ident!r}" + if "scope" in meta and meta["scope"] != ["file"]: + return None, None, "scope metadata must be scope:file" + if "decision" not in meta and ident is None: + return None, None, "@sc needs a decision citation or local-contract id" + return meta, ident, None + + +def _comment_body(line): + stripped = line.lstrip() + if not stripped.startswith("#"): + return None + return stripped[1:].lstrip() + + +def _comment_prose(lines, index, end=None): + stop = len(lines) if end is None else end + prose = [] + for line in lines[index + 1:stop]: + if not line.strip(): + break + body = _comment_body(line) + if body is None or body.startswith("@sc"): + break + if body: + prose.append(body.strip()) + return " ".join(prose) + + +def _ast_symbol(node, parents): + parts = [node.name] + parent = parents.get(node) + while parent is not None and not isinstance(parent, ast.Module): + if isinstance(parent, (ast.ClassDef, ast.FunctionDef, ast.AsyncFunctionDef)): + parts.append(parent.name) + parent = parents.get(parent) + return ".".join(reversed(parts)) + + +def _python_docstring_tags(path, source, tree): + lines = source.splitlines() + parents = {child: parent for parent in ast.walk(tree) + for child in ast.iter_child_nodes(parent)} + owners = [(tree, "")] + owners.extend((node, _ast_symbol(node, parents)) for node in ast.walk(tree) + if isinstance(node, (ast.ClassDef, ast.FunctionDef, ast.AsyncFunctionDef))) + tags, errors, consumed = [], [], set() + for owner, symbol in owners: + body = getattr(owner, "body", []) + if not body or not isinstance(body[0], ast.Expr) or not isinstance( + body[0].value, ast.Constant + ) or not isinstance(body[0].value.value, str): + continue + doc_node = body[0].value + for offset, doc_line in enumerate(doc_node.value.splitlines()): + if not doc_line.strip().startswith("@sc"): + continue + line_no = doc_node.lineno + offset + consumed.add(line_no) + meta, ident, error = _tag_line(doc_line) + if error: + errors.append(f"{path}:{line_no}: {error}") + continue + prose = [] + for part in doc_node.value.splitlines()[offset + 1:]: + if not part.strip() or part.strip().startswith("@sc"): + break + prose.append(part.strip()) + prose_text = " ".join(prose) + if ident is not None and not prose_text: + errors.append(f"{path}:{line_no}: local contract {ident} has no prose") + site = Site(path, getattr(owner, "lineno", 1), + getattr(owner, "end_lineno", len(lines)), + "python_declaration", symbol=symbol) + tags.append(Tag(path, line_no, tuple(meta.get("decision", ())), ident, + {k: v for k, v in meta.items() if k != "decision"}, + prose_text, site)) + return tags, errors, consumed + + +def _comment_site(path, lines, index, meta, tree=None, *, snakemake=False): + line_no = index + 1 + if meta.get("scope") == ["file"]: + if tree is not None or snakemake: + return None + has_content = any( + line.strip() and _comment_body(line) is None for line in lines + ) + if not has_content: + return None + return Site(path, 1, len(lines), "config", scope="file") + if snakemake: + for n in range(index + 1, len(lines)): + line = lines[n] + if not line.strip(): + return None + if line.lstrip().startswith("#"): + continue + start = n + 1 + if re.match(r"^\s*(?:rule|checkpoint)\s+[\w.-]+\s*:", line): + end = len(lines) + for j in range(n + 1, len(lines)): + if lines[j].strip() and not lines[j].startswith((" ", "\t", "#")): + end = j + break + return Site(path, start, end, "snakemake") + return Site(path, start, start, "snakemake") + return None + if tree is not None: + node = next((n for n in tree.body if getattr(n, "lineno", 0) > line_no), None) + if node is None: + return None + for between in lines[index + 1:node.lineno - 1]: + if not between.strip() or not between.lstrip().startswith("#"): + return None + names = [] + for target in getattr(node, "targets", []): + names.extend(_target_names(target)) + if hasattr(node, "target"): + names.extend(_target_names(node.target)) + return Site(path, node.lineno, getattr(node, "end_lineno", node.lineno), + "python_statement", symbol=names[0] if names else "") + + # Config paragraphs end at blank lines; comments before/inside a run are + # skipped, but a blank before the first setting leaves an empty site. + for n in range(index + 1, len(lines)): + line = lines[n] + if not line.strip(): + return None + comment = _comment_body(line) + if comment is not None: + if comment.startswith("@sc"): + return None + continue + start = n + 1 + header = re.match(r"^\s*\[([^]]+)\]\s*(?:[#;].*)?$", line) + if header: + end = len(lines) + for j in range(n + 1, len(lines)): + if re.match(r"^\s*\[[^]]+\]\s*(?:[#;].*)?$", lines[j]): + end = j + k = j - 1 + while k > n and _comment_body(lines[k]) is not None: + if _comment_body(lines[k]).startswith("@sc"): + end = k + break + k -= 1 + break + return Site(path, start, end, "config", section=header.group(1).strip(), scope="section") + end = len(lines) + for j in range(n + 1, len(lines)): + if not lines[j].strip(): + end = j + break + comment = _comment_body(lines[j]) + if comment is not None and comment.startswith("@sc"): + end = j + break + suffix = Path(path).suffix.lower() + section = "" + if suffix in {".ini", ".setools"}: + for prior in lines[:index]: + header = re.match(r"^\s*\[([^]]+)\]\s*(?:[#;].*)?$", prior) + if header: + section = header.group(1).strip() + if suffix == ".ini" and not section: + section = "DEFAULT" + return Site(path, start, end, "config", section=section, + scope="paragraph") + return None + + +def _parse_file_tags(path, root): + relative = Path(path).relative_to(root).as_posix() + source = Path(path).read_text(encoding="utf-8") + lines = source.splitlines() + suffix = Path(path).suffix.lower() + tags, errors = [], [] + is_snakemake = suffix == ".smk" or Path(path).name == "Snakefile" + tree = None + consumed = set() + comment_tokens = {} + if suffix == ".py": + try: + with warnings.catch_warnings(): + warnings.simplefilter("ignore", SyntaxWarning) + tree = ast.parse(source, filename=relative) + except SyntaxError as error: + return [], [f"{relative}: cannot parse tagged Python: {error}"] + found, issues, consumed = _python_docstring_tags(relative, source, tree) + tags.extend(found) + errors.extend(issues) + try: + comment_tokens = { + token.start[0]: (token.start[1], token.string) + for token in tokenize.generate_tokens(StringIO(source).readline) + if token.type == tokenize.COMMENT + } + except tokenize.TokenError as error: + return tags, errors + [f"{relative}: cannot tokenize Python comments: {error}"] + if suffix == ".py" or is_snakemake or suffix in _CONFIG_SUFFIXES: + if suffix == ".py": + tag_comments = [ + (line_no, column, comment[1:].lstrip()) + for line_no, (column, comment) in comment_tokens.items() + if comment[1:].lstrip().startswith("@sc") + ] + else: + tag_comments = [ + (index + 1, len(line) - len(line.lstrip()), body) + for index, line in enumerate(lines) + if (body := _comment_body(line)) is not None and body.startswith("@sc") + ] + for line_no, column, body in tag_comments: + index = line_no - 1 + if suffix == ".py" and line_no in consumed: + continue + if suffix == ".py" and column != 0: + errors.append(f"{relative}:{line_no}: Python statement tags must be module-level comments") + continue + meta, ident, error = _tag_line(body) + if error: + errors.append(f"{relative}:{line_no}: {error}") + continue + site = _comment_site(relative, lines, index, meta, tree, + snakemake=is_snakemake) + if site is None: + errors.append(f"{relative}:{line_no}: @sc tag governs no site") + continue + prose = _comment_prose(lines, index) + if ident is not None and not prose: + errors.append(f"{relative}:{line_no}: local contract {ident} has no prose") + tags.append(Tag(relative, line_no, tuple(meta.get("decision", ())), ident, + {k: v for k, v in meta.items() if k != "decision"}, + prose, site)) + return tags, errors + + +def scan_tags(root): + """Collect all supported-site tags and report malformed/duplicate tags. + + Symlinked files are skipped: their tags belong to the target, which is + scanned at its own path (e.g. workflow/config/cfis_image_sims/ -> cfis/). + """ + + root = Path(root) + tags, errors = [], [] + for path in sorted(root.rglob("*")): + if (not path.is_file() or path.is_symlink() + or any(part in _SKIP for part in path.parts)): + continue + if path.name == "CONTRACTS": + continue + if path.suffix.lower() not in _CONFIG_SUFFIXES | {".py", ".smk"} and path.name != "Snakefile": + continue + found, issues = _parse_file_tags(path, root) + tags.extend(found) + errors.extend(issues) + by_id = {} + for tag in tags: + if tag.ident: + by_id.setdefault(tag.ident, []).append(tag) + for ident, found in sorted(by_id.items()): + if len(found) > 1: + where = ", ".join(f"{tag.path}:{tag.line}" for tag in found) + errors.append(f"duplicate local-contract id {ident}: {where}") + return tags, errors + + +def decision_ids(record, prefix=""): + """Scoped decision IDs: bare at top level, dotted in sub-analyses.""" + + if not isinstance(record, dict): + return set() + result = {f"{prefix}{key}" for key in (record.get("decisions") or {})} + for name, analysis in (record.get("analyses") or {}).items(): + if isinstance(analysis, dict): + result.update(decision_ids(analysis, f"{prefix}{name}.")) + return result + + +def tag_errors(tags, record): + """Check citations and require every ASTRA decision to have a site.""" + + known = decision_ids(record) + cited = set() + errors = [] + for tag in tags: + for decision in tag.decisions: + cited.add(decision) + if decision not in known: + errors.append(f"{tag.path}:{tag.line}: @sc cites unknown decision {decision!r}") + errors.extend(f"{decision}: decision has no tagged site" for decision in sorted(known - cited)) + return errors + + +def _parse_values(rationale): + if not isinstance(rationale, str) or "Values:" not in rationale: + return (), None + if rationale.count("Values:") != 1: + return (), "rationale must contain at most one Values: sentence" + tail = rationale.rsplit("Values:", 1)[1].strip() + if not tail.endswith("."): + return (), "Values: sentence must end with a period" + entries = tuple(part.strip() for part in tail[:-1].split(";")) + if not entries or any(not entry for entry in entries): + return (), "Values: sentence contains an empty ref" + return entries, None + + +def _split_assertion(reference): + """Split one ``ref = value`` entry (not a quoted cut's ``==``).""" + + parts = re.split(r"\s+=\s*", reference.strip(), maxsplit=1) + ref = parts[0] + expected = parts[1].strip() if len(parts) == 2 else None + if not ref or re.search(r"\s|=", ref): + raise ValueError("expected a ref optionally followed by ' = value'") + if expected is None or not expected or expected.startswith("="): + raise ValueError("Values entries require a nonempty ' = value'") + return ref, expected + + +def _parse_reference(reference): + if "::" in reference: + path, selector = reference.split("::", 1) + return "python", path, selector + if "#" in reference: + path, selector = reference.split("#", 1) + return "config", path, selector + return "bare", "", reference + + +def _target_names(target): + if isinstance(target, ast.Name): + return [target.id] + if isinstance(target, ast.Attribute): + return [target.attr] + if isinstance(target, (ast.Tuple, ast.List)): + return [name for item in target.elts for name in _target_names(item)] + if isinstance(target, ast.Starred): + return _target_names(target.value) + return [] + + +@lru_cache(maxsize=128) +def _bindings(scope): + """Collect all bindings per name; value reads must not pick one silently.""" + + result = {} + + def visit(node): + if isinstance( + node, + (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef), + ): + result.setdefault(node.name, []).append(node) + return + if isinstance(node, ast.Lambda): + return + if isinstance(node, ast.Assign): + targets = node.targets + elif isinstance(node, (ast.AnnAssign, ast.AugAssign, ast.NamedExpr)): + targets = [node.target] + elif isinstance(node, (ast.For, ast.AsyncFor)): + targets = [node.target] + elif isinstance(node, (ast.With, ast.AsyncWith)): + targets = [item.optional_vars for item in node.items] + else: + targets = [] + for target in targets: + if target is not None: + for name in _target_names(target): + result.setdefault(name, []).append(node) + if isinstance(node, ast.ExceptHandler) and node.name: + result.setdefault(node.name, []).append(node) + for child in ast.iter_child_nodes(node): + visit(child) + + for statement in scope.body: + visit(statement) + return result + + +def _mutated_name(target): + """Base name of ``NAME[...] =`` / ``NAME.attr =`` (nested included).""" + + while isinstance(target, (ast.Subscript, ast.Attribute)): + target = target.value + if isinstance(target, ast.Name): + return target.id + return None + + +@lru_cache(maxsize=128) +def _mutations(scope): + """Names whose bound object is item- or attribute-assigned in ``scope``. + + Same lexical scope only, like ``_bindings``; method calls such as + ``.update()`` and mutation from other scopes are not seen. + """ + + names = set() + + def visit(node): + if isinstance( + node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef, ast.Lambda) + ): + return + if isinstance(node, ast.Assign): + targets = node.targets + elif isinstance(node, (ast.AnnAssign, ast.AugAssign)): + targets = [node.target] + elif isinstance(node, ast.Delete): + targets = node.targets + elif isinstance(node, (ast.For, ast.AsyncFor)): + targets = [node.target] + elif isinstance(node, (ast.With, ast.AsyncWith)): + targets = [item.optional_vars for item in node.items] + else: + targets = [] + stack = [target for target in targets if target is not None] + while stack: + target = stack.pop() + if isinstance(target, (ast.Tuple, ast.List)): + stack.extend(target.elts) + elif isinstance(target, ast.Starred): + stack.append(target.value) + else: + name = _mutated_name(target) + if name: + names.add(name) + for child in ast.iter_child_nodes(node): + visit(child) + + for statement in scope.body: + visit(statement) + return frozenset(names) + + +def _ini_parser(text, *, strict=True): + parser = configparser.ConfigParser( + interpolation=None, strict=strict, allow_no_value=True + ) + parser.optionxform = str.lower + parser.read_string(text) + return parser + + +@lru_cache(maxsize=16) +def _python_tree(text): + # Cache by source, not path: editing a file must invalidate the read. + return ast.parse(text) + + +def _code_selector(selector): + match = re.fullmatch(r"([\w.]+)(?:\[([\w.]+)\])?", selector) + if not match or any(not p.isidentifier() for p in match[1].split(".")): + raise ValueError(f"invalid Python selector {selector!r}") + keys = tuple(match[2].split(".")) if match[2] else () + if any(not key.isidentifier() for key in keys): + raise ValueError("dict paths need dot-separated identifier keys") + return match[1], keys + + +def _dict_entry(node, key): + """Select syntax, not a runtime value; never execute a dict() call.""" + + if isinstance(node, ast.Dict): + if any( + not isinstance(k, ast.Constant) or not isinstance(k.value, str) + for k in node.keys + ): + raise ValueError("dict selectors need literal string keys, no **") + items = [(k.value, v) for k, v in zip(node.keys, node.values)] + elif ( + isinstance(node, ast.Call) and isinstance(node.func, ast.Name) + and node.func.id == "dict" and not node.args + and all(k.arg is not None for k in node.keywords) + ): + items = [(k.arg, k.value) for k in node.keywords] + else: + raise ValueError("dict selectors need {...} or dict(key=value) syntax") + names = [name for name, _ in items] + if len(names) != len(set(names)): + raise ValueError("ambiguous duplicate dict keys") + if key not in names: + raise ValueError(f"dict key {key!r} is missing") + return dict(items)[key] + + +def _selected_python_node(tree, selector): + symbol, keys = _code_selector(selector) + scope = tree + parts = symbol.split(".") + for index, part in enumerate(parts): + declarations = _bindings(scope).get(part, []) + if len(declarations) != 1: + raise ValueError( + f"{symbol!r} needs one binding; found {len(declarations)} " + f"for {part!r}" + ) + node = declarations[0] + if index == len(parts) - 1 and part in _mutations(scope): + raise ValueError( + f"{symbol!r} is item- or attribute-assigned after binding" + ) + if index < len(parts) - 1: + if not isinstance( + node, (ast.ClassDef, ast.FunctionDef, ast.AsyncFunctionDef) + ): + raise ValueError(f"{part!r} is not a lexical scope") + scope = node + if not isinstance(node, (ast.Assign, ast.AnnAssign)): + raise ValueError(f"{symbol!r} is not a literal assignment") + targets = node.targets if isinstance(node, ast.Assign) else [node.target] + if any(not isinstance(target, ast.Name) for target in targets): + raise ValueError("value assertions need simple named assignment targets") + node = node.value + for key in keys: + node = _dict_entry(node, key) + return node + + +_NUMBER = re.compile(r"[+-]?(?:\d+(?:\.\d*)?|\.\d+)(?:[eE][+-]?\d+)?\Z") +# Boolean words each reader accepts. INI follows ConfigParser.getboolean, +# whose 1/0 spellings are handled at comparison (see _ini_bool); SExtractor +# and PSFEx add Y/N; Python literals are already bool, and the record spells +# them True/False. Elsewhere words stay text. +_INI_BOOLEANS = { + "yes": True, "true": True, "on": True, + "no": False, "false": False, "off": False, +} +_ASTROMATIC_BOOLEANS = {**_INI_BOOLEANS, "y": True, "n": False} +_PYTHON_BOOLEANS = {"true": True, "false": False} +_YAML_BOOLEANS = {"true": True, "false": False, "yes": True, "no": False, "on": True, "off": False} +_NO_BOOLEANS = {} + + +def _booleans_for(suffix): + if suffix == ".ini": + return _INI_BOOLEANS + if suffix in {".sex", ".psfex"}: + return _ASTROMATIC_BOOLEANS + if suffix == ".py": + return _PYTHON_BOOLEANS + if suffix in {".yaml", ".yml"}: + return _YAML_BOOLEANS + return _NO_BOOLEANS + + +def _ini_bool(want, got): + """getboolean also reads 1/0; accept them only against a bool expectation.""" + + if want[0] == "bool" and got[0] == "number" and got[1] in (0, 1): + return "bool", got[1] == 1 + return got + + +def _normalise_value(value, booleans=_ASTROMATIC_BOOLEANS): + """Use tagged atoms so boolean True cannot compare equal to number 1.""" + + if isinstance(value, bool): + return "bool", value + if isinstance(value, (int, float)): + number = Decimal(str(value)) + if not number.is_finite(): + raise ValueError("numeric values must be finite") + return "number", number + if isinstance(value, (list, tuple)): + elements = tuple(_normalise_value(item, booleans) for item in value) + if any(kind == "list" for kind, _ in elements): + raise ValueError("only flat lists are supported") + return "list", elements + if not isinstance(value, str): + raise ValueError("expected a number, boolean, string or flat list") + value = value.strip() + if not value: + raise ValueError("empty values/list elements are not supported") + if value[0] in "[{'\"": + # BaseLoader keeps even 5e-4 and Y as strings, avoiding YAML 1.1's + # inconsistent numeric/boolean coercions. It constructs no objects. + parsed = yaml.load(value, Loader=yaml.BaseLoader) + if isinstance(parsed, list): + return _normalise_value(parsed, booleans) + if not isinstance(parsed, str): + raise ValueError("expected a scalar or flat list, not a mapping") + # Quotes protect commas/operators; their contents are a single atom. + value = parsed.strip() + elif "," in value: + return _normalise_value(value.split(","), booleans) + if _NUMBER.fullmatch(value): + return "number", Decimal(value) + if value.lower() in booleans: + return "bool", booleans[value.lower()] + return "text", value + + +def _square_stamp(value): + if value[0] == "number": + return "list", (value, value) + return value + + +def universe_errors(record, universe): + """Check scoped decision IDs and options against the ASTRA record.""" + + record_decisions = _decisions(record) + pinned = _decisions(universe) + errors = [] + for location in sorted(pinned.keys() - record_decisions.keys()): + errors.append( + f"{location}: universe decision is absent from astra.yaml" + ) + for location in sorted(record_decisions.keys() - pinned.keys()): + errors.append( + f"{location}: astra.yaml decision is not pinned in the universe" + ) + for location in sorted(record_decisions.keys() & pinned.keys()): + definition = record_decisions[location] + options = ( + definition.get("options", {}) + if isinstance(definition, dict) + else {} + ) + if not isinstance(options, dict) or pinned[location] not in options: + errors.append( + f"{location}: pinned option {pinned[location]!r} is not in " + "ASTRA options" + ) + return errors + + +def _decisions(document, location=""): + scope = document if isinstance(document, dict) else {} + result = {} + for decision_id, definition in (scope.get("decisions") or {}).items(): + key = ( + f"{location}.decisions.{decision_id}" + if location + else f"decisions.{decision_id}" + ) + result[key] = definition + for analysis_id, analysis in (scope.get("analyses") or {}).items(): + child = ( + f"{location}.analyses.{analysis_id}" + if location + else f"analyses.{analysis_id}" + ) + if isinstance(analysis, dict): + result.update(_decisions(analysis, child)) + return result + + +def _iter_decision_defs(document, prefix=""): + if not isinstance(document, dict): + return + for ident, definition in (document.get("decisions") or {}).items(): + yield f"{prefix}{ident}", definition + for name, analysis in (document.get("analyses") or {}).items(): + if isinstance(analysis, dict): + yield from _iter_decision_defs(analysis, f"{prefix}{name}.") + + +def _sites_for(tags, decision): + unique = {} + for tag in tags: + if decision in tag.decisions and tag.site is not None: + unique[tag.site.identity] = tag.site + return list(unique.values()) + + +def _path_matches(site_path, qualifier): + path = Path(qualifier) + if path.is_absolute() or ".." in path.parts or not qualifier: + return False + normalized = path.as_posix().lstrip("./") + return site_path == normalized or site_path.endswith("/" + normalized) + + +def _split_config_key(selector, suffix, site): + if suffix in {".ini", ".setools"} and "." in selector: + return selector.rsplit(".", 1) + section = site.section if suffix in {".ini", ".setools"} else "" + return section, selector.rsplit(".", 1)[-1] + + +def _strip_config_comment(line, suffix): + stripped = line.strip() + if not stripped or stripped.startswith(("#", ";")): + return "" + if suffix != ".ini" and "#" in stripped: + stripped = stripped.split("#", 1)[0].strip() + return stripped + + +def _config_values_in_site(root, site, selector): + """Return this config site's value for selector, or None if it doesn't.""" + + target = Path(root) / site.path + text = target.read_text(encoding="utf-8") + suffix = target.suffix.lower() + if suffix in {".yaml", ".yml"}: + data = yaml.safe_load(text) + parts = selector.split(".") + node = data + for part in parts: + if not isinstance(node, dict) or part not in node: + return None + node = node[part] + key = parts[-1] + line_matches = any( + re.match(rf"^\s*{re.escape(key)}\s*:", line) + for line in text.splitlines()[site.start - 1:site.end] + ) + return node if line_matches else None + + lines = text.splitlines() + section, key = _split_config_key(selector, suffix, site) + if suffix == ".ini": + key = key.lower() + current_section = "DEFAULT" if suffix == ".ini" else "" + values, predicates = [], [] + for number, raw in enumerate(lines, 1): + stripped = _strip_config_comment(raw, suffix) + header = re.match(r"^\[([^]]+)\]$", stripped) + if header: + current_section = header.group(1).strip() + continue + if number < site.start or number > site.end or not stripped: + continue + if suffix in {".ini", ".setools"}: + wanted_section = section or (site.section if site.scope == "section" else "") + if wanted_section and current_section != wanted_section: + continue + if suffix == ".ini": + match = re.match(r"^([^:=\s][^:=]*?)\s*[:=]\s*(.*)$", stripped) + if not match or match.group(1).strip().lower() != key: + continue + value = match.group(2).strip() + base_indent = len(raw) - len(raw.lstrip()) + for continuation in lines[number:site.end]: + if not continuation.strip(): + break + if continuation.lstrip().startswith(("#", ";")): + continue + indent = len(continuation) - len(continuation.lstrip()) + if indent <= base_indent: + break + value += " " + continuation.strip() + predicate = False + elif suffix == ".setools": + match = re.match(rf"^{re.escape(key)}(?=$|\s|=|<|>)(.*)$", stripped) + if not match: + continue + value = match.group(1).strip() + predicate = value.startswith(("==", "!=", "<", ">")) + if value.startswith("=") and not predicate: + value = value[1:].strip() + elif suffix == ".param": + match = re.match(rf"^{re.escape(key)}(?:\s*\(([^)]*)\)|\s+(.*))?$", stripped) + if not match: + continue + value = match.group(1) if match.group(1) is not None else (match.group(2) or "") + predicate = False + else: + match = re.match(rf"^{re.escape(key)}(?=$|\s|=|\()(.*)$", stripped) + if not match: + continue + value = match.group(1).strip() + predicate = False + if value.startswith("="): + value = value[1:].strip() + if not value: + raise ValueError(f"no active value for {selector!r}") + values.append(value) + predicates.append(predicate) + if not values: + return None + if len(values) > 1 and not all(predicates): + raise ValueError(f"ambiguous active values for {selector!r}: {values!r}") + return values if len(values) > 1 else values[0] + + +def _section_exists(text, suffix, section): + if suffix == ".ini": + try: + parser = _ini_parser(text, strict=False) + except configparser.Error as error: + raise ValueError(f"cannot parse INI file: {error}") from error + return section == parser.default_section or parser.has_section(section) + return any(line.strip() == f"[{section}]" for line in text.splitlines()) + + +def _yaml_key_lines(lines, selector): + """Return lines for a simple nested YAML mapping selector.""" + + wanted = selector.split(".") + stack, found = [], [] + for number, raw in enumerate(lines, 1): + match = re.match(r"^(\s*)([^:#][^:]*?)\s*:\s*(?:.*)?$", raw) + if not match: + continue + indent = len(match.group(1)) + while stack and stack[-1][0] >= indent: + stack.pop() + key = match.group(2).strip().strip("'\"") + stack.append((indent, key)) + if [part for _, part in stack] == wanted: + found.append(number) + return found + + +def _active_config_lines(root, path, selector, site=None): + """Find active line settings for a key throughout one config file.""" + + target = Path(root) / path + suffix = target.suffix.lower() + lines = target.read_text(encoding="utf-8").splitlines() + if suffix in {".yaml", ".yml"}: + return _yaml_key_lines(lines, selector) + + file_site = site or Site(path, 1, len(lines), "config", scope="file") + section, key = _split_config_key(selector, suffix, file_site) + if suffix == ".ini": + key = key.lower() + current_section = "DEFAULT" if suffix == ".ini" else "" + found = [] + for number, raw in enumerate(lines, 1): + stripped = _strip_config_comment(raw, suffix) + if not stripped: + continue + header = re.match(r"^\[([^]]+)\]$", stripped) + if header: + current_section = header.group(1).strip() + continue + if suffix in {".ini", ".setools"} and section: + if current_section != section: + continue + if suffix == ".ini": + match = re.match(r"^([^:=\s][^:=]*?)\s*(?:[:=]\s*(.*))?$", stripped) + active_key = match.group(1).strip().lower() if match else None + elif suffix == ".setools": + match = re.match(rf"^{re.escape(key)}(?=$|\s|=|<|>)(.*)$", stripped) + active_key = key if match else None + elif suffix == ".param": + match = re.match( + rf"^{re.escape(key)}(?:\s*\(([^)]*)\)|\s+(.*))?$", stripped + ) + active_key = key if match else None + else: + match = re.match(rf"^{re.escape(key)}(?=$|\s|=|\()(.*)$", stripped) + active_key = key if match else None + if active_key == key: + found.append(number) + return found + + +def _config_absent_actual(root, path, selector): + """Return whether a config setting is active anywhere in its target scope.""" + + target = Path(root) / path + suffix = target.suffix.lower() + text = target.read_text(encoding="utf-8") + file_site = Site(path, 1, len(text.splitlines()), "config", scope="file") + section, key = _split_config_key(selector, suffix, file_site) + if suffix == ".ini": + key = key.lower() + try: + parser = _ini_parser(text, strict=False) + except configparser.Error as error: + raise ValueError(f"cannot parse INI file: {error}") from error + if section and not _section_exists(text, suffix, section): + return None + sections = [section] if section else [parser.default_section, *parser.sections()] + active_sections = [ + name for name in sections if parser.has_option(name, key) + ] + if section and section != parser.default_section: + explicit = parser._sections.get(section, {}) + if key not in explicit and parser.has_option(parser.default_section, key): + return f"active setting inherited from [{parser.default_section}]" + return ( + f"active setting in section(s) {active_sections!r}" + if active_sections else _NO_SETTING + ) + if suffix == ".setools" and section and not _section_exists( + text, suffix, section + ): + return None + active = _active_config_lines(root, path, selector) + return f"active setting on line(s) {active!r}" if active else _NO_SETTING + + +def _check_absent_entry(root, decision, ref, kind, qualifier, selector, tags): + """Check absence against a same-decision tag in the file or section.""" + + if kind == "python": + return ( + f"{decision}: ref {ref!r}: expected no active setting, actual " + " (absence refs apply only to config files)" + ) + matching_paths = set() + for tag in tags: + if decision not in tag.decisions or tag.site is None: + continue + site = tag.site + if qualifier and not _path_matches(site.path, qualifier): + continue + suffix = Path(site.path).suffix.lower() + if suffix not in _CONFIG_SUFFIXES: + continue + section, _ = _split_config_key(selector, suffix, site) + if suffix in {".ini", ".setools"} and section: + if site.scope != "file" and site.section != section: + continue + try: + text = (Path(root) / site.path).read_text(encoding="utf-8") + except OSError: + continue + if not _section_exists(text, suffix, section): + continue + matching_paths.add(site.path) + + if len(matching_paths) != 1: + actual = ( + "" if not matching_paths + else f"" + ) + return ( + f"{decision}: ref {ref!r}: expected no active setting, actual " + f"{actual} (absence requires a same-decision tag in the file or " + f"named section; found {len(matching_paths)})" + ) + + path, = matching_paths + try: + actual = _config_absent_actual(root, path, selector) + except (ValueError, OSError, configparser.Error, yaml.YAMLError) as error: + return ( + f"{decision}: ref {ref!r}: expected no active setting, " + f"actual : {error}" + ) + if actual is None: + section, _ = _split_config_key( + selector, Path(path).suffix.lower(), + Site(path, 1, 1, "config", scope="file"), + ) + return ( + f"{decision}: ref {ref!r}: expected no active setting, actual " + f" (section {section!r} does not exist)" + ) + if actual is _NO_SETTING: + return None + return f"{decision}: ref {ref!r}: expected no active setting, actual {actual}" + + +def _python_value_in_site(root, site, selector): + source = (Path(root) / site.path).read_text(encoding="utf-8") + tree = _python_tree(source) + try: + symbol, _ = _code_selector(selector) + except ValueError: + return None + direct = bool( + site.symbol + and (symbol == site.symbol or symbol.startswith(site.symbol + ".")) + ) + full_selector = ( + selector if direct else f"{site.symbol}.{selector}" + if site.symbol else selector + ) + try: + node = _selected_python_node(tree, full_selector) + except ValueError as error: + message = str(error) + if "needs one binding" in message: + found = re.search(r"found (\d+)", message) + if found and int(found.group(1)) > 1: + raise ValueError(f"ambiguous binding: {message}") from error + return None + if not direct: + return None + if "missing" in message: + return None + raise + try: + return ast.literal_eval(node) + except (ValueError, TypeError) as error: + raise ValueError("selected Python value is not a literal") from error + + +def _site_actual(root, site, kind, selector): + suffix = Path(site.path).suffix.lower() + if kind == "python": + if suffix != ".py": + return None + return _python_value_in_site(root, site, selector) + if suffix not in _CONFIG_SUFFIXES: + return None + return _config_values_in_site(root, site, selector) + + +def _value_comparison(expected, actual, suffix, selector): + booleans = _booleans_for(suffix) + want = _normalise_value(expected, booleans) + got = _normalise_value(actual, booleans) + if suffix == ".ini": + got = _ini_bool(want, got) + if (suffix, selector.rsplit(".", 1)[-1]) in { + (".param", "VIGNET"), (".psfex", "PSF_SIZE") + }: + want, got = _square_stamp(want), _square_stamp(got) + return want == got + + +def _check_value_entry(root, decision, reference, tags): + try: + ref, expected = _split_assertion(reference) + kind, qualifier, selector = _parse_reference(ref) + if kind == "bare": + if not selector or re.search(r"\s", selector): + raise ValueError("invalid unqualified ref") + elif not qualifier or Path(qualifier).is_absolute() or ".." in Path(qualifier).parts: + raise ValueError("path qualifier must be a relative path suffix") + except ValueError as error: + return f"{decision}: ref {reference!r}: expected , actual : {error}" + + if expected == ABSENT: + return _check_absent_entry( + root, decision, ref, kind, qualifier, selector, tags + ) + + candidates = [] + for site in _sites_for(tags, decision): + suffix = Path(site.path).suffix.lower() + expected_kind = "python" if suffix == ".py" else "config" + if kind != "bare" and kind != expected_kind: + continue + if qualifier and not _path_matches(site.path, qualifier): + continue + try: + actual = _site_actual(root, site, expected_kind, selector) + except (ValueError, OSError, SyntaxError, configparser.Error, yaml.YAMLError) as error: + return f"{decision}: ref {ref!r}: expected {expected!r}, actual : {error}" + if actual is not None: + if expected_kind == "config": + try: + active_lines = _active_config_lines( + root, site.path, selector, site + ) + except (ValueError, OSError, configparser.Error, yaml.YAMLError) as error: + return f"{decision}: ref {ref!r}: expected {expected!r}, actual : {error}" + outside = [ + number for number in active_lines + if not site.start <= number <= site.end + ] + if outside: + return ( + f"{decision}: ref {ref!r}: expected {expected!r}, " + f"actual {actual!r}; duplicate active setting outside " + f"the governed paragraph on line(s) {outside!r}" + ) + candidates.append((site, actual)) + + if len(candidates) != 1: + actual = "" if not candidates else f"" + return f"{decision}: ref {ref!r}: expected {expected!r}, actual {actual} (ref must resolve to exactly one tagged site; found {len(candidates)})" + + site, actual = candidates[0] + if expected == ABSENT: + if actual is _NO_SETTING: + return None + return f"{decision}: ref {ref!r}: expected no active setting, actual {actual!r}" + try: + suffix = Path(site.path).suffix.lower() + if _value_comparison(expected, actual, suffix, selector): + return None + return f"{decision}: ref {ref!r}: expected {expected!r}, actual {actual!r}" + except (ValueError, OSError, SyntaxError, configparser.Error, yaml.YAMLError) as error: + return f"{decision}: ref {ref!r}: expected {expected!r}, actual {actual!r}: {error}" + + +def value_errors(root, record, tags=None): + """Check every Values entry against exactly one site tagged for its decision.""" + + root = Path(root) + if tags is None: + tags, _ = scan_tags(root) + errors = [] + for decision, definition in _iter_decision_defs(record): + rationale = definition.get("rationale") if isinstance(definition, dict) else None + if not isinstance(rationale, str): + continue + entries, problem = _parse_values(rationale) + if problem: + errors.append(f"{decision}: {problem}") + continue + for entry in entries: + try: + _split_assertion(entry) + except ValueError as error: + errors.append(f"{decision}: Values ref {entry!r}: {error}") + continue + result = _check_value_entry(root, decision, entry, tags) + if result: + errors.append(result) + return errors + + +def _is_decision_marker_call(node): + function = node.func + return ( + isinstance(function, ast.Attribute) and function.attr == "decision" + and isinstance(function.value, ast.Attribute) and function.value.attr == "mark" + and isinstance(function.value.value, ast.Name) + and function.value.value.id == "pytest" + ) + + +def decision_markers(root): + root = Path(root) + markers, errors = [], [] + test_root = root / "tests" + if not test_root.is_dir(): + return markers, errors + for path in sorted(test_root.rglob("*.py")): + if any(part in _SKIP for part in path.parts): + continue + relative = path.relative_to(root).as_posix() + try: + tree = ast.parse(path.read_text(encoding="utf-8"), filename=relative) + except (OSError, SyntaxError) as error: + errors.append(f"{relative}: cannot parse decision markers: {error}") + continue + calls = sorted( + (node for node in ast.walk(tree) if isinstance(node, ast.Call) + and _is_decision_marker_call(node)), + key=lambda node: (node.lineno, node.col_offset), + ) + for call in calls: + where = f"{relative}:{call.lineno}" + if not call.args: + errors.append(f"{where}: decision marker needs literal string ids") + if call.keywords: + errors.append(f"{where}: decision markers accept positional ids only") + for argument in call.args: + if isinstance(argument, ast.Constant) and isinstance(argument.value, str): + markers.append(DecisionMarker(argument.value, relative, call.lineno)) + else: + errors.append(f"{where}: decision marker ids must be literal strings") + return markers, errors + + +def decision_marker_errors(markers, record): + known = decision_ids(record) + return [f"{marker.path}:{marker.line}: decision marker cites unknown decision {marker.decision!r}" + for marker in markers if marker.decision not in known] + + +def _rationale_sentence(definition): + text = str(definition.get("rationale", "")).strip() + text = re.sub(r"\s+", " ", text) + parts = re.split(r"(?<=[.!?])\s+", text, maxsplit=1) + return parts[0] if parts else "" + + +def _value_sentence(definition): + entries, problem = _parse_values(definition.get("rationale", "")) + return "Values: " + "; ".join(entries) + "." if entries and not problem else "" + + +def repository_errors(root): + """Run the bidirectional tag/value and marker checks for a repository.""" + + root = Path(root) + record = load_yaml(root / "astra.yaml") + tags, errors = scan_tags(root) + markers, marker_parse_errors = decision_markers(root) + errors = list(errors) + errors.extend(tag_errors(tags, record)) + errors.extend(value_errors(root, record, tags)) + errors.extend(marker_parse_errors) + errors.extend(decision_marker_errors(markers, record)) + universe_path = root / "universes" / "committed.yaml" + if universe_path.is_file(): + errors.extend(universe_errors(record, load_yaml(universe_path))) + return errors + + +_FORBID = re.compile(r"^\s*forbid:\s*(\S+)\s*->\s*(\S+)\s*$") +_CONTRACT_TAG = re.compile(r"^\s*@(sc|cc)(?:\s+\[[^]]*\])?\s+([\w][\w.-]*)\s*$") + + +def forbid_rules(contracts_file): + """Return ``(contract_id, source, target)`` import-boundary rules.""" + + rules, ident = [], None + for line in Path(contracts_file).read_text(encoding="utf-8").splitlines(): + match = _CONTRACT_TAG.match(line) + if match: + ident = match.group(2) + continue + match = _FORBID.match(line) + if match and ident: + rules.append((ident, *match.groups())) + return rules + + +def module_matches(name, pattern): + return fnmatch.fnmatchcase(name, pattern) or ( + pattern.endswith(".*") and name == pattern[:-2] + ) + + +def module_name(path, src_root): + parts = list(Path(path).relative_to(src_root).with_suffix("").parts) + if parts[-1] == "__init__": + parts.pop() + return ".".join(parts) + + +def imported_names(path, module): + """Yield ``(line, dotted_name)`` for imports in a Python module.""" + + with warnings.catch_warnings(): + warnings.simplefilter("ignore", SyntaxWarning) + tree = ast.parse(Path(path).read_text(encoding="utf-8")) + package = module.split(".") + if Path(path).name != "__init__.py": + package = package[:-1] + for node in ast.walk(tree): + if isinstance(node, ast.Import): + for alias in node.names: + yield node.lineno, alias.name + elif isinstance(node, ast.ImportFrom): + base = node.module or "" + if node.level: + parent = package[:len(package) - (node.level - 1)] + base = ".".join([*parent, *([base] if base else [])]) + yield node.lineno, base + for alias in node.names: + if alias.name != "*": + yield node.lineno, f"{base}.{alias.name}" + + +def import_violations(src_root, rules): + """Find imports forbidden by ``CONTRACTS`` rules.""" + + src_root = Path(src_root) + found = [] + for path in sorted(src_root.rglob("*.py")): + if any(part in _SKIP for part in path.parts): + continue + module = module_name(path, src_root) + for ident, source, target in rules: + if not module_matches(module, source): + continue + for line, name in imported_names(path, module): + if module_matches(name, target): + found.append( + f"{path.relative_to(src_root)}:{line}: {module} imports " + f"{name} (contract {ident})" + ) + return found + + +def _decision_description(record, decision): + for ident, definition in _iter_decision_defs(record): + if ident == decision: + label = definition.get("label", decision) if isinstance(definition, dict) else decision + return label, _rationale_sentence(definition), _value_sentence(definition) + return decision, "", "" + + +def _location_path(root, value): + raw = Path(value) + if raw.is_absolute(): + try: + return raw.resolve().relative_to(Path(root).resolve()).as_posix() + except ValueError: + return None + return raw.as_posix().lstrip("./") + + +def _print_location(root, record, tags, location): + line = None + match = re.match(r"^(.*):(\d+)$", location) + if match: + location, line = match.group(1), int(match.group(2)) + relative = _location_path(root, location) + if relative is None: + print(f"{location}: outside repository") + return + matches = [] + for tag in tags: + site = tag.site + if site is None or site.path != relative: + continue + if line is not None and not (site.start <= line <= site.end or tag.line == line): + continue + matches.append(tag) + by_decision = sorted({decision for tag in matches for decision in tag.decisions}) + print(f"{relative}" + (f":{line}" if line is not None else "")) + for decision in by_decision: + label, rationale, values = _decision_description(record, decision) + print(f" {decision} — {label}") + if rationale: + print(f" {rationale}") + if values: + print(f" {values}") + for tag in matches: + if tag.ident: + label = tag.meta.get("label", [""])[0] + suffix = f" [{label}]" if label else "" + print(f" Local contract {tag.ident}{suffix}: {tag.prose}") + if not matches: + print(" No tagged site governs this location.") + + +def _print_decision_sites(record, tags, decision): + label, rationale, values = _decision_description(record, decision) + print(f"{decision} — {label}") + if rationale: + print(f" {rationale}") + if values: + print(f" {values}") + sites = _sites_for(tags, decision) + if not sites: + print(" No tagged sites.") + for site in sorted(sites, key=lambda item: (item.path, item.start)): + print(f" {site.path}:{site.start}-{site.end} ({site.kind})") + for tag in tags: + if decision in tag.decisions and tag.site and tag.site.identity == site.identity and tag.ident: + print(f" Local contract {tag.ident}: {tag.prose}") + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("location", nargs="?", help="path[:line] to inspect") + parser.add_argument("--decision", help="list all sites for one decision") + parser.add_argument("--root", default=Path(__file__).resolve().parents[2]) + args = parser.parse_args(argv) + root = Path(args.root) + record = load_yaml(root / "astra.yaml") + tags, errors = scan_tags(root) + if args.decision: + _print_decision_sites(record, tags, args.decision) + elif args.location: + _print_location(root, record, tags, args.location) + else: + parser.error("supply a path[:line] or --decision ") + if errors: + print(f"\n{len(errors)} tag parse error(s):") + for error in errors: + print(f" {error}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tests/module/test_make_cat.py b/tests/module/test_make_cat.py index 63c759d2b..576ad2c95 100644 --- a/tests/module/test_make_cat.py +++ b/tests/module/test_make_cat.py @@ -16,6 +16,7 @@ import numpy as np import numpy.testing as npt +import pytest from astropy.io import fits from sqlitedict import SqliteDict @@ -354,14 +355,14 @@ def get_data(self): return {"N_EPOCH": self._n_epoch} -def _run_save_psf(galaxy_psf_path, obj_id, n_epoch): +def _run_save_psf(galaxy_psf_path, obj_id, n_epoch, n_epoch_slots=None): """Drive ``_save_psf_data`` and return its populated output dict.""" inst = object.__new__(SaveCatalogue) inst._obj_id = np.asarray(obj_id) inst._output_dict = {} inst._final_cat_file = _FinalCatStub(n_epoch) - inst._save_psf_data(str(galaxy_psf_path)) + inst._save_psf_data(str(galaxy_psf_path), n_epoch_slots=n_epoch_slots) return inst._output_dict @@ -455,6 +456,194 @@ def test_save_psf_data_fills_sentinel_for_absent_epochs(tmp_path): assert out[col][1] == -1, col +# --- _save_psf_data: fixed per-epoch slot count (N_EPOCH_SLOTS) --- + +# Per-family empty-slot sentinel: what a slot holds when no epoch fills it. +_PSF_SLOT_SENTINELS = { + "HSM_G1_PSF": -10.0, + "HSM_G2_PSF": -10.0, + "HSM_T_PSF": 0.0, + "HSM_FLAG_PSF": 1, + "HSM_M4_1_PSF": -10.0, + "HSM_M4_2_PSF": -10.0, + "HSM_RHO4_PSF": -1.0, + "EXP_ID": -1, + "CCD": -1, +} + + +class _ProcessCatStub(_FinalCatStub): + """FITSCatalogue stand-in for ``SaveCatalogue.process``; records add_col.""" + + def __init__(self, obj_id, n_epoch): + super().__init__(n_epoch) + self._number = np.asarray(obj_id) + self.cols = {} + + def open(self): + pass + + def close(self): + pass + + def get_data(self): + return {"NUMBER": self._number, "N_EPOCH": self._n_epoch} + + def add_col(self, name, data): + self.cols[name] = data + + +def _slot_numbers(out, family): + """The slot numbers ``n`` present in ``out`` for ``_n`` columns.""" + prefix = f"{family}_" + return sorted( + int(col[len(prefix):]) + for col in out + if col.startswith(prefix) and col[len(prefix):].isdigit() + ) + + +def test_save_psf_data_fixed_slots_pad_every_family(tmp_path): + """N_EPOCH_SLOTS fixes the slot count for every family, sentinel-padded. + + The campaign merge needs one schema across tiles, so a tile whose + objects all have far fewer epochs than N_EPOCH_SLOTS still writes + exactly slots 1..N_EPOCH_SLOTS for each per-epoch family, and every + slot no epoch fills holds that family's own sentinel. + """ + galaxy_psf_path = tmp_path / "galaxy_psf.sqlite" + per_obj = { + 101: {"2113864-7": _psf_epoch(0.01, 0.02, 0.5)}, + 202: { + "2113864-9": _psf_epoch(0.05, 0.06, 0.7), + "2358123-21": _psf_epoch(0.07, 0.08, 0.9), + }, + 303: "empty", + } + _write_galaxy_psf_cat(galaxy_psf_path, per_obj) + + # Driven through ``process``, the entry point the runner calls. + n_slots = 7 + cat = _ProcessCatStub([101, 202, 303], n_epoch=[1, 2, 0]) + sc = SaveCatalogue(cat, 3, _NullLogger()) + assert sc.process("psf", str(galaxy_psf_path), n_epoch_slots=n_slots) is None + out = cat.cols + + n_filled = [1, 2, 0] + for family, sentinel in _PSF_SLOT_SENTINELS.items(): + assert _slot_numbers(out, family) == list(range(1, n_slots + 1)), family + for row, filled in enumerate(n_filled): + for n in range(filled + 1, n_slots + 1): + assert out[f"{family}_{n}"][row] == sentinel, (family, row, n) + + +def test_save_psf_data_more_epochs_than_slots_raises(tmp_path): + """An object with more epochs than N_EPOCH_SLOTS raises, never truncates. + + Dropping the extra epochs would silently lose per-epoch PSF data; the + error names the object, its N_EPOCH and the slot count so the config + can be fixed. + """ + galaxy_psf_path = tmp_path / "galaxy_psf.sqlite" + per_obj = { + 101: {"2113864-7": _psf_epoch(0.01, 0.02, 0.5)}, + 404: { + "2113864-7": _psf_epoch(0.01, 0.02, 0.5), + "2229900-13": _psf_epoch(0.03, 0.04, 0.6), + "2358123-21": _psf_epoch(0.07, 0.08, 0.9), + }, + } + _write_galaxy_psf_cat(galaxy_psf_path, per_obj) + + with pytest.raises(ValueError) as excinfo: + _run_save_psf( + galaxy_psf_path, [101, 404], n_epoch=[1, 3], n_epoch_slots=2 + ) + msg = str(excinfo.value) + assert "404" in msg + assert "N_EPOCH=3" in msg + assert "N_EPOCH_SLOTS=2" in msg + + +def test_save_psf_data_exactly_slots_epochs_fits(tmp_path): + """An object whose epochs exactly fill N_EPOCH_SLOTS is written, not raised.""" + galaxy_psf_path = tmp_path / "galaxy_psf.sqlite" + per_obj = { + 404: { + "2113864-7": _psf_epoch(0.01, 0.02, 0.5), + "2229900-13": _psf_epoch(0.03, 0.04, 0.6), + }, + } + _write_galaxy_psf_cat(galaxy_psf_path, per_obj) + + out = _run_save_psf(galaxy_psf_path, [404], n_epoch=[2], n_epoch_slots=2) + + assert _slot_numbers(out, "EXP_ID") == [1, 2] + assert out["EXP_ID_2"][0] == 2229900 + npt.assert_allclose(out["HSM_G1_PSF_2"], [0.03]) + + +def test_save_psf_data_unset_slots_uses_tile_max_n_epoch_plus_one(tmp_path): + """Without N_EPOCH_SLOTS the slot count is the tile's max(N_EPOCH) + 1.""" + galaxy_psf_path = tmp_path / "galaxy_psf.sqlite" + per_obj = { + 101: {"2113864-7": _psf_epoch(0.01, 0.02, 0.5)}, + 202: { + "2113864-9": _psf_epoch(0.05, 0.06, 0.7), + "2229900-13": _psf_epoch(0.03, 0.04, 0.6), + "2358123-21": _psf_epoch(0.07, 0.08, 0.9), + }, + } + _write_galaxy_psf_cat(galaxy_psf_path, per_obj) + + out = _run_save_psf(galaxy_psf_path, [101, 202], n_epoch=[1, 3]) + + for family in _PSF_SLOT_SENTINELS: + assert _slot_numbers(out, family) == [1, 2, 3, 4], family + + +def test_save_psf_data_fixed_slots_keep_epoch_alignment(tmp_path): + """Under padding, slot n of every family still names the same epoch. + + Epochs fill slots 1..k in the galaxy_psf key order and padding sits + only in slots k+1..N_EPOCH_SLOTS, for every family alike; a flagged + epoch keeps its identity in its own slot while its HSM columns stay + at the sentinel. + """ + galaxy_psf_path = tmp_path / "galaxy_psf.sqlite" + epochs = [ + # (key, g1, g2, t, flag) in deliberately unsorted key order + ("2358123-21", 0.07, 0.08, 0.9, 0), + ("2113864-9", -10.0, -10.0, 0.0, 5), + ("2229900-13", 0.03, 0.04, 0.6, 0), + ] + per_obj = { + 505: {key: _psf_epoch(g1, g2, t, flag) for key, g1, g2, t, flag in epochs}, + } + _write_galaxy_psf_cat(galaxy_psf_path, per_obj) + + n_slots = 6 + out = _run_save_psf( + galaxy_psf_path, [505], n_epoch=[3], n_epoch_slots=n_slots + ) + + for n, (key, g1, g2, t, flag) in enumerate(epochs, start=1): + exp_id, ccd = (int(part) for part in key.split("-")) + assert out[f"EXP_ID_{n}"][0] == exp_id, n + assert out[f"CCD_{n}"][0] == ccd, n + if flag == 0: + npt.assert_allclose(out[f"HSM_G1_PSF_{n}"], [g1]) + npt.assert_allclose(out[f"HSM_G2_PSF_{n}"], [g2]) + npt.assert_allclose(out[f"HSM_T_PSF_{n}"], [t]) + assert out[f"HSM_FLAG_PSF_{n}"][0] == 0, n + else: + npt.assert_allclose(out[f"HSM_G1_PSF_{n}"], [-10.0]) + assert out[f"HSM_FLAG_PSF_{n}"][0] == 1, n + for n in range(len(epochs) + 1, n_slots + 1): + for family, sentinel in _PSF_SLOT_SENTINELS.items(): + assert out[f"{family}_{n}"][0] == sentinel, (family, n) + + def test_save_psf_data_carries_fourth_moments_per_epoch(tmp_path): """HSM_M4_1/M4_2/RHO4_PSF_n ride the same slots as HSM_G1_PSF_n. diff --git a/tests/science/test_additive_null.py b/tests/science/test_additive_null.py index c8440071b..001a24982 100644 --- a/tests/science/test_additive_null.py +++ b/tests/science/test_additive_null.py @@ -30,9 +30,12 @@ """ import numpy as np +import pytest from tests.helpers.metacal_sim import recover +pytestmark = pytest.mark.decision("shape_measurement.metacal_scheme") + SEEDS = list(range(8)) # deterministic ensemble; additive bias is statistical PSF_E1 = 0.05 # true PSF ellipticity the deconvolution must remove C_TOL = 1e-3 # |c| null (twin's published bound); mean ~5e-6, worst seed ~5e-5 diff --git a/tests/science/test_mbias.py b/tests/science/test_mbias.py index 4efb7c3dd..b9a7981fb 100644 --- a/tests/science/test_mbias.py +++ b/tests/science/test_mbias.py @@ -26,6 +26,8 @@ from tests.helpers.artifacts import emit_mbias_artifacts +pytestmark = pytest.mark.decision("shape_measurement.metacal_scheme") + # The GitHub Pages publish seam (see tests/_artifacts/README.md). _ARTIFACTS_DIR = Path(__file__).resolve().parents[1] / "_artifacts" diff --git a/tests/science/test_resolution_ladder.py b/tests/science/test_resolution_ladder.py index c1e0a6c2b..eb1e68e87 100644 --- a/tests/science/test_resolution_ladder.py +++ b/tests/science/test_resolution_ladder.py @@ -147,6 +147,7 @@ def rungs(): return ladder +@pytest.mark.decision("shape_measurement.metacal_scheme") def test_estimator_has_power(rungs): """Every resolved rung's σ_m is small enough that its assert has teeth. @@ -168,6 +169,7 @@ def test_estimator_has_power(rungs): ) +@pytest.mark.decision("shape_measurement.metacal_scheme") def test_resolved_rungs_unbiased(rungs): """On resolved rungs (ratio >= 0.5), ``|m|`` stays below a few x 1e-3. @@ -189,6 +191,7 @@ def test_resolved_rungs_unbiased(rungs): ) +@pytest.mark.decision("shape_measurement.metacal_scheme") def test_response_positive_all_rungs(rungs): """Every rung has a non-degenerate positive response ``R11 > 0.1``. diff --git a/tests/science/test_star_response.py b/tests/science/test_star_response.py index 23a984b6f..d52d3c251 100644 --- a/tests/science/test_star_response.py +++ b/tests/science/test_star_response.py @@ -33,7 +33,9 @@ """ import numpy as np +import pytest +pytestmark = pytest.mark.decision("shape_measurement.metacal_scheme") PSF_E1 = 0.05 # true PSF ellipticity the deconvolution must remove METACAL_STEP = 0.01 # ngmix MetacalBootstrapper default shear step diff --git a/tests/science/test_symmetry.py b/tests/science/test_symmetry.py index ca019543d..e44e769e3 100644 --- a/tests/science/test_symmetry.py +++ b/tests/science/test_symmetry.py @@ -22,9 +22,12 @@ class of bug a single-component m-bias test cannot see: a g1<->g2 swap, a status; this is a pure tripwire. """ import numpy as np +import pytest from tests.helpers.metacal_sim import recover +pytestmark = pytest.mark.decision("shape_measurement.metacal_scheme") + SEED = 42 INJECTED = 0.02 # per-component injected shear magnitude for the arms diff --git a/tests/unit/test_committed_configs_run.py b/tests/unit/test_committed_configs_run.py new file mode 100644 index 000000000..a20fbfb1f --- /dev/null +++ b/tests/unit/test_committed_configs_run.py @@ -0,0 +1,120 @@ +"""Exercise committed SExtractor/PSFEx configs against the real tools.""" + +from pathlib import Path +import shutil +import subprocess + +import numpy as np +import pytest +from astropy.io import fits +from astropy.wcs import WCS + + +ROOT = Path(__file__).resolve().parents[2] +CONFIG_DIR = ROOT / "workflow" / "config" / "cfis" +SEX = shutil.which("source-extractor") +PSFEX = shutil.which("psfex") + + +def _synthetic_images(directory): + """Write a WCS image with isolated stars and the maps used by both runners.""" + size = 512 + wcs = WCS(naxis=2) + wcs.wcs.crpix = [size / 2, size / 2] + wcs.wcs.cdelt = [-5.16e-5, 5.16e-5] + wcs.wcs.crval = [180.0, 30.0] + wcs.wcs.ctype = ["RA---TAN", "DEC--TAN"] + header = wcs.to_header() + header["GAIN"] = 1.0 + header["SATURATE"] = 60000.0 + header["PHOTZP"] = 30.0 + + rng = np.random.default_rng(42) + yy, xx = np.mgrid[:size, :size] + image = rng.normal(100.0, 3.0, (size, size)).astype(np.float32) + for x in np.linspace(55, size - 55, 4): + for y in np.linspace(55, size - 55, 4): + image += 5000.0 * np.exp( + -((xx - x) ** 2 + (yy - y) ** 2) / 8.0 + ) + + fits.PrimaryHDU(image, header).writeto(directory / "image.fits") + fits.PrimaryHDU(np.ones_like(image)).writeto(directory / "weight.fits") + fits.PrimaryHDU(np.zeros_like(image, dtype=np.int16)).writeto( + directory / "flag.fits" + ) + + +def _run_sextractor(directory, *, tile): + config = CONFIG_DIR / ("default_tile.sex" if tile else "default_exp.sex") + parameters = CONFIG_DIR / ( + "default_noimaflags.param" if tile else "default.param" + ) + convolution = CONFIG_DIR / ( + "gauss_3.0_7x7.conv" if tile else "default.conv" + ) + catalogue = directory / ("tile.cat" if tile else "exposure.cat") + command = [ + SEX, + "image.fits", + "-c", str(config), + "-PARAMETERS_NAME", str(parameters), + "-FILTER_NAME", str(convolution), + "-CATALOG_NAME", str(catalogue), + "-WEIGHT_IMAGE", "weight.fits", + "-FLAG_IMAGE", "NONE" if tile else "flag.fits", + "-VERBOSE_TYPE", "QUIET", + "-WRITE_XML", "N", + ] + result = subprocess.run( + command, + cwd=directory, + check=False, + capture_output=True, + text=True, + timeout=8, + ) + assert result.returncode == 0, result.stdout + result.stderr + with fits.open(catalogue) as hdus: + objects = next(hdu for hdu in hdus if hdu.name == "LDAC_OBJECTS") + assert objects.data is not None and len(objects.data) > 0 + return catalogue + + +def test_committed_convolution_filters_start_with_sextractor_directive(): + """SExtractor requires CONV on line 1 of every convolution file.""" + conv_files = sorted(CONFIG_DIR.rglob("*.conv")) + assert conv_files + for path in conv_files: + assert path.read_text(encoding="utf-8").splitlines()[0].startswith("CONV "), path + + +@pytest.mark.skipif(not (SEX and PSFEX), reason="source-extractor and psfex are required") +def test_committed_sextractor_and_psfex_configs_run(tmp_path): + """Both detection configs emit catalogues and the exposure catalogue fits a PSF.""" + _synthetic_images(tmp_path) + _run_sextractor(tmp_path, tile=True) + exposure_catalogue = _run_sextractor(tmp_path, tile=False) + + subprocess.run( + [ + PSFEX, + str(exposure_catalogue), + "-c", str(CONFIG_DIR / "default.psfex"), + # Smaller fixture stamp keeps this committed-config smoke test fast. + "-PSF_SIZE", "21,21", + "-PSF_SUFFIX", ".psf", + "-WRITE_XML", "N", + "-CHECKIMAGE_TYPE", "NONE", + "-CHECKPLOT_TYPE", "NONE", + "-VERBOSE_TYPE", "QUIET", + ], + cwd=tmp_path, + check=True, + capture_output=True, + text=True, + timeout=8, + ) + psf_file = exposure_catalogue.with_suffix(".psf") + assert psf_file.is_file() + assert psf_file.stat().st_size > 0 diff --git a/tests/unit/test_contracts.py b/tests/unit/test_contracts.py new file mode 100644 index 000000000..4091adb5e --- /dev/null +++ b/tests/unit/test_contracts.py @@ -0,0 +1,43 @@ +"""Preserve the utilities import-boundary contract.""" + +import textwrap +from pathlib import Path + +from tests.helpers.decisions import forbid_rules, import_violations + + +REPO_ROOT = Path(__file__).resolve().parents[2] + + +def _write(root, relative, text): + path = root / relative + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(textwrap.dedent(text), encoding="utf-8") + + +def test_forbidden_import_is_found(tmp_path): + _write(tmp_path, "pkg/utilities/CONTRACTS", """ + @cc no-up-imports + forbid: pkg.utilities.* -> pkg.modules.* + """) + _write(tmp_path, "pkg/utilities/good.py", "import os\n") + _write(tmp_path, "pkg/utilities/bad.py", "from ..modules import runner\n") + _write(tmp_path, "pkg/modules/runner.py", "from pkg.utilities import good\n") + + rules = forbid_rules(tmp_path / "pkg/utilities/CONTRACTS") + violations = import_violations(tmp_path, rules) + + assert rules == [("no-up-imports", "pkg.utilities.*", "pkg.modules.*")] + assert len(violations) == 2 + assert all("bad.py:1" in problem and "no-up-imports" in problem + for problem in violations) + + +def test_utilities_do_not_import_modules(): + contracts_file = REPO_ROOT / "src/shapepipe/utilities/CONTRACTS" + rules = forbid_rules(contracts_file) + assert [rule[0] for rule in rules] == ["utilities-do-not-import-modules"] + + violations = import_violations(REPO_ROOT / "src", rules) + message = "Forbidden imports:\n - " + "\n - ".join(violations) + assert not violations, message diff --git a/tests/unit/test_decisions.py b/tests/unit/test_decisions.py new file mode 100644 index 000000000..8c0f39766 --- /dev/null +++ b/tests/unit/test_decisions.py @@ -0,0 +1,693 @@ +"""Decision-tag parsing and the static Values resolver.""" + +from pathlib import Path + +import pytest + +from tests.helpers.decisions import ( + decision_ids, + decision_marker_errors, + decision_markers, + forbid_rules, + import_violations, + load_yaml, + main, + repository_errors, + scan_tags, + tag_errors, + value_errors, +) + + +REPO_ROOT = Path(__file__).resolve().parents[2] + + +def _record(rationale="Values: THRESH = 1.", decision="choice"): + return {"decisions": {decision: {"label": "Choice", "rationale": rationale}}} + + +def _write(path, text): + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(text, encoding="utf-8") + return path + + +def _config_tag(path, decision="choice", content="THRESH 1\n"): + return _write(path, f"# @sc [decision:{decision}]\n{content}") + + +def test_file_scope_tag_can_follow_a_required_config_header(tmp_path): + config = _write( + tmp_path / "filter.conv", + "CONV NORM\n# @sc [decision:choice,scope:file]\n1\n", + ) + tags, errors = scan_tags(tmp_path) + + assert errors == [] + assert len(tags) == 1 + assert tags[0].site.path == "filter.conv" + assert tags[0].site.scope == "file" + assert (tags[0].site.start, tags[0].site.end) == (1, 3) + + +def test_config_tags_govern_paragraph_and_section(tmp_path): + config = _write( + tmp_path / "settings.ini", + "# @sc [decision:choice]\n[SCIENCE]\nKEY = 1\n# comment\nOTHER = 2\n\n" + "# @sc [decision:choice]\n[OUTPUT]\nSAVE = True\n", + ) + tags, errors = scan_tags(tmp_path) + + assert errors == [] + assert len(tags) == 2 + science, output = tags + assert science.site.path == "settings.ini" + assert (science.site.start, science.site.end) == (2, 6) + assert science.site.section == "SCIENCE" + assert output.site.section == "OUTPUT" + assert output.site.scope == "section" + + +def test_section_tag_covers_the_entire_section_across_nested_key_tags(tmp_path): + _write( + tmp_path / "settings.ini", + "# @sc [decision:section_choice]\n[S]\nA = 1\n" + "# @sc [decision:key_choice]\nB = 2\n\n" + "# @sc [decision:next_choice]\n[N]\nC = 3\n", + ) + tags, errors = scan_tags(tmp_path) + + assert errors == [] + section_tag = next(tag for tag in tags if tag.decisions == ("section_choice",)) + assert section_tag.site.section == "S" + assert section_tag.site.start <= 6 <= section_tag.site.end + + +def test_prose_mentions_of_sc_are_not_tags(tmp_path): + _write( + tmp_path / "mod.py", + '"""An @sc citation points to a decision.\n\n' + "A paragraph that explains the @sc syntax.\n\n" + "@sc [decision:choice]\n\n\"\"\"\n", + ) + tags, errors = scan_tags(tmp_path) + assert errors == [] + assert len(tags) == 1 + assert tags[0].decisions == ("choice",) + + +def test_python_declaration_statement_and_snakemake_tags(tmp_path): + _write( + tmp_path / "src" / "mod.py", + '"""Module."""\n\n' + "# @sc [decision:choice]\nWIDTH = 51\n\n" + "def fit():\n" + ' """Fit.\n\n' + " @sc [decision:choice,label:coupling] fit-coupling\n" + " The local fit constraint is preserved.\n" + ' """\n' + " SCALE = 1.0\n", + ) + _write( + tmp_path / "workflow" / "rules.smk", + "# @sc [decision:choice]\nrule detect:\n output: 'catalogue'\n\n", + ) + tags, errors = scan_tags(tmp_path) + + assert errors == [] + assert {(tag.site.kind, tag.site.symbol) for tag in tags} == { + ("python_statement", "WIDTH"), + ("python_declaration", "fit"), + ("snakemake", ""), + } + contract = next(tag for tag in tags if tag.ident) + assert contract.ident == "fit-coupling" + assert contract.prose == "The local fit constraint is preserved." + + +def test_tag_grammar_rejects_bad_metadata_missing_sites_and_duplicate_contract_ids( + tmp_path, +): + _write( + tmp_path / "a.sex", + "# @sc [decision choice] malformed\nKEY 1\n\n" + "# @sc [decision:missing,label:x] same-id\n# Local prose.\nKEY 2\n\n" + "# @sc [decision:choice,label:x] same-id\n# Local prose.\nKEY 3\n\n" + "# @sc [decision:choice]", + ) + tags, errors = scan_tags(tmp_path) + + assert len(tags) == 2 + assert any("malformed @sc metadata" in error for error in errors) + assert any("duplicate local-contract id same-id" in error for error in errors) + assert any("same-id" in error for error in errors) + assert any("@sc tag governs no site" in error for error in errors) + + +def test_decision_citations_are_checked_in_both_directions(tmp_path): + record = { + "decisions": {"top": {}, "orphan": {}}, + "analyses": {"stage": {"decisions": {"inner": {}}}}, + } + _config_tag(tmp_path / "a.sex", "top") + _config_tag(tmp_path / "b.sex", "stage.inner") + _config_tag(tmp_path / "c.sex", "not-real") + tags, parse_errors = scan_tags(tmp_path) + + assert parse_errors == [] + assert decision_ids(record) == {"top", "orphan", "stage.inner"} + errors = tag_errors(tags, record) + assert any("unknown decision 'not-real'" in error for error in errors) + assert any("orphan: decision has no tagged site" in error for error in errors) + + +def test_repeated_decision_metadata_cites_multiple_real_decisions(tmp_path): + _write( + tmp_path / "shared.sex", + "# @sc [decision:first,decision:second]\nTHRESH 1\n", + ) + record = {"decisions": {"first": {}, "second": {}}} + tags, errors = scan_tags(tmp_path) + + assert errors == [] + assert tags[0].decisions == ("first", "second") + assert tag_errors(tags, record) == [] + + +def test_value_change_is_reported_with_decision_ref_expected_and_actual(tmp_path): + _config_tag(tmp_path / "detect.sex", content="THRESH 1.5\n") + record = _record("Detection. Values: THRESH = 1.") + tags, errors = scan_tags(tmp_path) + + assert errors == [] + problems = value_errors(tmp_path, record, tags) + assert len(problems) == 1 + for detail in ("choice", "THRESH", "expected", "1", "actual", "1.5"): + assert detail in problems[0] + + +def test_key_movement_inside_tagged_paragraph_preserves_value(tmp_path): + config = _config_tag( + tmp_path / "detect.sex", + content="# explanatory comment\nOTHER 2\nTHRESH 1\n", + ) + record = _record("Detection. Values: THRESH = 1.") + tags, errors = scan_tags(tmp_path) + + assert errors == [] + assert value_errors(tmp_path, record, tags) == [] + config.write_text( + "# @sc [decision:choice]\nTHRESH 1\nOTHER 2\n", encoding="utf-8" + ) + tags, errors = scan_tags(tmp_path) + assert errors == [] + assert value_errors(tmp_path, record, tags) == [] + + +def test_removed_tag_orphans_decision(tmp_path): + config = _config_tag(tmp_path / "detect.sex") + record = _record() + tags, _ = scan_tags(tmp_path) + assert tag_errors(tags, record) == [] + + config.write_text("THRESH 1\n", encoding="utf-8") + tags, _ = scan_tags(tmp_path) + assert tag_errors(tags, record) == ["choice: decision has no tagged site"] + + +def test_equal_values_at_multiple_sites_still_require_qualified_refs(tmp_path): + _config_tag(tmp_path / "default.param", content="VIGNET(51,51)\n") + _config_tag(tmp_path / "default_noimaflags.param", content="VIGNET(51,51)\n") + record = _record("Stamps. Values: VIGNET = 51.") + tags, errors = scan_tags(tmp_path) + + assert errors == [] + problem, = value_errors(tmp_path, record, tags) + assert "exactly one tagged site" in problem + assert "2" in problem + + +def test_deleting_one_of_two_vignet_tagged_sites_fails_its_assertion(tmp_path): + one = _config_tag(tmp_path / "default.param", content="VIGNET(51,51)\n") + two = _config_tag(tmp_path / "default_noimaflags.param", content="VIGNET(51,51)\n") + record = _record( + "Stamps. Values: default.param#VIGNET = 51; " + "default_noimaflags.param#VIGNET = 51." + ) + tags, errors = scan_tags(tmp_path) + assert errors == [] + assert value_errors(tmp_path, record, tags) == [] + + two.unlink() + tags, errors = scan_tags(tmp_path) + assert not errors + problems = value_errors(tmp_path, record, tags) + assert len(problems) == 1 + assert "default_noimaflags.param#VIGNET" in problems[0] + assert "found 0" in problems[0] + assert one.exists() + + +def test_absent_key_needs_only_a_same_decision_tag_in_the_file(tmp_path): + config = _write( + tmp_path / "settings.sex", + "# @sc [decision:choice]\nOTHER 2\n\nUNRELATED 3\n", + ) + record = _record("No fixed threshold. Values: settings.sex#THRESH = absent.") + tags, errors = scan_tags(tmp_path) + + assert errors == [] + assert value_errors(tmp_path, record, tags) == [] + + config.write_text( + "# @sc [decision:choice]\nOTHER 2\n\nTHRESH 1\n", + encoding="utf-8", + ) + tags, errors = scan_tags(tmp_path) + assert errors == [] + assert "expected no active setting" in value_errors(tmp_path, record, tags)[0] + + +def test_absent_key_fails_without_a_same_decision_tag_in_the_file(tmp_path): + _write(tmp_path / "settings.sex", "# @sc [decision:other]\nOTHER 2\n") + record = _record("No fixed threshold. Values: settings.sex#THRESH = absent.") + tags, errors = scan_tags(tmp_path) + + assert errors == [] + problem, = value_errors(tmp_path, record, tags) + assert "same-decision tag" in problem + assert "found 0" in problem + + +def test_ini_absence_sees_a_key_inherited_from_default(tmp_path): + _write( + tmp_path / "settings.ini", + "[DEFAULT]\nTHRESH = 1\n" + "# @sc [decision:choice]\n[SCIENCE]\nOTHER = 2\n", + ) + record = _record("No fixed threshold. Values: SCIENCE.THRESH = absent.") + tags, errors = scan_tags(tmp_path) + + assert errors == [] + problem, = value_errors(tmp_path, record, tags) + assert "expected no active setting" in problem + assert "DEFAULT" in problem + + +def test_duplicate_active_setting_outside_governed_paragraph_fails(tmp_path): + _write( + tmp_path / "settings.sex", + "# @sc [decision:choice]\nTHRESH 1\n\nTHRESH 1\n", + ) + record = _record("Threshold. Values: settings.sex#THRESH = 1.") + tags, errors = scan_tags(tmp_path) + + assert errors == [] + problem, = value_errors(tmp_path, record, tags) + assert "duplicate active setting outside the governed paragraph" in problem + + +def test_ini_duplicate_active_setting_outside_governed_paragraph_fails(tmp_path): + _write( + tmp_path / "settings.ini", + "[SCIENCE]\n# @sc [decision:choice]\nTHRESH = 1\n\n" + "OTHER = 2\nTHRESH = 1\n", + ) + record = _record("Threshold. Values: SCIENCE.THRESH = 1.") + tags, errors = scan_tags(tmp_path) + + assert errors == [] + problem, = value_errors(tmp_path, record, tags) + assert "duplicate active setting outside the governed paragraph" in problem + + +def test_ini_value_key_matching_is_case_insensitive(tmp_path): + _write( + tmp_path / "settings.ini", + "[SCIENCE]\n# @sc [decision:choice]\nmask_paths = stars.hsp\n", + ) + record = _record("Mask map. Values: SCIENCE.MASK_PATHS = stars.hsp.") + tags, errors = scan_tags(tmp_path) + + assert errors == [] + assert value_errors(tmp_path, record, tags) == [] + + +def test_ini_absence_detects_lowercase_pipeline_key(tmp_path): + _write( + tmp_path / "settings.ini", + "# @sc [decision:choice]\n[SCIENCE]\nmask_paths = stars.hsp\n", + ) + record = _record("No map. Values: SCIENCE.MASK_PATHS = absent.") + tags, errors = scan_tags(tmp_path) + + assert errors == [] + problem, = value_errors(tmp_path, record, tags) + assert "choice" in problem + assert "MASK_PATHS" in problem + assert "active setting" in problem + + +def test_ini_duplicate_matching_is_case_insensitive(tmp_path): + _write( + tmp_path / "settings.ini", + "[SCIENCE]\n# @sc [decision:choice]\nMASK_PATHS = stars.hsp\n" + "OTHER = 2\n\nmask_paths = other.hsp\n", + ) + record = _record("Mask map. Values: SCIENCE.MASK_PATHS = stars.hsp.") + tags, errors = scan_tags(tmp_path) + + assert errors == [] + problem, = value_errors(tmp_path, record, tags) + assert "choice" in problem + assert "duplicate active setting outside the governed paragraph" in problem + + +def test_absent_assertion_fails_when_its_scope_is_removed(tmp_path): + config = _write( + tmp_path / "settings.ini", + "# @sc [decision:choice]\n[SCIENCE]\nOTHER = 2\n\n", + ) + record = _record("No fixed threshold. Values: SCIENCE.THRESH = absent.") + tags, errors = scan_tags(tmp_path) + assert errors == [] + assert value_errors(tmp_path, record, tags) == [] + + # The tag no longer governs the named section; it cannot assert absence. + config.write_text( + "# @sc [decision:choice]\n[OTHER]\nKEY = 2\n", encoding="utf-8" + ) + tags, errors = scan_tags(tmp_path) + assert errors == [] + problem, = value_errors(tmp_path, record, tags) + assert "SCIENCE.THRESH" in problem + assert "found 0" in problem + + +def test_numeric_lists_and_astromatic_boolean_words_keep_reader_semantics(tmp_path): + _config_tag( + tmp_path / "values.sex", + content="THRESH 5e-4\nAPERTURE 2.5, 3.5\nFLAG Y\n", + ) + record = _record( + "Values. Values: THRESH = 0.0005; APERTURE = [2.50, 3.500]; FLAG = True." + ) + tags, errors = scan_tags(tmp_path) + + assert errors == [] + assert value_errors(tmp_path, record, tags) == [] + + +@pytest.mark.parametrize( + "actual, expected", + [ + ("1.000001", "1"), + ("1", "True"), + ("0", "False"), + ("51,51", "51"), + ("51,52", "51,51"), + ("1,2", "2,1"), + ("1,1,1", "1,1"), + ("1", "[1]"), + ("map_weight", "MAP_WEIGHT"), + ], +) +def test_normalization_does_not_hide_drift(tmp_path, actual, expected): + _config_tag(tmp_path / "config.sex", content=f"KEY {actual}\n") + record = _record(f"Value. Values: KEY = {expected}.") + tags, errors = scan_tags(tmp_path) + + assert errors == [] + problem, = value_errors(tmp_path, record, tags) + assert "expected" in problem and "actual" in problem + + +@pytest.mark.parametrize( + "contents, selector", + [ + ("X = get_value()", "X"), + ("X = 1 / 3", "X"), + ("X = 1\nX = 2", "X"), + ("X = 1\nX += 1", "X"), + ("X, Y = 1, 2", "X"), + ("def X():\n return 1", "X"), + ("X = {'a': 1, 'a': 2}", "X[a]"), + ("X = dict(**other)", "X[a]"), + ("X = {'a': 1, **other}", "X[a]"), + ("X = {variable: 1}", "X[a]"), + ("X = {'a': 1}\nX['a'] = 2", "X[a]"), + ("X = {'a': 1}\nX['b']['c'] = 2", "X[a]"), + ("X = {'a': 1}\nX['a'] += 1", "X[a]"), + ("X = {'a': 1}\ndel X['a']", "X[a]"), + ("X = 1\nX.attr = 2", "X"), + ("def f():\n X = {'a': 1}\n X['a'] = 2", "f.X[a]"), + ], +) +def test_python_nonliteral_or_ambiguous_values_fail_closed( + tmp_path, contents, selector +): + _write( + tmp_path / "constants.py", f"# @sc [decision:choice]\n{contents}\n" + ) + record = _record(f"Static value. Values: {selector} = 1.") + tags, errors = scan_tags(tmp_path) + + assert errors == [] + assert value_errors(tmp_path, record, tags) + + +def test_ini_multiline_value_keeps_continuation_lines(tmp_path): + _write( + tmp_path / "config.ini", + "# @sc [decision:choice]\n[SCIENCE]\nMODULE = alpha, beta,\n" + " gamma, delta\n", + ) + record = _record("Runner chain. Values: SCIENCE.MODULE = alpha,beta,gamma,delta.") + tags, errors = scan_tags(tmp_path) + + assert errors == [] + assert value_errors(tmp_path, record, tags) == [] + + +def test_config_boolean_semantics_and_setools_predicates(tmp_path): + ini = _write(tmp_path / "config.ini", "# @sc [decision:choice]\n[SCIENCE]\nENABLED = 1\n") + record = _record("Toggle. Values: SCIENCE.ENABLED = True.") + tags, errors = scan_tags(tmp_path) + assert errors == [] + assert value_errors(tmp_path, record, tags) == [] + + ini.write_text("# @sc [decision:choice]\n[SCIENCE]\nENABLED = Y\n", encoding="utf-8") + tags, _ = scan_tags(tmp_path) + assert value_errors(tmp_path, record, tags) + ini.write_text("# @sc [decision:choice]\n[SCIENCE]\nENABLED = False\n", encoding="utf-8") + tags, _ = scan_tags(tmp_path) + assert value_errors(tmp_path, record, tags) + + setools = _write( + tmp_path / "stars.setools", + "# @sc [decision:choice]\n[MASK:stars]\nMAG_AUTO > 18.\nMAG_AUTO < 22.\n", + ) + cuts = _record('Star cut. Values: stars.setools#MASK:stars.MAG_AUTO = ["> 18.", "< 22."].') + tags, errors = scan_tags(tmp_path) + assert errors == [] + assert value_errors(tmp_path, cuts, tags) == [] + setools.write_text( + "# @sc [decision:choice]\n[MASK:stars]\nMAG_AUTO >= 18.\nMAG_AUTO < 22.\n", + encoding="utf-8", + ) + tags, _ = scan_tags(tmp_path) + assert value_errors(tmp_path, cuts, tags) + + +def test_config_ref_ignores_unrelated_python_tagged_site(tmp_path): + _write( + tmp_path / "stars.setools", + '# @sc [decision:choice]\n[MASK:stars]\nFLAGS == 0\n', + ) + _write( + tmp_path / "other.py", + "# @sc [decision:choice]\nOTHER = 1\n", + ) + record = _record('Star flags. Values: MASK:stars.FLAGS = "== 0".') + tags, errors = scan_tags(tmp_path) + + assert errors == [] + assert value_errors(tmp_path, record, tags) == [] + + +def test_relative_python_reassignment_reports_ambiguous_binding(tmp_path): + _write( + tmp_path / "constants.py", + 'def fit():\n """Fit.\n\n @sc [decision:choice]\n """\n' + " WIDTH = 51\n WIDTH = 53\n", + ) + record = _record("Stamp. Values: WIDTH = 51.") + tags, errors = scan_tags(tmp_path) + assert errors == [] + problem, = value_errors(tmp_path, record, tags) + assert "choice" in problem + assert "ambiguous binding" in problem + assert "found 2" in problem + assert "found 0" not in problem + + +def test_reassigned_module_binding_reports_ambiguity_not_zero_sites(tmp_path): + _write( + tmp_path / "constants.py", + "# @sc [decision:choice]\nWIDTH = 51\nWIDTH = 53\n", + ) + record = _record("Stamp. Values: WIDTH = 51.") + tags, errors = scan_tags(tmp_path) + + assert errors == [] + problem, = value_errors(tmp_path, record, tags) + assert "ambiguous binding" in problem + assert "found 2" in problem + assert "found 0" not in problem + + +def test_python_qualified_selector_ignores_other_tagged_scopes(tmp_path): + _write( + tmp_path / "constants.py", + "# @sc [decision:choice]\nOTHER = 5\n\n" + 'def fit():\n """Fit.\n\n @sc [decision:choice]\n """\n' + " PARAMS = {'limits': {'T': 1}}\n", + ) + record = _record("Prior. Values: fit.PARAMS[limits.T] = 1.") + tags, errors = scan_tags(tmp_path) + + assert errors == [] + assert value_errors(tmp_path, record, tags) == [] + + +def test_python_literal_and_dict_selector_is_static(tmp_path): + _write( + tmp_path / "constants.py", + "raise RuntimeError('must not execute')\n" + "# @sc [decision:choice]\nCOMPLETENESS = {'run': {'expect': 40, 'warn': True}}\n", + ) + record = _record("Counts. Values: COMPLETENESS[run.expect] = 40.") + tags, errors = scan_tags(tmp_path) + assert errors == [] + assert value_errors(tmp_path, record, tags) == [] + + +def test_decision_markers_keep_unknown_id_check(tmp_path): + _write( + tmp_path / "tests" / "test_markers.py", + "import pytest\npytestmark = [pytest.mark.decision('choice')]\n" + "@pytest.mark.decision('missing')\ndef test_it():\n pass\n", + ) + markers, errors = decision_markers(tmp_path) + + assert errors == [] + assert [(marker.decision, marker.line) for marker in markers] == [ + ("choice", 2), ("missing", 3) + ] + assert decision_marker_errors(markers, _record()) == [ + "tests/test_markers.py:3: decision marker cites unknown decision 'missing'" + ] + + +def test_cli_reports_decision_and_local_contract_for_a_location(tmp_path, capsys): + _write(tmp_path / "astra.yaml", 'decisions:\n choice:\n label: Choice\n rationale: "First sentence. Values: THRESH = 1."\n') + _write( + tmp_path / "detect.sex", + "# @sc [decision:choice,label:coupling] threshold-coupling\n" + "# The threshold must remain coupled.\nTHRESH 1\n", + ) + + assert main(["detect.sex:3", "--root", str(tmp_path)]) == 0 + output = capsys.readouterr().out + assert "choice — Choice" in output + assert "First sentence." in output + assert "Values: THRESH = 1." in output + assert "threshold-coupling" in output + assert "The threshold must remain coupled." in output + + +def test_repository_decisions_have_sites_and_all_values_match(): + errors = repository_errors(REPO_ROOT) + assert not errors, errors + + +@pytest.mark.parametrize( + "path, pattern, replacement, decision, ref_fragment", + [ + ( + "workflow/config/cfis/config_tile_Sx.ini", + "WEIGHT_IMAGE = True", + "WEIGHT_IMAGE = False", + "detection.weight_map_usage", + "WEIGHT_IMAGE", + ), + ( + "workflow/config/cfis/config_tile_Sx.ini", + "MAKE_POST_PROCESS = True", + "MAKE_POST_PROCESS = False", + "detection.epoch_membership_ccd_bounds", + "MAKE_POST_PROCESS", + ), + ( + "workflow/config/cfis/star_selection.setools", + "[MASK:star_selection]\n", + "[MASK:star_selection]\nMASK_EXT == 0\n", + "masking.psf_star_mask_veto", + "MASK_EXT", + ), + ( + "workflow/config/cfis/default_tile.sex", + "SATUR_KEY", + "SATUR_LEVEL 50000\nSATUR_KEY", + "detection.saturation_level", + "SATUR_LEVEL", + ), + ( + "src/shapepipe/modules/ngmix_package/ngmix.py", + "\n boot = ngmix.metacal.", + "\n metacal_pars['step'] = 0.02\n boot = ngmix.metacal.", + "shape_measurement.metacal_scheme", + "metacal_pars[step]", + ), + ], +) +def test_real_record_catches_gate_and_absence_drift( + tmp_path, path, pattern, replacement, decision, ref_fragment +): + """Real-record mutations of gates and absences must break Values checks.""" + + record = load_yaml(REPO_ROOT / "astra.yaml") + tags, parse_errors = scan_tags(REPO_ROOT) + assert parse_errors == [] + for relative in {tag.path for tag in tags}: + source = REPO_ROOT / relative + if source.is_file(): + _write(tmp_path / relative, source.read_text(encoding="utf-8")) + assert value_errors(tmp_path, record) == [] + + target = tmp_path / path + text = target.read_text(encoding="utf-8") + assert text.count(pattern) == 1 + target.write_text(text.replace(pattern, replacement), encoding="utf-8") + problems = value_errors(tmp_path, record) + assert any( + decision in problem and ref_fragment in problem + for problem in problems + ), problems + + +def test_preserved_utilities_import_rule(tmp_path): + contracts = _write( + tmp_path / "pkg" / "utilities" / "CONTRACTS", + "@cc no-up-imports\nforbid: pkg.utilities.* -> pkg.modules.*\n", + ) + _write(tmp_path / "pkg" / "utilities" / "bad.py", "from ..modules import runner\n") + rules = forbid_rules(contracts) + assert rules == [("no-up-imports", "pkg.utilities.*", "pkg.modules.*")] + assert len(import_violations(tmp_path, rules)) == 2 + + +def test_decision_ids_cover_nested_analysis_paths(): + assert decision_ids( + {"decisions": {"outer": {}}, "analyses": {"child": {"decisions": {"inner": {}}}}} + ) == {"outer", "child.inner"} diff --git a/universes/committed.yaml b/universes/committed.yaml new file mode 100644 index 000000000..63b6e795b --- /dev/null +++ b/universes/committed.yaml @@ -0,0 +1,62 @@ +id: committed +description: The committed configuration on develop. +decisions: + per_unit_completeness: exact_counts + postage_stamp_size: px_51 + photometric_zeropoint: fixed_30_tiles_header_exposures +analyses: + masking: + decisions: + pixel_mask_source: instrument_flags_only + psf_star_mask_veto: instrument_flags_only + sky_mask_application: deferred_downstream + detection: + decisions: + detection_threshold_policy: megapipe_tiles + deblending_policy: megapipe_tiles + background_model: auto_megapipe_tiles + weight_map_usage: map_weight + zero_weight_interpolation: interp_all + spurious_detection_cleaning: clean_1 + blend_photometry_mask_type: correct + saturation_level: header_saturate + photometry_parameters: kron_25_35 + detection_source_mode: sx_nomask_single_image + epoch_membership_ccd_bounds: trimmed_bounds_33_2080 + preparation: + decisions: + astrometric_solution_source: delivered_headers + ccd_split_extent: all_40_hdus + epoch_provenance_from_tile_history: history_parse + object_position_columns: xwin_windowed + stamp_positioning_and_padding: round_and_zero_pad + star_selection_psf: + decisions: + star_selection_box: mode_centred_box + psf_train_validation_split: split_80_20_seeded + psfex_candidate_vetting: builtin_defaults + psf_modelling_software: psfex + psf_model_complexity: pixel_basis_deg2_per_ccd + psf_acceptance_thresholds: stars22_chi2_2 + shape_measurement: + decisions: + ngmix_seed_mode: position_seed + galaxy_model: gauss + fit_initialisation: prior_guess_t025_ntry5_2 + fit_priors: gpriorba04_flat + metacal_scheme: five_types_step001_fitgauss + centroid_source: wcs + epoch_flux_rescaling: fscale + psf_epoch_averaging: galaxy_weight_sum + galaxy_pixel_weights: rms_vignet_weights + psf_likelihood_noise: psf_noise_1em5 + megacam_ccd_flip: megapipe_flip + defect_fill: noise + blend_handling: none + epoch_masked_fraction_cut: one_third + central_defect_veto: disabled + catalogue_assembly: + decisions: + star_galaxy_classification: deferred_downstream + tile_overlap_handling: no_dedup_in_pipeline + failure_sentinels: sentinel_values diff --git a/workflow/config.yaml b/workflow/config.yaml index 76a7ee3d9..a220a4706 100644 --- a/workflow/config.yaml +++ b/workflow/config.yaml @@ -40,6 +40,7 @@ machines: nibi: base_dir: /project/def-mjhudson data: + # @sc [decision:star_selection_psf.psf_modelling_software] psf_model: psfex tile_list: $base_dir/cdaley/sp-products/$run/tiles.txt retrieve: symlink @@ -55,6 +56,7 @@ machines: candide: base_dir: /n17data/UNIONS/WL data: + # @sc [decision:star_selection_psf.psf_modelling_software] psf_model: psfex retrieve: vos container: /n17data/cdaley/containers/shapepipe_develop-runtime-20260718.sif @@ -62,6 +64,7 @@ machines: retrieve: symlink # True simulation PSF: no exposure PSF fit; fake_interp_runner # reads psf_dict below. + # @sc [decision:star_selection_psf.psf_modelling_software] psf_model: fake inputs: # $run is the branch dir (e.g. 1p2z_grid_1), so a run config diff --git a/workflow/config/cfis/config_MCCD.ini b/workflow/config/cfis/config_MCCD.ini index bf5e6852e..a195d5241 100644 --- a/workflow/config/cfis/config_MCCD.ini +++ b/workflow/config/cfis/config_MCCD.ini @@ -6,23 +6,31 @@ PREPROCESSED_OUTPUT_DIR = ./output OUTPUT_DIR = ./output INPUT_REGEX_FILE_PATTERN = star_split_ratio_80-*-*.fits INPUT_SEPARATOR = - +# @sc [decision:star_selection_psf.psf_modelling_software] MIN_N_STARS = 20 + OUTLIER_STD_MAX = 100. USE_SNR_WEIGHTS = False [INSTANCE] +# @sc [decision:star_selection_psf.psf_modelling_software] N_COMP_LOC = 8 D_COMP_GLOB = 8 + KSIG_LOC = 0.00 KSIG_GLOB = 0.00 FILTER_PATH = None D_HYB_LOC = 2 MIN_D_COMP_GLOB = None +# @sc [decision:star_selection_psf.psf_modelling_software] RMSE_THRESH = 1.25 + CCD_STAR_THRESH = 0.15 +# @sc [decision:star_selection_psf.psf_modelling_software] FP_GEOMETRY = CFIS [FIT] +# @sc [decision:star_selection_psf.psf_modelling_software] LOC_MODEL = hybrid PSF_SIZE = 6.2 PSF_SIZE_TYPE = R2 diff --git a/workflow/config/cfis/config_exp_Sp.ini b/workflow/config/cfis/config_exp_Sp.ini index dd27d6ccd..0b24a9562 100644 --- a/workflow/config/cfis/config_exp_Sp.ini +++ b/workflow/config/cfis/config_exp_Sp.ini @@ -75,4 +75,5 @@ NUMBERING_SCHEME = -0000000 OUTPUT_SUFFIX = image, weight, flag # Number of HDUs/CCDs of mosaic +# @sc [decision:preparation.ccd_split_extent] N_HDU = 40 diff --git a/workflow/config/cfis/config_exp_mccd.ini b/workflow/config/cfis/config_exp_mccd.ini index 5e40a5afa..3e1edfcdc 100644 --- a/workflow/config/cfis/config_exp_mccd.ini +++ b/workflow/config/cfis/config_exp_mccd.ini @@ -22,6 +22,7 @@ RUN_DATETIME = False [EXECUTION] # Module name, single string or comma-separated list of valid module runner names +# @sc [decision:star_selection_psf.psf_modelling_software] MODULE = sextractor_runner, mask_query_runner, setools_runner, mccd_preprocessing_runner, mccd_fit_val_runner, merge_starcat_runner, mccd_plots_runner @@ -61,9 +62,11 @@ TIMEOUT = 96:00:00 [SEXTRACTOR_RUNNER] # The split CCDs, and nothing else: ShapePipe generates no masks +# @sc [decision:masking.pixel_mask_source,decision:star_selection_psf.psf_modelling_software] INPUT_DIR = $SP_RUN/output/run_sp_exp_Sp/split_exp_runner/output # Read the instrument flag image split_exp wrote per CCD +# @sc [decision:masking.pixel_mask_source,decision:star_selection_psf.psf_modelling_software] FILE_PATTERN = image, weight, flag # Explicit extensions: a 3-entry FILE_PATTERN override must not fall back on @@ -84,6 +87,7 @@ DOT_CONV_FILE = $SP_CONFIG/default.conv WEIGHT_IMAGE = True # Use input flag image if True +# @sc [decision:masking.pixel_mask_source,decision:star_selection_psf.psf_modelling_software] FLAG_IMAGE = True # Use input PSF file if True diff --git a/workflow/config/cfis/config_exp_psfex.ini b/workflow/config/cfis/config_exp_psfex.ini index 495d9b78a..2c573e652 100644 --- a/workflow/config/cfis/config_exp_psfex.ini +++ b/workflow/config/cfis/config_exp_psfex.ini @@ -22,6 +22,7 @@ RUN_DATETIME = False [EXECUTION] # Module name, single string or comma-separated list of valid module runner names +# @sc [decision:star_selection_psf.psf_modelling_software] MODULE = sextractor_runner, mask_query_runner, setools_runner, psfex_runner, psfex_interp_runner # Run mode, SMP or MPI @@ -59,9 +60,11 @@ TIMEOUT = 96:00:00 [SEXTRACTOR_RUNNER] # The split CCDs, and nothing else: ShapePipe generates no masks +# @sc [decision:masking.pixel_mask_source] INPUT_DIR = $SP_RUN/output/run_sp_exp_Sp/split_exp_runner/output # Read the instrument flag image split_exp wrote per CCD +# @sc [decision:masking.pixel_mask_source] FILE_PATTERN = image, weight, flag # Explicit extensions: a 3-entry FILE_PATTERN override must not fall back on @@ -76,12 +79,16 @@ EXEC_PATH = source-extractor # SExtractor configuration files DOT_SEX_FILE = $SP_CONFIG/default_exp.sex DOT_PARAM_FILE = $SP_CONFIG//default.param +# @sc [decision:detection.detection_threshold_policy,label:effective-filter] exposure-dot-conv-overrides-filter +# DOT_CONV_FILE overrides FILTER_NAME in the exposure runner; keep the effective convolution kernel aligned with the intended PSF-star detection prescription. DOT_CONV_FILE = $SP_CONFIG/default.conv # Use input weight image if True +# @sc [decision:detection.weight_map_usage] WEIGHT_IMAGE = True # Use input flag image if True +# @sc [decision:masking.pixel_mask_source] FLAG_IMAGE = True # Use input PSF file if True @@ -96,15 +103,19 @@ DETECTION_IMAGE = False DETECTION_WEIGHT = False # True if photometry zero-point is to be read from exposure image header +# @sc [decision:photometric_zeropoint] ZP_FROM_HEADER = True # If ZP_FROM_HEADER is True, zero-point key name +# @sc [decision:photometric_zeropoint] ZP_KEY = PHOTZP # Background information from image header. # If BKG_FROM_HEADER is True, background value will be read from header. # In that case, the value of BACK_TYPE will be set atomatically to MANUAL. # This is used e.g. for the LSB images. +# @sc [decision:detection.background_model,label:override] header-background-replaces-auto +# BKG_FROM_HEADER replaces the AUTO background with a manual header value; enabling it changes the estimator rather than only its source. BKG_FROM_HEADER = False # LSB images: # BKG_FROM_HEADER = True @@ -127,6 +138,7 @@ SUFFIX = sexcat MAKE_POST_PROCESS = FALSE +# @sc [decision:masking.psf_star_mask_veto] [MASK_QUERY_RUNNER] INPUT_DIR = $SP_RUN/output/run_sp_exp_SxSePsf/sextractor_runner/output @@ -178,6 +190,7 @@ SETOOLS_CONFIG_PATH = $SP_CONFIG/star_selection.setools [PSFEX_RUNNER] # Use 80% sample for PSF model +# @sc [decision:star_selection_psf.psf_train_validation_split] FILE_PATTERN = star_split_ratio_80 NUMBERING_SCHEME = -0000000-0 @@ -191,6 +204,7 @@ DOT_PSFEX_FILE = $SP_CONFIG/default.psfex [PSFEX_INTERP_RUNNER] # Use 20% sample for PSF validation +# @sc [decision:star_selection_psf.psf_train_validation_split] FILE_PATTERN = star_split_ratio_80, star_split_ratio_20, psfex_cat FILE_EXT = .psf, .fits, .cat @@ -204,13 +218,16 @@ NUMBERING_SCHEME = -0000000-0 MODE = VALIDATION # Column names of position parameters +# @sc [decision:preparation.object_position_columns] POSITION_PARAMS = XWIN_IMAGE,YWIN_IMAGE # If True, measure and store ellipticity of the PSF (using moments) GET_SHAPES = True # Minimum number of stars per CCD for PSF model to be computed +# @sc [decision:star_selection_psf.psf_acceptance_thresholds] STAR_THRESH = 22 # Maximum chi^2 for PSF model to be computed on CCD +# @sc [decision:star_selection_psf.psf_acceptance_thresholds] CHI2_THRESH = 2 diff --git a/workflow/config/cfis/config_tile_Fe.ini b/workflow/config/cfis/config_tile_Fe.ini index 5fa45bead..e66f407ea 100644 --- a/workflow/config/cfis/config_tile_Fe.ini +++ b/workflow/config/cfis/config_tile_Fe.ini @@ -69,10 +69,12 @@ FILE_EXT = .fits NUMBERING_SCHEME = -000-000 # Column number of exposure name in FITS header +# @sc [decision:preparation.epoch_provenance_from_tile_history] COLNUM = 3 # Prefix to remove from exposure name. CFIS exposure names carry no # prefix -- the trailing "p" is a suffix (epoch letter), kept as part of # the exposure name, not stripped by this key. +# @sc [decision:preparation.epoch_provenance_from_tile_history] EXP_PREFIX = diff --git a/workflow/config/cfis/config_tile_Mc.ini b/workflow/config/cfis/config_tile_Mc.ini index 5920d0a91..9475a819c 100644 --- a/workflow/config/cfis/config_tile_Mc.ini +++ b/workflow/config/cfis/config_tile_Mc.ini @@ -79,6 +79,15 @@ FILE_EXT = .fits, .sqlite, .fits # sections below NUMBERING_SCHEME = -000-000 +# @sc [decision:catalogue_assembly.star_galaxy_classification] SM_DO_CLASSIFICATION = False SHAPE_MEASUREMENT_TYPE = ngmix + +# Save per-epoch PSF shapes and epoch identity (HSM_*_PSF_n, EXP_ID_n, +# CCD_n). The campaign merge needs one column schema across all tiles, so +# the slot count is fixed rather than per-tile max(N_EPOCH)+1. It must +# exceed the survey's maximum epochs per object with margin (smk-g7: max 6 +# over 8 inspected tiles); an object with more epochs than slots is an error. +SAVE_PSF_DATA = True +N_EPOCH_SLOTS = 12 diff --git a/workflow/config/cfis/config_tile_Ng_template.ini b/workflow/config/cfis/config_tile_Ng_template.ini index 04606636d..8ec39f5f1 100644 --- a/workflow/config/cfis/config_tile_Ng_template.ini +++ b/workflow/config/cfis/config_tile_Ng_template.ini @@ -101,6 +101,7 @@ NUMBERING_SCHEME = -000-000 # 1/RMS^2 inverse-variance ngmix weights. When set, the file must exist for # every tile (missing file -> error, no per-tile fallback); omit the option # entirely to fall back to the scalar sigma_mad noise estimate. +# @sc [decision:shape_measurement.galaxy_pixel_weights] BKG_RMS_VIGNET_PATH = $NGMIX_VIGNET_DIR/vignetmaker_runner_run_2/output/background_rms_vignet{file_number_string}.sqlite # Number of objects to batch save during processing, optional. Omit or set @@ -112,9 +113,11 @@ BKG_RMS_VIGNET_PATH = $NGMIX_VIGNET_DIR/vignetmaker_runner_run_2/output/backgrou SAVE_BATCH = 250 # Magnitude zero-point +# @sc [decision:photometric_zeropoint] MAG_ZP = 30.0 # Pixel scale in arcsec +# @sc [decision:shape_measurement.fit_priors] PIXEL_SCALE = 0.186 # ID_OBJ_MIN/MAX: this chunk's closed SExtractor NUMBER-column range, diff --git a/workflow/config/cfis/config_tile_PiViVi_mccd.ini b/workflow/config/cfis/config_tile_PiViVi_mccd.ini index d312ca494..864f790f9 100644 --- a/workflow/config/cfis/config_tile_PiViVi_mccd.ini +++ b/workflow/config/cfis/config_tile_PiViVi_mccd.ini @@ -21,6 +21,7 @@ RUN_DATETIME = False # Module name, single string or comma-separated list of valid module runner names #MODULE = mccd_interp_runner, +# @sc [decision:star_selection_psf.psf_modelling_software] MODULE = ${SP_PSF}_interp_runner, vignetmaker_runner, vignetmaker_runner # Parallel processing mode, SMP or MPI diff --git a/workflow/config/cfis/config_tile_PiViVi_psfex.ini b/workflow/config/cfis/config_tile_PiViVi_psfex.ini index 27a759528..62073d482 100644 --- a/workflow/config/cfis/config_tile_PiViVi_psfex.ini +++ b/workflow/config/cfis/config_tile_PiViVi_psfex.ini @@ -21,6 +21,7 @@ RUN_DATETIME = False # Module name, single string or comma-separated list of valid module runner names #MODULE = psfex_interp_runner, +# @sc [decision:star_selection_psf.psf_modelling_software] MODULE = ${SP_PSF}_interp_runner, vignetmaker_runner, vignetmaker_runner # Parallel processing mode, SMP or MPI @@ -93,15 +94,18 @@ NUMBERING_SCHEME = -000-000 MODE = MULTI-EPOCH # Column names of position parameters +# @sc [decision:preparation.object_position_columns] POSITION_PARAMS = XWIN_WORLD,YWIN_WORLD # If True, measure and store ellipticity of the PSF GET_SHAPES = True # Number of stars threshold +# @sc [decision:star_selection_psf.psf_acceptance_thresholds] STAR_THRESH = 22 # chi^2 threshold +# @sc [decision:star_selection_psf.psf_acceptance_thresholds] CHI2_THRESH = 2 # Multi-epoch mode parameters @@ -112,6 +116,7 @@ CHI2_THRESH = 2 ME_DOT_PSF_EXP_DIR = $SP_EXP # Input psf file pattern +# @sc [decision:star_selection_psf.psf_train_validation_split] ME_DOT_PSF_PATTERN = star_split_ratio_80 @@ -127,7 +132,9 @@ FILE_EXT = .fits, .fits # NUMBERING_SCHEME (optional) string with numbering pattern for input files NUMBERING_SCHEME = -000-000 +# @sc [decision:shape_measurement.defect_fill] MASKING = False + MASK_VALUE = 0 # Run mode for psfex interpolation: @@ -137,10 +144,12 @@ MASK_VALUE = 0 MODE = CLASSIC # Coordinate frame type, one in PIX (pixel frame), SPHE (spherical coordinates) +# @sc [decision:preparation.object_position_columns] COORD = PIX POSITION_PARAMS = XWIN_IMAGE,YWIN_IMAGE # Vignet size in pixels +# @sc [decision:postage_stamp_size] STAMP_SIZE = 51 # Output file name prefix, file name is _vignet.fits @@ -161,7 +170,9 @@ FILE_EXT = .fits, .sqlite, .txt # NUMBERING_SCHEME (optional) string with numbering pattern for input files NUMBERING_SCHEME = -000-000 +# @sc [decision:shape_measurement.defect_fill] MASKING = False + MASK_VALUE = 0 # Run mode for psfex interpolation: @@ -171,10 +182,12 @@ MASK_VALUE = 0 MODE = MULTI-EPOCH # Coordinate frame type, one in PIX (pixel frame), SPHE (spherical coordinates) +# @sc [decision:preparation.object_position_columns] COORD = SPHE POSITION_PARAMS = XWIN_WORLD,YWIN_WORLD # Vignet size in pixels +# @sc [decision:postage_stamp_size] STAMP_SIZE = 51 # Output file name prefix, file name is vignet.fits @@ -184,5 +197,8 @@ PREFIX = # run outputs. ME_IMAGE_EXP_DIR/ME_IMAGE_EXP_RUNNERS replace ME_IMAGE_DIR for # the v2.0 per-exposure pipeline; output dirs are discovered by scanning $SP_EXP. ME_IMAGE_EXP_DIR = $SP_EXP +# @sc [decision:masking.pixel_mask_source] ME_IMAGE_EXP_RUNNERS = split_exp_runner, split_exp_runner, split_exp_runner, sextractor_runner, sextractor_runner +# @sc [decision:masking.pixel_mask_source,label:coupling] flag-stamp-runner-order +# ME_IMAGE_PATTERN and ME_IMAGE_EXP_RUNNERS are parallel ordered lists; keep flag, image, weight, background and background-RMS entries paired. ME_IMAGE_PATTERN = flag, image, weight, background, background_rms diff --git a/workflow/config/cfis/config_tile_Sx.ini b/workflow/config/cfis/config_tile_Sx.ini index 4dccaf448..0c05c38b5 100644 --- a/workflow/config/cfis/config_tile_Sx.ini +++ b/workflow/config/cfis/config_tile_Sx.ini @@ -73,13 +73,19 @@ EXEC_PATH = source-extractor # SExtractor configuration files DOT_SEX_FILE = $SP_CONFIG/default_tile.sex +# @sc [decision:detection.detection_source_mode,label:override] tile-parameter-file-overrides-column-list +# DOT_PARAM_FILE overrides PARAMETERS_NAME; keep it pointed at default_noimaflags.param so tile detections do not request IMAFLAGS_ISO. DOT_PARAM_FILE = $SP_CONFIG/default_noimaflags.param +# @sc [decision:detection.detection_threshold_policy,label:effective-filter] tile-dot-conv-overrides-filter +# DOT_CONV_FILE overrides FILTER_NAME in the tile runner; keep the effective convolution kernel aligned with the intended detection prescription. DOT_CONV_FILE = $SP_CONFIG/gauss_3.0_7x7.conv # Use input weight image if True +# @sc [decision:detection.weight_map_usage] WEIGHT_IMAGE = True # Use input flag image if True +# @sc [decision:detection.detection_source_mode] FLAG_IMAGE = False # Use input PSF file if True @@ -87,14 +93,16 @@ PSF_FILE = False # Use distinct image for detection (SExtractor in # dual-image mode) if True +# @sc [decision:detection.detection_source_mode] DETECTION_IMAGE = False # Distinct weight image for detection (SExtractor # in dual-image mode) DETECTION_WEIGHT = False +# @sc [decision:photometric_zeropoint] ZP_FROM_HEADER = False - +# @sc [decision:detection.background_model] BKG_FROM_HEADER = False # Type of image check (optional), default not used, can be a list of @@ -109,10 +117,13 @@ SUFFIX = sexcat ## Post-processing # Necessary for tiles, to enable multi-exposure processing +# @sc [decision:detection.epoch_membership_ccd_bounds] MAKE_POST_PROCESS = True # World coordinate keywords, SExtractor output. Format: KEY_X,KEY_Y +# @sc [decision:preparation.object_position_columns] WORLD_POSITION = XWIN_WORLD,YWIN_WORLD # Number of pixels in x,y of a CCD. Format: Nx,Ny +# @sc [decision:detection.epoch_membership_ccd_bounds] CCD_SIZE = 33,2080,1,4612 diff --git a/workflow/config/cfis/default.conv b/workflow/config/cfis/default.conv index 2590b9cba..c4d9756f0 100644 --- a/workflow/config/cfis/default.conv +++ b/workflow/config/cfis/default.conv @@ -1,4 +1,5 @@ CONV NORM +# @sc [decision:detection.detection_threshold_policy,scope:file] # 3x3 ``all-ground'' convolution mask with FWHM = 2 pixels. 1 2 1 2 4 2 diff --git a/workflow/config/cfis/default.param b/workflow/config/cfis/default.param index 09ad8405e..3fe6978bc 100644 --- a/workflow/config/cfis/default.param +++ b/workflow/config/cfis/default.param @@ -59,6 +59,7 @@ FWHM_WORLD #FWHM assuming a gaussian core ELONGATION #A_IMAGE/B_IMAGE ELLIPTICITY #1 - B_IMAGE/A_IMAGE +# @sc [decision:postage_stamp_size] VIGNET(51,51) #Pixel data around detection [count] # For GaaP photometry diff --git a/workflow/config/cfis/default.psfex b/workflow/config/cfis/default.psfex index a9d1a906c..840f1ddfd 100644 --- a/workflow/config/cfis/default.psfex +++ b/workflow/config/cfis/default.psfex @@ -4,40 +4,58 @@ #-------------------------------- PSF model ---------------------------------- +# @sc [decision:star_selection_psf.psf_model_complexity] BASIS_TYPE PIXEL # NONE, PIXEL, GAUSS-LAGUERRE or FILE BASIS_NUMBER 20 # Basis number or parameter + BASIS_NAME basis.fits # Basis filename (FITS data-cube) BASIS_SCALE 1.0 # Gauss-Laguerre beta parameter NEWBASIS_TYPE NONE # Create new basis: NONE, PCA_INDEPENDENT # or PCA_COMMON NEWBASIS_NUMBER 8 # Number of new basis vectors +# @sc [decision:star_selection_psf.psf_model_complexity] PSF_SAMPLING 1. # Sampling step in pixel units (0.0 = auto) + PSF_PIXELSIZE 1.0 # Effective pixel size in pixel step units +# @sc [decision:star_selection_psf.psf_model_complexity] PSF_ACCURACY 0.01 # Accuracy to expect from PSF "pixel" values +# @sc [decision:postage_stamp_size] PSF_SIZE 51,51 # Image size of the PSF model +# @sc [decision:star_selection_psf.psfex_candidate_vetting] PSF_RECENTER N # Allow recentering of PSF-candidates Y/N ? +# @sc [decision:star_selection_psf.psf_model_complexity] MEF_TYPE INDEPENDENT # INDEPENDENT or COMMON #------------------------- Point source measurements ------------------------- +# @sc [decision:preparation.object_position_columns] CENTER_KEYS XWIN_IMAGE,YWIN_IMAGE # Catalogue parameters for source pre-centering +# @sc [decision:detection.photometry_parameters] PHOTFLUX_KEY FLUX_AUTO # Catalogue parameter for photometric norm. +# @sc [decision:detection.photometry_parameters] PHOTFLUXERR_KEY FLUXERR_AUTO # Catalogue parameter for photometric error #----------------------------- PSF variability ------------------------------- +# @sc [decision:preparation.object_position_columns,decision:star_selection_psf.psf_model_complexity] PSFVAR_KEYS XWIN_IMAGE,YWIN_IMAGE # Catalogue or FITS (preceded by :) params +# @sc [decision:star_selection_psf.psf_model_complexity] PSFVAR_GROUPS 1,1 # Group tag for each context key +# @sc [decision:star_selection_psf.psf_model_complexity] PSFVAR_DEGREES 2 # Polynom degree for each group + PSFVAR_NSNAP 9 # Number of PSF snapshots per axis HIDDENMEF_TYPE COMMON # INDEPENDENT or COMMON STABILITY_TYPE EXPOSURE # EXPOSURE or SEQUENCE #----------------------------- Sample selection ------------------------------ +# @sc [decision:star_selection_psf.star_selection_box,decision:star_selection_psf.psfex_candidate_vetting] SAMPLE_AUTOSELECT N # Automatically select the FWHM (Y/N) ? +# @sc [decision:star_selection_psf.psfex_candidate_vetting] BADPIXEL_FILTER N # Filter bad-pixels in samples (Y/N) ? + BADPIXEL_NMAX 0 # Maximum number of bad pixels allowed #----------------------- PSF homogeneisation kernel -------------------------- diff --git a/workflow/config/cfis/default_exp.sex b/workflow/config/cfis/default_exp.sex index b87275ecb..e68343b30 100644 --- a/workflow/config/cfis/default_exp.sex +++ b/workflow/config/cfis/default_exp.sex @@ -11,33 +11,46 @@ PARAMETERS_NAME default.param #------------------------------- Extraction ---------------------------------- DETECT_TYPE CCD # CCD (linear) or PHOTO (with gamma correction) +# @sc [decision:detection.detection_threshold_policy] DETECT_MINAREA 5 # min. # of pixels above threshold + DETECT_MAXAREA 0 # max. # of pixels above threshold (0=unlimited) +# @sc [decision:detection.detection_threshold_policy] THRESH_TYPE RELATIVE # threshold type: RELATIVE (in sigmas) # or ABSOLUTE (in ADUs) +# @sc [decision:detection.detection_threshold_policy] DETECT_THRESH 1.5 # or , in mag.arcsec-2 ANALYSIS_THRESH 1.5 # or , in mag.arcsec-2 +# @sc [decision:detection.detection_threshold_policy] FILTER Y # apply filter for detection (Y or N)? + FILTER_NAME default.conv FILTER_THRESH # Threshold[s] for retina filtering +# @sc [decision:detection.deblending_policy] DEBLEND_NTHRESH 32 # Number of deblending sub-thresholds DEBLEND_MINCONT 0.001 # Minimum contrast parameter for deblending +# @sc [decision:detection.spurious_detection_cleaning] CLEAN Y # Clean spurious detections? (Y or N)? CLEAN_PARAM 1.0 # Cleaning efficiency +# @sc [decision:detection.blend_photometry_mask_type] MASK_TYPE CORRECT # type of detection MASKing: can be one of # NONE, BLANK or CORRECT #-------------------------------- WEIGHTing ---------------------------------- +# @sc [decision:detection.weight_map_usage] WEIGHT_TYPE MAP_WEIGHT # type of WEIGHTing: NONE, BACKGROUND, # MAP_RMS, MAP_VAR or MAP_WEIGHT RESCALE_WEIGHTS Y # Rescale input weights/variances (Y/N)? + WEIGHT_IMAGE weight.fits # weight-map filename +# @sc [decision:detection.weight_map_usage] WEIGHT_GAIN Y # modulate gain (E/ADU) with weights? (Y/N) + WEIGHT_THRESH # weight threshold[s] for bad pixels #-------------------------------- FLAGging ----------------------------------- @@ -48,14 +61,18 @@ FLAG_TYPE OR # flag pixel combination: OR, AND, MIN, MAX #------------------------------ Photometry ----------------------------------- +# @sc [decision:detection.photometry_parameters] PHOT_APERTURES 5 # MAG_APER aperture diameter(s) in pixels PHOT_AUTOPARAMS 2.5, 3.5 # MAG_AUTO parameters: , + PHOT_PETROPARAMS 2.0, 3.5 # MAG_PETRO parameters: , # PHOT_AUTOAPERS 0.0,0.0 # , minimum apertures # for MAG_AUTO and MAG_PETRO +# @sc [decision:detection.photometry_parameters] PHOT_FLUXFRAC 0.5 # flux fraction[s] used for FLUX_RADIUS +# @sc [decision:detection.saturation_level] SATUR_KEY SATURATE # keyword for saturation level (in ADUs) MAG_ZEROPOINT 30.0 # magnitude zero-point @@ -71,12 +88,17 @@ STARNNW_NAME default.nnw #------------------------------ Background ----------------------------------- +# @sc [decision:detection.background_model] BACK_TYPE AUTO # AUTO or MANUAL + BACK_VALUE 0.0 # Default background value in MANUAL mode +# @sc [decision:detection.background_model] BACK_SIZE 64 # Background mesh: or , BACK_FILTERSIZE 3 # Background filter: or , +# @sc [decision:detection.background_model] BACKPHOTO_TYPE GLOBAL # can be GLOBAL or LOCAL + BACKPHOTO_THICK 24 # thickness of the background LOCAL annulus BACK_FILTTHRESH 0.0 # Threshold above which the background- # map filter operates @@ -120,6 +142,7 @@ WRITE_XML N # Write XML file (Y/N)? NTHREADS 1 # 1 single thread FITS_UNSIGNED N # Treat FITS integer values as unsigned (Y/N)? +# @sc [decision:detection.zero_weight_interpolation] INTERP_MAXXLAG 16 # Max. lag along X for 0-weight interpolation INTERP_MAXYLAG 16 # Max. lag along Y for 0-weight interpolation INTERP_TYPE ALL # Interpolation type: NONE, VAR_ONLY or ALL diff --git a/workflow/config/cfis/default_noimaflags.param b/workflow/config/cfis/default_noimaflags.param index b251f5b74..fff22641d 100644 --- a/workflow/config/cfis/default_noimaflags.param +++ b/workflow/config/cfis/default_noimaflags.param @@ -1,3 +1,4 @@ +# @sc [decision:detection.detection_source_mode,scope:file] NUMBER #Running object number EXT_NUMBER #FITS extension number @@ -56,6 +57,7 @@ FWHM_WORLD #FWHM assuming a gaussian core ELONGATION #A_IMAGE/B_IMAGE ELLIPTICITY #1 - B_IMAGE/A_IMAGE +# @sc [decision:postage_stamp_size] VIGNET(51,51) #Pixel data around detection [count] # For GaaP photometry diff --git a/workflow/config/cfis/default_tile.sex b/workflow/config/cfis/default_tile.sex index e706e81e4..ca7367c3e 100644 --- a/workflow/config/cfis/default_tile.sex +++ b/workflow/config/cfis/default_tile.sex @@ -11,33 +11,46 @@ PARAMETERS_NAME default.param #------------------------------- Extraction ---------------------------------- DETECT_TYPE CCD # CCD (linear) or PHOTO (with gamma correction) +# @sc [decision:detection.detection_threshold_policy] DETECT_MINAREA 3 # min. # of pixels above threshold + DETECT_MAXAREA 0 # max. # of pixels above threshold (0=unlimited) +# @sc [decision:detection.detection_threshold_policy] THRESH_TYPE RELATIVE # threshold type: RELATIVE (in sigmas) # or ABSOLUTE (in ADUs) +# @sc [decision:detection.detection_threshold_policy] DETECT_THRESH 1.0 # or , in mag.arcsec-2 ANALYSIS_THRESH 1.0 # or , in mag.arcsec-2 +# @sc [decision:detection.detection_threshold_policy] FILTER Y # apply filter for detection (Y or N)? + FILTER_NAME gauss_3.0_7x7.conv FILTER_THRESH # Threshold[s] for retina filtering +# @sc [decision:detection.deblending_policy] DEBLEND_NTHRESH 32 # Number of deblending sub-thresholds DEBLEND_MINCONT 0.002 # Minimum contrast parameter for deblending +# @sc [decision:detection.spurious_detection_cleaning] CLEAN Y # Clean spurious detections? (Y or N)? CLEAN_PARAM 1.0 # Cleaning efficiency +# @sc [decision:detection.blend_photometry_mask_type] MASK_TYPE CORRECT # type of detection MASKing: can be one of # NONE, BLANK or CORRECT #-------------------------------- WEIGHTing ---------------------------------- +# @sc [decision:detection.weight_map_usage] WEIGHT_TYPE MAP_WEIGHT # type of WEIGHTing: NONE, BACKGROUND, # MAP_RMS, MAP_VAR or MAP_WEIGHT RESCALE_WEIGHTS Y # Rescale input weights/variances (Y/N)? + WEIGHT_IMAGE weight.fits # weight-map filename +# @sc [decision:detection.weight_map_usage] WEIGHT_GAIN Y # modulate gain (E/ADU) with weights? (Y/N) + WEIGHT_THRESH # weight threshold[s] for bad pixels #-------------------------------- FLAGging ----------------------------------- @@ -48,17 +61,23 @@ FLAG_TYPE OR # flag pixel combination: OR, AND, MIN, MAX #------------------------------ Photometry ----------------------------------- +# @sc [decision:detection.photometry_parameters] PHOT_APERTURES 5 # MAG_APER aperture diameter(s) in pixels PHOT_AUTOPARAMS 2.5, 3.5 # MAG_AUTO parameters: , + PHOT_PETROPARAMS 2.0, 3.5 # MAG_PETRO parameters: , # PHOT_AUTOAPERS 0.0,0.0 # , minimum apertures # for MAG_AUTO and MAG_PETRO +# @sc [decision:detection.photometry_parameters] PHOT_FLUXFRAC 0.5 # flux fraction[s] used for FLUX_RADIUS +# @sc [decision:detection.saturation_level] SATUR_KEY SATURATE # keyword for saturation level (in ADUs) +# @sc [decision:photometric_zeropoint] MAG_ZEROPOINT 30.0 # magnitude zero-point + MAG_GAMMA 4.0 # gamma of emulsion (for photographic scans) GAIN_KEY GAIN # keyword for detector gain in e-/ADU @@ -71,13 +90,18 @@ STARNNW_NAME default.nnw #------------------------------ Background ----------------------------------- +# @sc [decision:detection.background_model] BACK_TYPE AUTO # AUTO or MANUAL + BACK_VALUE 0.0 # Default background value in MANUAL mode +# @sc [decision:detection.background_model] BACK_SIZE 512 # Background mesh: or , BACK_FILTERSIZE 9 # Background filter: or , +# @sc [decision:detection.background_model] BACKPHOTO_TYPE LOCAL # can be GLOBAL or LOCAL BACKPHOTO_THICK 30 # thickness of the background LOCAL annulus + BACK_FILTTHRESH 0.0 # Threshold above which the background- # map filter operates @@ -120,6 +144,7 @@ WRITE_XML N # Write XML file (Y/N)? NTHREADS 1 # 1 single thread FITS_UNSIGNED N # Treat FITS integer values as unsigned (Y/N)? +# @sc [decision:detection.zero_weight_interpolation] INTERP_MAXXLAG 16 # Max. lag along X for 0-weight interpolation INTERP_MAXYLAG 16 # Max. lag along Y for 0-weight interpolation INTERP_TYPE ALL # Interpolation type: NONE, VAR_ONLY or ALL diff --git a/workflow/config/cfis/final_cat.param b/workflow/config/cfis/final_cat.param index 00bcb3f73..6ebd1aae9 100644 --- a/workflow/config/cfis/final_cat.param +++ b/workflow/config/cfis/final_cat.param @@ -1,14 +1,18 @@ # coordinates +# @sc [decision:preparation.object_position_columns] XWIN_WORLD YWIN_WORLD # tile ID, for plot of tile-dependent additive bias. # Can maybe be removed. +# @sc [decision:catalogue_assembly.tile_overlap_handling] TILE_ID # flags FLAGS +# @sc [decision:detection.detection_source_mode] IMAFLAGS_ISO + NGMIX_MCAL_FLAGS # PSF ellipticity (original image PSF) diff --git a/workflow/config/cfis/gauss_3.0_7x7.conv b/workflow/config/cfis/gauss_3.0_7x7.conv index 527acbbb0..f184ebca5 100644 --- a/workflow/config/cfis/gauss_3.0_7x7.conv +++ b/workflow/config/cfis/gauss_3.0_7x7.conv @@ -1,4 +1,5 @@ CONV NORM +# @sc [decision:detection.detection_threshold_policy,scope:file] # 7x7 convolution mask of a gaussian PSF with FWHM = 3.0 pixels. 0.004963 0.021388 0.051328 0.068707 0.051328 0.021388 0.004963 0.021388 0.092163 0.221178 0.296069 0.221178 0.092163 0.021388 diff --git a/workflow/config/cfis/star_selection.setools b/workflow/config/cfis/star_selection.setools index 10ab60fd8..a237ca89a 100644 --- a/workflow/config/cfis/star_selection.setools +++ b/workflow/config/cfis/star_selection.setools @@ -28,28 +28,40 @@ ## config, which ships commented out; SETools has no bitwise operators, so the ## bit selection happens there and this file only ever tests for zero. +# @sc [decision:masking.psf_star_mask_veto,decision:star_selection_psf.star_selection_box] [MASK:preselect] + MAG_AUTO > 0 MAG_AUTO < 21 FWHM_IMAGE > 0.3 / 0.187 FWHM_IMAGE < 1.5 / 0.187 +# @sc [decision:detection.saturation_level] FLAGS == 0 + IMAFLAGS_ISO == 0 NO_SAVE +# @sc [decision:masking.psf_star_mask_veto] [MASK:flag] + +# @sc [decision:detection.saturation_level] FLAGS == 0 + IMAFLAGS_ISO == 0 NO_SAVE +# @sc [decision:masking.psf_star_mask_veto,decision:star_selection_psf.star_selection_box] [MASK:star_selection] # Star selection using the FWHM mode + MAG_AUTO > 18. MAG_AUTO < 22. FWHM_IMAGE <= mode(FWHM_IMAGE{preselect}) + 0.2 FWHM_IMAGE >= mode(FWHM_IMAGE{preselect}) - 0.2 +# @sc [decision:detection.saturation_level] FLAGS == 0 +# @sc [decision:masking.pixel_mask_source] IMAFLAGS_ISO == 0 [MASK:fwhm_mag_cut] @@ -63,7 +75,9 @@ NO_SAVE # Split the 'star_selection' sample into # two random sub-samples with ratio 80/20 [RAND_SPLIT:star_split] +# @sc [decision:star_selection_psf.psf_train_validation_split] RATIO = 20 + MASK = star_selection # The following selection is only used for plotting @@ -100,7 +114,10 @@ TYPE = scatter FORMAT = png X = X_IMAGE{star_selection} Y = Y_IMAGE{star_selection} +# @sc [decision:star_selection_psf.star_selection_box,label:diagnostic] stellar-diagnostic-matches-cut +# FWHM_FIELD plot labels and summary statistics must describe the size cut actually applied. SCATTER = FWHM_IMAGE{star_selection}*0.187 + MARKER = . LABEL = "FWHM (arcsec)" TITLE = "FWHM of stars" diff --git a/workflow/scripts/completeness.py b/workflow/scripts/completeness.py index 3b4cf3743..89cd63601 100644 --- a/workflow/scripts/completeness.py +++ b/workflow/scripts/completeness.py @@ -74,6 +74,7 @@ # stage -> {runner_subdir: {expect, [warn], [subpath]}} # exp_psf and tile_vignets are selected by $SP_PSF at check time. +# @sc [decision:per_unit_completeness] COMPLETENESS = { # --- tile prepare (phase A) --- # get_images counts are CONFIG-FLAVOR-DEPENDENT: the v2.0 bash table said 4/6 @@ -389,6 +390,12 @@ def _unit_from_run_dir(run_dir): def main(argv=None) -> int: + """Run the CLI and persist the per-unit verdict. + + @sc [label:policy] exact-counts-fail-the-unit + ``--job-rc`` can fail a stage even when product counts pass; the log records + every verdict, while the manifest is emitted only for success. + """ p = argparse.ArgumentParser(description="ShapePipe per-unit completeness check") sub = p.add_subparsers(dest="cmd", required=True) c = sub.add_parser("check", help="count products, write the manifest")