Persist PSF products and add the two campaign-level merges (exp_persist, star_cat_merge, final_cat_merge) - #879

Merged

cailmdaley merged 95 commits into

feat/persist-exp-products

Sep 28, 2026

Commits on Sep 9, 2026

feat(orchestration): the keep list, and the script that acts on it

authored and
cailmdaley
committed
681f156
feat(orchestration): exp_persist, between exp_psf and clean_exposure

authored and
cailmdaley
committed
de243a8
docs(orchestration): exp_persist in the rule list, and why the report omits it

authored and
cailmdaley
committed
4cf639f
feat(orchestration): exp_persist packs one tar per exposure, not loose copies

cailmdaley
and
claude
committed
1fa6455
fix(orchestration): persist_exp never leaves a .tmp behind on failure

cailmdaley
and
claude
committed
a14a2f3
docs(orchestration): drop references to the removed star-catalogue rules

cailmdaley
and
claude
committed
d91da36
feat(orchestration): the two campaign-level merges

cailmdaley
and
claude
committed
626aaba
fix(orchestration): seven defects in the campaign-level merges

cailmdaley
and
claude
committed
fc9aa3e
fix(create_final_cat): make the merged-catalogue writer reproducible

cailmdaley
and
claude
committed
d774cc3
perf(orchestration): size the two merges from the data, not from a guess

cailmdaley
and
claude
committed
98532ce
feat(orchestration): persist_exp keeps NAMED products, not globs

cailmdaley
and
claude
committed
d0ecfdd
perf(merge_starcat): accumulate arrays, not python lists of floats

cailmdaley
and
claude
committed
3b8c7d5
feat(orchestration): the star catalogue's inputs are not a user choice

cailmdaley
and
claude
committed
0ad6403
fix(cfis): two stale columns in final_cat.param, and no mask column at all

cailmdaley
and
claude
committed
90dfb00
perf(merge_starcat): two passes, so nothing is held twice

cailmdaley
and
claude
committed
a434c9a
feat(orchestration): final_cat_merge reconciles instead of rebuilding

cailmdaley
and
claude
committed
3c29825

Commits on Sep 10, 2026

fix(orchestration): eight defects found reviewing the merge work

cailmdaley
and
claude
committed
78d2ce9
feat(orchestration): the star catalogue becomes hdf5, reconciled like the tile one

cailmdaley
and
claude
committed
0fb514b
fix(orchestration): nine findings from the third review

cailmdaley
and
claude
committed
4ed9fb0
test(unit): property-based state machines for reconcile and persist_exp

cailmdaley
and
claude
committed
e86d8c8
fix(persist-exp): the PSF run dir is run_sp_exp_SxSePsf

cailmdaley
and
claude
committed
1a343e6

Commits on Sep 11, 2026

workflow: image simulations as an input mode of the real-data workflow

martinkilbinger
and
claude
committed
5b88f17
workflow: image-sims merge column list, and the merge reads the workflow layout

martinkilbinger
and
claude
committed
4e17fee
workflow: campaign-unique node-local tile store for image sims; candide excludes

martinkilbinger
and
claude
committed
716b160
workflow: port the MCCD exposure chain to the workflow grammar

martinkilbinger
and
claude
committed
fdc9db4

Commits on Sep 17, 2026

workflow: exp_psf reserves 2 cores, not 8, for the MCCD chain

martinkilbinger
and
claude
committed
836f91d
mccd_interp: test for N_EPOCH among the column names

martinkilbinger
and
claude
committed
ad80f23
workflow: MCCD validated through the full chain; drop the warn-only flags

martinkilbinger
and
claude
committed
2082dcf
workflow: tile_store_root run-config key moves the tile store's bind, per campaign

martinkilbinger
and
claude
committed
9085cd0
completeness.py: atomic write for stage log/manifest JSON
martinkilbinger
committed
7637b39
workflow/config.yaml: universal template, no cluster-specific paths
martinkilbinger
committed
7aababf
workflow: retrieve mode (symlink|vos) is a run-config key, not fixed in the ini
martinkilbinger
committed
d64f88e
config.yaml: restore real nibi defaults, keep the placeholder check as a guard
martinkilbinger
committed
c7bcfdd
Snakefile: check machine: against SP_PROFILE at parse time
martinkilbinger
committed
5be4c43
config.yaml: machines: table drives per-machine (and per-input_type) defaults
martinkilbinger
committed
29442d3
workflow: one run-config resolver for Snakefile, bin/sp and container.py

martinkilbinger
and
claude
committed
7293e3b
run_config: `run:` name, expanded as $run in run-config paths

martinkilbinger
and
claude
committed
084e305
added user run config template
martinkilbinger
committed
78cc0ca
bin/sp: -c/--config-file for the run config, instead of SP_RUN_CONFIG

martinkilbinger
and
claude
committed
d3b088a
bin/sp: document -c, the two-file merge and the environment in the header

martinkilbinger
and
claude
committed
746ca9c
improved commeents
martinkilbinger
committed
4e0bedc
final_cat_merge, star_cat_merge: record code provenance in the HDF5 attributes

cailmdaley
and
claude
committed
3e269c4

Commits on Sep 23, 2026

workflow smoothed; running until hdf5 file
martinkilbinger
committed
fbfdf3b

Commits on Sep 25, 2026

bin/sp: don't swallow -c/--config meant for the delegated command

cailmdaley
and
claude
committed
0b5fae7
workflow/README: add products_dir to the image-sims run-config example

cailmdaley
and
claude
committed
d22c4a0
run_template.yaml: make it resolve

cailmdaley
and
claude
committed
5ded2ca

Commits on Sep 26, 2026

docs(astra): record the pipeline's scientific decisions in astra.yaml

cailmdaley
and
claude
committed
dae9e1b
test(astra): validate decision anchors and universe pins
cailmdaley
committed
e901cc9
Merge remote-tracking branch 'origin/develop' into feat/persist-exp-products-v2
cailmdaley
committed
8e40991
merge: feat/workflow-image-sims (#894 @5ded2ca8) + develop into feat/persist-exp-products

cailmdaley
and
claude
committed
d84822f
workflow: final_cat_merge is the one merger, for data and image sims

cailmdaley
and
claude
committed
4b1e868
workflow: the campaign name is `run:`, and `run:` is required

cailmdaley
and
claude
committed
d7ce47f
workflow: no PSF persistence under psf_model: fake

cailmdaley
and
claude
committed
3386347
test(grammar): check the image-sims final_cat.param too

cailmdaley
and
claude
committed
89f00b5
Merge remote-tracking branch 'origin/docs/astra-decision-record' into HEAD
cailmdaley
committed
1685d65
docs(astra): record the pipeline's scientific decisions in astra.yaml

cailmdaley
and
claude
committed
09edd5e
test(astra): validate decision anchors and universe pins
cailmdaley
committed
206b48d
test(astra): resolve Snakemake rule anchors; JSON report mode

cailmdaley
and
claude
committed
9f9ec5e
docs(astra): rewrite the decision record against develop

cailmdaley
and
claude
committed
5368d9d
docs(claude): point the scientific-decisions section at the anchor test

cailmdaley
and
claude
committed
a6eff5e
feat(make_cat): fixed per-epoch slot count via N_EPOCH_SLOTS

cailmdaley
and
claude
committed
62a2699
feat(workflow): save fixed-slot per-epoch PSF data in tile make_cat

cailmdaley
and
claude
committed
512e7bf
fix(workflow): five review findings on the #894 merge

cailmdaley
and
claude
committed
b2a4d19
workflow: scientific contracts for the campaign products

cailmdaley
and
claude
committed
4958d66
test(workflow): the campaign name has one source, and products one root

cailmdaley
and
claude
committed
5fcc99e
test(final_cat_merge): columns, per-epoch slots, never-fit rows, missing names

cailmdaley
and
claude
committed
e768908
docs(astra): correct seven rationale claims against the code

cailmdaley
and
claude
committed
e3f6f4a
Merge remote-tracking branch 'origin/docs/astra-decision-record' into feat/persist-exp-products-v2
cailmdaley
committed
27b59f7
Merge remote-tracking branch 'origin/feat/make-cat-fixed-epoch-slots' into feat/persist-exp-products-v2
cailmdaley
committed
eb05d3a
Merge remote-tracking branch 'origin/docs/astra-decision-record' into feat/persist-exp-products-v2
cailmdaley
committed
d214966
config(cfis): final_cat.param carries the 12 per-epoch slots (EXP_ID, CCD, HSM_*_PSF)

cailmdaley
and
claude
committed
3358105
docs(astra): two lints this stack resolves — IMAFLAGS_ISO is not merged, the MCCD chain runs

cailmdaley
and
claude
committed
bac6eea
config(cfis): final_cat.param keeps NUMBER

cailmdaley
and
claude
committed
df20cf4
fix(workflow): the sims overlay resolves gauss_3.0_7x7.conv; `sp -- …` drops its separator

cailmdaley
and
claude
committed
05e7381
fix(workflow): refuse psf_model=mccd at parse time

cailmdaley
and
claude
committed
45a9f1b
fix(workflow): refuse reclamation when products_dir is run_dir

cailmdaley
and
claude
committed
16df7ae
fix(workflow): run_config expands to a fixed point and reports any leftover $

cailmdaley
and
claude
committed
96abec8
fix(persist_exp): the manifest records each member's sha256

cailmdaley
and
claude
committed
072b705
fix(hdf5_reconcile): one type per column across a campaign

cailmdaley
and
claude
committed
d179f2b
test(hdf5_reconcile): isolate schema invalidation from source stamps

cailmdaley
and
claude
committed
187968e
fix(build_index): readiness is the tiles row, not the retained edges

cailmdaley
and
claude
committed
70d7a20
fix(hdf5_reconcile): delete an abandoned tmp before the free-space check

cailmdaley
and
claude
committed
5a61aaf
fix(hdf5_reconcile): each provenance record replaces the last

cailmdaley
and
claude
committed
47b2ec5
fix(workflow): a reclaimed exposure's clean waits on its tar, not the manifest

cailmdaley
and
claude
committed
b15d442
test(conftest): the candide hostname test matches the whole bare name

cailmdaley
and
claude
committed
5b56a68
test(workflow): add an isolated Snakemake DAG driver

cailmdaley
and
codex
committed
33391a4
test(workflow): enforce campaign DAG scope and product custody

cailmdaley
and
codex
committed
7f5ecf2
test(workflow): pin prologues and shells at campaign boundaries

cailmdaley
and
codex
committed
0e279d1
test(workflow): the DAG harness follows the parse-time guards

cailmdaley
and
claude
committed
a846e08

Commits on Sep 28, 2026

Merge origin/develop into feat/make-cat-fixed-epoch-slots

cailmdaley
and
claude
committed
8209e26
Merge origin/develop into feat/persist-exp-products

cailmdaley
and
claude
committed
fedbf9c
Merge origin/feat/make-cat-fixed-epoch-slots into feat/persist-exp-products

cailmdaley
and
claude
committed
a2b37a8
test: seams and run_config follow the develop merge

cailmdaley
and
claude
committed
2d833fd
test(workflow): snakemake-driven checks run in their own CI step

cailmdaley
and
claude
committed
aa458b2
Merge origin/develop into feat/persist-exp-products

cailmdaley
and
claude
committed
5891cf6