diff --git a/CLAUDE.md b/CLAUDE.md index 4722a455b..1ac129708 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -81,19 +81,35 @@ Full detail: `docs/source/installation.md` and `docs/source/container.md`. - `src/shapepipe/` — the package (src-layout). `modules/` holds the pipeline modules and their `*_runner.py` wrappers; `pipeline/` is execution and file - I/O; `utilities/`; `canfar/` is CANFAR/cluster job orchestration. Console entry - points (`shapepipe_run`, `summary_run`, `canfar_*`) are defined under + I/O; `utilities/`; `canfar/` monitors CANFAR sessions. Console entry + points (`shapepipe_run`, `canfar_*`, `plot_coverage_map`) are defined under `[project.scripts]`. - `tests/` — the whole test suite, one discovery root: `module/` (per-module unit/property/integration tests), `unit/` (structural), `science/` (fast guardrails), `cluster/` (candide-only), `helpers/` (shared library code). See `tests/README.md`. +- `workflow/` — **the production orchestration**: a Snakemake workflow over the + same `shapepipe_run` calls. `bin/sp` is the entry point (`run`, `report`, + `container`, `cancel`); `Snakefile` plus `rules/{prepare,exposure,tile}.smk` + are the rule graph; `scripts/` holds the plain Python each rule + shells out to; `config.yaml` declares one run (tile list, the scratch and + persistent roots, the container); `config/cfis/` holds the committed ini + chain; `profiles/nibi/config.yaml` is the SLURM executor profile. Deep + reference: `workflow/README.md`. User-facing: `docs/source/workflow.md`. - `example/` — a runnable example pipeline (`example/config.ini`) on a single CFIS tile; doubles as the CI smoke test. - `scripts/` — shell / Python / notebook helpers (`sh/`, `python/`, `jupyter/`), symlinked onto `$PATH` inside the image. - `docs/` — Sphinx sources; the API docs are generated from docstrings. +**There is no concept of a catalogue version in the code.** Nothing branches on +`v1.3`..`v1.6` or `v2.0`, and there are no sky patches (`P1`..`P9`): a campaign +is a tile list, and the version of a catalogue is the git tag of the code that +produced it. The pre-Snakemake bash job +layer (`scripts/sh/run_job_sp_canfar_v2.0.bash`, `job_sp_canfar_v2.0.bash`, +`job_list_help.bash`, `functions.sh`) stays — it is version-free, and +sp_validation's image-simulation workflow calls it. + ## Development workflow **We are in rapid iteration.** Changes that require re-running the pipeline are fine; a PR that changes science defaults (detection parameters, cuts, module options) does not need a completed rerun to be accepted — state what we expect the rerun to show, and do the rerun when it comes up. diff --git a/Dockerfile b/Dockerfile index 400ee591f..f9b5bb233 100644 --- a/Dockerfile +++ b/Dockerfile @@ -139,6 +139,7 @@ RUN chmod -R go+rwX /app && \ uv pip install --no-deps -e . && \ for ext in .py .sh .bash; do \ for script in /app/scripts/*/*$ext; do \ + [ -e "$script" ] || continue; \ link_name=$(basename $script $ext); \ ln -s $script /usr/local/bin/$link_name; \ done; \ diff --git a/README.rst b/README.rst index fc1413031..c248e43c7 100644 --- a/README.rst +++ b/README.rst @@ -49,7 +49,8 @@ to start: - `Installation `_ — getting ShapePipe onto your machine or cluster. - `Basic execution `_ and `configuration `_ — running ``shapepipe_run`` and writing pipeline configs. - `Container workflow `_ — what's in the image and the ``pyproject.toml`` / ``uv.lock`` / ``Dockerfile`` layers. -- `Running on a cluster `_ — pulling the image and submitting jobs, with worked candide (SLURM) and CANFAR examples. +- `Running on a cluster `_ — pulling the image and submitting jobs, with a worked candide (SLURM) example. +- `The Snakemake workflow `_ — the production orchestration for a whole tile list, driven by ``workflow/bin/sp``. If you use ShapePipe in academic work, please cite Guinot et al. (2022) and Farrens et al. (2022). diff --git a/bin/canfar_submit_job.py b/bin/canfar_submit_job.py deleted file mode 100755 index 202e9d274..000000000 --- a/bin/canfar_submit_job.py +++ /dev/null @@ -1,3 +0,0 @@ -#!/usr/bin/env python -from shapepipe.canfar_run import run_job -run_job() diff --git a/bin/summary_run.py b/bin/summary_run.py deleted file mode 100755 index a59689e41..000000000 --- a/bin/summary_run.py +++ /dev/null @@ -1,3 +0,0 @@ -#!/usr/bin/env python -from shapepipe.summary_run import main -main() diff --git a/docs/source/clusters.md b/docs/source/clusters.md index 0dd189b74..ffae2b4b2 100644 --- a/docs/source/clusters.md +++ b/docs/source/clusters.md @@ -1,10 +1,9 @@ # Running on a Cluster ShapePipe runs the same way on every cluster: **through the container**. You -pull the image once, bind-mount your clone, and run `shapepipe_run` (or the -CANFAR submission tooling) inside it — there is no environment to install or -activate on the host. This page covers the shared pattern, then the specifics -for each supported machine. +pull the image once, bind-mount your clone, and run `shapepipe_run` inside it — +there is no environment to install or activate on the host. This page covers +the shared pattern, then the specifics for each supported machine. For what is *inside* the image and how it is built, see [Container Workflow](container.md). @@ -67,32 +66,12 @@ over unchanged. ## CANFAR -CANFAR submission does not go through a batch scheduler. Instead you submit -container jobs to CANFAR's headless system with the `canfar_submit_job` console -script (backed by the `canfar` library), and watch them with `canfar_monitor` / -`canfar_monitor_log`. Pipeline steps are **bit-coded** through `-j` (the same -scheme as `scripts/sh/job_sp_canfar.bash`), the PSF model is chosen with -`-p psfex|mccd`, and `-V` selects the image version: +ShapePipe has no CANFAR submission path. Production orchestration is the +Snakemake workflow in `workflow/` — see `workflow/README.md`. It is SLURM-only and has no CANFAR execution mode, so a +CANFAR campaign means moving to a SLURM site (nibi, candide). -```bash -# Submit pipeline step(s) for the configured tiles (bit-coded -j). -canfar_submit_job -j 1 -p psfex -V 1.1 - -# Monitor sessions/jobs and stream logs. -canfar_monitor -canfar_monitor_log -``` - -The full production run — input preparation, the per-step `-j` table, and -post-processing — is documented in the -[CANFAR production walkthrough](pipeline_canfar.md). - -```{note} -The CANFAR production submission scripts (`scripts/sh/job_sp_canfar*.bash`) still -run under the pre-container environment and are slated for the same -container-first cleanup the candide scripts received. Treat the walkthrough as -the current-but-evolving production procedure. -``` +`canfar_monitor` and `canfar_monitor_log` stay: they list, filter and destroy +CANFAR sessions, independently of how those sessions were started. ## ccin2p3 diff --git a/docs/source/conf.py b/docs/source/conf.py index eea9b8f43..d52836d43 100644 --- a/docs/source/conf.py +++ b/docs/source/conf.py @@ -104,6 +104,10 @@ myst_enable_extensions = ["html_image"] +# Generate slug anchors for h1-h3 so in-page links like +# `[Mask images](#mask-images)` resolve instead of warning. +myst_heading_anchors = 3 + # -- Options for HTML output ------------------------------------------------- # The theme to use for HTML and HTML Help pages. See the documentation for diff --git a/docs/source/dependencies.md b/docs/source/dependencies.md index 8c9afd630..84721d7d6 100644 --- a/docs/source/dependencies.md +++ b/docs/source/dependencies.md @@ -34,8 +34,7 @@ Data access & infrastructure (CANFAR / UNIONS): | Package | Purpose | |---------|---------| | [vos](https://github.com/opencadc/vostools) | CADC / CANFAR VOSpace access | -| [skaha](https://github.com/shinybrar/skaha) | CANFAR Science Platform sessions | -| canfar | CANFAR container-job submission | +| canfar | CANFAR session monitoring | | [astroquery](https://astroquery.readthedocs.io/) | external catalogue queries | | [cs_util](https://github.com/CosmoStat/cs_util) | shared CosmoStat utilities | | [sqlitedict](https://github.com/RaRe-Technologies/sqlitedict) | on-disk pipeline state | diff --git a/docs/source/pipeline_canfar.md b/docs/source/pipeline_canfar.md deleted file mode 100644 index 21416d375..000000000 --- a/docs/source/pipeline_canfar.md +++ /dev/null @@ -1,545 +0,0 @@ -# Running `ShapePipe` processing and post-processing pipelines on CANFAR - -Documentation to create ShapePipe output products for catalogues v1.x. - -## Initial Setup - -### CANFAR Login - -Login to the canfar system with - -```bash -canfar auth login -``` - -This can be done from at notebook or terminal within the canfar science portal, -or any remote terminal that has the canfar library installed. - -Check authentication status with - -```bash -canfar auth list -``` - -If not on "default", run - -```bash -canfar auth switch default -``` - -### Set variables (optional) - -Set the current patch in the shell as - -```bash -patch=P[1-9] -``` - -For convenience, the current PSF model can be set as environment variable, e.g.: - -```bash -psf="psfex" -``` - -Allowed are `psfex` and `mccd`. - -Setting the terminal title to display the patch can be useful for long jobs, to keep track of which terminal -runs which patch: - -```bash -echo -ne "\033]0;$patch\007" -``` - -### Prepare run directory - -First, go to the dedicated directory with - -```bash -cd /path/to/version/$patch -``` - -Next, set links to the tile number list and configuration directory: - -```bash -ln -s ~/shapepipe/auxdir/CFIS/tiles_202106/tiles_$patch.txt tile_numbers.txt -ln -s ~/shapepipe/example/cfis -``` - -Create output and debug log directories - -```bash -mkdir -p output -mkdir -p debug -``` - -Finally, create and link to central image storage directories for tiles and exposures: - -```bash -mkdir -p ~/cosmostat/v2/data_tiles/$patch -ln -s ~/cosmostat/v2/data_tiles/$patch data_tiles -mkdir -p ~/cosmostat/v2/data_exp/$patch -ln -s ~/cosmostat/v2/data_tiles/$patch data_exp -``` - -## `ShapePipe` processing - -Now, everything should be ready to start running `ShapePipe` for the weak lensing processing. The following -details all necessary steps. - -### Get Images - -We first download images, and in a second run create symbolic links with the proper pipeine naming scheme. - -#### Download and move tiles - -When running the main `ShapePipe` script `shapepipe_run`, the following env variable needs to point -to the current working directory - -```bash -export SP_RUN=`pwd` -``` - -Now we run the first module (`get_images_runner`) to download the tile images together with the weight files. -This run can get interrupted by VOSpace I/O or connection errors. In that case, -we move new files to the image storage directory, remove the previous (now void of images) run directory, -and update the run log file. We also check the number of previous and new tiles. - -```bash -shapepipe_run -c cfis/config_Git_vos.ini -ls -l data_tiles/ | wc -mv -i output/run_sp_Git_*/get_images_runner/output/CFIS.???.???.*fits* data_tiles -ls -l data_tiles/ | wc -rm -rf output/run_sp_Git_* -update_runs_log_file.py -``` - -Repeat the above block as needed. - -### Find Exposures - -With all tile images (= stacks) downloaded, we can inquire their headers to identify the exposures that were used -to create the stacks. This call to the pipeline also creates the symbolic links to the downloaded tile images. - -```bash -shapepipe_run -c cfis/config_GitFe_symlink.ini -``` - -(One could also run `Fe` alone.) - -### Download and Move Exposures - -The last module create exposure lists on output. These are now used to download all exposures. As for the tile downloads, -we have to account for VOSpace errors. - -```bash -shapepipe_run -c cfis/config_Gie_vos.ini -mv -i output/run_sp_Gie_*/get_images_runner/output/*.fits*fz data_exp -rm -rf output/run_sp_Gie_* -update_runs_log_file.py -``` - -Repeat the above by hand, or peform it in an automatic loop: - -```bash -while true; do - shapepipe_run -c cfis/config_Gie_vos.ini - ls -l data_exp/ | wc - mv -i output/run_sp_Gie_*/get_images_runner/output/*.fits*fz data_exp - ls -l data_exp/ | wc - rm -rf output/run_sp_Gie_* - update_runs_log_file.py -done -``` - -**Note:** Make sure that after all images are downloaded there is no `Gie` run in the output directory. -This would mess up later modules since `last:get_image_runner` could point to this run. - -### Create tile links again (necessary?) - -If necessary, e.g. because a previous `Git` run is no longer valid, re-create the symbolic links to the downloaded tiles with - -```bash -job_sp_canfar.bash -p $psf `cat tile_numbers.txt` -j 1 -r symlink -``` - -### Uncompress tile weights - -The downloaded tile weights are compressed. The following call uncompresses all. - -```bash -shapepipe_run -c cfis/config_tile_Uz.ini -``` - -### Masks - -There is no masking step. `ShapePipe` generates no masks: the sky-fixed -healsparse maps are queried once per object, by `mask_query` on the exposure -catalogues (`MASK_EXT`, recorded but not cut on) and by `make_cat` on the tile -catalogue (`MASK_` columns). Point the `MASK_PATHS` / `MASK_EXT_PATHS` -config entries at the maps and nothing else is needed — no star-catalogue -download, no rasterization, no `combine_runs.bash -c flag_*`. The only mask that -touches pixels is the instrument flag image shipped with each exposure, which -`split_exp` splits per CCD. - -## Tile detection - -We can finally run our first module using the canfar submission system. First, determine the number of optimal jobs such that at a given -time the allowed maximum of 512 running jobs is not exceeded. -Set as `N_PAR` (number of parallel jobs) a number between 1 and 8. - -```bash -canfar_submit_job -j 16 -f tile_numbers.txt -P N_PAR -v -s -``` - -Now, run the previous command with that number `JMAX` - -```bash -canfar_submit_job -j 16 -f tile_numbers.txt -P N_PAR -v -J JMAX -``` - -### Exposure Processing - -#### Option 0: Global split (deprecated; used for earlier v1.x patch runs) - -For this option, set `sp_local=0`. - -**TODO: ** Split Uz and SpMh - -For `sp_local=-` both `mh_local` (0, 1) are ok: - -```bash -export mh_local=0 -``` - -### Option 1: Local split exposures (recommended) - -Optional: Enable flags for local split processing and merge header runs as - -```bash -export sp_local=1 -export mh_local=1 -``` - -These flags are automatically set to 1 in the new job scripts. - - -Get single-HDU single-exposure IDs file (from missing 32 job): - -```bash -summary_run P$patch 32 -cp summary/missing_job_32_all.txt exp_shdu.txt -``` - -### Split exposures - -First, determine the number of maximum jobs with the option `-s` (see above). Then, submit with - -```bash -canfar_submit_job -j 2 -v -f exp_shdu.txt -v -P N_PAR -J JMAX -``` - -### Exposure detection - -```bash -canfar_submit_job -j 32 -f exp_shdu.txt -v -P N_PAR -J JMAX -``` - -### Tile preparation - -```bash -canfar_submit_job -j 64 -f tile_numbers.txt -``` - -### Tile shape measurement - -```bash -canfar_submit_job -j 128 -f tile_numbers.txt -``` - -### Merge sub-catalogues - -```bash -canfar_submit_job -j 256 -f tile_numbers.txt -``` - -### Create final catalogues - -```bash -canfar_submit_job -j 512 -f tile_numbers.txt -``` - -This was the last `ShapePipe` module to run for main processing. - - -### Merge all final catalogues - -The last step of `ShapePipe` processing is, per patch, to merget all final catalogues. This is done via a python script, as follows. -First, change to parent directory `/path/to/version` and run the following command for all patches - -```bash -patchnum=`tr $patch P ''` -create_final_cat.py -m final_cat_$patch.hdf5 -i . -p $patch/cfis/final_cat.param \ - -P $patchnum -o $patch/n_tiles_final.txt -v -``` - -## Additional `ShapePipe` processing - - -### Create star Catalogue - -We can additionaly create a combined star catalogue, with star shapes projecte from detector to world coordinates. -This is useful for validation and galaxy-PSF/star correlation diagnostics. - -#### Combine all PSF runs - -In each patch directory /path/to/version/$patch, run - -```bash -combine_runs.bash -p $psf -c psf -``` - -to create a single output directory of PSF files (symbolic links). - -Optionally, to create and plot results for this patch only: - -```bash -shapepipe_run -c $SP_CONFIG/config_Ms_$psf.ini -shapepipe_run -c $SP_CONFIG/config_Pl_$psf.ini -``` - -#### Collate star catalogues - -Collate all input validation PSF files into star catalogues, gathering positions -(X/Y/RA/DEC) and the MCCD CCD id. - -Note: HSM shapes are no longer rotated into world coordinates at this step. The -PSF/star ellipticities and sizes are now measured directly in sky coordinates -during PSF interpolation (galsim `FindAdaptiveMom(use_sky_coords=True)`), so -`collate_star_cat.py` only collates and passes the shapes through. - -> **v2.0 is patch-less.** Runs up to `v1.6` are organised in sky patches -> `P1`..`P`, each patch a run directory. `v2.0` removes the patch concept: a -> single run root, outputs directly under it. `v2.0` is the default; select an -> older layout with `-V` (e.g. `-V v1.6`). For `v2.0` the patch loop and the -> `-P` option no longer apply, and the patch token drops from the output -> filename (`validation_psf_conv-.fits` instead of -> `validation_psf_conv--.fits`). - -```bash -cd /path/to/version -mkdir star_cat -cd star_cat -``` - -For `v2.0` (the default), run once against the patch-less run root, producing -files `validation_psf_conv-.fits`: - -```bash -collate_star_cat.py -i .. -v -``` - -For `v1.x`, pass the version explicitly and run once per patch, creating a -directory per patch `P?` and producing files -`validation_psf_conv--.fits` (for the v1.4 setup only one file): - -```bash -collate_star_cat.py -i .. -V v1.6 -P $patchnum -v -``` - -Combine previously created files as links within one ShapePipe run directory (for the v1.4 setup only one link). -First (and optiohnal), create a subdir for a run and link to the input patches: - -```bash -cd /path/to/version/star_cat -mkdir v1.6 -ln -s ../P1 -ln -s ../P2 -... -``` - -Next, create links to all `validation_conv` runs: - -```bash -combine_runs.bash -p psfex -c psf_conv -``` - -Merge all converted star catalogues and create `final-starcat.fits`: - -```bash -export SP_RUN=`pwd` -shapepipe_run -c ~/shapepipe/example/cfis/config_Ms_psfex_conv.ini -``` - -Rename to general PSF and star catalogue used for all ("a") sub-versions: - - -```bash -cp output/run_sp_tile_Ms/merge_starcat_runner/output/full_starcat-0000000.fits \ - unions_shapepipe_psf_2024_v1.6.a.fits -``` - -The FITS file `CATTYPE` (newer version) should be `validation_psf_conf`. - -## Post-processing - -The following post-processing steps are performed with the library `sp_validation`. - -### Extract Information - -First, we extract all information from the final catalogue, per patch. We copy -the parameter file and set links to the catalogues and `ShapePipe` config directory. - -```bash -cd /path/to/version/$patch -cp ~/astro/repositories/github/sp_validation/notebooks/params.py . -ln -s /path/to/final_cat_$patchnum.hdf5 # not relative path ../final_cat_P$patchnum.hdf5 ! -ln -s output/run_sp_MsPl/mccd_merge_starcat_runner/output/full_starcat-0000000.fits -ln -s ~/astro/repositories/github/shapepipe/example/cfis -``` - -Then edit `params.py`: Set patch name; set `wrap_ra` for P2. - -Now we can run the script, recommended via job submission on candide. For large patches, -this requies a job with a large memory, e.g. with `mem=380000` - - -```bash -[squeue] python ~/astro/repositories/github/sp_validation/scripts/calibrate/extract_info.py -``` - -This creates a patch-wise comprehensive catalogue. - -### Create global comprehensive catalogues - -```bash -cd /patch/to/version -[squeue] python ~/astro/repositories/github/sp_validation/scripts/create_joint_comprehensive_cat.py \ - -v v1.6.c -v -p P1+P2+P3+P4+P5+P6+P7+P8+P9 -``` - -This creates the file `unions_shapepipe_comprehensive_2024_v1.6.c.hdf5`. - - -### Apply structural masks - -First, edit the Python script `~/astro/repositories/github/sp_validation/notebooks/demo_apply_hsp_masks.py` -to match catalogue name. Check the coverage mask input file (see below). -Run the script to apply the healsparse structural masks: - -```bash -[squeue] python ~/astro/repositories/github/sp_validation/notebooks/demo_apply_hsp_masks.py -``` - -This creates the file `unions_shapepipe_comprehensive_struct_2024_v1.6.c.hdf5`. - - -### Define sample, calibrate catalogue - -We are close to finally perform the last post-processing step, which is the calibration. First, the final galaxy sample -in question needs to be defined, with masks and cuts to apply from a `yaml` config file. A number of pre-defined files -can be found in `~/astro/repositories/github/sp_validation/calibration`. - -For example, to create `v1.6.6`, the steps are: - -```bash -cd /path/to/version -mkdir -p v1.6.6 -cd v1.6.6 -ln -s ~/astro/repositories/github/sp_validation/calibration/mask_v1.X.6.yaml config_mask.yaml -ln -s ..//unions_shapepipe_comprehensive_struct_2024_v1.6.c.hdf5 unions_shapepipe_comprehensive_struct_2024_v1.X.c.hdf5 -[squeue] python ~/astro/repositories/github/sp_validation/calibrate_comprehensive_cat.py -``` - -calibrate_comprehensive - - -```{note} -The matched-star-catalogue diagnostic (`merge_psf_cat.py`) has moved into -[`sp_validation`](https://github.com/CosmoStat/sp_validation) / `cosmo_val`; -see that repository for its current equivalent. The coverage-mask helpers -(`get_ccds_with_psf`, `download_headers`, `extract_field_corners`, -`build_coverage_map`, `plot_coverage_map`, and `summary_run`) are shipped by -ShapePipe and documented below. -``` - -### Create matched star catalogue - -For diagnostics, a catalogue with multi-epoch shapes measured by ngmix matched with the validation star catalogue is used. -This is created as follows: - -```bash -cd /path/to/version -merge_psf_cat.py [-V v1.6|-P P1+P2+...] -v -``` - -This creates the joint catalogue unions_shapepipe_star_2024_v1.6.a.fits . - -### Create coverage mask - -First, on canfar, move to the directory that has the patch subdirectories. - -```bash -cd /path/to/version -``` - -#### Get exposure numbers - -If the file `$patch/exp_numbers.txt` does not exist for a given patch, create it with the summary program - -```bash -summary_run $patch 1 -``` - -Now, create the list of single-CCD footprints that have a valid PSF model. -Only these CCDs are stamped into the mask, so the accumulated map counts, per -sky pixel, the number of exposures with a valid PSF. The list is written to -`ccds_with_psf_v1.6.txt`: - -```bash -get_ccds_with_psf -v -V v1.6 -o ccds_with_psf_v1.6.txt -``` - -Next, download the exposure headers; indicate (with `-d`) a directory of -already downloaded headers, which are linked so duplicate downloads are -skipped: - -```bash -download_headers -i ccds_with_psf_v1.6.txt -o headers_v1.6 -d headers_v1.3 -v -``` - -From the headers, the per-CCD corner coordinates are extracted. Pass the -CCD list with `-l` so only CCDs with a valid PSF are written: - -```bash -extract_field_corners -i headers_v1.6 -l ccds_with_psf_v1.6.txt -o exp_ra_dec_v1.6.txt -v -``` - -Then, build the healsparse coverage mask file as -```bash -build_coverage_map -i exp_ra_dec_v1.6.txt -o coverage_v1.6.x.hsp -c 128 -n 131072 -v -``` - -The healsparse resolutions (128, 131072) match the bit masks. - - -Use `plot_coverage_map` to plot a region of the coverage mask. For example, -the SGC region with a colorbar clipped to the 1–5 exposure range: - -```bash -plot_coverage_map -i coverage_v1.6.x.hsp -o coverage_v1.6_SGC.png \ - -C -R -20 -r 45 -D 18 -d 40 -m 1 -M 5 -v -``` - -Here `-R`/`-r` and `-D`/`-d` set the RA and Dec plot limits, `-m`/`-M` the -colorbar range, and `-C` adds the colorbar. Building and plotting for a range -of versions (SGC and NGC regions) is automated by -`build_and_plot_coverage_maps.sh`. - - -## Extra Utilities - -### Run in Terminal in Parallel - -```bash -cat IDs.txt | xargs -I {} -P 16 bash -c 'init_run_exclusive_canfar.sh -j 512 -e {}' -``` diff --git a/docs/source/pipeline_tutorial.md b/docs/source/pipeline_tutorial.md index ff1059e2d..eb5586dc3 100644 --- a/docs/source/pipeline_tutorial.md +++ b/docs/source/pipeline_tutorial.md @@ -2,13 +2,17 @@ ## Quick start -Run the entire pipeline on a single example CFIS image with tile ID 246.290: -1. [Install](installation.md) `ShapePipe` — the recommended path is the container image, which bundles everything needed to run the pipeline. -3. Run the job script +Production runs go through the [Snakemake workflow](workflow.md). Put the tile +IDs in the file named by `tile_list` in `workflow/config.yaml`, one +`IDra.IDdec` per line, and run + ```bash -job_sp 246.290 -j 127 +workflow/bin/sp run ``` +A single tile is a legal tile list and is the smallest useful end-to-end run. +The sections below describe what that produces, stage by stage. + ## Introduction The `ShapePipe` pipeline processes single-exposure images and stacked images. Input images have to be calibrated beforehand for astrometry and photometry. This tutorial of an entire `ShapePipe` run covers specifically images from CFIS, the Canada-France Imaging Survey. CFIS stacks are so-called tiles, which are the co-adds of on average three exposures in the r-band. @@ -69,21 +73,21 @@ Naming and numbering of the input files can closely follow the original image na sheared images. This information is used in post-processing to compute calibrated shear estimates via metacalibration. - Summary statistic files - The `SETools` module that creates samples of objects according to some user-defined selection criteria (see [Select stars](#select-stars)) also outputs ASCII + The `SETools` module that creates samples of objects according to some user-defined selection criteria (see [Detect objects](#detect-objects-on-tiles-and-process-stars-on-single-exposures)) also outputs ASCII files with user-defined summary statistics for each CCD, for example the number of selected stars, or mean and standard deviation of their FWHM. Example: `star_stat-2366993-18.txt` - Tile ID list - ASCII file with a tile number on each line. Used for the `get_image_runner` module to download CFIS images (see [Download tiles](#download-tiles)). + ASCII file with a tile number on each line. Used for the `get_image_runner` module to download CFIS images (see [Retrieve input images](#retrieve-input-images)). - Single-exposure name list ASCII file with a single-exposure name on each line. Produced by the `find_exposure_runner` module to identify single exposures that were used to create - a given tile. See [Find exposures](#find-exposures)). + a given tile. See [Retrieve input images](#retrieve-input-images)). - Plots The `SETools` module can also produce plots of the objects properties that were selected for a given CCD. The type of plot (histogram, scatter plot, ...) and quantities to plot as well as plot decorations can be specified in the - selection criteria config file (see [Select stars](#select-stars)). + selection criteria config file (see [Detect objects](#detect-objects-on-tiles-and-process-stars-on-single-exposures)). Example: `hist_mag_stars-2104133-5.png` - Log files @@ -94,7 +98,7 @@ Naming and numbering of the input files can closely follow the original image na `ShapePipe` splits the processing of CFIS images into several parts: These are the retrieval and preparation of input images, processing of single exposures, -processing of tile images, creation and upload (optional) of _final_ shape catalogues. +processing of tile images, and creation of _final_ shape catalogues. The following flowchart visualised the processing parts and steps. @@ -104,44 +108,58 @@ Below, the individual processing steps are described in detail. ### Input and output paths -All required paths are automatically set in the job script `job_sp`. +The workflow's rules export every path a config interpolates before each +`shapepipe_run` call, so nothing here needs setting by hand for a normal run. -If an example config file is run outside this script, -the following path variables might need to be defined. +If a config file is run outside the workflow, the following variables need to be +defined. - `$SP_RUN`: Run directory of `ShapePipe`. In general this is just `pwd`, and can be set via ```bash export SP_RUN=`pwd` ``` but on a cluster this directory might be different. - `$SP_CONFIG`: Path to configuration files. In our example this is `$SP_BASE/example/cfis`. +- `$SP_UNIT_NUM`: the tile or exposure this run is restricted to, in ShapePipe's + image-number convention — a leading dash, and dots replaced by dashes (tile + `210.282` becomes `-210-282`, exposure `2605805` becomes `-2605805`). The + configs interpolate it into `NUMBER_LIST`. In addition, the output path `$SP_RUN/output` needs to be created by the user before running `ShapePipe`. -### Job and pipeline scripts +### The rule chain -The job script to run the pipeline in its entity or in parts is `job_sp[.bash]`. Type -```bash -job_sp -h -``` -for all options. +There is no monolithic job script. Every processing step below is a **rule**, and +every rule instance is one `shapepipe_run` call on one unit — one tile, or one +single exposure — inside that unit's own directory: -This script creates the subdirectory `$SP_RUN/output` to store all pipeline outputs -(log files, diagnostics, statistics, output images, catalogues, single-exposure headers with WCS information). +| level | rule chain | +| --- | --- | +| tile, prepare phase | `tile_get_images` → `tile_uncompress` → `tile_find_exposures` | +| exposure | `exp_get_images` → `exp_split` → `exp_psf` → `exp_persist` → `clean_exposure` | +| tile, compute phase | `tile_exp_forest` → `tile_merge_headers` → `tile_detect` → `tile_vignets` → `tile_ngmix` → `tile_merge_cats` → `tile_make_cat` → `clean_tile` | + +Each rule calls -The job script automaticall performs a number of subsequent calls to the `ShapePipe` executable `shapepipe_run`, as ```bash shapepipe_run -c $SP_CONFIG/.ini ``` -The config file `.ini` contains the configuration for one or more modules. -See the main `ShapePipe` readme for more details. -The user specifies which steps are run with the command line option `-j JOB`. The integer value `JOB` -is bit-coded such that arbitrary combinations of steps can be run with a single call to `job_sp`. For -example, to run steps #1 and #2, type `job_sp -j 3`. +on a config file committed under `workflow/config/cfis/`, containing the +configuration for one or more modules. See [Configuration](configuration.md) for +what goes in one. To run a single rule and its prerequisites and nothing else, +name it as a snakemake target: `sp exp_psf ...`. + +Each unit directory holds an `output/` subdirectory with all pipeline outputs +(log files, diagnostics, statistics, output images, catalogues, single-exposure +headers with WCS information), beside the `manifests/` and `logs/` that are the +workflow's own bookkeeping. ### Select tiles -To run the job script, one or more CFIS tiles need to be chosen. If the tile IDs are known, they are provided to `job_sp` on the command line. +The tiles to process are the lines of the tile list. The campaign grows by +appending to that file: the tile-to-exposure index accumulates across +invocations, so adding tiles later changes which jobs exist without invalidating +completed work. If the tile IDs are not known a priori, they can be selected via sky coordinates, with the script `cfis_field_select`. For example, to find the tile number for a Planck cluster at R.A.=213.68 deg, dec=57.79 deg, run: @@ -151,41 +169,41 @@ cfis_field_select -i /path/to/shapepipe/auxdir/CFIS/tiles_202007/tiles_all_order The input text file (provide via the flag `-i`) contains a list of CFIS tiles, this can also be directory containing the tile FITS files. -The following sections describe the different steps that are performed with `job_sp`. +The following sections describe the steps the rule chain performs. ## Run the pipeline ### Retrieve input images -The command -```bash -job_sp TILE_ID -j 1 -``` -retrieves the image and weight corresponding to TILE_ID using the module `get_images`. -It then identifies the exposures that were used to create the tile image via the `find_exposures` runner. -Finally, another call to `get_images` retrieves the exposure images, weights, and flag files. +`tile_get_images` retrieves the image and weight corresponding to a tile ID +using the module `get_images`. `tile_find_exposures` then identifies the +exposures that were used to create the tile image via the `find_exposures` +runner, and `exp_get_images` retrieves each of those exposures' image, weight +and flag files. -For the retrieval method the user can choose betwen -- download from VOspace (`-r vos`); -- create symbolic link to existing file on disk (`-r symlink`). +For the retrieval method the user can choose between +- download from VOspace (`RETRIEVE = vos` in the config); +- create symbolic link to existing file on disk (`RETRIEVE = symlink`). Note that internet access is required for this step if the download method is `vos`. -An output directory `run_sp_GitFeGie` (in `output`) is created containing the results of `get_images` for tiles (`Git`), -`find_exposures` (`Fe`), and `get_images` for exposures (`Gie`). +These three rules are the workflow's PREPARE phase. The exposure half of the run +cannot be scheduled before them, because which exposures exist is something +`find_exposures` has to say first — which is why one run is two snakemake +invocations over one Snakefile (see [the workflow page](workflow.md)). ## Prepare input images -With -```bash -job_sp TILE_ID -j 2 -``` -the compressed tile weight image is uncompressed via the `uncompress_fits` module. Then, the single-exposure images, weight, and flags are split into single-exposure single-CCD file -(one FITS file per CCD) with `split_exp`. -Finally, the headers of all single-exposure single-CCD files are merged into a single `sqlite` file, to store the WCS information of the input exposures. +`tile_uncompress` uncompresses the compressed tile weight image via the +`uncompress_fits` module. + +`exp_split` splits each single-exposure image, weight and flag into +single-exposure single-CCD files (one FITS file per CCD) with `split_exp`, and +writes that exposure's WCS information to `headers-.npy`. -Two output directories are created, `run_sp_Uz` for `uncompress_fits`, and `run_sp_exp_SpMh` for the output of the modules -`split_exp` (`Sp`) and `merge_headers` (`Mh`). +`tile_merge_headers` then merges the headers of the exposures overlapping one +tile into a single `sqlite` file, so the tile-side modules can look up the WCS of +any of them. ## Masks @@ -233,23 +251,23 @@ with `FLAG_IMAGE = False`. ## Detect objects on tiles and process stars on single exposures -The call -```bash -job_sp TILE_ID -j 8 -``` -performs a number of steps. First, objects on the tiles are deteced with the `sextractor` runner. -Next, the following tasks are run on the single-exposure single-CCD images: -- Objects are deteced with `sextractor`. +`tile_detect` detects objects on the tile with the `sextractor` runner. + +`exp_psf` runs the single-exposure single-CCD chain: +- Objects are detected with `sextractor`. - Star candidates are selected via `setools`. -- The PSF model is created, either with `psfex` for PSFex, or - with `mccd_preprocessing` and `mccd_fit_val` for MCCD. -- The PSF model is interpolated to star positions for validation. For the PSFEx model, this is done - via a call to `psfex_interp`. For MCCD, the modules `merge_starcat`, `mccd_plots`, and - `mccd_interp` are called. +- The PSF model is created with `psfex`. +- The PSF model is interpolated to star positions for validation, via a call to + `psfex_interp`. -The output directory for both the `mccd` and `psfex` options is `run_sp_tile_Sx_exp_SxSePsf`. -This stores the output of SExtractor on the tiles (`tile_Sx`), on the exposures (`exp_Sx`), -`setools` (`Se`), and the Psf model (`Psf`). +The output directory is `run_sp_exp_SxSePsfPi`, holding the output of SExtractor +on the exposures (`Sx`), `setools` (`Se`), the PSF model (`Psf`) and the +validation interpolation (`Pi`). + +`exp_persist` then copies the products named by `persist_exp:` in +`workflow/config.yaml` — by default the `validation_psf-*.fits` that the rho and +tau statistics are computed from — off scratch onto the persistent root, before +the purge or `clean_exposure` can take them. The following plots show an example of a single CCD, in the center of the focal plane. @@ -297,59 +315,39 @@ several runs. ## Galaxy selection -The focus of the next step, -```bash -job_sp TILE_ID -j 16 -``` -is the selection of galaxies as extended objects compared to the PSF. -First, the PSF model is interpolated to galaxy positions, according to the PSF model -with `psfex_interp` or `mccd_interp`. Next, postage stamps around galaxies -of the weights maps are created via `vignetmaker`. Finally, postage stamps -around galaxies of single-exposure data is extracted with another call -to `vignetmaker`. - -The output directory is -- `run_sp_MiViSmVi` if the PSF model is `mccd`; -- `run_sp_tile_PsViSmVi` for the `PSFEx` PSF model. +`tile_vignets` selects galaxies as extended objects compared to the PSF. +First, the PSF model is interpolated to galaxy positions with `psfex_interp`. +Next, postage stamps around galaxies of the weight maps are created via +`vignetmaker`. Finally, postage stamps around galaxies of single-exposure data +are extracted with another call to `vignetmaker` — `Pi`, `Vi`, `Vi`, hence the +config name `config_tile_PiViVi_psfex.ini`. -This corresponds to the MCCD/PSFex interpolation (`Mi`/`Pi`), `vignetmaker` (`Vi`), -and the second call to `vignetmaker` (`Vi`). +`tile_vignets` reads the exposure products through the symlink forest that +`tile_exp_forest` built for this tile, and is the last stage that touches them: +once every consuming tile has its vignets, an exposure store becomes +reclaimable. ## Shape measurement -The call -```bash -job_sp TILE_ID -j 32 -``` -computes galaxy shapes using the multi-epoch model-fitting method `ngmix`. At the same time, -shapes of artifically sheared galaxies are obtained for metacalibration. +`tile_ngmix` computes galaxy shapes using the multi-epoch model-fitting method +`ngmix`. At the same time, shapes of artificially sheared galaxies are obtained +for metacalibration. -Shape measurement is performed in parallel for each tile, the number of processes can be specified -by the user with the option `--nsh_jobs NJOB`. This creates `NJOB` output directories `run_sp_tile_ngmix_Ngu`. -with `X` = 1 ... `NJOB` containing the result of `ngmix`. +Shape measurement is split into parallel chunks: `ngmix_chunks:` in +`workflow/config.yaml` sets how many, and each chunk is its own rule instance +writing its own `run_sp_tile_ngmix_Ngu` output directory. ## Paste catalogues -The last real processing step is -```bash -job_sp TILE_ID -j 64 -``` -This task first merges the `NJOB` parallel `ngmix` output files from the previous step into -one output file. Then, previously obtained information are pasted into a _final_ shape catalogue via `make_cat`. -Included are galaxy detection and basic measurement parameters, the PSF model at -galaxy positions, and the shape measurement. - -Two output directories are created. -The first one is `run_sp_tile_Ms` for the `merge_sep` run. -The second is `run_sp_tile_Mc` for the `make_cat` task; the name is the same for both the `MCCD` and `PSFEx` PSF model. - - -## Upload results - -Optionally, after the pipeline is finished, results can be uploaded to VOspace via -```bash -job_sp TILE_ID -j 128 -``` +`tile_merge_cats` merges the parallel `ngmix` output files from the previous +step into one file with `merge_sep`, in `run_sp_Ms`. +`tile_make_cat` then pastes the previously obtained information into a _final_ +shape catalogue via `make_cat`, in `run_sp_Mc`. Included are galaxy detection +and basic measurement parameters, the PSF model at galaxy positions, and the +shape measurement. The rule publishes that +catalogue onto the persistent root as +`/tiles///final_cat-.fits` — the campaign's +product, and also the marker that says this tile is finished. diff --git a/docs/source/post_processing.md b/docs/source/post_processing.md deleted file mode 100644 index f6af7f8e2..000000000 --- a/docs/source/post_processing.md +++ /dev/null @@ -1,76 +0,0 @@ -# Post-processing - -This page shows all required steps of post-processing the results from one or -more `ShapePipe` runs. Post-processing combines various individual `ShapePipe` -output files, and creates joint results, for example combining individual tile -catalogues into a large sky area. The output of post-processing is a joint _shape -catalogue_, containing all required information to create a calibrated shear -catalogue via _metacalibration_), a joint star catalogue, and PSF diagnostic plots. - - - ---- - -```{note} -This page documents the **legacy** post-processing used for pre-v1.4 runs on the -canfar VM system. PSF validation and the scale-dependent diagnostics -(ρ-statistics) now live in -[`sp_validation`](https://github.com/CosmoStat/sp_validation) rather than in -ShapePipe; the steps below are retained for reference and for reproducing older -runs. -``` - -The following steps were used for pre-v1.4 runs performed on the canfar VM system. - -1. Optional: Split output into sub-samples - - An optional intermediate step is to create directories for sub-samples, for example one directory - for each patch on the sky. This will create symbolic links to the results `.tgz` files downloaded in - the previous step. For example, to create the subdir `tiles_W3` with links to result files to `all` for - those tiles contained in the list `tiles_W3.txt`, do: - ```bash - create_sample_results --input_IDs tiles_W3.txt -i . all -o tiles_W3 -v - ``` - The following steps will then be done in the directory `tiles_W3`. - -2. Run PSF diagnostics, create merged catalogue - - Type - ```bash - post_proc_sp -p PSF - ``` - to automatically perform a number of post-processing steps. Choose the PSF model with the option - `-p psfex|mccd`. In detail, these are (and can also be done individually - by hand): - - 1. Analyse psf validation files - - ```bash - combine_runs -t psf -p PSF - ``` - with options as for `post_proc_sp`. - This script creates a new combined psf run in the ShapePipe `output` directory, by identifying all psf validation files - and creating symbolic links. The run log file is updated. - - 3. Merge the individual PSF validation files into one catalogue, and create - plots of the PSF and its residuals in the focal plane as a diagnostic of - the overall PSF model: - ```bash - shapepipe_run -c /path/to/shapepipe/example/cfis/config_MsPl_PSF.ini - ``` - The scale-dependent PSF diagnostics that propagate to the shear correlation - function (the ρ-statistics, {cite:p}`rowe:10`, {cite:p}`jarvis:16`) are no - longer computed here — they have moved to - [`sp_validation`](https://github.com/CosmoStat/sp_validation). - - 4. Merge final output files - - Create a single main shape catalogue: - ```bash - merge_final_cat -i -p -v - ``` - Choose as input directory `input_dir` the `make_cat` output of the runs - being combined. A default parameter file `` is - `/path/to/shapepipe/workflow/config/cfis/final_cat.param`. - On success, the file `./final_cat.npy` is created. Depending on the number of - input tiles, this file can be several tens of Gb large. diff --git a/docs/source/toc.rst b/docs/source/toc.rst index 01c5a1596..5e4113728 100644 --- a/docs/source/toc.rst +++ b/docs/source/toc.rst @@ -34,14 +34,7 @@ :caption: Running on a cluster clusters - pipeline_canfar - -.. toctree:: - :hidden: - :titlesonly: - :caption: Miscellaneous - - post_processing + workflow .. toctree:: :hidden: diff --git a/docs/source/workflow.md b/docs/source/workflow.md index 624a9819d..196aeeb46 100644 --- a/docs/source/workflow.md +++ b/docs/source/workflow.md @@ -1,24 +1,169 @@ -# Snakemake Workflow +# The Snakemake workflow -The end-to-end CFIS production run is driven by a -[Snakemake](https://snakemake.readthedocs.io) workflow that lives in the -`workflow/` directory of the repository. It replaces the older bit-coded bash -job chain (`job_sp`) that this documentation previously described. +Production ShapePipe runs — a tile list, not a tile — are orchestrated by the +Snakemake workflow in `workflow/`. It is the only orchestration for real data; +the bit-coded bash job scripts (`run_job_sp_canfar_v2.0.bash`, +`job_sp_canfar_v2.0.bash`) remain only as the entry point of sp_validation's +image-simulation chain. -The workflow owns: +**Module code is untouched by it.** Every rule ultimately calls +`shapepipe_run -c ` on the config chains described in +[Configuration](configuration.md); the workflow supplies the scheduling, the +per-unit isolation, the bookkeeping and the reclamation around them. Reading +[Basic execution](basic_execution.md) first is still the right order. -- the per-module ShapePipe configuration files, in `workflow/config/cfis/`, - which is what `$SP_CONFIG` points at; -- the rule graph and job dispatch, in `workflow/Snakefile` and - `workflow/rules/`; -- the run-level settings, in `workflow/config.yaml`. +This page is the orientation. The deep reference — design rationale, every +rule, the profile, the two-roots argument — is +[`workflow/README.md`](https://github.com/CosmoStat/shapepipe/blob/develop/workflow/README.md) +in the repository, and the living design record is +[issue #848](https://github.com/CosmoStat/shapepipe/issues/848). -Full usage — dependencies, profiles, how to launch a run, and how the rules map -onto the ShapePipe modules — is documented in `workflow/README.md` in the -repository. +## Quick start -```{note} -The PSF-validation and post-processing configurations that the workflow does -not yet cover are kept separately in `example/cfis/`; see the README there and -[Post-processing](post_processing.md). +```bash +# One-time, on a SHARED filesystem: the SLURM executor re-invokes this python +# inside every job, so a node-local path (/tmp) will not do. +uv venv /project///snakemake-env --python 3.12 +source /project///snakemake-env/bin/activate +uv pip install 'snakemake>=9,<10' 'snakemake-executor-plugin-slurm>=2.7,<3' + +# Write a run config that sets at least `run:`, the campaign's name; +# workflow/config.yaml's machines: table supplies the rest. + +workflow/bin/sp run -c my_run.yaml # bring products on disk up to date with the tile list +workflow/bin/sp report -c my_run.yaml # the success/failure tables, any time, mid-run is fine +workflow/bin/sp container status # which image the jobs will run inside +workflow/bin/sp cancel # scancel this workflow's jobs ``` + +The run config is merged on top of `workflow/config.yaml`; `SP_PROFILE` and +`input_type:` (`data` or `image_sims`) select the `machines:` entry that fills +in whatever it leaves unset. The README's *Run configuration* section has the +full resolution. + +`workflow/bin/sp` is the entry point for everything. It loads the apptainer +module and the venv, snapshots the code it is about to launch, and sets the +state directory, the SLURM profile and `SP_PHASE`. **Bare `snakemake` outside +`sp` is unsupported** — it skips all of that, including the phase variable the +Snakefile needs to build its index at parse time. Any verb `sp` does not +recognise passes straight through to snakemake with the right profile and state +directory, which is the escape hatch for `sp --unlock`, `sp --dag` or +`sp exp_psf ...`. + +## The two static invocations + +The exposure job set is *derived from data the run itself produces* — a tile's +`find_exposures` output — so it cannot live in the same static DAG that produces +it. `sp run` is therefore two snakemake invocations over one Snakefile: + +1. **PREPARE** (`snakemake prepare_all_tiles`) — the per-tile static chain + `tile_get_images → tile_uncompress → tile_find_exposures`, with `keep-going` + so tile failures stay independent. A nonzero exit is a warning, not a fatal + error: tiles that lost their exposure list are dropped at the compute parse, + and `SP_MISSING_THRESHOLD` is the real gate on how many may be lost. +2. **COMPUTE** (`snakemake all`) — this invocation's *parse* builds the + tile↔exposure index (`run_index.sqlite`) and the DAG then runs the full + exposure and tile chains: + + | level | chain | + | --- | --- | + | exposure | `exp_get_images → exp_split → exp_psf → exp_persist → clean_exposure` | + | tile | `tile_exp_forest → tile_merge_headers → tile_detect → tile_vignets → tile_ngmix → tile_merge_cats → tile_make_cat → clean_tile` | + | campaign | `star_cat_merge`, `final_cat_merge` | + +`sp run` chains both and fails if either phase did. The index **accumulates** +across invocations, so appending tiles to `tile_list` later changes which jobs +exist without invalidating completed work. + +Every rule instance is its own SLURM job (`executor: slurm` in +`profiles/nibi/config.yaml`) carrying that rule's own attempt-scaled resources, +and runs inside the container via snakemake's software-deployment method — the +workflow never calls apptainer itself. + +## Manifests are the currency + +The atom is **one rule == one `shapepipe_run` on one unit** (one tile, or one +exposure), in its own sharded store: `/tiles/<2-char prefix>//` or +`/exp///`. + +A rule's single declared output is not its product files but that unit's +**manifest**, `/manifests/.json`. Per-CCD declaration would +mean millions of paths at DR6 scale, and a missing CCD is frequently +legitimate. + +Manifests are *success-only*. After each run, `completeness.py check` writes its +full verdict — per-runner counts against floors, scraped failure reasons, the +`shapepipe_run` exit status — to the rule's `log:` +(`/logs/.json`) **every** time, and additionally to the +declared manifest **only** when that verdict is a success. So a manifest on disk +means "this stage succeeded", and a resume after an unclean death can never +schedule downstream work on top of a failure. + +Completeness is a **count floor, not a taxonomy**: a stage is a real failure iff +a mandatory runner produced fewer products than its floor. Per-CCD attrition +between the floor and the expected count is tolerated and recorded. There is no +error-signature whitelist. + +## `sp report` + +`sp report` reads the index and those two directories — the manifest for +success, the log for failure, a unit with neither ran nothing — and prints the +per-stage tables plus a `run_report.json` beside the index. It is deliberately +**not** a DAG node: a report rule declaring every tile's output would be a +descendant of every job, so one hard failure under `--keep-going` would poison +its cone and the report would never run in exactly the situation it exists for. +It is a plain script, runnable at any time including mid-run, and the COMPUTE +invocation's `onsuccess`/`onerror` hooks fire it automatically so every run ends +with one. + +## `sp container` + +`sp container` owns which image the jobs run inside, in one resolution order +shared by the CLI and the workflow: **sandbox → your cached SIF → the +machine's `container:` in `config.yaml`'s `machines:` table**. + +```bash +sp container status # layers present, active one, revision vs HEAD +sp container pull # ghcr.io/cosmostat/shapepipe:develop +sp container sandbox # unpack the SIF writable (opt-in) +sp container exec --writable pip install +sp container resolve # just the path the workflow will run +``` + +The cached SIF is a private pull, so nobody else's refresh moves the ground +under your running jobs; the sandbox is the escape hatch for work needing a +package the image does not carry yet. `status` places the image's +`org.opencontainers.image.revision` label against this checkout's HEAD. +`pull` needs the network — run it on a login node, never from a batch job. + +See [Container workflow](container.md) for how the images are built. + +## Durable products + +Two roots. `run_dir` is scratch: the bulk per-unit stores, sized to finish +inside the purge window. `products_dir` is persistent and holds the low-volume +things worth keeping: + +- **Final catalogues** — `/tiles///final_cat-.fits`, + mirroring the scratch shard structure. This file is also the *tile-finished + marker* that lets a completed tile stop pinning its exposures, which is the + second reason it cannot live on scratch. +- **Merged catalogues** — `/final_cat_.hdf5` (every tile's + final catalogue, one dataset per tile) and `full_starcat_.hdf5` (every + exposure's PSF validation catalogue, the rho/tau statistics input), named by + the run config's `run:` and reconciled rather than rebuilt when tiles are + appended. +- **PSF products** — `exp_persist` packs each exposure's psfex_interp + `validation_psf-*.fits` (always) and the products named by `persist_exp:` in + `config.yaml` (by default `psf_model`) into one uncompressed tar per exposure, + plus a manifest recording the members. It answers the scratch purge, not the + workflow's own reclamation, so it runs whether or not `clean:` is on. Under + `psf_model: fake` there is no PSF product and it does not run. +- **The index and the report** — `run_index.sqlite`, `missing.json`, + `run_report.json`. + +Exposure stores are reclaimed by the in-DAG `clean_exposure` rule rather than by +`temp()`: exposures overlap tiles, so `temp()` would cascade destructive reruns +whenever a tile is appended. Reclamation leaves a `cleaned.json` tombstone +holding the absorbed manifests, so a reclaimed exposure still reports its +per-runner counts and blocks no tile. diff --git a/example/cfis/config_valjoint_Pl_mccd.ini b/example/cfis/config_valjoint_Pl_mccd.ini index 086690815..81c444b55 100644 --- a/example/cfis/config_valjoint_Pl_mccd.ini +++ b/example/cfis/config_valjoint_Pl_mccd.ini @@ -39,7 +39,7 @@ INPUT_DIR = ./SP OUTPUT_DIR = ./output # FILE_PATTERN (opional) list of string patterns to identify input files -FILE_PATTERN = unions_shapepipe_psf_2022_v1.0.2 +FILE_PATTERN = unions_shapepipe_psf_2022 # FILE_EXT (opional) list of string extensions to identify input files FILE_EXT = .fits diff --git a/example/cfis_image_sims/config_tile_PiViVi_canfar_sx.ini b/example/cfis_image_sims/config_tile_PiViVi_canfar_sx.ini index 26564e4fa..5658df35d 100644 --- a/example/cfis_image_sims/config_tile_PiViVi_canfar_sx.ini +++ b/example/cfis_image_sims/config_tile_PiViVi_canfar_sx.ini @@ -120,8 +120,8 @@ STAMP_SIZE = 51 PREFIX = # Additional parameters for path and file pattern corresponding to single-exposure -# run outputs. ME_IMAGE_EXP_DIR/ME_IMAGE_EXP_RUNNERS replace ME_IMAGE_DIR for -# the v2.0 per-exposure pipeline; output dirs are discovered by scanning $SP_EXP. +# run outputs. Output dirs are discovered by scanning $SP_EXP for the +# exposures listed in the exp_numbers input file. ME_IMAGE_EXP_DIR = $SP_EXP ME_IMAGE_EXP_RUNNERS = split_exp_runner, split_exp_runner, split_exp_runner ME_IMAGE_PATTERN = flag, image, weight diff --git a/pyproject.toml b/pyproject.toml index 53f0dae61..edf81bafb 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -37,7 +37,6 @@ dependencies = [ "python-pysap>=0.3", "PyQt5", "pyqtgraph", - "skaha>=1.7", "skyproj", "sqlitedict>=2.0", "termcolor", @@ -80,11 +79,8 @@ dev = ["shapepipe[doc,jupyter,lint,plot,release,test,fitsio]"] [project.scripts] shapepipe_run = "shapepipe.shapepipe_run:main" -summary_run = "shapepipe.summary_run:main" -canfar_submit_job = "shapepipe.canfar_run:run_job" canfar_monitor = "shapepipe.canfar_run:run_log" canfar_monitor_log = "shapepipe.canfar_run:run_monitor_log" -get_ccds_with_psf = "shapepipe.get_ccds_run:run_ccd_psf_handler" download_headers = "shapepipe.coverage_run:run_download_headers" extract_field_corners = "shapepipe.coverage_run:run_extract_corners" build_coverage_map = "shapepipe.coverage_run:run_build_coverage" diff --git a/scripts/README.rst b/scripts/README.rst index 2b8067cb0..482aaea78 100644 --- a/scripts/README.rst +++ b/scripts/README.rst @@ -1,19 +1,77 @@ Scripts directory ================= -This directory contain scripts that are used on pipeline outputs or to prepare -files. They are not run throught the pipeline framework. For more details on how -to run each them see below. +Helpers that operate on pipeline inputs and outputs but are **not** pipeline +modules: nothing here is run by ``shapepipe_run``, and nothing here is part of +the Snakemake workflow's rule graph (that is ``workflow/scripts/``). -Python scripts -============== +Everything under ``scripts/*/`` with a ``.py``, ``.sh`` or ``.bash`` extension is +symlinked onto ``$PATH`` inside the container image under its bare name, so +``cfis_field_select`` and ``python scripts/python/cfis_field_select.py`` are the +same command. -1. `create_log_exp_headers`_ +Each script's own ``--help`` and module docstring is the reference; this file is +only a map. Run any of them with ``-h`` first. -create_log_exp_headers -====================== +``python/`` — pipeline inputs +----------------------------- -This as to run after the module `split_exp_runner` it will create a master log -file containing all the WCS information for each CCDs of each single exposures. -To run the script : -`python create_log_exp_headers.py path/to/split_exp_runner/output path/to/srcipt/output_dir` +- ``cfis_field_select.py`` — select CFIS tiles or exposures by sky coordinates + or by name, and emit the ID list. This is how a tile list for + ``workflow/config.yaml`` is made. +- ``check_tile_coverage.py`` — flag input simulation tiles whose weight maps are + mostly empty, as a YAML exclude list. Called from sp_validation's image-sims + orchestration, not from here. + +``python/`` — pipeline outputs +------------------------------ + +- ``collate_star_cat.py`` — collate the per-exposure PSF validation catalogues + into the input ``merge_starcat`` wants for the rho/tau statistics. +- ``create_final_cat.py`` — merge per-tile final catalogues into one HDF5 file + for sp_validation. +- ``stats_global.py`` — histograms and tables from the SETools per-CCD summary + statistics of a whole run. + +``python/`` — image simulations +------------------------------- + +The image-simulation chain is still driven by the pre-Snakemake bash layer, +which sp_validation's ``image_sims.smk`` calls into. These exist for it and are +not used by real-data runs. + +- ``init_run_v2.0.py`` — lay out a run directory for that layer. +- ``update_runs_log_file.py`` — rebuild ``log_run_sp.txt`` from the run + directories on disk, which that layer needs because it deletes run dirs. +- ``test_tile_det.py`` + ``test_tile_det.cfg`` — drive tile detection over it. + +``python/`` — the ngmix status dashboard +---------------------------------------- + +Orthogonal to everything above: ``build_status.py``, ``build_history.py``, +``plot_trends.py``, ``run_breakdown_grid.py``, ``plot_breakdown_grid.py`` and +``plot_s4_ablations.py`` build the shape-measurement status page and its +trend/ablation figures. + +``sh/`` +------- + +- ``shapepipe_run_example.sh`` — run the bundled example pipeline against a + writable copy of ``/app/example``. This is the container's CI smoke test. +- ``apptainer_noslurm.sh`` — ``apptainer exec`` with the SLURM/PMI/PMIX/OMPI + environment stripped, for MPI inside a SLURM job. +- ``run_job_sp_canfar_v2.0.bash``, ``job_sp_canfar_v2.0.bash``, + ``job_list_help.bash``, ``functions.sh`` — the bit-coded bash job layer. Kept + only for image simulations (above); real-data runs go through ``workflow/``. + +``jupyter/`` +------------ + +Scratch notebooks: ``wcs.ipynb``, ``test_centroid_shift.py``. + +``validation/`` +--------------- + +``centroid/`` — the metacalibration multiplicative-bias validation of the ngmix +centroid handling: ``run_all.sh`` drives ``centroid_bias_v2.py`` over the three +cases in ``configs/``. diff --git a/scripts/jupyter/summary_run.ipynb b/scripts/jupyter/summary_run.ipynb deleted file mode 100644 index 44f2f2545..000000000 --- a/scripts/jupyter/summary_run.ipynb +++ /dev/null @@ -1,399 +0,0 @@ -{ - "cells": [ - { - "cell_type": "code", - "execution_count": null, - "id": "130112a4-f2ca-4d26-b884-d8b054676f9c", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "%load_ext autoreload\n", - "%autoreload 3\n", - "%reload_ext autoreload" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "id": "196beca5-10a1-4cf5-9462-be145167cc70", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "ename": "ModuleNotFoundError", - "evalue": "No module named 'shapepipe'", - "output_type": "error", - "traceback": [ - "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", - "\u001b[0;31mModuleNotFoundError\u001b[0m Traceback (most recent call last)", - "Cell \u001b[0;32mIn[1], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;21;01mshapepipe\u001b[39;00m\n\u001b[1;32m 2\u001b[0m \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01mshapepipe\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01mutilities\u001b[39;00m\u001b[38;5;21;01m.\u001b[39;00m\u001b[38;5;21;01msummary\u001b[39;00m \u001b[38;5;28;01mimport\u001b[39;00m \u001b[38;5;241m*\u001b[39m\n", - "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'shapepipe'" - ] - } - ], - "source": [ - "import shapepipe\n", - "from shapepipe.utilities.summary import *" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "ca63c72d-212c-463e-a792-71efbac0b908", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "# Setting\n", - "patch = \"P7\"\n", - "\n", - "verbose = False" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "id": "dcb5604c-d61f-4705-8295-63875455cadb", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "# Load parameters\n", - "%run ~/shapepipe/scripts/python/summary_params_pre_v2" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "id": "e69b7dab-1fea-4fcc-a8d9-0720e1d628c3", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "Checking main directory = /arc/home/kilbinger/cosmostat/v2/pre_v2/psfex/P7\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Set job info for patch P7\n" - ] - } - ], - "source": [ - "jobs, list_tile_IDs = set_jobs_v2_pre_v2(patch, verbose)" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "id": "2c3a9dde-cf88-493f-926e-7ae7e8e10916", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "# Initialize runtime dicionary.\n", - "par_runtime = init_par_runtime(list_tile_IDs)" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "id": "1e9c1487-3cec-4394-9fcf-c12e92a0f984", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "# No effect in notebook\n", - "#print_par_runtime(par_runtime, verbose=verbose)" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "id": "b7c63a22-ead1-4d6a-b081-a74ade515439", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "module expected found miss_expl missing uniq_miss fr_found\n", - "====================================================================================================\n" - ] - } - ], - "source": [ - "# Start program\n", - "job_data.print_stats_header()" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "id": "4720ae18-0633-4646-b392-b1b24e0294c3", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - " (Job 1)\n", - "get_images_runner_run_1 462 462 0 0 0.0 100.0%\n", - "find_exposures_runner 231 231 0 0 0.0 100.0%\n", - "get_images_runner_run_2 537 0 0 537 179.0 0.0%\n" - ] - } - ], - "source": [ - "for key in \"1\":\n", - " jobs[key].print_intro()\n", - " jobs[key].check_numbers(par_runtime=par_runtime, indices=[0, 1])\n", - "\n", - " all_exposures = get_all_exposures(jobs[key]._paths_in_dir[1], verbose=verbose)\n", - " par_runtime[\"n_exposures\"] = len(all_exposures)\n", - " par_runtime[\"list_exposures\"] = all_exposures\n", - "\n", - " jobs[key].check_numbers(par_runtime, indices=[2])" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "id": "f149f404-64e7-4d92-8f54-f300ed620130", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "# Update runtime dictionary with numbers of exposures\n", - "par_runtime = update_par_runtime_after_find_exp(par_runtime, all_exposures)" - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "id": "78a9065f-8983-41cf-a34c-21892fc52dd2", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "# Get all keys after \"1\"\n", - "keys = sorted(jobs.keys(), key=int)\n", - "_ = keys.pop(0)" - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "id": "79e39954-1155-4ca3-b0b2-64bc5670db53", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - " (Job 2)\n", - "uncompress_fits_runner 1268 1268 0 0 0.0 100.0%\n", - "merge_headers_runner 0 0 0 0 0.0 100.0%\n", - "split_exp_runner 137940 137940 0 0 0.0 100.0%\n", - " (Job 4)\n", - "mask_runner 1268 1268 0 0 0.0 100.0%\n", - " (Job 8)\n", - "mask_runner 45600 45600 0 0 0.0 100.0%\n", - " (Job 16)\n", - "sextractor_runner 2536 2536 0 0 0.0 100.0%\n", - " (Job 32)\n", - "sextractor_runner 91200 91200 0 0 0.0 100.0%\n", - "setools_runner 91200 91032 0 168 84.0 99.8%\n", - "psfex_runner 91200 91032 0 168 84.0 99.8%\n", - " (Job 64)\n", - "psfex_interp_runner 1268 1268 0 0 0.0 100.0%\n", - "vignetmaker_runner_run_1 1268 1268 0 0 0.0 100.0%\n", - "spread_model_runner 1268 1268 0 0 0.0 100.0%\n", - "vignetmaker_runner_run_2 5072 5072 0 0 0.0 100.0%\n", - " (Job 128)\n", - "ngmix_runner 1268 1225 0 43 43.0 96.6%\n", - "ngmix_runner 1268 1216 0 52 52.0 95.9%\n", - "ngmix_runner 1268 1216 0 52 52.0 95.9%\n", - "ngmix_runner 1268 1217 0 51 51.0 96.0%\n", - "ngmix_runner 1268 1228 0 40 40.0 96.8%\n", - "ngmix_runner 1268 1216 0 52 52.0 95.9%\n", - "ngmix_runner 1268 1216 0 52 52.0 95.9%\n", - "ngmix_runner 1268 1216 0 52 52.0 95.9%\n", - " (Job 256)\n", - "merge_sep_cats_runner 1268 0 0 1268 1268.0 0.0%\n", - "make_cat_runner 1268 0 0 1268 1268.0 0.0%\n", - " (Job 1024)\n", - "psfex_interp_runner 45600 41132 0 4468 4468.0 90.2%\n" - ] - } - ], - "source": [ - "for key in keys:\n", - " jobs[key].print_intro()\n", - " jobs[key].check_numbers(par_runtime=par_runtime)" - ] - }, - { - "cell_type": "code", - "execution_count": 65, - "id": "b3d51a05-ecca-420b-b8b3-1fb2b1ec9fe3", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - " (Job 128)\n", - "ngmix_runner 1268 1267 0 1 1.0 99.9%\n", - "ngmix_runner 1268 1265 0 3 3.0 99.8%\n", - "ngmix_runner 1268 1267 0 1 1.0 99.9%\n", - "ngmix_runner 1268 1267 0 1 1.0 99.9%\n", - "ngmix_runner 1268 1267 0 1 1.0 99.9%\n", - "ngmix_runner 1268 1266 0 2 2.0 99.8%\n", - "ngmix_runner 1268 1268 0 0 0.0 100.0%\n", - "ngmix_runner 1268 1266 0 2 2.0 99.8%\n" - ] - } - ], - "source": [ - "## Update some runs\n", - "for key in [\"128\"]:\n", - " jobs[key].print_intro()\n", - " jobs[key].check_numbers(par_runtime=par_runtime)" - ] - }, - { - "cell_type": "code", - "execution_count": 64, - "id": "67b50a61-e3cc-4559-941d-f39c6a200294", - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - " (Job 128)\n", - "ngmix_runner 1268 1267 0 1 1.0 99.9%\n", - "ngmix_runner 1268 1265 0 3 3.0 99.8%\n", - "ngmix_runner 1268 1267 0 1 1.0 99.9%\n", - "ngmix_runner 1268 1267 0 1 1.0 99.9%\n", - "ngmix_runner 1268 1267 0 1 1.0 99.9%\n", - "ngmix_runner 1268 1266 0 2 2.0 99.8%\n", - "ngmix_runner 1268 1268 0 0 0.0 100.0%\n", - "ngmix_runner 1268 1266 0 2 2.0 99.8%\n" - ] - } - ], - "source": [ - "for key in [\"128\"]:\n", - " jobs[key].print_intro()\n", - " \n", - " \n", - " jobs[key].check_numbers(par_runtime=par_runtime)" - ] - }, - { - "cell_type": "code", - "execution_count": 19, - "id": "affa8293-daf9-4d2b-9215-fe19f8e2c1e2", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "session = Session()" - ] - }, - { - "cell_type": "code", - "execution_count": 36, - "id": "269496d1-cd89-4d13-a5e4-41b897669e22", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "ids = [session[\"id\"] for session in session.fetch(kind=\"headless\")]" - ] - }, - { - "cell_type": "code", - "execution_count": 38, - "id": "80af8dff-98c7-4db4-8bcc-06936e1875cf", - "metadata": { - "tags": [] - }, - "outputs": [ - { - "ename": "RuntimeError", - "evalue": "This event loop is already running", - "output_type": "error", - "traceback": [ - "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m", - "\u001b[0;31mRuntimeError\u001b[0m Traceback (most recent call last)", - "\u001b[0;32m/tmp/ipykernel_69/559116804.py\u001b[0m in \u001b[0;36m\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0msession\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdestroy\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mids\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m", - "\u001b[0;32m~/.local/lib/python3.7/site-packages/skaha/session.py\u001b[0m in \u001b[0;36mdestroy\u001b[0;34m(self, id)\u001b[0m\n\u001b[1;32m 264\u001b[0m \u001b[0marguments\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m{\u001b[0m\u001b[0;34m\"url\"\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mserver\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0;34m\"/\"\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mvalue\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 265\u001b[0m \u001b[0mloop\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mget_event_loop\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 266\u001b[0;31m \u001b[0mresults\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mloop\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_until_complete\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mscale\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msession\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdelete\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0marguments\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 267\u001b[0m \u001b[0mresponses\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mDict\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbool\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 268\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mindex\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0midentity\u001b[0m \u001b[0;32min\u001b[0m \u001b[0menumerate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mid\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/opt/conda/lib/python3.7/asyncio/base_events.py\u001b[0m in \u001b[0;36mrun_until_complete\u001b[0;34m(self, future)\u001b[0m\n\u001b[1;32m 561\u001b[0m \"\"\"\n\u001b[1;32m 562\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_check_closed\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 563\u001b[0;31m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_check_runnung\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 564\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 565\u001b[0m \u001b[0mnew_task\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mfutures\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0misfuture\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfuture\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n", - "\u001b[0;32m/opt/conda/lib/python3.7/asyncio/base_events.py\u001b[0m in \u001b[0;36m_check_runnung\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m 521\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_check_runnung\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 522\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mis_running\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 523\u001b[0;31m \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'This event loop is already running'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m 524\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mevents\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_get_running_loop\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m 525\u001b[0m raise RuntimeError(\n", - "\u001b[0;31mRuntimeError\u001b[0m: This event loop is already running" - ] - } - ], - "source": [ - "session.destroy(ids[0])" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "66a3ed14-8aaf-4028-b933-10ecb7376d68", - "metadata": {}, - "outputs": [], - "source": [] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.11.9" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/scripts/python/clear_ngmix_prev.py b/scripts/python/clear_ngmix_prev.py deleted file mode 100644 index a1a7e802c..000000000 --- a/scripts/python/clear_ngmix_prev.py +++ /dev/null @@ -1,155 +0,0 @@ -#!/usr/bin/env python3 -""" -Clean up previous ngmix run directories for completed tiles. -Removes *_prev directories when the current run has finished. -""" - -import argparse -import os -import re -import shutil -import glob -from pathlib import Path -from astropy.io import fits - - -def get_fits_info(fits_file): - """Get number of objects (NAXIS2 in HDU 1) and number of HDUs.""" - try: - with fits.open(fits_file) as hdul: - n_hdus = len(hdul) - # Get NAXIS2 from HDU 1 (second HDU, index 1) - if len(hdul) > 1: - n_objects = hdul[1].header.get("NAXIS2", "N/A") - else: - n_objects = "N/A (no HDU 1)" - return n_objects, n_hdus - except Exception as e: - return f"Error: {e}", "N/A" - - -def check_finished(log_file): - """Check if 'finished' appears in the log file.""" - try: - with open(log_file, "r") as f: - content = f.read() - return "finished" in content - except Exception as e: - print(f"Warning: Could not read {log_file}: {e}") - return False - - -def main(): - parser = argparse.ArgumentParser( - description="Clean up previous ngmix run directories for completed tiles." - ) - parser.add_argument( - "-n", "--dry-run", - action="store_true", - help="Show what would be removed without actually removing files" - ) - args = parser.parse_args() - - if args.dry_run: - print("dry run mode: No files will be removed\n") - - base_dir = Path("tile_runs") - - # Pattern to match tile IDs like 123.456 - tile_id_pattern = re.compile(r"^\d+\.\d+$") - - # Find all tile ID directories - if not base_dir.exists(): - print(f"Error: {base_dir} does not exist") - return - - tile_dirs = [ - d - for d in base_dir.iterdir() - if d.is_dir() and tile_id_pattern.match(d.name) - ] - - print(f"Found {len(tile_dirs)} tile directories") - - removed_count = 0 - finished_count = 0 - - for tile_dir in sorted(tile_dirs): - tile_id = tile_dir.name - - # Construct path to ngmix_runner directory - ngmix_dir = ( - tile_dir / "output" / "run_sp_tile_ngmix_Ng1u" / "ngmix_runner" - ) - - if not ngmix_dir.exists(): - continue - - # Check for process-*.log files - log_pattern = str(ngmix_dir / "logs" / "process-*.log") - log_files = glob.glob(log_pattern) - - if not log_files: - continue - - # Check if any log file contains "finished" - is_finished = any(check_finished(log_file) for log_file in log_files) - - if is_finished: - finished_count += 1 - - # Path to the _prev directory to remove - prev_dir = tile_dir / "output" / "run_sp_tile_ngmix_Ng1u_prev" - - if prev_dir.exists(): - - # File info of new - ngmix_out_dir = ngmix_dir / "output" - fits_files = list(ngmix_out_dir.glob("ngmix-*.fits")) - if fits_files: - for fits_file in fits_files: - n_objects, n_hdus = get_fits_info(fits_file) - else: - n_objects, n_hdus = (-1, -1) - - # File info of prev - prev_out_dir = prev_dir / "output" - fits_files = list(prev_out_dir.glob("ngmix-*.fits")) - if fits_files: - for fits_file in fits_files: - n_objects_prev, n_hdus_prev = get_fits_info(fits_file) - else: - n_objects_prev, n_hdus_prev = (-1, -1) - - action = "Would remove" if args.dry_run else "Removing" - print( - f"Tile {tile_id}: {action} {prev_dir} (new: {n_objects}" - + f" {n_hdus}) (prev: {n_objects_prev} {n_hdus_prev})" - ) - - if n_objects < n_objects_prev: - print(f"Warning: {tile_id}") - - if not args.dry_run: - try: - shutil.rmtree(prev_dir) - removed_count += 1 - except Exception as e: - print(f"Error removing {prev_dir}: {e}") - else: - removed_count += 1 - else: - pass - else: - print(f"Tile {tile_id}: Not finished") - - print(f"\nSummary:") - print(f" Tiles with finished runs: {finished_count}") - if args.dry_run: - print(f" Previous directories that would be removed: {removed_count}") - else: - print(f" Previous directories removed: {removed_count}") - - -if __name__ == "__main__": - main() diff --git a/scripts/python/collate_star_cat.py b/scripts/python/collate_star_cat.py index f7ab25b41..a3ccc2a2e 100755 --- a/scripts/python/collate_star_cat.py +++ b/scripts/python/collate_star_cat.py @@ -2,29 +2,25 @@ """COLLATE STAR CATALOGUES. -Collate the per-exposure PSF validation catalogues into star catalogues: gather -positions (X/Y/RA/DEC), assign the MCCD focal-plane CCD id, and merge into -``validation_psf_conv`` FITS files. - -Catalogue layout depends on the major version (``-V``). Runs up to ``v1.6`` are -organised in sky patches (``P1``..``P``): input runs live under -``/P/output`` and outputs are named -``validation_psf_conv--.fits``. ``v2.0`` removes the patch concept: -input runs live under a single ``/output`` root and outputs drop the patch -token (``validation_psf_conv-.fits``). - -HSM ellipticities and sizes are no longer rotated here. PSFEx and the in-repo -MCCD interpolation now measure adaptive moments directly in world coordinates -(galsim ``FindAdaptiveMom(use_sky_coords=True)``), so the WCS-Jacobian shape -rotation this script used to perform is redundant and has been removed. - -Caveat: the MCCD ``PSF_MOM_LIST``/``STAR_MOM_LIST`` columns are produced by the -external ``mccd`` fit-validation code (``mccd.auxiliary_fun.mccd_validation``), -which still measures HSM moments in the pixel frame. Those shapes are therefore -still rotated into world coordinates here, via the WCS-Jacobian rotation, until -``mccd`` itself adopts ``use_sky_coords``. This is the one branch that keeps the -rotation; the in-repo PSFEx and MCCD-interpolation paths measure adaptive -moments directly in world coordinates upstream and pass them straight through. +Collate the per-exposure PSFEx validation catalogues into star catalogues: +gather positions (X/Y/RA/DEC) and the HSM PSF and star shapes, tag each row +with its CCD number, and merge one ``validation_psf_conv-.fits`` per +single-exposure run. + +HSM ellipticities and sizes are not rotated here. ``psfex_interp`` measures +adaptive moments directly in world coordinates (galsim +``FindAdaptiveMom(use_sky_coords=True)``), so the WCS-Jacobian shape rotation +this script used to perform is redundant and has been removed; only positions +are collated. + +Caveat -- no workflow rule produces these catalogues yet. The input layout +below is the pre-workflow one: a link farm of ``psfex_interp`` run directories +under a single input root. The Snakemake workflow instead persists +``validation_psf--.fits`` into the per-exposure ``exp_persist`` +tars on the products root, and ``merge_starcat_runner`` still wants the +collated ``validation_psf_conv-*`` files for the rho/tau statistics. +Repointing this script at the persist tars (or replacing it with a rule) is a +follow-up, and a bigger change than this strip. """ import sys @@ -37,51 +33,27 @@ import numpy as np from astropy.io import fits -import galsim from cs_util import args as cs_args from cs_util import logging -def collate_paths(input_base_dir, output_base_dir, patch): - """Collate Paths. +# Input layout: single-exposure run directories matching ``EXP_RUN_GLOB`` under +# ``/output``, each holding the PSF interpolation output in +# ``PSF_INTERP_SUBDIR``. +EXP_RUN_GLOB = "run_sp_combined_psf*" +PSF_INTERP_SUBDIR = "psfex_interp_runner/output" - Return the ``(input run dir, output dir)`` for a patch. ``patch`` is None - for the patch-less v2.0 layout, which drops the ``P`` token; v1.x - passes the patch number. - Parameters - ---------- - input_base_dir : str - input base directory - output_base_dir : str - output base directory - patch : str or None - patch number, or None for the patch-less v2.0 layout - - Returns - ------- - tuple - input run directory and output directory - - """ - if patch is None: - return f"{input_base_dir}/output/", output_base_dir - return f"{input_base_dir}/P{patch}/output/", f"{output_base_dir}/P{patch}" - - -def output_filename(file_pattern, patch, idx): +def output_filename(file_pattern, idx): """Output Filename. - Build the collated catalogue filename. ``patch`` is None for the patch-less - v2.0 layout, which drops the patch token. + Build the collated catalogue filename. Parameters ---------- file_pattern : str input file pattern (e.g. ``validation_psf``) - patch : str or None - patch number, or None for the patch-less v2.0 layout idx : int exposure run index @@ -91,340 +63,7 @@ def output_filename(file_pattern, patch, idx): output catalogue file name """ - patch_token = "" if patch is None else f"{patch}-" - return f"{file_pattern}_conv-{patch_token}{idx}.fits" - - -def transform_shape(mom_list, jac): - """Transform Shape. - - Transform shape (ellipticity and size) using a Jacobian. - - Parameters - ---------- - mom_list : list - input moment measurements; each list element contains - first and second ellipticity component and size - jac : galsim.JacobianWCS - Jacobian transformation matrix information - - Returns - ------- - list - transformed shape parameters, which are - first and second ellipticity component and size - - """ - scale, shear, theta, flip = jac.getDecomposition() - - sig_tmp = mom_list[2] * scale - shape = galsim.Shear(g1=mom_list[0], g2=mom_list[1]) - if flip: - # The following output is not observed - print("FLIP!") - shape = galsim.Shear(g1=-shape.g1, g2=shape.g2) - shape = galsim.Shear(g=shape.g, beta=shape.beta + theta) - shape = shear + shape - - return shape.g1, shape.g2, sig_tmp - - -class Loc2Glob(object): - r"""Change from local to global coordinates. - - Class to pass from local coordinates to global coordinates under - CFIS (CFHT) MegaCam instrument. The geometrical informcation of the - instrument is encoded in this function. - - Parameters - ---------- - x_gap : int - Gap between the CCDs along the horizontal direction; - default is ``70`` (MegaCam value) - y_gap : int - Gap between the CCDs along the vertical direction; - Default is ``425`` (MegaCam value) - x_npix : int - Number of pixels per CCD along the horizontal direction; - default is ``2048`` (MegaCam value) - y_npix : int - Number of pixels per CCD along the vertical direction; - default to ``4612`` (MegaCam value) - ccd_tot : int - Total number of CCDs; - default to ``40`` (MegaCam value) - - Notes - ----- - This is the geometry of MegaCam. Watch out with the conventions ba,ab that means where - is the local coordinate system origin for each CCD. - For more info check out MegaCam's instrument webpage. - - Examples - -------- - 'COMMENT (North on top, East to the left)', - 'COMMENT --------------------------', - 'COMMENT ba ba ba ba ba ba ba ba ba', - 'COMMENT 00 01 02 03 04 05 06 07 08', - 'COMMENT --------------------------------', - 'COMMENT ba ba ba ba ba ba ba ba ba ba ba', - 'COMMENT 36 09 10 11 12 13 14 15 16 17 37', - 'COMMENT --------------*-----------------', - 'COMMENT 38 18 19 20 21 22 23 24 25 26 39', - 'COMMENT ab ab ab ab ab ab ab ab ab ab ab', - 'COMMENT --------------------------------', - 'COMMENT 27 28 29 30 31 32 33 34 35', - 'COMMENT ab ab ab ab ab ab ab ab ab', - 'COMMENT __________________________' - """ - - def __init__( - self, x_gap=70, y_gap=425, x_npix=2048, y_npix=4612, ccd_tot=40 - ): - r"""Initialize with instrument geometry.""" - self.x_gap = x_gap - self.y_gap = y_gap - self.x_npix = x_npix - self.y_npix = y_npix - self.ccd_tot = ccd_tot - - def loc2glob_img_coord(self, ccd_n, x_coor, y_coor): - """loc2glob Img Coord. - - Go from the local to the global img (pixel) coordinate system. - - Global system with (0,0) in the intersection of ccds [12,13,21,22]. - - Parameters - ---------- - ccd_n: int - CCD number of the considered positions - x_coor: float - Local coordinate system hotizontal value - y_coor: float - Local coordinate system vertical value - - Returns - ------- - glob_x_coor: float - Horizontal position in global coordinate system - glob_y_coor: float - Vertical position in global coordinate system - - """ - # Flip axes - x_coor, y_coor = self.flip_coord(ccd_n, x_coor, y_coor) - - # Calculate the shift - x_shift, y_shift = self.shift_coord(ccd_n) - - # Return new coordinates - return x_coor + x_shift, y_coor + y_shift - - def flip_coord(self, ccd_n, x_coor, y_coor): - r"""Change of coordinate convention. - - So that all of them are coherent on the global coordinate system. - So that the origin is on the south-west corner. - Positive: South to North ; West to East. - """ - if ccd_n < 18 or ccd_n in [36, 37]: - x_coor = self.x_npix - x_coor + 1 - y_coor = self.y_npix - y_coor + 1 - else: - pass - - return x_coor, y_coor - - def x_coord_range(self): - r"""Return range of the x coordinate.""" - max_x = self.x_npix * 6 + self.x_gap * 5 - min_x = self.x_npix * (-5) + self.x_gap * (-5) - return min_x, max_x - - def y_coord_range(self): - r"""Return range of the y coordinate.""" - max_y = self.y_npix * 2 + self.y_gap * 1 - min_y = self.y_npix * (-2) + self.y_gap * (-2) - return min_y, max_y - - def shift_coord(self, ccd_n): - r"""Provide the shifting. - - It is needed to go from the local coordinate - system origin to the global coordinate system origin. - """ - if ccd_n < 9: - # first row - x_shift = (ccd_n - 4) * (self.x_gap + self.x_npix) - y_shift = self.y_gap + self.y_npix - return x_shift, y_shift - - elif ccd_n < 18: - # second row, non-ears - x_shift = (ccd_n - 13) * (self.x_gap + self.x_npix) - y_shift = 0.0 - return x_shift, y_shift - - elif ccd_n < 27: - # third row non-ears - x_shift = (ccd_n - 22) * (self.x_gap + self.x_npix) - y_shift = -1.0 * (self.y_gap + self.y_npix) - return x_shift, y_shift - - elif ccd_n < 36: - # fourth row - x_shift = (ccd_n - 31) * (self.x_gap + self.x_npix) - y_shift = -2.0 * (self.y_gap + self.y_npix) - return x_shift, y_shift - - elif ccd_n < 37: - # ccd= 36 ears, second row - x_shift = (-5.0) * (self.x_gap + self.x_npix) - y_shift = 0.0 - return x_shift, y_shift - - elif ccd_n < 38: - # ccd= 37 ears, second row - x_shift = 5.0 * (self.x_gap + self.x_npix) - y_shift = 0.0 - return x_shift, y_shift - - elif ccd_n < 39: - # ccd= 38 ears, third row - x_shift = (-5.0) * (self.x_gap + self.x_npix) - y_shift = -1.0 * (self.y_gap + self.y_npix) - return x_shift, y_shift - - elif ccd_n < 40: - # ccd= 39 ears, third row - x_shift = 5.0 * (self.x_gap + self.x_npix) - y_shift = -1.0 * (self.y_gap + self.y_npix) - return x_shift, y_shift - - -class Glob2CCD(object): - r"""Get the CCD ID number from the global coordinate position. - - The Loc2Glob() object as input is the one that defines the instrument's - geometry. - - Parameters - ---------- - loc2glob: Loc2Glob object - Object with the desired focal plane geometry. - with_gaps: bool - If add the gaps to the CCD area. - """ - - def __init__(self, loc2glob, with_gaps=True): - # Save loc2glob object - self.loc2glob = loc2glob - self.with_gaps = with_gaps - self.ccd_list = np.arange(self.loc2glob.ccd_tot) - # Init edges defininf the CCDs - self.edge_x_list, self.edge_y_list = self.build_all_edges() - - def build_all_edges(self): - """Build the edges for all the CCDs in the focal plane.""" - edge_xy_list = [] - for idx in (0, 1): - edge_list = np.array( - [self.build_edge(ccd_n)[idx] for ccd_n in self.ccd_list] - ) - edge_xy_list.append(edge_list) - - return edge_xy_list - - def build_edge(self, ccd_n): - """Build the edges of the `ccd_n` in global coordinates.""" - if self.with_gaps: - corners = np.array( - [ - [-self.loc2glob.x_gap / 2, -self.loc2glob.y_gap / 2], - [ - self.loc2glob.x_npix + self.loc2glob.x_gap / 2, - -self.loc2glob.y_gap / 2, - ], - [ - -self.loc2glob.x_gap / 2, - self.loc2glob.y_npix + self.loc2glob.y_gap / 2, - ], - [ - self.loc2glob.x_npix + self.loc2glob.x_gap / 2, - self.loc2glob.y_npix + self.loc2glob.y_gap / 2, - ], - ] - ) - else: - corners = np.array( - [ - [0, 0], - [self.loc2glob.x_npix, 0], - [0, self.loc2glob.y_npix], - [self.loc2glob.x_npix, self.loc2glob.y_npix], - ] - ) - - glob_corners = np.array( - [ - self.loc2glob.loc2glob_img_coord(ccd_n, pos[0], pos[1]) - for pos in corners - ] - ) - - edge_xy = [] - for idx in (0, 1): - edge = np.array( - [np.min(glob_corners[:, idx]), np.max(glob_corners[:, idx])] - ) - edge_xy.append(edge) - - return edge_xy - - def is_inside(self, x, y, edge_x, edge_y): - """Is the position inside the edges. - - Return True if the position is within the rectangle - defined by the edges. - - Parameters - ---------- - x: float - Horizontal position in global coordinate system. - y: float - Vertical position in global coordinate system. - edge_x: np.ndarray - Edge defined as `np.array([min_x, max_x])`. - edge_y: np.ndarray - Edge defined as `np.array([min_y, max_y])`. - """ - if ( - (x > edge_x[0]) - and (x < edge_x[1]) - and (y > edge_y[0]) - and (y < edge_y[1]) - ): - return True - else: - return False - - def get_ccd_n(self, x, y): - """Returns the CCD number from the position `(x, y)`. - - Returns `None` if the position is not found. - """ - bool_list = np.array( - [ - self.is_inside(x, y, edge_x, edge_y) - for edge_x, edge_y in zip(self.edge_x_list, self.edge_y_list) - ] - ) - - try: - return self.ccd_list[bool_list][0] - except Exception: - return None + return f"{file_pattern}_conv-{idx}.fits" class Convert(object): @@ -457,38 +96,25 @@ def params_default(self): self._params = { "input_base_dir": ".", "output_base_dir": ".", - "version_cat": "v2.0", "mode": "merge", - "patches": "", - "psf": "psfex", "file_pattern_psfint": "validation_psf", } self._short_options = { "input_base_dir": "-i", - "version_cat": "-V", "mode": "-m", - "psf": "-p", - "patches": "-P", } self._types = {} self._help_strings = { "input_base_dir": ( - "input base dir; for v1.x runs are expected in" - + " /P/output, for v2.0 (patch-less) in" + "input base dir; single-exposure runs are expected in" + " /output; default is {}" ), - "version_cat": ( - "catalogue major version, allowed are v1.3, v1.4, v1.5, v1.6," - + " v2.0; v2.0 is patch-less; default is {}" - ), "mode": ( "run mode, allowed are 'merge', 'test'; default is" + " '{}'" ), - "psf": "PSF model, allowed are 'psfex' and 'mccd'; default is {}", - "patches": "(list of) input patches; ignored for v2.0", } # Output column names with types @@ -508,127 +134,67 @@ def params_default(self): ("CCD_NB", int), ] - # Extra columns for MCCD:737 - self._dt_mccd = self._dt.copy() - self._dt_mccd.append(("GLOB_X", float)) - self._dt_mccd.append(("GLOB_Y", float)) - - def update_params(self): - """Update Params. - - Update parameters. - - """ - if self._params["psf"] == "psfex": - #self._params["sub_dir_pattern"] = "run_sp_exp_202" - self._params["sub_dir_pattern"] = "run_sp_combined_psf" - self._params["sub_dir_psfint"] = "psfex_interp_runner" - elif self._params["psf"] == "mccd": - self._params["sub_dir_pattern"] = "run_sp_exp_SxSePsf_202" - self._params["sub_dir_psfint"] = "mccd_fit_val_runner" - self._params["sub_dir_setools"] = "setools_runner/output/mask" - else: - raise ValueError(f"Invalid PSF model {self._params['psf']}") - self._params["sub_dir_psfint"] = ( - f"{self._params['sub_dir_psfint']}/output" - ) - def run(self): """Run. Main processing function. """ - # Guard against a mistyped version silently falling through to the - # v1.x patch loop (e.g. ``-V v2`` or ``-V 2.0``). - allowed_versions = ("v1.3", "v1.4", "v1.5", "v1.6", "v2.0") - if self._params["version_cat"] not in allowed_versions: - raise ValueError( - f"Invalid version {self._params['version_cat']}; allowed are" - + f" {', '.join(allowed_versions)}" - ) - - # v2.0 removes the patch concept: a single patch-less run root. For - # v1.x, iterate over the requested sky patches as before. ``patch`` is - # None in the patch-less case, which drops the patch token from the - # input path and the output filename. - if self._params["version_cat"] == "v2.0": - patch_nums = [None] - elif self._params["mode"] == "test": - patch_nums = ["3", "4"] - else: - patch_nums = cs_args.my_string_split(self._params["patches"]) - do_parallel = True - # Loop over patches - for patch in patch_nums: + output_dir = self._params["output_base_dir"] + if not os.path.isdir(output_dir): + os.makedirs(output_dir, exist_ok=True) - patch_dir, output_dir = collate_paths( - self._params["input_base_dir"], - self._params["output_base_dir"], - patch, - ) - print("Running patch-less (v2.0)" if patch is None else f"Running patch: {patch}") - - if not os.path.isdir(output_dir): - os.makedirs(output_dir, exist_ok=True) + subdirs = f"{self._params['input_base_dir']}/output/{EXP_RUN_GLOB}" + exp_run_dirs = glob.glob(subdirs) + n_exp_runs = len(exp_run_dirs) + print(f"Found {n_exp_runs} input single-exposure run(s) ({subdirs})") - subdirs = f"{patch_dir}/{self._params['sub_dir_pattern']}*" - exp_run_dirs = glob.glob(subdirs) + if self._params["mode"] == "test": + exp_run_dirs = exp_run_dirs[:2] n_exp_runs = len(exp_run_dirs) print( - f"Found {n_exp_runs} input single-exposure run(s) for patch" - + f" {patch_dir} ({subdirs})" + f"test mode: only using {n_exp_runs} input single-exposure" + + f" runs" ) - if self._params["mode"] == "test": - exp_run_dirs = exp_run_dirs[:2] - n_exp_runs = len(exp_run_dirs) - print( - f"test mode: only using {n_exp_runs} input single-exposure" - + f" runs" + # Loop over exposure runs + if not do_parallel: + for idx_exp, exp_run_dir in tqdm( + enumerate(exp_run_dirs), + total=n_exp_runs, + disable=self._params["verbose"], + ): + self.transform_exposures(output_dir, idx_exp, exp_run_dir) + else: + res = Parallel(n_jobs=-1, backend="loky")( + delayed(self.transform_exposures)( + output_dir, idx_exp, exp_run_dir ) - - # Loop over exposure runs - if not do_parallel: for idx_exp, exp_run_dir in tqdm( enumerate(exp_run_dirs), total=n_exp_runs, disable=self._params["verbose"], - ): - self.transform_exposures( - output_dir, patch, idx_exp, exp_run_dir - ) - else: - res = Parallel(n_jobs=-1, backend="loky")( - delayed(self.transform_exposures)( - output_dir, patch, idx_exp, exp_run_dir - ) - for idx_exp, exp_run_dir in tqdm( - enumerate(exp_run_dirs), - total=n_exp_runs, - disable=self._params["verbose"], - ) ) + ) - def transform_exposures(self, output_dir, patch, idx, exp_run_dir): - """Transform exposures. + def transform_exposures(self, output_dir, idx, exp_run_dir): + """Transform Exposures. - Transform shapes for exposure for a given run (input exp run dir). + Collate the PSF validation catalogues of one single-exposure run + (input exp run dir) into one output catalogue. """ output_path = ( f"{output_dir}/" - + output_filename( - self._params["file_pattern_psfint"], patch, idx - ) + + output_filename(self._params["file_pattern_psfint"], idx) ) if os.path.exists(output_path): print(f"Skipping transform_exposures, file {output_path} exists") return - psf_dir = f"{exp_run_dir}/{self._params['sub_dir_psfint']}" + psf_dir = f"{exp_run_dir}/{PSF_INTERP_SUBDIR}" try: all_files = os.listdir(psf_dir) if self._params["verbose"]: @@ -645,11 +211,7 @@ def transform_exposures(self, output_dir, patch, idx, exp_run_dir): tmp = re.findall(r"\d+", file_name) - if self._params["psf"] == "psfex": - exp_name, ccd_id = int(tmp[0]), int(tmp[1]) - elif self._params["psf"] == "mccd": - exp_name = int(tmp[0]) - ccd_id = -1 + exp_name, ccd_id = int(tmp[0]), int(tmp[1]) if self._params["verbose"]: print("Match found ", exp_name, ccd_id) @@ -657,194 +219,42 @@ def transform_exposures(self, output_dir, patch, idx, exp_run_dir): psf_file_path = f"{psf_dir}/{file_name}" try: - if self._params["psf"] == "psfex": - psf_file_hdus = fits.open(psf_file_path, memmap=False) - psf_file = psf_file_hdus[2].data - psf_file_hdus.close() - mod = "RA" - else: - psf_file = fits.getdata(psf_file_path, 1, memmap=True) - mod = "RA_LIST" + psf_file_hdus = fits.open(psf_file_path, memmap=False) + psf_file = psf_file_hdus[2].data + psf_file_hdus.close() except Exception: continue - if self._params["psf"] == "psfex": - # HSM ellipticities and sizes are measured directly in world - # coordinates upstream (FindAdaptiveMom use_sky_coords=True), so - # they are passed straight through; only positions are collated. - exp_cat = np.array( - list( - map( - tuple, - np.array( - [ - psf_file["X"], - psf_file["Y"], - psf_file["RA"], - psf_file["DEC"], - psf_file["E1_PSF_HSM"], - psf_file["E2_PSF_HSM"], - psf_file["SIGMA_PSF_HSM"], - psf_file["FLAG_PSF_HSM"], - psf_file["E1_STAR_HSM"], - psf_file["E2_STAR_HSM"], - psf_file["SIGMA_STAR_HSM"], - psf_file["FLAG_STAR_HSM"], - np.ones_like(psf_file["RA"], dtype=int) - * ccd_id, - ] - ).T.tolist(), - ) - ), - dtype=self._dt, - ) - cat_list.append(exp_cat) - - else: - l2g = Loc2Glob() - g2c = Glob2CCD(l2g) - new_ccd_id = np.array( - [ - int( - g2c.get_ccd_n( - psf_file["GLOB_POSITION_IMG_LIST"][ii, 0], - psf_file["GLOB_POSITION_IMG_LIST"][ii, 1], - ) - ) - for ii in range(len(psf_file)) - ] - ) - - # Local-to-CCD position: subtract each CCD's focal-plane shift. - new_x = np.zeros_like(psf_file[mod]) - new_y = np.zeros_like(psf_file[mod]) - - # The MCCD PSF_MOM_LIST/STAR_MOM_LIST columns come from the - # external mccd fit-validation code, which still measures HSM - # moments in the pixel frame; rotate them into world coordinates - # via the per-CCD WCS Jacobian. This rotation stays until mccd - # itself adopts use_sky_coords (see the module docstring). The - # in-repo PSFEx / MCCD-interpolation paths are already in world - # coordinates and are passed through unrotated. - new_e1_psf = np.zeros_like(psf_file[mod]) - new_e2_psf = np.zeros_like(psf_file[mod]) - new_sig_psf = np.zeros_like(psf_file[mod]) - new_e1_star = np.zeros_like(psf_file[mod]) - new_e2_star = np.zeros_like(psf_file[mod]) - new_sig_star = np.zeros_like(psf_file[mod]) - new_flag_psf = np.zeros_like(psf_file[mod]) - new_flag_star = np.zeros_like(psf_file[mod]) - for ccd_id in range(40): - m_ccd_id = new_ccd_id == ccd_id - if sum(m_ccd_id) == 0: - continue - - x_shift, y_shift = l2g.shift_coord(ccd_id) - - new_x[m_ccd_id] = ( - psf_file["GLOB_POSITION_IMG_LIST"][:, 0][m_ccd_id] - - x_shift + # HSM ellipticities and sizes are measured directly in world + # coordinates upstream (FindAdaptiveMom use_sky_coords=True), so + # they are passed straight through; only positions are collated. + exp_cat = np.array( + list( + map( + tuple, + np.array( + [ + psf_file["X"], + psf_file["Y"], + psf_file["RA"], + psf_file["DEC"], + psf_file["E1_PSF_HSM"], + psf_file["E2_PSF_HSM"], + psf_file["SIGMA_PSF_HSM"], + psf_file["FLAG_PSF_HSM"], + psf_file["E1_STAR_HSM"], + psf_file["E2_STAR_HSM"], + psf_file["SIGMA_STAR_HSM"], + psf_file["FLAG_STAR_HSM"], + np.ones_like(psf_file["RA"], dtype=int) + * ccd_id, + ] + ).T.tolist(), ) - new_y[m_ccd_id] = ( - psf_file["GLOB_POSITION_IMG_LIST"][:, 1][m_ccd_id] - - y_shift - ) - - header_file_path = ( - self._params["sub_dir_setools"] - + self._params["file_pattern_psfint"] - + f"{exp_name}-{ccd_id}.fits" - ) - try: - header_file = fits.getdata(header_file_path, 1) - except Exception: - continue - header = fits.Header.fromstring( - "\n".join(header_file[0][0]), sep="\n" - ) - wcs = galsim.AstropyWCS(header=header) - - g1_psf_tmp_l = [] - g2_psf_tmp_l = [] - sig_psf_tmp_l = [] - g1_star_tmp_l = [] - g2_star_tmp_l = [] - sig_star_tmp_l = [] - flag_psf_tmp_l = [] - flag_star_tmp_l = [] - - for obj in psf_file[m_ccd_id]: - try: - jac = wcs.jacobian( - world_pos=galsim.CelestialCoord( - ra=obj["RA_LIST"] * galsim.degrees, - dec=obj["DEC_LIST"] * galsim.degrees, - ) - ) - except Exception: - flag_star_tmp_l.append(16) - flag_psf_tmp_l.append(16) - g1_psf_tmp_l.append(0) - g2_psf_tmp_l.append(0) - sig_psf_tmp_l.append(0) - g1_star_tmp_l.append(0) - g2_star_tmp_l.append(0) - sig_star_tmp_l.append(0) - continue - g1_psf_tmp, g2_psf_tmp, sig_psf_tmp = transform_shape( - obj["PSF_MOM_LIST"], jac - ) - - g1_psf_tmp_l.append(g1_psf_tmp) - g2_psf_tmp_l.append(g2_psf_tmp) - sig_psf_tmp_l.append(sig_psf_tmp) - flag_psf_tmp_l.append(obj["PSF_MOM_LIST"][3]) - - g1_star_tmp, g2_star_tmp, sig_star_tmp = ( - transform_shape(obj["STAR_MOM_LIST"], jac) - ) - g1_star_tmp_l.append(g1_star_tmp) - g2_star_tmp_l.append(g2_star_tmp) - sig_star_tmp_l.append(sig_star_tmp) - flag_star_tmp_l.append(obj["STAR_MOM_LIST"][3]) - - new_e1_psf[m_ccd_id] = g1_psf_tmp_l - new_e2_psf[m_ccd_id] = g2_psf_tmp_l - new_sig_psf[m_ccd_id] = sig_psf_tmp_l - new_flag_psf[m_ccd_id] = flag_psf_tmp_l - new_e1_star[m_ccd_id] = g1_star_tmp_l - new_e2_star[m_ccd_id] = g2_star_tmp_l - new_sig_star[m_ccd_id] = sig_star_tmp_l - new_flag_star[m_ccd_id] = flag_star_tmp_l - - exp_cat = np.array( - list( - map( - tuple, - np.array( - [ - new_x, - new_y, - psf_file["RA_LIST"], - psf_file["DEC_LIST"], - new_e1_psf, - new_e2_psf, - new_sig_psf, - psf_file["PSF_MOM_LIST"][:, 3], - new_e1_star, - new_e2_star, - new_sig_star, - psf_file["STAR_MOM_LIST"][:, 3], - new_ccd_id, - psf_file["GLOB_POSITION_IMG_LIST"][:, 0], - psf_file["GLOB_POSITION_IMG_LIST"][:, 1], - ] - ).T.tolist(), - ) - ), - dtype=self._dt_mccd, - ) - cat_list.append(exp_cat) + ), + dtype=self._dt, + ) + cat_list.append(exp_cat) del psf_file @@ -852,10 +262,10 @@ def transform_exposures(self, output_dir, patch, idx, exp_run_dir): return # Finalize catalogue - patch_cat = np.concatenate(cat_list) + star_cat = np.concatenate(cat_list) hdul = fits.HDUList() hdul.append(fits.PrimaryHDU()) - hdul.append(fits.BinTableHDU(patch_cat)) + hdul.append(fits.BinTableHDU(star_cat)) # Write catalogue hdul.writeto( @@ -874,7 +284,6 @@ def run_convert(*args): obj = Convert() obj.set_params_from_command_line(args) - obj.update_params() obj.run() diff --git a/scripts/python/distribute_tiles.py b/scripts/python/distribute_tiles.py deleted file mode 100755 index 338286002..000000000 --- a/scripts/python/distribute_tiles.py +++ /dev/null @@ -1,289 +0,0 @@ -#!/usr/bin/env python -""" -distribute_tiles.py - -Uses canfar.helpers.distributed.chunk to automatically distribute tiles -across replicas, then processes each tile assigned to this replica. -""" - -import os -import sys -import subprocess -from multiprocessing import Pool -import fcntl -from canfar.helpers.distributed import chunk - - -def parse_arguments(args): - """ - Extract the file IDs from command line arguments. - - Args: - args (list of str): Original command line arguments (e.g., sys.argv[1:]) - - Returns: - str or None: The value passed with -f, or None if not present - """ - # Default values - parsed = {"dry_run": 0, "parallel_jobs": 1} - - i = 0 - while i < len(args): - if args[i] == '-f' and i + 1 < len(args): - parsed['file_ids'] = args[i + 1] - i += 2 - elif args[i] == '--batch_num' and i + 1 < len(args): - parsed['batch_num'] = int(args[i + 1]) - i += 2 - elif args[i] == '--batch_tot' and i + 1 < len(args): - parsed['batch_tot'] = int(args[i + 1]) - i += 2 - elif args[i] == '--batch_size' and i + 1 < len(args): - parsed['batch_size'] = int(args[i + 1]) - i += 2 - elif args[i] in ('-n', '--dry_run') and i + 1 < len(args): - parsed['dry_run'] = int(args[i + 1]) - i += 2 - elif args[i] == '--parallel_jobs' and i + 1 < len(args): - parsed['parallel_jobs'] = int(args[i + 1]) - i += 2 - elif args[i] == "--debug_out" and i + 1 < len(args): - parsed["debug_out"] = args[i + 1] - i += 2 - else: - i += 1 - - return parsed - - -def get_my_tiles(all_tiles, batch_num, batch_tot, batch_size): - """ - Calculate which tiles this replica should process based on global position. - Args: - all_tiles: List of all tile IDs - batch_num: Current batch number (1-indexed) - batch_tot: Total number of batches - batch_size: Number of replicas per batch - Returns: - list: Tiles assigned to this replica - - """ - # Get local replica info from environment - local_replica_id = int(os.environ.get('REPLICA_ID', 1)) - - start_idx = (batch_num - 1) * batch_size - end_idx = min(batch_num * batch_size, len(all_tiles)) - - batch_tiles = all_tiles[start_idx:end_idx] - print(f"Batch {batch_num}/{batch_tot}, local replica {local_replica_id}/{batch_size}") - print(f"Batch processes tiles {start_idx + 1} to {end_idx} ({len(batch_tiles)} tiles)") - - # Use chunk() to distribute this batch's tiles among local replicas - # chunk() will use REPLICA_ID and REPLICA_COUNT automatically - my_tiles = list(chunk(batch_tiles)) - - return my_tiles - -def get_tile_list(file_ids): - """Read all tile IDs from file. - - Args: - file_ids: Path to file containing tile IDs - - Returns: - list: List of tile ID strings - """ - with open(file_ids, 'r') as f: - tiles = [line.strip() for line in f if line.strip()] - return tiles - - -def build_process_command(tile_id, original_args): - """ - Build command for a single tile by replacing -f with -e . - - Args: - tile_id (str): Tile ID to process - original_args (list of str): Original CLI arguments - - Returns: - list of str: Command ready for subprocess - """ - cmd = [f"{os.environ['HOME']}/shapepipe/scripts/sh/init_run_exclusive_canfar.sh"] - - # Transform arguments: replace -f with -e , - # skip batch arguments, parallel_jobs, and dry_run if not 0, 1 - i = 0 - while i < len(original_args): - if original_args[i] == '-f' and i + 1 < len(original_args): - cmd.extend(['-e', tile_id]) - i += 2 - elif original_args[i] in ('--batch_num', '--batch_tot', '--batch_size', '--parallel_jobs'): - i += 2 - elif original_args[i] in ('-n', '--dry_run'): - if i + 1 < len(original_args) and original_args[i + 1] in ("0", "1"): - cmd.extend(["-n", original_args[i + 1]]) - i += 2 - else: - cmd.append(original_args[i]) - i += 1 - - return cmd - - -def process_single_tile(args_tuple): - """ - Process a single tile (designed for multiprocessing). - - Args: - args_tuple: Tuple of (tile_id, tile_num, total_tiles, original_args, dry_run, msg_batch) - - Returns: - tuple: (tile_id, success, error_message) - """ - tile_id, tile_num, total_tiles, original_args, dry_run, msg_batch = args_tuple - - print(f"{'='*5} Processing {msg_batch}tile num/total={tile_num}/{total_tiles}: ID={tile_id} {'='*5}") - - # Build command - cmd = build_process_command(tile_id, original_args) - print(f"Command: {' '.join(cmd)}") - - # Execute command - try: - if dry_run != 2: - result = subprocess.run(cmd, check=True, capture_output=True, text=True) - print(f"✓ Successfully processed tile {tile_id}") - return (tile_id, True, None) - else: - print(f"dry_run=2") - return (tile_id, True, None) - except subprocess.CalledProcessError as e: - error_msg = f"Failed to process tile {tile_id}: {e}" - print(f"✗ {error_msg}", file=sys.stderr) - return (tile_id, False, error_msg) - except Exception as e: - error_msg = f"Unexpected error processing tile {tile_id}: {e}" - print(f"✗ {error_msg}", file=sys.stderr) - return (tile_id, False, error_msg) - - -def print_debug(pat, tile_list, out_path, verbose=False): - - local_replica_id = int(os.environ.get('REPLICA_ID', 1)) - - with open(out_path, "a") as f: - # Exclusive lock for save parallel use - fcntl.flock(f.fileno(), fcntl.LOCK_EX) - try: - # Build output string - output = f"{pat} distribute_tiles REPLICA_ID={local_replica_id}, tiles=" - output += " ".join(tile_list) + "\n" - - # Write to file - f.write(output) - - # Optionally write to stdout - if verbose: - print(output, end="") - finally: - # Unlock - fcntl.flock(f.fileno(), fcntl.LOCK_UN) - - -def main(): - - # Debug file line pattern - pat = "- " - - # Parse arguments - args = parse_arguments(sys.argv[1:]) - - if not "file_ids" in args: - print("Error: -f must be provided", file=sys.stderr) - sys.exit(1) - else: - file_ids = args["file_ids"] - - # Read all tiles - print(f"Reading tile list from {file_ids}") - all_tiles = get_tile_list(file_ids) - print(f"Total tiles in file: {len(all_tiles)}") - - print( - "REPLICA_ID, REPLICA_COUNT=", - os.environ.get('REPLICA_ID'), - os.environ.get('REPLICA_COUNT') - ) - - - # Check if we're in multi-batch mode - if "batch_num" in args and "batch_tot" in args and "batch_size" in args: - # Multi-batch mode: calculate global distribution - my_tile_list = get_my_tiles( - all_tiles, - args['batch_num'], - args['batch_tot'], - args['batch_size'] - ) - msg_batch = f"batch {args['batch_num']}/{args['batch_tot']} " - - else: - # Use chunk() to get tiles for this replica - print("Using chunk() to determine tiles for this replica...") - my_tile_list = list(chunk(all_tiles)) - msg_batch = "" - - if "debug_out" in args: - print_debug(f"{pat}{msg_batch}", my_tile_list, args["debug_out"], verbose=True) - - print(f"This replica assigned {len(my_tile_list)} tiles") - print(f"Parallel jobs: {args['parallel_jobs']}") - - if len(my_tile_list) > 0: - print(f"First tile: {my_tile_list[0]}") - if len(my_tile_list) > 1: - print(f"Last tile: {my_tile_list[-1]}") - else: - print("Only one tile") - - # Process each tile assigned to this replica - success_count = 0 - failure_count = 0 - - # Prepare arguments for parallel processing - process_args = [ - (tile_id, i, len(my_tile_list), sys.argv[1:], args["dry_run"], msg_batch) - for i, tile_id in enumerate(my_tile_list, 1) - ] - - if args['parallel_jobs'] > 1: - # Parallel processing - print(f"Processing tiles in parallel with {args['parallel_jobs']} workers") - with Pool(processes=args['parallel_jobs']) as pool: - results = pool.map(process_single_tile, process_args) - else: - # Sequential processing (original behavior) - print("Processing tiles sequentially") - results = [process_single_tile(arg) for arg in process_args] - - # Count successes and failures - for tile_id, success, error_msg in results: - if success: - success_count += 1 - else: - failure_count += 1 - - # Summary - print( - f"{'='*5} Processing completed tot/suc/fail=" - + f"{len(my_tile_list)}/{success_count}/{failure_count}" - ) - - # Exit with error if any failures - if failure_count > 0: - sys.exit(1) - - -if __name__ == '__main__': - main() diff --git a/scripts/python/get_ccds_with_psf.py b/scripts/python/get_ccds_with_psf.py deleted file mode 100755 index e284e2640..000000000 --- a/scripts/python/get_ccds_with_psf.py +++ /dev/null @@ -1,59 +0,0 @@ -#!/usr/bin/env python3 - -"""GET_CCDS_WITH_PSF - -Obtain list of CCDs (single-exposure single-HDU files) for which valid PSF information -is available. This can serve to create a footprint coverage mask. - -Author: Martin Kilbinger - -""" - -import sys - -from shapepipe.utilities.ccd_psf_handler import CcdPsfHandler - - -def run_ccd_psf_handler(args=None): - """Run CCD PSF Handler. - - Create instance and run the CCD PSF handler. - - Parameters - ---------- - args : list, optional - command line arguments - - Returns - ------- - int - exit code - - """ - # Create instance - obj = CcdPsfHandler() - - return obj.run(args=args) - - -def main(argv=None): - """Main. - - Main program. - - Parameters - ---------- - argv : list, optional - command line arguments - - Returns - ------- - int - exit code - - """ - return run_ccd_psf_handler(args=argv) - - -if __name__ == "__main__": - sys.exit(main(sys.argv)) diff --git a/scripts/python/get_number_objects.py b/scripts/python/get_number_objects.py deleted file mode 100755 index 98422457c..000000000 --- a/scripts/python/get_number_objects.py +++ /dev/null @@ -1,243 +0,0 @@ -#!/usr/bin/env python - -# -*- coding: utf-8 -*- - -"""Script get_number_objects.py - -Get number of objects in a (last-run SExtractor) catalogue. - -:Author: Martin Kilbinger - -""" - -import sys -import copy -import glob - -from optparse import OptionParser -from astropy.io import fits - -from shapepipe.pipeline.run_log import get_last_dir, get_all_dirs -from shapepipe.utilities import cfis - - -class param: - """General class to store (default) variables""" - - def __init__(self, **kwds): - self.__dict__.update(kwds) - - def print(self, **kwds): - print(self.__dict__) - - def var_list(self, **kwds): - return vars(self) - - -def params_default(): - """Set default parameter values. - - Returns - ------- - class param - parameter values - - """ - p_def = param( - input_path=".", - input_name_base="final_cat", - hdu_num=1, - ) - - return p_def - - -def parse_options(p_def): - """Parse command line options. - - Parameters - ---------- - p_def: class param - parameter values - - Returns - ------- - list - command line options - command line str - - """ - usage = "%prog [OPTIONS]" - parser = OptionParser(usage=usage) - - # IO - parser.add_option( - "-i", - "--input_path", - dest="input_path", - type="string", - default=p_def.input_path, - help=f"input path, default='{p_def.input_path}'", - ) - parser.add_option( - "-n", - "--input_name_base", - dest="input_name_base", - type="string", - default=p_def.input_name_base, - help=f"input name base, default='{p_def.input_name_base}'", - ) - parser.add_option( - "-l", - "--list_tile_ID_path", - dest="tile_ID_list_path", - type="string", - default=None, - help=f"tile ID list, default: Use all data in input files", - ) - - # Control - parser.add_option( - "-p", - "--param_path", - dest="param_path", - type="string", - default=None, - help="parameter file path, default=None", - ) - - parser.add_option( - "", - "--hdu_num", - dest="hdu_num", - type="int", - default=p_def.hdu_num, - help=f"input HDU number, default='{p_def.hdu_num}'", - ) - - parser.add_option( - "-v", - "--verbose", - dest="verbose", - action="store_true", - help="verbose output", - ) - - options, args = parser.parse_args() - - return options, args - - -def check_options(options): - """Check command line options. - - Parameters - ---------- - options: tuple - Command line options - - Returns - ------- - bool - Result of option check. False if invalid option value. - - """ - return True - - -def update_param(p_def, options): - """Return default parameter, updated and complemented according to options. - - Parameters - ---------- - p_def: class param - parameter values - optiosn: tuple - command line options - - Returns - ------- - class param - updated paramter values - - """ - param = copy.copy(p_def) - - # Update keys in param according to options values - for key in vars(param): - if key in vars(options): - setattr(param, key, getattr(options, key)) - - # Add remaining keys from options to param - for key in vars(options): - if not key in vars(param): - setattr(param, key, getattr(options, key)) - - # Do extra stuff if necessary - - return param - - -def main(argv=None): - - # Set default parameters - p_def = params_default() - - # Command line options - options, args = parse_options(p_def) - - if check_options(options) is False: - return 1 - - param = update_param(p_def, options) - - # Save command line arguments to log file - f_log = cfis.log_command(argv, close_no_return=False) - - pattern = "sexcat" - run_log_file = "output/log_run_sp.txt" - - # For v1 - # module = 'sextractor_runner_run_1' - - # For v2 - module = "sextractor_runner" - - # Get all sextractor output directories - all_dir = get_all_dirs(run_log_file, module) - paths = [] - - # Find tile runs - for path in all_dir: - if "run_sp_tile_Sx" in path: - paths.append(path) - paths = sorted(paths) - - if len(paths) == 0: - # No previous tile SExtractor run found - return -1 - - # Get latest run - last_dir = paths[-1] - - # Get all output SExtractor catalogues - file_list = glob.glob(f"{last_dir}/{pattern}*.fits") - if len(file_list) == 0: - raise ValueError(f"No files {last_dir}/{pattern}*.fits found") - - # Add up number of objects over all catalogues - n_obj = 0 - hdu_no = -1 - for fpath in file_list: - hdu_list = fits.open(fpath) - header = hdu_list[-1].header - n_obj += int(header["NAXIS2"]) - - # Compute average - n_obj = int(n_obj / len(file_list)) - - print(n_obj) - - -if __name__ == "__main__": - sys.exit(main(sys.argv)) diff --git a/scripts/python/link_to_exp_for_tile.py b/scripts/python/link_to_exp_for_tile.py deleted file mode 100755 index 5b2a5f0e2..000000000 --- a/scripts/python/link_to_exp_for_tile.py +++ /dev/null @@ -1,467 +0,0 @@ -#!/usr/bin/env python - -# -*- coding: utf-8 -*- - -"""Script link_to_exp_for_tile.py - -:Description: Link to exposure and PSF catalogue - for a given tile. - -:Author: Martin Kilbinger - -""" - -import os -import sys -import re -import copy - -from optparse import OptionParser - - -class param: - """General class to store (default) variables""" - - def __init__(self, **kwds): - self.__dict__.update(kwds) - - def print(self, **kwds): - print(self.__dict__) - - def var_list(self, **kwds): - return vars(self) - - -def params_default(): - """Params Default. - - Set default parameter values. - - Returns - ------- - class param - parameter values - - """ - p_def = param( - tile_base_dir=".", - exp_base_dir=".", - sp_local=0, - ) - - return p_def - - -def parse_options(p_def): - """Parse Options. - - Parse command line options. - - Parameters - ---------- - p_def: class param - parameter values - - Returns - ------- - list - command line options - command line str - - """ - usage = "%prog [OPTIONS]" - parser = OptionParser(usage=usage) - - # IO - parser.add_option( - "-i", - "--input_tile_dir", - dest="tile_base_dir", - type="string", - default=p_def.tile_base_dir, - help=f"input tile base directory, default='{p_def.tile_base_dir}'", - ) - parser.add_option( - "-t", - "--tile_ID", - dest="tile_ID", - type="string", - help=f"input tile ID", - ) - parser.add_option( - "-I", - "--input_exp_dir", - dest="exp_base_dir", - type="string", - default=p_def.exp_base_dir, - help=f"input exposure base directory, default='{p_def.exp_base_dir}'", - ) - parser.add_option( - "-s", - "--sp_local", - dest="sp_local", - type="int", - default=p_def.sp_local, - help=f"local runi of split_exposure_runner, default='{p_def.sp_local}'", - ) - parser.add_option( - "-v", - "--verbose", - dest="verbose", - action="store_true", - help="verbose output", - ) - - options, args = parser.parse_args() - - return options, args - - -def check_options(options): - """Check Options. - - Check command line options. - - Parameters - ---------- - options: tuple - Command line options - - Returns - ------- - bool - Result of option check. False if invalid option value. - - """ - return True - - -def update_param(p_def, options): - """Update Param. - - Return default parameter, updated and complemented according to options. - - Parameters - ---------- - p_def: class param - parameter values - optiosn: tuple - command line options - - Returns - ------- - class param - updated paramter values - - """ - param = copy.copy(p_def) - - # Update keys in param according to options values - for key in vars(param): - if key in vars(options): - setattr(param, key, getattr(options, key)) - - # Add remaining keys from options to param - for key in vars(options): - if not key in vars(param): - setattr(param, key, getattr(options, key)) - - # Do extra stuff if necessary - - return param - - -# TODO: move to cs_util -def matching_subdirs(base_dir, pattern): - """Matching Subdirs. - - Return all subdirectories whose name start with a given string pattern. - - Parameters - ---------- - base_dir: str - base directory - pattern: str - pattern to match beginning of subdir name - - Returns - ------- - list - matched subdirectory names - - """ - subdirs = [] - if os.path.exists(base_dir): - # Loop over directory entries - for entry in os.listdir(base_dir): - full_path = os.path.join(base_dir, entry) - # Append if match - if os.path.isdir(full_path) and entry.startswith(pattern): - subdirs.append(full_path) - else: - print(f"Warning: {base_dir} does not exist, continuing...") - - # Sort according to creation date - subdirs.sort(key=os.path.getctime) - - return subdirs - - -def get_tile_out_dir(tile_base_dir, tile_ID): - """Get Tile Out Dir. - - Return output directory path for a given tile ID. - - Parameters - ---------- - tile_base_dir: str - base directory of tile runs - tile_ID: str - tile ID - - Returns - ------- - str - path - - """ - return f"{tile_base_dir}/{tile_ID}/output" - - -def get_exp_IDs(tile_base_dir, tile_ID, verbose=False): - """Get Exp IDs. - - Return exposure IDs used for given tile. - - Parameters - ---------- - tile_base_dir: str - base directory of tile runs - tile_ID: str - tile ID - verbose: bool, optional - verbose output if ``True``; default is ``False`` - - Returns - ------- - list - exposure IDs - - """ - # Get tile output path - tile_out_dir = get_tile_out_dir(tile_base_dir, tile_ID) - - # Get subdirectories with runs "Fe" (find exposures) - pattern = "run_sp_GitFeGie" - subdirs = matching_subdirs(tile_out_dir, pattern) - - # Raise error if not exactly one run - if len(subdirs) == 0: - raise IOError( - f"No matching directory '{pattern}' in {tile_out_dir} found" - ) - if len(subdirs) != 1: - raise IOError( - f"Exactly one directory natching {pattern} in {tile_out_dir} " - + f"expected, not {len(subdirs)}" - ) - - # Replace dot with dash in tile ID - tile_ID_sp = re.sub(r"\.", "-", tile_ID) - # Get output file of find_exposure_runner for this tile ID - exp_ID_file = ( - f"{subdirs[0]}/find_exposures_runner/output/" - + f"exp_numbers-{tile_ID_sp}.txt" - ) - - exp_IDs = [] - # Read file to get exposure IDs - with open(exp_ID_file) as f_in: - for line in f_in: - name = line.strip() - # Remove any letter - ID = re.sub("[a-zA-Z]", "", name) - exp_IDs.append(ID) - - if verbose: - print("Exposures: ", exp_IDs) - return exp_IDs - - -def get_exp_single_HDU_IDs(exp_IDs, n_CPU): - """Get Exp Single HDU IDs. - - Return all single-HDU IDs for given exposures. - - Parameters - ---------- - exp_IDs: list - input exposure IDs, list of str - n_CPU: int - number of CPUs (=HDUs) of an exposure; n_CPU=40 for MegaCAM - - Returns - ------- - list - list of single-HDU exposure IDs - - """ - exp_shdu_IDs = [] - # Loop over input exposure IDs - for exp_ID in exp_IDs: - # Append int up to n_CPU to each - for idx in range(n_CPU): - ID = f"{exp_ID}-{idx}" - exp_shdu_IDs.append(ID) - - return exp_shdu_IDs - - -def get_paths(exp_base_dir, exp_shdu_IDs, pattern): - """Get Paths. - - Return (newest) subdirectory for each input single-exposure HDU ID that matches pattern. - - Parameters - ---------- - exp_base_dir: str - base directory for (single-HDU) exposure runs - exp_shdu_IDs: list - single-HDU exposure IDs; list of str - pattern: str - pattern to match beginning of subdir names - Returns - ------- - list - matching paths; list of str, one entry for each input single-exp ID - - """ - paths = [] - # Loop over single-HDU exposure IDs - for exp_shdu_ID in exp_shdu_IDs: - - # output path of runs - name = f"{exp_base_dir}/{exp_shdu_ID}/output" - path = os.path.abspath(name) - - # get matching subdirs - subdirs = matching_subdirs(path, pattern) - n_subdirs = len(subdirs) - - if n_subdirs != 1: - msg = ( - f"Exactly one directory matching {pattern} in {path} expected," - + f" not {n_subdirs}" - ) - - # If more than one found: sort by name = sort by date - subdirs = sorted(subdirs) - - # No match - if n_subdirs == 0: - continue - - # Append matching subdir; if more than one append newest - paths.append(f"{subdirs[-1]}") - - return paths - - -def create_links_paths(tile_base_dir, tile_ID, paths, verbose=False): - """Create Links Paths. - - Create links to paths. - - Parameters - ---------- - tile_base_dir: str - base directory for tile runs - tile_ID: str - tile ID - paths: list - paths; list of str - verbose: bool, optional - verbose output if ``True``; default is ``False`` - - """ - # Get tile output path - tile_out_dir = get_tile_out_dir(tile_base_dir, tile_ID) - - # Loop over paths - for path in paths: - - # Get destination = tile output dir + path tail (part of path after last slash) - head, tail = os.path.split(path) - src = path - dst = f"{tile_out_dir}/{tail}" - - if os.path.exists(dst): - - src_existing = os.readlink(dst) - if src_existing == src: - # destination already points to source: skip - if verbose: - print( - f"Warning: {src} <- {dst} already exists, no link created" - ) - continue - else: - # destination points to different source: create links with added index to distinguish - # from existing one(s) - idx = 1 - dst_orig = dst - while True: - # Find destination name with lowest appended index that does not exist - dst = f"{dst_orig}_{idx}" - if os.path.exists(dst): - idx += 1 - else: - # Found: create new link - if verbose: - print(f"link {src} <- {dst}") - os.symlink(src, dst) - break - else: - - # destination does not exist: create link - if verbose: - print(f"link {src} <- {dst}") - os.symlink(src, dst) - - -def main(argv=None): - - # Set default parameters - p_def = params_default() - - # Command line options - options, args = parse_options(p_def) - - if check_options(options) is False: - return 1 - - param = update_param(p_def, options) - - tile_base_dir = param.tile_base_dir - exp_base_dir = param.exp_base_dir - tile_ID = param.tile_ID - n_CPU = 40 - verbose = param.verbose - - exp_IDs = get_exp_IDs(tile_base_dir, tile_ID, verbose=verbose) - exp_shdu_IDs = get_exp_single_HDU_IDs(exp_IDs, n_CPU) - - # Note: psfex P3 is mostly run_sp_exp_SxSePsf - patterns = ["run_sp_exp_SxSePsfPi"] # , "run_sp_exp_Pi"] - if param.sp_local == 1: - patterns.append("run_sp_exp_Sp_shdu") - for pattern in patterns: - paths = get_paths(exp_base_dir, exp_shdu_IDs, pattern) - - create_links_paths(tile_base_dir, tile_ID, paths, verbose=verbose) - - return 0 - - -if __name__ == "__main__": - sys.exit(main(sys.argv)) diff --git a/scripts/python/merge_final_cat.py b/scripts/python/merge_final_cat.py deleted file mode 100755 index 9d9c10d21..000000000 --- a/scripts/python/merge_final_cat.py +++ /dev/null @@ -1,396 +0,0 @@ -#!/usr/bin/env python - -# -*- coding: utf-8 -*- - -"""Script merge_final_cat.py - -Merge all final catalogues, created by ShapePipe module -``make_catalogue_runner``, into a joined numpy binary file. - -:Authors: Axel Guinot, Martin Kilbinger - -""" - -from astropy.io import fits -import numpy as np -import os -import sys -import re -import copy - -from optparse import OptionParser - -from shapepipe.utilities import cfis - - -class param: - """General class to store (default) variables""" - - def __init__(self, **kwds): - self.__dict__.update(kwds) - - def print(self, **kwds): - print(self.__dict__) - - def var_list(self, **kwds): - return vars(self) - - -def params_default(): - """Params Default. - - Set default parameter values. - - Returns - ------- - class param - parameter values - - """ - p_def = param( - input_path=".", - input_name_base="final_cat", - hdu_num=1, - ) - - return p_def - - -def parse_options(p_def): - """Parse Options. - - Parse command line options. - - Parameters - ---------- - p_def: class param - parameter values - - Returns - ------- - list - command line options - command line str - - """ - usage = "%prog [OPTIONS]" - parser = OptionParser(usage=usage) - - # IO - parser.add_option( - "-i", - "--input_path", - dest="input_path", - type="string", - default=p_def.input_path, - help=f"input path, default='{p_def.input_path}'", - ) - parser.add_option( - "-n", - "--input_name_base", - dest="input_name_base", - type="string", - default=p_def.input_name_base, - help=f"input name base, default='{p_def.input_name_base}'", - ) - parser.add_option( - "-l", - "--list_tile_ID_path", - dest="tile_ID_list_path", - type="string", - default=None, - help=f"tile ID list, default: Use all data in input files", - ) - - # Control - parser.add_option( - "-p", - "--param_path", - dest="param_path", - type="string", - default=None, - help="parameter file path, default=None", - ) - - parser.add_option( - "", - "--hdu_num", - dest="hdu_num", - type="int", - default=p_def.hdu_num, - help=f"input HDU number, default='{p_def.hdu_num}'", - ) - - parser.add_option( - "-v", - "--verbose", - dest="verbose", - action="store_true", - help="verbose output", - ) - - options, args = parser.parse_args() - - return options, args - - -def check_options(options): - """Check Options. - - Check command line options. - - Parameters - ---------- - options: tuple - Command line options - - Returns - ------- - bool - Result of option check. False if invalid option value. - - """ - return True - - -def update_param(p_def, options): - """Update Param. - - Return default parameter, updated and complemented according to options. - - Parameters - ---------- - p_def: class param - parameter values - optiosn: tuple - command line options - - Returns - ------- - class param - updated paramter values - - """ - param = copy.copy(p_def) - - # Update keys in param according to options values - for key in vars(param): - if key in vars(options): - setattr(param, key, getattr(options, key)) - - # Add remaining keys from options to param - for key in vars(options): - if not key in vars(param): - setattr(param, key, getattr(options, key)) - - # Do extra stuff if necessary - - return param - - -# MKDEBUG TODO: remove this function, duplicate in create_final_cat.py -def read_param_file(path, verbose=False): - """Read Param File. - - Return parameter list read from file. - - Parameters - ---------- - path: str - input file name - verbose: bool, optional, default=False - verbose output if True - - Returns - ------- - list of str - parameter names - - """ - param_list = [] - - if path: - - with open(path) as f: - for line in f: - if line.startswith("#"): - continue - entry = line.rstrip() - if not entry or entry == "": - continue - param_list.append(entry) - - if verbose: - if len(param_list) > 0: - print(f"Copying {len(param_list)} columns", end="") - else: - print("Copying all columns", end="") - print(" into merged catalogue") - - # Check for multiples - multiples = [] - for param in param_list: - if param_list.count(param) > 1: - multiples.append(param) - - if len(multiples) > 0: - print( - "The following parameters are more than one times " - "in the parameter file: ", - end="", - ) - for m in multiples: - print(m, end=" ") - print() - raise ValueError("Multiple identical keys found") - - return param_list - - -def get_data(path, hdu_num, param_list): - """Get Data. - - Return data of selected columns from FITS file. - - Parameters - ---------- - path: str - input file name - hdu_num: int - HDU number - param_list: list of str - parameters to be extracted. If none, copy - all columns - - Returns - ------- - numpy array - data columns - - """ - hdu_list = fits.open(path) - hdu = hdu_list[hdu_num] - - if param_list: - cols = [] - for p in param_list: - cols.append(hdu.columns[p]) - coldefs = fits.ColDefs(cols) - hdu_new = fits.BinTableHDU.from_columns(coldefs) - d = hdu_new.data - else: - d = hdu.data - - return d - - -def main(argv=None): - - - # Set default parameters - p_def = params_default() - - # Command line options - options, args = parse_options(p_def) - - if check_options(options) is False: - return 1 - - param = update_param(p_def, options) - - if param.verbose: - print("Start") - - # Save command line arguments to log file - f_log = cfis.log_command(argv, close_no_return=False) - - path = param.input_path - - if param.verbose: - print("Read parameter file") - param.param_list = read_param_file(param.param_path, verbose=param.verbose) - - # read (optional) input tile ID file - if param.tile_ID_list_path: - if param.verbose: - print("Read tile ID list") - tile_ID_list = cfis.read_list(param.tile_ID_list_path) - - if param.verbose: - print("Find input catalogue FITS files") - l = os.listdir(path=path) - ext = "fits" - lpath = [] - for this_l in l: - - add_this_l = False - - # mark to add if correct extension, matches input pattern, not `.npy` file - if ( - this_l.endswith(ext) - and (f"{param.input_name_base}" in this_l) - and (".npy" not in this_l) - ): - add_this_l = True - - # unmark to add if no in (optional) input tile ID file - if param.tile_ID_list_path: - nix, niy = cfis.get_tile_number(this_l) - tile_ID = f"{nix}.{niy}" - if tile_ID not in tile_ID_list: - add_this_l = False - if add_this_l: - lpath.append(os.path.join(path, this_l)) - - if param.verbose: - print(f"{len(lpath)} files files to merge found") - - count = 0 - - # Determine number of columns and keys from first catalogue file - d_tmp = get_data(lpath[0], param.hdu_num, param.param_list) - d = np.zeros(d_tmp.shape, dtype=d_tmp.dtype) - for key in d_tmp.dtype.names: - d[key] = d_tmp[key] - count = count + 1 - if param.verbose: - print(f"File '{lpath[0]}' copied ({count}/{len(lpath)})") - - # merge remaining catalogue files - for fname in lpath[1:]: - - try: - d_tmp = get_data(fname, param.hdu_num, param.param_list) - dd = np.zeros(d_tmp.shape, dtype=d.dtype) - - for key in d_tmp.dtype.names: - dd[key] = d_tmp[key] - - count = count + 1 - if param.verbose: - print(f"File '{fname}' copied ({count}/{len(lpath)})") - - d = np.concatenate((d, dd)) - except: - print( - f"Error while adding file '{fname}', {len(dd)} objects" - " not in final cat" - ) - - # Save merged catalogue as numpy binary file - if param.verbose: - print("Saving merged catalogue") - np.save(f"{param.input_name_base}.npy", d) - - msg = f"{count} catalog files merged with success" - if param.verbose: - print(msg) - print(msg, file=f_log) - - f_log.close() - - return 0 - - -if __name__ == "__main__": - sys.exit(main(sys.argv)) diff --git a/scripts/python/summary_tiles.py b/scripts/python/summary_tiles.py deleted file mode 100755 index 9addbb017..000000000 --- a/scripts/python/summary_tiles.py +++ /dev/null @@ -1,183 +0,0 @@ -#!/usr/bin/env python - -import sys -import os -import re -import numpy as np - -from tqdm import tqdm - - -def get_input_IDs_patch(tiles, patch): - - # Read in files from summary output with missing CCDs. - # Create and update CCD name set. - fname = f"{patch}/tile_numbers.txt" - with open(fname) as f: - lines = f.readlines() - for line in lines: - ID = line.rstrip("\n") - tiles.append( - { - "ID": ID, - "patch": patch, - } - ) - - return tiles - - -def check_zero_weight(tiles, patches): - - for patch in tqdm(patches, desc="check zero weight"): - - fname = f"{patch}/summary/special_job_16_sextractor_runner_5.txt" - - if os.path.exists(fname): - - with open(fname) as f: - lines = f.readlines() - for line in lines: - m = re.search("process-(\d{3})-(\d{3})\.log", line) - if m: - ID = f"{m[1]}.{m[2]}" - my_tile = next((tile for tile in tiles if tile.get("ID") == ID), None) - my_tile["zero_weight"] = 1 - - return tiles - - -def check_exceptions(tiles, patches): - - for patch in tqdm(patches, desc="check processing exceptions"): - - fname = f"{patch}/exceptions.txt" - - if os.path.exists(fname): - - with open(fname) as f: - lines = f.readlines() - for line in lines: - m = re.search("(\d{3})\.(\d{3})\s+\t+(.*)", line) - if m: - ID = f"{m[1]}.{m[2]}" - my_tile = next((tile for tile in tiles if tile.get("ID") == ID), None) - my_tile[m[3]] = 1 - - return tiles - - -def get_final(tiles, patches): - - n_final = {} - - for patch in tqdm(patches, desc="check processing exceptions"): - - n_final[patch] = 0 - - fname = f"{patch}/n_tiles_final.txt" - if os.path.exists(fname): - with open(fname) as f: - lines = f.readlines() - for line in lines: - num = line.rstrip("\n") - n_final[patch] = int(num) - - n_final["all"] = sum(n_final.values()) - - return n_final - - -def fill_True(tiles, keys): - - - for tile in tiles: - for key in keys: - tile[key] = 0 - - return tiles - - -def summary(patches, keys): - - tiles = [] - - n_final = get_final(tiles, patches) - - # Loop over patches - for patch in tqdm(patches, desc="read input tile lists"): - tiles = get_input_IDs_patch(tiles, patch) - - tiles = fill_True(tiles, keys) - - tiles = check_zero_weight(tiles, patches) - - tiles = check_exceptions(tiles, patches) - - return tiles, n_final - -def print_tile(tile, f_out, keys): - - for key in keys: - print(f"{tile[key]}", file=f_out, end=" ") - print(file=f_out) - - -def output_tiles(tiles, path, keys): - - with open(path, "w") as f_out: - - for tile in tiles: - print_tile(tile, f_out, keys) - - -def print_summary(tiles, patches, n_final, keys, keys_sum, nform): - - for key in keys_sum: - print(key.rjust(nform, " "), end="") - print("tot_miss".rjust(nform, " ")) - - for patch in patches: - tiles_patch = [entry for entry in tiles if entry.get("patch") == patch] - print_summary_patch(tiles_patch, patch, n_final[patch], keys, nform) - print_summary_patch(tiles, "all", n_final["all"], keys, nform) - -def print_summary_patch(tiles, patch, n_final, keys, nform): - - n_tiles = len(tiles) - print(patch.rjust(nform, " "), end="") - print(str(n_tiles).rjust(nform, " "), end="") - print(str(n_final).rjust(nform, " "), end="") - n_all = 0 - for key in keys: - n = sum(v[key] for v in tiles) - print(str(n).rjust(nform, " "), end="") - n_all += n - print(str(n_all).rjust(nform, " "), f"{n_all / n_tiles:10.2%}", end="") - print() - - -def main(argv): - - n_patch = 8 - patches = [f'P{x}' for x in np.arange(n_patch) + 1] - - keys = ["zero_weight", "aborted", "large_num_det"] - - nform = 15 - - tiles, n_final = summary(patches, keys) - - output_path = "tile_summary.txt" - - keys_ext = ["ID", "patch"] + keys - output_tiles(tiles, output_path, keys_ext) - - keys_sum = ["patch", "n_tiles", "n_final"] + keys - print_summary(tiles, patches, n_final, keys, keys_sum, nform) - - return 0 - - -if __name__ == "__main__": - sys.exit(main(sys.argv)) diff --git a/scripts/sh/build_and_plot_coverage_maps.sh b/scripts/sh/build_and_plot_coverage_maps.sh deleted file mode 100755 index b7ea9c8b2..000000000 --- a/scripts/sh/build_and_plot_coverage_maps.sh +++ /dev/null @@ -1,95 +0,0 @@ -#!/bin/bash - -set -euo pipefail - -# Build and plot per-CCD coverage (nexp) maps for a range of catalogue -# versions. Each map counts, per sky pixel, the number of exposures with a -# valid PSF model. The per-version flow is: -# 0. get_ccds_with_psf -> ccds_with_psf_.txt (valid CCD IDs) -# 1. download_headers -> per-exposure header text files -# 2. extract_field_corners-> exp_ra_dec_.txt (per-CCD corners) -# 3. build_coverage_map -> coverage_.x.hsp -# 4. plot_coverage_map -> SGC / NGC plots -# Steps 0-2 are expected to have been run already on canfar (they need -# VOSpace access); this script rebuilds from the extracted per-CCD corners and -# plots. Uncomment the marked lines to run the full chain. - -# Configuration variables - -## Catalogue versions -VERSIONS=("v1.3" "v1.4" "v1.5" "v1.6") - -## Output directory of plots -OUTPUT_DIR="coverage_map_plots" - -# healsparse resolution. nside=131072 gives ~0.1" per pixel, chosen to -# match the UNIONS bit-mask resolution so coverage and mask align pixel- -# wise. CoverageMapBuilder defaults to nside=2048 for lighter-weight -# offline use; override here for the production build. -BUILD_NSIDE=131072 -BUILD_CHANNELS=128 - -# Common parameters -VERBOSE="-v" - -## Colorbar -PLOT_COLORBAR="-C" -PLOT_MIN=1 -PLOT_MAX=5 - -# Plot parameters -## SGC region -SGC_RA_MIN=-20 -SGC_RA_MAX=45 -SGC_DEC_MIN=18 -SGC_DEC_MAX=40 - -## NGC region -NGC_RA_MIN=110 -NGC_RA_MAX=270 -NGC_DEC_MIN=28 -NGC_DEC_MAX=90 - -# Create output directory if it doesn't exist -mkdir -p "${OUTPUT_DIR}" - -# Loop over versions -for VERSION in "${VERSIONS[@]}"; do - echo "Processing ${VERSION}..." - - # Define file paths - CCD_LIST="ccds_with_psf_${VERSION}.txt" - HEADER_DIR="headers_${VERSION}" - INPUT_CORNERS="exp_ra_dec_${VERSION}.txt" - COVERAGE_MAP="coverage_${VERSION}.x.hsp" - - # Step 0-2: build the per-CCD corner file (canfar / VOSpace only). - # Uncomment to run the full chain from scratch. - # get_ccds_with_psf -V "${VERSION}" -o "${CCD_LIST}" ${VERBOSE} - # download_headers -i "${CCD_LIST}" -o "${HEADER_DIR}" ${VERBOSE} - # extract_field_corners -i "${HEADER_DIR}" -l "${CCD_LIST}" \ - # -o "${INPUT_CORNERS}" ${VERBOSE} - - # Build coverage map - echo " Building coverage map from ${INPUT_CORNERS}..." - CMD="build_coverage_map -i ${INPUT_CORNERS} -o ${COVERAGE_MAP} -c ${BUILD_CHANNELS} -n ${BUILD_NSIDE} ${VERBOSE}" - echo "$CMD" - $CMD - - # Plot SGC region - echo " Plotting SGC region..." - CMD="plot_coverage_map -i ${COVERAGE_MAP} -o ${OUTPUT_DIR}/coverage_${VERSION}_SGC.png ${VERBOSE} ${PLOT_COLORBAR} -R ${SGC_RA_MIN} -r ${SGC_RA_MAX} -D ${SGC_DEC_MIN} -d ${SGC_DEC_MAX} -m ${PLOT_MIN} -M ${PLOT_MAX}" - echo "$CMD" - $CMD - - # Plot NGC region - echo " Plotting NGC region..." - CMD="plot_coverage_map -i ${COVERAGE_MAP} -o ${OUTPUT_DIR}/coverage_${VERSION}_NGC.png ${VERBOSE} ${PLOT_COLORBAR} -R ${NGC_RA_MIN} -r ${NGC_RA_MAX} -D ${NGC_DEC_MIN} -d ${NGC_DEC_MAX} -m ${PLOT_MIN} -M ${PLOT_MAX}" - echo "$CMD" - $CMD - - echo " Done with ${VERSION}" - echo -done - -echo "All versions processed successfully!" diff --git a/scripts/sh/canfar_async_job.sh b/scripts/sh/canfar_async_job.sh deleted file mode 100755 index 7128e7734..000000000 --- a/scripts/sh/canfar_async_job.sh +++ /dev/null @@ -1,40 +0,0 @@ -#!/bin/bash - -source $HOME/shapepipe/scripts/sh/functions.sh - - -# 1. Extract the line from -f file -# Save original arguments -ARGS=("$@") - -# Find -f argument and get its value; remove -f and -S -FILE_IDS="" -NEW_ARGS=() -i=0 -while [[ $i -lt $# ]]; do - arg="${ARGS[i]}" - case "$arg" in - -f) - ((i++)) - FILE_IDS="${ARGS[i]}" - ;; - -S) - ((i++)) # skip next argument (value of -S) - ;; - *) - NEW_ARGS+=("$arg") - ;; - esac - ((i++)) -done - -if [[ -z "$FILE_IDS" ]]; then - echo "Error: -f must be provided" - exit 1 -fi - -# Get the line for this replica -LINE=$(get_line_from_file -f "$FILE_IDS") - -# 2. Call init_run.sh with remaining args + -e line -$HOME/shapepipe/scripts/sh/init_run_exclusive_canfar.sh "${NEW_ARGS[@]}" -e "$LINE" diff --git a/scripts/sh/combine_runs.bash b/scripts/sh/combine_runs.bash deleted file mode 100755 index 91b2ca42a..000000000 --- a/scripts/sh/combine_runs.bash +++ /dev/null @@ -1,224 +0,0 @@ -#!/usr/bin/env bash - -# Name: combine_runs.bash -# Description: Create new shapepipe run directory with -# links to source files from combined existing runs -# Author: Martin Kilbinger - - -# Command line arguments - -## Default values -cat='final' -psf="mccd" - -## Help string -usage="Usage: $(basename "$0") [OPTIONS] -\n\nOptions:\n - -h\tthis message\n - -p, --psf MODEL\n - \tPSF model, allowed are 'psfex', 'mccd', 'setools', default='$psf'\n - -c, --cat TYPE\n - \tCatalogue type, allowed are 'final', 'flag_tile', 'flag_exp', \n - \t'psf', 'psf_conv', 'image', 'shdu', default='$cat'\n -" - -## Parse command line -while [ $# -gt 0 ]; do - case "$1" in - -h) - echo -ne $usage - exit 0 - ;; - -c|--cat) - cat="$2" - shift - ;; - -p|--psf) - psf="$2" - shift - ;; - *) - echo -ne $usage - exit 1 - ;; - esac - shift -done - - -## Check options -if [ "$cat" != "final" ] \ - && [ "$cat" != "flag_tile" ] \ - && [ "$cat" != "tile_detection" ] \ - && [ "$cat" != "flag_exp" ] \ - && [ "$cat" != "psf" ] \ - && [ "$cat" != "psf_conv" ] \ - && [ "$cat" != "image" ] \ - && [ "$cat" != "shdu" ]; then - echo "cat (option -c) needs to be 'final', 'tile_detection', 'flag_tile', 'flag_exp', 'psf', 'psf_conv', 'shdu', or 'image'" - exit 2 -fi - -## Check options -if [ "$psf" != "psfex" ] \ - && [ "$psf" != "mccd" ] \ - && [ "$psf" != "setools" ]; then - echo "PSF (option -p) needs to be 'psfex' or 'mccd'" - exit 2 -fi - - -## Functions -function link_s () { - target=$1 - link_name=$2 - - if [ -L "$link_name" ]; then - echo "link with name $link_name already exists, skipping..." - let "n_skipped+=1" - else - echo "create link $target <- $link_name" - ln -s $target $link_name - let "n_created+=1" - fi -} - - -# Start program - -n_skipped=0 -n_created=0 - -pwd=`pwd` -out_base="output" - -# Set paths: -## run_out: target output new run directory -## run_in: source input run base directory -## module: source input module runner sub-directory -## pattern: source file pattern - -run_out="run_sp_combined_$cat" - -if [ "$cat" == "final" ]; then - - # v1 - #run_in="$pwd/$out_base/run_sp_tile_Mc_*" - # v2 - run_in="$pwd/tile_runs/*/$out_base/run_sp_tile_Mc_*" - - module="make_catalog_runner" - pattern="final_cat-*" - -elif [ "$cat" == "tile_detection" ]; then - - run_in="$pwd/P?/tile_runs/*/$out_base/run_sp_tile_Sx_*" - module="sextractor_runner" - pattern="sexcat-*" - -elif [ "$cat" == "flag_tile" ]; then - - # v1 - #run_in="$pwd/$out_base/run_sp_MaMa_*/mask_runner_run_1" - # v2 - run_in="$pwd/$out_base/run_sp_tile_Ma_*" - run_out="run_sp_Ma_tile" - - module="mask_runner" - pattern="pipeline_flag-*" - -elif [ "$cat" == "flag_exp" ]; then - - # v1 - #run_in="$pwd/$out_base/run_sp_MaMa_*/mask_runner_run_2" - # v2 - run_in="$pwd/$out_base/run_sp_exp_Ma_*" - run_out="run_sp_Ma_exp" - - module="mask_runner" - pattern="pipeline_flag-*" - -elif [ "$cat" == "image" ]; then - - run_in="$pwd/$out_base/run_sp_Git_*" - module="get_images_runner" - pattern="CFIS_image-*" - -elif [ "$cat" == "psf" ]; then - - #MKDEBUG TODO: add option - # v1 - #run_in="$pwf/$out_base/run_sp_exp_Pi_*" - # v2 - #run_in="$pwd/exp_runs/*/$out_base/run_sp_exp_Pi_*" - run_in="$pwd/exp_runs/*/$out_base/run_sp_exp_SxSePsfPi_*" - - pattern="validation_psf-*" - if [ "$psf" == "psfex" ]; then - module="psfex_interp_runner" - elif [ "$psf" == "setools" ]; then - module="setools_runner" - else - module="mccd_interp_runner" - fi - -elif [ "$cat" == "psf_conv" ]; then - - #run_in="$pwd/../P?" - run_in="$pwd" - pattern="validation_psf_conv-*" - module="psfex_interp_runner" - -elif [ "$cat" == "shdu" ]; then - - run_in="$pwd/$out_base/run_sp_exp_Sp_shdu_*" - module="split_exp_runner" - pattern="headers-*" - -else - - echo "Invalid catalogue type $cat" - exit 2 - -fi - - -OUTPUT="$pwd/$out_base/$run_out" -mkdir -p $OUTPUT - - -# Create links - -## target directory -outdir=$OUTPUT/$module/output -mkdir -p $outdir - -## identify source files - -# The following can result in an "Argument list too long" error -#FILES=(`find $run_in -type f -name "$pattern" -print0 | xargs -0 echo`) - -i=0 -for dir in $run_in; do - FILES=(`find -L $dir -type f -name "$pattern" -print0 | xargs -0 echo`) - - #echo "$dir $pattern" - - ## Look over source files - for file in ${FILES[@]}; do - - target=$file - link_name=$outdir/`basename $file` - link_s $target $link_name - ((i=i+1)) - - done - -done - -#echo " $n_files target files, $i links created/skipped" -echo " $i total, "$n_skipped skipped, "$n_created links created" - -# Update log file -update_runs_log_file.py diff --git a/scripts/sh/functions.sh b/scripts/sh/functions.sh index f6763c5b0..777f1a557 100644 --- a/scripts/sh/functions.sh +++ b/scripts/sh/functions.sh @@ -1,97 +1,12 @@ -# Global variables -SSL=~/.ssl/cadcproxy.pem -SESSION=https://ws-uv.canfar.net/skaha/v0/session -IMAGE=images.canfar.net/unions/shapepipe -NAME=shapepipe +# Shell helper sourced by run_job_sp_canfar_v2.0.bash. +# +# command() reads $VERBOSE, $debug_out, $pat and $STOP as free variables set +# by its caller; the two defaults below cover callers that set neither. -version="2.0" -cmd_remote="$HOME/shapepipe/scripts/sh/run_job_sp_canfar_v2.0.bash" pat="---- " STOP=0 -get_line_from_file() { - local file="" - local OPTIND opt - - # Parse options: look for -f - while getopts ":f:" opt; do - case $opt in - f) - file="$OPTARG" - ;; - \?) - echo "Unknown option: -$OPTARG", ", continuing" - #return 1 - ;; - :) - echo "Option -$OPTARG requires an argument." - return 1 - ;; - esac - done - - # Check that file and REPLICA_ID are set - if [[ -z "$file" ]]; then - echo "Error: -f option not provided." - return 1 - fi - if [[ -z "$REPLICA_ID" ]]; then - echo "Error: REPLICA_ID not set." - return 1 - fi - - # Check file exists - if [[ ! -f "$file" ]]; then - echo "Error: file '$file' not found." - return 1 - fi - - # Extract the line corresponding to REPLICA_ID (1-based index) - sed -n "${REPLICA_ID}p" "$file" -} - - - -# Add session and image IDs to log files -function update_session_logs() { - echo $my_session >> session_IDs.txt - echo "$my_session $ID" >> session_image_IDs.txt -} - -function call_curl() { - my_name=$1 - my_job=$2 - my_psf=$3 - my_ID=$4 - my_N_SMP=$5 - my_dry_run=$6 - my_dir=$7 - my_debug_out=$8 - my_scratch=$9 - my_test_arg=${10} - - my_arg="-j $my_job -p $my_psf -e $my_ID -N $my_N_SMP -n $my_dry_run -d $my_dir --debug_out $my_debug_out -S $my_scratch $my_test_arg" - - if [ "$my_dry_run" == "0" ]; then - my_session=`curl -E $SSL "$SESSION?$RESOURCES" -d "image=$IMAGE:$version" -d "name=${my_name}" -d "cmd=$cmd_remote" --data-urlencode "args=${my_arg[@]}"` - fi - - cmd=("curl" "-E" "$SSL" "$SESSION?$RESOURCES" "-d" "image=$IMAGE:$version" "-d" "name=${my_name}" "-d" "cmd=$cmd_remote" "--data-urlencode" "args=\"${my_arg}\"") - - if [ -n "$my_debug_out" ]; then - echo "${pat}call_curl $my_name $my_arg" >> $my_debug_out - echo "${pat}Running ${cmd[@]} (dry_run=$my_dry_run)" >> $my_debug_out - fi - echo "${cmd[@]} (dry_run=$my_dry_run)" - - - # Running $cmd does not work due to unknown problems with passing of args - - update_session_logs -} - - ## Print string, executes command, and prints return value. function command () { cmd=$1 @@ -142,57 +57,3 @@ function command () { fi fi } - - -function get_kind_from_job() { - my_job=$1 - - job_to_test=2 - kind="none" - - # loop over possible job numbers - while [ $job_to_test -le 1024 ]; do - - (( do_job = $job & $job_to_test )) - if [[ $do_job != 0 ]]; then - - if [ $job_to_test == 32 ]; then - if [ "$kind" == "tile" ]; then - echo "Error: Invalid job $job. mixing tile and exp kinds" - exit 6 - fi - - # job=32 -> set kind to exp - kind="exp" - elif [ $job_to_test == 2 ]; then - if [ "$kind" == "tile" ]; then - echo "Error: Invalid job $job. mixing tile and exp kinds" - exit 6 - fi - - kind="exp" - elif [ $job_to_test == 8 ]; then - if [ "$kind" == "tile" ]; then - echo "Error: Invalid job $job. mixing tile and exp kinds" - exit 6 - fi - - kind="exp" - else - if [ "$kind" == "exp" ]; then - echo "Error: Invalid job $job. mixing tile and exp kinds" - exit 6 - fi - - # job != 32 -> set kind to tile - kind="tile" - fi - - fi - - # Multiply job number by two to get next bitwise number - job_to_test=$((job_to_test * 2)) - done - - echo $kind -} diff --git a/scripts/sh/init_run_exclusive_canfar.sh b/scripts/sh/init_run_exclusive_canfar.sh deleted file mode 100755 index 556153b0d..000000000 --- a/scripts/sh/init_run_exclusive_canfar.sh +++ /dev/null @@ -1,642 +0,0 @@ -#!/bin/bash - -# init_run_exclusive_canfar.sh - -# Command line arguments -## Default values -job=-1 -ID=-1 -N_SMP=1 -dry_run=0 -dir=`pwd` -debug_out=-1 -scratch=-1 -fix=0 -test_only=0 -sm=1 - -# mh_local is 0 (1) if merge_header_runner is run on all exposures, -# which is standard so far (run on exposures of given tile only; new) -mh_local=1 - -# sp_local is 0 (1) is split_headers_runner and mask_runner is run -# on all exposures (locally). Not 100% automatic yet. -sp_local=1 -VERBOSE=1 - -pat="-- " - - -## Help string -usage="Usage: $(basename "$0") -j JOB -e ID [OPTIONS] -\n\nOptions:\n - -h\tthis message\n - -j, --job JOB\tRUnning JOB, bit-coded\n - -e, --exclusive ID - \timage ID\n - -p, --psf MODEL\n - \tPSF model, one in ['psfex'|'mccd'], default='$psf'\n - -m, --mh_local MH\n - \tmerge header file local (MH=1) or global (MH=0); default is $mh_local\n - -s, --sp_local SP\n - \tsplit local run local (SP=1) or global (SP=0); default is $sp_local\n - --sm SM\n - \tWith (SM=1; default) or without (SM=0) spread model input\n - -N, --N_SMP N_SMOp\n - \tnumber of jobs (SMP mode only), default from original config files\n - -d, --directory\n - \trun directory, default is pwd ($dir)\n - -S, --scratch\n - \tprocessing scratch directory, default is None ($scratch)\n - -F, --fix FIX\n - \tfix missing data (re-download tile, unzip) for FIX=1; default is $fix\n - -n, --dry_run LEVEL\n - \tdry run (LEVEL=1), no actual processing; default is $dry_run\n - --debug_out PATH\n - \tdebug output file PATH, default not used\n - --test\n - \ttest mode, no processing\n -" - -## Help if no arguments -if [ -z $1 ]; then - echo -ne $usage - exit 1 -fi - -## Parse command line -while [ $# -gt 0 ]; do - case "$1" in - -h) - echo -ne $usage - exit 0 - ;; - -j|--job) - job="$2" - shift - ;; - -e|--exclusive) - ID="$2" - shift - ;; - -p|--psf) - psf="$2" - shift - ;; - -m|--mh_local) - mh_local="$2" - shift - ;; - -s|--sp_local) - sp_local="$2" - shift - ;; - --sm) - sm="$2" - shift - ;; - -N|--N_SMP) - N_SMP="$2" - shift - ;; - -d|--directory) - dir="$2" - shift - ;; - -S|--scratch) - scratch="$2" - shift - ;; - -n|--dry_run) - dry_run="$2" - shift - ;; - -F|--fix) - fix="$2" - shift - ;; - --debug_out) - debug_out="$2" - shift - ;; - --test) - test_only=1 - ;; - esac - shift -done - - -# functions -function message() { - msg=$1 - my_debug_out=$2 - my_exit=$3 - - echo $msg - if [ "$my_debug_out" != "-1" ]; then - echo ${pat}$msg >> $my_debug_out - fi - - if [ "$my_exit" != "-1" ]; then - echo "${pat}exiting with code $my_exit" >> $my_debug_out - exit $my_exit - fi -} - -# Write an updated copy of a shapepipe config with NUMBER_LIST set to the -# given image ID, expressed in the numbering scheme (leading dash, dots -> -# dashes). Replaces the retired shapepipe_run -e/--exclusive flag (#746). -function set_config_number_list() { - local config_orig=$1 - local config_upd=$2 - local _id=$3 - - local number="-$(echo $_id | tr '.' '-')" - local config_tmp="${config_upd}.tmp" - - if grep -q "^NUMBER_LIST" "$config_orig"; then - perl -pe 's/^NUMBER_LIST\s*=.*/NUMBER_LIST = '$number'/' "$config_orig" > "$config_tmp" - else - perl -pe 's/^\[FILE\][ \t]*$/[FILE]\nNUMBER_LIST = '$number'/' "$config_orig" > "$config_tmp" - fi - if ! grep -q "^NUMBER_LIST = $number$" "$config_tmp"; then - echo "set_config_number_list: failed to set NUMBER_LIST in $config_orig" >&2 - exit 1 - fi - mv "$config_tmp" "$config_upd" -} - - -# Init message -message "test=$test_only" $debug_out -1 -if [ "$test_only" == "1" ]; then - echo "MKDEBUG test" - msg="init_run_exclusive.py script test mode, exiting." - ex=0 -else - msg="init_run_exclusive.py script processing mode, starting." - ex=-1 -fi -message "$msg" $debug_out $ex - - -## Check options -message "checking options" $debug_out -1 - -if [ "$job" == "-1" ]; then - message "No job indicated, use option -j" $debug_out 2 -fi - -if [ "$ID" == "-1" ]; then - message "No image ID indicated, use option -e" $debug_out 3 -fi - -if [ "$psf" != "psfex" ] && [ "$psf" != "mccd" ]; then - message "PSF (option -p) needs to be 'psfex' or 'mccd'" $debug_out 4 -fi - -if [ "$mh_local" != "0" ] && [ "$mh_local" != "1" ]; then - message "mh_local (option -m) needs to be 0 or 1" $debug_out 5 -fi - -if [ "$sp_local" != "0" ] && [ "$sp_local" != "1" ]; then - message "sp_local (option -m) needs to be 0 or 1" $debug_out 6 -fi - -if [ "$dry_run" != "0" ] && [ "$dry_run" != 1 ]; then - message "dry_run must be 0 or 1, not $dry_run" $debug_out 8 -fi - - -# Start script - -source $HOME/shapepipe/scripts/sh/functions.sh - -msg="Starting $(basename "$0") `date` ID=$ID" -message "$msg" $debug_out -1 -#message "`date`" $debug_out -1 -#message "ID=$ID" $debug_out -1 - -# Set kind -kind=$(get_kind_from_job $job) - -if [ "$kind" == "none" ]; then - message "Error: invalid job $job" $debug_out 5 -fi - -message "kind=$kind" $debug_out -1 - - -if [ "$dry_run" == "1" ]; then - message "running in dry run mode" $debug_out -1 -else - message "not running in dry run mode" $debug_out -1 -fi - -CONDA_PREFIX=$HOME/.conda/envs/shapepipe -PATH=$PATH:$CONDA_PREFIX/bin - -cd $dir - -if [ ! -d ${kind}_runs ]; then - command "mkdir ${kind}_runs" $dry_run -fi - -if [ "$fix" == "1" ]; then - message "Fixing missing data" $debug_out -1 - - message "Download tile images" $debug_out -1 - cd data_tiles - file_name=CFIS.$ID.r.fits - if [ ! -e $file_name ]; then - message "Downloading $file_name" $debug_out -1 - vcp vos:cfis/tiles_DR5/$file_name . - else - message "File $file_name exists, skipping" $debug_out -1 - fi - file_name=CFIS.$ID.r.weight.fits.fz - if [ ! -e $file_name ]; then - message "Downloading $file_name" $debug_out -1 - vcp vos:cfis/tiles_DR5/$file_name . - else - message "File $file_name exists, skipping" $debug_out -1 - fi - cd .. - - message "link to tiles..." $debug_out -1 - cd output/run_sp_GitFeGie_202*/get_images_runner_run_1/output - IDt=`echo $ID | tr "." "-"` - path_tiles=/arc/home/kilbinger/cosmostat/v2/pre_v2/$psf/$patch/data_tiles - link_name=CFIS_image-$IDt.fits - if [ ! -e $link_name ]; then - message "Creating link $link_name" $debug_out -1 - # force to overwrite broken link - ln -sf $path_tiles/CFIS.$ID.r.fits $link_name - else - message "Link $link_name exists, skipping" $debug_out -1 - fi - - link_name=CFIS_weight-$IDt.fitsfz - if [ ! -e $link_name ]; then - message "Creating link $link_name" $debug_out -1 - ln -sf $path_tiles/CFIS.$ID.r.weight.fits.fz $link_name - else - message "Link $link_name exists, skipping" $debug_out -1 - fi - - command "cd ../../../.." $dry_run - - message "Unzip weight ($dry_run)" $debug_out -1 - command "cd tile_runs/$ID" $dry_run - export SP_RUN=`pwd` - command "set_config_number_list cfis/config_tile_Uz.ini config_tile_Uz_upd.ini $ID" $dry_run - command "shapepipe_run -c config_tile_Uz_upd.ini" $dry_run - - cd $dir -else - message "Not fixing missing data" $debug_out -1 -fi - - -cd ${kind}_runs - -if [ ! -d "$ID" ]; then - command "mkdir $ID" $dry_run -fi - -cd $ID -pwd - -# Point cfis to local link, to be independent of platform -ln -sf ~/shapepipe/example/cfis - - -if [ ! -d "output" ]; then - command "mkdir output" $dry_run -fi - -cd output - - -# Update links to global run directories (GiFeGie) -# New: 27/11/2024: Remove link to Uz, conflict with fix -for my_dir in $dir/output/run_sp_[G]*; do - command "ln -sf $my_dir" $dry_run -done - -# The following could be done also with fix=1 -if [ "$fix" == "0" ] && [ "$kind" == "tile" ]; then - if [ ! -e "run_sp_Uz*" ]; then - - message "previous uncompress run not found..." $debug_out -1 - - # Remove broken links - for my_dir in run_sp_Uz*; do - if [ -L $my_dir ]; then - command "rm $my_dir" $dry_run - message "remove broken link $my_dir" $debug_out -1 - fi - done - - # Create working link - for my_dir in $dir/output/run_sp_Uz*; do - command "ln -sf $my_dir" $dry_run - message "create valid link $my_dir" $debug_out -1 - done - cd .. - command "update_runs_log_file.py" $dry_run - cd output - fi -fi - - -# Combined flags - -## Tiles -command "ln -sf $dir/output/run_sp_Ma_tile" $dry_run - -if [ "$sp_local" == "0" ]; then - # Exposures - command "ln -sf $dir/output/run_sp_Ma_exp" $dry_run -#else - #command "rm -f $dir/output/run_sp_Ma_exp" $dry_run -fi - -# Check for existing exp SpMh dir -dir_SpMh="run_sp_exp_SpMh" -if [[ -d "$dir/output/$dir_SpMh" ]]; then - # create link - command "ln -sf $dir/output/$dir_SpMh" $dry_run -else - message "No global SpMh directory found" $debug_out -1 - if [ -L "$dir_SpMh" ] && [ ! -e "$dir_SpMh" ]; then - message "Remove broken link $dir_SpMh" $debug_out -1 - rm $dir_SpMh - fi -fi - -(( do_job = $job & 2 )) -if [ $do_job != 0 ] && [ "$sp_local" == "1" ]; then - message "run local sp" $debug_out -1 - command "rm -rf run_sp_Gie*" $dry_run - command "rm -rf run_sp_exp_Sp*" $dry_run - - # Create new get_image dir - new_dir="run_sp_Gie/get_images_runner/output" - command "rm -rf $new_dir" $dry_run - command "mkdir -p $new_dir" $dry_run - - # Link to image, weight, and flag file of current exposure - - # Remove HDU extension - command "cd $new_dir" $dry_run - exp_ID=$(echo "$ID" | sed 's/-[0-9]\{1,2\}//') - for file in $dir/output//run_sp_GitFeGie_20*/get_images_runner_run_2/output/*${exp_ID}.* ; do - echo $file - command "ln -s $file" $dry_run - done - command "cd ../../.." $dry_run - - # Run Sp - command "cd .." $dry_run - if [ "$scratch" != "-1" ]; then - command "cd ../.." $dry_run - command "mkdir -p $scratch/exp_runs" $dry_run - command "cp -R exp_runs/$ID $scratch/exp_runs" $dry_run - command "cd $scratch/exp_runs/$ID" $dry_run - fi - command "update_runs_log_file.py" $dry_run - export SP_RUN=`pwd` - command "set_config_number_list cfis/config_exp_Sp.ini config_exp_Sp_upd.ini $exp_ID" $dry_run - command "shapepipe_run -c config_exp_Sp_upd.ini" $dry_run - - # Only keep CCD of this ID - command "mkdir -p output/run_sp_exp_Sp_shdu/split_exp_runner/output" $dry_run - command "mv output/run_sp_exp_Sp/split_exp_runner/output/*$ID.* output/run_sp_exp_Sp_shdu/split_exp_runner/output" $dry_run - command "mv output/run_sp_exp_Sp/split_exp_runner/output/headers* output/run_sp_exp_Sp_shdu/split_exp_runner/output" $dry_run - command "rm -rf output/run_sp_exp_Sp" $dry_run - if [ "$scratch" != "-1" ]; then - command "mv output/run_sp_exp_Sp_shdu $dir/exp_runs/$ID/output" $dry_run - command "cd .." $dry_run - command "rm -rf $ID" $dry_run - command "cd $dir" $dry_run - fi - command "update_runs_log_file.py" $dry_run - cd output - - if [ "$job" == "2" ]; then - msg="Finishing $(basename "$0") after job=2 `date` ID=$ID" - message "$msg" $debug_out 0 - fi - -fi - -if [ "$kind" == "tile" ] && [ "$sp_local" == "1" ]; then - echo "New (for P9): skipping link_to_exp_for_tile.py" - cd ../../.. - #command "link_to_exp_for_tile.py -t $ID -i tile_runs -I exp_runs -s $sp_local" $dry_run - cd tile_runs/$ID - #command "combine_runs.bash -p psfex -c shdu" $dry_run - cd output -fi - - -if [ "$mh_local" == "0" ]; then - if [ ! -f log_exp_headers.sqlite ]; then - # Global Mh and file does not exist -> symlink to - # gllobal mh file - message "creating global link to exp headers (mh_local=0)" $debug_out -1 - command "ln -s $dir/output/log_exp_headers.sqlite" $dry_run - else - message "global link to exp headers (mh_local=0) exists" $debug_out -1 - fi -else - # Local Mh - message "not creating global link to exp headers (mh_local=1)" $debug_out -1 - if [ "$ID" == "-1" ]; then - message "ID needs to be given (option -e) for mh_local" $debug_out 6 - fi - - # Check and remove symbolic (global) mh file link - if [ -L log_exp_headers.sqlite ]; then - # Local Mh and symlink -> remove previous link to - # (potentially incomplete) global mh file - message "Removing previous mh sym link" $debug_out -1 - command "rm log_exp_headers.sqlite" $dry_run - else - message "no mh link found" $debug_out -1 - fi - - # Check size of existing header file - if [ -e log_exp_headers.sqlite ]; then - size=$(stat -c %s log_exp_headers.sqlite) - if (( size > 15000 )); then - message "Found valid local mh file, continuing" $debug_out -1 - else - message "Existing local mh file looks invalid, deleting" $debug_out -1 - rm -f log_exp_headers.sqlite - fi - fi - - if [ ! -e log_exp_headers.sqlite ]; then - message "Creating local mh file" $debug_out -1 - - cd .. - command "update_runs_log_file.py" $dry_run - export SP_RUN=`pwd` - command "shapepipe_run -c cfis/config_exp_Mh.ini" $dry_run - cd output - else - message "Found local mh file, continuing" $debug_out -1 - fi -fi - - -(( do_job = $job & 8 )) -if [ $do_job != 0 ] && [ "$sp_local" == "1" ]; then - # Remove previous local Ma runs - message "cdsclient = $CDSCLIENT" $debug_out -1 - command "rm -rf run_sp_exp_Ma*" $dry_run -fi - -(( do_job = $job & 16 )) -if [[ $do_job != 0 ]]; then - # Remove previous Sx runs - command "rm -rf run_sp_tile_Sx_*" $dry_run -fi - -# Update links to exposure run directories, which were created in job 32 -(( do_job = $job & 64 )) -if [[ $do_job != 0 ]]; then - if [ "$kind" == "tile" ]; then - - # Remove previous runs of this job - rm -rf run_sp_tile_PsViSmVi* - fi -fi - -(( do_job = $job & 128 )) -if [[ $do_job != 0 ]]; then - - echo - - cat_ngmix="run_sp_tile_ngmix_Ng1u/ngmix_runner/output/ngmix-*.fits" - dir_ngmix_prev="run_sp_tile_ngmix_Ng1u_prev/ngmix_runner/output" - cat_ngmix_prev="$dir_ngmix_prev/ngmix-*.fits" - - # Remove if empty - if [ ! -s $cat_ngmix ]; then - echo "Removing empty file $cat_ngmix" - rm $cat_ngmix - fi - if [ ! -s $cat_ngmix_prev ]; then - echo "Removing empty file $cat_ngmix_prev" - rm $cat_ngmix_prev - fi - - # Check whether ngmix output exists - if [ -e $cat_ngmix ]; then - message "ngmix output catalogue exists" $debug_out -1 - - # Check whether previous ngmix directory and output cat exist - exists="1" - if [ ! -d $dir_ngmix_prev ]; then - exists="0" - elif [ ! -e "$cat_ngmix_prev" ]; then - exists="1" - fi - if [ "$exists" == "0" ]; then - message "Moving to previous batch-save dir (does not exist yet)" $debug_out -1 - command "mkdir -p $dir_ngmix_prev" $dry_run - command "mv $cat_ngmix $dir_ngmix_prev" $dry_run - else - # Compare file sizes - size_cat_ngmix=$(stat -c%s $cat_ngmix) - size_cat_ngmix_prev=$(stat -c%s $cat_ngmix_prev) - if [ "$size_cat_ngmix" -gt "$size_cat_ngmix_prev" ]; then - message "Moving to batch-save dir, overwriting smaller batch-save cat" $debug_out -1 - command "mv $cat_ngmix $dir_ngmix_prev" $dry_run - else - message "Previous batch-save dir not smaller, removing ngmix output" $debug_out -1 - command "rm $cat_ngmix" $dry_run - fi - fi - else - # Whether or not previous ngmix exists, job_sp_canfar will handle it - message "No ngmix output exists, continuing..." $debug_out -1 - fi - - echo -fi - -(( do_job = $job & 256 )) -if [[ $do_job != 0 ]]; then - - # Remove previous runs of this job - rm -rf run_sp_tile_Ms_20??-* - -fi - -(( do_job = $job & 512 )) -if [[ $do_job != 0 ]]; then - - # Remove previous runs of this job - rm -rf run_sp_tile_Mc_20??-* - -fi - - -cd .. - -# Update log file -command update_runs_log_file.py $dry_run - -echo -n "pwd: " -pwd - -echo -n "environment: " -echo $CONDA_PREFIX - -# To avoid (new?) qt error with setools (-j 32) -export DISPLAY=:1.0 - - -if [ "$scratch" != "-1" ]; then - # Copy inputs to scratch - command "mkdir -p $scratch/${kind}_runs" $dry_run - cd ../.. - command "pwd" $dry_run - command "cp -R ${kind}_runs/$ID $scratch/${kind}_runs" $dry_run - command "cd $scratch/${kind}_runs/$ID" $dry_run -fi - -command "job_sp_canfar.bash -p psfex -j $job -e $ID --n_smp $N_SMP --nsh_jobs $N_SMP --debug_out $debug_out --sm $sm " $dry_run - -if [ "$scratch" != "-1" ]; then - cd ../.. - if [ "$job" == "16" ]; then - command "mv ${kind}_runs/$ID/output/run_sp_Sx_* $dir/${kind}_runs/$ID/output" $dry_run - elif [ "$job" == "32" ]; then - command "mv ${kind}_runs/$ID/output/run_sp_exp_SxSe* $dir/${kind}_runs/$ID/output" $dry_run - elif [ "$job" == "64" ]; then - command "mv ${kind}_runs/$ID/output/run_sp_tile_PsViSm** $dir/${kind}_runs/$ID/output" $dry_run - elif [ "$job" == "128" ]; then - command "mv ${kind}_runs/$ID/output/run_sp_tile_ngmix* $dir/${kind}_runs/$ID/output" $dry_run - else - echo "scratch mode with job=$job not implemented yet, exiting..." - exit 8 - fi - - command "rm -rf ${kind}_runs/$ID" $dry_run - command "cd $dir/${kind}_runs/$ID" $dry_run - command "update_runs_log_file.py" $dry_run - command "cd $dir" $dry_run -fi - -cd $dir - -#msg="End $(basename "$0")" - -msg="Finished $(basename "$0") `date` ID=$ID" -message "$msg" $debug_out -1 diff --git a/scripts/sh/job_sp_canfar.bash b/scripts/sh/job_sp_canfar.bash deleted file mode 100755 index e64ed1e5f..000000000 --- a/scripts/sh/job_sp_canfar.bash +++ /dev/null @@ -1,610 +0,0 @@ -#!/bin/bash - -# Name: job_sp_canfar.bash -# Description: General script to process one or more tiles -# with all contributing exposures. -# This works as job submission script for -# the canfar batch system. -# called in interactive mode on a virtual -# machine. -# Author: Martin Kilbinger - - -# Command line arguments -## Default values -job=255 -config_dir=$HOME/shapepipe/example/cfis -psf='mccd' -retrieve='vos' -star_cat_for_mask='onthefly' -exclusive='' -results='cosmostat/kilbinger/results_v2' -n_smp=-1 -nsh_jobs=8 -debug_out=-1 -sm=1 - -pat="--- " - -## Help string -usage="Usage: $(basename "$0") [OPTIONS] [TILE_ID] -\n\nOptions:\n - -h\tthis message\n - -j, --job JOB\tRunning JOB, bit-coded\n - \t 1: retrieve images (online if method=vos)\n - \t 2: prepare images (offline)\n - \t 4: mask tiles (online if star_cat_for_mask=onthefly)\n - \t 8: mask exposures (online if star_cat_for_mask=onthefly)\n - \t 16: detection of galaxies on tiles (offline)\n - \t 32: processing of stars on exposures (offline)\n - \t 64: galaxy selection on tiles (offline)\n - \t 128: shapes and morphology (offline)\n - \t 256: paste catalogues (offline)\n - -c, --config_dir DIR\n - \t config file directory, default='$config_dir'\n - -p, --psf MODEL\n - \tPSF model, one in ['psfex'|'mccd'], default='$psf'\n - -r, --retrieve METHOD\n - \tmethod to retrieve images, allowed are 'vos', 'symlink', default='$retrieve'\n - -s, --star_cat_for_mask\n - \tcatalogue for masking bright stars, allowed are 'onthefly', 'save',\n - \tdefault is '${star_cat_for_mask}'\n - --sm SM\n - \tWith (SM=1; default) or without (SM=0) spread model input\n - -e, --exclusive ID\n - \texclusive input filer number string ID (default: None)\n - -o, --output_dir\n - \toutput (upload) directory on vos:cfis, default='$results'\n - -n, --n_smp N_SMP\n - \tnumber of jobs (SMP mode only), default from original config files\n - --nsh_jobs NJOB\n - \tnumber of objects per parallel shape module call, \n - \tdefault: optimal number is computed\n - --debug_out PATH\n - \tdebug output file PATH, default not used\n - TILE_ID_i\n - \ttile ID(s), e.g. 283.247 214.242, only with '-j 1'\n -" - -## Help if no arguments -if [ -z $1 ]; then - echo -ne $usage - exit 1 -fi - -## Parse command line -while [ $# -gt 0 ]; do - case "$1" in - -h) - echo -ne $usage - exit 0 - ;; - -j|--job) - job="$2" - shift - ;; - -c|--config_dir) - config_dir="$2" - shift - ;; - -p|--psf) - psf="$2" - shift - ;; - -r|--retrieve) - retrieve="$2" - shift - ;; - -s|--star_cat_for_mask) - star_cat_for_mask="$2" - shift - ;; - --sm) - sm="$2" - shift - ;; - -e|--exclusive) - exclusive="$2" - shift - ;; - -o|--output_dir) - results="$2" - shift - ;; - -n|--n_smp) - n_smp="$2" - shift - ;; - --nsh_jobs) - nsh_jobs="$2" - shift - ;; - --debug_out) - debug_out="$2" - shift - ;; - esac - shift -done - -## Check options -if [ "$psf" != "psfex" ] && [ "$psf" != "mccd" ]; then - echo "PSF (option -p) needs to be 'psfex' or 'mccd'" - exit 2 -fi - -if [ "$star_cat_for_mask" != "onthefly" ] && [ "$star_cat_for_mask" != "save" ]; then - echo "Star cat for mask (option -s) needs to be 'onthefly' or 'save'" - exit 4 -fi - -if [ "$retrieve" != "vos" ] && [ "$retrieve" != "symlink" ]; then - echo "method to retrieve images (option -r) needs to be 'vos' or 'symlink'" - exit 5 -fi - -if [ "$debug_out" != "-1" ]; then - echo $pat`date` >> $debug_out - echo "${pat}Starting $(basename "$0")" >> $debug_out -fi - -CONDA_PREFIX=/arc/home/kilbinger/.conda/envs/shapepipe -PATH=$PATH:$CONDA_PREFIX/bin - -# For tar archives. TODO: Should be unique to each job -export ID="test" - -## Paths - -## Path variables used in shapepipe config files - -# Run path and location of input image directories -export SP_RUN=`pwd` - -# Config file path -export SP_CONFIG=$SP_RUN/cfis -export SP_CONFIG_MOD=$SP_RUN/cfis_mod - -## Other variables - -# Output -OUTPUT=$SP_RUN/output - -# For tar archives -output_rel=`realpath --relative-to=. $OUTPUT` - -# Stop on error, default=1 -STOP=1 - -# Verbose mode (1: verbose, 0: quiet) -VERBOSE=1 - -# VCP options -export CERTFILE=$HOME/.ssl/cadcproxy.pem -export VCP="vcp --certfile=$CERTFILE" - - -## Functions - -# Print string, executes command, and prints return value. -function command () { - cmd=$1 - str=$2 - - RED='\033[0;31m' - GREEN='\033[0;32m' - NC='\033[0m' # No Color - # Color escape characters show up in log files - #RED='' - #GREEN='' - #NC='' - - - if [ "$debug_out" != "-1" ]; then - echo "${pat}pwd = `pwd`" >> $debug_out - echo "${pat}SP_RUN = $SP_RUN" >> $debug_out - echo "${pat}SP_CONFIG = $SP_CONFIG" >> $debug_out - fi - - if [ $# == 2 ]; then - if [ $VERBOSE == 1 ]; then - echo "$str: running '$cmd'" - fi - if [ "$debug_out" != "-1" ]; then - echo "${pat}Running $cmd" >> $debug_out - fi - - $cmd - - else - if [ $VERBOSE == 1 ]; then - echo "$str: running '$cmd $4 \"$5 $6\"'" - fi - if [ "$debug_out" != "-1" ]; then - echo "${pat}Running $cmd $4 \"$5 $6\"" >> $debug_out - fi - - $cmd $4 "$5 $6" - - fi - - res=$? - - if [ "$debug_out" != "-1" ]; then - echo "${pat}exit code = $res" >> $debug_out - fi - - if [ $VERBOSE == 1 ]; then - if [ $res == 0 ]; then - echo -e "${GREEN}success, return value = $res${NC}" - else - echo -e "${RED}error, return value = $res${NC}" - if [ $STOP == 1 ]; then - echo "${RED}exiting 'canfar_sp.bash', error in command '$cmd'${NC}" - exit $res - else - echo "${RED}continuing 'canfar_sp.bash', error in command '$cmd'${NC}" - fi - fi - fi -} - -# Run shapepipe command. If error occurs, upload sp log files before stopping script. -function command_sp() { - local cmd=$1 - local str=$2 - - command "$1" "$2" -} - -# Set up config file and call shapepipe_run -function command_cfg_shapepipe() { - local config_name=$1 - local str=$2 - local _n_smp=$3 - local _exclusive=$4 - - config_upd=$(set_config_n_smp $config_name $_n_smp) - - # Run a single image ID via NUMBER_LIST in an updated config copy; - # replaces the retired shapepipe_run -e/--exclusive flag (#746) - if [ "$_exclusive" != "" ]; then - set_config_number_list "$config_upd" "$SP_CONFIG_MOD/$config_name" "$_exclusive" - config_upd="$SP_CONFIG_MOD/$config_name" - fi - - local cmd="shapepipe_run -c $config_upd" - command_sp "$cmd" "$str" -} - -# Tar and upload files to vos -function upload() { - base=$1 - shift - ID=$1 - shift - verbose=$1 - shift - upl=("$@") - - echo "Counting upload files" - n_upl=(`ls -l ${upl[@]} | wc`) - if [ $n_upl == 0 ]; then - if [ $STOP == 1 ]; then - echo "Exiting script, no file found for '$base' tar ball" - exit 3 - fi - fi - tar czf ${base}_${ID}.tgz ${upl[@]} - command "$VCP ${base}_${ID}.tgz vos:cfis/$results" "Upload tar ball" -} - -# Upload log files -function upload_logs() { - id=$1 - verbose=$2 - - upl="$output_rel/*/*/logs $output_rel/*/logs" - upload "logs" "$id" "$verbose" "${upl[@]}" -} - -function set_config_n_smp() { - local config_name=$1 - local _n_smp=$2 - - local config_orig="$SP_CONFIG/$config_name" - - if [[ $_n_smp != -1 ]]; then - # Update SMP batch size - local config_upd="$SP_CONFIG_MOD/$config_name" - update_config $config_orig $config_upd "SMP_BATCH_SIZE" $_n_smp - else - # Keep original config file - local config_upd=$config_orig - fi - - # Set "return" value (stdout) - echo "$config_upd" -} - -# Update config file -function update_config() { - local config_orig=$1 - local config_upd=$2 - local key=$3 - local val_upd=$4 - - cat $config_orig \ - | perl -ane 's/'$key'\s+=.+/'$key' = '$val_upd'/; print' > $config_upd -} - -# Write an updated copy of a shapepipe config with NUMBER_LIST set to the -# given image ID, expressed in the numbering scheme (leading dash, dots -> -# dashes). Replaces the retired shapepipe_run -e/--exclusive flag (#746). -function set_config_number_list() { - local config_orig=$1 - local config_upd=$2 - local _id=$3 - - local number="-$(echo $_id | tr '.' '-')" - local config_tmp="${config_upd}.tmp" - - if grep -q "^NUMBER_LIST" "$config_orig"; then - perl -pe 's/^NUMBER_LIST\s*=.*/NUMBER_LIST = '$number'/' "$config_orig" > "$config_tmp" - else - perl -pe 's/^\[FILE\][ \t]*$/[FILE]\nNUMBER_LIST = '$number'/' "$config_orig" > "$config_tmp" - fi - if ! grep -q "^NUMBER_LIST = $number$" "$config_tmp"; then - echo "set_config_number_list: failed to set NUMBER_LIST in $config_orig" >&2 - exit 1 - fi - mv "$config_tmp" "$config_upd" -} - -### Start ### - -echo "Start processing" - -# Create input and output directories -mkdir -p $SP_RUN -cd $SP_RUN -mkdir -p $OUTPUT -mkdir -p $SP_CONFIG_MOD - -# Processing - - -### Retrieve config files -if [[ $config_dir == *"vos:"* ]]; then - command_sp "$VCP $config_dir ." "Retrieve shapepipe config files" -else - if [[ ! -L cfis ]]; then - command_sp "ln -s $config_dir cfis" "Retrieve shapepipe config files" - fi -fi - - -## Retrieve config files and images (online if retrieve=vos) -## Retrieve and save star catalogues for masking (if star_cat_for_mask=save) -(( do_job = $job & 1 )) -if [[ $do_job != 0 ]]; then - - ### Retrieve files - command_cfg_shapepipe \ - "config_GitFeGie_$retrieve.ini" \ - "Retrieve images" \ - -1 \ - $exclusive - - ### Retrieve and save star catalogues for masking - if [ "$star_cat_for_mask" == "save" ]; then - #### For tiles - mkdir $SP_RUN/star_cat_tiles - command_sp \ - "create_star_cat $SP_RUN/output/run_sp_GitFeGie_*/get_images_runner_run_1/output $SP_RUN/star_cat_tiles" \ - "Save star cats for masking (tile)" - - #### For single-exposures - mkdir $SP_RUN/star_cat_exp - command_sp \ - "create_star_cat $SP_RUN/output/run_sp_GitFeGie_*/get_images_runner_run_2/output $SP_RUN/star_cat_exp exp" \ - "Save star cats for masking (exp)" - fi - -fi - -## Prepare images (offline) -(( do_job = $job & 2 )) -if [[ $do_job != 0 ]]; then - - ### Uncompress tile weights - command_cfg_shapepipe "config_tile_Uz.ini" "Run shapepipe (uncompress tile weights)" $n_smp $exclusive - - ### Split images into single-HDU files, merge headers for WCS info - command_cfg_shapepipe \ - "config_exp_SpMh.ini" \ - "Run shapepipe (split images, merge headers)" \ - $n_smp \ - $exclusive - -fi - -## Mask tiles: add star, halo, and Messier object masks (online if "star_cat_for_mask" is "onthefly") -(( do_job = $job & 4 )) -if [[ $do_job != 0 ]]; then - - ### Mask tiles - command_cfg_shapepipe \ - "config_tile_Ma_$star_cat_for_mask.ini" \ - "Run shapepipe (mask tiles)" \ - $n_smp \ - $exclusive - -fi - -## Mask exposures: add star, halo, and Messier object masks (online if "star_cat_for_mask" is "onthefly") -(( do_job = $job & 8 )) -if [[ $do_job != 0 ]]; then - - ### Mask exposures - command_cfg_shapepipe \ - "config_exp_Ma_$star_cat_for_mask.ini" \ - "Run shapepipe (mask exposures)" \ - $n_smp \ - $exclusive - -fi - - -## Remaining exposure processing (offline) -(( do_job = $job & 16 )) -if [[ $do_job != 0 ]]; then - - ### Object detection on tiles - command_cfg_shapepipe \ - "config_tile_Sx.ini" \ - "Run shapepipe (tile detection)" \ - $n_smp \ - $exclusive - -fi - -## Exposure processing (offline) -(( do_job = $job & 32 )) -if [[ $do_job != 0 ]]; then - - ### Star detection, selection, PSF model. setools can exit with an error for CCD with insufficient stars, - ### the script should continue - STOP=0 - command_cfg_shapepipe \ - "config_exp_${psf}.ini" \ - "Run shapepipe (exp $psf)" \ - $n_smp \ - $exclusive - STOP=1 - -fi - -## Process tiles up to shape measurement -(( do_job = $job & 64 )) -if [[ $do_job != 0 ]]; then - - ### PSF model letter: 'P' (psfex) or 'M' (mccd) - letter=${psf:0:1} - Letter=${letter^} - command_cfg_shapepipe \ - "config_tile_${Letter}iViSmVi_canfar.ini" \ - "Run shapepipe (tile PsfInterp=$Letter}: up to ngmix+galsim)" \ - $n_smp \ - $exclusive - -fi - -## Shape measurement (offline) -(( do_job = $job & 128 )) -if [[ $do_job != 0 ]]; then - - ### Prepare config files - n_min=0 - n_obj=`get_number_objects` - if [ "$n_obj" == "-1" ]; then - echo "No tile SExtractor run found, exiting after et_number_objects call" - exit 10 - fi - nsh_step=`echo "$(($n_obj/$nsh_jobs))"` - - n_max=$((nsh_step - 1)) - for k in $(seq 1 $nsh_jobs); do - cat $SP_CONFIG/config_tile_Ng_template_batch.ini | \ - perl -ane \ - 's/(ID_OBJ_MIN =) X/$1 '$n_min'/; s/(ID_OBJ_MAX =) X/$1 '$n_max'/; s/NgXu/Ng'$k'u/; s/X_interp/'$psf'_interp/g; print' \ - > $SP_CONFIG_MOD/config_tile_Ng${k}u.ini - n_min=$((n_min + nsh_step)) - if [ "$k" == $((nsh_jobs - 1)) ]; then - n_max=-1 - else - n_max=$((n_min + nsh_step - 1)) - fi - done - - ### Shapes, run $nsh_jobs parallel processes - VERBOSE=0 - for k in $(seq 1 $nsh_jobs); do - - # if output dir for subrun exists but no output: re-run - ngmix_run=$OUTPUT/"run_sp_tile_ngmix_Ng${k}u/ngmix_runner" - if [ -e "$ngmix_run" ]; then - ngmix_out="$ngmix_run/output" - n_out=`ls -rlt $ngmix_out | wc -l` - if [ "$n_out" -lt 2 ]; then - command \ - "rm -rf $OUTPUT/run_sp_tile_ngmix_Ng${k}u" \ - "Re-running existing empty ngmix subrun $k" - command_sp \ - "shapepipe_run -c $SP_CONFIG_MOD/config_tile_Ng${k}u.ini" \ - "Run shapepipe (tile: ngmix $k)" & - else - echo "Skipping existing non-empty ngmix subrun $k" - fi - else - command_sp \ - "shapepipe_run -c $SP_CONFIG_MOD/config_tile_Ng${k}u.ini" \ - "Run shapepipe (tile: ngmix $k)" & - fi - done - wait - VERBOSE=1 - -fi - -## Create final catalogues (offline) -(( do_job = $job & 256 )) -if [[ $do_job != 0 ]]; then - - cat $SP_CONFIG/config_merge_sep_cats_template.ini | \ - perl -ane \ - 's/(N_SPLIT_MAX =) X/$1 '$nsh_jobs'/; print' \ - > $SP_CONFIG_MOD/config_tile_Ms.ini - - ### Merge separated shapes catalogues - command_sp \ - "shapepipe_run -c $SP_CONFIG_MOD/config_tile_Ms.ini" \ - "Run shapepipe (tile: merge sep cats)" \ - "$VERBOSE" \ - "$ID" -fi - -(( do_job = $job & 512 )) -if [[ $do_job != 0 ]]; then - - # spread_model suffix for config file with or without SM input - if [ "$sm" == "0" ]; then - suff_sm="_nosm" - else - suff_sm="" - fi - - ### Merge all relevant information into final catalogue - command_cfg_shapepipe \ - "config_make_cat_$psf${suff_sm}.ini" \ - "Run shapepipe (tile: create final cat $psf)" \ - $n_smp \ - $exclusive - -fi - -# MKDEBUG: Putting Mh at the end for now, could be integrated before 16. -(( do_job = $job & 1024 )) -if [[ $do_job != 0 ]]; then - - command_cfg_shapepipe \ - "config_exp_Mh.ini" \ - "Run shapepipe (merge exp headers)" \ - $n_smp \ - $exclusive - -fi - -if [ "$debug_out" != "-1" ]; then - echo "${pat}End $(basename "$0") ID=$exclusive success" >> $debug_out -fi diff --git a/scripts/sh/job_sp_canfar_v2.0.bash b/scripts/sh/job_sp_canfar_v2.0.bash index 592cae70d..e0fb28a01 100755 --- a/scripts/sh/job_sp_canfar_v2.0.bash +++ b/scripts/sh/job_sp_canfar_v2.0.bash @@ -1,6 +1,6 @@ #!/bin/bash -# Name: job_sp_canfar.bash +# Name: job_sp_canfar_v2.0.bash # Description: General script to process one or more tiles # with all contributing exposures. # This works as job submission script for diff --git a/scripts/sh/post_proc_sp.bash b/scripts/sh/post_proc_sp.bash deleted file mode 100755 index 930d5b79d..000000000 --- a/scripts/sh/post_proc_sp.bash +++ /dev/null @@ -1,86 +0,0 @@ -#!/usr/bin/env bash - -# Name: post_proc_sp.bash -# Description: Post-process downloaded canfar results and -# creates final merge catalog. -# Author: Martin Kilbinger -# Date: 06/2020 -# Package: shapepipe - -# The shapepipe python virtual environment needs to -# be active to run this script. - - -# Command line arguments - -## Default values -psf='mccd' - -## Help string -usage="Usage: $(basename "$0") [OPTIONS] -\n\nOptions:\n - -h\tthis message\n - -p, --psf MODEL\n - \tPSF model, one in ['psfex'|'mccd'], default='$psf'\n -" - -## Parse command line -while [ $# -gt 0 ]; do - case "$1" in - -h) - echo -ne $usage - exit 0 - ;; - -p|--psf) - psf="$2" - shift - ;; - *) - echo -ne $usage - exit 1 - ;; - esac - shift -done - -## Check options -if [ "$psf" != "psfex" ] && [ "$psf" != "mccd" ]; then - echo "PSF (option -p) needs to be 'psfex' or 'mccd'" - exit 2 -fi - -# Paths -export SP_RUN=. -SP_BASE=$HOME/astro/repositories/github/shapepipe -SP_CONFIG=$SP_BASE/example/cfis - - -# To download results from canfar, use -# -# canfar_download_results.sh -# -# On candide this needs to be done on -# the login node. - -# To Un-tar all .tgz results files, use -# -# $SP_BASE/scripts/sh/untar_results.sh - - -# PSF - -## Collect all psfinterp results -combine_runs -p $psf -t psf - -## Merge all psfinterp results and compute PSF residuals -shapepipe_run -c $SP_CONFIG/config_MsPl_$psf.ini - - -# Galaxies - -## Prepare output directory with links to all 'final_cat' result files -combine_runs - -## Merge final output files to single mother catalog -input_final=output/run_sp_combined/make_catalog_runner/output -merge_final_cat -i $input_final -p $SP_CONFIG/final_cat.param -v diff --git a/scripts/sh/remove_duplicates_tiles.sh b/scripts/sh/remove_duplicates_tiles.sh deleted file mode 100755 index 6f789fe6e..000000000 --- a/scripts/sh/remove_duplicates_tiles.sh +++ /dev/null @@ -1,55 +0,0 @@ -#!/usr/bin/env bash - -# Script to remove duplicate tile runs, keeping only the newest -# Run from directory containing P?/tile_runs subdirectories - -count=0 -total_removed=0 - -echo "Starting duplicate tile run removal..." -echo "Looking for patches in P?/tile_runs structure..." -echo "" - -# Process each patch directory -for patch_dir in P*/tile_runs; do - if [ ! -d "$patch_dir" ]; then - continue - fi - - patch_name=$(dirname "$patch_dir") - echo "Processing patch: $patch_name" - - # Process each tile directory within the patch - for dir in "$patch_dir"/*; do - if [ ! -d "$dir" ]; then - continue - fi - - # Check each of the three run types - for run_type in run_sp_tile_PsViSmVi run_sp_tile_Mc run_sp_tile_Ms; do - if compgen -G "$dir/output/${run_type}*" > /dev/null; then - n=$(ls -dt "$dir/output/${run_type}"* 2>/dev/null | wc -l) - if [ "$n" -gt 1 ]; then - ((n_remove=n-1)) - echo " Found $n copies of $run_type in $(basename "$dir"), removing $n_remove oldest" - - # Remove all but the newest - ls -dt "$dir/output/${run_type}"* | tail -n "$n_remove" | while read old_run; do - echo " Removing: $(basename "$old_run")" - rm -rf "$old_run" - ((total_removed++)) - done - - ((count++)) - fi - fi - done - done - echo "" -done - -echo "========================================" -echo "Summary:" -echo " Processed tile directories: $count" -echo " Total run directories removed: $total_removed" -echo "========================================" diff --git a/src/shapepipe/canfar/canfar_submit.py b/src/shapepipe/canfar/canfar_submit.py deleted file mode 100644 index 685584b99..000000000 --- a/src/shapepipe/canfar/canfar_submit.py +++ /dev/null @@ -1,576 +0,0 @@ -# canfar_submit_job.py - -# Submit job with the canfar client - -import os -import sys -import asyncio -import math - -from canfar.sessions import Session -from canfar.sessions import AsyncSession -from datetime import datetime - -from cs_util import args as cs_args -from cs_util import logging - - -class Job(object): - """Class Job. - - Handles job submission with the canfar system. - - """ - - def __init__(self): - - self.params_default() - - self._patch = os.environ["patch"] if "patch" in os.environ else "P0" - - # Maximum replicas per batch (CANFAR limit is 512) - self._max_replicas_per_batch = 512 - if self._max_replicas_per_batch != 512: - print(f"Setting the maximum number of replicas to the non-standard {self._max_replicas_per_batch}") - - # Default number of cores, can be overwritten by -c or -P (parallel jobs) options - self._cores_default = 2 - - - def set_params_from_command_line(self, args): - """Set Params From Command line. - - Only use when calling using python from command line. - Does not work from ipython or jupyter. - - """ - # Read command line options - options = cs_args.parse_options( - self._params, - self._short_options, - self._types, - self._help_strings, - ) - self._params = options - - # Save calling command - logging.log_command(args) - - def params_default(self): - """Params Default. - - Set default parameters. - - """ - self._params = { - "job": -1, - "exclusive": None, - "file_IDs": None, - "psf": "psfex", - "n": "0", - "debug_out": None, - "dry_run": 0, - "log": "log_jobs.txt", - "sync": "async", - "cores": 2, - "ram": 4, - "parallel_jobs": 1, - "jobs_per_session": 0, - "stats": False, - "version": "1.1", - } - - self._short_options = { - "job": "-j", - "exclusive": "-e", - "file_IDs": "-f", - "psf": "-p", - "dry_run": "-n", - "log": "-l", - "sync": "-S", - "cores": "-c", - "ram": "-r", - "parallel_jobs": "-P", - "jobs_per_session": "-J", - "stats": "-s", - "version": "-V", - } - - self._types = { - "job": "int", - "cores": "int", - "ram": "int", - "parallel_jobs": "int", - "jobs_per_session": "int", - "stats": "bool", - } - - self._help_strings = { - "job": "Running JOB, bit-coded, default is {}", - "exclusive": "Run exclusively given image", - "file_IDs": "file containing IDs", - "psf": "PSF model, allowed are 'psfex' and 'mccd'; default is {}", - "dry_run": "if dry run > 0 no actual processing; allowed are 2, 1, 0; default is {}", - "debug_out": "debug output file path, default:set automatically", - "log": "output log file with job and tile IDs", - "sync": "job submission mode, allowed are async, sync; default is {}", - "cores": "number of CPU cores per replica; default is {} or overwritten by -P", - "ram": "RAM in GB per replica; default is {}", - "parallel_jobs": "number of tiles to process in parallel per replica; default is {}", - "jobs_per_session": ( - "number of jobs (tiles) per session; if 0, create one session per job (default is {});" - + " Number of replicas is #FILE_IDs / JOBS_PER_SESSION" - ), - "stats": "print stats of sessions/batches/replicas and exit", - "version": "shapepipe image version; default is {}", - } - - def update_params(self): - """Update Params. - - Update parameters. - """ - - # Set version and image - version = self._params["version"] - self._image = f"images.canfar.net/unions/shapepipe:{version}" - - # If not given, set number of cores to number of parallel jobs; - # else set to default. - if self._params["cores"] == -1: - if self._params["parallel_jobs"] != 1: - self._params["cores"] = self._params["parallel_jobs"] - else: - self._params["cores"] = self._cores_default - - def check_params(self): - """Check Params. - - Check command line arguments and parameters. - - Returns - ------- - bool - status of checks; True (False) if passed (failed) - - """ - if self._params["job"] == -1: - raise ValueError(f"need to specify job (option -j)") - - if self._params["sync"] == "async" and not self._params["file_IDs"]: - raise ValueError("asynchronous mode only possible when tile_IDs file given (-f)") - - if not self._params["exclusive"] and not self._params["file_IDs"]: - raise ValueError("No image ID(s) indicated, use option -e ID or -f file_IDs") - - def set_options_base(self): - """Set Options Base. - - Set basic options from command line options and parameters. - - """ - opt = {} - - cwd = os.getcwd() - - # asynchronous mode - if self._params["sync"] == "async": - opt["-f"] = f"-f {cwd}/{self._params['file_IDs']}" - - # Options - opt["j"] = f"-j {self._params['job']}" - opt["p"] = f"-p {self._params['psf']}" - - # Set identifier for debug file name - if self._params["exclusive"]: - deb_ID = self._params["exclusive"] - else: - deb_ID = os.path.basename(self._params["file_IDs"]) - - if self._params['debug_out']: - opt["debug_out"] = f"--debug_out {cwd}/debug/{self._params['debug_out']}" - else: - opt["debug_out"] = f"--debug_out {cwd}/debug/debug_{deb_ID}.txt" - - opt["d"] = f"-d {cwd}" - opt["m_s"] = "-m 1 -s 1" if self._patch in ("P8", "P9") else "" - if self._params["dry_run"] != 0: - opt["n"] = f"-n {self._params['dry_run']}" - if self._params["parallel_jobs"] > 1: - opt["P"] = f"--parallel_jobs {self._params['parallel_jobs']}" - - options = "" - for key in opt: - options = f"{options} {opt[key]}" - - self._options_base = options - - def set_job_name(self, suf=None): - """Set Job Name. - - Set job name as it will appear on the canfar submission system. - - Parameters - ---------- - suf : str, optional - suffix, default is ``None`` - - """ - if suf is None: - suf = self._params["exclusive"] - - suf1 = suf.replace(".", "-") - - job_name = f"sp-{self._patch}-j{self._params['job']}-{suf1}" - - return job_name - - def set_command(self, mode): - """Set Command. - - Set shell command to run. - - Parameters - ---------- - mode: str - job submission mode, allowed are "async_single" (for asynchronous - submission of single job), "async_bulk" (asynchronous submission of many jobs), - "sync" (synchronous submission of one or more jobs) - - """ - if mode == "async_single": - self._cmd = f"{os.environ['HOME']}/shapepipe/scripts/sh/canfar_async_job.sh" - elif mode == "async_bulk": - self._cmd = f"{os.environ['HOME']}/shapepipe/scripts/python/distribute_tiles.py" - elif mode == "sync": - self._cmd = f"{os.environ['HOME']}/shapepipe/scripts/sh/init_run_exclusive_canfar.sh" - - def get_tile_IDs(self): - """Get Tile IDs. - - Return tile ID information. - - Returns - ------- - str - tile_ID(s) - int - length of tile ID list - str - suffix - - """ - if self._params["exclusive"]: - tile_IDs = [self._params["exclusive"]] - suf = None - else: - with open(self._params["file_IDs"], "r") as f: - tile_IDs = [line.strip() for line in f] - suf = "" - - return tile_IDs, len(tile_IDs), suf - - async def run_async(self): - """Run Async. - - Run asynchronous job submission. - - """ - async with AsyncSession() as session: - - _, total_n, suf = self.get_tile_IDs() - - # Calculate number of replicas (sessions) based on jobs_per_session - if self._params["jobs_per_session"] > 0: - # Each replica processes multiple jobs - num_replicas = math.ceil(total_n / self._params["jobs_per_session"]) - print(f"Distributing {total_n} jobs across {num_replicas} sessions") - print(f"Each session will process ~{self._params['jobs_per_session']} jobs") - else: - # Old behavior: one replica per job - num_replicas = total_n - print(f"Creating {num_replicas} sessions (one per job)") - - # Calculate number of batches needed - num_batches = math.ceil(num_replicas / self._max_replicas_per_batch) - - if num_batches == 1: - # Submit single batch - self.set_command(mode="async_bulk") - print(f"Submitting {num_replicas} replicas in a single batch") - return await self._submit_single_batch(num_replicas, total_n) - else: - # Submit multiple batches as chunks - self.set_command(mode="async_bulk") - print(f"Splitting into {num_batches} batches (max {self._max_replicas_per_batch} replicas per batch)") - return await self._submit_multiple_batches(num_replicas, total_n, num_batches) - - async def _submit_single_batch(self, num_replicas, total_n): - """Submit Single Batch. - - Submit a single batch of jobs. - - Parameters - ---------- - num_replicas: int - number of replicas (sessions) to create - total_n: int - total number of jobs (tiles) to process - - Returns - ------- - list - IDs of submitted sessions - """ - print(f"Submitting {num_replicas} replicas to process {total_n} jobs") - - job_name = self.set_job_name(suf="") - options = self._options_base.lstrip() - print(f"Running '{self._cmd} {options}'") - - async with AsyncSession() as session: - try: - sessions = await session.create( - name=job_name, - image=self._image, - cmd=self._cmd, - args=options, - replicas=num_replicas, - cores=self._params["cores"], - ram=self._params["ram"], - ) - print(f"✓ Batch submitted successfully: {len(sessions)} sessions created") - print(f"Each session will process ~{math.ceil(total_n / num_replicas)} jobs using chunk()") - print("Sessions = ", sessions) - except Exception as e: - print(f"❌ CANFAR session.create() failed: {type(e).__name__}: {e}") - raise - - return sessions - - async def _submit_multiple_batches(self, num_replicas, total_n, num_batches): - """Submit Multiple Batches. - - Submit multiple batches of jobs. - - Parameters - ---------- - num_replicas: int - total number of replicas (sessions) to create - total_n: int - total number of jobs (tiles) to process - num_batches: int - number of batches to split replicas into - - Each batch creates a subset of replicas. chunk() automatically - distributes all tiles across all replicas. - - Returns - ------- - list - IDs of submitted sessions - - """ - all_sessions = [] - - async with AsyncSession() as session: - for batch_num in range(1, num_batches + 1): - # Calculate batch size (number of replicas in this batch) - if batch_num < num_batches: - batch_size = self._max_replicas_per_batch - else: - # Last batch gets the remainder - batch_size = num_replicas - (num_batches - 1) * self._max_replicas_per_batch - - print(f"\n--- Batch {batch_num}/{num_batches} ---") - print(f"Submitting {batch_size} replicas") - print(f"Total {num_replicas} replicas will process {total_n} jobs") - print(f"chunk() will distribute jobs across all replicas") - - job_name = self.set_job_name(suf=f"b{batch_num}") - options = ( - f"{self._options_base} --batch_num {batch_num}" - + f" --batch_tot {num_batches}" - + f" --batch_size {batch_size}" - ) - options = options.lstrip() - print(f"Running '{self._cmd} {options}'") - - try: - sessions = await session.create( - name=job_name, - image=self._image, - cmd=self._cmd, - args=options, - replicas=batch_size, - cores=self._params["cores"], - ram=self._params["ram"], - - ) - print(f"✓ Batch {batch_num} submitted: {len(sessions)} sessions created") - print(sessions) - all_sessions.extend(sessions) - - # Small delay between batches to avoid overwhelming the system - if batch_num < num_batches: - await asyncio.sleep(1) - - except Exception as e: - print(f"❌ Batch {batch_num} failed: {type(e).__name__}: {e}") - # Continue with other batches even if one fails - continue - - print(f"\n=== Submission complete ===") - print(f"Total sessions created: {len(all_sessions)} replicas (for {total_n} jobs)") - print(f"Each replica will process ~{math.ceil(total_n / num_replicas)} jobs") - - return all_sessions - - def run_no_async(self): - """Run No Async. - - Run synchronous job submission. - - Returns - ------- - list - IDs of submitted jobs - - """ - session = Session() - - self.set_command(mode="sync") - - tile_IDs, _, _ = self.get_tile_IDs() - - job_ids = [] - for tile_ID in tile_IDs: - - job_name = self.set_job_name(tile_ID) - - options = f"{self._options_base} -e {tile_ID}" - - # Remove leading whitespace, problem in passing args below - options = options.lstrip() - - print(f"Running '{self._cmd} {options}'") - - # Submit flexible job (default - auto-scaling) - job_id = session.create( - name=job_name, - image=self._image, - cmd=self._cmd, - args=options, - ) - - print(f"Submitted job: {job_id}") - job_ids.append(job_id) - - return job_ids - - def write_job_tile_IDs(self, job_ids): - """Write Job Tile IDs - - Write tile IDs of jobs to log file. - - Parameters - ---------- - jobs_ids: list - IDs of submitted jobs. - - """ - tile_IDs, _, _ = self.get_tile_IDs() - - cwd = os.getcwd() - log_path = f"{cwd}/{self._params['log']}" - - print(f"Writing job and tile ID log file {log_path}") - with open(log_path, "w") as log: - for job_id, tile_ID in zip(job_ids, tile_IDs): - print(job_id, tile_ID, file=log) - - def print_stats(self): - """Print Stats. - - Print statistics of (to be submitted) jobs, helpfule to fine-tune parameters - before the actual submission. - - """ - # Total number of jobs (= length of input ID file) - _, total_n, _ = self.get_tile_IDs() - - if self._params["jobs_per_session"] == 0: - # Compute max jobs per session to fit in single batch - self._params["jobs_per_session"] = math.ceil(total_n / self._max_replicas_per_batch) - print(f"Max jobs per session (-J) = {self._params['jobs_per_session']}") - - # Number of sessions - n_session = math.ceil(total_n / self._params["jobs_per_session"]) - - # Number of batches - n_batch = math.ceil(n_session / self._max_replicas_per_batch) - - # Number of jobs per session - n_jobs = math.ceil(self._params["jobs_per_session"] / n_batch) - - n_jobs_serial = max( - math.ceil(n_jobs / self._params["parallel_jobs"]), - 1, - ) - - print( - f"Number of jobs = {total_n}\t\t(#{self._params['file_IDs']})") - print( - f"Number of sessions = {n_session:5d}\t\t({total_n} /" - + f" {self._params['jobs_per_session']})" - ) - print( - f"Number of batches = {n_batch:5d}\t\t({n_session} /" - + f" {self._max_replicas_per_batch})" - ) - print( - f"Number of jobs per session = {n_jobs:5d}" - + f"\t\t({self._params['jobs_per_session']} / {n_batch}" - ) - print( - f"Number of serial jobs = {n_jobs_serial:5d}\t\t({n_jobs}" - + f" / {self._params['parallel_jobs']})" - ) - - def run(self, args=None): - """Run. - - Run instance. - - Parameters - ---------- - args: list, optional - command line arguments, default is ``None`` - - """ - obj = self - - if args is None: - args = sys.argv - obj.set_params_from_command_line(args) - obj.update_params() - obj.check_params() - - obj.set_options_base() - - if self._params["stats"]: - obj.print_stats() - sys.exit(0) - - # Initialize session manager - self._session = Session() - - if obj._params["sync"] == "async": - print("Async mode") - job_ids = asyncio.run(obj.run_async()) - else: - print("Sync mode") - job_ids = obj.run_no_async() - print(f"Submitting jobs: done") - - obj.write_job_tile_IDs(job_ids) diff --git a/src/shapepipe/canfar_run.py b/src/shapepipe/canfar_run.py index 8330b8013..8f54c08ef 100644 --- a/src/shapepipe/canfar_run.py +++ b/src/shapepipe/canfar_run.py @@ -4,25 +4,7 @@ # executables are created by pyproject.toml. import sys -from shapepipe.canfar import canfar_submit, canfar_monitor, canfar_log_monitor - -def run_job(args=None): - """Run Job. - - Handles job submission with the canfar library - - Parameters - ---------- - args : list, optional - command line arguments, default is ``None`` - - """ - # Create instance - obj = canfar_submit.Job() - - # Run instance - obj.run(args=args) - +from shapepipe.canfar import canfar_monitor, canfar_log_monitor def run_log(args=None): """Run Log. diff --git a/src/shapepipe/get_ccds_run.py b/src/shapepipe/get_ccds_run.py deleted file mode 100644 index fb18b0a74..000000000 --- a/src/shapepipe/get_ccds_run.py +++ /dev/null @@ -1,44 +0,0 @@ -#!/usr/bin/env python3 - -"""GET_CCDS_WITH_PSF - -Obtain list of CCDs (single-exposure single-HDU files) for which valid PSF information -is available. This can serve to create a footprint coverage mask. - -Author: Martin Kilbinger - -""" - -import sys - -from shapepipe.utilities.ccd_psf_handler import CcdPsfHandler - - -def run_ccd_psf_handler(args=None): - """Run CCD PSF Handler. - - Create instance and run the CCD PSF handler. - - Parameters - ---------- - args : list, optional - command line arguments - - Returns - ------- - int - exit code - - """ - # Create instance - - obj = CcdPsfHandler() - - return obj.run(args=args) - - -def main(argv=None): - """Main. - """ - # A scripts to call the ccd psf class is created by pyproject.toml - return 0 diff --git a/src/shapepipe/modules/merge_headers_package/__init__.py b/src/shapepipe/modules/merge_headers_package/__init__.py index ed8416790..cd47d6a3d 100644 --- a/src/shapepipe/modules/merge_headers_package/__init__.py +++ b/src/shapepipe/modules/merge_headers_package/__init__.py @@ -4,9 +4,10 @@ :Author: Axel Guinot -:Parent module: ``split_exp_runner`` +:Parent module: ``find_exposures_runner`` -:Input: Numpy binary files (``.npy``) with single-exposure header information +:Input: An ``exp_numbers`` text file listing the tile's exposures; the + per-exposure header files (``.npy``) are found under ``EXP_BASE_DIR`` :Output: Single SQL file with combined header information @@ -20,8 +21,10 @@ Module-specific config file entries =================================== -OUTPUT_PATH : str, optional - Overrides the default module output directory under ``[FILE]:OUTPUT_DIR`` +EXP_BASE_DIR : str + Root of the per-exposure work directories; the header ``.npy`` files are + collected from ``///`` for every exposure + listed in the input ``exp_numbers`` file """ diff --git a/src/shapepipe/modules/merge_headers_runner.py b/src/shapepipe/modules/merge_headers_runner.py index d7b0bd035..21a2ec08b 100644 --- a/src/shapepipe/modules/merge_headers_runner.py +++ b/src/shapepipe/modules/merge_headers_runner.py @@ -34,38 +34,32 @@ def merge_headers_runner( output_dir = run_dirs["output"] w_log.info(f"output_dir = {output_dir}") - if config.has_option(module_config_sec, "EXP_BASE_DIR"): - # Tile-level mode: input is an exp_numbers txt file; collect header - # files from each per-exposure work directory via get_exp_output_files. - exp_base_dir = config.getexpanded(module_config_sec, "EXP_BASE_DIR") + # The input is an exp_numbers txt file (find_exposures_runner); the header + # files themselves are collected from each per-exposure work directory + # under EXP_BASE_DIR via get_exp_output_files. + exp_base_dir = config.getexpanded(module_config_sec, "EXP_BASE_DIR") - # In serial mode several tiles' exp_numbers files can arrive in a - # single call; merge each tile into its own per-tile sqlite file. - for exp_numbers_file in (item[0] for item in input_file_list): - w_log.info( - f"Tile-level merge: collecting headers from {exp_base_dir} " - f"using {exp_numbers_file}" - ) - headers_file_list = get_exp_output_files( - exp_base_dir, - exp_numbers_file, - "split_exp_runner", - "headers", - ".npy", - w_log=w_log, - ) - # Extract tile number from the exp_numbers filename, e.g. - # "exp_numbers-284.272-1.000.txt" -> "-284.272-1.000" - base = os.path.splitext(os.path.basename(exp_numbers_file))[0] - tile_number = re.sub(r"^exp_numbers", "", base) - merge_headers(headers_file_list, output_dir, tile_number) - w_log.info( - f"Merged {len(headers_file_list)} exposure header files" - ) - else: - # Per-exposure mode: input_file_list already contains the header files. - merge_headers(input_file_list, output_dir) - w_log.info(f"Merged {len(input_file_list)} input file headers") + # In serial mode several tiles' exp_numbers files can arrive in a + # single call; merge each tile into its own per-tile sqlite file. + for exp_numbers_file in (item[0] for item in input_file_list): + w_log.info( + f"Collecting headers from {exp_base_dir} " + f"using {exp_numbers_file}" + ) + headers_file_list = get_exp_output_files( + exp_base_dir, + exp_numbers_file, + "split_exp_runner", + "headers", + ".npy", + w_log=w_log, + ) + # Extract tile number from the exp_numbers filename, e.g. + # "exp_numbers-284.272-1.000.txt" -> "-284.272-1.000" + base = os.path.splitext(os.path.basename(exp_numbers_file))[0] + tile_number = re.sub(r"^exp_numbers", "", base) + merge_headers(headers_file_list, output_dir, tile_number) + w_log.info(f"Merged {len(headers_file_list)} exposure header files") # No return objects return None, None diff --git a/src/shapepipe/modules/psfex_interp_package/__init__.py b/src/shapepipe/modules/psfex_interp_package/__init__.py index 1c3b51ca4..2b6a161b1 100644 --- a/src/shapepipe/modules/psfex_interp_package/__init__.py +++ b/src/shapepipe/modules/psfex_interp_package/__init__.py @@ -32,14 +32,18 @@ Threshold of stars under which the PSF is not interpolated CHI2_THRESH : int Threshold for chi squared (:math:`\chi^2`) -ME_DOT_PSF_DIR : str - Module name of last run producing PSFEx PSF model files, for multi-epoch - processing. The specifier "last:" is not required +ME_DOT_PSF_EXP_DIR : str + Root of the per-exposure work directories, for multi-epoch processing; + the ``psfex_runner`` output directory of every exposure listed in the + input ``exp_numbers`` file is resolved beneath it ME_DOT_PSF_PATTERN : str Input file name pattern for PSFEx PSF model files, for multi-epoch processing -ME_LOG_WCS : str - Path to world coordinate system log file (``*sqlite``) + +In ``MULTI-EPOCH`` mode the input file list is positional and carries three +entries -- the galaxy catalogue, the world-coordinate-system log +(``*sqlite``) and the ``exp_numbers`` file -- set through ``FILE_PATTERN`` +and ``FILE_EXT``, not through module-specific keys. """ diff --git a/src/shapepipe/modules/psfex_interp_runner.py b/src/shapepipe/modules/psfex_interp_runner.py index 67c21adfc..5627d5d20 100644 --- a/src/shapepipe/modules/psfex_interp_runner.py +++ b/src/shapepipe/modules/psfex_interp_runner.py @@ -10,7 +10,6 @@ from shapepipe.modules.psfex_interp_package import psfex_interp from shapepipe.pipeline.exp_utils import get_exp_output_dirs -from shapepipe.pipeline.run_log import get_last_dir, get_all_dirs @module_runner( @@ -62,38 +61,22 @@ def psfex_interp_runner( # Run in MULTI-EPOCH mode elif mode == "MULTI-EPOCH": - # Fetch multi-epoch parameters - if config.has_option(module_config_sec, "ME_DOT_PSF_EXP_DIR"): - # v2.0: locate psfex_runner output dirs via the $SP_EXP tree - exp_base_dir = config.getexpanded( - module_config_sec, "ME_DOT_PSF_EXP_DIR" - ) - if len(input_file_list) < 3: - raise ValueError( - "ME_DOT_PSF_EXP_DIR requires the exposure-numbers file" - + " as a third input; add 'exp_numbers' to FILE_PATTERN" - + f" and FILE_EXT in the [{module_config_sec}] config" - + " section." - ) - exp_numbers_file = input_file_list[2] - dot_psf_dirs = get_exp_output_dirs( - exp_base_dir, exp_numbers_file, "psfex_runner", w_log - ) - else: - module = config.getexpanded( - module_config_sec, - "ME_DOT_PSF_DIR", + # Fetch multi-epoch parameters. The psfex_runner output dirs are + # located through the per-exposure work tree. + exp_base_dir = config.getexpanded( + module_config_sec, "ME_DOT_PSF_EXP_DIR" + ) + if len(input_file_list) < 3: + raise ValueError( + "ME_DOT_PSF_EXP_DIR requires the exposure-numbers file" + + " as a third input; add 'exp_numbers' to FILE_PATTERN" + + f" and FILE_EXT in the [{module_config_sec}] config" + + " section." ) - module_name = module.split(":")[-1] - if "last" in module: - dot_psf_dirs = [get_last_dir(run_dirs["run_log"], module_name)] - elif "all" in module: - dot_psf_dirs = get_all_dirs(run_dirs["run_log"], module_name) - else: - raise ValueError( - "Expected qualifier 'last:' or 'all' before module" - + f" '{module}' in config entry 'ME_DOT_PSF_DIR'" - ) + exp_numbers_file = input_file_list[2] + dot_psf_dirs = get_exp_output_dirs( + exp_base_dir, exp_numbers_file, "psfex_runner", w_log + ) dot_psf_pattern = config.get( module_config_sec, diff --git a/src/shapepipe/modules/vignetmaker_package/__init__.py b/src/shapepipe/modules/vignetmaker_package/__init__.py index b24f8397e..27c679c23 100644 --- a/src/shapepipe/modules/vignetmaker_package/__init__.py +++ b/src/shapepipe/modules/vignetmaker_package/__init__.py @@ -35,15 +35,22 @@ Run mode for module, options are ``CLASSIC`` or ``MULTI-EPOCH`` PREFIX : str or list Output file name prefix(es) -ME_IMAGE_DIR : list - Module names of last run producing single-exposure flags, images, weights, - and SExtractor background images, for multi-epoch processing. The specifier - "last:" is not required +ME_IMAGE_EXP_DIR : str + Root of the per-exposure work directories, for multi-epoch processing; + each runner output directory is resolved beneath it for every exposure + listed in the input ``exp_numbers`` file +ME_IMAGE_EXP_RUNNERS : list + Names of the runners producing the single-exposure flags, images, weights + and SExtractor background images, in the same order as + ``ME_IMAGE_PATTERN`` ME_IMAGE_PATTERN : list Input file name patterns for flag, image, weight, and SExtractor background files, for multi-epoch processing -ME_LOG_WCS : str - Path to world coordinate system log file (``*sqlite``) + +In ``MULTI-EPOCH`` mode the input file list is positional and carries three +entries -- the galaxy catalogue, the world-coordinate-system log +(``*sqlite``) and the ``exp_numbers`` file -- set through ``FILE_PATTERN`` +and ``FILE_EXT``, not through module-specific keys. """ diff --git a/src/shapepipe/modules/vignetmaker_runner.py b/src/shapepipe/modules/vignetmaker_runner.py index a31d818ae..85df0cffe 100644 --- a/src/shapepipe/modules/vignetmaker_runner.py +++ b/src/shapepipe/modules/vignetmaker_runner.py @@ -10,7 +10,6 @@ from shapepipe.modules.vignetmaker_package import vignetmaker as vm from shapepipe.pipeline.exp_utils import get_exp_output_dirs -from shapepipe.pipeline.run_log import get_last_dir, get_all_dirs @module_runner( @@ -105,44 +104,27 @@ def vignetmaker_runner( elif mode == "MULTI-EPOCH": # Multi-epoch exposures - if config.has_option(module_config_sec, "ME_IMAGE_EXP_DIR"): - # v2.0: locate runner output dirs via the $SP_EXP tree - exp_base_dir = config.getexpanded( - module_config_sec, "ME_IMAGE_EXP_DIR" + # Locate the runner output dirs through the per-exposure work tree. + exp_base_dir = config.getexpanded( + module_config_sec, "ME_IMAGE_EXP_DIR" + ) + if len(input_file_list) < 3: + raise ValueError( + "ME_IMAGE_EXP_DIR requires the exposure-numbers" + + " file as a third input; add 'exp_numbers' to" + + " FILE_PATTERN and FILE_EXT in the" + + f" [{module_config_sec}] config section." ) - if len(input_file_list) < 3: - raise ValueError( - "ME_IMAGE_EXP_DIR requires the exposure-numbers" - + " file as a third input; add 'exp_numbers' to" - + " FILE_PATTERN and FILE_EXT in the" - + f" [{module_config_sec}] config section." - ) - exp_numbers_file = input_file_list[2] - exp_runner_names = config.getlist( - module_config_sec, "ME_IMAGE_EXP_RUNNERS" + exp_numbers_file = input_file_list[2] + exp_runner_names = config.getlist( + module_config_sec, "ME_IMAGE_EXP_RUNNERS" + ) + image_dirs = [] + for runner_name in exp_runner_names: + dirs = get_exp_output_dirs( + exp_base_dir, exp_numbers_file, runner_name, w_log ) - image_dirs = [] - for runner_name in exp_runner_names: - dirs = get_exp_output_dirs( - exp_base_dir, exp_numbers_file, runner_name, w_log - ) - image_dirs.append(dirs) - else: - # v1: run-log based lookup via symlinked exposure run dirs - modules = config.getlist(module_config_sec, "ME_IMAGE_DIR") - image_dirs = [] - for module in modules: - module_name = module.split(":")[-1] - if "last" in module: - dirs = [get_last_dir(run_dirs["run_log"], module_name)] - elif "all" in module: - dirs = get_all_dirs(run_dirs["run_log"], module_name) - else: - raise ValueError( - "Expected qualifier 'last:' or 'all' before module" - + f" '{module}' in config entry 'ME_IMAGE_DIR'" - ) - image_dirs.append(dirs) + image_dirs.append(dirs) image_pattern = config.getlist( module_config_sec, diff --git a/src/shapepipe/pipeline/exp_utils.py b/src/shapepipe/pipeline/exp_utils.py index b934f0bb3..889ff31c1 100644 --- a/src/shapepipe/pipeline/exp_utils.py +++ b/src/shapepipe/pipeline/exp_utils.py @@ -1,11 +1,10 @@ """EXPOSURE UTILITIES. Utility functions for accessing per-exposure runner outputs from the -tile level. In the v2.0 pipeline, each single exposure is processed in -its own work directory ``///``. Tile-level -modules that need files produced by a per-exposure runner use the -functions here to discover those files by scanning the contributing -exposure directories. +tile level. Each single exposure is processed in its own work directory +``///``. Tile-level modules that need files +produced by a per-exposure runner use the functions here to discover those +files by scanning the contributing exposure directories. :Author: Martin Kilbinger @@ -40,7 +39,7 @@ def get_exp_output_files( ---------- exp_base_dir : str Root directory that contains all per-exposure work directories, - e.g. ``/arc/home/kilbinger/v2.0/exp`` + e.g. ``/exp`` exp_numbers_file : str Path to a text file listing one exposure ID per line, e.g. ``exp_numbers-301-279.txt`` @@ -91,7 +90,8 @@ def get_exp_output_files( missing = [] for exp_id in exp_ids: - # Directory structure mirrors run_job_canfar_v2.0.sh: + # Directory structure mirrors the workflow's per-exposure store, built + # by rule tile_exp_forest / workflow/scripts/build_forest.py: # exp_prefix = first 2 chars of exp_id (e.g. "21") # exp_base = exp_id without trailing letter if present (e.g. "2113864"), # or full exp_id for numeric-only ids (image sims) diff --git a/src/shapepipe/summary_run.py b/src/shapepipe/summary_run.py deleted file mode 100755 index 39d60ce08..000000000 --- a/src/shapepipe/summary_run.py +++ /dev/null @@ -1,102 +0,0 @@ -#!/usr/bin/env python - -import argparse -import sys -import os - -from shapepipe.utilities import summary - -from shapepipe.utilities import summary_params_pre_v2 as summary_params - - -def run(patch, job_exclusive=None, verbose=False): - - jobs, list_tile_IDs_dot = summary_params.set_jobs_v2_pre_v2(patch, verbose) - - list_tile_IDs = summary.job_data.replace_dot_dash(list_tile_IDs_dot) - - # Numbers updated at runtime - par_runtime = summary.init_par_runtime(list_tile_IDs) - - summary.job_data.print_stats_header() - - exp_IDs_path = "exp_numbers.txt" - if os.path.exists(exp_IDs_path): - # Read exposure ID list if file exists - all_exposures = summary.get_IDs_from_file(exp_IDs_path) - par_runtime = summary.update_par_runtime_after_find_exp( - par_runtime, all_exposures - ) - - if ( - not os.path.exists(exp_IDs_path) - or not job_exclusive - or int(job_exclusive) & 1 - ): - # Run job 1 if exposure ID list file does not exist or - # job_exclusive is 1 or not set - key = "1" - jobs[key].print_intro() - jobs[key].check_numbers(par_runtime=par_runtime, indices=[0, 1]) - - all_exposures = summary.get_all_exposures( - jobs[key]._paths_in_dir[1], verbose=True - ) - par_runtime = summary.update_par_runtime_after_find_exp( - par_runtime, all_exposures - ) - - jobs[key].write_IDs_to_file("exp_numbers.txt", all_exposures) - - jobs[key].check_numbers(par_runtime, indices=[2]) - - summary.print_par_runtime(par_runtime, verbose=verbose) - - # Get all keys after "1" - keys = sorted(jobs.keys(), key=int) - _ = keys.pop(0) - - for key in keys: - if job_exclusive and not int(key) & int(job_exclusive): - continue - jobs[key].print_intro() - jobs[key].check_numbers(par_runtime=par_runtime) - - return 0 - - -def parse_args(argv=None): - parser = argparse.ArgumentParser( - description=( - 'Print summary of ShapePipe job status for a given UNIONS patch.' - ), - ) - parser.add_argument( - 'patch', - help='Patch identifier (e.g. P3, P9).', - ) - parser.add_argument( - 'job_exclusive', - nargs='?', - default=None, - help=( - 'Bitmask selecting which jobs to run; ' - + 'if omitted, all jobs run.' - ), - ) - parser.add_argument( - '-v', - '--verbose', - action='store_true', - help='Verbose output.', - ) - return parser.parse_args(argv) - - -def main(argv=None): - args = parse_args(argv) - run(args.patch, args.job_exclusive, args.verbose) - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/src/shapepipe/utilities/__init__.py b/src/shapepipe/utilities/__init__.py index 42e234003..8173e49b5 100644 --- a/src/shapepipe/utilities/__init__.py +++ b/src/shapepipe/utilities/__init__.py @@ -7,4 +7,4 @@ """ -__all__ = ["file_system", "cfis", "galaxy", "mask_query", "summary"] +__all__ = ["file_system", "cfis", "galaxy", "mask_query"] diff --git a/src/shapepipe/utilities/ccd_psf_handler.py b/src/shapepipe/utilities/ccd_psf_handler.py deleted file mode 100644 index d5b72cc19..000000000 --- a/src/shapepipe/utilities/ccd_psf_handler.py +++ /dev/null @@ -1,388 +0,0 @@ -"""CCD_PSF_HANDLER - -Obtain list of CCDs (single-exposure single-HDU files) for which valid PSF -information is available. This can serve to create a footprint coverage mask. - -Author: Mike Hudson, Martin Kilbinger - -""" - -import glob -import os -import re -import sys - -import numpy as np - -from cs_util import args as cs_args -from cs_util import logging - -from shapepipe.utilities import summary - - -class CcdPsfHandler(object): - """CCD PSF Handler Class. - - Handles extraction of CCDs with valid PSF information from shapepipe - patches. - """ - - def __init__(self): - """Initialize the handler.""" - self.params_default() - - def params_default(self): - """Set default parameters and command line options.""" - - self._params = { - "version_cat": "v1.6", - "n_CCD": 40, - "output": None, - } - - self._short_options = { - "version_cat": "-V", - "n_CCD": "-n", - "output": "-o", - } - - self._types = { - "n_CCD": "int", - } - - self._help_strings = { - "version_cat": "catalogue major version, allowed are v1.3, v1.4, v1.5, v1.6; default is {}", - - "n_CCD": "number of CCDs per exposure; default is {}", - "output": "output file path; default is ccds_with_psf_.txt", - } - - def set_params_from_command_line(self, args): - """Set Params From Command line. - - Only use when calling using python from command line. - Does not work from ipython or jupyter. - - Parameters - ---------- - args : list - command line arguments - - """ - # Read command line options - options = cs_args.parse_options( - self._params, - self._short_options, - self._types, - self._help_strings, - args=args, - ) - self._params = options - - # Save calling command to log_get_ccds_with_psf; args excludes the program - # name, which log_command takes from argv[0] - logging.log_command(["get_ccds_with_psf", *args]) - - def update_params(self): - """Update parameters. - - Set derived parameters based on input parameters. - """ - # Determine number of patches based on version - version = self._params["version_cat"] - if version in ("v1.3", "v1.4"): - n_patch = 7 - elif version == "v1.5": - n_patch = 8 - elif version == "v1.6": - n_patch = 9 - else: - raise ValueError(f"Invalid version {version}") - - self._params["n_patch"] = n_patch - self._params["patches"] = [f"P{x}" for x in np.arange(n_patch) + 1] - - # Set output file if not specified - if self._params["output"] is None: - self._params["output"] = f"ccds_with_psf_{version}.txt" - - def check_params(self): - """Check parameters for validity.""" - # Add any parameter validation logic here - pass - - def get_lines(self, fname): - """Get Lines. - - Return list of lines read from a text file. - - Parameters - ---------- - fname : str - input file name - - Returns - ------- - list - IDs - - """ - IDs = [] - with open(fname) as f: - lines = f.readlines() - for line in lines: - IDs.append(line.rstrip()) - - return IDs - - def get_exp_shdu_missing(self, patches): - """Get Exp Shdu Missing. - - Returns set of missing CCDs (single-exposure single-HDU IDs) from a list of patches. - - Parameters - ---------- - patches : list - input patches - - Returns - ------- - set - missing CCD IDs - - """ - exp_shdu_missing_all = set() - - for patch in patches: - - path_exp_shdu_missing = f"{patch}/summary/missing_job_32_all.txt" - exp_shdu_missing = self.get_lines(path_exp_shdu_missing) - - print( - f"Patch {patch}: Found {len(exp_shdu_missing)} missing ccds", - end="; ", - ) - - exp_shdu_missing_all.update(exp_shdu_missing) - - print(f"cumulative {len(exp_shdu_missing_all)} missing ccds") - - print() - - return exp_shdu_missing_all - - def get_exp(self, patches): - """Get Exp. - - Return set of exposures from a list of patches. - - Parameters - ---------- - patches : list - input patches - - Returns - ------- - set - exposure IDs - - """ - exp_all = set() - - for patch in patches: - - path_exp = f"{patch}/exp_numbers.txt" - exp = self.get_lines(path_exp) - - print(f"Patch {patch}: Found {len(exp)} exposures", end="; ") - - exp_all.update(exp) - - print(f"cumulative {len(exp_all)} exposures") - - print() - - return exp_all - - def get_ccds_with_psf(self, patches, n_CCD=40): - """Get CCDs With PSF. - - Return set of CCDs with valid PSF from a list of patches. A CCD has a - valid PSF if it appears among the exposures' single-HDU IDs but not in - the per-patch missing-CCD lists. - - Parameters - ---------- - patches : list - input patches - n_CCD : int - number of CCDs per exposure - - Returns - ------- - set - CCD IDs with valid PSF - - """ - # Get missing CCDs - print("=== get missing CCDs ===") - exp_shdu_missing_all = self.get_exp_shdu_missing(patches) - - # Get all exposures used in tiles - print("=== get exposures ===") - exp_all = self.get_exp(patches) - - # Turn exposures into exposure-single-HDU names (CCDs) - exp_shdu_all = set(summary.get_all_shdus(exp_all, n_CCD)) - - # Subtract the CCDs whose PSF model is missing - exp_shdu_with_psf = exp_shdu_all - exp_shdu_missing_all - - print( - f"Found {len(exp_shdu_all)} CCDs, " - f"{len(exp_shdu_missing_all)} missing, " - f"{len(exp_shdu_with_psf)} with valid PSF" - ) - - return exp_shdu_with_psf - - def get_ccds_with_psf_method_v1_3(self, patches, n_CCD=40): - """Get CCDs With PSF Method v1.3. - - Return set of CCDs with valid PSF by scanning run directories. - Finds star_selection files and checks corresponding stat files - for valid (non-nan) FWHM values. - - Parameters - ---------- - patches : list - input patches - n_CCD : int - number of CCDs per exposure (unused, kept for API consistency) - - Returns - ------- - set - CCD IDs with valid PSF - - """ - exp_shdu_all = set() - - for patch in patches: - # Find all star_selection files - mask_pattern = ( - f"{patch}/output/run_sp_exp_SxSePs*/" - f"setools_runner/output/mask/star_selection-*.fits" - ) - mask_files = glob.glob(mask_pattern) - - print( - f"Patch {patch}: Found {len(mask_files)} star_selection FITS" - + " files" - ) - - for mask_file in mask_files: - # Extract exp_shdu from filename - basename = os.path.basename(mask_file) - match = re.match(r"star_selection-(.+)\.fits", basename) - if not match: - print(f"Warning: Non-matching file name {basename}") - continue - exp_shdu = match.group(1) - - # Get corresponding stat file path - # Replace mask dir with stat/stat dir in the path - stat_file = mask_file.replace( - "/mask/star_selection-", - "/stat/star_stat-", - ).replace(".fits", ".txt") - - #print("MKDEBUG ", exp_shdu, stat_file) - - # Check if stat file exists and has valid FWHM - if not os.path.exists(stat_file): - print("MKDEBUG stat file ", stat_file, " does not exist") - continue - - # Read stat file and check for nan in "Mean star fwhm selected" - has_nan = False - with open(stat_file) as f: - for line in f: - if "Mean star fwhm selected" in line: - if "nan" in line.lower(): - has_nan = True - break - - if not has_nan: - #print("MKDEBUG append", exp_shdu) - exp_shdu_all.add(exp_shdu) - - print(f"After patch {patch}: {len(exp_shdu_all)} CCDs with valid PSF") - - print(f"Found {len(exp_shdu_all)} CCDs with valid PSF") - - return exp_shdu_all - - def save(self, IDs, path): - """Save. - - Save list of IDs to text file. - - Parameters - ---------- - IDs : set - input IDs - path : str - output file name - - """ - with open(path, "w") as f_out: - for ID in IDs: - print(ID, file=f_out) - - def run(self, args=None): - """Run. - - Main execution method. - - Parameters - ---------- - args : list, optional - command line arguments - - Returns - ------- - int - exit code (0 for success) - - """ - if args is None: - args = sys.argv[1:] - - # Set parameters from command line - self.set_params_from_command_line(args) - self.update_params() - self.check_params() - - # Get parameters - patches = self._params["patches"] - version = self._params["version_cat"] - n_CCD = self._params["n_CCD"] - output = self._params["output"] - - print( - f"=== get_ccds_with_psf for version {version}, patches {patches} ===" - ) - - if self._params["version_cat"] == "v1.3": - print("=== method for 1.3: exp with PSF star sample === ") - exp_shdu_all = self.get_ccds_with_psf_method_v1_3(patches, n_CCD) - else: - print("=== method for >= 1.4: exp_list - missing === ") - exp_shdu_all = self.get_ccds_with_psf(patches, n_CCD) - - self.save(exp_shdu_all, output) - - print(f"Results saved to {output}") - - return 0 diff --git a/src/shapepipe/utilities/summary.py b/src/shapepipe/utilities/summary.py deleted file mode 100755 index dd5009f3f..000000000 --- a/src/shapepipe/utilities/summary.py +++ /dev/null @@ -1,832 +0,0 @@ -"""SUMMARY - -Author: Martin Kilbinger - -""" - -import sys -import os -import re -import fnmatch - -import logging - -from collections import Counter - -from tqdm import tqdm - - -def init_par_runtime(list_tile_IDs): - - # Numbers updated at runtime - par_runtime = {} - - par_runtime["n_tile_IDs"] = len(list_tile_IDs) - par_runtime["list_tile_IDs"] = list_tile_IDs - - return par_runtime - - -def update_par_runtime_after_find_exp(par_runtime, all_exposures): - - # Single-exposure images - par_runtime["n_exposures"] = len(all_exposures) - par_runtime["list_exposures"] = all_exposures - - # Single-HDU single exposure images - n_CCD = 40 - par_runtime["n_shdus"] = get_par_runtime(par_runtime, "exposures") * n_CCD - par_runtime["list_shdus"] = get_all_shdus(all_exposures, n_CCD) - - return par_runtime - - -def get_IDs_from_file(path): - """Get IDs From File. - - Return IDs from text file. Removes letters and replaces - dots "." with dashes "-". - - Parameters - ---------- - path: str - input file path - - Returns - -------- - list - IDs - - """ - numbers = [] - with open(path) as f_in: - for line in f_in: - entry = line.rstrip() - number = re.sub("[a-zA-Z]", "", entry) - numbers.append(number) - - return numbers - - -def get_all_exposures(exp_number_file_list, verbose=False): - """Get All Exposures. - - Return all exposure names from a list of text files. - - Parameters - ---------- - exp_number_list: list - input file names - - """ - exposures = set() - for idx, path in enumerate(exp_number_file_list): - exps = get_IDs_from_file(path) - exposures.update(exps) - - return list(exposures) - - -def get_all_shdus(exposures, n_CCD): - """Get All SHDUs. - - Return all single-exposure single-HDU (CCD) IDs. - - Parameters - ---------- - exposures: list - exposure names - n_CCD: int - number of CCDs per exposure - - Returns - -------- - list - single-exposure single-HDU IDs - - """ - shdus = [] - for exposure in exposures: - for idx_CCD in range(n_CCD): - shdus.append(f"{exposure}-{idx_CCD}") - - return shdus - - -def set_as_list(item=None, n=None, default=1): - """Set As List. - - Return input as list. - - Parameters - ----------- - item: str, int, or list, optional - input item(s); default is None, in which - case the return is [1] * n - n: int, optional - number of list items to return, default is None, - in which case the number will be set to 1. If item and - n are not None, n has to be equal to len(item) - default: int, optional - value to return if item is not given; - default is 1 - - Raises - ------- - IndexError - if n != len(item) - - Returns - ------- - list - input item(s) as list - """ - my_n = n or 1 - - if not item: - result = [default] * my_n - elif not isinstance(item, list): - result = [item] * my_n - else: - result = item - if len(item) != my_n: - raise IndexError(f"item has length {len(item)} != {n}") - - return result - - -def check_special_one(module, path): - - ngmix_finished = False - - with open(path) as f_in: - lines = f_in.readlines() - for line in lines: - entry = line.rstrip() - - if module == "setools_runner": - m = re.search("Nb stars = (\S*)", line) - if m: - value = int(m[1]) - if value < 2: - code = 0 - msg = ( - f"Not enough stars for random split:" - + f" #stars = {value}" - ) - return msg, code - break - m = re.search("Mode computation failed", line) - if m: - code = 1 - msg = "Mode computation of stellar locus failed" - return msg, code - - if module == "psfex_interp_runner": - m = re.search("Key N_EPOCH not found", line) - if m: - code = 2 - msg = "N_EPOCH not in SEx cat, rerun job 16" - return msg, code - - m = re.search( - "ValueError: cannot reshape array of size 0 into shape", - line, - ) - if m: - code = 3 - msg = "found array of size 0" - return msg, code - - if module == "mask_runner": - m = re.search("Empty or corrupt FITS file", line) - if m: - code = 4 - msg = "empty or corrult FITS file" - return msg, code - - if module == "sextractor_runner": - m = re.search("sextracted 0", line) - if m: - code = 5 - msg = "No object detected (weight might be 0 everywhere)" - return msg, code - m = re.search("astropy\.wcs\.wcs\.NoConvergence", line) - if m: - code = 8 - msg = "WCS world2pix did not converge" - return msg, code - - if module == "ngmix_runner": - m = re.search("finished", line) - if m: - ngmix_finished = True - break - - if module == "merge_sep_cats_runner": - m = re.search("Input catalogue", line) - if m: - code = 7 - msg = "One or more ngmix catalogues not found for merge" - return msg, code - - if module == "ngmix_runner" and not ngmix_finished: - code = 6 - msg = "ngmix incomplete" - return msg, code - - - return None, None - - -class job_data(object): - """Job Data. - - Class to handle a job. - - Parameters - ---------- - bit: int - bit-coded job number - run_dir: str or list - run directory(ies) - modules: list - module names - key_expected: int or str - number of expected output files; if str: will be updated - with runtime value - n_mult: int or list, optional - multiplicity of output files, default `None`, in which - case it is set to 1 - pattern: list, optional - if not None, file pattern to match; defafult is `None` - path_main: str, optional - main (left-most) part of output directory, default is "." - path_left: str, optional - left (first) part of output directory, default is "./output" - output_subdirs: str, optional - output subdirectories if not `None`; default is `None` - path_right: str, optional - right (last) part of output subdir suffix if not `None`; - default is `None` - path_output: str, optional - module output path, default is "output" - output_path_missing_IDs: list, optional - output path of missing ID, if `None` (default) will be - given by job bit and module. - special: bool, optional - if True check output file content for special messages; - default is False - verbose: bool, optional - verbose output if True; default is False - - """ - - def __init__( - self, - bit, - run_dir, - modules, - key_expected, - n_mult=None, - pattern=None, - path_main=".", - path_left="output", - output_subdirs=None, - path_right=None, - path_output="output", - output_path_missing_IDs=None, - special=False, - verbose=False, - ): - self._bit = bit - self._run_dir = set_as_list(item=run_dir, n=len(modules)) - self._modules = modules - self._key_expected = set_as_list(item=key_expected, n=len(modules)) - self._n_mult = set_as_list(item=n_mult, n=len(modules)) - self._pattern = set_as_list(item=pattern, n=len(modules), default="") - self._path_main = path_main - self._path_left = path_left - self._output_subdirs = output_subdirs or [""] - self._path_right = set_as_list( - path_right, - len(modules), - default=".", - ) - self._path_output = set_as_list( - path_output, - len(modules), - default="output", - ) - self._output_path_missing_IDs = output_path_missing_IDs - self._special = set_as_list( - special, - len(modules), - default=False, - ) - self._path_right = set_as_list(path_right, len(modules), default=".") - self._output_path_missing_IDs = output_path_missing_IDs - self._verbose = verbose - - def print_intro(self): - """Print Intro. - - Print header line for job statistics. - - """ - logging.info(f" # Job {self._bit}:") - - @classmethod - def print_stats_header(self): - """Print Stats Header. - - Print overall header information for stats output. - - """ - logging.info( - "module expected found" - + " missing uniq_miss fr_found" - ) - logging.info("=" * 100) - - def print_stats( - self, - module, - n_expected, - n_found, - n_special, - n_missing, - idx, - ): - """Print Stats. - - Print output file statistics. - - Parameters - ---------- - module: str - module name - n_expected: int - number of expected files - n_found: int - number of found files - n_special: int - number of special cases - n_missing: int - number of missing files - idx: int - module index - - """ - module_str = module - - if not self._special[idx]: - if n_expected > 0: - fraction_found = n_found / n_expected - else: - fraction_found = 1 - - n_missing_per_mult = n_missing / self._n_mult[idx] - - else: - module_str = f"{module_str} (special)" - n_found = n_special - n_missing = -1 - n_missing_per_mult = -1 - fraction_found = n_found / n_expected - n_expected = -1 - - logging.info( - f"{module_str:30s} {n_expected:9d} {n_found:9d}" - + f" {n_missing:9d}" - + f" {n_missing_per_mult:9.1f} {fraction_found:9.1%}" - ) - - @classmethod - def is_ID_in_str(self, ID, path): - if ID in path: - return True - - @classmethod - def is_not_in_any(self, ID, list_str): - return not any(ID in string for string in list_str) - - @classmethod - def replace_dot_dash(self, numbers): - - results = [re.sub("\.", "-", number) for number in numbers] - - return results - - @classmethod - def replace_dash_dot_if_tile(self, numbers): - - pattern = re.compile(r"(\d{3})-(\d{3})") - results = [pattern.sub(r"\1.\2", number) for number in numbers] - - return results - - @classmethod - def get_unique(self, names): - n_all = len(names) - names_unique = list(set(names)) - n_unique = len(names_unique) - - if n_all != n_unique: - if True: # self._verbose: - logging.warning( - f"{n_all - n_unique} duplicates removed from {n_all} IDs" - ) - - return names_unique - - @classmethod - def write_IDs_to_file(self, output_path, IDs): - """Write IDs to file. - - Write list if image IDs to text file. - - Parameters - ---------- - output_path: str - output file path - IDs: list - image IDs - - """ - IDs_dot = self.replace_dash_dot_if_tile(IDs) - if len(IDs_dot) > 0: - # Write IDs to file - with open(output_path, "w") as f_out: - for ID in IDs_dot: - print(ID, file=f_out) - elif os.path.exists(output_path): - # Remove preivous obsolete ID file - os.unlink(output_path) - - def check_special(self, module, idx): - - messages = {} - - if self._special[idx]: - - # Loop over input file names and paths - for name, path in zip(self._names_in_dir[idx], self._paths_in_dir[idx]): - - # Check if special case is found - msg, code = check_special_one(module, path) - if msg: - # First time occurance: create empty list for this code - if code not in messages: - messages[code] = [] - - # Append file name, message, and code - messages[code].append(f"{name} {code} {msg}") - - if len(messages) > 0: - # Loop over codes = key in messages dict - for code in messages: - # Create output file for this code - output_path = ( - f"{self._path_main}/summary/special_job_{self._bit}" - + f"_{module}_{code}.txt" - ) - # Write all messages - with open(output_path, "w") as f_out: - for msg in messages[code]: - print(msg, file=f_out) - - # Count all special cases = sum of cases over all codes - n_all = sum([len(messages[code]) for code in messages]) - return n_all - - def output_missing( - self, - module, - idx, - par_runtime=None, - ): - """Output Missing. - - Writes IDs of missing images to disk. - - """ - key_expected = self._key_expected[idx] - names_in_dir = self._names_in_dir[idx] - paths_in_dir = self._paths_in_dir[idx] - n_mult = self._n_mult[idx] - - list_expected = get_par_runtime(par_runtime, key_expected, kind="list") - - # Count image IDs in names that were found earlier - - # Get file name pattern - if module != "split_exp_runner" or (self._bit != 2 and self._bit != 4096): - pattern = re.compile(r"(?:\d{3}-\d{3}|\d{7}-\d+|\d{7})") - else: - # split_exp_runner with sp_local=0: input is exp, output is shdu - # (images) and exp (header); ignore hdu number. - # If sp_local=1 set bit to != 2 - # Update 11/2025: No longer working for P9 4096. Solution: set n_mult=3. - pattern = re.compile( - r"(?:\d{3}-\d{3}|\d{7})" - ) - - ## Extract image IDs from names - IDs = [] - for name, path in zip(names_in_dir, paths_in_dir): - - match = pattern.search(name) - if match: - ID = match.group() - IDs.append(ID) - else: - msg = f"No ID found in {name}" - #raise ValueError(msg) - print(f"Warning: {msg}, continuing") - - # For split_exp_runner P8, IDs now contain exps and sdus, - # not matching mult. - - ## Count occurences - ID_counts = Counter(IDs) - - ## Add to missing if ocurence less than n_mult - missing_IDs = [] - for ID in list_expected: - if ID_counts[ID] < n_mult: - missing_IDs.append(ID) - - n_all = len(missing_IDs) - missing_IDs_unique = self.get_unique(missing_IDs) - - if not self._output_path_missing_IDs: - # Default name using bit and module - output_path = ( - f"{self._path_main}/summary/missing_job_{self._bit}" - + f"_{module}.txt" - ) - else: - # User-defined name (e.g. ngmix_runner_X) - output_path = self._output_path_missing_IDs[idx] - self.write_IDs_to_file(output_path, missing_IDs_unique) - - return missing_IDs_unique - - def output_missing_job(self): - output_path = ( - f"{self._path_main}/summary/missing_job_{self._bit}_all.txt" - ) - - missing_IDs_all = set(self._missing_IDs_job) - - self.write_IDs_to_file(output_path, missing_IDs_all) - - @classmethod - def get_last_full_path(self, base_and_subdir, matches): - """Get Last Full Path - - Return full path of last file in list. - - """ - # Sort according to creation time - matches_sorted = sorted( - matches, - key=lambda entry: entry.name, - ) - - # Get most recent one - last = matches_sorted[-1] - - # Get full path - full_path = os.path.join(base_and_subdir, last.name) - - return full_path - - @classmethod - def get_module_output_dir(self, full_path, module, path_output): - """Get Module Output Dir. - - Return output directory name for given module. - - """ - directory = f"{full_path}/{module}/{path_output}" - - return directory - - def get_matches_final(self, directory, idx): - - # Loop over files - # os.path.whether exists is twice faster than try/except - - if os.path.exists(directory): - pattern = f"{self._pattern[idx]}*" - for entry2 in os.scandir(directory): - if ( - entry2.is_file() - and (fnmatch.fnmatch(entry2.name, pattern)) - and entry2.stat().st_size > 0 - ): - # Append matching files - self._names_in_dir[idx].append(entry2.name) - self._paths_in_dir[idx].append( - os.path.join(directory, entry2.name) - ) - - def get_names_in_dir(self, iterable, module, idx): - - # Initialise output file names and paths - self._names_in_dir[idx] = [] - self._paths_in_dir[idx] = [] - - # Loop over subdirs - for jdx, subdir in enumerate(iterable): - base_and_subdir = ( - f"{self._path_main}/" - + f"{self._path_left}/{subdir}/" - + f"{self._path_right[idx]}" - ) - if self._verbose: - print(f"**** base_and_subdir {base_and_subdir}") - - if os.path.isdir(base_and_subdir): - - matches = [] - - # Loop over entries (files and dirs) - with os.scandir(base_and_subdir) as entries: - for entry in entries: - - # Append directory name if matches module - if ( - entry.name.startswith(self._run_dir[idx]) - and not entry.name.endswith("prev") - ): - matches.append(entry) - - # This entry does not match module -> next - if not matches: - continue - - if self._verbose: - print("**** Matching entries: ", end="") - for match in matches: - print(match.name) - - full_path = self.get_last_full_path( - base_and_subdir, - matches, - ) - - # Get module output directory - directory = self.get_module_output_dir( - full_path, - module, - self._path_output[idx], - ) - if self._verbose: - print(f"**** Output dir = {directory}") - - # Find matching file names and paths - self.get_matches_final(directory, idx) - else: - if self._verbose: - print(f"Directory {base_and_subdir} not found") - - def update_subdirs(self, par_runtime): - """Update Subdirs. - - Update subdir names with runtime information if required. - - """ - if not isinstance(self._output_subdirs, list): - self._output_subdirs = get_par_runtime( - par_runtime, self._output_subdirs, kind="list" - ) - - def check_numbers(self, par_runtime=None, indices=None): - """Check Numbers. - - Check output file numbers and IDs. - - Parameters - ---------- - par_runtime : dict, optional - runtime parameter. default is None - indices: list, optional - if not None (default), only check modules corresponding - to indices - - """ - # Update subdirs if not already set as list - self.update_subdirs(par_runtime) - - # Initialise variables - self._names_in_dir = {} - self._paths_in_dir = {} - self._missing_IDs_job = [] - n_missing_job = 0 - - # Loop over modules - for idx, module in enumerate(self._modules): - if indices is not None and idx not in indices: - continue - - if self._verbose: - print(f"** module {module}") - - # Look over subdirs - iterable = self._output_subdirs - if len(iterable) > 1 and self._verbose: - iterable = tqdm(iterable, desc="subdirs", leave=False) - - if self._verbose: - print(f"*** subdirs {self._output_subdirs}") - - # Get output file names and paths - self.get_names_in_dir( - iterable, - module, - idx, - ) - - # If expected is string: Update parameter with runtime value - # and set as integer - if isinstance(self._key_expected[idx], str): - n_expected_base = get_par_runtime( - par_runtime, self._key_expected[idx], kind="n" - ) - else: - n_expected_base = self._key_expected[idx] - - # Get some numbers - n_found = len(self._names_in_dir[idx]) - n_expected = n_expected_base * self._n_mult[idx] - n_missing = n_expected - n_found - - n_special = self.check_special( - module, - idx, - ) - - # Print statistics - self.print_stats( - module, - n_expected, - n_found, - n_special, - n_missing, - idx, - ) - - # Write missing IDs for module to file - if n_missing > 0: - missing_IDs = self.output_missing( - module, - idx, - par_runtime=par_runtime, - ) - n_missing_job += n_missing - self._missing_IDs_job.extend(missing_IDs) - - # Empty line after job - logging.info("") - - # Write missing IDs for entire job to file - # if n_missing_job > 0: - self.output_missing_job() - - -def get_par_runtime(par_runtime, key, kind="n"): - """Get Par RunTime. - - Return runtime parameter value. - - Parameters - ---------- - par_runtime: dict - runtime parameter - key: str - key - - """ - combined_key = f"{kind}_{key}" - - return par_runtime[combined_key] - - -def print_par_runtime(par_runtime, verbose=True): - # Print runtime parameter values - if True: - logging.info("") - logging.info("===========") - logging.info("par_runtime") - logging.info("-----------") - for key, value in par_runtime.items(): - if not key.startswith("list"): - logging.info(f"{key:30s} {value:6d}") - else: - # logging.info(f"{key:30s} {len(value):6d} entries") - pass - logging.info("===========") - logging.info("") diff --git a/src/shapepipe/utilities/summary_params_pre_v2.py b/src/shapepipe/utilities/summary_params_pre_v2.py deleted file mode 100644 index ea67442e3..000000000 --- a/src/shapepipe/utilities/summary_params_pre_v2.py +++ /dev/null @@ -1,276 +0,0 @@ -# Parameters for summary run - -import logging -import os - -from shapepipe.utilities import summary - - -def set_jobs_v2_pre_v2(patch, verbose): - """Return information about shapepipe jobs""" - print(f"Set job info for patch {patch}") - - # Main input and output directory - path_main = f"{os.environ['HOME']}/cosmostat/v2/pre_v2/psfex/{patch}" - - # Logging - path = f"{path_main}/summary" - if not os.path.isdir(path): - os.mkdir(path) - log_file_name = f"{path}/summary_log.txt" - handlers = [ - logging.FileHandler(log_file_name, mode="w"), - logging.StreamHandler(), - ] - logging.basicConfig( - level=logging.INFO, format="%(message)s", handlers=handlers - ) - - logging.info(f"Checking main directory = {path_main}") - - # Tile IDs - tile_ID_path = f"{path_main}/tile_numbers.txt" - - ## Tile IDs with dots - list_tile_IDs_dot = summary.get_IDs_from_file(tile_ID_path) - - jobs = {} - - # Set the first job (retrieve images) - - # With "CFIS_" only the linked images are counted. The original - # ones do not match the IDdash pattern. - # If images were downloaded in several runs: - # - Only copy original images, then (re-)set links in SP numbering format - # - get_images_runner_run_[12] consistent - # - remove previous output dirs since only last is searched - jobs["1"] = summary.job_data( - 1, - "run_sp_GitFeGie", - [ - "get_images_runner_run_1", - "find_exposures_runner", - "get_images_runner_run_2", - ], - ["tile_IDs", "tile_IDs", "exposures"], - pattern=["CFIS_", "", ""], - n_mult=[2, 1, 3], - path_main=path_main, - path_left="output", - verbose=verbose, - ) - - if patch in ("P8", "P9"): - jobs["2"] = summary.job_data( - 2, - ["run_sp_Uz", "run_sp_exp_Sp_shdu"], - ["uncompress_fits_runner", "split_exp_runner"], - ["tile_IDs", "shdus"], - n_mult=[1, 3], - path_main=path_main, - path_left=["output", "exp_runs"], - output_subdirs=[None, "shdus"], - path_right=[None, "output"], - verbose=verbose, - ) - jobs["4096"] = summary.job_data( - 4096, - ["run_sp_exp_Sp_shdu"], - ["split_exp_runner"], - ["shdus"], - n_mult=4, - path_main=path_main, - path_left="exp_runs", - output_subdirs="shdus", - path_right="output", - verbose=verbose, - ) - else: - jobs["2"] = summary.job_data( - 2, - ["run_sp_Uz", "run_sp_exp_SpMh"], - ["uncompress_fits_runner", "split_exp_runner"], - ["tile_IDs", "shdus"], - n_mult=[1, 121], - path_main=path_main, - path_left="output", - verbose=verbose, - ) - - jobs["4"] = summary.job_data( - 4, - ["run_sp_Ma_tile"], - ["mask_runner"], - ["tile_IDs"], - path_main=path_main, - path_left="output", - verbose=verbose, - ) - - if patch not in ("P8", "P9"): - jobs["8"] = summary.job_data( - 8, - ["run_sp_Ma_exp"], - ["mask_runner"], - ["shdus"], - path_main=path_main, - path_left="output", - verbose=verbose, - ) - else: - jobs["8"] = summary.job_data( - 8, - ["run_sp_exp_Ma"], - ["mask_runner"], - ["shdus"], - n_mult=[1], - path_main=path_main, - path_left="exp_runs", - output_subdirs= "shdus", - path_right="output", - verbose=verbose, - ) - - - jobs["16"] = summary.job_data( - 16, - "run_sp_tile_Sx", - ["sextractor_runner", "sextractor_runner"], - "tile_IDs", - n_mult=[2, 1], - path_main=path_main, - path_left="tile_runs", - path_output=["output", "logs"], - output_subdirs=[f"{tile_ID}/output" for tile_ID in list_tile_IDs_dot], - special=[False, True], - verbose=verbose, - ) - - jobs["32"] = summary.job_data( - 32, - [ - "run_sp_exp_SxSePsf", - "run_sp_exp_SxSePsf", - "run_sp_exp_SxSePsf", - "run_sp_exp_SxSePsf", - "run_sp_exp_SxSePsf", - ], - [ - "sextractor_runner", - "setools_runner", - "setools_runner", - "setools_runner", - "psfex_runner", - ], - "shdus", - n_mult=[2, 2, 1, 1, 2], - path_main=path_main, - path_left="exp_runs", - output_subdirs="shdus", - path_right="output", - path_output=[ - "output", - "output/rand_split", - "output/stat", - "logs", - "output", - ], - special=[False, False, True, True, False], - verbose=verbose, - ) - - jobs["64"] = summary.job_data( - "64", - "run_sp_tile_PsViSmVi", - [ - "psfex_interp_runner", - "psfex_interp_runner", - "vignetmaker_runner_run_1", - "spread_model_runner", - "vignetmaker_runner_run_2", - ], - "tile_IDs", - n_mult=[1, 1, 1, 1, 4], - path_main=path_main, - path_left="tile_runs", - output_subdirs=[f"{tile_ID}/output" for tile_ID in list_tile_IDs_dot], - path_output=["output", "logs", "output", "output", "output"], - special=[False, True, False, False, False], - verbose=verbose, - ) - - if patch in ("P2", "P5", "P8", "P9"): - n_sh = 1 - else: - n_sh = 8 - run_dirs = [f"run_sp_tile_ngmix_Ng{idx+1}u" for idx in range(n_sh)] - - # Add special (unfinished run) - run_dirs.append("run_sp_tile_ngmix_Ng1u") - - output_path_missing_IDs = [ - f"{path_main}/summary/missing_job_128_ngmix_runner_{idx+1}.txt" - for idx in range(n_sh + 1) - ] - jobs["128"] = summary.job_data( - "128", - run_dirs, - ["ngmix_runner"] * (n_sh + 1), - "tile_IDs", - path_main=path_main, - path_left="tile_runs", - output_subdirs=[f"{tile_ID}/output" for tile_ID in list_tile_IDs_dot], - path_output=["output"] * n_sh + ["logs"], - special=[False] * n_sh + [True], - output_path_missing_IDs=output_path_missing_IDs, - verbose=verbose, - ) - - jobs["256"] = summary.job_data( - "256", - "run_sp_tile_Ms", - ["merge_sep_cats_runner"] * 2, - "tile_IDs", - path_main=path_main, - path_left="tile_runs", - path_output=["output", "logs"], - special=[False, True], - output_subdirs=[f"{tile_ID}/output" for tile_ID in list_tile_IDs_dot], - verbose=verbose, - ) - - jobs["512"] = summary.job_data( - "512", - ["run_sp_tile_Mc"], - ["make_cat_runner"], - "tile_IDs", - path_main=path_main, - path_left="tile_runs", - output_subdirs=[f"{tile_ID}/output" for tile_ID in list_tile_IDs_dot], - verbose=verbose, - ) - - # Post-processing - jobs["1024"] = summary.job_data( - "1024", - ["run_sp_combined_final"], - ["make_catalog_runner"], - "tile_IDs", - path_main=path_main, - path_left="output", - verbose=verbose, - ) - - jobs["2048"] = summary.job_data( - "2048", - "run_sp_combined_psf", - ["psfex_interp_runner"], - "shdus", - path_main=path_main, - path_left="output", - verbose=verbose, - ) - - return jobs, list_tile_IDs_dot - - diff --git a/tests/module/test_collate_star_cat.py b/tests/module/test_collate_star_cat.py deleted file mode 100644 index 20d52ce23..000000000 --- a/tests/module/test_collate_star_cat.py +++ /dev/null @@ -1,70 +0,0 @@ -"""UNIT TESTS FOR STAR-CATALOGUE COLLATION PATHS. - -Pin the patch vs patch-less (v2.0) path and filename convention of -``scripts/python/collate_star_cat.py``. Runs up to v1.6 carry a ``P`` -token in both the input run directory and the output filename; v2.0 is -patch-less (``patch is None``) and drops that token, reading from a single -``/output`` root and writing ``validation_psf_conv-.fits`` — the -name still matched by the downstream ``validation_psf_conv-*`` glob. -""" - -import importlib.util -from pathlib import Path - -import pytest - -# The collation script lives under scripts/python (not an importable package), -# so load it by path. -_SCRIPT = ( - Path(__file__).resolve().parents[2] - / "scripts" - / "python" - / "collate_star_cat.py" -) -_spec = importlib.util.spec_from_file_location("collate_star_cat", _SCRIPT) -collate_star_cat = importlib.util.module_from_spec(_spec) -_spec.loader.exec_module(collate_star_cat) - - -@pytest.mark.parametrize( - "patch, exp_input, exp_output", - [ - ("3", "in/P3/output/", "out/P3"), - (None, "in/output/", "out"), - ], -) -def test_collate_paths(patch, exp_input, exp_output): - """v1.x carries the P token; v2.0 (patch None) drops it.""" - assert collate_star_cat.collate_paths("in", "out", patch) == ( - exp_input, - exp_output, - ) - - -@pytest.mark.parametrize( - "patch, expected", - [ - ("3", "validation_psf_conv-3-0.fits"), - (None, "validation_psf_conv-0.fits"), - ], -) -def test_output_filename(patch, expected): - """The patch token is present for v1.x and absent for v2.0.""" - assert collate_star_cat.output_filename("validation_psf", patch, 0) == expected - - -def test_output_filename_matches_downstream_glob(): - """Both layouts stay under the downstream ``validation_psf_conv-*`` glob.""" - for patch in ("1", None): - assert collate_star_cat.output_filename( - "validation_psf", patch, 5 - ).startswith("validation_psf_conv-") - - -@pytest.mark.parametrize("bad", ["v2", "2.0", "v1.7", ""]) -def test_invalid_version_raises(bad): - """A mistyped -V is rejected rather than falling through to v1.x.""" - obj = collate_star_cat.Convert() - obj._params["version_cat"] = bad - with pytest.raises(ValueError): - obj.run() diff --git a/tests/module/test_coverage.py b/tests/module/test_coverage.py index f7808a375..9d1595887 100644 --- a/tests/module/test_coverage.py +++ b/tests/module/test_coverage.py @@ -1,9 +1,8 @@ """UNIT / PROPERTY TESTS FOR THE COVERAGE-MASK FEATURE. Covers the pure and lightly-fixtured logic behind the per-CCD coverage nexp -masks: exposure-number parsing, the CCD-list -> unique-exposure reduction, the -handler's missing-CCD subtraction (pinned against the real ID format), per-CCD -corner extraction from plain and fpack-compressed multi-HDU headers, +masks: exposure-number parsing, the CCD-list -> unique-exposure reduction, +per-CCD corner extraction from plain and fpack-compressed multi-HDU headers, ``--ccd_list`` filtering, the per-CCD resume path, the builder's per-CCD row parsing, the accumulated exposure-count (nexp) contract, the RA-wrap and pole guards, the power-of-two ``nside`` validation, the atomic-download rename, and @@ -22,8 +21,6 @@ from hypothesis import given from hypothesis import strategies as st -from shapepipe.utilities import summary -from shapepipe.utilities.ccd_psf_handler import CcdPsfHandler from shapepipe.utilities.coverage_map_builder import ( CoverageMapBuilder, unwrap_ra, @@ -216,55 +213,6 @@ def failing_copy(source, target, head=True): assert not (tmp_path / "42.txt.part").exists() -# --------------------------------------------------------------------------- -# CcdPsfHandler.get_ccds_with_psf — missing-CCD subtraction -# --------------------------------------------------------------------------- - -def test_get_ccds_with_psf_subtracts_missing(monkeypatch): - """Valid CCDs are all exposure single-HDUs minus the missing set. - - The real ``summary.get_all_shdus`` is used so the cross-component - ``-`` ID format is pinned end to end. - """ - handler = CcdPsfHandler() - - # Two exposures, 3 CCDs each -> 6 candidate CCDs; two are missing. - monkeypatch.setattr(handler, "get_exp", lambda patches: {"100", "200"}) - monkeypatch.setattr( - handler, - "get_exp_shdu_missing", - lambda patches: {"100-1", "200-2"}, - ) - - result = handler.get_ccds_with_psf(["P1"], n_CCD=3) - - # get_all_shdus yields "-" for ccd in range(n_CCD). - assert result == {"100-0", "100-2", "200-0", "200-1"} - # Guard the assumption that the missing IDs share the produced format. - assert set(summary.get_all_shdus({"100"}, 3)) == {"100-0", "100-1", "100-2"} - - -@pytest.mark.parametrize( - ("version", "n_patch"), - [("v1.3", 7), ("v1.4", 7), ("v1.5", 8), ("v1.6", 9)], -) -def test_version_to_patch_count(version, n_patch): - """Each v1.x catalogue version maps to its patch count.""" - handler = CcdPsfHandler() - handler._params["version_cat"] = version - handler.update_params() - assert handler._params["n_patch"] == n_patch - assert len(handler._params["patches"]) == n_patch - - -def test_invalid_version_raises(): - """An unknown catalogue version fails loudly.""" - handler = CcdPsfHandler() - handler._params["version_cat"] = "v9.9" - with pytest.raises(ValueError, match="v9.9"): - handler.update_params() - - # --------------------------------------------------------------------------- # image-shape resolution (fpack ZNAXIS vs plain NAXIS) # --------------------------------------------------------------------------- @@ -684,7 +632,6 @@ def test_run_help_flag_exits_cleanly(monkeypatch): @pytest.mark.parametrize( "runner, prog", [ - (CcdPsfHandler, "get_ccds_with_psf"), (HeaderDownloader, "download_headers"), (FieldCornersExtractor, "extract_field_corners"), (CoverageMapBuilder, "build_coverage_map"), diff --git a/uv.lock b/uv.lock index 4f3d9e20c..212dd82c8 100644 --- a/uv.lock +++ b/uv.lock @@ -3185,7 +3185,6 @@ dependencies = [ { name = "pyqtgraph" }, { name = "python-dateutil" }, { name = "python-pysap" }, - { name = "skaha" }, { name = "skyproj" }, { name = "sqlitedict" }, { name = "termcolor" }, @@ -3277,7 +3276,6 @@ requires-dist = [ { name = "python-pysap", specifier = ">=0.3" }, { name = "ruff", marker = "extra == 'lint'" }, { name = "shapepipe", extras = ["doc", "jupyter", "lint", "plot", "release", "test", "fitsio"], marker = "extra == 'dev'" }, - { name = "skaha", specifier = ">=1.7" }, { name = "skyproj" }, { name = "skyproj", marker = "extra == 'plot'" }, { name = "sphinx", marker = "extra == 'doc'" }, @@ -3309,24 +3307,6 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/b7/ce/149a00dd41f10bc29e5921b496af8b574d8413afcd5e30dfa0ed46c2cc5e/six-1.17.0-py2.py3-none-any.whl", hash = "sha256:4721f391ed90541fddacab5acf947aa0d3dc7d27b2e1e8eda2be8970586c3274", size = 11050, upload-time = "2024-12-04T17:35:26.475Z" }, ] -[[package]] -name = "skaha" -version = "1.7.0" -source = { registry = "https://pypi.org/simple" } -dependencies = [ - { name = "defusedxml" }, - { name = "httpx" }, - { name = "pydantic" }, - { name = "rich" }, - { name = "toml" }, - { name = "typer" }, - { name = "vos" }, -] -sdist = { url = "https://files.pythonhosted.org/packages/76/62/a733508ff88200c41459f5cdb72a797eb8795c833b4e9d51d31b5efe4073/skaha-1.7.0.tar.gz", hash = "sha256:ef9d69e7a4da8653cdbde3e62f18caef137130064ccb6fabed57c7c6f8bf8ef9", size = 54414, upload-time = "2025-05-28T21:17:03.631Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/d9/d3/a3c1d569ae714ed9985adccefdfe2d6dd9f8ad7a297bd8fa4c7ed30c587b/skaha-1.7.0-py3-none-any.whl", hash = "sha256:68b0d3c925b98bf145c5f695237474fe1cea07cf9b69b4a9bae9910375dfa01a", size = 40795, upload-time = "2025-05-28T21:17:02.296Z" }, -] - [[package]] name = "skyproj" version = "2.6.1" diff --git a/workflow/README.md b/workflow/README.md index 02eb6c418..0a7468563 100644 --- a/workflow/README.md +++ b/workflow/README.md @@ -1,13 +1,21 @@ # ShapePipe Snakemake orchestration -Snakemake workflow that orchestrates real-data ShapePipe runs. It replaces the -`curl_canfar_local.sh → run_job_sp_canfar_v2.0.bash → job_sp_canfar_v2.0.bash` -bash layers and the per-site sbatch reimplementations. **Module code is -untouched**: rules call `shapepipe_run -c ` on the existing config -chains. Design and rationale: +Snakemake workflow that orchestrates real-data ShapePipe runs. It is the one +production orchestration: there is no CANFAR submission front end and no +per-site sbatch layer beside it. The two Gen-2 bash scripts +(`run_job_sp_canfar_v2.0.bash → job_sp_canfar_v2.0.bash`) are still in the tree +and still installed (`pyproject.toml`'s `[tool.setuptools] script-files`): +sp_validation's image-simulation workflow calls `run_job_sp_canfar_v2.0.bash` by +path, so they stay until that chain is ported or parked. **Module code is untouched**: rules call +`shapepipe_run -c ` on the existing config chains. Design and rationale: [CosmoStat/shapepipe#848](https://github.com/CosmoStat/shapepipe/issues/848) (the design document). +**There is no concept of a catalogue version in the code.** No path branches on +`v1.3`..`v1.6` or `v2.0`, and there are no sky patches: a campaign is a tile +list, and the version of a catalogue is the git tag of the code that produced +it. + Use `workflow/bin/sp` for everything. Bare `snakemake all` outside `sp` is unsupported: `sp` sets the state directory, the SLURM profile, and `SP_PHASE`, which the Snakefile needs to build the tile/exposure index at @@ -235,14 +243,16 @@ workflow/ scripts/ build_index.py prepare-phase run_index.sqlite builder (plain script) build_forest.py per-tile exposure symlink forest (group-compatible shell) - completeness.py the ported count table (shared by sp_rule + run_report) + completeness.py the ported count-floor table + the `check` every rule ends with run_report.py standalone report (NOT a DAG node; run_report hooks call it) container.py image layers + the resolution order behind `sp container` (stdlib-only) + ngmix_range.py the ngmix chunk partition: written once by tile_vignets, read by each chunk persist_exp.py ONE exposure's keepable PSF products -> one tar on products_dir (the exp_persist rule) hdf5_reconcile.py bring an hdf5 catalogue into agreement with a campaign (shared by both merges) merge_star_cat.py ALL exposures' validation_psf, out of the tars -> full_starcat_.hdf5 merge_final_cat.py ALL tiles' final_cat -> final_cat_.hdf5 (the final_cat_merge rule) clean_exposure.py ONE exposure's store + manifests + logs -> tombstone (the clean_exposure rule) + clean_tile.py ONE finished tile's store -> tombstone (the clean_tile rule) profiles/nibi/config.yaml SLURM executor; apptainer SDM; per-user jobs cap; keep-going ``` @@ -264,10 +274,13 @@ profiles/nibi/config.yaml SLURM executor; apptainer SDM; per-user jobs cap; kee output natively and never touches its log, which is why the profile runs *without* `keep-incomplete`. `sp report` reads both dirs — the manifest for success, the log for failure — and a unit with neither ran nothing. -- **Completeness is an exact-count check, not a taxonomy.** After a run, - `sp_rule.py` counts products per mandatory runner against - `completeness.py`'s expected count and exits nonzero below it. A shortfall - below `expect` fails unless `warn` is set. No 3-class taxonomy, no +- **Completeness is a count floor, not a taxonomy.** There is no per-unit + wrapper script: the Snakefile's `unit_pre()` builds the unit's furniture and + clears the stage's run dir as bash inlined into the rule's `params`, and + `sp_shell()` composes every rule's shell as *prologue, one `shapepipe_run`, + one `completeness.py check`*. That check counts products per mandatory runner + against `completeness.py`'s floor and exits nonzero below it. Per-CCD + attrition between floor and `expect` is tolerated. No 3-class taxonomy, no error-signature whitelist. `--keep-going` isolates a failure to its own DAG cone. - **Stores are sharded.** Every tile/exposure runs its own `shapepipe_run` diff --git a/workflow/config/cfis/config_exp_Gie.ini b/workflow/config/cfis/config_exp_Gie.ini index 9f69fbaee..b6be929c0 100644 --- a/workflow/config/cfis/config_exp_Gie.ini +++ b/workflow/config/cfis/config_exp_Gie.ini @@ -95,5 +95,3 @@ N_TRY = 3 # Retrieve command options, optional RETRIEVE_OPTIONS = --certfile=$HOME/.ssl/cadcproxy.pem - -#CHECK_EXISTING_DIR = $SP_RUN/output/run_sp_Gie_prev diff --git a/workflow/config/cfis/config_exp_Sp.ini b/workflow/config/cfis/config_exp_Sp.ini index 0b24a9562..3326dd89c 100644 --- a/workflow/config/cfis/config_exp_Sp.ini +++ b/workflow/config/cfis/config_exp_Sp.ini @@ -34,8 +34,8 @@ LOG_NAME = log_sp # Runner log file name, optional, default: shapepipe_runs RUN_LOG_NAME = log_run_sp -# NUMBER_LIST selects this unit; the workflow sets SP_UNIT_NUM to the -# dashed exposure ID (exp split is a "tile-scheme" stage per sp_rule.py). +# NUMBER_LIST selects this unit; the workflow sets SP_UNIT_NUM to the dashed +# exposure ID (the Snakefile's unit_num(), exported by unit_pre()). NUMBER_LIST = $SP_UNIT_NUM # Input directory, containing input files, single string or list of names with length matching FILE_PATTERN diff --git a/workflow/config/cfis/config_tile_PiViVi_mccd.ini b/workflow/config/cfis/config_tile_PiViVi_mccd.ini index 864f790f9..a13b72935 100644 --- a/workflow/config/cfis/config_tile_PiViVi_mccd.ini +++ b/workflow/config/cfis/config_tile_PiViVi_mccd.ini @@ -172,8 +172,8 @@ STAMP_SIZE = 51 PREFIX = # Additional parameters for path and file pattern corresponding to single-exposure -# run outputs. ME_IMAGE_EXP_DIR/ME_IMAGE_EXP_RUNNERS replace ME_IMAGE_DIR for -# the v2.0 per-exposure pipeline; output dirs are discovered by scanning $SP_EXP. +# run outputs. Output dirs are discovered by scanning $SP_EXP for the +# exposures listed in the exp_numbers input file. ME_IMAGE_EXP_DIR = $SP_EXP ME_IMAGE_EXP_RUNNERS = split_exp_runner, split_exp_runner, split_exp_runner, sextractor_runner, sextractor_runner ME_IMAGE_PATTERN = flag, image, weight, background, background_rms diff --git a/workflow/config/cfis/config_tile_PiViVi_psfex.ini b/workflow/config/cfis/config_tile_PiViVi_psfex.ini index 62073d482..225f34c78 100644 --- a/workflow/config/cfis/config_tile_PiViVi_psfex.ini +++ b/workflow/config/cfis/config_tile_PiViVi_psfex.ini @@ -110,9 +110,9 @@ CHI2_THRESH = 2 # Multi-epoch mode parameters -# Root directory of per-exposure work directories; replaces ME_DOT_PSF_DIR -# for v2.0 per-exposure pipeline. psfex_runner/output/ dirs are discovered -# by scanning $SP_EXP for the exposures listed in the exp_numbers input file. +# Root directory of per-exposure work directories. psfex_runner/output/ +# dirs are discovered by scanning $SP_EXP for the exposures listed in the +# exp_numbers input file. ME_DOT_PSF_EXP_DIR = $SP_EXP # Input psf file pattern @@ -194,8 +194,8 @@ STAMP_SIZE = 51 PREFIX = # Additional parameters for path and file pattern corresponding to single-exposure -# run outputs. ME_IMAGE_EXP_DIR/ME_IMAGE_EXP_RUNNERS replace ME_IMAGE_DIR for -# the v2.0 per-exposure pipeline; output dirs are discovered by scanning $SP_EXP. +# run outputs. Output dirs are discovered by scanning $SP_EXP for the +# exposures listed in the exp_numbers input file. ME_IMAGE_EXP_DIR = $SP_EXP # @sc [decision:masking.pixel_mask_source] ME_IMAGE_EXP_RUNNERS = split_exp_runner, split_exp_runner, split_exp_runner, sextractor_runner, sextractor_runner