Exposure-level HealSparse maps: instrument defects and exposure count

astra.yaml

@@ -215,6 +215,9 @@ analyses:
       - id: sky_masks
         type: data
         source: UNIONS healsparse mask maps
+      - id: exposure_psf_ccds
+        type: data
+        source: per-exposure split WCS (headers-<exp>.npy) and the persisted validation_psf members (exp_persist)
     outputs:
       - id: masked_measurement_inputs
         type: data
@@ -225,7 +228,97 @@ analyses:
           catalogue's MASK_n<bit> columns.
         inputs: [exposure_flags, sky_masks]
         decisions: [pixel_mask_source, psf_star_mask_veto, sky_mask_application, mask_default_cut]
+      - id: defect_map
+        type: data
+        format: hsp
+        description: >-
+          The campaign's instrument-flag defect map: a boolean healsparse
+          map, True = masked, the union of one fragment per exposure. A
+          product; nothing in the workflow reads it. Data runs only.
+        inputs: [exposure_flags]
+        decisions: [defect_map_from_flags]
+      - id: nexp_map
+        type: data
+        format: hsp
+        description: >-
+          The campaign's exposure-count map: a uint16 healsparse map at the
+          defect map's resolution holding, per pixel, the number of exposures
+          with a valid PSF model covering it. The count sp_validation's
+          npoint cut reads. A product; nothing in the workflow reads it.
+          Built for every fitted-PSF campaign; image simulations on the true
+          PSF record no footprints and skip it.
+        inputs: [exposure_psf_ccds]
+        decisions: [nexp_map_valid_psf_ccds]
     decisions:
+      nexp_map_valid_psf_ccds:
+        label: The exposure-count map stamps only CCDs with a valid PSF model
+        rationale: >-
+          exp_footprint records the four sky corners of each CCD whose
+          validation_psf catalogue exp_persist packed; psfex_interp writes that
+          catalogue on success and returns without it on NOT_ENOUGH_STARS,
+          BAD_CHI2 or FILE_NOT_FOUND, so the member list is the valid-PSF set.
+          Corners come from the split's WCS at pixel edges. nexp_map stamps
+          value 1 per CCD polygon into a uint16 map; the CCDs of one exposure
+          do not overlap, so a pixel's value is the number of exposures that
+          can contribute a PSF-corrected epoch there. It is built from every
+          record on the products root, reclaimed exposures included, and
+          written raw (no median smoothing). There is one count: of exposures,
+          not of pointings. The resolution is the mask ladder's, shared with
+          the defect map.
+          Values:
+          config.yaml#exposure_maps.nexp.enabled = true.
+        default: valid_psf_ccds
+        options:
+          valid_psf_ccds:
+            label: Count exposures whose CCD has a valid PSF model
+          all_ccds:
+            label: Count every exposure whose CCD covers the pixel
+            excluded: true
+            excluded_reason: >-
+              Counts epochs that cannot contribute a shape: a CCD without a
+              PSF model yields no PSF-corrected measurement, so the count
+              overstates depth exactly where the PSF fit failed.
+      defect_map_from_flags:
+        label: Instrument flags rasterized conservatively into a healsparse defect map
+        rationale: >-
+          The per-CCD flag images (bad columns, saturated
+          pixels, bleed trails; the exposure_flags input) otherwise never leave the
+          pixel domain, and the footprint built from CCD-corner WCS cannot
+          subtract them. exp_defect_map samples every nonzero-flag CCD pixel
+          on an oversample x oversample grid spanning its full extent,
+          corners included (exposure_maps.defect.oversample 3 in workflow/config.yaml),
+          maps the samples through the CCD's image-split WCS, and masks every
+          healpix pixel they touch, at the mask ladder's convention (nside
+          131072 over coverage 128, True = masked). Any flag bit counts.
+          defect_map_merge ORs the campaign's fragments into
+          defect_map/defect_map_<campaign>.hsp, with a sidecar recording which exposures
+          are in; a removed or changed fragment forces a rebuild, since a
+          union cannot be un-OR-ed. On 2079612p CCD 0, oversample 2, 3 and 5
+          miss 169, 54 and 5 of 14229 reference healpix pixels. The map is a
+          product no workflow stage consumes (config_tile_Mc.ini's
+          MASK_EXT_PATHS does not name it); image simulations get none, their
+          flag images being all zero.
+          Values:
+          config.yaml#exposure_maps.nside = 131072;
+          config.yaml#exposure_maps.nside_coverage = 128;
+          config.yaml#exposure_maps.defect.oversample = 3.
+        default: rasterize_conservative_os3
+        options:
+          rasterize_conservative_os3:
+            label: "Any-touch rasterization, 3x3 samples per CCD pixel"
+          centre_sampling:
+            label: Mask a healpix pixel only where its centre is flagged
+            excluded: true
+            excluded_reason: >-
+              Erases one-pixel bad columns, the thin geometry #878 exists to
+              keep.
+          none_footprint_from_wcs:
+            label: No defect map; footprint from CCD-corner WCS alone
+            excluded: true
+            excluded_reason: >-
+              The footprint silently includes defective pixels: percent-level
+              area, in exactly the thin small-scale geometry a window function
+              needs.
       pixel_mask_source:
         label: Only the instrument flag image reaches pixels
         rationale: >-

docs/source/exposure_maps.md

@@ -0,0 +1,33 @@
+# Exposure-level maps
+
+A workflow campaign builds two HealSparse maps from its exposures, beside the
+merged catalogues on the products root. Both are per-exposure records combined
+by one campaign job, at the mask ladder's resolution (`nside` 131072 over
+`nside_coverage` 128, set once under `exposure_maps:` in
+`workflow/config.yaml`), and both are produced automatically whenever the input
+can support them — no configuration is needed.
+
+| map | chain | product | built for |
+| --- | --- | --- | --- |
+| defect | `exp_split` → `exp_defect_map` → `defect_map_merge` | `defect_map/defect_map_<run>.hsp` | data runs |
+| exposure count | `exp_persist` → `exp_footprint` → `nexp_map` | `nexp_map/nexp_map_<run>.hsp` | fitted-PSF runs |
+
+**The defect map** is boolean, `True` = masked: every healpix pixel touched by
+a flagged CCD pixel (bad columns, saturated pixels, bleed trails) of any
+exposure. It carries the instrument flags, which otherwise never leave the pixel
+domain, into the same form as the sky masks. Image simulations' flag images are
+blank, so sims build none.
+
+**The exposure-count map** counts, per pixel, the exposures with a valid PSF
+model covering it — the count behind sp_validation's `npoint >= 3` cut.
+`exp_footprint` records the sky corners of each CCD whose PSF fit succeeded;
+`nexp_map` stamps every record on the products root into the map, so it grows as
+tiles are appended. `psf_model: fake` fits no PSF, so sims build none.
+`exposure_maps.nexp.enabled: false` opts a campaign out; the map is rebuilt
+whole, so a campaign appended in many small batches may prefer to build it once
+at the end.
+
+Nothing in the workflow reads either map back. Plot the count map by hand with
+`plot_coverage_map -i <products_dir>/nexp_map/nexp_map_<run>.hsp ...`, using the
+sky windows under `exposure_maps.nexp.plot`. The design and the measurements
+are in `workflow/README.md`.

docs/source/toc.rst

@@ -26,6 +26,7 @@
   configuration
   testing
   workflow
+  exposure_maps
   pipeline_tutorial
 
 .. toctree::

pyproject.toml

@@ -85,9 +85,6 @@ canfar_submit_job   = "shapepipe.canfar_run:run_job"
 canfar_monitor      = "shapepipe.canfar_run:run_log"
 canfar_monitor_log  = "shapepipe.canfar_run:run_monitor_log"
 get_ccds_with_psf   = "shapepipe.get_ccds_run:run_ccd_psf_handler"
-download_headers    = "shapepipe.coverage_run:run_download_headers"
-extract_field_corners = "shapepipe.coverage_run:run_extract_corners"
-build_coverage_map  = "shapepipe.coverage_run:run_build_coverage"
 plot_coverage_map   = "shapepipe.coverage_run:run_plot_coverage"
 
 [tool.uv]

src/shapepipe/coverage_run.py

@@ -1,79 +1,20 @@
 """COVERAGE_RUN
 
-Call coverage processing classes.
+Console entry point for plotting a coverage map.
+
+Building the exposure-count map is the Snakemake ``nexp_map`` rule's job;
+plotting a finished ``.hsp`` is a human act on a durable product, so it stays a
+hand-run command, on the same argument that keeps ``run_report.py`` out of the
+DAG. The plot windows for the UNIONS SGC and NGC fields are in
+``workflow/config.yaml``'s ``exposure_maps.nexp.plot`` block.
 
 Author: Martin Kilbinger <martin.kilbinger@cea.fr>
 
 """
 
-import sys
-
-from shapepipe.utilities.header_downloader import HeaderDownloader
-from shapepipe.utilities.field_corners_extractor import FieldCornersExtractor
-from shapepipe.utilities.coverage_map_builder import CoverageMapBuilder
 from shapepipe.utilities.coverage_plotter import CoveragePlotter
 
 
-def run_download_headers(args=None):
-    """Run Download Headers.
-
-    Download FITS headers from VOSpace for exposures in a CCD list.
-
-    Parameters
-    ----------
-    args : list, optional
-        command line arguments
-
-    Returns
-    -------
-    int
-        exit code
-
-    """
-    obj = HeaderDownloader()
-    return obj.run(args=args)
-
-
-def run_extract_corners(args=None):
-    """Run Extract Corners.
-
-    Extract per-CCD sky-footprint corner coordinates from FITS headers.
-
-    Parameters
-    ----------
-    args : list, optional
-        command line arguments
-
-    Returns
-    -------
-    int
-        exit code
-
-    """
-    obj = FieldCornersExtractor()
-    return obj.run(args=args)
-
-
-def run_build_coverage(args=None):
-    """Run Build Coverage.
-
-    Build HealSparse coverage maps from per-CCD corner coordinates.
-
-    Parameters
-    ----------
-    args : list, optional
-        command line arguments
-
-    Returns
-    -------
-    int
-        exit code
-
-    """
-    obj = CoverageMapBuilder()
-    return obj.run(args=args)
-
-
 def run_plot_coverage(args=None):
     """Run Plot Coverage.
 
@@ -92,13 +33,3 @@ def run_plot_coverage(args=None):
     """
     obj = CoveragePlotter()
     return obj.run(args=args)
-
-
-def main(argv=None):
-    """Main.
-
-    Main program.
-
-    """
-    # Scripts to call coverage classes are created by pyproject.toml
-    return 0