Remove the spread model from ShapePipe

astra.yaml

@@ -1907,27 +1907,22 @@ analyses:
       star_galaxy_classification:
         label: Star/galaxy separation deferred out of the pipeline
         rationale: >-
-          Classification is off and no spread-model input is wired, so the
-          catalogue ships every object and separation happens downstream. The
-          dormant make_cat classifier computes class = sm + 2 sm_err, with
-          stars at |class| < SM_STAR_THRESH and galaxies at class >
-          SM_GAL_THRESH, both read from config (the function defaults, 0.003
-          and 0.01, are bypassed); the committed config sets neither key, so
-          enabling classification alone raises. Unlike Guinot+22
-          (guinot22_spread_model_cut), it applies no s > 0 or magnitude cut.
-          Values:
-          MAKE_CAT_RUNNER.SM_DO_CLASSIFICATION = False.
+          [HARDCODED] ShapePipe does no star/galaxy classification: make_cat
+          keeps every detection, and the catalogue carries no classification
+          column. Separation happens downstream, in sp_validation's size-based
+          selection. The spread-model classifier Guinot+22 used
+          (guinot22_spread_model_cut) is removed from the pipeline.
         default: deferred_downstream
         options:
           deferred_downstream:
             label: No in-pipeline classification; catalogue ships all objects
           spread_model_inline:
             label: spread_model classification in make_cat
             insights: [guinot22_spread_model_cut]
-            description: >-
-              Not wired; needs spread_model_runner after the PSF
-              interpolation, its output added to make_cat's inputs, and both
-              thresholds set.
+            excluded: true
+            excluded_reason: >-
+              It did not work reliably when last run, and it separates stars
+              from galaxies worse than the downstream classification.
       tile_overlap_handling:
         label: Tile-overlap duplicates neither removed nor flagged
         rationale: >-

docs/source/pipeline_tutorial.md

@@ -304,17 +304,16 @@ job_sp TILE_ID -j 16
 is the selection of galaxies as extended objects compared to the PSF.
 First, the PSF model is interpolated to galaxy positions, according to the PSF model
 with `psfex_interp` or `mccd_interp`. Next, postage stamps around galaxies
-of the weights maps are created via `vignetmaker`. Then, the spread model
-is computed by the `spread_model` module. Finally, postage stamps
+of the weights maps are created via `vignetmaker`. Finally, postage stamps
 around galaxies of single-exposure data is extracted with another call
 to `vignetmaker`.
 
 The output directory is
 - `run_sp_MiViSmVi` if the PSF model is `mccd`;
 - `run_sp_tile_PsViSmVi` for the `PSFEx` PSF model.
 
-This corresponds to the MCCD/PSFex interpolation (`Mi`/`Pi`), `vignetmaker` (`Vi`), `spread_model` (`Sm`), and the
-second call to `vignetmaker` (`Vi`).
+This corresponds to the MCCD/PSFex interpolation (`Mi`/`Pi`), `vignetmaker` (`Vi`),
+and the second call to `vignetmaker` (`Vi`).
 
 
 ## Shape measurement
@@ -340,7 +339,7 @@ job_sp TILE_ID -j 64
 This task first merges the `NJOB` parallel `ngmix` output files from the previous step into
 one output file. Then, previously obtained information are pasted into a _final_ shape catalogue via `make_cat`.
 Included are galaxy detection and basic measurement parameters, the PSF model at
-galaxy positions, the spread-model classification, and the shape measurement.
+galaxy positions, and the shape measurement.
 
 Two output directories are created.
 The first one is `run_sp_tile_Ms` for the `merge_sep` run.

docs/source/refs.bib

@@ -193,21 +193,6 @@ @article{dalcin:11
     keywords = "Python, MPI, PETSc",
 }
 
-@article{desai:12,
-   title={THE BLANCO COSMOLOGY SURVEY: DATA ACQUISITION, PROCESSING, CALIBRATION, QUALITY DIAGNOSTICS, AND DATA RELEASE},
-   volume={757},
-   ISSN={1538-4357},
-   url={http://dx.doi.org/10.1088/0004-637X/757/1/83},
-   DOI={10.1088/0004-637x/757/1/83},
-   number={1},
-   journal={The Astrophysical Journal},
-   publisher={IOP Publishing},
-   author={Desai, S. and Armstrong, R. and Mohr, J. J. and Semler, D. R. and Liu, J. and Bertin, E. and Allam, S. S. and Barkhouse, W. A. and Bazin, G. and Buckley-Geer, E. J. and et al.},
-   year={2012},
-   month={Sep},
-   pages={83}
-}
-
 @ARTICLE{ellison:19,
        author = {{Ellison}, Sara L. and {Viswanathan}, Akshara and {Patton}, David R. and {Bottrell}, Connor and {McConnachie}, Alan W. and {Gwyn}, Stephen and {Cuillandre}, Jean-Charles},
         title = "{A definitive merger-AGN connection at z {\ensuremath{\sim}} 0 with CFIS: mergers have an excess of AGN and AGN hosts are more frequently disturbed}",
@@ -574,18 +559,6 @@ @INPROCEEDINGS{marmo:08
       adsnote = {Provided by the SAO/NASA Astrophysics Data System}
 }
 
-@article{mohr:12,
-   title={The Dark Energy Survey data processing and calibration system},
-   url={http://dx.doi.org/10.1117/12.926785},
-   DOI={10.1117/12.926785},
-   journal={Software and Cyberinfrastructure for Astronomy II},
-   publisher={SPIE},
-   author={Mohr, Joseph J. and Armstrong, Robert and Bertin, Emmanuel and Daues, Greg and Desai, Shantanu and Gower, Michelle and Gruendl, Robert and Hanlon, William and Kuropatkin, Nikolay and Lin, Huan and et al.},
-   editor={Radziwill, Nicole M. and Chiozzi, GianlucaEditors},
-   year={2012},
-   month={Sep}
-}
-
 @InProceedings{pandas:10,
   author    = { {W}es {M}c{K}inney },
   title     = { {D}ata {S}tructures for {S}tatistical {C}omputing in {P}ython },

example/cfis_image_sims/config_tile_Mc_psfex.ini

@@ -70,10 +70,4 @@ FILE_EXT = .fits, .sqlite, .fits
 # sections below
 NUMBERING_SCHEME = -000-000
 
-# Star/galaxy spread-model classification. The image-sims pipeline has no
-# spread_model_runner stage (make_cat inputs are sextractor + fake_psf +
-# ngmix only), so there is nothing to classify — disable it. The newer
-# make_cat_runner reads this key unconditionally (NoOptionError if absent).
-SM_DO_CLASSIFICATION = False
-
 SHAPE_MEASUREMENT_TYPE = ngmix

example/unions_800/cat_matched.param

@@ -15,13 +15,6 @@ NGMIX_MCAL_FLAGS
 NGMIX_G1_PSF_ORIG_NOSHEAR
 NGMIX_G2_PSF_ORIG_NOSHEAR
 
-# spread class
-SPREAD_CLASS
-
-# spread model flag and error
-SPREAD_MODEL
-SPREADERR_MODEL
-
 # Number of epochs (exposures)
 N_EPOCH
 NGMIX_N_EPOCH

src/shapepipe/modules/make_cat_package/__init__.py

@@ -7,7 +7,6 @@
 :Parent modules:
 
 - ``sextractor_runner``
-- ``spread_model_runner``
 - ``psfex_interp_runner`` or ``mccd_interp_runner``
 - ``ngmix_runner``
 
@@ -21,30 +20,16 @@
 This module creates a *final* catalogue combining the output of various
 previous module runs. This gathers all relevant information on the measured
 galaxies for weak-lensing post-processing. This includes galaxy detection and
-basic measurement parameters, the PSF model at galaxy positions, the
-spread-model classification, and the shape measurement. Each object is
+basic measurement parameters, the PSF model at galaxy positions, and the
+shape measurement. Every detected object is kept: the catalogue carries no
+star/galaxy classification, which is done downstream. Each object is
 tagged with its source tile via the ``TILE_ID`` column; objects duplicated
 across overlapping tiles are not deduplicated here, and are left to
 downstream selection.
 
 Module-specific config file entries
 ===================================
 
-SM_DO_CLASSIFICATION : bool, optional
-    Adds spread-model star/galaxy classification flag as the column
-    ``SPREAD_CLASS`` to the output if ``True``
-SM_STAR_THRESH : float, optional
-    Threshold :math:`s_{\rm star, thresh}` for star selection; object is
-    classified as a star if
-    :math:`|x s + 2 \sigma_s | < s_{\textrm{star, thresh}}`
-    where :math:`s` is the spread model and :math:`\sigma_s` is the spread
-    model error; default value is ``0.003``
-SM_GAL_THRESH : float, optional
-    Threshold :math:`s_{\rm gal, thresh}` for galaxy selection; object is
-    classified as a galaxy if
-    :math:`s + 2 \sigma_s  > s_{\textrm{gal, thresh}}` where :math:`s` is the
-    spread model and :math:`\sigma_s` is the spread model error; default value
-    is ``0.01``
 SHAPE_MEASUREMENT_TYPE : list
     Shape measurement method; the only valid option is ``ngmix`` (the knob is
     retained as the extension point for a future estimator family)

src/shapepipe/modules/make_cat_package/make_cat.py

@@ -145,78 +145,6 @@ def save_sextractor_data(final_cat_file, sexcat_path, remove_vignet=True):
     return cat_size
 
 
-def save_sm_data(
-    final_cat_file,
-    sexcat_sm_path,
-    do_classif=True,
-    star_thresh=0.003,
-    gal_thresh=0.01,
-    n_obj=-1,
-):
-    r"""Save Spread-Model Data.
-
-    Save the spread-model data into the final catalogue.
-
-    Parameters
-    ----------
-    final_cat_file : file_io.FITSCatalogue
-        Final catalogue
-    sexcat_sm_path : str
-        Path to spread-model catalogue to save. If ``None``, spread_model is
-        set to 99
-    do_classif : bool
-        If ``True`` objects will be classified into stars, galaxies, and other,
-        using the classifier
-        :math:`{\rm class} = {\rm sm} + 2 * {\rm sm}_{\rm err}`
-    star_thresh : float
-        Threshold for star selection; object is classified as star if
-        :math:`|{\rm class}| <` ``star_thresh``
-    gal_thresh : float
-        Threshold for galaxy selection; object is classified as galaxy if
-        :math:`{\rm class} >` ``gal_thresh``
-    nobj : int, optional
-        Number of objects, only used if sexcat_sm_path is ``-1``
-
-    Returns
-    -------
-    int
-        Number of objects saved
-    @sc [decision:catalogue_assembly.star_galaxy_classification]
-    """
-    final_cat_file.open()
-
-    if sexcat_sm_path is not None:
-        sexcat_sm_file = file_io.FITSCatalogue(
-            sexcat_sm_path,
-            SEx_catalogue=True,
-        )
-        sexcat_sm_file.open()
-
-        sm = np.copy(sexcat_sm_file.get_data()["SPREAD_MODEL"])
-        sm_err = np.copy(sexcat_sm_file.get_data()["SPREADERR_MODEL"])
-
-        sexcat_sm_file.close()
-
-    else:
-        sm = np.ones(n_obj) * 99
-        sm_err = np.ones(n_obj) * 99
-
-    final_cat_file.add_col("SPREAD_MODEL", sm)
-    final_cat_file.add_col("SPREADERR_MODEL", sm_err)
-
-    if do_classif:
-        obj_flag = np.ones_like(sm, dtype="int16") * 2
-        classif = sm + 2.0 * sm_err
-        obj_flag[np.where(np.abs(classif) < star_thresh)] = 0
-        obj_flag[np.where(classif > gal_thresh)] = 1
-
-        final_cat_file.add_col("SPREAD_CLASS", obj_flag)
-
-    final_cat_file.close()
-
-    return n_obj
-
-
 def parse_mask_ext_paths(paths_str):
     """Parse Mask Ext Paths.
 

src/shapepipe/modules/make_cat_runner.py

@@ -16,17 +16,15 @@
     version="1.1",
     input_module=[
         "sextractor_runner",
-        "spread_model_runner",
         "psfex_interp_runner",
         "ngmix_runner",
     ],
     file_pattern=[
         "tile_sexcat",
-        "sexcat_sm",
         "galaxy_psf",
         "ngmix",
     ],
-    file_ext=[".fits", ".fits", ".sqlite", ".fits"],
+    file_ext=[".fits", ".sqlite", ".fits"],
     depends=["numpy", "sqlitedict"],
 )
 def make_cat_runner(
@@ -43,35 +41,7 @@ def make_cat_runner(
 
     @sc [decision:masking.sky_mask_application]
     """
-    # Set input file paths
-    if len(input_file_list) == 3:
-        # No spread model input
-        (
-            tile_sexcat_path,
-            galaxy_psf_path,
-            shape1_cat_path,
-        ) = input_file_list
-        sexcat_sm_path = None
-    else:
-        # With spread model input
-        (
-            tile_sexcat_path,
-            sexcat_sm_path,
-            galaxy_psf_path,
-            shape1_cat_path,
-        ) = input_file_list[0:4]
-
-    # Fetch classification options
-    do_classif = config.getboolean(
-        module_config_sec,
-        "SM_DO_CLASSIFICATION",
-    )
-    if do_classif:
-        star_thresh = config.getfloat(module_config_sec, "SM_STAR_THRESH")
-        gal_thresh = config.getfloat(module_config_sec, "SM_GAL_THRESH")
-    else:
-        star_thresh = None
-        gal_thresh = None
+    tile_sexcat_path, galaxy_psf_path, shape1_cat_path = input_file_list
 
     # Fetch shape measurement type
     shape_type_list = config.getlist(
@@ -100,29 +70,9 @@ def make_cat_runner(
 
     # Save SExtractor data
     w_log.info("Save SExtractor data")
-    n_obj = make_cat.save_sextractor_data(final_cat_file, tile_sexcat_path)
-    cat_size_sextractor = n_obj
-
-    # Save spread-model data
-    if sexcat_sm_path is None:
-        w_log.info("No sm cat input, setting spread model to 99")
-    else:
-        w_log.info("Save spread-model data")
-        cat_size_sm = make_cat.save_sm_data(
-            final_cat_file,
-            sexcat_sm_path,
-            do_classif,
-            star_thresh,
-            gal_thresh,
-            n_obj=n_obj
-        )
-
-        if cat_size_sextractor != cat_size_sm:
-            w_log(
-                f"Warnign: SExtractor catalogue {tile_sexcat_path} has different size"
-                + f" ({cat_size_sextractor} than spread_model catalogue"
-                + f" {sexcat_sm_path} ({cat_size_sm})"
-            )
+    cat_size_sextractor = make_cat.save_sextractor_data(
+        final_cat_file, tile_sexcat_path
+    )
 
     # Save shape data
     sc_inst = make_cat.SaveCatalogue(final_cat_file, cat_size_sextractor, w_log)