Write metacal flag columns as int32; reduce_mem never narrows integers

[cailmdaley]

The comprehensive catalogue should hold ngmix's metacal flag bitmasks exactly: NGMIX_MCAL_FLAGS and the five per-type NGMIX_FLAGS_{NOSHEAR,1P,1M,2P,2M}. ngmix 2.4.1 uses bits 0–15, from NO_ATTEMPT = 1 to ZERO_DOF = 2**15. Two things on develop stop the bitmasks from surviving:

• Column format. Both parameter files (scripts/calibration/params.py, workflow/image_sims/params_im_sim.py) write NGMIX_MCAL_FLAGS as FITS I, a signed int16, so bit 15 overflows. The per-type columns fall through to float64, because their format entry is keyed FLAGS_* and matches no column.
• JointCat with reduce_mem. dtype_out narrows every int32 column outside a short keep-list to int8, so any value above 127 wraps with no warning.

The galaxy cut NGMIX_MCAL_FLAGS == 0 still rejects these objects, because a wrapped value is still nonzero. What goes wrong is the bit content: you cannot tell from the catalogue why metacal failed. That matters once CosmoStat/shapepipe#854 lands. It flags the objects ngmix never fit (1.03% of the 64-tile smk-g7 catalogue, which currently carry all-zero flags) with ngmix's own bits, e.g. LM_FUNC_NOTFINITE = 2**12, and does not add a custom bit.

Design. All six bitmask columns are written as J (int32). It is the smallest standard signed FITS integer that holds bits 0–15; FITS has no plain unsigned 16-bit format. reduce_mem now only narrows float64 to float32, keeps RA/Dec at float64, and leaves every integer column alone. The alternative, a longer keep-list, would silently break the next integer column someone adds. NGMIX_MCAL_TYPES_FAIL is a count in [0, 5], not a bitmask, so it stays I.

Size. On the v1.5 comprehensive catalogue (585M rows, 290 B/row, 170 GB), only NGMIX_MCAL_FLAGS grows: int16 → int32, +2 B/row, about +1.2 GB (+0.7%). The five per-type columns go from float32 to int32, the same width. No other column changes: the remaining integer columns are already int16/int32 and were not being narrowed, and patch is int8 by construction.

Changes

• Both parameter files: NGMIX_MCAL_FLAGS and NGMIX_FLAGS_* → J, with the per-type format key corrected.
• catalog_builders.py: JointCat.dtype_out as described above. The int8 path and the keep-list are removed.
• catalog.py: an @sc metacal-flag-width contract in the write_shape_catalog docstring.

Tests (test_catalog_flag_roundtrip.py, run in the CI image)

• The flag values 0, 2**12, 2**15, 2**15|2**12|8 and 2**16-1 go in as float64, as in ShapePipe's final catalogue. They are written to all six columns through both parameter files, to FITS and HDF5, with reduce_mem on and off. The test checks exact values, int32 storage, and that dtype_out leaves them intact. All 10 cases pass.
• A second test checks that reduce_mem never narrows integers and still narrows non-coordinate float64.
• Negative controls:
  • Reverting NGMIX_MCAL_FLAGS to I fails the 4 data cases.
  • Restoring develop's catalog_builders.py and data parameter file fails 7 of 10, including every reduce_mem=True case.

For reviewers. This PR does not depend on #854. ngmix's existing ZERO_DOF bit already needs it, so the two can merge in either order. Existing comprehensive catalogues are not rewritten; the fix applies to catalogues built after merge.

— Claude (Opus) on behalf of Cail

🤖 Generated with Claude Code

[cailmdaley]

does 854 have to use 2**30, or was that an arbitrary choice? could this PR be avoided if 854 used a smaller bit?

[cailmdaley]

Martin grumpiliy agrees, let's make sure the int32 is propagated back to the shapepipe PR

[cailmdaley]

LGTM