Skip to content

Add Stage 12 parallel simulation workers - #682

Draft
anth-volk wants to merge 13 commits into
mainfrom
stage-12-modal-worker-foundation
Draft

anth-volk wants to merge 13 commits into
mainfrom
stage-12-modal-worker-foundation

Conversation

@anth-volk

@anth-volk anth-volk commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

Fixes #681

Same-repository draft PR check

  • This PR is a draft.
  • This PR is from a branch in PolicyEngine/policyengine-sim-api, not a personal fork.

Summary

  • Add a separately named Stage 12 Modal application with a report coordinator and distinct baseline and reform simulation calls.
  • Publish and consume a separate v2 version manifest populated from PolicyEngine.py .py bundle metadata.
  • Keep the existing Modal HTTP routing service and versioned v1 executors authoritative for all user-visible results.
  • Add temporary authenticated direct Stage 12 submission and status routes, excluded from OpenAPI and documented for removal in Stage 14.
  • When STAGE12_ENABLED=1, automatically submit a temporary comparison run after a supported production annual society-wide report is accepted.
  • Have Cloud Run prepare deterministic parent metadata in memory, perform no Stage 12 database write during submission, and wait at most five seconds only for Modal to acknowledge the coordinator invocation.
  • Have the Modal coordinator atomically create or resolve the parent, suppress duplicate active or complete runs, own all parent and child writes, and run calculation, aggregation, production-result retrieval, and comparison after the Cloud Run response is released.
  • Compare the complete production and Stage 12 aggregate result objects exactly and write private comparison metadata without returning it to users.
  • Remove the shared runtime-control object, process-local dispatch queue, one-time control initializer, periodic database cleanup function, and unused artifact-prefix deletion helper.
  • Validate the actual restricted PostgreSQL credential, required comparison columns, comparison-status transition, and Modal service-account storage credential during deployment.
  • Limit each Stage 12 country worker function and the report-coordinator function to 10 Modal containers.

Configuration and deployment behavior

STAGE12_ENABLED is the only Stage 12 execution setting:

  • missing or 0: automatic Stage 12 runs are disabled and direct Stage 12 POST submissions return HTTP 503 before any database write or Modal invocation;
  • 1: every supported newly accepted annual society-wide report starts a Stage 12 run, and authenticated direct submissions are accepted; and
  • any other value: the Cloud Run service fails startup validation.

The authenticated Stage 12 status route remains available for existing runs whenever its resources are configured. The beta and prod GitHub environments both currently define STAGE12_ENABLED=0. Stage 12 workers and the separate v2 manifest remain deployed when execution is disabled. Changing the value requires another Cloud Run deployment and does not change the existing production forwarding destination.

Cloud Run and the Modal v2 application use the same environment-specific restricted PostgreSQL runtime credential. Cloud Run uses it only for temporary status reads. The Modal report coordinator owns all temporary parent and child writes. Neither runtime receives schema-migration permission.

Deployment dependency

PolicyEngine/policyengine-api#3842 must merge and its additive v2 Alembic revision must run before this PR merges because this runtime writes the new comparison-result columns. Deployment reads the configured database's schema metadata and exercises comparison_status inside a rolled-back canary transaction, so a missing migration fails before the Cloud Run revision is promoted. The production restricted database URL secret and environment configuration must also pass deployment validation. This PR does not require or perform a public traffic transition.

Testing

  • simulation contract: 105 passed
  • Cloud Run simulation entry service: 139 passed
  • simulation executor: 571 passed, 24 skipped, 2 deselected
  • Ruff checks for every changed Python module
  • complete simulation-entry Pyright check
  • focused source Pyright checks for the changed executor modules
  • infrastructure shell syntax and deployment-contract tests from the preceding implementation commits
  • git diff --check

The repository-wide contract and executor Pyright commands still report pre-existing errors in unrelated modules and deliberately invalid test inputs; the changed source modules pass focused type analysis.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add the Stage 12 parallel v2 simulation path

1 participant