Skip to content

refactor(storage): group MySQL schema into readmodel and pipeline packages by owning service - #737

Closed
roychying wants to merge 1 commit into
mainfrom
chenghan.ying/schema-ownership-filegroups
Closed

roychying wants to merge 1 commit into
mainfrom
chenghan.ying/schema-ownership-filegroups

Conversation

@roychying

Copy link
Copy Markdown
Contributor

Why?

The gateway and orchestrator own disjoint halves of this schema. That is already the documented design — doc/rfc/submitqueue/status-list-api.md says:

The gateway owns the append-only request log and three new logical read models. The orchestrator's request and change > stores are pipeline working state with different retention semantics, so neither API reads them.

schema/README.md groups the same four tables under a "Gateway request read model" heading, and the code agrees exactly: the gateway tree reaches for those four Storage accessors and no others, the orchestrator tree for the other nine, with no crossover through the shared core/ helpers.

But the split exists only in prose. The whole schema is one filegroup, so a deployment that gives each service its own database has no way to provision just that service's tables — it gets all 13 in both, including tables the service
never reads. This makes the ownership line expressible as a build target.

What?

Two Bazel packages, by owning service:

  • schema/readmodel/ — gateway read model: the append-only request log plus the three materialized projections (4 tables).
  • schema/pipeline/ — orchestrator pipeline working state: requests, batches, builds, changes, speculation (9 tables).

:schema remains the union and remains the default, so a deployment colocating both services on one database is unaffected. The subpackages are for deployments that separate them; they do not assert that anyone must.

Ownership is expressed by file location rather than a list in BUILD.bazel, so adding a table means putting its .sql in the owning package — nothing to keep in sync, and a misfile is visible in the diff path. A checked-in list would silently default a forgotten table into the wrong group, which downstream means creating it in the wrong database.

Three call sites globbed *.sql at a single level and had to learn the tree:

  • test/testutil/schema.goApplySchema walks recursively, so callers keep passing the schema root. Log lines now show the path relative to that root.
  • tool/linter/queueshard — new findSchemaFiles walks each schema root. schemaRoots is unchanged, so a future subpackage needs no linter edit.
  • Makefile — the local-init-submitqueue-schemas glob.

Test Plan

  • bazel build of the schema packages and every integration, e2e, testutil and linter dependent — clean.
  • queueshard reports "All 25 tables are shardable", identical to main, so nothing fell out of the linter's view.
  • make check-gazelle clean; gofmt, go vet, and go test ./tool/linter/queueshard/... pass.
  • ApplySchema's walk verified against a built runfiles tree — where the subdirectories are real directories with symlinked leaf files — and finds all 13.

Issue

@behinddwalls behinddwalls left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

tbh, i don't think it makes sense to separate them like this... each storage extension need to hold on to its schema in itself... the way i think of by moving the extension inside gateway and orchestrator with the split view, not just for storage but for all the extensions i would say.. that way it will have 1:1 mapping for things which are being used with-in gateway and orchstrator, we can do storage first and then others after that

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants