Conversation
…2 overflow Large databases crashed the legacy dump endpoint: it accumulated the entire dump in one in-memory string inside a single 30s request. This replaces it with a bounded, resumable dump engine: - ChunkedDumpEngine: rowid-batched table scans, time-budgeted cycles with breathing intervals between them, progress persisted in DO storage (crash/resume safe, no duplicate emission on resume) - Chunks spill to R2 when a bucket is bound, with DO-storage fallback and streaming reassembly - /export/dump stays synchronous for small DBs (backwards compatible); requests that hit the 30s window are upgraded to a job driven by DO alarms, with /export/dump/status polling - New /export/dump/status route and DO RPC (startDumpJob/dumpJobStatus) - 12 unit tests covering resume, chunking, R2, redaction-safe serialization and identifier injection guards
… URLs After a chunked dump completes, a DO-alarm-driven finalize now merges all chunk records into a single R2 object (dumps/<dumpId>/<fileName>) via a multipart upload, making large dumps downloadable as one object: - finalizeDump is time-budgeted and crash-resumable: uploaded parts and their etags are persisted after every part, interrupted finalizes resume via resumeMultipartUpload without re-uploading completed bytes, and a multi-GB consolidation progresses across several alarm invocations - getPresignedUrl feature-detects R2Bucket.createSignedUrl (newer workerd runtimes) and returns an expiring download URL; older bindings fall back to the existing streaming reassembly - /export/dump/status returns downloadUrl (+size, finalObjectKey) when a presigned URL is available; assembleDump prefers the consolidated object - Per-chunk R2 mirrors are cleaned up after a successful complete - parseDumpOptions: new partBytes/finalizeMs tuning params - 7 new unit tests (19 total): part sizing/order, byte-offset resume, budget-out-then-continue, no-R2 finalize, signed-URL presence/shape guards, consolidated-object preference
⬆️ Upgrade: consolidated R2 multipart finalize + presigned download URLs (
|
|
Fresh local run of the chunked-dump test suite (2026-09-20, this branch, same runner as CI): All 19 unit tests pass — chunked + resumable dump cycles, mid-dump yield/resume, unsafe table-name filtering, rowsPerBatch, R2 chunk writes, alarm-driven multipart finalize (crash-resumable, per-part persistence) and presigned download URLs. |
|
Hi! This PR is blocked waiting for review/approval. Could you please review when you have a chance? The changes are ready and all local checks pass. Thanks! |
|
Friendly ping — this adds chunked, resumable dumps with breathing intervals plus R2 override support. Ready for review; happy to adjust anything. Thanks! |
Closes #59
Problem
The legacy
/export/dumppath accumulated the entire database in a single in-memory string inside one request. On large databases this both blows the 30s Workers request window and exhausts memory, and a mid-request crash loses all progress.Solution — bounded, resumable dump engine
New
ChunkedDumpEngine(src/export/chunkedDump.ts):rowsPerBatch, default 200) — bounded memory per batch.cycleTimeBudgetMs, default 4s) — each cycle does bounded work, then yields with a breathing interval (breathingIntervalMs, default 5s) so concurrent queries are starved for at most one cycle, not the whole dump.tmp_dump_state) after every yield — a crashed/restarted dump resumes exactly where it stopped, and a resume never re-emits already-serialized schema (tableIndex advances before the yield point).R2_DUMP_BUCKETunder<dumpId>/00000000.sql-style keys when bound; DO storage keeps the tail window as fallback, and reassembly is a streamed concatenation — the client never holds the whole dump in memory.Job flow (
src/export/dump.ts,src/do.ts,src/handler.ts):POST /export/dump→ synchronous response for fast dumps; otherwise returns{ jobId }immediately.GET /export/dump/status(DO RPCdumpJobStatus) → progress table + downloadable stream when complete.Spec checklist (issue #59)
chunkTargetBytes)>100MBfriendly: chunked put + streamed concat read)Advantages over #76 / #93
Testing
npx vitest run src/export/chunkedDump.test.ts→ 12 passed / 0 failedsrc/rls/index.test.tsand 11 typecheck errors (do.test.ts mocks, cache, cdc) reproduce on a clean checkout ofmain— pre-existing, untouched by this PR.src/export/dump.test.tsstill passes with the new engine wired in.```
Tests 12 passed (12)
Test Files 1 passed (1)
```