Skip to content

feat(cache): restore task results from the remote cache - #756

Merged
wan9chi merged 8 commits into
mainfrom
remote-cache-restore
Sep 27, 2026
Merged

wan9chi merged 8 commits into
mainfrom
remote-cache-restore

Conversation

@wan9chi

@wan9chi wan9chi commented Sep 24, 2026 •

Copy link
Copy Markdown
Member

Motivation

#755 uploads task results to the remote cache, but nothing reads them back. Restoring them lets a task that ran on another machine, or before vp cache clean, replay instead of running again.

Changes

  • vt_remote_cache adds fetch (POST {endpoint}/fetch), which returns an exact match, a fallback match with its key, or no match, and download (GET {endpoint}/blob/{blob_id}), which streams the blob's chunks.
  • After a local miss in read or read-write mode, vp run fetches the entry. An exact entry is validated like a local one. Its archive is downloaded to a .tmp file while being checked from the same stream, then renamed, recorded locally, and replayed as a hit. Hits never upload.
  • A fallback, a failed validation, or a failed read is a miss. If the local cache had an entry, its miss reason stays; otherwise the reason comes from the remote cache, such as remote cache fetch failed or downloaded archive is corrupt. Read failures aren't warnings, and --verbose and --last-details show their causes like other errors (refactor(reporter): save errors as a chain of causes #761). After a failed read, the .tmp file is removed and nothing is recorded.

@wan9chi
wan9chi force-pushed the remote-cache-restore branch from bc8a890 to f607879 Compare September 24, 2026 19:03
@wan9chi
wan9chi added this pull request to stack #751 September 24, 2026 19:03
@github-actions

github-actions Bot commented Sep 24, 2026 •

Copy link
Copy Markdown

fspy benchmark

linux

dynamic/launch             change  -0.68%  [-12.89% .. +16.60%]  overhead  +295.50%
dynamic/access             change  -0.42%  [ -5.38% ..  +7.19%]  overhead   +23.02%
dynamic/access-relative    change  +0.30%  [ -2.40% ..  +3.66%]  overhead   +73.69%
dynamic/access-contended   change  -2.99%  [-13.39% ..  +7.78%]  overhead   +52.31%
static/launch              change  +1.54%  [ -7.66% .. +13.17%]  overhead  +706.43%
static/access              change  -0.23%  [ -2.03% ..  +1.71%]  overhead +1472.91%
static/access-relative     change  -0.34%  [ -2.34% ..  +1.90%]  overhead +1915.78%
static/access-contended    change  -0.80%  [ -1.62% ..  +1.78%]  overhead +1833.17%

macos

dynamic/launch             change  +0.09%  [ -5.04% ..  +6.50%]  overhead  +243.59%
dynamic/access             change  +0.34%  [-25.95% .. +104.52%]  overhead    +5.32%
dynamic/access-relative    change  -0.61%  [-18.95% ..  +3.67%]  overhead  +261.70%
dynamic/access-contended   change  +2.02%  [-27.33% .. +24.98%]  overhead    +4.72%

windows

dynamic/launch             change  -0.89%  [ -7.60% ..  +7.13%]  overhead   +26.11%
dynamic/access             change  +0.00%  [ -1.82% ..  +2.43%]  overhead    +1.30%
dynamic/access-relative    change  +0.18%  [ -1.27% ..  +2.00%]  overhead    +1.30%
dynamic/access-contended   change  -0.22%  [ -7.45% ..  +4.03%]  overhead    +1.23%

@wan9chi
wan9chi force-pushed the remote-cache-restore branch 3 times, most recently from 1935fe9 to 5ff9dff Compare September 25, 2026 02:28
@wan9chi
wan9chi force-pushed the remote-cache-restore branch from 5ff9dff to 75d1019 Compare September 25, 2026 02:29
@wan9chi
wan9chi removed this pull request from stack #751 September 25, 2026 02:34
@wan9chi
wan9chi added this pull request to stack #759 September 25, 2026 02:35
@wan9chi
wan9chi force-pushed the remote-cache-restore branch 5 times, most recently from e440c57 to ead9cc6 Compare September 27, 2026 12:24
@wan9chi
wan9chi force-pushed the remote-cache-restore branch from ead9cc6 to 1ff7a83 Compare September 27, 2026 12:40
Base automatically changed from remote-cache-upload to main September 27, 2026 12:45
@wan9chi
wan9chi force-pushed the remote-cache-restore branch from 1ff7a83 to ee2b6c4 Compare September 27, 2026 12:45
wan9chi added a commit that referenced this pull request Sep 27, 2026
## Motivation

The run summary saved each kind of error in its own way. Task errors used a six-variant enum whose message prefixes were rebuilt by hand when rendering. Upload failures picked their details separately for each client error variant. Task communication errors kept only their top message. #756 adds another set of these rules for remote cache read failures.

`SavedError` replaces all of them. It keeps an error's message and the messages of its sources, so any error can be saved without its own conversion. `UploadError` now derives `thiserror`, which exposes its sources.

The compact summary still shows only the message. `--verbose` and `--last-details` now show each cause on its own line below it:

```
      ⚠ Not uploaded to the remote cache: network error
        ↳ error sending request for url (http://127.0.0.1:0/projects/test/store)
        ↳ client error (Connect)
        ↳ tcp connect error
        ↳ <os error>
```

The `✗` line printed after a failed task still shows the whole chain on one line. The e2e redaction now also hides OS error messages at the start of a cause line.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@wan9chi
wan9chi force-pushed the remote-cache-restore branch from 4ddb9ad to 348fc6a Compare September 27, 2026 13:30
wan9chi and others added 5 commits September 27, 2026 23:44
After a local miss in `read` or `read-write` mode, fetch the entry from
the remote cache. An exact entry that passes validation is restored: its
output archive is downloaded into the cache directory, checked by
decoding it, and the entry is recorded locally before it replays as a
hit. Hits never upload.

A fallback entry, a failed validation, or a failed read is a miss. When
the local cache has no entry for the task, the miss reason comes from the
remote cache. Read failures produce no warning.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
An error while validating a remote entry now makes the lookup a miss with
the reason "remote cache entry couldn't be validated", instead of failing
the task. Local validation is unchanged.

Exact fetch responses must include `blob_id`, using null for no blob, so a
response without it is a malformed response rather than a hit that
restores no outputs.

The decision from a fetch result to a hit or miss moves into
`remote::resolve`, with unit tests for each outcome.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The client returns the blob's chunks instead of writing them to a file.
Each chunk is written to the archive file and passed to the archive check,
which reads the chunks on a blocking thread as they arrive, so the archive
isn't read back from disk. A failed check stops the download.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
wan9chi and others added 3 commits September 27, 2026 23:44
Follow #761: a read failure's miss carries the error, and the run summary
saves it as a `SavedError`. The miss reason is the error's message, such as
`remote cache fetch failed`. `--verbose` and `--last-details` show each cause,
such as the network error, on its own line below it.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Fetching and uploading each name every access mode, so adding one fails
to compile until both decide what it does.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The archive is downloaded to a `.tmp` file and renamed once the check
passes. The chunks flow one way: from the network to the check on a
blocking thread, which writes each one to the file as it takes it. The
file is closed before it's renamed or removed, even after a network error.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@wan9chi
wan9chi force-pushed the remote-cache-restore branch from 0a55996 to aefe44a Compare September 27, 2026 15:46
@wan9chi
wan9chi marked this pull request as ready for review September 27, 2026 16:31
@wan9chi
wan9chi merged commit d5c659c into main Sep 27, 2026
19 checks passed
@wan9chi
wan9chi deleted the remote-cache-restore branch September 27, 2026 16:32
wan9chi added a commit that referenced this pull request Sep 27, 2026
## Motivation

A review of #755 and #756 found a few small problems in remote cache
requests and restores.

## Changes

- Redirects aren't followed. A store redirected to a page that answers
200, such as a login page, now fails with the redirect's status instead
of counting as an upload.
- The system proxy settings are used on macOS and Windows, as with
reqwest's default features. Proxy environment variables already worked.
- Network errors leave out the request URL, so credentials in the
endpoint, such as `?token=…`, aren't saved in the run summary or shown
by `--last-details`.
- An exact entry whose value has an output archive but no blob ID, or a
blob ID but no archive, is a miss with the reason `remote cache entry's
blob doesn't match its value`. Before, an entry without a blob ID was
recorded and replayed without its outputs.
- A download stops as soon as the archive check fails, instead of
waiting for the server's next chunk.

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant