diff --git a/CHANGELOG.md b/CHANGELOG.md index 44a7007eb..17a3d0c85 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,7 @@ - **Fixed** An invalid glob in `--filter` no longer shows its error message twice ([#763](https://github.com/voidzero-dev/vite-task/pull/763)). - **Changed** The detailed summary from `vp run --verbose` and `vp run --last-details` now shows each underlying cause of an error on its own line ([#761](https://github.com/voidzero-dev/vite-task/pull/761)). -- **Added** Remote caching. Configure an endpoint with the workspace's `cache: { remote: { url } }` or `VP_REMOTE_CACHE_URL`, and choose access with `--remote-cache=off|read|read-write` or `VP_REMOTE_CACHE`. The default is `read` with an endpoint and `off` without one. After a local cache miss, `vp run` looks the task up in the remote cache and, on a hit, restores its outputs and caches it locally. The task output and the run summary show which hits came from the remote cache. A failed read is just a cache miss, with the failure as its reason. In `read-write` mode, `vp run` also uploads the results of successful, cacheable tasks after caching them locally. A failed upload doesn't fail the task; the run summary shows a warning instead. Ctrl-C, or a failing task, stops remote cache requests right away, and a task still being looked up doesn't start. Tasks can opt out with `cache: { remote: false }`. Requests use the proxy environment variables or, on macOS and Windows, the system proxy settings ([#727](https://github.com/voidzero-dev/vite-task/pull/727), [#755](https://github.com/voidzero-dev/vite-task/pull/755), [#756](https://github.com/voidzero-dev/vite-task/pull/756), [#757](https://github.com/voidzero-dev/vite-task/pull/757), [#764](https://github.com/voidzero-dev/vite-task/pull/764), [#771](https://github.com/voidzero-dev/vite-task/pull/771)). +- **Added** Remote caching. Configure an endpoint with the workspace's `cache: { remote: { url } }` or `VP_REMOTE_CACHE_URL`, and choose access with `--remote-cache=off|read|read-write` or `VP_REMOTE_CACHE`. The default is `read` with an endpoint and `off` without one. After a local cache miss, `vp run` looks the task up in the remote cache and, on a hit, restores its outputs and caches it locally. The task output and the run summary show which hits came from the remote cache. A failed read is just a cache miss, with the failure as its reason. In `read-write` mode, `vp run` also uploads the results of successful, cacheable tasks after caching them locally. A failed upload doesn't fail the task; the run summary shows a warning instead. Ctrl-C, or a failing task, stops remote cache requests right away, and a task still being looked up doesn't start. Tasks can opt out with `cache: { remote: false }`. Requests use the proxy environment variables or, on macOS and Windows, the system proxy settings ([#727](https://github.com/voidzero-dev/vite-task/pull/727), [#755](https://github.com/voidzero-dev/vite-task/pull/755), [#756](https://github.com/voidzero-dev/vite-task/pull/756), [#757](https://github.com/voidzero-dev/vite-task/pull/757), [#764](https://github.com/voidzero-dev/vite-task/pull/764), [#771](https://github.com/voidzero-dev/vite-task/pull/771), [#772](https://github.com/voidzero-dev/vite-task/pull/772)). - **Fixed** On Windows, environment variable names used by `vp run` now match regardless of ASCII letter case. Assignments in task commands override earlier assignments and inherited variables spelled differently, and `FORCE_COLOR`, `VP_RUN_CONCURRENCY_LIMIT`, and variables requested through `@voidzero-dev/vite-task-client` are found under any spelling ([#747](https://github.com/voidzero-dev/vite-task/pull/747)). - **Changed** A task's cache settings now go inside `cache`, e.g. `cache: { env: ["NODE_ENV"], input: ["src/**"] }`; `cache: true` is the same as `cache: {}`. `env`, `untrackedEnv`, `input`, and `output` are no longer supported at the top level of a task ([#749](https://github.com/voidzero-dev/vite-task/pull/749)). - **Fixed** Cached tasks on macOS no longer intermittently fail with exit 2 and `oils I/O error (main): No such process` when a fast command finishes before the shell gets scheduled. The bundled shell that runs task commands is updated to Oils 0.38.0, which fixes this race ([#702](https://github.com/voidzero-dev/vite-task/issues/702), [#703](https://github.com/voidzero-dev/vite-task/pull/703)). diff --git a/crates/vt/src/session/cache/remote.rs b/crates/vt/src/session/cache/remote.rs index 159563046..18008f28e 100644 --- a/crates/vt/src/session/cache/remote.rs +++ b/crates/vt/src/session/cache/remote.rs @@ -97,20 +97,20 @@ pub(super) struct Restore { /// Turn the result of a fetch into an entry to restore or a miss. `validate` /// checks an exact entry against the current execution. A fallback's miss -/// reason compares its key with `cache_key`. A failed fetch, an entry that -/// doesn't decode or whose blob doesn't match its value, or a validation error -/// is a read failure. +/// reason compares its key with `cache_key`, and no match is a miss without +/// an entry. A failed fetch, an entry that doesn't decode or whose blob +/// doesn't match its value, or a validation error is a read failure. #[expect( clippy::result_large_err, reason = "`CacheMiss` is intentionally large, and a lookup returns it once" )] pub(super) fn resolve( - fetched: Result, + fetched: Result, ReadError>, cache_key: &CacheEntryKey, validate: impl FnOnce(&CacheEntryValue) -> anyhow::Result>, ) -> Result { let (value, blob_id) = match fetched.map_err(ReadError::into_miss)? { - Fetched::Exact { value, blob_id } => { + Some(Fetched::Exact { value, blob_id }) => { let value: CacheEntryValue = deserialize_cache(&value) .map_err(|err| ReadError::CorruptValue(err).into_miss())?; if value.output_archive.is_some() != blob_id.is_some() { @@ -118,11 +118,11 @@ pub(super) fn resolve( } (value, blob_id) } - Fetched::Fallback { key } => { + Some(Fetched::Fallback { key }) => { let key = decode_key(&key).map_err(ReadError::into_miss)?; return Err(CacheMiss::FingerprintMismatch(key.into_mismatch(cache_key))); } - Fetched::NotFound => return Err(CacheMiss::NotFound), + None => return Err(CacheMiss::NotFound), }; match validate(&value) { Ok(None) => Ok(Restore { value, blob_id }), @@ -150,15 +150,15 @@ impl RemoteClients { } /// Fetch the entry stored under `cache_key`, falling back to the entry - /// last stored for `execution_cache_key`. Stops when `cancel_token` is - /// cancelled. + /// last stored for `execution_cache_key`. Returns `None` if neither key + /// matched. Stops when `cancel_token` is cancelled. pub(super) async fn fetch( &self, endpoint: &Arc, cache_key: &CacheEntryKey, execution_cache_key: &ExecutionCacheKey, cancel_token: &CancellationToken, - ) -> Result { + ) -> Result, ReadError> { let client = self.client(endpoint).map_err(ReadError::Fetch)?; let key = encode_key(cache_key)?; let secondary_key = encode_key(execution_cache_key)?; @@ -430,7 +430,8 @@ mod tests { fn exact_entry_that_validates_is_restored() { let key = cache_key(ResolvedGlobConfig::default_auto()); let restore = - resolve(Ok(exact(&cache_value())), &key, validate_against(BTreeMap::new())).unwrap(); + resolve(Ok(Some(exact(&cache_value()))), &key, validate_against(BTreeMap::new())) + .unwrap(); assert_eq!(restore.blob_id.as_deref(), Some("1")); assert_eq!(restore.value.std_outputs[0].content, b"built\n"); assert_eq!(restore.value.duration, Duration::from_millis(5)); @@ -440,7 +441,7 @@ mod tests { fn exact_entry_that_fails_validation_is_a_mismatch() { let current_inputs = BTreeMap::from([(RelativePathBuf::new("src/a.txt").unwrap(), 1)]); let miss = resolve( - Ok(exact(&cache_value())), + Ok(Some(exact(&cache_value()))), &cache_key(ResolvedGlobConfig::default_auto()), validate_against(current_inputs), ) @@ -466,7 +467,7 @@ mod tests { .tracked_env_queries .insert(TrackedEnvQuery::Glob(Str::from("PROBE_[")), BTreeMap::new()); let miss = resolve( - Ok(exact(&value)), + Ok(Some(exact(&value))), &cache_key(ResolvedGlobConfig::default_auto()), validate_against(BTreeMap::new()), ) @@ -480,7 +481,7 @@ mod tests { let mut input_config = ResolvedGlobConfig::default_auto(); input_config.positive_globs.insert(Str::from("src/**")); let miss = resolve( - Ok(Fetched::Fallback { key: stored_key }), + Ok(Some(Fetched::Fallback { key: stored_key })), &cache_key(input_config), not_validated, ) @@ -492,9 +493,9 @@ mod tests { } #[test] - fn not_found_is_a_miss_without_an_entry() { + fn no_match_is_a_miss_without_an_entry() { let key = cache_key(ResolvedGlobConfig::default_auto()); - let miss = resolve(Ok(Fetched::NotFound), &key, not_validated).unwrap_err(); + let miss = resolve(Ok(None), &key, not_validated).unwrap_err(); assert!(matches!(miss, CacheMiss::NotFound), "{miss:?}"); } @@ -512,7 +513,7 @@ mod tests { let mut trailing = serialize_cache(&cache_value()).unwrap(); trailing.push(0); for value in [b"not a cache value".to_vec(), trailing] { - let fetched = Ok(Fetched::Exact { value, blob_id: None }); + let fetched = Ok(Some(Fetched::Exact { value, blob_id: None })); let miss = resolve(fetched, &key, not_validated).unwrap_err(); assert_eq!(read_failure(miss), "remote cache value is corrupt"); } @@ -523,7 +524,8 @@ mod tests { let key = cache_key(ResolvedGlobConfig::default_auto()); let without_archive = CacheEntryValue { output_archive: None, ..cache_value() }; for (value, blob_id) in [(cache_value(), None), (without_archive, Some(Str::from("1")))] { - let fetched = Ok(Fetched::Exact { value: serialize_cache(&value).unwrap(), blob_id }); + let value = serialize_cache(&value).unwrap(); + let fetched = Ok(Some(Fetched::Exact { value, blob_id })); let miss = resolve(fetched, &key, not_validated).unwrap_err(); assert_eq!(read_failure(miss), "remote cache entry's blob doesn't match its value"); } @@ -542,7 +544,7 @@ mod tests { let mut garbage = encode_header().unwrap(); garbage.extend(b"not a cache key"); for stored_key in [b"not a cache key".to_vec(), other_header, garbage] { - let fetched = Ok(Fetched::Fallback { key: stored_key }); + let fetched = Ok(Some(Fetched::Fallback { key: stored_key })); let miss = resolve(fetched, &key, not_validated).unwrap_err(); assert_eq!(read_failure(miss), "remote cache key is corrupt"); } diff --git a/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache/snapshots/corrupt_archive.md b/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache/snapshots/corrupt_archive.md index 98ad169ed..a6d400f99 100644 --- a/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache/snapshots/corrupt_archive.md +++ b/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache/snapshots/corrupt_archive.md @@ -5,7 +5,7 @@ ``` $ vtt write-file dist/output.txt built -[remote-cache] POST /fetch 200 not_found +[remote-cache] POST /fetch 404 [remote-cache] POST /store 200 ``` diff --git a/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache/snapshots/fallback.md b/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache/snapshots/fallback.md index 25d43608e..25f5c1d5a 100644 --- a/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache/snapshots/fallback.md +++ b/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache/snapshots/fallback.md @@ -5,7 +5,7 @@ ``` $ vtt write-file dist/output.txt built -[remote-cache] POST /fetch 200 not_found +[remote-cache] POST /fetch 404 [remote-cache] POST /store 200 ``` diff --git a/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache/snapshots/local_and_remote_hits.md b/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache/snapshots/local_and_remote_hits.md index 5941e8e06..c0a4b271f 100644 --- a/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache/snapshots/local_and_remote_hits.md +++ b/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache/snapshots/local_and_remote_hits.md @@ -5,7 +5,7 @@ ``` $ vtt write-file dist/output.txt built -[remote-cache] POST /fetch 200 not_found +[remote-cache] POST /fetch 404 [remote-cache] POST /store 200 ``` diff --git a/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache/snapshots/read.md b/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache/snapshots/read.md index 3f11f5be7..aa4c5fe2c 100644 --- a/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache/snapshots/read.md +++ b/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache/snapshots/read.md @@ -5,7 +5,7 @@ ``` $ vtt write-file dist/output.txt built -[remote-cache] POST /fetch 200 not_found +[remote-cache] POST /fetch 404 [remote-cache] POST /store 200 ``` diff --git a/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache/snapshots/read_write.md b/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache/snapshots/read_write.md index 70d75c261..699c9871f 100644 --- a/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache/snapshots/read_write.md +++ b/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache/snapshots/read_write.md @@ -7,7 +7,7 @@ The fetch finds no entry. The new execution is uploaded with one store request. ``` $ vtt write-file dist/output.txt built -[remote-cache] POST /fetch 200 not_found +[remote-cache] POST /fetch 404 [remote-cache] POST /store 200 ``` diff --git a/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache/snapshots/restore.md b/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache/snapshots/restore.md index 9d8b31327..16e50049a 100644 --- a/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache/snapshots/restore.md +++ b/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache/snapshots/restore.md @@ -5,7 +5,7 @@ ``` $ vtt write-file dist/output.txt built -[remote-cache] POST /fetch 200 not_found +[remote-cache] POST /fetch 404 [remote-cache] POST /store 200 ``` diff --git a/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache_backend/snapshots/entries_and_associations.md b/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache_backend/snapshots/entries_and_associations.md index 5f9be001f..9b5ad0152 100644 --- a/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache_backend/snapshots/entries_and_associations.md +++ b/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache_backend/snapshots/entries_and_associations.md @@ -3,8 +3,8 @@ ## `remote-cache-server cbor-http POST /fetch --cbor '{"key": '\''A'\'', "secondary_key": '\''S'\''}'` ``` -{"status": 200, "content_type": "application/cbor", "body": {"kind": "not_found"}} -[remote-cache] POST /fetch 200 not_found +{"status": 404, "content_type": "text/plain; charset=utf-8", "body": "Not found"} +[remote-cache] POST /fetch 404 ``` ## `remote-cache-server cbor-http POST /store --form-cbor 'metadata={"key": '\''A'\'', "secondary_key": '\''S'\'', "value": '\''VA'\''}'` diff --git a/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache_backend/snapshots/invalid_requests.md b/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache_backend/snapshots/invalid_requests.md index d294904b2..9d468db74 100644 --- a/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache_backend/snapshots/invalid_requests.md +++ b/crates/vt_bin/tests/e2e_snapshots/fixtures/remote_cache_backend/snapshots/invalid_requests.md @@ -133,6 +133,6 @@ Failed stores did not replace the value or blob. Failed stores did not publish a secondary association. ``` -{"status": 200, "content_type": "application/cbor", "body": {"kind": "not_found"}} -[remote-cache] POST /fetch 200 not_found +{"status": 404, "content_type": "text/plain; charset=utf-8", "body": "Not found"} +[remote-cache] POST /fetch 404 ``` diff --git a/crates/vt_remote_cache/README.md b/crates/vt_remote_cache/README.md index a9bb658f0..150cc62fb 100644 --- a/crates/vt_remote_cache/README.md +++ b/crates/vt_remote_cache/README.md @@ -4,14 +4,14 @@ Client for the [remote cache server API](https://github.com/voidzero-dev/vite-ta `Client::new` takes the configured endpoint, which can include a namespace path, such as `https://cache.example.com/projects/my-project`. Each operation appends its route to that path, so a store goes to `https://cache.example.com/projects/my-project/store`. Endpoints that aren't HTTP or HTTPS URLs are rejected. -`Client::fetch` sends a key and a secondary key as a CBOR map of byte strings. It decodes the response into `Fetched`: an exact match with its value and blob ID, a fallback match with the key it's stored under, or no match. An exact match must include `blob_id`, which is null when there's no blob. The fallback's value and blob ID aren't decoded. +`Client::fetch` sends a key and a secondary key as a CBOR map of byte strings. A 404 response means neither key matched, so `fetch` returns `None` without decoding the body. The body of a 200 response is a CBOR map tagged by `kind`, and `Fetched` decodes only two kinds: `exact`, an exact match with its value and blob ID, and `fallback`, a fallback match with the key it's stored under. Any other kind is a malformed response. An exact match must include `blob_id`, which is null when there's no blob. The fallback's value and blob ID aren't decoded. `Client::download` gets a blob by its ID. After a 200 response, it returns a `Download`, from which the caller reads the blob's chunks as they arrive. `Client::store` sends a multipart request: a CBOR `metadata` part with the key, secondary key, and value as byte strings, and an optional `blob` part streamed from a file. The response body isn't decoded. -Only HTTP 200 counts as success for every operation. Redirects aren't followed, so a redirect fails like any other status. +Only HTTP 200 counts as success for every operation, except that a fetch also accepts 404 as no match. Redirects aren't followed, so a redirect fails like any other status. reqwest configures TLS. It uses the process's default rustls crypto provider, which the client installs as ring unless one is already installed, and verifies certificates with the operating system's verifier. Requests go through the proxy set in `HTTPS_PROXY`, `HTTP_PROXY`, or `ALL_PROXY`, except for hosts in `NO_PROXY`. Without those variables, the system proxy settings are used on macOS and Windows. Connections time out after 10 seconds. Reads time out after 60 seconds, and until the response headers arrive, that limit also covers sending the request. -`Error` names the kind of failure: an invalid endpoint, a client that couldn't be created, a blob file that couldn't be read, a network error (including timeouts and responses that end early), a status other than 200, or a malformed fetch response. Its messages contain no OS-specific details, so they can be shown to users as is. The details are in the source: the underlying error, the parse error for an endpoint that isn't a URL, or the message in an error response's body. Network errors leave out the request URL, since the endpoint may contain credentials, such as a token in its query. +`Error` names the kind of failure: an invalid endpoint, a client that couldn't be created, a blob file that couldn't be read, a network error (including timeouts and responses that end early), a status other than 200 (or 404, for a fetch), or a malformed fetch response. Its messages contain no OS-specific details, so they can be shown to users as is. The details are in the source: the underlying error, the parse error for an endpoint that isn't a URL, or the message in an error response's body. Network errors leave out the request URL, since the endpoint may contain credentials, such as a token in its query. diff --git a/crates/vt_remote_cache/src/lib.rs b/crates/vt_remote_cache/src/lib.rs index 51912d38f..c5f91b2f9 100644 --- a/crates/vt_remote_cache/src/lib.rs +++ b/crates/vt_remote_cache/src/lib.rs @@ -43,11 +43,12 @@ pub enum Error { /// credentials, such as a token in its query. #[error("network error")] Network(#[source] reqwest::Error), - /// The server responded with a status other than 200. The source is the - /// message in the response body, if any. + /// The server responded with a status other than 200, or for a fetch, + /// other than 200 or 404. The source is the message in the response body, + /// if any. #[error("HTTP status {}", .0.as_u16())] Status(StatusCode, #[source] Option), - /// The response body isn't a fetch response. + /// The body of a 200 fetch response isn't an exact or fallback match. #[error("malformed response")] MalformedResponse(#[source] ciborium::de::Error), } @@ -95,7 +96,7 @@ struct FetchRequest<'a> { secondary_key: &'a [u8], } -/// The result of a fetch. +/// The body of a 200 fetch response: an exact or fallback match. #[derive(Debug, PartialEq, Eq, Deserialize)] #[serde(tag = "kind", rename_all = "snake_case")] pub enum Fetched { @@ -117,8 +118,6 @@ pub enum Fetched { #[serde(with = "serde_bytes")] key: Vec, }, - /// Neither key matched an entry. - NotFound, } /// A client for one remote cache endpoint. @@ -160,13 +159,15 @@ impl Client { } /// Fetch the entry stored under `key` with `POST {endpoint}/fetch`, - /// falling back to the entry associated with `secondary_key`. + /// falling back to the entry associated with `secondary_key`. Returns + /// `None` for a 404 response, which means neither key matched. /// /// # Errors /// /// Returns an error if the request fails, the server responds with a - /// status other than 200, or the response isn't a fetch response. - pub async fn fetch(&self, key: &[u8], secondary_key: &[u8]) -> Result { + /// status other than 200 or 404, or the body of a 200 response isn't an + /// exact or fallback match. + pub async fn fetch(&self, key: &[u8], secondary_key: &[u8]) -> Result, Error> { let body = encode_cbor(&FetchRequest { key, secondary_key }); let response = self .http @@ -176,8 +177,14 @@ impl Client { .send() .await .map_err(network_error)?; + if response.status() == StatusCode::NOT_FOUND { + // Read the body so the connection can be reused. It doesn't matter + // if that fails, because the status alone is the answer. + let _ = response.bytes().await; + return Ok(None); + } let body = check_status(response).await?.bytes().await.map_err(network_error)?; - decode_fetched(&body) + decode_fetched(&body).map(Some) } /// Start downloading the blob `blob_id` with @@ -367,9 +374,6 @@ mod tests { decode_fetched(&fallback).unwrap(), Fetched::Fallback { key: b"stored key".to_vec() } ); - - let not_found = cbor_map(vec![("kind", "not_found".into())]); - assert_eq!(decode_fetched(¬_found).unwrap(), Fetched::NotFound); } #[test] @@ -377,6 +381,8 @@ mod tests { for body in [ b"\xffnot cbor".to_vec(), cbor_map(vec![("kind", "unknown".into())]), + // A miss is a 404 response, not a kind. + cbor_map(vec![("kind", "not_found".into())]), // The value must be a byte string. cbor_map(vec![ ("kind", "exact".into()), @@ -458,10 +464,17 @@ mod tests { async fn fetch_posts_the_keys_as_cbor() { let listener = TcpListener::bind("127.0.0.1:0").unwrap(); let client = client_for(&listener); - let body = cbor_map(vec![("kind", "not_found".into())]); + let body = cbor_map(vec![ + ("kind", "exact".into()), + ("value", ciborium::Value::Bytes(b"v".to_vec())), + ("blob_id", ciborium::Value::Null), + ]); let server = std::thread::spawn(move || serve_once(&listener, "HTTP/1.1 200 OK", &body)); - assert_eq!(client.fetch(b"k", b"s").await.unwrap(), Fetched::NotFound); + assert_eq!( + client.fetch(b"k", b"s").await.unwrap(), + Some(Fetched::Exact { value: b"v".to_vec(), blob_id: None }) + ); let request = server.join().unwrap(); assert!(request.starts_with(b"POST /projects/test/fetch HTTP/1.1\r\n")); @@ -470,6 +483,31 @@ mod tests { assert!(request.ends_with(&[b"\r\n\r\n".as_slice(), &keys].concat())); } + #[tokio::test] + async fn fetch_of_a_missing_entry_is_none() { + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let client = client_for(&listener); + let server = std::thread::spawn(move || { + serve_once(&listener, "HTTP/1.1 404 Not Found", b"Not found") + }); + + assert_eq!(client.fetch(b"k", b"s").await.unwrap(), None); + server.join().unwrap(); + } + + #[tokio::test] + async fn fetch_ignores_an_incomplete_404_body() { + let listener = TcpListener::bind("127.0.0.1:0").unwrap(); + let client = client_for(&listener); + // The connection closes before the announced length arrives. + let server = std::thread::spawn(move || { + serve_raw_once(&listener, b"HTTP/1.1 404 Not Found\r\ncontent-length: 100\r\n\r\nNot") + }); + + assert_eq!(client.fetch(b"k", b"s").await.unwrap(), None); + server.join().unwrap(); + } + #[tokio::test] async fn fetch_fails_on_an_error_status() { let listener = TcpListener::bind("127.0.0.1:0").unwrap(); diff --git a/packages/tools/README.md b/packages/tools/README.md index f5489c440..6d815e236 100644 --- a/packages/tools/README.md +++ b/packages/tools/README.md @@ -11,17 +11,18 @@ remote-cache-server cbor-http POST /fetch --cbor "{\"key\": 'A', \"secondary_key `remote-cache-server COMMAND [ARGS...]` starts the backend on a free loopback port and runs the command with `VP_REMOTE_CACHE_URL` set to the endpoint, `http://127.0.0.1:/projects/test`. The fixed base path gives every endpoint a namespace path. The wrapper takes no options and passes all arguments to the command unchanged. The command inherits stdio. When it exits, the server stops and the wrapper exits with the command's exit code. -After the command exits, the wrapper prints one line to stderr for each request it served, in the order of the responses. Each line has the method, the path below the base path, and the status. Fetch responses add their kind: +After the command exits, the wrapper prints one line to stderr for each request it served, in the order of the responses. Each line has the method, the path below the base path, and the status. Successful fetch responses add their kind: ```text -[remote-cache] POST /fetch 200 not_found +[remote-cache] POST /fetch 404 [remote-cache] POST /store 200 +[remote-cache] POST /fetch 200 exact [remote-cache] GET /blob/1 200 ``` State persists in `remote-cache/` in the current directory, so consecutive commands share it. Each E2E case has its own directory and state. `state.json` holds the entries, associations, and next blob ID, with keys and values hex-encoded. Each blob is a file in `remote-cache/blobs/` named by its blob ID. Blob IDs are sequential strings and continue across invocations, keeping snapshots deterministic. -The backend implements `POST /fetch`, `POST /store`, and `GET /blob/{blob_id}` from the [remote cache server API](https://github.com/voidzero-dev/vite-task/pull/713). Keys, values, and blobs are opaque bytes without length limits. There is no authentication. +The backend implements `POST /fetch`, `POST /store`, and `GET /blob/{blob_id}` from the [remote cache server API](https://github.com/voidzero-dev/vite-task/pull/713). A fetch that matches neither key gets a `404` with the plain-text body `Not found`. Keys, values, and blobs are opaque bytes without length limits. There is no authentication. ## CBOR HTTP client diff --git a/packages/tools/src/remote-cache/server.ts b/packages/tools/src/remote-cache/server.ts index 88a1b6f14..76a14e33a 100644 --- a/packages/tools/src/remote-cache/server.ts +++ b/packages/tools/src/remote-cache/server.ts @@ -97,9 +97,10 @@ function cbor(response: ServerResponse, value: unknown): void { /** * A test backend that keeps its state in `directory`: entries and associations * in `state.json`, and each blob in `blobs/` under its ID. Keys, values, and - * blobs remain opaque bytes. After each response, `logRequest` receives a line - * with the method, the route below `basePath`, the status, and for fetch - * responses, the kind. + * blobs remain opaque bytes. A fetch that matches neither key gets a plain-text + * 404. After each response, `logRequest` receives a line with the method, the + * route below `basePath`, the status, and for successful fetch responses, the + * kind. */ export function createCacheServer({ basePath, @@ -119,7 +120,7 @@ export function createCacheServer({ const associations = new Map(Object.entries(state.associations)); let nextBlobId = state.next_blob_id; - /** Respond to `request`, returning the kind of a fetch response. */ + /** Respond to `request`, returning the kind of a successful fetch response. */ async function handle( request: IncomingMessage, response: ServerResponse, @@ -159,8 +160,7 @@ export function createCacheServer({ }); return 'fallback'; } - cbor(response, { kind: 'not_found' }); - return 'not_found'; + throw new RequestError(404, 'Not found'); } if (mediaType(contentType) !== 'multipart/form-data') {