[pull] master from apify:master

docs/guides/result_storage.mdx

@@ -21,12 +21,15 @@ In Crawlee, the key-value store is represented by the <ApiLink to="core/class/Ke
 The data is stored in the directory specified by the `CRAWLEE_STORAGE_DIR` environment variable as follows:
 
 ```
-{CRAWLEE_STORAGE_DIR}/key_value_stores/{STORE_ID}/{KEY}.{EXT}
+{CRAWLEE_STORAGE_DIR}/key_value_stores/{STORE_ID}/{KEY}
+{CRAWLEE_STORAGE_DIR}/key_value_stores/{STORE_ID}/{KEY}.__metadata__.json
 ```
 
 :::note
 
-`{STORE_ID}` is the name or the ID of the key-value store. The default key-value store has ID `default`. The `{KEY}` is the key of the record and `{EXT}` corresponds to the MIME content type of the data value.
+`{STORE_ID}` is the name or the ID of the key-value store. The default key-value store has ID `default`. The `{KEY}` is the key of the record, used as the filename verbatim — no extension is derived from the content type, which is recorded in the `{KEY}.__metadata__.json` sidecar instead.
+
+A file you drop into the directory by hand is adopted as a record when the store is opened, keyed by its filename (see the [v4 upgrading guide](../upgrading/upgrading-to-v4#out-of-band-key-value-files-eg-a-hand-placed-inputjson) for the run-input exception). Files added while the crawler is running are not picked up — write those with `setValue()`.
 
 :::
 

docs/upgrading/upgrading_v4.md

@@ -1866,14 +1866,20 @@ Because the in-memory queue lives entirely within a single process and is never
 
 #### Out-of-band key-value files (e.g. a hand-placed `INPUT.json`)
 
-Keys are literal. `aaa` and `aaa.json` are two distinct keys, and `FileSystemStorageBackend` never infers a key from a file's extension. In v3 a hand-placed `aaa.json` in the store directory was readable as `aaa`; in v4 it is not read, not listed, and gets deleted by the purge of the default store on start like any other untracked file.
+Keys are literal. `aaa` and `aaa.json` are two distinct keys, and `FileSystemStorageBackend` never infers a key from a file's extension — in v3 a hand-placed `aaa.json` was readable as `aaa`, in v4 it is not.
 
-The one exception is the run input. `FileSystemStorageBackend` only fully tracks records it wrote itself (those have a `<key>.__metadata__.json` sidecar), but for the `INPUT` key (and the configured `inputKey`) it still reads a value file placed in the store directory out-of-band — such as a hand-written or platform-provided `INPUT.json` — by probing the requested key plus the `.json` extension. A few behaviors around these "bare" files changed in v4:
+What v4 does instead is *adopt* value files that turn up in a store directory without the `<key>.__metadata__.json` sidecar that marks a record — the Apify CLI's input, a project template, a v3 store directory, a file you dropped in with an editor. Opening the store writes the missing sidecar (the value bytes are never touched), and from then on the file is an ordinary record: read by `getValue`, enumerated by `listKeys`, removed by `deleteValue`. Two rules decide the key:
 
-- **Only `INPUT` and `INPUT.json` are probed.** v3 also fell back to `INPUT.txt` and `INPUT.bin`. In v4 those files are not read (`getValue('INPUT')` returns `undefined`), are not listed, and are no longer exempt from the purge of the default store on start, so they get deleted like any other untracked file. Rename a `.txt`/`.bin` input to `INPUT.json` (or drop the extension) before upgrading.
-- **Extensionless bare files report `application/octet-stream`.** In v3 a bare value file with no extension was read as `text/plain`. In v4 the client is a plain byte transport and only infers a content type from a real extension, so an extensionless file now comes back as `application/octet-stream`. Give the file a `.json` extension if you need a more specific type.
-- **Malformed bare files are no longer silently swallowed.** In v3 a bare `INPUT.json` containing invalid JSON was treated as a missing record (`getValue` returned `undefined`). In v4 the raw bytes are returned verbatim and parsing happens in the `KeyValueStore` frontend, so a malformed value now surfaces a parse error at read time instead of looking absent.
-- **Bare files are enumerated by `listKeys` under their actual on-disk name.** A bare `INPUT.json` shows up in `listKeys` as `INPUT.json` and reads back cleanly under that key via `getValue` / `recordExists` / `getPublicUrl`; the logical `INPUT` lookup keeps resolving the same file as well. An extensionless bare file is listed as `INPUT`. If both a tracked `INPUT` record and a bare `INPUT.json` exist, the tracked record wins and the bare variant is not listed. Everything `listKeys` needs is read from the filesystem index, so this no longer triggers the per-read O(n) directory scans the v3 fallback performed.
+- In the **default** store, the run-input keys (`INPUT` and the configured `inputKey`) claim a bare `INPUT` or `INPUT.json`. The key is `INPUT` while the file keeps its name, so `listKeys` reports `INPUT`, `getValue('INPUT.json')` is `undefined`, and `getPublicUrl('INPUT')` points at `INPUT.json`. If both files are present, opening the store fails instead of guessing which one is the input.
+- Every other sidecar-less file becomes a record **keyed by its filename**, in every store: a hand-placed `some-key.json` is the key `some-key.json`, and so is an `INPUT.json` in a store other than the default one. Dotfiles are skipped.
+
+A `.json` file is adopted as `application/json; charset=utf-8` and anything else as `application/octet-stream`; there is no content sniffing. Adopted records are subject to the purge of the default store on start like any other record — only the run-input keys are spared.
+
+Beyond the literal keys, three v3 behaviors are gone:
+
+- **`INPUT.txt` and `INPUT.bin` are no longer the input.** v3 probed those extensions too. In v4 they are adopted under their own names, so `getValue('INPUT')` returns `undefined` and the purge on start deletes them. Rename such an input to `INPUT.json` (or drop the extension) before upgrading.
+- **Extensionless input files report `application/octet-stream`.** In v3 a bare value file with no extension was read as `text/plain`. Give the file a `.json` extension if you need a more specific type.
+- **Malformed input files are no longer silently swallowed.** In v3 an `INPUT.json` containing invalid JSON was treated as a missing record (`getValue` returned `undefined`). In v4 the raw bytes are returned verbatim and parsing happens in the `KeyValueStore` frontend, so a malformed value surfaces a parse error at read time instead of looking absent.
 
 ## Only if you tuned autoscaling
 

packages/fs-storage/package.json

@@ -42,7 +42,7 @@
         "access": "public"
     },
     "dependencies": {
-        "@crawlee/fs-storage-native": ">=0.2.1-beta.0 <0.3",
+        "@crawlee/fs-storage-native": ">=0.2.1 <0.3",
         "@crawlee/types": "workspace:*",
         "@crawlee/utils": "workspace:*",
         "zod": "catalog:"

packages/fs-storage/src/file-system-storage.ts

@@ -6,6 +6,7 @@ import type { CrawleeLogger } from '@crawlee/types';
 import { parseArgument, schemas } from '@crawlee/utils/internal';
 import { z } from 'zod';
 
+import type { AdoptionCandidate } from '@crawlee/fs-storage-native';
 import { DatasetBackend } from './resource-clients/dataset.js';
 import { KeyValueStoreBackend } from './resource-clients/key-value-store.js';
 import { RequestQueueBackend } from './resource-clients/request-queue.js';
@@ -33,6 +34,17 @@ const DEFAULT_STORAGE_ALIAS = '__default__';
 /** The directory the default storage lives in, one level below `datasets` / `key_value_stores` / etc. */
 const DEFAULT_STORAGE_DIRECTORY = 'default';
 
+/** The conventional run-input key, always treated as one alongside the configured `inputKey`. */
+const DEFAULT_INPUT_KEY = 'INPUT';
+
+/**
+ * Content types declared for adopted files. The native client infers nothing from extensions, and
+ * neither do we beyond this: a `.json` file is JSON, anything else is bytes for the
+ * {@apilink KeyValueStore} frontend to make sense of.
+ */
+const ADOPTED_JSON_CONTENT_TYPE = 'application/json; charset=utf-8';
+const ADOPTED_BINARY_CONTENT_TYPE = 'application/octet-stream';
+
 export interface FileSystemStorageOptions {
     /**
      * Path to directory where the data will be saved.
@@ -64,11 +76,11 @@ export interface FileSystemStorageOptions {
     /**
      * The key the run input is read from — Crawlee's `inputKey` (`CRAWLEE_INPUT_KEY`).
      *
-     * Like the conventional `INPUT`, this key may live in the default key-value store as a bare value
+     * Like the conventional `INPUT`, this key may arrive in the default key-value store as a bare value
      * file with no metadata sidecar (e.g. the Apify CLI writes the effective input to `__CLI_INPUT.json`
-     * and points the run at that key). It is therefore readable out-of-band (`<key>`, `<key>.json`) and
-     * preserved when the default store is purged, exactly like `INPUT`,
-     * which is always kept regardless of this setting.
+     * and points the run at that key). Such a file is adopted into a record under this key when the
+     * store is opened, and the key is preserved when the default store is purged — exactly like
+     * `INPUT`, which is always treated as an input key regardless of this setting.
      *
      * @default 'INPUT'
      */
@@ -90,7 +102,8 @@ export class FileSystemStorageBackend implements storage.StorageBackend {
     readonly requestQueuesDirectory: string;
     readonly logger?: CrawleeLogger;
     readonly requestQueueAccess: 'single' | 'shared';
-    readonly #inputKey: string;
+    /** `INPUT` plus the configured `inputKey`, deduplicated. */
+    readonly #inputKeys: string[];
 
     readonly #keyValueStoreBackendCache: KeyValueStoreBackend[] = [];
     readonly #datasetBackendCache: DatasetBackend[] = [];
@@ -104,7 +117,7 @@ export class FileSystemStorageBackend implements storage.StorageBackend {
 
         this.logger = logger;
         this.requestQueueAccess = requestQueueAccess;
-        this.#inputKey = inputKey;
+        this.#inputKeys = [...new Set([DEFAULT_INPUT_KEY, inputKey])];
 
         this.localDataDirectory = localDataDirectory;
         this.datasetsDirectory = resolve(this.localDataDirectory, 'datasets');
@@ -182,13 +195,21 @@ export class FileSystemStorageBackend implements storage.StorageBackend {
 
         const nativeBackend = await (
             await importNativeModule()
-        ).FileSystemKeyValueStoreClient.open(id, name, alias, this.localDataDirectory);
+        ).FileSystemKeyValueStoreClient.open(
+            id,
+            name,
+            alias,
+            this.localDataDirectory,
+            // useTestClock — always real wall-clock outside of native tests.
+            undefined,
+            this.#adoptionCandidates(cacheKey === DEFAULT_STORAGE_DIRECTORY),
+        );
         const newStore = await KeyValueStoreBackend.create({
             name: alias ? undefined : (name ?? cacheKey),
             cacheKey,
             nativeBackend,
             logger: this.logger,
-            inputKey: this.#inputKey,
+            inputKeys: this.#inputKeys,
         });
         this.#keyValueStoreBackendCache.push(newStore);
 
@@ -230,6 +251,40 @@ export class FileSystemStorageBackend implements storage.StorageBackend {
         return newStore;
     }
 
+    /**
+     * What the native `open` may turn into records: value files sitting in the store directory with no
+     * metadata sidecar, written out-of-band by a CLI, a project template, a v3 Crawlee or a text editor.
+     *
+     * Each run-input key claims a bare `<key>` or `<key>.json` first — that is the layout the Apify CLI
+     * and the templates produce, and the key must end up being `INPUT` rather than `INPUT.json`. Only
+     * the default store holds the run input, so elsewhere an `INPUT.json` is just a file named
+     * `INPUT.json`, adopted by the trailing sweep like every other one.
+     *
+     * Once adopted, the file is an ordinary record: readable, listed, deletable, and — in a run-scoped
+     * store — purged on start unless its key is a run-input key.
+     */
+    #adoptionCandidates(isDefaultStore: boolean): AdoptionCandidate[] {
+        const inputCandidates: AdoptionCandidate[] = isDefaultStore
+            ? this.#inputKeys.map((key) => ({
+                  key,
+                  files: [
+                      { filename: key, contentType: ADOPTED_BINARY_CONTENT_TYPE },
+                      { filename: `${key}.json`, contentType: ADOPTED_JSON_CONTENT_TYPE },
+                  ],
+              }))
+            : [];
+
+        return [
+            ...inputCandidates,
+            {
+                files: [
+                    { filename: '*.json', contentType: ADOPTED_JSON_CONTENT_TYPE },
+                    { filename: '*', contentType: ADOPTED_BINARY_CONTENT_TYPE },
+                ],
+            },
+        ];
+    }
+
     async storageExists(id: string, type: 'Dataset' | 'KeyValueStore' | 'RequestQueue'): Promise<boolean> {
         let backends: (KeyValueStoreBackend | DatasetBackend | RequestQueueBackend)[];
         let baseDir: string;