diff --git a/src/content/docs/authoring/execution.mdx b/src/content/docs/authoring/execution.mdx index f16b4cd..bf9385b 100644 --- a/src/content/docs/authoring/execution.mdx +++ b/src/content/docs/authoring/execution.mdx @@ -48,7 +48,8 @@ A requirement check (see [Requirements](/authoring/manifest-reference/#requireme Every job gets a private **work directory** (`$FG_WORK_DIR`), and a checkout of the project at the app's **pinned commit** is symlinked into it as `repo`. The `working_dir` runnable field controls which of these the command runs from: -- **`repo`** (default for most runnables) — the script `cd`s into `$FG_WORK_DIR/repo` (or the manifest's subdirectory within it). Use this when the command relies on the project being the current directory — for example Pixi tasks, or scripts that reference sibling files with relative paths. +- **`manifest`** (default for most runnables) — the script `cd`s into the manifest's directory within the checkout: `$FG_WORK_DIR/repo` when the manifest is at the repo root, or its subdirectory when the manifest lives deeper (e.g. a monorepo). Use this when the command relies on being run from beside the manifest — for example Pixi tasks, or scripts that reference sibling files with relative paths. +- **`repo`** — the script `cd`s into the checkout root (`$FG_WORK_DIR/repo`) regardless of where the manifest lives. Use this for a manifest in a subdirectory whose command still needs to run from the top of the project. - **`work`** (default for **container** runnables) — the script `cd`s into `$FG_WORK_DIR`. The project is still reachable via the `repo` symlink (e.g. `nextflow run repo`). Use this for tools that should write their outputs into a clean per-job directory rather than the project checkout (which is shared by every job running the same version of the app). The checkout is per-version: jobs of an app that hasn't been updated share one checkout (so e.g. a Pixi environment builds once and is reused), and updating the app gives subsequent jobs a fresh checkout of the new commit while running jobs keep the one they started with. @@ -61,7 +62,7 @@ runnables: working_dir: work ``` -Containers default to `work` because the project checkout is **not** bind-mounted into the container (only the work directory and your file/directory parameters are), so a container's current directory should be the work dir. A container runnable that genuinely needs the project can set `working_dir: repo` — Fileglancer then bind-mounts the checkout into the container automatically so the command can run from it and read the project's files. +Containers default to `work` because the project checkout is **not** bind-mounted into the container (only the work directory and your file/directory parameters are), so a container's current directory should be the work dir. A container runnable that genuinely needs the project can set `working_dir: manifest` or `working_dir: repo` — Fileglancer then bind-mounts the checkout into the container automatically so the command can run from it and read the project's files. ## Environment Variables @@ -202,7 +203,7 @@ apptainer exec --bind /home/user/.fileglancer/jobs/1-lolcow-say "$SIF_PATH" \ 'Hello from Fileglancer!' ``` -**Bind mounts** are auto-detected from `file` and `directory` parameters — the effective values the command is built from, so manifest defaults and parameters declared in `env_parameters` are bound too, not just values the user typed. The job's working directory is always bound, and when `working_dir: repo` is set, the cached repo clone is bound as well. Use `bind_paths` to add extra paths: +**Bind mounts** are auto-detected from `file` and `directory` parameters — the effective values the command is built from, so manifest defaults and parameters declared in `env_parameters` are bound too, not just values the user typed. The job's working directory is always bound, and when `working_dir` is `manifest` or `repo`, the cached repo clone is bound as well. Use `bind_paths` to add extra paths: ```yaml container: ghcr.io/org/image:tag diff --git a/src/content/docs/authoring/manifest-reference.mdx b/src/content/docs/authoring/manifest-reference.mdx index 35cf7b4..148011b 100644 --- a/src/content/docs/authoring/manifest-reference.mdx +++ b/src/content/docs/authoring/manifest-reference.mdx @@ -13,7 +13,7 @@ The `runnables.yaml` manifest describes your app. This page documents its struct |-------|------|----------|-------------| | `name` | string | yes | Display name shown in the Fileglancer UI | | `description` | string | no | Short description of the app | -| `repo_url` | string | no | GitHub URL of a separate repository containing the tool code (see [Separate Tool Repo](/authoring/overview/#separate-tool-repo)). Must be a valid GitHub repository URL — the manifest is rejected at load time otherwise | +| `repo_url` | string | no | GitHub URL of a separate repository containing the tool code (see [Separate Tool Repo](/authoring/overview/#separate-tool-repo)). Only takes effect when it names a *different* repository (`owner/repo`); pointing it at the manifest's own repo is ignored, and the job runs from the manifest's branch and pinned commit. Must be a valid GitHub repository URL — the manifest is rejected at load time otherwise | | `requirements` | list of strings | no | Tools that must be available in the job environment (see [Requirements](#requirements)) | | `runnables` | list of objects | yes | One or more runnable definitions (see [Runnables](#runnables)). A manifest with an empty `runnables` list is rejected | @@ -38,7 +38,7 @@ Each runnable defines a single command that users can launch. If a manifest has | `container` | string | no | Container image URL for Apptainer (see [Containers](/authoring/execution/#containers-apptainer)) | | `bind_paths` | list of strings | no | Additional paths to bind-mount into the container (requires `container`) | | `container_args` | string | no | Default extra arguments for container exec (e.g. `--nv`), overridable at launch time | -| `working_dir` | string | no | Where the command runs: `repo` (the cloned project, optionally the manifest's subdirectory) or `work` (the job's work directory). Defaults to `work` for container runnables and `repo` otherwise (see [Execution Environment](/authoring/execution/#working-directory)) | +| `working_dir` | string | no | Where the command runs: `manifest` (the manifest's directory inside the cloned project), `repo` (the cloned project's root), or `work` (the job's work directory). Defaults to `work` for container runnables and `manifest` otherwise (see [Execution Environment](/authoring/execution/#working-directory)) | | `auto_url` | boolean | no | Service runnables only. When `true`, Fileglancer publishes the service URL for you once `$FG_SERVICE_PORT` is accepting connections — bind your service to `$FG_SERVICE_PORT` and you need no URL-writing code (see [Services](/authoring/services/)) | | `service_url_suffix` | string | no | Service runnables with `auto_url`. Text appended to `http://$FG_HOSTNAME:$FG_SERVICE_PORT` when publishing — a path and/or query, e.g. `/?access_token=${FG_SERVICE_TOKEN}` for one-click auth. May contain literal URL text and the placeholders `${FG_SERVICE_TOKEN}`, `${FG_SERVICE_PORT}`, `${FG_HOSTNAME}` (braces required); nothing else (see [Services](/authoring/services/)) | | `requirements` | list of strings | no | Additional tool requirements specific to this runnable, merged with manifest-level requirements | diff --git a/src/content/docs/authoring/overview.mdx b/src/content/docs/authoring/overview.mdx index 08a17ec..8468be7 100644 --- a/src/content/docs/authoring/overview.mdx +++ b/src/content/docs/authoring/overview.mdx @@ -130,6 +130,8 @@ When `repo_url` is set: - The discovery repo (containing `runnables.yaml`) is used only for manifest metadata. - The tool repo (`repo_url`) is cloned separately and used as the working directory for the job. +`repo_url` only triggers this separate-repo behavior when it points to a *different* repository (a different `owner/repo`); the branch is ignored in that comparison. If `repo_url` names the **same** repository the manifest lives in, it is not treated as separate — the job runs from the manifest's own branch and pinned commit. So a manifest on a non-default branch does not need `repo_url` pointing back at itself; doing so with a bare URL (no `/tree/`) would otherwise imply the repo's default branch. Omit `repo_url` unless the code genuinely lives in another repository. + Repos are cloned into a per-user cache, and every app is **pinned to an exact commit** when it is added. Jobs run from an immutable per-commit checkout of the pinned code — they never pull automatically, and nothing can change a job's code while it runs. To get the newest code, use the **Update** action for the app in the UI, which pulls the latest commits on the app's revision, re-reads the manifest, and re-pins the app (both the manifest repo and the `repo_url` tool repo, when one is declared). Updates are per-app: other apps sharing the same repository keep their own pinned commits. ## Where to Go Next diff --git a/src/content/docs/authoring/parameters.mdx b/src/content/docs/authoring/parameters.mdx index 02665d5..3577a18 100644 --- a/src/content/docs/authoring/parameters.mdx +++ b/src/content/docs/authoring/parameters.mdx @@ -27,7 +27,8 @@ Each parameter with a `flag` field becomes a CLI flag appended to the base comma | `raw` | boolean | no | If `true`, the value is appended to the command without shell quoting. Validated against shell metacharacters for safety. Default: `false` | | `value_separator` | string | no | How a flagged parameter is joined to its value: `"space"` emits `--flag 'value'`, `"equals"` emits `--flag='value'`. Use `equals` for tools (like Nextflow) that would otherwise misparse a value starting with `-` as another flag. Default: `"space"` | | `boolean_style` | string | no | Only valid on `boolean` parameters. `"flag"` emits the bare flag when true and omits it when false; `"value"` emits an explicit `--flag true` / `--flag false` (or `--flag=true` / `--flag=false` with `value_separator: equals`). Use `value` for tools where omitting a false switch would leave a default of true in effect. Default: `"flag"` | -| `exists` | boolean | no | Only valid on `file` and `directory` parameters. When `true` (the default), the path must exist and be readable before launch. Set `false` for outputs the job creates: the existence check is skipped, and `directory` parameters are created (as the user, within an allowed file share) before launch, so a home default like `~/.fileglancer/logs` works on first launch. Default: `true` | +| `exists` | boolean | no | Only valid on `file`, `directory`, and `path` parameters. When `true` (the default), the path must exist and be readable before launch. Set `false` for outputs the job creates: the existence check is skipped, and `directory` parameters are created (as the user, within an allowed file share) before launch, so a home default like `~/.fileglancer/logs` works on first launch. Default: `true` | +| `associations` | object | no | Only valid on `file`, `directory`, and `path` parameters. Declares which file types this parameter accepts, so the file browser can offer the app when the user navigates to a matching file or directory. See [File Associations](#file-associations) | ## Parameter Types @@ -39,17 +40,64 @@ Each parameter with a `flag` field becomes a CLI flag appended to the base comma | `boolean` | Checkbox | `--flag` (if true, omitted if false); `--flag true` / `--flag false` with `boolean_style: value` | N/A | Must be true/false | | `file` | Text input + file browser | `--flag '/path/to/file'` | `'/path/to/file'` | Must be an absolute path that exists, is readable, and points to a file on the server (unless `exists: false`) | | `directory` | Text input + directory browser | `--flag '/path/to/dir'` | `'/path/to/dir'` | Must be an absolute path that exists, is readable, and points to a directory on the server (unless `exists: false`) | +| `path` | Text input + file/directory browser | `--flag '/path/to/item'` | `'/path/to/item'` | Must be an absolute path that exists and is readable on the server (unless `exists: false`). Accepts either a file or a directory — no type check. Useful for inputs that take both, e.g. TIFF files and OME-Zarr directories | | `enum` | Dropdown select | `--flag 'chosen_value'` | `'chosen_value'` | Value must be one of the `options` list | -**Notes on `file` and `directory` types:** +**Notes on `file`, `directory`, and `path` types:** - The UI provides a browse button alongside the text input, so users can pick paths from the Fileglancer file browser instead of typing them. -- Paths are validated server-side before job submission (must exist, be accessible, and match the expected file/folder type, unless `exists: false`). +- Paths are validated server-side before job submission (must exist, be accessible, and match the expected file/folder type, unless `exists: false`; `path` parameters accept either kind). - Both absolute paths (`/data/images`) and home-relative paths (`~/output`) are accepted. - Shell metacharacters (`;`, `&`, `|`, `` ` ``, `$`, `(`, `)`, etc.) are rejected for safety. -- Set `exists: false` on paths the job creates, such as an output or logs directory. The path is still validated for file-share containment, but not for existence. If the path already exists, Fileglancer still checks that it matches the expected file/folder type. Fileglancer additionally creates `exists: false` directories (as the user, within an allowed file share) just before launch, so a default like `~/.fileglancer/logs` works out of the box and overrides that point at a new directory are created too. Overrides pointing at an existing directory are a no-op. This never creates anything outside a file share. `file` parameters with `exists: false` skip the existence check but nothing is created for them. +- Set `exists: false` on paths the job creates, such as an output or logs directory. The path is still validated for file-share containment, but not for existence. If the path already exists, Fileglancer still checks that it matches the expected file/folder type. Fileglancer additionally creates `exists: false` directories (as the user, within an allowed file share) just before launch, so a default like `~/.fileglancer/logs` works out of the box and overrides that point at a new directory are created too. Overrides pointing at an existing directory are a no-op. This never creates anything outside a file share. `file` and `path` parameters with `exists: false` skip the existence check but nothing is created for them (a `path` parameter's kind is ambiguous, so Fileglancer cannot know whether to create a directory). - Apps auto-detected from a Nextflow pipeline's `nextflow_schema.json` follow the same convention as [nf-schema](https://nextflow-io.github.io/nf-schema/): a path parameter only requires existence when the schema property sets `"exists": true`. +## File Associations + +`file`, `directory`, and `path` parameters can declare `associations`: the file types they accept. When a user navigates to a matching file or directory in the Fileglancer file browser, a **Compatible apps** panel appears above the listing. Clicking the app opens its launch form with the browsed path already filled into the associated parameter. + +| Field | Type | Required | Description | +|-------|------|----------|-------------| +| `extensions` | list of strings | no | Name suffixes to match, e.g. `[.tif, .czi]`. Case-insensitive; a missing leading dot is added automatically. Multi-part suffixes match against the end of the name, so `.ome.zarr` works | +| `capabilities_manifest` | string | no | Repo-relative path to a [capability manifest](https://github.com/BioImageTools/capability-manifest) YAML file describing the OME-Zarr data the app supports — the same standard Fileglancer uses to pick compatible OME-Zarr web viewers. Accepts either a full manifest (`viewer:` + `capabilities:` sections) or a bare capabilities mapping. Resolved when the app manifest is loaded, so the pinned app snapshot carries its capabilities with it. Only valid on `directory` and `path` parameters | +| `label` | string | no | Display label shown in the file browser panel, e.g. `TIFF images`. Defaults to the parameter's `name` | + +At least one of `extensions` or `capabilities_manifest` must be declared. Capability manifests are a separate standard and always live in their own file — they cannot be inlined in `runnables.yaml`. + +### Matching Rules + +- A `file` parameter only matches files, a `directory` parameter only matches directories, and a `path` parameter matches both. +- For a Zarr dataset (detected by content, not by name), declared capabilities are **authoritative**: the app is offered only when the compatibility check passes, even if `.zarr` is also listed in `extensions`. Plain (non-OME) Zarr never matches a capabilities rule, since compatibility cannot be verified. +- Everything else matches by case-insensitive name suffix. + +This lets one input register many file types with different handling. For example, an image converter that accepts TIFF and CZI files by extension, plus the OME-Zarr datasets it can actually read: + +```yaml +# runnables.yaml +parameters: + - flag: --input + name: Input image + type: path + required: true + associations: + extensions: [.tiff, .czi, .zarr] + capabilities_manifest: capabilities.yaml + label: Convertible images +``` + +```yaml +# capabilities.yaml (next to runnables.yaml) +capabilities: + ome_zarr_versions: [0.4, 0.5] + compression_codecs: [blosc, zstd, gzip] +``` + +This app appears for every `.tiff` and `.czi` file. For Zarr datasets, the capability manifest decides: an OME-Zarr 0.4 dataset compressed with blosc matches, while an unsupported version or codec hides the app — even though `.zarr` is in the extensions list. + + + ## Flag Forms Parameters support three flag styles: diff --git a/src/content/docs/workflows/apps-and-jobs.mdx b/src/content/docs/workflows/apps-and-jobs.mdx index 135d44d..a268e8b 100644 --- a/src/content/docs/workflows/apps-and-jobs.mdx +++ b/src/content/docs/workflows/apps-and-jobs.mdx @@ -117,7 +117,7 @@ Once an app is added, you can launch it as a job on the compute cluster. 3. **Fill in parameters on the launch form**
- The form shows the app's parameters with their types, defaults, and descriptions. For `file` and `directory` parameters, a browse button lets you pick paths from the Fileglancer file browser. Required parameters are marked and validated before submission. + The form shows the app's parameters with their types, defaults, and descriptions. For `file`, `directory`, and `path` parameters, a browse button lets you pick paths from the Fileglancer file browser. Required parameters are marked and validated before submission. 4. **Review the cluster and environment settings (optional)**
Switch to the **Cluster** tab to adjust CPU, memory, walltime, and queue requests, along with any extra scheduler arguments. The **Environment** tab lets you set environment variables, pre/post-run scripts, and tool-specific options like Nextflow profiles. @@ -127,6 +127,10 @@ Once an app is added, you can launch it as a job on the compute cluster. +### Launching from the File Browser + +Apps can register **file associations** — the file types they handle, declared in their manifest (see [File Associations](/authoring/parameters/#file-associations)). When you navigate to a matching file or directory in the file browser, a **Compatible apps** panel appears above the listing, next to any OME-Zarr viewer links. Clicking an app opens its launch form with the browsed path already filled into the matching parameter; review the remaining settings and submit as usual. For Zarr datasets, apps that declare OME-Zarr capabilities are shown only when the dataset is actually compatible, the same way viewer links are filtered. + ### Tips for Launching - **Running the latest code**: Each launch runs the exact commit the app is pinned to, so repeated runs are reproducible. If the app's code is actively developed, watch for the amber **Update available** tag on the app's card and use the **Update** button to move to the newest commit before launching (see [Updating an App](#updating-an-app)). @@ -166,7 +170,7 @@ The Jobs page displays a table with the following columns: Click an entry point name in the jobs table (or select **View Details** from the actions menu) to open the detail view. This page shows: - **Header** — app name, entry point, and status badge, with a back arrow that returns to the jobs list -- **Overview tab** — a summary of the job: its app (linked to the app's detail page), entry point, the exact commit the job executed (**Version**, linked to GitHub), status, timestamps (submitted, started, finished, exit code), and resource requests +- **Overview tab** — a summary of the job: its app (linked to the app's detail page), entry point, the exact commit the job executed (**Commit**, linked to GitHub), status, timestamps (submitted, started, finished, exit code), and resource requests - **Parameters tab** — the parameter values that were used for this job - **Script tab** — the generated shell script that was submitted to the cluster, with a link to the script file on disk - **Output Log tab** — the job's stdout, streamed in near-real-time while the job is running. Includes a download button