Optional per-dataset metadata file passed to opt-in pipelines#1764
Open
mattdawkins wants to merge 4 commits into
Open
Optional per-dataset metadata file passed to opt-in pipelines#1764mattdawkins wants to merge 4 commits into
mattdawkins wants to merge 4 commits into
Conversation
…elines Add an optional metadata file (.json/.txt/.csv) to single-image and multi-camera image-sequence imports, mirroring how calibration files are optionally attached to stereo datasets — but without the stereo gating. DIVE stores a link to the file per dataset and hands it to pipelines that opt in via a new `# Metadata File: <block>:<key>` pipe header. Built for the sea-lion add-on, whose registration pipelines read it as `-s stabilizer:flight_log=<path>`. Pipe header / contract: - Parse `# Metadata File: <kwiver-key>` in both pipe-metadata readers (server pipeline_discovery.extract_pipe_metadata, desktop extractPipeMetadata) into PipeMetadata.metadataFileKey. Pipes that do not declare it receive nothing, so other pipelines are unaffected. Desktop: - Import dialog "Metadata File (Optional)" picker (single) and a non-stereo-gated ImportMultiCamMetadata picker (multicam). - Store the file in the project dir at import (JsonMeta.metadataFile / metadataOriginalName) and inject `-s <key>="<path>"` in viame.ts for pipes that opt in. Web + server: - Upload the file as a marked Girder item and record a folder-level metadataFileItemId marker (create_multicam metadataFileId for multicam; a new POST dive_dataset/:id/metadata_file endpoint for single). - crud_rpc.run_pipeline resolves the item for opt-in pipes; the run task downloads it and appends the `-s <key>=<path>` override for single and multicam alike. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sea-lion registration reads the frame list from the stabilizer's own image_list / frame_list params, not just the input reader's video_filename, so DIVE must pass the list in two places. Add two opt-in pipe headers: # Image List Keys: stabilizer:image_list (primary / first-camera manifest) # Frame List Keys: stabilizer:frame_list (comma-joined per-camera manifests) Both pipe-metadata readers parse them into PipeMetadata.imageListKeys / frameListKeys. At run time DIVE binds the same absolute manifest path(s) it feeds input:video_filename to each declared key — for single-image and multicam runs, on desktop (viame.ts) and web/server (tasks.py). Pipes that do not declare the headers are unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…eys) Collapse the two image-list headers into one. `# Image List Keys:` now binds the run's input image list(s) to every listed key with the same value — one list per camera (single-camera: the one manifest; multicam: the per-camera manifests comma-joined). Removes the separate primary/first-camera vs comma-joined distinction and the `# Frame List Keys:` header, so a pipe simply points image_list and frame_list at the same input lists DIVE already writes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Image-list keys are now per-camera pointers to a single line-separated file, not
a comma-separated list. `# Image List Keys:` templates may contain a `{cam}`
placeholder, which DIVE expands to the 1-based camera index — e.g.
`stabilizer:image_list{cam}` becomes `-s stabilizer:image_list1=<cam1 list>
-s stabilizer:image_list2=<cam2 list> ...`. A key without `{cam}` gets the first
camera's list. Single-image runs bind one list (image_list1). Matches the
colmap_registration process, which now takes one image_list<i> per camera.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an optional metadata file (
.json/.txt/.csv) to single-image and multi-camera image-sequence imports, mirroring how calibration files are optionally attached to stereo datasets — but without the stereo gating. DIVE keeps a link to the file per dataset and hands it to pipelines that opt in.Built for the sea-lion add-on, whose registration pipelines consume it as
-s stabilizer:flight_log=<path>.How it works
A pipe opts in with a header directive:
Both pipe-metadata readers (server
pipeline_discovery.extract_pipe_metadataand desktopextractPipeMetadata) parse this intoPipeMetadata.metadataFileKey. At run time DIVE appends-s <key>="<path>"only for pipes that declare the header, so all other pipelines are unaffected.Changes
Contract / parsing
# Metadata File: <block>:<key>→metadataFileKey(server + desktop + shared types); added to the description-block stop conditions.Desktop
ImportMultiCamMetadatapicker in the multicam dialog.JsonMeta.metadataFile/metadataOriginalName);viame.tsinjects-s <key>="<path>"for opt-in pipes (single + multicam).Web + server
metadataFileItemIdmarker (viacreate_multicammetadataFileIdfor multicam, and a newPOST dive_dataset/:id/metadata_fileendpoint for single).crud_rpc.run_pipelineresolves the item for opt-in pipes; the run task downloads it and appends the override for single and multicam alike.Testing
vite build(web) andelectron-vite build(desktop) both succeed.eslintclean ondive-common/andplatform/.py_compileclean; pipe-header parser verified against the sea-lion pipes.File is not definedenv failures are unaffected).🤖 Generated with Claude Code