[FSTORE-2075] Document the tag lifecycle, job and dataset tags, and keywords - #629
Open
jimdowling wants to merge 7 commits into
Open
[FSTORE-2075] Document the tag lifecycle, job and dataset tags, and keywords#629jimdowling wants to merge 7 commits into
jimdowling wants to merge 7 commits into
Conversation
…eywords https://hopsworks.atlassian.net/browse/FSTORE-2075 Tags now cover jobs and datasets alongside the feature store artifacts, tag schemas can be deprecated and deleted rather than living for ever, every attachment records when it was made, and the keyword API is a documented surface rather than three CLI commands. The pages here were written for the five artifact types that existed before and told readers that a schema is immutable, which is no longer true. The tags guide gains the two new artifact types, the attachment time and the methods that expose it, and the freeze on tags attached to files inside a dataset. Its immutability warning becomes a statement about the JSON definition, which is still fixed, and points at the new lifecycle page for what an administrator can do instead. Two new user guides cover the schema lifecycle (deprecate, restore, usage, delete, force delete, and why a plain delete against an older backend is refused by the client) and keywords. The mandatory tags guide gains datasets, which are enforced at creation and exempt the datasets Hopsworks provisions for a project. The search concept page lists the four asset types that became searchable. A new administration page covers dataset tag storage: the write window the upgrade takes for the migration, the activation gate that holds the two cluster-wide operations until every node runs the new code, the admission policy that fences a rollback, and the cut-over itself, which pauses dataset tag writes while the API restarts and leaves reads untouched. Signed-off-by: Jim Dowling <jim@hopsworks.ai> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
https://hopsworks.atlassian.net/browse/FSTORE-2075 The page was written from the specification before the Job existed, and the implementation settled three details differently. The Job reads the cut-over status before anything else, so a database that is already canonical exits immediately and a window an earlier attempt left open is resumed rather than opened twice; both matter to an operator deciding whether re-running is safe, so they are now step 1. It scales both API deployments, not one. And accepting an unfenced rollback is the tagLifecycle.cutover.acceptUnfencedRollback value, not a command line flag. Also a note that Helm 4 needs --force-conflicts on the upgrade: server side apply refuses fields owned by other field managers, which a running cluster always has, and the failure lands before any hook runs. Pre-existing Helm 4 behaviour, but the operator following this page is the one who hits it. Signed-off-by: Jim Dowling <jim@hopsworks.ai> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR expands the Hopsworks docs to cover the new tag lifecycle functionality, extend tagging/search to jobs and datasets, and document keywords and dataset-tag storage behavior as part of FSTORE-2075.
Changes:
- Extend the Tags guide to include jobs/datasets, tag attachment timestamps, and updated search semantics.
- Add new guides for Tag Schema Lifecycle and Keywords.
- Add an admin guide for the dataset-tag storage migration/cut-over, and update search/mandatory-tag docs accordingly.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| mkdocs.yml | Adds nav entries for the new tag/keyword/lifecycle/admin documentation pages. |
| docs/user_guides/fs/tags/tags.md | Updates tagging docs for jobs/datasets, tag timestamp metadata APIs, and expanded search coverage. |
| docs/user_guides/fs/tags/tag_schema_lifecycle.md | New page documenting admin lifecycle operations: deprecate/restore/usage/delete/force delete. |
| docs/user_guides/fs/tags/keywords.md | New page documenting keyword APIs, metadata timestamps, CLI commands, and concurrency caveats. |
| docs/user_guides/fs/tags/mandatory_tags.md | Adds datasets as a supported mandatory-tag artifact type and documents dataset-specific behavior. |
| docs/setup_installation/admin/dataset_tag_storage.md | New admin guide describing the dataset tag storage migration, activation gate, rollback fencing, and cut-over steps. |
| docs/concepts/projects/search.md | Updates searchable asset list to include models/deployments/jobs/datasets and clarifies keyword applicability. |
Suppressed comments (2)
docs/user_guides/fs/tags/tag_schema_lifecycle.md:30
- Internal cross-references should use heading IDs (not relative .md paths) to avoid breaking under mike versioning; see .claude/docs/content.md:21-27.
A schema that is currently registered as a [mandatory tag](mandatory_tags.md) cannot be deprecated.
docs/user_guides/fs/tags/keywords.md:94
- Internal cross-references should use heading IDs (not relative .md paths) to avoid breaking under mike versioning; see .claude/docs/content.md:21-27.
Attach a tag with the tag commands, described in the [Tags](tags.md) guide.
| !!! warning "Schema definitions cannot be edited" | ||
| The JSON schema of a tag schema cannot be changed after it is created, because the values already attached were validated against the original definition. | ||
| A schema can be deprecated so that it accepts no new attachments, and it can be deleted once nothing references it. | ||
| Both are administrator actions and are described in the [Tag schema lifecycle](tag_schema_lifecycle.md) guide. |
|
|
||
| ## Introduction | ||
|
|
||
| A tag schema is defined once and then attached to artifacts across every project on the cluster, as described in the [Tags](tags.md) guide. |
| A tag schema can be mandatory for any combination of feature groups, feature views, training datasets, models, deployments and datasets. | ||
| For example, a `data_owner` schema can be marked mandatory for models and deployments only, leaving the others unaffected. | ||
|
|
||
| A [deprecated](tag_schema_lifecycle.md) schema cannot be registered as mandatory, because nobody is allowed to attach it. |
| ## Introduction | ||
|
|
||
| A keyword is a single user-defined word attached to a feature group, feature view or training dataset. | ||
| Keywords are free text with no schema behind them, which makes them the lighter alternative to [tags](tags.md): use a keyword to make an artifact easier to find, and a tag when the metadata has to be structured and validated. |
jimdowling
marked this pull request as ready for review
August 7, 2026 16:08
https://hopsworks.atlassian.net/browse/FSTORE-2075 Every helm setting on the dataset tag storage page was written without its subchart prefix, so an operator following the page could not run any of it. tagLifecycle belongs to the hopsworks subchart, and the umbrella chart validates user-supplied values against its own schema first: helm upgrade --set tagLifecycle.cutover.run=true fails with "additional properties 'tagLifecycle' not allowed" before a single template renders. The cluster run that verified the cut-over used hopsworks.tagLifecycle.cutover.run, which is why the gap survived the end-to-end test. All four settings now carry the prefix, and the cut-over section says why it is there so the next person does not read it as a typo. Signed-off-by: Jim Dowling <jim@hopsworks.ai> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
https://hopsworks.atlassian.net/browse/FSTORE-2075 The search concept page states that a cluster-wide search reaches projects you are not a member of, without saying that this is a setting. The migration in this ticket seeds cross_project_global_search_enabled true, and a multi-tenant operator is expected to set it false, which restricts the global endpoint to the caller's accessible projects in-query. An administrator reading the page had no way to learn the variable existed, and a reader of the page was told as unconditional a behaviour their cluster may not have. Signed-off-by: Jim Dowling <jim@hopsworks.ai> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
https://hopsworks.atlassian.net/browse/FSTORE-2075 Review round ten called "reads are never interrupted" what it was: an availability claim the quiesce falsifies, since both API deployments are at zero for the restart and nothing answers while they are down. The page now states the short full outage plainly, and that reads serve normally only once the pods are back. Also added, matching the hardened Job in the chart PR: the recovery step for an attempt killed mid-window, the HorizontalPodAutoscaler refusal with the instruction to pause GitOps reconciliation of the replica counts, the fence re-check at commit with the exact-expression comparison, and a paragraph on what downgrading a FRESH installation without restoring its database costs, since a fresh install is database-canonical from the start and sits past the same one-way boundary without ever running a cut-over. Signed-off-by: Jim Dowling <jim@hopsworks.ai> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…mise https://hopsworks.atlassian.net/browse/FSTORE-2075 Review round eleven asked for the product contract the fresh-install fence decision rests on, in writing: downgrading any database-canonical cluster below this release without restoring or reinstalling its database is unsupported, and a fresh installation is such a cluster from day one. The page now says so plainly, with the admission policy as the opt-in refusal for operators who want more than a rule. The write-window escape also stops being free: disabling it renders a pre-upgrade check that refuses the migration while any API pod runs or an autoscaler targets the API deployments, and the page explains why a load-balancer window alone is not a write window. The failure-step reference in the cut-over runbook also moved from step 6 to steps 7-8, matching the renumbered Job. Signed-off-by: Jim Dowling <jim@hopsworks.ai> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
https://hopsworks.atlassian.net/browse/FSTORE-2075 The cut-over guide told operators the Job refuses while a tag is quarantined without saying what a quarantined tag is, how to clear one, or what waiving costs. It now shows the waive call, and states the part that matters: the waiver applies to the value that was inspected and to no other, so a value that changes before the grant is refused and one that changes afterwards brings the record back for review. An operator who thinks they waived "the tag" would be surprised by that; one who reads this will not be. Reviewed-by: Claude Opus 5 (1M context) Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DLDai4Z5QRoBTJpikoToZ7
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ticket: FSTORE-2075
Documentation for the tag lifecycle work. The code PRs are ee#3191, api#1079, front#2023, helm#2164 and loadtest#967.
What changed
user_guides/fs/tags/tags.md— jobs and datasets are now taggable artifacts, with a Python example for each. The attachment time of a tag is exposed throughget_tag_metadata/get_tags_metadata, described in its own section. Tags attached to a file inside a dataset are frozen for writes, which gets a warning box. The "Immutable" warning becomes a statement about the JSON definition, which is still fixed, and points at the new lifecycle page. The search section lists the artifact types that are indexed and links jobs and datasets to the Python search API.user_guides/fs/tags/tag_schema_lifecycle.md(new) — deprecate, restore, what the usage report counts and whycompleteis always false, delete, force delete and its threshold, and why a plain delete against an older backend is refused by the client rather than sent.user_guides/fs/tags/keywords.md(new) — the keyword API on feature groups, feature views and training datasets, the attachment times, the cluster vocabulary, the CLI commands, and the note thatadd_keywordsis read-modify-write.user_guides/fs/tags/mandatory_tags.md— datasets as a sixth artifact type: enforced at creation, exempting the datasets Hopsworks provisions for a project, withmissingMandatoryTagson read for datasets that predate the policy.concepts/projects/search.md— models, deployments, jobs and datasets added to the searchable asset list, with the note that keywords do not apply to jobs and datasets.setup_installation/admin/dataset_tag_storage.md(new) — the administrator's view of moving dataset tags from HopsFS extended attributes into the database: the write window the upgrade takes for the migration and its two failure paths, the activation gate, the admission policy that fences a rollback and why it is off by default, and the cut-over, which pauses dataset tag writes for about five minutes while the API restarts and never interrupts reads.Verification
mkdocs build -sgreen, built against theFSTORE-2075-tag-lifecycle-timestampsbranch ofhopsworks-api.markdownlint-cli2green on the changed files.hopsworks-docs snakeoilrun overdocs/; no reformatting left outstanding on the changed files.The strict build initially failed with sixteen unresolved cross-references to
hopsworks.tag.Tag.created_on, coming from docstrings in thehopsworks-apiPR. Fixed there (fc108e88d), since a property gets no alias anchor under the aliased module.Not covered
The cut-over page names
tagLifecycle.cutover.run, which lands with the chart PR. Hold this PR until that value is merged, or the page names a value that does not exist yet.🤖 Generated with Claude Code