Skip to content

fix: persist preview and preview_rows fields after datastore push#270

Closed
Copilot wants to merge 2 commits into
mainfrom
copilot/fix-preview-fields-dataset
Closed

fix: persist preview and preview_rows fields after datastore push#270
Copilot wants to merge 2 commits into
mainfrom
copilot/fix-preview-fields-dataset

Conversation

Copy link
Copy Markdown

Copilot AI commented Mar 10, 2026

preview and preview_rows resource fields were being set via resource_update before datastore_create (with calculate_record_count=True), which internally modifies the resource and overwrites our fields.

Changes

  • Reorder operations in both MetadataStage.process() and jobs_legacy.py: call send_resource_to_datastore() first, then _update_resource_metadata() / update_resource()
  • Re-fetch resource via dsu.get_resource() before setting our fields, so we operate on the latest state post-datastore_create
# Before (fields get clobbered):
self._update_resource_metadata(context)      # sets preview, preview_rows
dsu.send_resource_to_datastore(...)           # datastore_create overwrites resource

# After (fields persist):
dsu.send_resource_to_datastore(...)           # datastore_create does its thing
self._update_resource_metadata(context)       # re-fetches resource, then sets fields

…preview fields

The preview and preview_rows fields were being set on the resource via
resource_update, but then immediately overwritten when datastore_create
(called via send_resource_to_datastore with calculate_record_count=True)
internally modified the resource metadata.

Fix: Move _update_resource_metadata() call to AFTER send_resource_to_datastore()
and re-fetch the resource to get the latest state. This ensures our preview
fields are set last and persist correctly.

Applied to both the new pipeline (metadata.py) and legacy code (jobs_legacy.py).

Co-authored-by: minhajuddin2510 <68331751+minhajuddin2510@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix missing preview fields in dataset after push fix: persist preview and preview_rows fields after datastore push Mar 10, 2026
@minhajuddin2510 minhajuddin2510 marked this pull request as ready for review March 10, 2026 12:34
jqnatividad added a commit that referenced this pull request May 16, 2026
fix(metadata): persist preview/preview_rows after datastore_create (rebases #270)
@jqnatividad
Copy link
Copy Markdown
Collaborator

Superseded by #292 (commit 20e835b). The Prefect 3 migration deleted jobs_legacy.py (the file this PR also patched), which is why this PR couldn't be cleanly rebased from the fork. #292 cherry-picks the still-applicable metadata.py half and credits the original work in the commit body. Thanks @copilot-swe-agent!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants