Skip to content

AIR CLI: implement air run --watch (stream logs until completion) - #6080

Open
riddhibhagwat-db wants to merge 1 commit into
air-clifrom
air-run-watch
Open

AIR CLI: implement air run --watch (stream logs until completion)#6080
riddhibhagwat-db wants to merge 1 commit into
air-clifrom
air-run-watch

Conversation

@riddhibhagwat-db

Copy link
Copy Markdown
Contributor

Changes & Why

After submitting a workload, --watch follows the run's logs to completion and exits with the run's outcome, reusing the same Bricklens-with-MLflow-fallback pipeline as air logs.

  • Text mode: prints "Submitted run", the dashboard link, and "Monitoring run and streaming logs...", then streams the logs.
  • JSON mode: emits a SUBMITTED event with the run id, a STATUS event on each lifecycle transition, the streamed LOG/ALERT events, and a closing terminal-status envelope (SUCCESS/FAILED/CANCELED) — matching the Python CLI's --watch JSONL contract.
  • Without --watch, the plain submit path now prints a tip about --watch.
  • STATUS events are watch-scoped (opt-in via logRequest.onStatusChange), so the merged air logs output is unchanged.

--dry-run still takes precedence over --watch (nothing is submitted or streamed).

Tests

Unit tests (experimental/air/cmd/)

  • logbricklens_test.go: Bricklens client query/path serialization + time_unix_nano parsing
  • logstream_test.go: fallback classification, status projection, --minutes/tail math, page dedup/ordering, retry-then-fallback, JSONL/ALERT emit, Ctrl-C exit
  • logmlflow_test.go: MLflow chunk discovery/listing, attempt-prefix layout, no-logs exit-code parity
  • logs_test.go: air logs command: flag validation, completed-run tail, Bricklens→MLflow fallback, past-retry static view
  • run_watch_test.go: air run --watch: text stream, JSON SUBMITTED→STATUS→LOG→terminal envelope, failed-run exit code, dry-run precedence

Acceptance tests (acceptance/experimental/air/)

  • logs/ : text/JSON streaming, --minutes, --lines, --lines 0, --retry, mutual-exclusion errors, invalid id, negative node, --download-to
  • logs-mlflow-fallback/ : Bricklens FEATURE_DISABLED → MLflow fallback → no-logs (text & JSON)
  • run/ : dry-run, --override, config validation, --watch ignored under --dry-run
  • run-submit/ : real submit payload + --watch tip line
  • help/ : air --help, air logs --help command-tree pins

Manual verification:
Properly monitors and outputs logs from runs on manual test:
Screenshot 2026-07-27 at 3 00 15 PM

After submitting a workload, --watch follows the run's logs to completion and
exits with the run's outcome, reusing the same Bricklens-with-MLflow-fallback
pipeline as `air logs`.

- Text mode: prints "Submitted run", the dashboard link, and "Monitoring run
  and streaming logs...", then streams the logs.
- JSON mode: emits a SUBMITTED event with the run id, a STATUS event on each
  lifecycle transition, the streamed LOG/ALERT events, and a closing
  terminal-status envelope (SUCCESS/FAILED/CANCELED) — matching the Python CLI's
  --watch JSONL contract.
- Without --watch, the plain submit path now prints a tip about --watch.
- STATUS events are watch-scoped (opt-in via logRequest.onStatusChange), so the
  merged `air logs` output is unchanged.

--dry-run still takes precedence over --watch (nothing is submitted or streamed).

Co-authored-by: Isaac
@github-actions

Copy link
Copy Markdown
Contributor

Approval status: pending

/acceptance/experimental/air/ - needs approval

Files: acceptance/experimental/air/run-submit/output.txt, acceptance/experimental/air/run/output.txt, acceptance/experimental/air/run/script
Eligible: @apeforest, @bfontain, @lu-wang-dl, @panchalhp-db, @vinchenzo-db, @maggiewang-db, @ben-hansen-db, @pardis-beikzadeh-db

/experimental/air/ - needs approval

4 files changed
Eligible: @apeforest, @bfontain, @lu-wang-dl, @panchalhp-db, @vinchenzo-db, @maggiewang-db, @ben-hansen-db, @pardis-beikzadeh-db

Any maintainer (@andrewnester, @anton-107, @denik, @pietern, @shreyas-goenka, @simonfaltum, @renaudhartert-db, @janniklasrose, @lennartkats-db) can approve all areas.
See OWNERS for ownership rules.

@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

Integration test report

Commit: caa28f8

Run: 30309220941

Env ❌​FAIL 🟨​KNOWN 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
❌​ aws-cli-is linux 8 1 3 4 320 1035 7:28
❌​ aws-cli-is windows 8 1 3 4 322 1033 20:19
❌​ azure-cli-is linux 8 1 3 4 320 1034 7:18
❌​ azure-cli-is windows 8 1 3 4 322 1032 14:48
❌​ gcp-cli-is linux 8 4 4 319 1036 11:08
❌​ gcp-cli-is windows 8 4 4 321 1034 20:19
16 interesting tests: 8 FAIL, 4 KNOWN, 4 SKIP
Test Name aws-cli-is linux aws-cli-is windows azure-cli-is linux azure-cli-is windows gcp-cli-is linux gcp-cli-is windows
🟨​ TestAccept 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K 🟨​K
❌​ TestAccept/bundle/deploy/mlops-stacks ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/deploy/mlops-stacks/DATABRICKS_BUNDLE_ENGINE=direct ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/deploy/mlops-stacks/DATABRICKS_BUNDLE_ENGINE=terraform ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/generate/auto-bind ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/generate/auto-bind/DATABRICKS_BUNDLE_ENGINE=terraform ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
🙈​ TestAccept/bundle/invariant/no_drift 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
❌​ TestAccept/bundle/resources/grants/volumes ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/grants/volumes/DATABRICKS_BUNDLE_ENGINE=direct ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
❌​ TestAccept/bundle/resources/grants/volumes/DATABRICKS_BUNDLE_ENGINE=terraform ❌​F ❌​F ❌​F ❌​F ❌​F ❌​F
🙈​ TestAccept/bundle/resources/vector_search_endpoints/drift/recreated_same_name 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/bundle/resources/vector_search_indexes/recreate/embedding_dimension 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🙈​ TestAccept/ssh/connection 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestFetchRepositoryInfoAPI_FromRepo 💚​R 💚​R 💚​R 💚​R 🟨​K 🟨​K
🟨​ TestFetchRepositoryInfoAPI_FromRepo/root 💚​R 💚​R 💚​R 💚​R 🟨​K 🟨​K
🟨​ TestFetchRepositoryInfoAPI_FromRepo/subdir 💚​R 💚​R 💚​R 💚​R 🟨​K 🟨​K

@maggiewang-db maggiewang-db left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you do a manual test with the fallback mlflow path?

// envelope after streaming. Mirrors the Python CLI's --watch JSONL contract.
out := cmd.OutOrStdout()
printSubmittedEvent(out, runIDStr, dashboardURL)
terminalStatus := "FAILED"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isaac claims this terminalStatus is never updated in the fallback mlflow code path. Can you double check?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants