Fixes 5963: migrate BigQuery CLI E2E coverage to the v2 framework

[ulixius9]

Describe your changes:

Fixes open-metadata/openmetadata-collate#5963

Stacked on #33946 (base branch fix/bigquery-types-fk-constraints-profiler); GitHub retargets this to main once that merges.

I migrated the BigQuery CLI E2E coverage (cli_e2e/test_cli_bigquery.py and test_cli_bigquery_multiple_project.py) to the v2 framework from #27949, because the v2 path asserts persisted OpenMetadata state strictly instead of v1's status-count floors. The strict assertions found the six connector/profiler bugs fixed in #33946. The v1 tests and their py-cli-e2e-tests.yml matrix entries stay until the agreed stability window completes.

Type of change:

• Improvement

High-level design:

Follows cli_e2e_v2/CONNECTORS.md with no shared-core changes. The new code is under ingestion/tests/cli_e2e_v2/bigquery/:

• Source ownership (source.py, conftest.py): BigQuery can't run in a container, so each test creates its own dataset e2e_bq_<uuid> (label owner=cli-e2e-v2, 24h table expiration as a safety net) in two real projects, and deletes it with its contents on success and failure. Every invocation carries an anchored schemaFilterPattern for the owned datasets; a schema filter without explicit includes is rejected, so no run can ingest unowned project data.
• Auth: service-account key via E2E_BQ_* env vars by reference (CI default). E2E_BQ_AUTH=adc runs locally with Application Default Credentials (gcp_adc).
• Config (connector.py): single-project runs set billingProjectId to the second project, as v1 did; multi-project runs use a projectId list.
• Expectations (baseline.py, expected.py): authored independently of the connector's parsers, with an explicit BigQuery type map. Types are strict by review decision (NUMERIC → NUMERIC, JSON → JSON).
• Inventory: 22 contracts, mapped v1 → v2 in the README:
  • catalog: single- and multi-project;
  • filters: table ×4, schema ×2, database;
  • deletion, repeat ingestion, procedures, FKs, view + column lineage, PII tagging;
  • profiler metrics, system metrics, default latest-partition profiling;
  • sample limit, native sample values, sample replacement;
  • tableDiff.

Not included (needs maintainer authorization): .github/workflows/py-cli-e2e-tests-v2.yml must allowlist bigquery and pass the existing TEST_BQ_* secrets to the BigQuery job only; meta/test_ci_workflow.py needs the matching accepted case. Also note the job's 60-minute timeout: the suite takes about 9 minutes with -n 6 and longer serially.

Tests:

Use cases covered

• Every behaviour asserted by the two v1 BigQuery tests, plus stored procedure code, NOT ENFORCED foreign keys, repeat ingestion keeping IDs, native sample values for all BigQuery types and sample replacement.
• Fixture safety: datasets isolated and removed on success and on seed failure; both projects writable; constraints visible in INFORMATION_SCHEMA.

Unit tests

• ingestion/tests/cli_e2e_v2/meta/test_bigquery_cases.py (18 offline tests: owned-dataset scoping, credentials by reference, ADC config, invalid invocations, DQ config shape, strict type map, system-profile and partition checks).
• Full offline meta suite passes; --e2e-contract-check --collect-only collects all 22 contracts exactly once.

Backend integration tests

• Not applicable (no backend API changes).

Ingestion integration tests

• Added ingestion/tests/cli_e2e_v2/bigquery/ (live suite).

Playwright (UI) tests

• Not applicable (no UI changes).

Manual testing performed

1. Local OpenMetadata 2.0.0-SNAPSHOT via Docker.
2. Ran E2E_BQ_AUTH=adc E2E_BQ_PROJECT_ID=open-metadata-beta E2E_BQ_PROJECT_ID2=modified-leaf-330420 python -m pytest ingestion/tests/cli_e2e_v2/bigquery --e2e-contract-check -n 6 on top of Fixes 5963: migrate BigQuery CLI E2E to v2 and fix the bugs it found #33946: 25/26 passed in 9m16s. The remaining failure (catalog.multi-project) is a local IAM gap: my user lacks project-level bigquery.tables.list on the second project, so the region-scoped lifecycle query gets a 403. The CI service account should not hit this.
3. Verified no leaked e2e_bq_* datasets or e2e_* services after the runs.

UI screen recording / screenshots:

Not applicable.

Checklist:

• I have read the CONTRIBUTING document.

• My PR title is Fixes <issue-number>: <short explanation>

• My PR is linked to a GitHub issue via Fixes #<issue-number> above. — cross-repo link to the Collate tracking issue.

• I have commented on my code, particularly in hard-to-understand areas.

• For JSON Schema changes: I updated the migration scripts or explained why it is not needed. — no schema changes.

• For UI changes: I attached a screen recording and/or screenshots above. — no UI changes.

• I have added tests (unit / integration / Playwright as applicable) and listed them above.

• I have added tests around the new logic.

• For connector/ingestion changes: I updated the documentation. — ingestion/tests/cli_e2e_v2/README.md.

[gitar-bot]

💡 Quality: Use model_str() instead of .root for Pydantic RootModel strings

procedures_have_bodies reads item.name.root and native_sample_rows reads name.root. That goes against the ingestion guideline to convert RootModel values with model_str() from metadata.ingestion.ometa.utils. The file already imports model_str, so switching both call sites costs nothing.

Use model_str for the RootModel conversions:

procedure = next((item for item in snapshot.procedures if model_str(item.name) == "sp_active_customer_count"), None)
...
names = [model_str(name) for name in table.sampleData.columns]

_{Was this helpful? React with 👍 / 👎}

[gitar-bot]

Code Review 👍 Approved with suggestions 0 closed / 1 findings

🟡 Medium risk · Adds live BigQuery E2E fixtures that create, mutate, and delete isolated datasets.

Migrates BigQuery CLI E2E coverage to the v2 framework with comprehensive fixture safety (dataset isolation, cleanup on success and failure) and 22 test contracts covering catalog ingestion, filters, deletion, profiling, sampling, and lineage. Consider using model_str() instead of .root for Pydantic RootModel strings in checks.py to align with ingestion guidelines.

💡 Quality: Use model_str() instead of .root for Pydantic RootModel strings

📄 ingestion/tests/cli_e2e_v2/bigquery/checks.py:41 📄 ingestion/tests/cli_e2e_v2/bigquery/checks.py:155

procedures_have_bodies reads item.name.root and native_sample_rows reads name.root. That goes against the ingestion guideline to convert RootModel values with model_str() from metadata.ingestion.ometa.utils. The file already imports model_str, so switching both call sites costs nothing.

Use model_str for the RootModel conversions

procedure = next((item for item in snapshot.procedures if model_str(item.name) == "sp_active_customer_count"), None)
...
names = [model_str(name) for name in table.sampleData.columns]

🤖 Prompt for agents

Code Review: Migrates BigQuery CLI E2E coverage to the v2 framework with comprehensive fixture safety (dataset isolation, cleanup on success and failure) and 22 test contracts covering catalog ingestion, filters, deletion, profiling, sampling, and lineage. Consider using `model_str()` instead of `.root` for Pydantic RootModel strings in `checks.py` to align with ingestion guidelines.

1. 💡 Quality: Use model_str() instead of .root for Pydantic RootModel strings
   Files: ingestion/tests/cli_e2e_v2/bigquery/checks.py:41, ingestion/tests/cli_e2e_v2/bigquery/checks.py:155

   `procedures_have_bodies` reads `item.name.root` and `native_sample_rows` reads `name.root`. That goes against the ingestion guideline to convert RootModel values with `model_str()` from `metadata.ingestion.ometa.utils`. The file already imports `model_str`, so switching both call sites costs nothing.

   Fix (Use model_str for the RootModel conversions):
   procedure = next((item for item in snapshot.procedures if model_str(item.name) == "sp_active_customer_count"), None)
   ...
   names = [model_str(name) for name in table.sampleData.columns]

Review coverage

📋 Rules No rules evaluated

🧪 Functional validation Not enabled · Set up

🤖 Auto-approval Not enabled · Set up

Options

Display: compact → Counting what did not apply, without listing it.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source}

[github-actions]

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

• Linked issue open-metadata/openmetadata-collate#5963 does not exist or is not accessible.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.