Fixes 5963: migrate BigQuery CLI E2E to v2 and fix the bugs it found

[ulixius9]

Describe your changes:

Fixes open-metadata/openmetadata-collate#5963

I migrated the BigQuery CLI E2E coverage (cli_e2e/test_cli_bigquery.py and test_cli_bigquery_multiple_project.py) to the v2 framework from #27949, because v2 asserts persisted OpenMetadata state strictly instead of v1's status-count floors. Those strict assertions, run against real BigQuery, exposed six connector/profiler bugs, which this PR also fixes (with a regression unit test for each). The v1 tests and their py-cli-e2e-tests.yml matrix entries stay until the agreed stability window completes. (Includes #33947, merged into this branch.)

Type of change:

• Improvement

High-level design:

1. Connector and profiler fixes

Each fix is at the shared root cause rather than per caller:

Bug	Fix	Scope
NUMERIC/BIGNUMERIC → INT, JSON → VARCHAR	bigquery/metadata.py reflects them as sqlalchemy.NUMERIC / sqlalchemy.JSON; new profiler/orm/types/bigquery_json.py (BigQueryJSON, pass-through processors — the BigQuery dialect has no JSON deserializer and the DB-API already returns dicts) wired into the BigQuery ORM converter and NOT_COMPUTE	BigQuery
System metrics: sibling-table DML attributed to every table	system/bigquery/system.py: restore the per-table filter broken in #22044 (x or -1 > 0 and …, then noqa: RUF021), same shape as Snowflake	BigQuery
FK referred FQN built as svc.None.schema.table	common_db_source._prepare_foreign_constraints falls back to the current database when referred_database is absent, mirroring the existing referred_schema fallback	every connector with supportsDatabase that doesn't report it (BigQuery, Databricks, Trino, Presto, Vertica, Synapse, DB2, Unity Catalog, …)
ARRAY/STRUCT/MAP columns have no constraint	sql_column_handler: complex branch calls _get_column_constraints like the primitive branch	BigQuery, Databricks, Trino, Hive, Athena, ClickHouse
Column._set_parent() missing 'all_names' on STRUCT subfields	sampler/sqlalchemy/bigquery/sampler.py: SQLAlchemy 2 arguments, as #32549 did for the profiler interface	BigQuery
Unique count KeyError: 'struct_col.x'; intermittent "session is provisioning a new connection"	SQAProfilerInterface._compute_query_metrics: address a STRUCT subfield by path when it isn't a sample column, and build the grouped query on the metric thread's session instead of the shared self.session	struct part BigQuery; session race all SQLAlchemy connectors

Rollout: on the next ingestion, existing BigQuery NUMERIC/JSON columns change data type, complex columns gain a constraint, and previously dropped FKs appear — one round of change events per affected column/table; nothing is removed.

2. BigQuery CLI E2E v2 suite

Follows cli_e2e_v2/CONNECTORS.md with no shared-core changes. The new code is under ingestion/tests/cli_e2e_v2/bigquery/:

• Source ownership (source.py, conftest.py): BigQuery can't run in a container, so each test creates its own dataset e2e_bq_<uuid> (label owner=cli-e2e-v2, 24h table expiration as a safety net) in two real projects, and deletes it with its contents on success and failure. Every invocation carries an anchored schemaFilterPattern for the owned datasets; a schema filter without explicit includes is rejected, so no run can ingest unowned project data.
• Auth: service-account key via E2E_BQ_* env vars by reference (CI default). E2E_BQ_AUTH=adc runs locally with Application Default Credentials (gcp_adc).
• Config (connector.py): single-project runs set billingProjectId to the second project, as v1 did; multi-project runs use a projectId list.
• Expectations (baseline.py, expected.py): authored independently of the connector's parsers, with an explicit BigQuery type map. Types are strict by review decision (NUMERIC → NUMERIC, JSON → JSON).
• Inventory: 22 contracts, mapped v1 → v2 in the README:
  • catalog: single- and multi-project;
  • filters: table ×4, schema ×2, database;
  • deletion, repeat ingestion, procedures, FKs, view + column lineage, PII tagging;
  • profiler metrics, system metrics, default latest-partition profiling;
  • sample limit, native sample values, sample replacement;
  • tableDiff.

Not included (needs maintainer authorization): .github/workflows/py-cli-e2e-tests-v2.yml must allowlist bigquery and pass the existing TEST_BQ_* secrets to the BigQuery job only; meta/test_ci_workflow.py needs the matching accepted case. Also note the job's 60-minute timeout: the suite takes about 9 minutes with -n 6 and longer serially.

Tests:

Use cases covered

• BigQuery NUMERIC(10,2)/BIGNUMERIC ingest as NUMERIC with precision/scale, JSON as JSON; INT64/STRING unchanged.
• JSON columns sample as dicts and are skipped for metrics.
• System profile for a table excludes DML on sibling tables, other datasets and other projects.
• A FK without referred_database resolves in the current database.
• Nullable/required ARRAY and STRUCT columns get NULL/NOT_NULL.
• STRUCT subfield sampling and unique count over a sample that selects only the parent struct.
• Unique count executes on the metric thread's session.
• Every behaviour asserted by the two v1 BigQuery tests, plus stored procedure code, NOT ENFORCED foreign keys, repeat ingestion keeping IDs, native sample values for all BigQuery types and sample replacement.
• Fixture safety: datasets isolated and removed on success and on seed failure; both projects writable; constraints visible in INFORMATION_SCHEMA.

Unit tests

• I added unit tests for the new/changed logic — each failed before its fix with the exact production error.
• Files added: tests/unit/topology/database/test_bigquery_column_types.py, tests/unit/topology/database/test_complex_column_constraints.py, tests/unit/metadata/ingestion/source/database/bigquery/profiler/test_bigquery_system_profile.py
• Files updated: tests/unit/topology/database/test_common_db_source.py, tests/unit/observability/profiler/sqlalchemy/bigquery/test_bigquery_profiler_sql.py, tests/unit/observability/profiler/sqlalchemy/bigquery/test_bigquery_sampling.py
• Coverage (coverage run -m pytest over the tests above): sampler/sqlalchemy/bigquery/sampler.py 96%, orm/types/bigquery_json.py 91%, orm/converter/bigquery/converter.py 71%, metrics/system/bigquery/system.py 68% (uncovered lines are the pre-existing JOBS query path; the changed filter is fully covered). Changed lines in the large shared files are covered by the new tests.
• Regression: tests/unit/topology/database, tests/unit/source/database, profiler/sampler suites — identical failures/errors to a clean main checkout (all from optional connector packages missing locally), plus the new tests passing.
• ingestion/tests/cli_e2e_v2/meta/test_bigquery_cases.py (18 offline tests: owned-dataset scoping, credentials by reference, ADC config, invalid invocations, DQ config shape, strict type map, system-profile and partition checks).
• Full offline meta suite passes; --e2e-contract-check --collect-only collects all 22 contracts exactly once.

Backend integration tests

• Not applicable (no backend API changes).

Ingestion integration tests

• Added ingestion/tests/cli_e2e_v2/bigquery/: the live suite (real BigQuery → CLI → OpenMetadata) that found and verified the fixes.

Playwright (UI) tests

• Not applicable (no UI changes).

Manual testing performed

1. Local OpenMetadata 2.0.0-SNAPSHOT via Docker; BigQuery projects open-metadata-beta + modified-leaf-330420 with Application Default Credentials.
2. E2E_BQ_AUTH=adc E2E_BQ_PROJECT_ID=open-metadata-beta E2E_BQ_PROJECT_ID2=modified-leaf-330420 python -m pytest ingestion/tests/cli_e2e_v2/bigquery --e2e-contract-check -n 6:
   • 13/26 passing before the fixes, 25/26 after (9m16s).
   • The remaining failure (catalog.multi-project) is a local IAM gap: my user lacks project-level bigquery.tables.list on the second project, so the region-scoped lifecycle query gets a 403. The CI service account should not hit it.
3. Five back-to-back profiler workflows on partitioned + STRUCT tables: 0 failures (previously failed on the first run with the session race).
4. Verified no leaked e2e_bq_* datasets or e2e_* services after the runs.

UI screen recording / screenshots:

Not applicable.

Checklist:

• I have read the CONTRIBUTING document.

• My PR title is Fixes <issue-number>: <short explanation>

• My PR is linked to a GitHub issue via Fixes #<issue-number> above. — cross-repo link to the Collate tracking issue.

• I have commented on my code, particularly in hard-to-understand areas.

• For JSON Schema changes: I updated the migration scripts or explained why it is not needed. — no schema changes.

• For UI changes: I attached a screen recording and/or screenshots above. — no UI changes.

• I have added tests (unit / integration / Playwright as applicable) and listed them above.

• I have added tests around the new logic.

• For connector/ingestion changes: I updated the documentation. — ingestion/tests/cli_e2e_v2/README.md.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[github-actions]

❌ PR checklist incomplete

This PR cannot be merged until the following are addressed on its linked issue:

• Linked issue open-metadata/openmetadata-collate#5963 does not exist or is not accessible.

The fields live on the linked issue in the Shipping project (open the issue → right sidebar → Projects). After you set them, re-run this check (or push a commit) — issue/project changes do not re-trigger it automatically.

Maintainers can bypass this check by adding the skip-pr-checks label.

[github-actions]

✅ Playwright Results — workflow succeeded

Validated commit 8519c4bca9c4b17ef127243f6476eefb6d304a18 in Playwright run 36105123692, attempt 1.

✅ 110 passed · ❌ 0 failed · 🟡 1 flaky · ⏭️ 0 skipped · 🧰 0 lifecycle flaky

Performance

Blocking targets: ✅ met · Optimization targets: 🟡 in progress

Shard-job maxima below are not the full workflow wall time; the linked run includes build, fixture, planning, and reporting.

🕒 Full workflow signal wall (to summary) 50m 16s

⏱️ Max setup 4m 16s · max shard execution 12m 34s · max shard-job elapsed before upload 19m 45s · reporting 15s

🌐 106.68 requests/attempt · 1.81 app boots/UI scenario · 0.00% common-shard skew

Optimization targets still in progress:

• Application boot ratio was 1.81 per UI scenario (228 boots / 126 scenarios; convergence target: at most 1).

Shard	Passed	Failed	Flaky	Skipped	Lifecycle failed	Lifecycle flaky
✅ Shard chromium-01	46	0	0	0	0	0
🟡 Shard ingestion-01	31	0	1	0	0	0
✅ Shard ingestion-02	33	0	0	0	0	0

🟡 1 flaky test(s) (passed on retry)

• Features/IncidentManager.spec.ts › Complete Incident lifecycle with table owner (shard ingestion-01, 1 retry)

📦 Download artifacts

How to debug locally

# Download playwright-test-results-<shard> artifact and unzip
npx playwright show-trace path/to/trace.zip    # view trace

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[gitar-bot]

Code Review ✅ Approved

🟡 Medium risk · Shared ingestion fixes alter catalog types, constraints, and profiling across connectors

Fixes six BigQuery ingestion and profiler bugs exposed by strict E2E assertions: NUMERIC/JSON columns now ingest as correct types, system metrics properly filter to per-table DML, foreign keys without referred_database resolve in the current database, complex columns gain NULL constraints, STRUCT subfield sampling works correctly, and a session-race in metric computation is eliminated. Comprehensive unit tests cover all fixes with high coverage; integration testing verified against a real BigQuery project. No issues found.

Review coverage

📋 Rules No rules evaluated

🧪 Functional validation Not enabled · Set up

🤖 Auto-approval Not enabled · Set up

Options

Display: compact → Counting what did not apply, without listing it.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source}

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ingestion'

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
60.6% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud