Skip to content

FIX: Restore Linux ARM64 Conda builds on Ubuntu 26.04 - #905

Open
Jahnvi Thakkar (jahnvi480) wants to merge 10 commits into
mainfrom
jahnvi/github-conda-qemu-fix
Open

Jahnvi Thakkar (jahnvi480) wants to merge 10 commits into
mainfrom
jahnvi/github-conda-qemu-fix

Conversation

@jahnvi480

@jahnvi480 Jahnvi Thakkar (jahnvi480) commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Work Item / Issue Reference

AB#50320

GitHub Issue: N/A (maintainer change; ADO reference above)


Summary

Port the focused fix from ADO PR !9017, address the GitHub review feedback, and trigger only the Official Conda build after a successful main wheel build.

Root cause: Conda build 192868, sourced from ADO main 988922bff2602334adf5e0d2cc8e39288746fb53, failed QEMU setup on Ubuntu 26.04 because qemu-user-static became a virtual package with no installation candidate. Correct Linux ARM64 wheels were present, but the non-executable target Python then entered macOS-only extraction and reported a misleading missing-wheel error.

Changes

  • Install concrete qemu-user and qemu-user-binfmt packages with libc6-arm64-cross; restart systemd binfmt registration and check the emulator, ARM64 loader, enabled handler and registered interpreter. Compare resolved interpreter paths so valid Ubuntu 24.04 binfmt-P symlinks work while broken or different interpreters fail.
  • Restrict interpreter-free wheel extraction to osx-*; failed Linux interpreter execution now reports an actionable QEMU/binfmt/loader diagnostic.
  • Emit a named setup-success output only after every QEMU check passes, and require it before the ARM64 build. Keep native Linux artifact publication independent. The regression test now requires the prerequisite installation command exactly once and rejects its omission.
  • After successful Build-Release-Package-Pipeline completion on refs/heads/main, automatically start an Official Conda-Build-Pipeline run. This trigger waits for the full upstream pipeline and retains its exact wheel producer run.
  • Do not equate that downstream Official label with production eligibility. The existing release provenance gate now requires the recorded wheel producer's effective Official mode, enabled SDL/signing settings, and completed/succeeded per-job ESRP wheel scans and Component Governance. Windows jobs also require their native scan, signing, Authenticode and signed-wheel verification; the ODBC wheel scan is required too. Missing, skipped, failed, wrong-job or previous-attempt evidence blocks production. Non-publishing validation reports ineligibility without granting publication.
  • Automation stops after the Conda build. No downstream release, rehearsal or publication is automatically triggered. The Conda release pipeline is restored exactly to its pre-automation configuration: manual invocation, publishToConda=false by default, explicit production opt-in and all existing release safeguards.
  • Add focused provisioning, build-trigger, manual-release and producer-security regressions and update the documentation.

Runner selection, native build behavior, ARM64 failure isolation, required release matrices, native audits, intended macOS extraction, Windows settings and publisher implementation remain unchanged. Provenance is tightened, not bypassed. Incomplete ARM64 artifacts still block release. Production still requires the existing approvals, restricted publishing group and exclusive lock.

Source equivalence and follow-ups: The initial port commit a0023d4a0e18407a259610cd6181481690404c1d exactly reproduced all five changed blobs from ADO !9017 head 3f6a1c65d08dbf47b070cd605963b1cca42aeb45, relative to ADO base 988922bff2602334adf5e0d2cc8e39288746fb53. Those baseline blobs also matched GitHub main 98ed24fbc6023ae014bbd52eda1b34024f7e7d5b. ADO !9017 was already merged when refreshed read-only. Follow-up commits address review feedback and the requested build-only trigger. The previously added automatic release/rehearsal changes were removed in a new commit, without rewriting history. GitHub main 6907546ebb2eab52341eafc4d1cad2e45333735e is merged. The eight-file PR diff contains no change to OneBranchPipelines/conda-release-pipeline.yml; the additional provenance module/test changes address the latest security-evidence comments.

Current validation: latest review fixes on merged main

  • python -m pytest --noconftest -p no:cacheprovider tests/test_027_conda_release_metadata.py tests/test_035_conda_macho_assert.py tests/test_036_conda_provenance.py -k "linux_qemu_setup or unix_recipe or posix_python_selection or conda_build_completion or conda_release_remains_manual or automatic_conda or publication_lock or dry_release or test_036_conda_provenance" -q - 477 passed, 528 deselected.
  • Before implementing these review fixes: python -m pytest --noconftest -p no:cacheprovider tests/test_027_conda_release_metadata.py tests/test_036_conda_provenance.py -k "linux_qemu_setup or wheel_security_evidence or scheduled_official_wheel" -q --tb=line - 18 failed, 7 passed, 927 deselected, reproducing the symlink rejection and missing producer-security enforcement.
  • Read-only real-run snapshots: the new wheel-security check accepts successful scheduled Official run 188915, rejects NonOfficial main run 192784, and still rejects 192784's missing 22 ESRP wheel scans when only its in-memory test label is changed to Official. No recorded evidence or ADO run was modified.
  • The release YAML Git blob exactly matches both main and pre-automation commit 425a707b (340f327c5c0f524397492646ee87444a78a79732). No downstream automation was reintroduced.
  • A disposable mutation removing the package-install command is still rejected by the strengthened ready-case test.
  • .\.venv\Scripts\python.exe -m pre_commit run black-check --all-files --hook-stage pre-push - passed with the repository-pinned Black hook.
  • bash -n conda/mssql-python/build.sh and bash -n with the extracted QEMU setup block on stdin - passed.
  • PyYAML parsing of the build, restored release and POSIX step pipeline YAML files - passed.
  • git diff --check - passed.

The source-only tests used Linux CPython 3.13.15 / pytest 9.1.1 in a disposable container with networking disabled, all capabilities dropped and a read-only checkout. The repository's documented --noconftest audit path requires neither a native extension build nor SQL Server. Initial-port validation separately reproduced all eight original regressions and passed its 30-case focused selection.

Historical ADO evidence (not rerun for this port): The source PR records official Ubuntu package/path verification, an offline QEMU 10.2.1 / ARM64 glibc-loader smoke check, and an exact-head ADO YAML preview.

Limits and rollout: The build-trigger YAML must reach the ADO repository's configured default branch, with existing pipeline-resource authorizations, before automatic Conda builds can run. No ADO definition or permission was changed, and no pipeline was manually queued, retried or previewed for these follow-ups. Hosted trigger delivery, systemd/binfmt registration and a complete emulated ARM64 Python/Conda build remain untested; local source tests and YAML parsing are not an ADO template compilation or hosted qualification. The full driver/integration suite was not run.

Manual rehearsal keeps its existing non-publishing authentication-probe behavior. No automatic release/rehearsal, package publication, merge or auto-merge is requested.

Port the focused QEMU provisioning and Linux interpreter failure handling from ADO PR 9017.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 17, 2026 18:05
@github-actions github-actions Bot added the pr-size: medium Moderate update size label Nov 17, 2026
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

PR Performance Report

✅ No regression detected

No consistent slowdowns detected across all 2 environments.

0 IMPROVEMENTS 0 SLOWDOWNS 2/2 ENVIRONMENTS

Coverage: 2 of 2 environments completed. Advisory result; does not block merging.

Performance diagnostics

Phase times are inclusive diagnostics and must not be added together. They identify where measured time changed, not why it changed.

No affected phases or call-count changes were recorded.

All database tasks and timings

Unix / SQL Server 2022

Database task Before After Paired change Result
Connection opening 10.496 ms 10.427 ms -2.4% no signal
SELECT queries 1.147 ms 1.029 ms -3.1% no signal
Row insertion 30.862 ms 30.285 ms -2.6% no signal
Executemany inserts 172.872 ms 164.278 ms -5.0% no signal
Fetch-all queries 120.453 ms 120.993 ms +2.7% no signal
Row-by-row fetching 4.949 ms 4.987 ms +0.6% no signal
Batched row fetching 116.500 ms 118.147 ms +1.4% no signal
Transaction commit and rollback 106.084 ms 105.100 ms +0.0% no signal
Arrow row fetching 91.617 ms 90.444 ms -1.1% no signal
100,000-row insertion 352.050 ms 345.730 ms +0.7% no signal
Row fetching in batches of 100 121.336 ms 121.937 ms -0.7% no signal
Row fetching in batches of 10,000 142.126 ms 132.058 ms +2.6% no signal
Repeated positional queries 32.105 ms 32.015 ms +0.7% no signal
Repeated named-parameter queries 32.358 ms 32.567 ms -0.2% no signal
Legacy 100,000-row insertion 256.428 ms 259.715 ms +0.6% no signal
Insertion with explicit input sizes 391.612 ms 395.271 ms +2.4% no signal
Joined aggregation queries 186.886 ms 183.053 ms -2.0% no signal
Large joined-result fetching 202.533 ms 202.811 ms +0.5% no signal
1.2-million-row fetching 3500.444 ms 3493.622 ms -0.2% no signal
Common table expression queries 5.350 ms 5.268 ms -0.8% no signal
256 KiB VARCHAR(MAX) / fetchall() 1.422 ms 1.314 ms -4.8% no signal
10,000 scalar values / fetchval() (debug disabled) 17.033 ms 17.029 ms -0.0% no signal
10,000 output-converted rows / fetchall() 19.990 ms 19.615 ms -1.6% no signal
10,000 output-converted rows / fetchmany(1000) 23.948 ms 23.467 ms -2.0% no signal
10,000 INT/NVARCHAR rows / fetchmany(1) 24.659 ms 24.621 ms -0.3% no signal
10,000 rows x 32 columns / dict(row._mapping) (warm) 32.169 ms 31.899 ms -1.9% no signal

Unix / SQL Server 2025

Database task Before After Paired change Result
Connection opening 82.550 ms 81.874 ms -0.3% no signal
SELECT queries 1.161 ms 1.035 ms -10.6% no signal
Row insertion 30.762 ms 30.842 ms +0.3% no signal
Executemany inserts 155.945 ms 157.649 ms +1.3% no signal
Fetch-all queries 121.368 ms 120.666 ms +0.5% no signal
Row-by-row fetching 4.945 ms 5.010 ms +1.7% no signal
Batched row fetching 118.825 ms 118.041 ms +1.2% no signal
Transaction commit and rollback 107.583 ms 107.791 ms +0.3% no signal
Arrow row fetching 90.869 ms 90.216 ms +1.7% no signal
100,000-row insertion 344.839 ms 344.755 ms +0.1% no signal
Row fetching in batches of 100 121.002 ms 121.839 ms -0.2% no signal
Row fetching in batches of 10,000 132.634 ms 141.792 ms +6.6% no signal
Repeated positional queries 31.381 ms 31.692 ms +0.3% no signal
Repeated named-parameter queries 31.830 ms 32.008 ms -0.3% no signal
Legacy 100,000-row insertion 271.246 ms 258.485 ms +0.7% no signal
Insertion with explicit input sizes 387.466 ms 396.179 ms +2.7% no signal
Joined aggregation queries 164.882 ms 165.816 ms -0.3% no signal
Large joined-result fetching 194.013 ms 196.733 ms +0.6% no signal
1.2-million-row fetching 3539.678 ms 3524.476 ms -0.7% no signal
Common table expression queries 5.262 ms 5.278 ms -1.1% no signal
256 KiB VARCHAR(MAX) / fetchall() 1.478 ms 1.967 ms +35.8% no signal
10,000 scalar values / fetchval() (debug disabled) 16.651 ms 16.682 ms -0.2% no signal
10,000 output-converted rows / fetchall() 19.349 ms 19.594 ms -0.2% no signal
10,000 output-converted rows / fetchmany(1000) 23.440 ms 24.514 ms +4.6% no signal
10,000 INT/NVARCHAR rows / fetchmany(1) 24.332 ms 24.654 ms +2.0% no signal
10,000 rows x 32 columns / dict(row._mapping) (warm) 32.273 ms 31.466 ms -1.4% no signal
Build and measurement details

ADO build 193259

PR head: 2098c959811757050255b3da6c883944898cc61f
Base: d257fdd86dc0c8f09462981cde0cf76e38b7d0d7
Measured merge: 69704b18410709e921d23eb802b4be531657f10e

  • Unix / SQL Server 2022: Python 3.12.3, x86_64, SQL 16.0.4325.4; 5 paired comparisons and 1 warmup.
  • Unix / SQL Server 2025: Python 3.12.3, x86_64, SQL 17.0.5065.9; 5 paired comparisons and 1 warmup.

A consistent change requires more than 20% median paired movement, at least 1 ms between the median runtimes, and at least 80% of pairs exceeding the relative threshold in the same direction. A slowdown without enough pair agreement is reported as inconsistent.

The displayed change is the median of paired before-and-after ratios. It is not recalculated from the two displayed median runtimes.

Both revisions use profiling-enabled builds on the same agent and database, with alternating order and discarded warmups. Results are diagnostic and do not represent production-wheel latency.

Raw samples and logs are attached to the ADO run as profiler-* artifacts.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The ARM64 build can proceed after a failed interpreter check, and the regression test does not verify that the prerequisite install command ran.

2 open issues
What changed in this PR

Restores Linux ARM64 Conda build provisioning on Ubuntu 26.04 and prevents failed Linux Python execution from entering the macOS-only wheel-extraction path.

Changes:

  • Replaces qemu-user-static with concrete QEMU packages and adds emulator, loader and binfmt checks.
  • Restricts interpreter-free extraction to osx-* targets and adds actionable Linux emulation errors.
  • Adds eight regression cases and documents the provisioning requirements.

Hosted binfmt registration and a complete emulated ARM64 build remain unverified.

File Description
tests/​test_035_conda_macho_assert.py Adds three cases rejecting failed Linux Python execution before wheel extraction.
tests/​test_027_conda_release_metadata.py Adds five mocked QEMU provisioning and validation cases.
OneBranchPipelines/​conda-build-pipeline.yml Updates QEMU packages, restarts binfmt registration and checks prerequisites.
conda/​README.md Documents Linux ARM64 provisioning and extraction restrictions.
conda/​mssql-python/​build.sh Limits extraction fallback to macOS and reports Linux emulation failures.

Review effort: Balanced

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

📊 Code Coverage Report

🔥 Diff Coverage

100%


🎯 Overall Coverage

86%


📈 Total Lines Covered: 11690 out of 13576
📁 Project: mssql-python


Diff Coverage

Diff: main...HEAD, staged and unstaged changes

No lines with coverage information in this diff.


📋 Files Needing Attention

📉 Files with overall lowest coverage (click to expand)
mssql_python.pybind.performance_counter.hpp: 0.7%
mssql_python.pybind.logger_bridge.cpp: 57.9%
mssql_python.pybind.ddbc_bindings.h: 64.5%
mssql_python.pybind.logger_bridge.hpp: 70.8%
mssql_python._native_fetch.py: 77.7%
mssql_python.__init__.py: 80.8%
mssql_python.pybind.ddbc_bindings.cpp: 81.4%
mssql_python.pybind.row_mapping_cache.hpp: 81.5%
mssql_python.pybind.connection.connection_pool.cpp: 82.5%
mssql_python.pybind.connection.connection.cpp: 83.1%

🔗 Quick Links

⚙️ Build Summary 📋 Coverage Details

View Azure DevOps Build

Browse Full Coverage Report

Require the setup success output before running the ARM64 leg, preserve native artifact publication, and assert prerequisite package installation.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Trigger from successful main wheel builds and exact Conda producer runs. Restrict production publication to explicit manual opt-in, preserving provenance, matrix and approval gates.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 17, 2026 18:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Official Conda releases may accept artifacts from wheels missing required scans or signatures, while successful automatic rehearsals report failures.

2 open issues
2 resolved since last review

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings November 18, 2026 04:23

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Official Conda builds can make unscanned wheels or unsigned Windows binaries production-eligible, while the ARM64 readiness check can reject valid symlinked emulators.

4 open issues

Review effort: Balanced

Remove downstream release/rehearsal completion triggering and restore the original manual release configuration. Preserve the main wheel-to-Official-Conda-build trigger and QEMU review fixes.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 18, 2026 07:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Official Conda builds can include unscanned wheels or unsigned Windows binaries, and the QEMU path check can skip ARM64 builds despite working emulation.

3 open issues
1 resolved since last review

Review effort: Balanced

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 18, 2026 08:19
@github-actions github-actions Bot added pr-size: large Substantial code update and removed pr-size: medium Moderate update size labels Nov 18, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The provisioning test invokes unmocked readlink -e, which is unsupported on stock macOS and Alpine, causing pytest jobs, including the macOS wheel build, to fail.

1 open issue
3 resolved since last review

Review effort: Balanced

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 18, 2026 09:46

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The Azure DevOps trigger, hosted binfmt/QEMU setup, and emulated ARM64 Conda build require end-to-end hosted qualification beyond source-only review.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

Copilot AI balanced review requested due to automatic review settings November 18, 2026 11:31

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Hosted Azure DevOps trigger and template execution, binfmt registration, and the full emulated ARM64 Conda build remain unverified and require human qualification.

No open issues 🎉

Review effort: Balanced

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed to evaluate how missing security evidence affects validation and production eligibility.

No open issues 🎉

Review effort: Balanced

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-size: large Substantial code update

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants