Commit 69d4757
FEAT: Integrate mssql_py_core wheel installation into Dev workflow PR validation, official and release pipelines (#440)
### Work Item / Issue Reference
>
[AB#42668](https://sqlclientdrivers.visualstudio.com/c6d89619-62de-46a0-8b46-70b92a84d85e/_workitems/edit/42668)
-------------------------------------------------------------------
### Summary
# PR Review Guide — Integrate mssql-py-core into mssql-python
Testing: Official build at
https://sqlclientdrivers.visualstudio.com/mssql-python/_build?definitionId=2199&_a=summary
PR pipeline works.
---
## What this PR does
Ships `mssql_py_core` (the Rust-native bulk-copy engine) inside every
`mssql-python` wheel. After this PR, `import mssql_py_core` works out of
the box after `pip install mssql-python`.
### Before / After
```
BEFORE AFTER
────── ─────
┌─────────────────────────┐ ┌─────────────────────────┐
│ mssql-rs repo │ │ mssql-rs repo │
│ (builds mssql_py_core) │ │ (builds mssql_py_core) │
└────────┬────────────────┘ └────────┬────────────────┘
│ │
│ NuGet publish │ NuGet publish
▼ ▼
┌─────────────────────┐ ┌─────────────────────┐
│ Azure Artifacts │ │ Azure Artifacts │
│ (nupkg with .whl) │ │ (nupkg with .whl) │
└─────────────────────┘ └────────┬────────────┘
│
(no connection) install-mssql-py-core.sh/.ps1
│ downloads + extracts
▼
┌─────────────────────────┐ ┌─────────────────────────┐
│ mssql-python repo │ │ mssql-python repo │
│ │ │ │
│ setup.py bdist_wheel │ │ setup.py bdist_wheel │
│ └── mssql_python/ │ │ ├── mssql_python/ │
│ │ │ └── mssql_py_core/ ◄─── NEW
└────────┬────────────────┘ └────────┬────────────────┘
│ │
▼ ▼
┌─────────────────────┐ ┌─────────────────────┐
│ .whl (PyPI) │ │ .whl (PyPI) │
│ └── mssql_python/ │ │ ├── mssql_python/ │
└─────────────────────┘ │ └── mssql_py_core/ │
└─────────┬───────────┘
│
┌─────────▼───────────┐
│ Release pipelines │
│ validate version │
│ is NOT dev/alpha │
└─────────────────────┘
```
### Key design decisions
| Decision | Rationale |
|---|---|
| `mssql_py_core` binary is pre-built in `mssql-rs`, not compiled here |
It depends on OpenSSL 3 / glibc 2.34 (manylinux_2_34). Building it
inside mssql-python's 2_28 container isn't possible. |
| `mssql-python` wheels stay on `manylinux_2_28` | Broader distro
compatibility (RHEL 8, Debian 10+, Ubuntu 20.04+). The pre-built `.so`
from `mssql-rs` is just byte-copied in. |
| Version pinned in `eng/versions/mssql-py-core.version` | Single source
of truth. All install scripts read from here. Release pipelines validate
it. |
| Release pipelines reject pre-release versions | Prevents accidentally
shipping `dev`/`alpha`/`beta`/`rc` builds of `mssql_py_core` to PyPI. |
---
## Recommended review order
The PR has three areas. Review them in this order:
### Area 1 — Install infrastructure (core of this PR)
Read these together — they form a single pipeline:
| # | File | What changed |
|---|---|---|
| 1 |
[eng/versions/mssql-py-core.version](eng/versions/mssql-py-core.version)
| **New.** Single-line version file (`0.1.0-dev.20260222.140833`). All
scripts read from here. |
| 2 |
[eng/scripts/resolve_nuget_feed.py](eng/scripts/resolve_nuget_feed.py) |
**New.** Fetches NuGet v3 service index, prints `PackageBaseAddress`
URL. Used by the shell script. |
| 3 | [eng/scripts/extract_wheel.py](eng/scripts/extract_wheel.py) |
**New.** Extracts `mssql_py_core/` entries from a `.whl` (ZIP), skipping
`.dist-info` and `.libs`. Used by both PS1 and SH. |
| 4 |
[eng/scripts/install-mssql-py-core.ps1](eng/scripts/install-mssql-py-core.ps1)
| **New.** Windows install script: read version → resolve feed →
download nupkg → find matching wheel → extract → verify import. |
| 5 |
[eng/scripts/install-mssql-py-core.sh](eng/scripts/install-mssql-py-core.sh)
| **New.** Unix install script: same flow as PS1, plus musl/glibc
detection for Alpine. |
| 6 |
[OneBranchPipelines/scripts/validate-release-versions.ps1](OneBranchPipelines/scripts/validate-release-versions.ps1)
| **New.** Reads a `.version` file, rejects `dev\|alpha\|beta\|rc`.
Called by OneBranch release pipelines. |
### Area 2 — Pipeline integration
These files wire the install scripts into every pipeline:
| # | File | What changed |
|---|---|---|
| 7 |
[eng/pipelines/steps/install-mssql-py-core.yml](eng/pipelines/steps/install-mssql-py-core.yml)
| **New.** Step template with 3 variants: `windows`, `unix`,
`container`. Calls PS1 or SH. |
| 8 |
[eng/pipelines/pr-validation-pipeline.yml](eng/pipelines/pr-validation-pipeline.yml)
| **Modified.** Adds `install-mssql-py-core.yml` template ref in all 12
test jobs (Windows, macOS, Linux containers ×2, RHEL ×2, Alpine ×2,
Azure SQL ×3, codecov). |
| 9 |
[OneBranchPipelines/stages/build-linux-single-stage.yml](OneBranchPipelines/stages/build-linux-single-stage.yml)
| **Modified.** Adds `install-mssql-py-core.sh` call inside the
per-Python-version build loop. Adds `curl` to package installs. Updates
comment about manylinux_2_28 vs 2_34. |
| 10 |
[OneBranchPipelines/stages/build-windows-single-stage.yml](OneBranchPipelines/stages/build-windows-single-stage.yml)
| **Modified.** Adds PowerShell install step before testing. |
| 11 |
[OneBranchPipelines/stages/build-macos-single-stage.yml](OneBranchPipelines/stages/build-macos-single-stage.yml)
| **Modified.** Adds bash install step before testing. |
| 12 |
[OneBranchPipelines/jobs/consolidate-artifacts-job.yml](OneBranchPipelines/jobs/consolidate-artifacts-job.yml)
| **Modified.** `checkout: self` instead of `checkout: none` so we can
copy the version file into `dist/` for traceability. |
| 13 |
[OneBranchPipelines/official-release-pipeline.yml](OneBranchPipelines/official-release-pipeline.yml)
| **Modified.** Adds version validation step — calls
`validate-release-versions.ps1` to block pre-release versions before
ESRP release. |
| 14 |
[OneBranchPipelines/dummy-release-pipeline.yml](OneBranchPipelines/dummy-release-pipeline.yml)
| **Modified.** Same version validation gate as official pipeline. |
| 15 |
[OneBranchPipelines/build-release-package-pipeline.yml](OneBranchPipelines/build-release-package-pipeline.yml)
| **Trivial.** Comment fix (`manylinux_2_28` → `manylinux_2_34`). |
### Area 3 — Build packaging and tests
| # | File | What changed |
|---|---|---|
| 16 | [setup.py](setup.py) | **Modified.** Adds
`validate_mssql_py_core()` pre-flight check, includes `mssql_py_core` in
`packages` and `package_data`, uses `manylinux_2_28` tags. |
| 17 | [tests/test_019_bulkcopy.py](tests/test_019_bulkcopy.py) |
**New.** Basic bulkcopy integration test — insert 3 rows via
`_bulkcopy()`, verify round-trip. |
### Ancillary
| # | File | What changed |
|---|---|---|
| 18 | [.gitignore](.gitignore) | Adds `mssql_py_core/` (extracted at
dev time, not checked in). |
---
## Things to look for
1. **Version file path consistency** — every script resolves
`eng/versions/mssql-py-core.version` relative to repo root. Verify the
consolidation job copies it correctly into the artifact.
2. **musl/glibc detection in `install-mssql-py-core.sh`** — Alpine uses
musl. The script gracefully exits 0 (skip) when no musllinux wheel is
available rather than failing the build.
3. **`setup.py` validation** — `validate_mssql_py_core()` runs at
wheel-build time. Must not block `pip install` from PyPI (the check only
runs during `bdist_wheel`).
4. **`manylinux_2_28` everywhere in `mssql-python`** — `setup.py`
platform tags and `build-linux-single-stage.yml` container selection
both use `manylinux_2_28`. The `mssql_py_core` `.so` is built on
`manylinux_2_34` in the `mssql-rs` repo but byte-copied in here.
5. **Release pipeline gating** — both OneBranch release pipelines
(`OneBranchPipelines/official-release-pipeline.yml` and
`OneBranchPipelines/dummy-release-pipeline.yml`) validate the version
file from build artifacts and reject pre-release versions before ESRP
submission. The script lives at
`OneBranchPipelines/scripts/validate-release-versions.ps1`.
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: gargsaumya <saumyagarg.100@gmail.com>1 parent 865a379 commit 69d4757
17 files changed
Lines changed: 892 additions & 16 deletions
File tree
- OneBranchPipelines
- jobs
- scripts
- stages
- eng
- pipelines
- steps
- scripts
- versions
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
61 | 61 | | |
62 | 62 | | |
63 | 63 | | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
64 | 67 | | |
65 | 68 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
174 | 174 | | |
175 | 175 | | |
176 | 176 | | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
177 | 185 | | |
178 | 186 | | |
179 | 187 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
26 | 26 | | |
27 | 27 | | |
28 | 28 | | |
29 | | - | |
| 29 | + | |
| 30 | + | |
30 | 31 | | |
31 | 32 | | |
32 | 33 | | |
| |||
112 | 113 | | |
113 | 114 | | |
114 | 115 | | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
115 | 123 | | |
116 | 124 | | |
117 | 125 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
177 | 177 | | |
178 | 178 | | |
179 | 179 | | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
180 | 188 | | |
181 | 189 | | |
182 | 190 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
104 | 104 | | |
105 | 105 | | |
106 | 106 | | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
107 | 110 | | |
108 | 111 | | |
109 | 112 | | |
| |||
126 | 129 | | |
127 | 130 | | |
128 | 131 | | |
129 | | - | |
| 132 | + | |
130 | 133 | | |
131 | 134 | | |
132 | | - | |
| 135 | + | |
133 | 136 | | |
134 | 137 | | |
135 | 138 | | |
| |||
138 | 141 | | |
139 | 142 | | |
140 | 143 | | |
141 | | - | |
| 144 | + | |
142 | 145 | | |
143 | 146 | | |
144 | 147 | | |
| |||
208 | 211 | | |
209 | 212 | | |
210 | 213 | | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
211 | 217 | | |
212 | 218 | | |
213 | 219 | | |
| |||
273 | 279 | | |
274 | 280 | | |
275 | 281 | | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
276 | 285 | | |
277 | 286 | | |
278 | 287 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
169 | 169 | | |
170 | 170 | | |
171 | 171 | | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
172 | 181 | | |
173 | 182 | | |
174 | 183 | | |
| |||
186 | 195 | | |
187 | 196 | | |
188 | 197 | | |
| 198 | + | |
189 | 199 | | |
190 | 200 | | |
191 | 201 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
238 | 238 | | |
239 | 239 | | |
240 | 240 | | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
241 | 252 | | |
242 | 253 | | |
243 | 254 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
230 | 230 | | |
231 | 231 | | |
232 | 232 | | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
233 | 237 | | |
234 | 238 | | |
235 | 239 | | |
| |||
497 | 501 | | |
498 | 502 | | |
499 | 503 | | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
500 | 508 | | |
501 | 509 | | |
502 | 510 | | |
| |||
669 | 677 | | |
670 | 678 | | |
671 | 679 | | |
| 680 | + | |
| 681 | + | |
| 682 | + | |
| 683 | + | |
| 684 | + | |
| 685 | + | |
672 | 686 | | |
673 | 687 | | |
674 | 688 | | |
| |||
984 | 998 | | |
985 | 999 | | |
986 | 1000 | | |
| 1001 | + | |
| 1002 | + | |
| 1003 | + | |
| 1004 | + | |
| 1005 | + | |
| 1006 | + | |
987 | 1007 | | |
988 | 1008 | | |
989 | 1009 | | |
| |||
1192 | 1212 | | |
1193 | 1213 | | |
1194 | 1214 | | |
| 1215 | + | |
| 1216 | + | |
| 1217 | + | |
| 1218 | + | |
| 1219 | + | |
| 1220 | + | |
1195 | 1221 | | |
1196 | 1222 | | |
1197 | 1223 | | |
| |||
1411 | 1437 | | |
1412 | 1438 | | |
1413 | 1439 | | |
| 1440 | + | |
| 1441 | + | |
| 1442 | + | |
| 1443 | + | |
| 1444 | + | |
| 1445 | + | |
1414 | 1446 | | |
1415 | 1447 | | |
1416 | 1448 | | |
| |||
1638 | 1670 | | |
1639 | 1671 | | |
1640 | 1672 | | |
| 1673 | + | |
| 1674 | + | |
| 1675 | + | |
| 1676 | + | |
| 1677 | + | |
| 1678 | + | |
1641 | 1679 | | |
1642 | 1680 | | |
1643 | 1681 | | |
| |||
1883 | 1921 | | |
1884 | 1922 | | |
1885 | 1923 | | |
| 1924 | + | |
| 1925 | + | |
| 1926 | + | |
| 1927 | + | |
| 1928 | + | |
| 1929 | + | |
1886 | 1930 | | |
1887 | 1931 | | |
1888 | 1932 | | |
| |||
2005 | 2049 | | |
2006 | 2050 | | |
2007 | 2051 | | |
| 2052 | + | |
| 2053 | + | |
| 2054 | + | |
| 2055 | + | |
2008 | 2056 | | |
2009 | 2057 | | |
2010 | 2058 | | |
| |||
2047 | 2095 | | |
2048 | 2096 | | |
2049 | 2097 | | |
| 2098 | + | |
| 2099 | + | |
| 2100 | + | |
| 2101 | + | |
2050 | 2102 | | |
2051 | 2103 | | |
2052 | 2104 | | |
| |||
2118 | 2170 | | |
2119 | 2171 | | |
2120 | 2172 | | |
| 2173 | + | |
| 2174 | + | |
| 2175 | + | |
| 2176 | + | |
| 2177 | + | |
| 2178 | + | |
2121 | 2179 | | |
2122 | 2180 | | |
2123 | 2181 | | |
| |||
2212 | 2270 | | |
2213 | 2271 | | |
2214 | 2272 | | |
| 2273 | + | |
| 2274 | + | |
| 2275 | + | |
| 2276 | + | |
2215 | 2277 | | |
2216 | 2278 | | |
2217 | 2279 | | |
| |||
0 commit comments