Skip to content

Commit 1e2f202

Browse files
bewithgauravbewithgaurav
authored
FEAT: Enhance pipeline dependencies and add Python < 3.13 (#111)
### ADO Work Item Reference <!-- Insert your ADO Work Item ID below (e.g. AB#37452) --> > [AB#37821](https://sqlclientdrivers.visualstudio.com/c6d89619-62de-46a0-8b46-70b92a84d85e/_workitems/edit/37821) ------------------------------------------------------------------- ### Summary <!-- Insert your Copilot Generated Summary below --> This pull request updates the build and validation pipelines to improve compatibility, streamline database setup, and enhance testing workflows. Key changes include updating Python version specifications, adding support for LocalDB and Docker-based SQL Server configurations, and ensuring pytest is run with proper database connections. ### Updates to Python version specifications: * Updated Python version from `3.13.5` to `3.13` across multiple jobs in `eng/pipelines/build-whl-pipeline.yml` and `eng/pipelines/pr-validation-pipeline.yml`, reflecting the availability of the new version. [[1]](diffhunk://#diff-a871d64acfb78366b4b077045db53551c2df7d622854569f5f6780ad2ec6f911L58-R63) [[2]](diffhunk://#diff-296c8f902bbd70f34ee1c8c32383c8c99165fe4c8e5b0f234f8f22246e56a621L17-R17) [[3]](diffhunk://#diff-296c8f902bbd70f34ee1c8c32383c8c99165fe4c8e5b0f234f8f22246e56a621L91-R87) ### Database setup enhancements: * Added steps to start a LocalDB instance, create a database, and configure a user for testing in `eng/pipelines/build-whl-pipeline.yml`. * Integrated Docker-based SQL Server setup in both pipelines, including pulling the SQL Server container, starting it, and verifying its operational status. [[1]](diffhunk://#diff-a871d64acfb78366b4b077045db53551c2df7d622854569f5f6780ad2ec6f911R275-R320) [[2]](diffhunk://#diff-296c8f902bbd70f34ee1c8c32383c8c99165fe4c8e5b0f234f8f22246e56a621L100-R133) ### Testing improvements: * Ensured pytest runs with appropriate database connection strings for LocalDB and Docker-based SQL Server setups in `eng/pipelines/build-whl-pipeline.yml` and `eng/pipelines/pr-validation-pipeline.yml`. Removed reliance on Azure SQL Database for testing on macOS. [[1]](diffhunk://#diff-a871d64acfb78366b4b077045db53551c2df7d622854569f5f6780ad2ec6f911R134-R140) [[2]](diffhunk://#diff-296c8f902bbd70f34ee1c8c32383c8c99165fe4c8e5b0f234f8f22246e56a621L154-R150) ### Expanded Python version support: * Added jobs for Python versions `3.10`, `3.11`, and `3.12` with `universal2` architecture for macOS in `eng/pipelines/build-whl-pipeline.yml`. <!-- ### PR Title Guide > For feature requests FEAT: (short-description) > For non-feature requests like test case updates, config updates , dependency updates etc CHORE: (short-description) > For Fix requests FIX: (short-description) > For doc update requests DOC: (short-description) > For Formatting, indentation, or styling update STYLE: (short-description) > For Refactor, without any feature changes REFACTOR: (short-description) > For release related changes, without any feature changes RELEASE: #<RELEASE_VERSION> (short-description) -->
1 parent 25b3330 commit 1e2f202

2 files changed

Lines changed: 133 additions & 53 deletions

File tree

‎eng/pipelines/build-whl-pipeline.yml‎

Lines changed: 92 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -55,14 +55,12 @@ jobs:
5555

5656
# Python 3.13
5757
py313_x64:
58-
# TODO: Remove this once Python 3.13.5 is available in ADO
59-
pythonVersion: '3.13.5'
58+
pythonVersion: '3.13'
6059
shortPyVer: '313'
6160
architecture: 'x64'
6261
targetArch: 'x64'
6362
py313_arm64:
64-
# TODO: Remove this once Python 3.13.5 is available in ADO
65-
pythonVersion: '3.13.5'
63+
pythonVersion: '3.13'
6664
shortPyVer: '313'
6765
architecture: 'x64'
6866
targetArch: 'arm64'
@@ -83,6 +81,22 @@ jobs:
8381
pip install cmake pybind11
8482
displayName: 'Install dependencies'
8583
84+
# Start LocalDB instance
85+
- powershell: |
86+
sqllocaldb create MSSQLLocalDB
87+
sqllocaldb start MSSQLLocalDB
88+
displayName: 'Start LocalDB instance'
89+
90+
# Create database and user
91+
- powershell: |
92+
sqlcmd -S "(localdb)\MSSQLLocalDB" -Q "CREATE DATABASE TestDB"
93+
sqlcmd -S "(localdb)\MSSQLLocalDB" -Q "CREATE LOGIN testuser WITH PASSWORD = '$(DB_PASSWORD)'"
94+
sqlcmd -S "(localdb)\MSSQLLocalDB" -d TestDB -Q "CREATE USER testuser FOR LOGIN testuser"
95+
sqlcmd -S "(localdb)\MSSQLLocalDB" -d TestDB -Q "ALTER ROLE db_owner ADD MEMBER testuser"
96+
displayName: 'Setup database and user'
97+
env:
98+
DB_PASSWORD: $(DB_PASSWORD)
99+
86100
- task: DownloadPipelineArtifact@2
87101
condition: eq(variables['targetArch'], 'arm64')
88102
inputs:
@@ -115,8 +129,15 @@ jobs:
115129
call keep_single_arch.bat $(targetArch)
116130
117131
cd ..\..
118-
displayName: 'Build PYD for $(targetArch)'
119-
132+
displayName: 'Build PYD for $(targetArch)'
133+
134+
# Run pytests before packaging
135+
- script: |
136+
python -m pytest -v
137+
displayName: 'Run pytests'
138+
env:
139+
DB_CONNECTION_STRING: 'Server=(localdb)\MSSQLLocalDB;Database=TestDB;Uid=testuser;Pwd=$(DB_PASSWORD);TrustServerCertificate=yes'
140+
120141
# Copy the built .pyd file to staging folder for artifacts
121142
- task: CopyFiles@2
122143
inputs:
@@ -193,6 +214,25 @@ jobs:
193214
shortPyVer: '313'
194215
# Always use universal2 for macOS
195216
targetArch: 'universal2'
217+
218+
# Python 3.12 (universal2 for both arm64 and x86_64)
219+
py312_universal2:
220+
pythonVersion: '3.12'
221+
shortPyVer: '312'
222+
targetArch: 'universal2'
223+
224+
# Python 3.11 (universal2 for both arm64 and x86_64)
225+
py311_universal2:
226+
pythonVersion: '3.11'
227+
shortPyVer: '311'
228+
targetArch: 'universal2'
229+
230+
# Python 3.10 (universal2 for both arm64 and x86_64)
231+
py310_universal2:
232+
pythonVersion: '3.10'
233+
shortPyVer: '310'
234+
targetArch: 'universal2'
235+
196236
steps:
197237
# Use correct Python version and architecture for the current job
198238
- task: UsePythonVersion@0
@@ -231,7 +271,53 @@ jobs:
231271
Contents: '*.so'
232272
TargetFolder: '$(Build.ArtifactStagingDirectory)/ddbc-bindings'
233273
displayName: 'Place .so file into artifacts directory'
274+
275+
- script: |
276+
brew update
277+
brew install docker colima
278+
279+
# Start Colima with extra resources
280+
colima start --cpu 4 --memory 8 --disk 50
281+
282+
# Optional: set Docker context (usually automatic)
283+
docker context use colima >/dev/null || true
284+
285+
# Confirm Docker is operational
286+
docker version
287+
docker ps
288+
displayName: 'Install and start Colima-based Docker'
289+
290+
- script: |
291+
# Pull and run SQL Server container
292+
docker pull mcr.microsoft.com/mssql/server:2022-latest
293+
docker run \
294+
--name sqlserver \
295+
-e ACCEPT_EULA=Y \
296+
-e MSSQL_SA_PASSWORD="${DB_PASSWORD}" \
297+
-p 1433:1433 \
298+
-d mcr.microsoft.com/mssql/server:2022-latest
299+
300+
# Starting SQL Server container…
301+
for i in {1..30}; do
302+
docker exec sqlserver \
303+
/opt/mssql-tools18/bin/sqlcmd \
304+
-S localhost \
305+
-U SA \
306+
-P "$DB_PASSWORD" \
307+
-C -Q "SELECT 1" && break
308+
sleep 2
309+
done
310+
displayName: 'Pull & start SQL Server (Docker)'
311+
env:
312+
DB_PASSWORD: $(DB_PASSWORD)
234313
314+
# Run Pytest to ensure the bindings work correctly
315+
- script: |
316+
python -m pytest -v
317+
displayName: 'Run Pytest to validate bindings'
318+
env:
319+
DB_CONNECTION_STRING: 'Driver=ODBC Driver 18 for SQL Server;Server=localhost;Database=master;Uid=SA;Pwd=$(DB_PASSWORD);TrustServerCertificate=yes'
320+
235321
# Build wheel package for universal2
236322
- script: |
237323
python -m pip install --upgrade pip

‎eng/pipelines/pr-validation-pipeline.yml‎

Lines changed: 41 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -14,11 +14,7 @@ jobs:
1414
steps:
1515
- task: UsePythonVersion@0
1616
inputs:
17-
# TODO: Remove this once Python 3.13 is available in ADO
18-
# ADO indexing will take some time to reflect 3.13 as 3.13.5, right now it is pointing to 3.13.4.
19-
# We're specifying to use Python 3.13.5 since 3.13.4 has issues with compilation
20-
# See https://github.com/python/cpython/issues/135151, next release should fix it.
21-
versionSpec: '3.13.5'
17+
versionSpec: '3.13'
2218
addToPath: true
2319
githubToken: $(GITHUB_TOKEN)
2420
displayName: 'Use Python 3.13'
@@ -88,7 +84,7 @@ jobs:
8884
steps:
8985
- task: UsePythonVersion@0
9086
inputs:
91-
versionSpec: '3.13.5'
87+
versionSpec: '3.13'
9288
addToPath: true
9389
displayName: 'Use Python 3.13 on macOS'
9490

@@ -97,44 +93,44 @@ jobs:
9793
brew install cmake
9894
displayName: 'Install CMake'
9995
100-
# - script: |
101-
# brew update
102-
# brew install docker colima
103-
104-
# # Start Colima with extra resources
105-
# colima start --cpu 4 --memory 8 --disk 50
106-
107-
# # Optional: set Docker context (usually automatic)
108-
# docker context use colima >/dev/null || true
109-
110-
# # Confirm Docker is operational
111-
# docker version
112-
# docker ps
113-
# displayName: 'Install and start Colima-based Docker'
114-
115-
# - script: |
116-
# # Pull and run SQL Server container
117-
# docker pull mcr.microsoft.com/mssql/server:2022-latest
118-
# docker run \
119-
# --name sqlserver \
120-
# -e ACCEPT_EULA=Y \
121-
# -e MSSQL_SA_PASSWORD="${DB_PASSWORD}" \
122-
# -p 1433:1433 \
123-
# -d mcr.microsoft.com/mssql/server:2022-latest
124-
125-
# # Starting SQL Server container…
126-
# for i in {1..30}; do
127-
# docker exec sqlserver \
128-
# /opt/mssql-tools18/bin/sqlcmd \
129-
# -S localhost \
130-
# -U SA \
131-
# -P "$DB_PASSWORD" \
132-
# -C -Q "SELECT 1" && break
133-
# sleep 2
134-
# done
135-
# displayName: 'Pull & start SQL Server (Docker)'
136-
# env:
137-
# DB_PASSWORD: $(DB_PASSWORD)
96+
- script: |
97+
brew update
98+
brew install docker colima
99+
100+
# Start Colima with extra resources
101+
colima start --cpu 4 --memory 8 --disk 50
102+
103+
# Optional: set Docker context (usually automatic)
104+
docker context use colima >/dev/null || true
105+
106+
# Confirm Docker is operational
107+
docker version
108+
docker ps
109+
displayName: 'Install and start Colima-based Docker'
110+
111+
- script: |
112+
# Pull and run SQL Server container
113+
docker pull mcr.microsoft.com/mssql/server:2022-latest
114+
docker run \
115+
--name sqlserver \
116+
-e ACCEPT_EULA=Y \
117+
-e MSSQL_SA_PASSWORD="${DB_PASSWORD}" \
118+
-p 1433:1433 \
119+
-d mcr.microsoft.com/mssql/server:2022-latest
120+
121+
# Starting SQL Server container…
122+
for i in {1..30}; do
123+
docker exec sqlserver \
124+
/opt/mssql-tools18/bin/sqlcmd \
125+
-S localhost \
126+
-U SA \
127+
-P "$DB_PASSWORD" \
128+
-C -Q "SELECT 1" && break
129+
sleep 2
130+
done
131+
displayName: 'Pull & start SQL Server (Docker)'
132+
env:
133+
DB_PASSWORD: $(DB_PASSWORD)
138134
139135
- script: |
140136
python -m pip install --upgrade pip
@@ -151,9 +147,7 @@ jobs:
151147
python -m pytest -v --junitxml=test-results.xml --cov=. --cov-report=xml --capture=tee-sys --cache-clear
152148
displayName: 'Run pytest with coverage'
153149
env:
154-
# Temporarily Use Azure SQL Database connection string for testing purposes since Docker takes too long to install & start in MacOS
155-
DB_CONNECTION_STRING: $(AZURE_CONNECTION_STRING)
156-
# DB_CONNECTION_STRING: 'Driver=ODBC Driver 18 for SQL Server;Server=localhost;Database=master;Uid=SA;Pwd=$(DB_PASSWORD);TrustServerCertificate=yes'
150+
DB_CONNECTION_STRING: 'Driver=ODBC Driver 18 for SQL Server;Server=localhost;Database=master;Uid=SA;Pwd=$(DB_PASSWORD);TrustServerCertificate=yes'
157151
DB_PASSWORD: $(DB_PASSWORD)
158152
159153
- task: PublishTestResults@2

0 commit comments

Comments
 (0)