Skip to content

MAINT: Batch integration tests #2433

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jbolor21 wants to merge 1 commit into
base: main
Choose a base branch
from
+31 −3
Open

MAINT: Batch integration tests #2433

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions .azuredevops/integration-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,17 @@ jobs:
newDir: integration_test_directory
testsFolder: integration
makeTarget: integration-test
# Run each group in a fresh pytest process to release memory between groups.
testBatches:
- tests/integration/ai_recruiter
- tests/integration/cli
- tests/integration/converter
- tests/integration/datasets
- tests/integration/embeddings
- tests/integration/executor
- tests/integration/executors
- tests/integration/memory
- tests/integration/message_normalizer
- tests/integration/scenarios
- tests/integration/score
- tests/integration/targets
17 changes: 15 additions & 2 deletions .azuredevops/test-job-template.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,9 @@ parameters:
type: string
- name: makeTarget
type: string
- name: testBatches
type: object
default: []

jobs:
- job: ${{ parameters.jobName }}
Expand Down Expand Up @@ -114,7 +117,16 @@ jobs:
echo "Azure SQL Database access token cached successfully."

# Run tests
make ${{ parameters.makeTarget }}
test_batches="${{ join(' ', parameters.testBatches) }}"
if [ -n "$test_batches" ]; then
batch_number=0
for test_batch in $test_batches; do
batch_number=$((batch_number + 1))
make ${{ parameters.makeTarget }} INTEGRATION_TESTS="$test_batch" JUNIT_XML="junit/test-results-${batch_number}.xml"

@rlundeen2 Richard Lundeen (rlundeen2) Aug 19, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rich agrees with this comment but it is copilot generated:

This loop returns only the final make command's exit status. If an earlier batch fails and a later batch passes, the Bash step succeeds, and PublishTestResults@2 does not fail the job by default. Please stop on the first failure with set -e, or collect the statuses and exit nonzero after all batches run.

done
else
make ${{ parameters.makeTarget }}
fi
- bash: |
rm -f ~/.pyrit/.env ~/.pyrit/.env.local
name: clean_up_env_files
Expand All @@ -123,4 +135,5 @@ jobs:
condition: always()
inputs:
testResultsFormat: 'JUnit'
testResultsFiles: 'junit/test-results.xml'
testResultsFiles: 'junit/test-results*.xml'
mergeTestResults: true
3 changes: 2 additions & 1 deletion Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ UNIT_TESTS:=tests/unit
INTEGRATION_TESTS:=tests/integration
PARTNER_INTEGRATION_TESTS:=tests/partner_integration
END_TO_END_TESTS:=tests/end_to_end
JUNIT_XML?=junit/test-results.xml

all: pre-commit

Expand Down Expand Up @@ -67,7 +68,7 @@ unit-test-diff-cover:
uv run python -m diff_cover.diff_cover_tool coverage.xml --compare-branch=origin/main --diff-range-notation=.. --fail-under=90

integration-test:
$(CMD) pytest $(INTEGRATION_TESTS) --cov=$(PYMODULE) $(INTEGRATION_TESTS) --cov-report xml --junitxml=junit/test-results.xml --doctest-modules
$(CMD) pytest $(INTEGRATION_TESTS) --cov=$(PYMODULE) --cov-report xml --junitxml=$(JUNIT_XML) --doctest-modules

end-to-end-test:
$(CMD) pytest $(END_TO_END_TESTS) -v --junitxml=junit/test-results.xml
Expand Down
Loading