Repository navigation
Conversation
Identify dependency builds by their sources, patches, target, resolved MSVC toolchain, SDK, and build flags. Keep Python 3.11's OpenSSL uplink variant separate. For libffi, hash prepare_libffi.bat from the pinned CPython source and restrict unrecognized scripts to that source archive. Add commands to report archive paths or build a single dependency. Reuse a matching archive when present and publish a fresh archive atomically on a miss. Keep filenames short while matching against the full fingerprint.
Emit one producer for each OpenSSL or libffi configuration shared by multiple selected Windows builds. Group by target, runner, and toolchain. Keep the Python 3.11 OpenSSL uplink variant separate and group libffi builds by the expected hash of CPython's preparation script. Skip producers for dry runs and configurations with a single consumer. Preserve the default Windows build on unlabeled pull requests.
Run dependency producers for configurations shared by multiple Windows builds. Restore each archive using the fingerprint computed on the consumer's runner, and build locally when a producer or cache restore fails. Install Cygwin only when libffi must be built. Discard partial restores so an incomplete archive cannot be mistaken for a cache hit.
jjhelmus
added
platform:windows
Contributor
Author
|
Requiring the dependencies for all architectures and configurations to be built before starting any CPython build is not good. It delays initial results and limits parallelism. Working to break this block up into smaller tasks that can be depended on. |
Contributor
Author
|
Compared to a recent merge, this saved ~5.5 hours of CI time but requires ~9 additional minutes for the first complete CPython build (~x2 time to first complete build). |
The full Windows matrix makes every CPython job wait for all 14 OpenSSL and libffi builds. Split the matrix into target and toolchain groups and call a reusable workflow for each group.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Contributor
Author
|
By splitting dependency groups by architecture the CI time savings were similar but the first CPython build completed less than a minute after that in a reference run. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Build OpenSSL and libffi once for each configuration shared by multiple Windows CPython jobs.
Archives are used only when its full fingerprint matches their source, target, toolchain, SDK, and build inputs.
A missing producer or cache entry falls back to a local build.
Cygwin is installed only when libffi needs to be built.
The full release matrix has 60 Windows CPython jobs, requiring 60 OpenSSL and libffi builds. This change reduces this to 7 of each, avoiding 53 builds. Based on previous build time for these libraries, this change is estimated to save ~7.5 hours of CI time for a full build matrix. Importantly this should speed up the ARM64 Windows builds which often take the longest, although the time for the first complete build is not significantly effected.