Conversation
The TPC-H and TPC-DS dataset caches were keyed on the hash of pr_build_linux.yml, so every workflow edit regenerated both datasets and left useless entries that evict main's. The generators were also unpinned: GenTPCHData cloned tpch-dbgen HEAD and the tpcds-kit checkout had no ref, so a generator change could not rotate a dataset key. Pin both generators and key the datasets on their inputs instead: - TPC-H key is the hash of GenTPCHData.scala; the generator now checks out a fixed tpch-dbgen commit before applying the stdout patch. - TPC-DS key embeds the pinned tpcds-kit commit, which the checkout now names explicitly. Save both from main only, matching the cache-write policy for other large caches. check_tpc_dataset_caches() in dev/ci/check-ci-config.py asserts all of the above so the old shape cannot come back.
github-actions
Bot
added
build
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #6102.
The TPC-H and TPC-DS dataset caches were keyed on
hashFiles('.github/workflows/pr_build_linux.yml'), so any workflow editregenerated both datasets and left entries that a later run can never restore.
The generators were also unpinned:
GenTPCHDataclonedtpch-dbgenHEAD andthe
tpcds-kitcheckout had noref:.This pins both generators and keys the datasets on their inputs:
GenTPCHData.scala; the generator checks out a fixedtpch-dbgencommit before applying the stdout patch.tpcds-kitcommit, which the checkout namesexplicitly.
actions/cache/restoreplus a save step guarded tomain, matching the cache-write policy used for other large caches.check_tpc_dataset_caches()indev/ci/check-ci-config.pyasserts the keyshape, the pinned
ref:, and that the TPC caches are no longer bareactions/cache@vN.Testing: the new guard fails on the previous revision with five findings and
passes after the change (
CI config checks passed);dev/ci/check-suites.pyalso passes.