Repository navigation
astrta medium: Separate validation progress from solver performance benchmarks - #6
Merged
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
astrta medium:
Validation wall time included an expensive SymPy generated-facts check that never entered either substituted backend. It also measured a changing workload: fixing an early assertion lets a test execute more queries. This change makes correctness progress and fixed-workload performance independently trackable, so optimization can proceed alongside support for more cases.
test_query::test_known_facts_consistentfrom routine backend comparisons and ASV progress runs, using normalized test identities.--include-controlsand plainpytest validationretain the full upstream suite; other slow tests remain selected.--output, uploaded by CI. Version the pass-count series for the 104-to-103 population change and rename the wall-time series to validation execution cost.Validation:
No solver implementation or upstream test bodies change. Optional backend-only timing instrumentation is unnecessary for the fixed-workload performance track and is not added.
Implements the assessment and requested changes.
Closes tilorc-bot#1