Repository navigation
perf(ts_test): share npm binding sets by content and find hidden runfiles paths in one sorted pass - #386
Merged
Conversation
…iles paths in one sorted pass
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
In the monorepo, the
bazel-frontend-unitandbazel-workers-testslanes usually start with a cold Bazel server, and on a PR with no web changes the whole lane is analysis. About 70% of the profiled analysis time is this ruleset's Starlark, mostly_ts_test_impl. This PR removes the biggest pieces of repeated work there without changing any action.Binding sets shared by content.
runtime_npm_contextsinternednpm_binding_setsby"<owner> <lookup directory>", so every directory got its own copy even when it resolved to the same map.//web/shared:shared_testwrote 1,093 sets, of which 6 were distinct: 17.3 MB of its 19.4 MB launcher config. Each config went throughjson.encodeandctx.actions.write(which compresses large contents) during analysis. Sets are now also deduplicated by their sorted items. Across the 118 test programs in the two lanes, the written files shrink from 4.4 GB to 3.1 GB.Hidden runfiles paths in one sorted pass.
_runfiles_pathsand_declare_native_viewchecked every runfiles path against all of its ancestor prefixes, building each prefix with"/".join(parts[:depth]), which is quadratic in depth per path._beneath_anothersorts the paths by components, so each subtree directly follows its root, and marks a path hidden when it starts with the last root seen. The visible set and its insertion order are unchanged.One runfiles flattening per test.
ts_testcalledrunfiles_scope_pathstwice on the same runfiles, once in_ancestor_scope_linksand once for the module check. Each call flattened and filtered the whole runfiles tree. It now computesrunfiles_visible_pathsonce and passes it to both. It recomputes only when scopes were staged in between, because staging changes the runfiles.Cheaper path strings.
runfiles_scope_pathswalks each demanded path up withrpartitionand stops at the first ancestor already recorded.rlocation_pathreadsshort_pathonce and returns without building and strippingexternal/.runtime_scope_destinationscuts the trailing components withrpartitioninstead of splitting and re-joining the whole runtime path.Equivalence. I ran
bazel aquery --include_file_write_contents --include_commandline --include_artifactsoverdeps()of all 118 test programs inbazel-frontend-unitandbazel-workers-tests, with this repo as--override_moduleat the monorepo's pin (5d5fba8) with and without the change. I compared the dumps per action: 35,500 actions. With everybindings_setindex resolved to its map, every action, command line and written file is identical. Without the binding-set change the raw dumps are byte-identical.Measurements. I measured cold analysis (
bazel build --nobuild, afterbazel shutdown) of those 118 test programs on a shared 16-core workstation. I alternated pinned and patched runs to spread the noise, and recorded the Bazel server's total CPU from/proc. The patched run was faster in all six pairs.Profile by profile, Starlark CPU went from about 226 s to 189 s.
writeandjson.encodefell from 39 s to 18 s, and_runfiles_pathsfrom 32 s to 7 s cumulative.I haven't run this repo's own test suite locally; the workstation is too loaded for the nested-Bazel tests. CI here should cover the analysis tests that exercise
runfiles_scope_paths.//tests/compile_layout/nested_scope:scoped_test_npm_contexts_testpinned the oldnpm_binding_setsline, which listed the samezodmap three times. It now expects[{zod}, {}], with every context'sbindings_setat 0, the same resolved bindings.