Repository navigation
Default auron.tokio.worker.threads.per.cpu to 1 to size per-task runtimes by task CPUs - #2587
Open
warrenzhu25 wants to merge 1 commit into
Open
warrenzhu25 wants to merge 1 commit into
warrenzhu25 wants to merge 1 commit into
Conversation
…imes by task CPUs Which issue does this PR close? Closes #<issue> (to be filed: per-task Tokio runtime defaults to all host cores, causing thread explosion and slower short tasks) Rationale for this change Each native task builds its own multi-threaded Tokio runtime (native-engine/auron/src/rt.rs) sized auron.tokio.worker.threads.per.cpu * spark.task.cpus. The current default of 0 leaves the worker count unset, so Tokio uses every host core for every task, regardless of spark.task.cpus. Each worker thread also attaches to the JVM on start to set the classloader and task context. When a task finishes, its runtime is torn down with shutdown_background(), which does not wait for the threads to exit. On many-core hosts with many short tasks, threads from finished tasks accumulate faster than they exit. That wastes CPU and memory, adds per-task thread start/stop overhead, and can exhaust the OS thread limit: - TPC-H SF10 Q4 with 3000 shuffle partitions on a 64-core host (local[16]): the JVM peaked at ~74,000 threads (auron-native-stage-*), and the user's total thread count reached 131,041 of a 131,072 limit. - One run failed with "OS can't spawn worker thread: Resource temporarily unavailable (os error 11)". Another hung: all final-stage tasks were blocked in JniBridge.nextBatch while native threads sat idle. What changes are included in this PR? - AuronConfiguration.TOKIO_WORKER_THREADS_PER_CPU: default 0 -> 1, so a task's runtime gets spark.task.cpus worker threads. The description now explains that 0 means "all host cores for every task". - rt.rs: the native fallbacks used when the conf can't be read become 1 instead of 0, consistent with the JVM-side defaults. Users who relied on the old behavior can set spark.auron.tokio.worker.threads.per.cpu=0. Are there any user-facing changes? Yes. The default number of Tokio worker threads per task changes from "all host cores" to spark.task.cpus. There are no API changes. How was this patch tested? 64-core host, Spark 3.5.8 / Scala 2.13, local[16], release build, using the dev/auron-it TPC-DS harness at SF10. Both configs used the same jar and differ only in this setting; runs were interleaved (old, new, old, new) and per-query medians compared. TPC-DS SF10, 101 queries per.cpu=0 (old) per.cpu=1 (new) total time 1010.7 s 919.6 s (-9%) total excluding q72 ~412 s ~308 s (-25%) median per-query ratio 0.72 geomean per-query ratio 0.663 queries >15% faster/slower 31 / 0 q72 (~600 s, dominated by spill I/O) is unchanged and makes up ~60% of the total. Same-config run-to-run geomean noise was 1-4%. Largest gains: q77 10.8 s -> 2.0 s, q76 2.2 s -> 0.6 s, q83 2.0 s -> 0.7 s, q64 11.9 s -> 6.7 s. Short queries gain most, because the saved cost is per task. Thread usage on TPC-H SF10 Q4 with 3000 shuffle partitions: ~74,000 JVM threads with the old default (crash and hang observed) vs a peak of 227 with the new default, and the query completes in 47 s. Correctness: a --result-check run with the new default matched vanilla Spark on 99/103 queries. The other four fail identically with the old default: q14a/q14b (existing Scala 2.13 ClassCastException, apache#2386) and q65/q71 (same rows, different order, because of ties in their ORDER BY). cargo fmt --check and spotless:check (auron-core) pass. Was this patch authored or co-authored using generative AI tooling? Yes. Generated-by: Claude Opus 5.5
warrenzhu25
force-pushed
the
auron-tokio-worker-threads-default
branch
from
October 19, 2026 02:54
a630a4c to
fcb60bb
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
auron.tokio.worker.threads.per.cpufrom 0 to 1, so each task's Tokio runtime is sized tospark.task.cpusinstead of all host CPU cores.unwrap_orfallbacks innative-engine/auron/src/rt.rswith the new defaults.Why
Each task creates its own Tokio runtime. With the default of 0, every task spawns one worker thread per host core, which creates a very large number of threads on many-core machines.
Testing
Not run; this changes a default value and its fallback only.