Backends: Read autosplit_reserve for every exllamav3 split mode - #450
Open
sashko-zakharchuk wants to merge 1 commit into
Open
Backends: Read autosplit_reserve for every exllamav3 split mode#450sashko-zakharchuk wants to merge 1 commit into
sashko-zakharchuk wants to merge 1 commit into
Conversation
The autosplit_reserve parse sat inside the autosplit branch, so any other split mode kept the 96 MB __init__ default and the configured value never reached exllamav3. That covers single GPU, tensor parallel and gpu_split_auto: false. Move the parse ahead of the GPU count and split mode branching so it is read once, and pass it at the three load_gen call sites. A scalar is normalised to a one element list there: config.yml and the load API reject a scalar, but a model folder's tabby_config.yml is merged without pydantic, so it reaches the container from there and would otherwise raise TypeError on every split mode once the parse moves. exllamav3 takes reserve_per_device or use_per_device and asserts on both, so a model whose placement is already set by gpu_split or draft_gpu_split still gets the split and not the reserve. That discard is now logged for either one, and only when the value differs from the shipped default so a stock config stays quiet. Also fixes a draft model with draft_gpu_split and no gpu_split, which sent both arguments and tripped that assert before loading. Note that a configured reserve now takes effect where it was previously replaced with the default, so a large value on one GPU or with gpu_split_auto: false can turn a config that loads today into an Insufficient VRAM in split failure. Under tensor parallel a reserve at or above free VRAM on every device is worse: the exllamav3 allocator raises an exhausted budget 10% at a time and never converges from a non-positive one, so the load spins rather than raising. Document what a negative reserve does. exllamav3 uses it to exclude a device from the model split, which already applied on the autosplit path but was described nowhere in the tabbyAPI config.
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Is your pull request related to a problem? Please describe.
Part of #405.
autosplit_reserveonly reaches exllamav3 on multi GPU autosplit, because theparse sits inside
elif gpu_split_auto and not self.use_tp:. One GPU, tensor parallel andgpu_split_auto: falseall keep the 96 MB__init__default, andgpu_splitnulls the listoutright, so every other configuration discards the value.
Arguments arriving at
load_genforautosplit_reserve: [1024, 1024], before and after (MB asconfigured; the code divides by 1024 and passes GB):
The
gpu_splitrow staysNonedeliberately: exllamav3 asserts onbool(reserve_per_device) and bool(use_per_device), so only one of the two can be sent. It nowsays so instead of discarding the value with no warning, for
draft_gpu_splitas well, and onlywhen the value differs from the shipped default so a stock config stays quiet.
The last row is a separate bug:
draft_gpu_splitset withoutgpu_splitsent both arguments forthe draft model and could not load at all.
The parse also now accepts a scalar, normalising
96to[96], which is what the shipped defaultalready is. On main a scalar raises
TypeError: 'int' object is not iterableon the autosplitpath.
config.ymland the load API reject a scalar with aValidationError, but a model folder'stabby_config.ymlis merged byapply_load_defaultswithout going through pydantic, so it doesreach the container from there.
Why should this feature be added?
The one reproduction attempt in #405 set
gpu_split, which is tested ahead of the autosplitbranch and applies with or without tensor parallel, so it sent
Noneand could not have shownthe reserve doing anything either way. That config sets
tensor_parallelandgpu_splittogether with a non-default reserve, so it is one of the configurations that now gets the
message instead of silence.
This PR does not close #405. The reporter's own reproduction steps are multi GPU autosplit, which is
the one row that already forwards the list; that symptom is exllamav3 side, where the reserve is a
load-time cap applied to a device only when the loader first places a module on it and lifted
again when loading finishes. The report also names tensor parallel; that path now receives the
configured value instead of the 96 MB default, though exllamav3 subtracts it there from each
device's free VRAM to size the split rather than holding it back.
Examples
The reserve parse moves ahead of the GPU count and split mode branching, so it is read once. The
three
load_gencall sites then pass whichever ofreserve_per_device/use_per_deviceapplies.
Additional context
What this does not change: with
gpu_splitset the reserve is still not sent, and under tensorparallel exllamav3 subtracts the reserve from each device's free memory (
model_tp.py:448) anduses the result as a budget for a proportional split, raising every budget by 10% and retrying if
it runs out (
model_tp_alloc.py:88), which never terminates if the budget is already non-positive.So it does not bound allocation there the way it does on the autosplit path. Those are library side
and out of scope here.
Upgrade risks, all from the same cause, that a configured value now takes effect where it
was previously replaced:
gpu_split_auto: falsenow binds. Someone whose configloads today because their value was silently swapped for 96 MB can hit
Insufficient VRAM in split for model and cacheafter this lands. That is the fix doing what itsays, but it will look like a regression to whoever hits it.
exllamav3 subtracts the reserve from each device's free memory with no floor (
model_tp.py:448)and feeds the result to
TPAllocator.initial_split, whose retry loop raises an exhausted budgetby 10% and tries again (
model_tp_alloc.py:88); from a non-positive budget it never converges,so the load spins after the TP workers have spawned instead of raising. Upstream never reached
this because TP always got the 96 MB default. A short list will not normally trigger it, since
exllamav3 pads the missing entries with its own 0.5 GB, though an unlisted device with under
0.5 GB free goes non-positive too. A fix for the loop is up as
TP: Raise instead of looping forever when no device has a positive budget turboderp-org/exllamav3#288, and I can guard it on the tabbyAPI side as well if you would
rather not rely on the library here.
"exclude this device" (
model/model.py:390). On one GPU that leaves no devices and the loadstops with an
IndexErrorfrom the library, where before the value was ignored and it loaded.config.ymlaccepts negatives, so this is reachable. Negatives already worked this way on themulti GPU autosplit path but were documented nowhere on the tabbyAPI side, so the config
description now states what they do. I stopped at documenting instead of filtering or rejecting
them in code, and am happy to add either.
An empty list also changes: it now reaches exllamav3 on every path that forwards the list, where
it pads to that library's own 0.5 GB per device, so a single GPU reserves 512 MB where it
previously reserved 96 MB. It also differs from the shipped default, so with
gpu_splitset ittrips the new warning. Module placement did not change in my tests.
The table comes from running
create()andload_model_sync()with exllamav3 stubbed out andrecording what reaches
load_gen, for 1 and 4 GPU counts.I also loaded real EXL3 quants on two and three GPUs on exllamav3 1.4.2, before and after, through
all three
load_gencall sites. Withgpu_split_auto: falseand a reserve that binds on device 0,current main puts all 18 modules there; with this change 16 of the 18 move to the second device.
The split model generates correctly afterwards, and tensor parallel loads and generates under
both the native and nccl backends. The vision path behaves the same way on a Qwen3-VL quant, 33 of
33 vision modules on device 0 before and 17/16 after. The draft path raises
Cannot specify both memory usage and memory reserveon main and loads here. The autosplit andgpu_splitpaths placemodules identically before and after. The warnings fire only for
gpu_splitordraft_gpu_splitwith a non-default reserve, and not on a stock config.
ruff checkandruff formatclean, and the repo's unit tests pass. Harnesses available ifuseful.