Skip to content

Backends: Read autosplit_reserve for every exllamav3 split mode - #450

Open
sashko-zakharchuk wants to merge 1 commit into
theroyallab:mainfrom
sashko-zakharchuk:fix-405
Open

Backends: Read autosplit_reserve for every exllamav3 split mode#450
sashko-zakharchuk wants to merge 1 commit into
theroyallab:mainfrom
sashko-zakharchuk:fix-405

Conversation

@sashko-zakharchuk

Copy link
Copy Markdown
Contributor

Is your pull request related to a problem? Please describe.

Part of #405. autosplit_reserve only reaches exllamav3 on multi GPU autosplit, because the
parse sits inside elif gpu_split_auto and not self.use_tp:. One GPU, tensor parallel and
gpu_split_auto: false all keep the 96 MB __init__ default, and gpu_split nulls the list
outright, so every other configuration discards the value.

Arguments arriving at load_gen for autosplit_reserve: [1024, 1024], before and after (MB as
configured; the code divides by 1024 and passes GB):

config (multi GPU unless noted)     before                after
one GPU                             [96]                  [1024, 1024]
tensor_parallel: true               [96]                  [1024, 1024]
gpu_split_auto: false               [96]                  [1024, 1024]
autosplit, multi GPU                [1024, 1024]          [1024, 1024]
gpu_split: [24, 24]                 None, no warning      None, warns
draft_gpu_split without gpu_split   assert at load        loads

The gpu_split row stays None deliberately: exllamav3 asserts on
bool(reserve_per_device) and bool(use_per_device), so only one of the two can be sent. It now
says so instead of discarding the value with no warning, for draft_gpu_split as well, and only
when the value differs from the shipped default so a stock config stays quiet.

The last row is a separate bug: draft_gpu_split set without gpu_split sent both arguments for
the draft model and could not load at all.

The parse also now accepts a scalar, normalising 96 to [96], which is what the shipped default
already is. On main a scalar raises TypeError: 'int' object is not iterable on the autosplit
path. config.yml and the load API reject a scalar with a ValidationError, but a model folder's
tabby_config.yml is merged by apply_load_defaults without going through pydantic, so it does
reach the container from there.

Why should this feature be added?

The one reproduction attempt in #405 set gpu_split, which is tested ahead of the autosplit
branch and applies with or without tensor parallel, so it sent None and could not have shown
the reserve doing anything either way. That config sets tensor_parallel and gpu_split
together with a non-default reserve, so it is one of the configurations that now gets the
message instead of silence.

This PR does not close #405. The reporter's own reproduction steps are multi GPU autosplit, which is
the one row that already forwards the list; that symptom is exllamav3 side, where the reserve is a
load-time cap applied to a device only when the loader first places a module on it and lifted
again when loading finishes. The report also names tensor parallel; that path now receives the
configured value instead of the 96 MB default, though exllamav3 subtracts it there from each
device's free VRAM to size the split rather than holding it back.

Examples

The reserve parse moves ahead of the GPU count and split mode branching, so it is read once. The
three load_gen call sites then pass whichever of reserve_per_device / use_per_device
applies.

Additional context

What this does not change: with gpu_split set the reserve is still not sent, and under tensor
parallel exllamav3 subtracts the reserve from each device's free memory (model_tp.py:448) and
uses the result as a budget for a proportional split, raising every budget by 10% and retrying if
it runs out (model_tp_alloc.py:88), which never terminates if the budget is already non-positive.
So it does not bound allocation there the way it does on the autosplit path. Those are library side
and out of scope here.

Upgrade risks, all from the same cause, that a configured value now takes effect where it
was previously replaced:

  • A large reserve on one GPU or with gpu_split_auto: false now binds. Someone whose config
    loads today because their value was silently swapped for 96 MB can hit
    Insufficient VRAM in split for model and cache after this lands. That is the fix doing what it
    says, but it will look like a regression to whoever hits it.
  • Under tensor parallel a reserve at or above free VRAM on every device is worse than an error.
    exllamav3 subtracts the reserve from each device's free memory with no floor (model_tp.py:448)
    and feeds the result to TPAllocator.initial_split, whose retry loop raises an exhausted budget
    by 10% and tries again (model_tp_alloc.py:88); from a non-positive budget it never converges,
    so the load spins after the TP workers have spawned instead of raising. Upstream never reached
    this because TP always got the 96 MB default. A short list will not normally trigger it, since
    exllamav3 pads the missing entries with its own 0.5 GB, though an unlisted device with under
    0.5 GB free goes non-positive too. A fix for the loop is up as
    TP: Raise instead of looping forever when no device has a positive budget turboderp-org/exllamav3#288, and I can guard it on the tabbyAPI side as well if you would
    rather not rely on the library here.
  • A negative entry now reaches those paths too, and exllamav3 treats a negative reserve as
    "exclude this device" (model/model.py:390). On one GPU that leaves no devices and the load
    stops with an IndexError from the library, where before the value was ignored and it loaded.
    config.yml accepts negatives, so this is reachable. Negatives already worked this way on the
    multi GPU autosplit path but were documented nowhere on the tabbyAPI side, so the config
    description now states what they do. I stopped at documenting instead of filtering or rejecting
    them in code, and am happy to add either.

An empty list also changes: it now reaches exllamav3 on every path that forwards the list, where
it pads to that library's own 0.5 GB per device, so a single GPU reserves 512 MB where it
previously reserved 96 MB. It also differs from the shipped default, so with gpu_split set it
trips the new warning. Module placement did not change in my tests.

The table comes from running create() and load_model_sync() with exllamav3 stubbed out and
recording what reaches load_gen, for 1 and 4 GPU counts.

I also loaded real EXL3 quants on two and three GPUs on exllamav3 1.4.2, before and after, through
all three load_gen call sites. With gpu_split_auto: false and a reserve that binds on device 0,
current main puts all 18 modules there; with this change 16 of the 18 move to the second device.
The split model generates correctly afterwards, and tensor parallel loads and generates under
both the native and nccl backends. The vision path behaves the same way on a Qwen3-VL quant, 33 of
33 vision modules on device 0 before and 17/16 after. The draft path raises Cannot specify both memory usage and memory reserve on main and loads here. The autosplit and gpu_split paths place
modules identically before and after. The warnings fire only for gpu_split or draft_gpu_split
with a non-default reserve, and not on a stock config.

ruff check and ruff format clean, and the repo's unit tests pass. Harnesses available if
useful.

The autosplit_reserve parse sat inside the autosplit branch, so any other split
mode kept the 96 MB __init__ default and the configured value never reached
exllamav3. That covers single GPU, tensor parallel and gpu_split_auto: false.
Move the parse ahead of the GPU count and split mode branching so it is read
once, and pass it at the three load_gen call sites. A scalar is normalised to a
one element list there: config.yml and the load API reject a scalar, but a model
folder's tabby_config.yml is merged without pydantic, so it reaches the container
from there and would otherwise raise TypeError on every split mode once the parse
moves.

exllamav3 takes reserve_per_device or use_per_device and asserts on both, so a
model whose placement is already set by gpu_split or draft_gpu_split still gets
the split and not the reserve. That discard is now logged for either one, and
only when the value differs from the shipped default so a stock config stays
quiet.

Also fixes a draft model with draft_gpu_split and no gpu_split, which sent both
arguments and tripped that assert before loading.

Note that a configured reserve now takes effect where it was previously replaced
with the default, so a large value on one GPU or with gpu_split_auto: false can
turn a config that loads today into an Insufficient VRAM in split failure. Under
tensor parallel a reserve at or above free VRAM on every device is worse: the
exllamav3 allocator raises an exhausted budget 10% at a time and never converges
from a non-positive one, so the load spins rather than raising.

Document what a negative reserve does. exllamav3 uses it to exclude a device
from the model split, which already applied on the autosplit path but was
described nowhere in the tabbyAPI config.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] EXL3 reserve allocations ignored for device IDs > 0

1 participant