Repository navigation
Enabled KV, QKV, HQKV, BHQKV, KV_HEADPAR, KV_BATCH_FOLD* blocking support for weightfree in causallm models - #1373
Closed
Anusha V.S Bhamidipati (quic-abhamidi) wants to merge 49 commits into
Conversation
In this PR, we are adding the support for DFlash (Block Diffusion for Flash Speculative Decoding) approach that significantly accelerate LLM inference on Qualcomm platforms. DFlash replaces autoregressive draft models with a diffusion-based block drafter that predicts an entire block of tokens in a single parallel pass, resulting in constant drafting latency independent of block size. # Key benefits: 5–6× lossless speedup over baseline decoding ~2.5× faster than EAGLE-3 Higher acceptance length enabled by KV injection of target hidden states into every draft layer Improved accelerator utilization and reduced end-to-end latency Verification is performed by the frozen target model, guaranteeing output identical to standard decoding. # Use-case(s) High-throughput and low-latency LLM serving Long chain-of-thought reasoning workloads (e.g., math and planning) Code generation and agentic inference loops Platform differentiation via advanced speculative decoding on AI100 #Requirements DFlash speculative decoding with block diffusion drafting and target verification Extraction and fusion of hidden states from multiple target layers during prefill KV cache injection of fused target context into all draft layers Integration with QEfficient, vLLM, and other applications Support current architecture: Target: single QPC for prefill + decode (expandable to DA serving) Draft: lightweight QPC --------- Signed-off-by: Vahid Janfaza <vjanfaza@qti.qualcomm.com> Signed-off-by: FNU Annanya <fannanya@qti.qualcomm.com> Co-authored-by: FNU Annanya <fannanya@qti.qualcomm.com>
PyTorch loses original FX source metadata when Dynamo retraces GraphModule subfunctions through invoke_subgraph. This causes ONNX function nodes to reference generated tracing locations instead of the original QEff model files and source lines. Add a process-scoped monkey patch in torch_patches that retraces **GraphModule** subfunctions through torch.fx.Interpreter under preserve_node_meta(). The original PyTorch function is restored after export, and concurrent exports are serialized because the patched API is process global. Apply the workaround to both regular Dynamo export and weight-free Dynamo export paths. **Validated with:** - tests/dynamo/test_transforms.py: 13 passed - Normal Dynamo tiny Qwen2 export, compile, and inference - Weight-free Dynamo tiny Qwen2 export, compile, and inference - ONNX function metadata containing original QEff source locations Signed-off-by: vbaddi <vbaddi@qti.qualcomm.com>
Signed-off-by: Mohit Soni <mohisoni@qti.qualcomm.com> Co-authored-by: Mohit Soni <mohisoni@qti.qualcomm.com>
Updated numpy version from numpy==1.26.4 -> numpy>=2.2.6,<3, and protobuf==6.31.0 --> protobuf==6.33.6 Also deprecated python 3.8 support as NumPy 2.2.6 requires Python ≥3.10 --------- Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
Remove all five Kimi K2.5 QAIC xfail. Stabilize Kimi K2.5 test initialization and logits parity. Force full CI while disabling LLM impact-test selection. --------- Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
Issue: Decode Compilation Fails with Missing ONNX External Data File Error, current behavior picks qpc_key = "lang_qpc_path" in place of "lang_decode_qpc_path" under continuous batching, which breaks vllm_qaic - To fix name mismatch between vllm and qeff, removed continuous_batching condition check from modeling auto.py - Change prefill seq len from 1 to 32 in kimi CB tests --------- Signed-off-by: Mamta Singh <mamtsing@qti.qualcomm.com>
Add `vision_batch_size` to the dual-QPC compile API and VLM specialization builders. Use `vision_batch_size` for vision specs and dynamic axes while retaining `batch_size/full_batch_size` for language prefill and decode specs. When omitted, `vision_batch_size` defaults to `batch_size` for compatibility. --------- Signed-off-by: vbaddi <vbaddi@qti.qualcomm.com> Signed-off-by: vtirumal <vtirumal@qti.qualcomm.com> Co-authored-by: vtirumal <vtirumal@qti.qualcomm.com>
Made two fixes to performance, relevant to Qwen3.6 but applicable to all models that use the generic blocked interface: - KV head parallel softmax decode - mask creation split to be partially outside of kv blocking loop - Online prefill - correctly uses num_cores instead of always defaulting to num kv groups --------- Signed-off-by: Kushal Dulla <kdulla@qti.qualcomm.com>
## Summary - Add a pre-merge reported reproducer config suite under `tests/reproducer_configs/`. - Capture reported import/download/export/compile/inference cases as explicit `RegressionScenario` entries. - Run the real QEfficient stage for active scenarios instead of placeholder option checks. - Use `QEFF_REPRODUCER_SCENARIO` for targeted local runs. - Add `tests/reproducer_configs/README.md` with the common run commands. ## Notes - Compile-stage scenarios either run as reduced tests or stay gated behind official-model/full-model settings. - MDP scenarios generate local compiler dump/intersection artifacts instead of relying on reporter-machine paths. - Compile kwargs are checked before long compile jobs start. - The runtime report records the Hugging Face cache selected for the run. - The reduced run still emits an existing non-fatal qconfig warning: `Object of type PosixPath is not JSON serializable`; QPC `programqpc.bin` artifacts were still produced and validated. ## Validation - Ran lint/format checks for the reproducer test file. - Checked full test collection for the reproducer suite. - Checked targeted collection with `QEFF_REPRODUCER_SCENARIO=hf-transfer-import-order`. - Ran catalog guard tests for scenario count, uniqueness, and active scenario coverage. - Earlier full reproducer-suite run passed with reduced scenarios: `18 passed, 34 skipped`. - Earlier model quickcheck gate passed: `185 passed, 3 skipped`. --------- Signed-off-by: Anuj Gupta <anujgupt@qti.qualcomm.com> Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com> Co-authored-by: Rishin Raj <rishinr@qti.qualcomm.com>
Added a pre-compile MDP sanitization step in mdp_generator.py to remove alias-prone node ordering conflicts before compile. This specifically targets cases with: - use_onnx_subfunctions=True - blocking enabled In this mode, compiler can create dotted aliases (for example /node.), which can trigger partition-order failures (consumer before producer). The fix prunes conflicting top-level nodes in MDP before compile. Verified on models: - qwen3_5_moe - gemma4 - qwen3_moe --------- Signed-off-by: Mohit Soni <mohisoni@qti.qualcomm.com> Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com> Co-authored-by: Mohit Soni <mohisoni@qti.qualcomm.com> Co-authored-by: Rishin Raj <rishinr@qti.qualcomm.com>
Remove all QNN hardware tests from the test suites. --------- Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
This PR is a clone of qualcomm#982. This PR adds the PagedAttention (https://arxiv.org/pdf/2309.06180) support for all CausalLM models in QEfficient. The major change is that KV cache is not treated as a contiguous memory under this implementation but rather a collection of blocks which can reside in a non-contiguous fashion inside the memory. This forces cache scatter and gather operations to happen per KV block. ### Summary of changes compared to BlockedKV: 1. The cache shape changes from [BS, num_kv_heads, CL, dh] to [total_num_kv_blocks, num_kv_heads, kv_block_size, dh]. 2. num_kv_blocks = -(-ctx_len // kv_block_size) = physical blocks required for 1 batch element in K cache. 3. Total_num_kv_blocks = BS (<kv_batch_size>) * num_kv_blocks = total physical blocks available for K cache. 4. 2 new inputs **block_table [BS, num_kv_blocks]** and **slot_id [BS]** are passed as inputs to the ONNX. 4) a) block_id is each entry in the block_table and points to the physical K/V block that needs to be read/written corresponding to (position_id // kv_block_size)th entry in block_table. ‘-1’ signifies invalid/unallocated block. 4) b) slot_id denotes how many entries are already filled in currently active block => read up to / write after (slot_id – 1) 5. Limitation - Cache writes to only 1 block at a time per batch element => CPL is less than or equal to kv_block_size. Hence, cache writes should not cross the block boundary. 6. vLLM provides KV Cache Manager implementation which maintains the KV cache block_table with logical to physical block mapping and slot_id for location mapping within the active block. --------- Signed-off-by: Vaibhav Verma <vaibverm@qti.qualcomm.com> Signed-off-by: Shivani Athavale <athavale@qti.qualcomm.com> Signed-off-by: Aditya Dhananjay Jadhav <aditjadh@qti.qualcomm.com> Co-authored-by: Shivani Athavale <athavale@qti.qualcomm.com> Co-authored-by: Aditya Dhananjay Jadhav <aditjadh@qti.qualcomm.com> Co-authored-by: Rishin Raj <rishinr@qti.qualcomm.com>
## Summary - Remove disaggregation tests from the causal-model stage coverage. - Populate the Disagg column from the dedicated disaggregation stage. - Include disagg coverage for both causal and VLM models. - Add the QAIC DISAGG stage to the stage summary. - Add the missing disagg_dma marker to stage detection. Generated `report.html` using artifacts from [QEfficient Preflight Test qualcomm#768](https://re-qranium-ci.qualcomm.com/job/Execute_Test/job/Apps/job/Qefficient-Preflight-Test/768/) and attached it to this PR for review. [ci_report.html](https://github.com/user-attachments/files/32435928/ci_report.html) --------- Signed-off-by: Vishali Senthilkumar <vishsent@qti.qualcomm.com> Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com> Co-authored-by: Rishin Raj <rishinr@qti.qualcomm.com>
Fixes silent image corruption when a VLM is loaded with `bfloat16` dtype. Code cast `pixel_values/image_masks` numerically to float16 instead of bit-reinterpreting them, producing garbage on real-cloat16-compiled compiled vision bindings. This PR adds QAICInferenceSession.binding_is_bfloat16() and switches all affected vision-input casting sites to bit cast when the biding is bfloat16, leaving float16 bindings untouched. Changes verified on AI200 Hardware for `Qwen3.6-27B` and `Qwen2.5-VL-32B` model. --------- Signed-off-by: Asmita Goswami <asmigosw@qti.qualcomm.com> Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com> Co-authored-by: Rishin Raj <rishinr@qti.qualcomm.com>
Fix dual-QPC VLM sampler export to read vocab_size from the language decoder configuration. This supports nested configurations such as Qwen2.5-VL. --------- Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com>
…m#1271) ## Summary Adds `artifacts=True` support for QEfficient CausalLM and ImageTextToText compile/generate flows. This mode prepares the files needed by external harnesses, such as perf-compare, without invoking `qaic-compile` or running inference. It keeps the normal compile/generate behavior unchanged when `artifacts` is not set. ## What This Produces For compile: ```text qpc-<hash>/ qaic-compile.sh specializations.json custom_io.yaml hashed_compile_params.json mdp*.json ``` For generate: ```text qpc-<hash>/ io/ aic_batch_io.json data/*.raw ``` The generated qaic-compile.sh can be run later by an external harness. The qpc/ directory is not created during artifacts mode; it is created only when the replay script is executed. ## API ### CausalLM ```python qpc_path = model.compile( tokenizer=tokenizer, batch_size=1, full_batch_size=4, ctx_len=128, prefill_seq_len=32, mxfp6=False, mxint8=False, artifacts=True, ) io_path = model.generate( tokenizer=tokenizer, prompts=["Hello"], device_group=[0, 1, 2, 3], artifacts=True, ) ``` ### ImageTextToText ```python model.compile( processor=processor, batch_size=1, ctx_len=1024, prefill_seq_len=128, artifacts=True, ) vision_io = model.generate( processor=processor, messages=messages, skip_lang=True, artifacts=True, ) language_io = model.generate( processor=processor, messages=messages, skip_vision=True, artifacts=True, ) ``` For dual-QPC models, generate one runner input bundle per component. If compile was called with `skip_lang=True` or `skip_vision=True`, pass the same component selection to generate(). ## Backward Compatibility Existing compile and generate paths are unchanged unless artifacts=True is passed. The feature is currently enabled for: - QEffAutoModelForCausalLM - QEffAutoModelForImageTextToText Other Auto classes are not changed in this PR. --------- Signed-off-by: vbaddi <vbaddi@qti.qualcomm.com> Signed-off-by: Mamta Singh <mamtsing@qti.qualcomm.com> Co-authored-by: Mamta Singh <mamtsing@qti.qualcomm.com>
configurable GDN chunk size during export --------- Signed-off-by: vbaddi <vbaddi@qti.qualcomm.com> Signed-off-by: Mohit Soni <mohisoni@qti.qualcomm.com> Co-authored-by: Mohit Soni <mohisoni@qti.qualcomm.com>
QEFFAutoModelForCausalLM.generate() accepted `device_id` (singular) while all other model classes (QEFFAutoModel, QEFFAutoModelForImageTextToText, QEFFAutoModelForSpeechSeq2Seq, QEFFAutoModelForCTC, QAICInferenceSession) already used `device_ids` (plural). All accept List[int], so the singular name was misleading and required callers to look up the correct kwarg per class. Rename `device_id` → `device_ids` uniformly in: - QEFFAutoModelForCausalLM.generate() signature + docstring - cloud_ai_100_exec_kv() / latency_stats_bertstyle() in text_generation_inference.py - TextGeneration / TextGenerationMultiQPC constructors - VisionLanguageGeneration.__init__() + self.device_ids attribute - QEFFLoRAForCausalLM.generate() - cloud/infer.py, cloud/execute.py, utils/run_utils.py call sites - examples: basic_inference, moe_inference, continuous_batching, multi_adapter, on_device_sampling, text_inference_cpp device_utils.get_available_device_id() is a different function (returns a single free device ID for probing); it is intentionally left unchanged. --------- Signed-off-by: shagsood <shagsood@qti.qualcomm.com>
Signed-off-by: Zack McKevitt <zmckevit@qti.qualcomm.com>
qualcomm#1327) Threads the runtime `ProfilingHandle` device-profiling support in `QAICInferenceSession` through `cloud_ai_100_exec_kv/TextGeneration/QEffTextGenerationBase`, exposing `profiling_type/profiling_output_dir` end-to-end via `model.generate()` and as `--profiling-type`/`--profiling-output-dir` flags in `examples/text_generation/basic_inference.py`. profiling_type takes "stats", "latency", "trace", and "raw_device_stats". --------- Signed-off-by: Asmita Goswami <asmigosw@qti.qualcomm.com> Signed-off-by: Rishin Raj <rishinr@qti.qualcomm.com> Co-authored-by: Rishin Raj <rishinr@qti.qualcomm.com>
…qualcomm#1272) ## Summary - Automatically selects a representative language-layer prefix when `enable_proxy=True`. - Ensures every distinct attention and dense/MoE layer signature occurs at least twice. - Supports causal LM, MoE, VLM, and mixed linear/full-attention configurations. - Keeps VLM vision towers intact while proxying language embeddings and LM heads. - Preserves existing behavior when proxy mode is disabled. - Makes four-layer GPT-OSS export as two decoder subfunctions with two calls each. ## Usage ```python model = QEFFAutoModelForCausalLM.from_pretrained( model_id, enable_proxy=True, ) model = QEFFAutoModelForImageTextToText.from_pretrained( model_id, enable_proxy=True, ) ``` No manual num_hidden_layers override is required. ## Validation - Proxy and VLM focused tests: 8 passed - Blocking transform tests: 17 passed - GPT-OSS regression: two decoder functions, two calls each --------- Signed-off-by: vbaddi <vbaddi@qti.qualcomm.com> Co-authored-by: Rishin Raj <rishinr@qti.qualcomm.com> Co-authored-by: Mamta Singh <mamtsing@qti.qualcomm.com>
read_weight_map() (checkpoint_utils.py):
- Remove shard-scan fallback that opened every shard file just to
reconstruct an index already available in model.safetensors.index.json.
- Single-file model (exactly 1 shard, no index): open once and return.
- Zero safetensors: FileNotFoundError with clear message.
- Multiple shards, no index: FileNotFoundError identifying malformed
checkpoint — callers no longer silently scan on broken inputs.
CheckpointTransformPipeline.apply() (base/checkpoint_transforms.py):
- Drop silent .bin → safetensors rewrite which doubled I/O and disk
usage with no user visibility.
- Raise ValueError immediately when .bin files are present so callers
know exactly what to fix.
- Remove now-unused convert_bin_to_safetensors import.
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
…(TODO-6)
base/checkpoint_transforms.py:
- Add detect_group_transform_id(config, weight_map, hash_params) -> str.
Uses config.num_experts as the MoE gate and inspects weight_map key
patterns to identify the checkpoint layout without opening any shard
files. Returns stable TRANSFORM_ID strings that will map 1:1 to the
group transform classes introduced in the pipeline redesign.
- Simplify _checkpoint_manifest(): remove original_source field (source_dir
always equals src after Stage ①), add active_group field, bump version to 2.
- Simplify _checkpoint_file_fingerprint(): drop label parameter.
- Simplify _clear_stale_prepared_dir(): drop source_dir parameter.
- Read weight_map once at the top of apply() so detection and transform
selection share the same dict.
export.py:
- Add _prepared_checkpoint_hash(model_ref, dtype, active_group_id,
moe_flavour) -> 12-char sha256. Different model flavours (dense vs MoE,
decode vs expert_parallel, different quantizations) now hash to different
prepared directories and never silently overwrite each other.
- _prepare_checkpoint_for_weight_free_export() detects the active group
transform upfront, computes the hash, and uses it in prepared_name instead
of the plain dtype suffix.
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
…DO-5)
- Pipeline passes weight_map to transform.apply() so transforms never
re-read the index themselves.
- MoEExpertStackingCheckpointTransform: replace Phase 1 ThreadPoolExecutor
shard scan with O(N) dict iteration on the passed weight_map.
- GptOssMxfp4ExpertDequantSplitCheckpointTransform: same.
- MoEFusedExpertSplitCheckpointTransform: replace inline index re-read
duplication (TODO-5) with read_weight_map() fallback.
- GraniteMoeFusedExpertSplitCheckpointTransform: same pattern.
All four transforms accept weight_map as an explicit Optional parameter
and fall back to read_weight_map(src) only when called without it.
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
…n (TODO-1)
detect_group_transform() returns the active layout transform class directly
(not a string ID), eliminating the is_applicable() selector loop.
Pipeline execute logic:
transforms_to_run = [active_layout_transform, DtypeConversionCheckpointTransform]
for each: apply(remaining_keys), collect result, shrink consumed set
Each layout transform (MoEExpertStacking, GptOssMxfp4Dequant,
MoEFusedExpertSplit) gains:
+ TRANSFORM_ID: stable cache hash identifier
+ get_consumed_keys(): declares which keys it owns
- Phase 3 removed: base key dtype conversion deleted from layout transforms
- sentinel.touch(), write_index(), copy_checkpoint_aux_files() removed
- apply() now returns Dict[str, str] instead of bool
DtypeConversionCheckpointTransform gains TRANSFORM_ID + get_consumed_keys()
and always runs on remaining keys after any layout transform.
Stage ⑤ (finalise): sidecar copy, write_index, manifest, sentinel now
handled once in the pipeline after all transforms complete.
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
…roach
Replaces MoEFusedExpertSplitCheckpointTransform and the now-deleted
GraniteMoeFusedExpertSplitCheckpointTransform with one class:
FusedExpertSplitCheckpointTransform:
_get_key_remap(weight_map): detects architecture from index.json keys.
input_linear.weight present → GraniteMoE remap
otherwise → Mixtral fused (already canonical, no remap)
build_canonical_maps(weight_map, remap) produces:
canonical_index: {canonical_key → shard_file} WHERE to find tensor
key_translation: {canonical_key → actual_key} WHAT to ask shard for
Single pass — loads by actual key, writes by canonical output key.
No shape reads — split dim determined from bias key presence in canonical_index.
_checkpoint_transforms now has 4 entries; no architecture-specific subclasses.
detect_group_transform() delegates pre-stacked detection to is_applicable().
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
Each layout transform now declares resolve_onnx_key(onnx_key, index):
MoEExpertStackingCheckpointTransform: direct + .mlp.→.block_sparse_moe.
FusedExpertSplitCheckpointTransform: direct + .mlp.→.block_sparse_moe.
GptOssMxfp4ExpertDequantSplit: direct match only
find_checkpoint_key() updated:
1. Universal HF prefix rules (unchanged)
2. active_transform.resolve_onnx_key() — explicit per-architecture
3. Legacy MoE alias fallback (kept for old checkpoints)
Hardcoded .mlp./.gate.weight/.router.weight rules removed from resolver.
promote_initializers_and_build_spec() detects active_transform from
qeff_model and passes it to find_checkpoint_key().
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
Each layout transform now declares resolve_onnx_key(onnx_key, index):
MoEExpertStackingCheckpointTransform: direct + .mlp.→.block_sparse_moe.
FusedExpertSplitCheckpointTransform: direct + .mlp.→.block_sparse_moe.
GptOssMxfp4ExpertDequantSplit: direct match only
find_checkpoint_key() updated:
1. Universal HF prefix rules (unchanged)
2. active_transform.resolve_onnx_key() — explicit per-architecture
3. Legacy MoE alias fallback (kept for old checkpoints)
Hardcoded .mlp./.gate.weight/.router.weight rules removed from resolver.
promote_initializers_and_build_spec() detects active_transform from
qeff_model and passes it to find_checkpoint_key().
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
Bug 1: Old MoEFusedExpertSplitCheckpointTransform and GraniteMoeFusedExpertSplitCheckpointTransform class definitions were never removed after Stage ⑤ aliased them to FusedExpertSplitCheckpoint Transform. The redefinitions overrode the aliases, restored the old bool return type, and broke the Stage ④ pipeline contract. Deleted. Bug 2: FusedExpertSplitCheckpointTransform.get_consumed_keys() returned canonical keys (experts.gate_up_proj) but the pipeline's remaining computation filters against original weight_map keys (input_linear.weight for GraniteMoE). Mismatch meant DtypeConversion would re-process expert keys. Fixed to iterate original keys and map to canonical for matching. Signed-off-by: Amar <amarshar@qti.qualcomm.com>
- find_checkpoint_key(): add lm_head.weight → wte.weight fallback for
GPT-family models where lm_head is not stored in the checkpoint.
- checkpoint_key_resolver: read active_group TRANSFORM_ID from prepared
checkpoint manifest (.checkpoint_prepared.json) instead of re-running
detect_group_transform() on the prepared weight_map (which has output
keys, not the original expert keys).
- GptOssMxfp4ExpertDequantExpertParallelCheckpointTransform: new transform
for GptOss MXFP4 with expert_parallel prefill — dequant + repack
[E,H,I] → [E/P,P,H,I] via pack_moe_weights_for_expert_parallel().
- detect_group_transform(): use config.model_type == 'gpt_oss' to route
GptOss (cleaner than _blocks key scan), with expert_parallel routing.
- _load_prepared_tensors: scan shards when index.json is absent (needed
for unit tests that call transforms directly without the pipeline).
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
1. _find_transform_by_id(): add expert-parallel transform IDs so promote_initializers_and_build_spec() can recover the active_transform from the manifest and apply resolve_onnx_key() (.mlp.→.block_sparse_moe.). - moe_expert_parallel_stacking_v1 → MoEExpertParallelStackingCheckpointTransform - gptoss_mxfp4_dequant_expert_parallel_v1 → GptOssMxfp4ExpertDequantExpertParallelCheckpointTransform 2. _prepared_checkpoint_hash(): include moe_prefill_num_pipeline_stages and moe_prefill_num_parallelized_experts so two expert_parallel exports with different P/E values produce different prepared directories. 3. flavours.py: restore expand_as for matched_idx; torch.full_like on int32_max_scalar was an accidental change — the original compiler-preferred workaround uses expand_as on a scalar to avoid ConstantOfShape(INT32_MAX). Signed-off-by: Amar <amarshar@qti.qualcomm.com>
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
…agg working Signed-off-by: Amar <amarshar@qti.qualcomm.com>
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
Signed-off-by: Amar <amarshar@qti.qualcomm.com>
Signed-off-by: smedhe <smedhe@qti.qualcomm.com>
Signed-off-by: smedhe <smedhe@qti.qualcomm.com>
…king Signed-off-by: smedhe <smedhe@qti.qualcomm.com>
Signed-off-by: smedhe <smedhe@qti.qualcomm.com>
…port for weightfree in causallm models. Signed-off-by: Anusha V.S Bhamidipati <abhamidi@qti.qualcomm.com>
Anusha V.S Bhamidipati (quic-abhamidi)
changed the title
Anusha V.S Bhamidipati (quic-abhamidi)
changed the title
Anusha V.S Bhamidipati (quic-abhamidi)
changed the title
Anusha V.S Bhamidipati (quic-abhamidi)
changed the title
| split_model_io=True, | ||
| use_onnx_subfunctions=True, | ||
| qaic_config=qaic_config, | ||
| # comp_ctx_lengths_prefill=comp_ctx_lengths_prefill, |
There was a problem hiding this comment.
Can we remove the commented lines 78,79?
| split_model_io=True, | ||
| use_onnx_subfunctions=True, | ||
| qaic_config=qaic_config, | ||
| # comp_ctx_lengths_prefill=comp_ctx_lengths_prefill, |
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The blocked QAIC matrix covers:
gemma,gemma2,glm4_moe,gpt_oss,granite,granitemoe,llama,mistral,mixtral,qwen2,qwen3,qwen3_moe, andstarcoder2for non-kv_batch_foldmodes.kv_batch_foldis limited toqwen3_5/tiny-random/qwen3.5, where batch-fold decode support is explicit.Also adds fixes for Dynamo export/runtime issues found during validation:
torch._dynamo.is_compiling()batch_indexkv_batch_foldqwen3_5-kv_batch_foldwithuse_onnx_subfunctions=Trueis currently skipped with TODOQRANIUMSW-64622, pending compiler support for nested loop constructs generated bytorch.while_loopin ONNX subfunctions.kv_batch_foldis covered and passing withuse_onnx_subfunctions=False.Validation
92cases9casesqwen3_5 kv_batch_foldwithout ONNX subfunctions: passed