Conversation
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/23266
Note: Links to docs will display an error until the docs builds have been completed. ✅ You can merge normally! (1 Unrelated Failure)As of commit 0fe4ca4 with merge base 273cb33 ( BROKEN TRUNK - The following job failed but were present on the merge base:👉 Rebase onto the `viable/strict` branch to avoid these failures
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
|
|
|
@pcwu0329 has exported this pull request. If you are a Meta employee, you can view the originating Diff in D122298673. |
This PR needs a
|
Summary:
`replace_with_constant_node` names propagated constants
`_prop_tensor_constant{len(exported_program.constants)}`, then checks for a
collision. Both the check and the max-suffix rescan look only at
`exported_program.constants`.
A program re-entering this pass can already carry a `_prop_tensor_constant*` in
`exported_program.state_dict` from an earlier run. That name is invisible to
both, so the pass reuses it and writes its tensor into `constants` alongside the
stale `state_dict` entry. Emission then resolves the placeholder against
`state_dict` before `constants` (`exir/emit/_emitter.py:2044`), picks up the
stale tensor, and fails the size check at `_emitter.py:513`:
InternalError: Tensor spec has buffer of size 48, but expected nbytes of 4
The spec is correct at 4 bytes; the tensor bound to it is the wrong one. Note
the existing comment already reads "If prop_constant_tensor_fqn already exists
in the state dict" -- the implementation never matched that intent.
Widen the collision check and the suffix rescan to span `state_dict` as well as
`constants`.
Observed on a Turing/Coleman hand-tracking model compiled with
`fx_options.allow_quant_dequant_on_tce=True`, which folds a scalar `1` from a
`1 - valid_multi_view` expression whose name was already taken in `state_dict`.
Instrumenting the pass showed the collision directly:
fqn=_prop_tensor_constant20 new_nbytes=4
already_in_constants=False already_in_state_dict=True
With the fix the model compiles to a .pte (3 TCE delegates, 84.19% delegation);
without it emission aborts.
Differential Revision: D122298673
a0df710 to
e17590d
Compare
Summary:
`replace_with_constant_node` names propagated constants
`_prop_tensor_constant{len(exported_program.constants)}`, then checks for a
collision. Both the check and the max-suffix rescan look only at
`exported_program.constants`.
A program re-entering this pass can already carry a `_prop_tensor_constant*` in
`exported_program.state_dict` from an earlier run. That name is invisible to
both, so the pass reuses it and writes its tensor into `constants` alongside the
stale `state_dict` entry. Emission then resolves the placeholder against
`state_dict` before `constants` (`exir/emit/_emitter.py:2044`), picks up the
stale tensor, and fails the size check at `_emitter.py:513`:
InternalError: Tensor spec has buffer of size 48, but expected nbytes of 4
The spec is correct at 4 bytes; the tensor bound to it is the wrong one. Note
the existing comment already reads "If prop_constant_tensor_fqn already exists
in the state dict" -- the implementation never matched that intent.
Widen the collision check and the suffix rescan to span `state_dict` as well as
`constants`.
Observed on a Turing/Coleman hand-tracking model compiled with
`fx_options.allow_quant_dequant_on_tce=True`, which folds a scalar `1` from a
`1 - valid_multi_view` expression whose name was already taken in `state_dict`.
Instrumenting the pass showed the collision directly:
fqn=_prop_tensor_constant20 new_nbytes=4
already_in_constants=False already_in_state_dict=True
With the fix the model compiles to a .pte (3 TCE delegates, 84.19% delegation);
without it emission aborts.
Differential Revision: D122298673
e17590d to
d19e3bb
Compare
Summary: Pull Request resolved: pytorch#23266 `replace_with_constant_node` names propagated constants `_prop_tensor_constant{len(exported_program.constants)}`, then checks for a collision. Both the check and the max-suffix rescan look only at `exported_program.constants`. A program re-entering this pass can already carry a `_prop_tensor_constant*` in `exported_program.state_dict` from an earlier run. That name is invisible to both, so the pass reuses it and writes its tensor into `constants` alongside the stale `state_dict` entry. Emission then resolves the placeholder against `state_dict` before `constants` (`exir/emit/_emitter.py:2044`), picks up the stale tensor, and fails the size check at `_emitter.py:513`: InternalError: Tensor spec has buffer of size 48, but expected nbytes of 4 The spec is correct at 4 bytes; the tensor bound to it is the wrong one. Note the existing comment already reads "If prop_constant_tensor_fqn already exists in the state dict" -- the implementation never matched that intent. Widen the collision check and the suffix rescan to span `state_dict` as well as `constants`. Observed on a Turing/Coleman hand-tracking model compiled with `fx_options.allow_quant_dequant_on_tce=True`, which folds a scalar `1` from a `1 - valid_multi_view` expression whose name was already taken in `state_dict`. Instrumenting the pass showed the collision directly: fqn=_prop_tensor_constant20 new_nbytes=4 already_in_constants=False already_in_state_dict=True With the fix the model compiles to a .pte (3 TCE delegates, 84.19% delegation); without it emission aborts. Differential Revision: D122298673
d19e3bb to
0fe4ca4
Compare
Summary:
replace_with_constant_nodenames propagated constants_prop_tensor_constant{len(exported_program.constants)}, then checks for acollision. Both the check and the max-suffix rescan look only at
exported_program.constants.A program re-entering this pass can already carry a
_prop_tensor_constant*inexported_program.state_dictfrom an earlier run. That name is invisible toboth, so the pass reuses it and writes its tensor into
constantsalongside thestale
state_dictentry. Emission then resolves the placeholder againststate_dictbeforeconstants(exir/emit/_emitter.py:2044), picks up thestale tensor, and fails the size check at
_emitter.py:513:InternalError: Tensor spec has buffer of size 48, but expected nbytes of 4
The spec is correct at 4 bytes; the tensor bound to it is the wrong one. Note
the existing comment already reads "If prop_constant_tensor_fqn already exists
in the state dict" -- the implementation never matched that intent.
Widen the collision check and the suffix rescan to span
state_dictas well asconstants.Observed on a Turing/Coleman hand-tracking model compiled with
fx_options.allow_quant_dequant_on_tce=True, which folds a scalar1from a1 - valid_multi_viewexpression whose name was already taken instate_dict.Instrumenting the pass showed the collision directly:
fqn=_prop_tensor_constant20 new_nbytes=4
already_in_constants=False already_in_state_dict=True
With the fix the model compiles to a .pte (3 TCE delegates, 84.19% delegation);
without it emission aborts.
Differential Revision: D122298673