Arm backend: Added real calibration for NFRU model

[Michiel-Olieslagers]

This patch contains the following changes:

• Added real calibration for NFRU model
• Sped up dataset generation for NSS & NFRU models
• Reworked NFRU QAT testing to use use real calibration data
• Parameterised QAT testing

Change-Id: Ifd9d1b41bbba52b200be4eba079363d12c7fb2af

cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/23182

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

❗ 1 Active SEVs

There are 1 currently active SEVs. If your PR is affected, please view them below:

• Long queue for g5 runnners

❌ 5 New Failures, 8 Pending, 1 Unrelated Failure

As of commit 4e4799c with merge base f3626e6 ():

NEW FAILURES - The following jobs have failed:

• pull / android / run-emulator (gh)
  java.lang.NullPointerException: Attempt to invoke virtual method 'java.util.List android.os.storage.StorageManager.getVolumes()' on a null object reference
• pull / test-arm-backend-no-driver (test_pytest_models_tosa) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• pull / unittest / macos / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1
• trunk / test-arm-backend-vkml (test_pytest_models_vkml) / linux-job (gh)
  RuntimeError: Command bash /__w/_temp/exec_script failed with exit code 1
• trunk / test-models-macos-coreml (mv2) / macos-job (gh)
  RuntimeError: Command bash /Users/ec2-user/runner/_work/_temp/exec_script failed with exit code 1

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

• Windows MSVC Build / build-windows-arm64 (gh) (trunk failure)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[metascroy]

@Michiel-Olieslagers this is breaking CI (e.g., the pull / test-arm-backend-no-driver (test_pytest_models_tosa) / linux-job). Are you working on a forward fix?

An agent says the following:

The failing job ran the PR’s merge commit, 22b43b2, and reports:
AttributeError: 'TosaPipelineINT' object has no attribute 'set_quantization_calibration'
The PR adds _set_nfru_calibration_samples and rewrites _set_nss_calibration_samples to call pipeline.set_quantization_calibration(...). However, that method does not exist on TosaPipelineINT or its base classes at the merge commit. The PR contains no pipeline implementation change.
Previously, NSS configured the quantization stage directly, assigning dynamic_shapes and calibration_samples. The refactor replaced that working code with the missing helper.
This explains all five failures: NFRU real-data PTQ, QAT, and a16w8, plus NSS real-data PTQ and QAT. Random-data variants bypass the helper and pass. These failures happen before pipeline.run(), so they are not numerical accuracy or infrastructure failures.
The fix is to implement the shared set_quantization_calibration method on BasePipeline, or restore direct quantization-stage configuration in both helpers. No changes made.

[psiddh]

@Michiel-Olieslagers this is breaking CI (e.g., the pull / test-arm-backend-no-driver (test_pytest_models_tosa) / linux-job). Are you working on a forward fix?

An agent says the following:

The failing job ran the PR’s merge commit, 22b43b2, and reports: AttributeError: 'TosaPipelineINT' object has no attribute 'set_quantization_calibration' The PR adds _set_nfru_calibration_samples and rewrites _set_nss_calibration_samples to call pipeline.set_quantization_calibration(...). However, that method does not exist on TosaPipelineINT or its base classes at the merge commit. The PR contains no pipeline implementation change. Previously, NSS configured the quantization stage directly, assigning dynamic_shapes and calibration_samples. The refactor replaced that working code with the missing helper. This explains all five failures: NFRU real-data PTQ, QAT, and a16w8, plus NSS real-data PTQ and QAT. Random-data variants bypass the helper and pass. These failures happen before pipeline.run(), so they are not numerical accuracy or infrastructure failures. The fix is to implement the shared set_quantization_calibration method on BasePipeline, or restore direct quantization-stage configuration in both helpers. No changes made.

#23265 , potential fwd fix ?

[zingo]

@metascroy and @psiddh I don't think anyone is working until tomorrow, I'm back from vacation tomorrow also (CET timeframe) and will see if there is anything I can do and/or check progress of the mentioend fix in one of the other PRs that @Michiel-Olieslagers mentioned.
Feel free to emergency revert the 4 NSS/NFRU PR that got merged if needed. We can reland it again with the fix.
We are sorry for the problems caused.

[zingo]

Update:

@psiddh tried to fix the problem during the night and manage to fix 2/3 problems
#23265
#23268
Left after this are merged seem to only be a few numerical/high "mean abs error" problems.