diff --git a/content/learning-paths/mobile-graphics-and-gaming/deploy-deit-tiny-with-vgf/_index.md b/content/learning-paths/mobile-graphics-and-gaming/deploy-deit-tiny-with-vgf/_index.md new file mode 100644 index 0000000000..474d8f72e7 --- /dev/null +++ b/content/learning-paths/mobile-graphics-and-gaming/deploy-deit-tiny-with-vgf/_index.md @@ -0,0 +1,74 @@ +--- +title: Classify pet images with DeiT-Tiny and Arm VGF using ExecuTorch +description: Fine-tune DeiT-Tiny, export a quantized model with the Arm VGF backend, and classify a pet image using ExecuTorch and the ML SDK for Vulkan. + +draft: true +cascade: + draft: true + +minutes_to_complete: 120 + +who_is_this_for: This Learning Path is for machine learning developers who want to deploy an image classifier through the Arm VGF backend and run it with the ML SDK for Vulkan. + +learning_objectives: + - Prepare ExecuTorch, the ML SDK for Vulkan, and a VGF runner on a Linux host + - Fine-tune DeiT-Tiny on the Oxford-IIIT Pet dataset and export a quantized VGF program + - Classify a pet image with the VGF-backed ExecuTorch program + - Inspect the predicted breed and confirm VGF execution + +prerequisites: + - A Linux development machine with an aarch64 or x86_64 processor + - A working Vulkan 1.3 or later GPU driver with shaderFloat64 support for the packaged ML emulation layer + - Python 3.12, Git, a C++ compiler, and Make + - Familiarity with Python virtual environments, PyTorch, and model training + - Internet access and disk space for the source code, SDK, Oxford-IIIT Pet dataset, and model checkpoints + +author: Usamah Zaheer + +generate_summary_faq: true +rerun_summary: false +rerun_faqs: false + +skilllevels: Advanced +subjects: ML +armips: + - Mali +tools_software_languages: + - ExecuTorch + - PyTorch + - Python + - VGF + - Vulkan + - CMake + - Hugging Face +operatingsystems: + - Linux + +further_reading: + - resource: + title: ExecuTorch VGF image classification example + link: https://github.com/pytorch/executorch/tree/9dfe4086846ad372b8b78976586ee1857a0c6d13/examples/arm/image_classification_example_vgf + type: website + - resource: + title: ExecuTorch Arm VGF backend documentation + link: https://github.com/pytorch/executorch/blob/9dfe4086846ad372b8b78976586ee1857a0c6d13/docs/source/backends/arm-vgf/arm-vgf-overview.md + type: documentation + - resource: + title: ML SDK for Vulkan + link: https://github.com/arm/ai-ml-sdk-for-vulkan + type: website + - resource: + title: DeiT-Tiny model card + link: https://huggingface.co/facebook/deit-tiny-patch16-224 + type: documentation + - resource: + title: Oxford-IIIT Pet dataset on Hugging Face + link: https://huggingface.co/datasets/timm/oxford-iiit-pet + type: website + +### FIXED, DO NOT MODIFY +# ================================================================================ +weight: 1 # _index.md always has weight of 1 to order correctly +layout: "learningpathall" # All files under learning paths have this same wrapper +learning_path_main_page: "yes" # This should be surfaced when looking for related content. Only set for _index.md of learning path content. +--- diff --git a/content/learning-paths/mobile-graphics-and-gaming/deploy-deit-tiny-with-vgf/_next-steps.md b/content/learning-paths/mobile-graphics-and-gaming/deploy-deit-tiny-with-vgf/_next-steps.md new file mode 100644 index 0000000000..727b395ddd --- /dev/null +++ b/content/learning-paths/mobile-graphics-and-gaming/deploy-deit-tiny-with-vgf/_next-steps.md @@ -0,0 +1,8 @@ +--- +# ================================================================================ +# FIXED, DO NOT MODIFY THIS FILE +# ================================================================================ +weight: 21 # The weight controls the order of the pages. _index.md always has weight 1. +title: "Next Steps" # Always the same, html page title. +layout: "learningpathall" # All files under learning paths have this same wrapper for Hugo processing. +--- diff --git a/content/learning-paths/mobile-graphics-and-gaming/deploy-deit-tiny-with-vgf/deit_vgf_helper.py b/content/learning-paths/mobile-graphics-and-gaming/deploy-deit-tiny-with-vgf/deit_vgf_helper.py new file mode 100644 index 0000000000..96f355cd26 --- /dev/null +++ b/content/learning-paths/mobile-graphics-and-gaming/deploy-deit-tiny-with-vgf/deit_vgf_helper.py @@ -0,0 +1,197 @@ +"""Supporting commands for the DeiT-Tiny VGF Learning Path, not ExecuTorch.""" + +import argparse +import json +import math +import struct +from pathlib import Path + + +MODEL_ID = "facebook/deit-tiny-patch16-224" +MODEL_REVISION = "b3428f18dcc7b543470d07f14b4a4157815d1880" +DATASET_ID = "timm/oxford-iiit-pet" +DATASET_REVISION = "089695c834a7deb60505b7cc506672db1c31a6aa" +NUM_CLASSES = 37 +INPUT_SHAPE = (1, 3, 224, 224) + + +def validate_labels(metadata): + if not isinstance(metadata, dict): + raise ValueError("Expected a JSON object containing breed labels") + labels = metadata["id2label"] + if not isinstance(labels, dict) or set(labels) != { + str(index) for index in range(NUM_CLASSES) + }: + raise ValueError("Expected breed labels for class IDs 0 through 36") + if not all(isinstance(label, str) and label for label in labels.values()): + raise ValueError("Breed labels must be nonempty strings") + return labels + + +def checkpoint(args): + model_dir = args.work_dir / "deit-tiny-oxford-pet/final_model" + config = json.loads((model_dir / "config.json").read_text(encoding="utf-8")) + validate_labels(config) + source = model_dir / "model.safetensors" + target = model_dir / "pytorch_model.bin" + if source.is_file(): + import torch + from safetensors.torch import load_file + + # Refresh the export copy after retraining, even if an older .bin exists. + weights = load_file(str(source)) + torch.save(weights, target) + print(f"Export weights: {target}") + print(f"Original weights preserved: {source}") + elif target.is_file() and target.stat().st_size: + print(f"Using existing export weights: {target}") + else: + raise FileNotFoundError( + f"No model.safetensors or nonempty pytorch_model.bin in {model_dir}. " + "Run train_deit.py first." + ) + print(f"Export checkpoint ready: {model_dir}") + + +def prepare(args): + if args.sample_index < 0: + raise ValueError("--sample-index must be zero or greater") + model_dir = args.work_dir / "deit-tiny-oxford-pet/final_model" + config = json.loads((model_dir / "config.json").read_text(encoding="utf-8")) + labels = validate_labels(config) + + from datasets import load_dataset + from transformers import AutoImageProcessor + + dataset = load_dataset(DATASET_ID, revision=DATASET_REVISION, split="test") + if args.sample_index >= len(dataset): + raise ValueError(f"--sample-index must be less than {len(dataset)}") + sample = dataset[args.sample_index] + expected_id = int(sample["label"]) + if labels[str(expected_id)] != dataset.features["label"].names[expected_id]: + raise ValueError( + "Checkpoint breed labels do not match the Oxford-IIIT Pet dataset" + ) + image = sample["image"].convert("RGB") + processor = AutoImageProcessor.from_pretrained( + MODEL_ID, revision=MODEL_REVISION, use_fast=True + ) + pixels = processor(image, return_tensors="pt")["pixel_values"].contiguous() + if tuple(pixels.shape) != INPUT_SHAPE: + raise ValueError( + f"Expected input shape {INPUT_SHAPE}, got {tuple(pixels.shape)}" + ) + + image.save(args.work_dir / "input.jpg") + pixels.detach().cpu().numpy().astype("&1 | tee arm_test/deit_vgf/export.log +``` + +The script exports the floating-point graph, calibrates symmetric INT8 post-training quantization, and evaluates the quantized model in PyTorch. It then delegates supported operations through the Arm VGF backend and writes the `.pte` file. + +## Check the export result + +Find the accuracy result and the export confirmation in the log, then check that the program exists: + +```bash +grep -E 'Top-1 accuracy|Exported model saved' arm_test/deit_vgf/export.log +test -s arm_test/deit_vgf/deit_quantized_vgf.pte +``` + +The script reports `Top-1 accuracy on 100 test samples:` followed by the result from your run. A successful export also reports the output path, and `test -s` exits successfully when that file is nonempty. + +The reported accuracy measures the quantized PyTorch model before VGF execution. The training log evaluates a different number of test images, so those two values alone do not measure the accuracy change caused by quantization. Use the same evaluation images when investigating that change. + +The `.pte` includes its VGF delegate data. You do not need to supply a separate `.vgf` file to the ExecuTorch runner. + +## What you've accomplished + +You have produced a quantized VGF-backed program and recorded its host accuracy. Next, you will classify a pet image with the runner you built during setup. diff --git a/content/learning-paths/mobile-graphics-and-gaming/deploy-deit-tiny-with-vgf/prepare-the-environment.md b/content/learning-paths/mobile-graphics-and-gaming/deploy-deit-tiny-with-vgf/prepare-the-environment.md new file mode 100644 index 0000000000..8bc6072176 --- /dev/null +++ b/content/learning-paths/mobile-graphics-and-gaming/deploy-deit-tiny-with-vgf/prepare-the-environment.md @@ -0,0 +1,112 @@ +--- +title: Prepare ExecuTorch and build the VGF runner +description: Set up the DeiT example dependencies and ML SDK for Vulkan, then build a VGF-enabled ExecuTorch runner on your Linux host. +weight: 3 +layout: "learningpathall" +--- + +## Get the ExecuTorch source + +The commands use public ExecuTorch revision `9dfe4086846ad372b8b78976586ee1857a0c6d13`. This keeps the example, exporter, and runtime source aligned. + +Create a new workspace, then clone the source and select that revision: + +```bash +mkdir deit-vgf-workspace +cd deit-vgf-workspace +git clone https://github.com/pytorch/executorch.git executorch +cd executorch +git checkout 9dfe4086846ad372b8b78976586ee1857a0c6d13 +git submodule sync --recursive +git submodule update --init --recursive +``` + +Keep the repository directory named `executorch`: the build checks this name at the pinned revision. Run all remaining commands from this repository root, in the same shell. + +## Install the Python dependencies + +Create a Python 3.12 environment and install the current source checkout with VGF dependencies: + +```bash +python3.12 -m venv .venv +source .venv/bin/activate +python -m pip install --upgrade pip +./install_executorch.sh --optional-dependency vgf +``` + +Install the example's training and evaluation dependencies after the main installer: + +```bash +python -m pip install -r examples/arm/image_classification_example_vgf/requirements.txt +``` + +The example pins Transformers 5.3.0. The ExecuTorch installer supplies the matching PyTorch stack, Datasets, and CMake dependencies. + +## Configure the ML SDK for Vulkan + +Read the tooling's license terms before accepting them. Install the ML SDK components and Vulkan SDK, then load the generated environment: + +```bash +./examples/arm/setup.sh \ + --i-agree-to-the-contained-eula \ + --disable-ethos-u-deps \ + --enable-mlsdk-deps +source examples/arm/arm-scratch/setup_path.sh +``` + +This revision uses ML SDK packages version `0.10.0`. The setup script configures the model converter, VGF library, and Vulkan emulation layers. Your GPU driver must already be installed and working. + +{{% notice Note %}} +This flow uses the packaged emulation layer, which needs `shaderFloat64` support at this revision. If setup reports missing support, use a compatible Linux host for these commands. The [ML SDK source-build helper](https://github.com/pytorch/executorch/blob/9dfe4086846ad372b8b78976586ee1857a0c6d13/backends/arm/scripts/setup-mlsdk-from-source.sh) documents the separate source-build route for other configurations. +{{% /notice %}} + +## Check the environment + +Check export prerequisites and confirm that the Vulkan tools can see your GPU: + +```bash +python -m executorch.backends.arm.vgf.check_env --aot +command -v model-converter +command -v glslc +vulkaninfo --summary +``` + +Resolve any `FAIL` entries before continuing. The tool paths should belong to this environment, and the Vulkan summary should identify your GPU and driver. + +## Build the host runner + +Keep the Python environment active and the generated `setup_path.sh` sourced. Configure a separate build directory for the VGF runner: + +```bash +cmake -S . -B cmake-out-deit-vgf \ + -DCMAKE_BUILD_TYPE=Release \ + -DEXECUTORCH_BUILD_EXTENSION_DATA_LOADER=ON \ + -DEXECUTORCH_BUILD_EXTENSION_TENSOR=ON \ + -DEXECUTORCH_BUILD_KERNELS_QUANTIZED=ON \ + -DEXECUTORCH_BUILD_XNNPACK=OFF \ + -DEXECUTORCH_BUILD_VULKAN=ON \ + -DEXECUTORCH_BUILD_VGF=ON \ + -DEXECUTORCH_ENABLE_LOGGING=ON \ + -DPython3_EXECUTABLE="$(command -v python)" + +cmake --build cmake-out-deit-vgf --target executor_runner --parallel 4 +``` + +`EXECUTORCH_BUILD_VGF` includes the Arm VGF delegate. The Vulkan option enables the associated runtime components. The build produces `cmake-out-deit-vgf/executor_runner` for your host architecture. + +## Save the Learning Path helper + +Create a directory for your artifacts, and preserve failures when logging command output: + +```bash +mkdir -p arm_test/deit_vgf +set -o pipefail +``` + +Download the [DeiT-Tiny Learning Path helper](../deit_vgf_helper.py) and save it as `arm_test/deit_vgf/deit_vgf_helper.py` in your ExecuTorch checkout. If your browser displays the source, use **Save as** to save it with the `.py` extension. Review the file before running it. + +The helper belongs to this Learning Path, not the ExecuTorch example. It handles checkpoint compatibility, image preparation, and result decoding without changing the example. Training, export, and inference still use ExecuTorch's existing scripts and runner. + +## What you've accomplished + +You have prepared the tools, built the VGF runner, and saved the helper. Next, you will fine-tune the classifier and prepare its checkpoint for export. diff --git a/content/learning-paths/mobile-graphics-and-gaming/deploy-deit-tiny-with-vgf/run-the-example.md b/content/learning-paths/mobile-graphics-and-gaming/deploy-deit-tiny-with-vgf/run-the-example.md new file mode 100644 index 0000000000..5943301781 --- /dev/null +++ b/content/learning-paths/mobile-graphics-and-gaming/deploy-deit-tiny-with-vgf/run-the-example.md @@ -0,0 +1,69 @@ +--- +title: Classify a pet image and verify VGF execution +description: Run a pet image through your VGF-backed ExecuTorch program, inspect the predicted breed, and confirm successful host execution. +weight: 6 +layout: "learningpathall" +aliases: + - /learning-paths/mobile-graphics-and-gaming/deploy-deit-tiny-with-vgf/validate-the-results/ +--- + +## Prepare a pet image + +Keep your Python environment active and `setup_path.sh` sourced. Use the Learning Path helper to prepare the first image in the dataset's test split: + +```bash +python arm_test/deit_vgf/deit_vgf_helper.py prepare +``` + +The helper uses the same pinned image processor as the exporter. It saves `input.bin`, `input.jpg`, and `reference.json` under `arm_test/deit_vgf/`. + +`input.bin` contains the normalized float32 tensor in `[1, 3, 224, 224]` batch, channel, height, width order. The runner reads this tensor, not the JPEG. Open `input.jpg` in an image viewer to inspect the pet you will classify. + +## Run inference through VGF + +Execute the exported program with your input file and save the output scores: + +```bash +set -o pipefail +./cmake-out-deit-vgf/executor_runner \ + --model_path=arm_test/deit_vgf/deit_quantized_vgf.pte \ + --inputs=arm_test/deit_vgf/input.bin \ + --output_file=arm_test/deit_vgf/prediction \ + 2>&1 | tee arm_test/deit_vgf/runtime.log +``` + +A successful run reports `Model executed successfully` and writes `arm_test/deit_vgf/prediction-0.bin`. The runner appends `-0.bin` for the first output tensor. + +Keep `--inputs`: without it, the generic runner fills the input tensor with ones instead of classifying your pet image. + +## Inspect the breed prediction + +Decode the saved scores and verify VGF execution: + +```bash +python arm_test/deit_vgf/deit_vgf_helper.py inspect +``` + +The helper checks for 37 finite output scores, then maps the largest score to a breed. It prints `Expected breed`, `VGF prediction`, and `Matches dataset label`. It also checks the runtime log for `Entered VGF init` and `Model executed successfully` before reporting `VGF execution: confirmed`. + +A valid prediction and confirmed VGF execution complete the deployment workflow. A matching dataset label means the model recognizes this image; a mismatch does not by itself indicate a deployment failure. + +{{% notice Note %}} +One image does not measure dataset accuracy. The export log reports quantized PyTorch accuracy, not VGF accuracy across the test set. This host emulation run also does not establish performance on an Arm GPU. +{{% /notice %}} + +## Optional: compare with the floating-point model + +Run the original fine-tuned model on the same input and compare its winning class with the VGF result: + +```bash +python arm_test/deit_vgf/deit_vgf_helper.py inspect --compare-fp32 +``` + +The helper adds the floating-point prediction and whether the two predictions match. Quantization can change the winning class. Compare more images before drawing conclusions about accuracy or numerical equivalence. + +To classify another test image, repeat `prepare` with `--sample-index 1`, then rerun inference and inspection. Each preparation replaces the previous input artifacts. The helper rejects predictions and logs that predate the prepared image, so you must rerun `executor_runner` before inspecting a new image. + +## What you've accomplished + +You have fine-tuned DeiT-Tiny, exported a VGF-backed program, and classified a pet image with the host runtime. Your model, input, prediction, and logs are in `arm_test/deit_vgf/`. You can now reuse the `.pte` to classify other test images. diff --git a/content/learning-paths/mobile-graphics-and-gaming/deploy-deit-tiny-with-vgf/train-the-model.md b/content/learning-paths/mobile-graphics-and-gaming/deploy-deit-tiny-with-vgf/train-the-model.md new file mode 100644 index 0000000000..ee6a70e052 --- /dev/null +++ b/content/learning-paths/mobile-graphics-and-gaming/deploy-deit-tiny-with-vgf/train-the-model.md @@ -0,0 +1,37 @@ +--- +title: Fine-tune DeiT-Tiny on pet images +description: Train a DeiT-Tiny classifier for 37 pet breeds and prepare its checkpoint for the VGF export script. +weight: 4 +layout: "learningpathall" +--- + +## Train the pet classifier + +The training script loads `facebook/deit-tiny-patch16-224` and replaces its classification head for the dataset's 37 breeds. It uses a fixed dataset revision and seed, reserving ten percent of the training split for validation. + +Run three training epochs and save the log: + +```bash +python examples/arm/image_classification_example_vgf/model_export/train_deit.py \ + --output-dir arm_test/deit_vgf/deit-tiny-oxford-pet \ + --num-epochs 3 \ + 2>&1 | tee arm_test/deit_vgf/train.log +``` + +The first run downloads the model weights and dataset. At completion, the script prints `Test set accuracy:` and saves the selected model under `arm_test/deit_vgf/deit-tiny-oxford-pet/final_model/`. + +Record the accuracy from your run. Training speed and final accuracy depend on your environment; a single fixed accuracy value is not a completion requirement. + +## Prepare the checkpoint for export + +At the pinned revision, the trainer saves `model.safetensors`, but `export_deit.py` loads with `use_safetensors=False`. Use the Learning Path helper to create the compatible PyTorch weight file: + +```bash +python arm_test/deit_vgf/deit_vgf_helper.py checkpoint +``` + +The helper creates `pytorch_model.bin` in `final_model/` without changing the trained weights. Keep `config.json` beside the weights because it contains the model configuration and breed labels. + +## What you've accomplished + +You have fine-tuned DeiT-Tiny and prepared a checkpoint that the example exporter can load. Next, you will quantize the model and generate the `.pte` program. diff --git a/content/learning-paths/mobile-graphics-and-gaming/deploy-deit-tiny-with-vgf/understand-the-workflow.md b/content/learning-paths/mobile-graphics-and-gaming/deploy-deit-tiny-with-vgf/understand-the-workflow.md new file mode 100644 index 0000000000..8e30b7e1aa --- /dev/null +++ b/content/learning-paths/mobile-graphics-and-gaming/deploy-deit-tiny-with-vgf/understand-the-workflow.md @@ -0,0 +1,46 @@ +--- +title: Understand the DeiT-Tiny deployment workflow +description: Follow a pet classifier from DeiT-Tiny fine-tuning through quantization and VGF export to inference with the ML SDK for Vulkan. +weight: 2 +layout: "learningpathall" +--- + +## Classify a pet image with VGF + +You will fine-tune DeiT-Tiny, a Data-efficient Image Transformer, to recognize 37 cat and dog breeds. You will then export the classifier with the Arm VGF backend and run a pet image through ExecuTorch. + +You will use the [ExecuTorch example's](https://github.com/pytorch/executorch/tree/9dfe4086846ad372b8b78976586ee1857a0c6d13/examples/arm/image_classification_example_vgf) training and export scripts, then run inference with `executor_runner`. A downloadable Learning Path helper handles checkpoint compatibility, image preparation, and breed decoding. You don't need to edit the example. + +## Follow the model through the pipeline + +Follow these stages: + +1. Prepare the Linux environment and build the VGF runner +2. Fine-tune DeiT-Tiny on Oxford-IIIT Pet images +3. Quantize the model and export a VGF-backed ExecuTorch `.pte` file +4. Classify a pet image and confirm VGF execution + +The exporter uses `VgfCompileSpec("TOSA-1.0+INT")`. TOSA, the Tensor Operator Set Architecture, describes the graph that the ML SDK model converter compiles into VGF. The `.pte` includes this delegate data. + +The model uses these tensor interfaces: + +| Tensor | Shape | Data | +|---|---|---| +| Input | `[1, 3, 224, 224]` | Preprocessed RGB image, float32 | +| Output | `[1, 37]` | One float32 class score per breed | + +Quantization applies inside the model. You still supply the normalized floating-point image expected by its exported input. + +## Understand the execution target + +The ML SDK for Vulkan supplies emulation layers for the Arm tensor and data graph extensions. These let you develop the VGF workflow on a compatible host GPU before integrating it with a device application. + +You will validate host execution and a breed prediction. Timing from this emulation workflow does not establish performance on an Arm GPU, and this example does not deploy an Android application. + +{{% notice Note %}} +Allow additional time for dependency downloads, builds, and three training epochs. CPU training can take substantially longer than the estimated reading and setup time. +{{% /notice %}} + +## What you've learned + +You know how the fine-tuned model becomes a VGF-backed ExecuTorch program and what the host run demonstrates. Next, you will prepare the development environment.