Skip to content

feat: add U2NetP foreground segmentation model presets #1587

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
barhanc merged 8 commits into from
Oct 29, 2026
Merged

feat: add U2NetP foreground segmentation model presets #1587

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
88286d0
Add U2NetP foreground segmentation model presets
barhanc Oct 29, 2026
ed0b8c5
Cache download sizes for U2NetP benchmark variants
barhanc Oct 29, 2026
9c337af
Shorten U2NetP model description
barhanc Oct 29, 2026
3d9cf77
Add selective XNNPACK INT8 variant for U2NetP
barhanc Oct 29, 2026
7b508c0
Add device-validated U2NetP Vulkan variants
barhanc Oct 29, 2026
0d0af03
Keep only the U2NetP Vulkan FP16 variant
barhanc Oct 29, 2026
0a8e005
Restore standard Vulkan-first selection for U2NetP
barhanc Oct 29, 2026
fa7a6dd
Share U2NetP configuration through a dedicated options constant
barhanc Oct 29, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .cspell-wordlist.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -447,3 +447,4 @@ grayscale
Premul
eurollm
doclayout
netp
7 changes: 6 additions & 1 deletion apps/benchmarks/scripts/variant-sizes.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -493,5 +493,10 @@
"https://huggingface.co/software-mansion/all-mpnet-base-v2-ExecuTorch/resolve/v0.11.0/vulkan/all_mpnet_base_v2_vulkan_fp16.pte": 217937154,
"https://huggingface.co/software-mansion/all-MiniLM-L6-v2-ExecuTorch/resolve/v0.11.0/vulkan/all_minilm_l6_v2_vulkan_fp16.pte": 45208066,
"https://huggingface.co/software-mansion/all-MiniLM-L6-v2-ExecuTorch/resolve/v0.11.0/xnnpack/all_minilm_l6_v2_xnnpack_fp32.pte": 90391296,
"https://huggingface.co/software-mansion/all-MiniLM-L6-v2-ExecuTorch/resolve/v0.11.0/coreml/all_minilm_l6_v2_coreml_fp16.pte": 45460244
"https://huggingface.co/software-mansion/all-MiniLM-L6-v2-ExecuTorch/resolve/v0.11.0/coreml/all_minilm_l6_v2_coreml_fp16.pte": 45460244,
"https://huggingface.co/software-mansion/U2NetP-ExecuTorch/resolve/v1.0.0/xnnpack/u2netp_xnnpack_fp32.pte": 4610948,
"https://huggingface.co/software-mansion/U2NetP-ExecuTorch/resolve/v1.0.0/coreml/u2netp_coreml_fp16.pte": 3272420,
"https://huggingface.co/software-mansion/U2NetP-ExecuTorch/resolve/v1.0.0/coreml/u2netp_coreml_fp32.pte": 5476362,
"https://huggingface.co/software-mansion/U2NetP-ExecuTorch/resolve/v1.0.0/xnnpack/u2netp_xnnpack_int8.pte": 4007684,
"https://huggingface.co/software-mansion/U2NetP-ExecuTorch/resolve/v1.0.0/vulkan/u2netp_vulkan_fp16.pte": 2437516
}
25 changes: 24 additions & 1 deletion apps/computer-vision/app/segmentation/index.tsx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import React, { useState, useRef } from 'react';
import { View, Text, ScrollView } from 'react-native';
import { View, Text, ScrollView, Platform } from 'react-native';
import { useSafeAreaInsets } from 'react-native-safe-area-context';
import { commonStyles, theme } from '../../theme';
import {
Expand All @@ -23,6 +23,29 @@ const SEGMENTATION_OPTIONS: ModelOption[] = [
label: 'Selfie Seg (FP32)',
value: models.semanticSegmentation.SELFIE_SEGMENTATION.XNNPACK_FP32,
},
{
label: 'U²-Netp (XNNPACK FP32)',
value: models.semanticSegmentation.U2NETP.XNNPACK_FP32,
},
{
label: 'U²-Netp (XNNPACK INT8)',
value: models.semanticSegmentation.U2NETP.XNNPACK_INT8,
},
{
label: 'U²-Netp (CoreML FP16)',
value: models.semanticSegmentation.U2NETP.COREML_FP16,
disabled: Platform.OS !== 'ios' || Number.parseInt(String(Platform.Version), 10) < 18,
},
{
label: 'U²-Netp (CoreML FP32)',
value: models.semanticSegmentation.U2NETP.COREML_FP32,
disabled: Platform.OS !== 'ios' || Number.parseInt(String(Platform.Version), 10) < 18,
},
{
label: 'U²-Netp (Vulkan FP16)',
value: models.semanticSegmentation.U2NETP.VULKAN_FP16,
disabled: Platform.OS !== 'android',
},
{
label: 'LRASPP MobileNet V3 (INT8)',
value: models.semanticSegmentation.LRASPP_MOBILENET_V3_LARGE.XNNPACK_INT8,
Expand Down
13 changes: 7 additions & 6 deletions docs/docs/02-extensions/computer-vision/06-semantic-segmentation.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,12 +165,13 @@ See [Worklets & Threading](../../03-core-and-advanced/06-worklets-and-threading.

The library provides ready-to-use segmentation models from the [Software Mansion HuggingFace Semantic Segmentation Collection](https://huggingface.co/collections/software-mansion/semantic-segmentation), accessible via [`models.semanticSegmentation`](../../06-api-reference/variables/models.md#semanticsegmentation):

| Model Family | Variants | Classes / Labels | Size Range | Supported Backends | Notes |
| :---------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :---------------------------------------------------------------------------------------- | :----------------- | :----------------------------- | :----------------------------------------------------------------------- |
| **Selfie Segmentation** | [`Portrait`](../../06-api-reference/variables/models.md#semanticsegmentationselfie_segmentation), [`Landscape`](../../06-api-reference/variables/models.md#semanticsegmentationselfie_segmentation_landscape) | Person / Background | 0.5 MB – 0.6 MB | XNNPACK (CPU), Core ML (Apple) | Real-time front-camera portrait background replacement and blur effects. |
| **LRASPP MobileNetV3** | [See](../../06-api-reference/variables/models.md#semanticsegmentationlraspp_mobilenet_v3_large) | [`PASCAL_VOC_LABELS`](../../06-api-reference/variables/PASCAL_VOC_LABELS.md) (21 classes) | 3.4 MB – 12.3 MB | XNNPACK (CPU), Core ML (Apple) | Lightweight multi-class scene segmentation with low CPU overhead. |
| **DeepLabV3** | [`ResNet50`](../../06-api-reference/variables/models.md#semanticsegmentationdeeplab_v3_resnet50), [`ResNet101`](../../06-api-reference/variables/models.md#semanticsegmentationdeeplab_v3_resnet101), [`MobileNetV3`](../../06-api-reference/variables/models.md#semanticsegmentationdeeplab_v3_mobilenet_v3_large) | [`PASCAL_VOC_LABELS`](../../06-api-reference/variables/PASCAL_VOC_LABELS.md) (21 classes) | 40.4 MB – 223.6 MB | XNNPACK (CPU), Core ML (Apple) | High-fidelity dense pixel classification for complex scenes. |
| **FCN** | [`ResNet50`](../../06-api-reference/variables/models.md#semanticsegmentationfcn_resnet50), [`ResNet101`](../../06-api-reference/variables/models.md#semanticsegmentationfcn_resnet101) | [`PASCAL_VOC_LABELS`](../../06-api-reference/variables/PASCAL_VOC_LABELS.md) (21 classes) | 34.0 MB – 198.1 MB | XNNPACK (CPU), Core ML (Apple) | Fully Convolutional Networks baseline for dense multi-class parsing. |
| Model Family | Variants | Classes / Labels | Size Range | Supported Backends | Notes |
| :---------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :---------------------------------------------------------------------------------------- | :----------------- | :----------------------------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Selfie Segmentation** | [`Portrait`](../../06-api-reference/variables/models.md#semanticsegmentationselfie_segmentation), [`Landscape`](../../06-api-reference/variables/models.md#semanticsegmentationselfie_segmentation_landscape) | Person / Background | 0.5 MB – 0.6 MB | XNNPACK (CPU), Core ML (Apple) | Real-time front-camera portrait background replacement and blur effects. |
| **LRASPP MobileNetV3** | [See](../../06-api-reference/variables/models.md#semanticsegmentationlraspp_mobilenet_v3_large) | [`PASCAL_VOC_LABELS`](../../06-api-reference/variables/PASCAL_VOC_LABELS.md) (21 classes) | 3.4 MB – 12.3 MB | XNNPACK (CPU), Core ML (Apple) | Lightweight multi-class scene segmentation with low CPU overhead. |
| **DeepLabV3** | [`ResNet50`](../../06-api-reference/variables/models.md#semanticsegmentationdeeplab_v3_resnet50), [`ResNet101`](../../06-api-reference/variables/models.md#semanticsegmentationdeeplab_v3_resnet101), [`MobileNetV3`](../../06-api-reference/variables/models.md#semanticsegmentationdeeplab_v3_mobilenet_v3_large) | [`PASCAL_VOC_LABELS`](../../06-api-reference/variables/PASCAL_VOC_LABELS.md) (21 classes) | 40.4 MB – 223.6 MB | XNNPACK (CPU), Core ML (Apple) | High-fidelity dense pixel classification for complex scenes. |
| **FCN** | [`ResNet50`](../../06-api-reference/variables/models.md#semanticsegmentationfcn_resnet50), [`ResNet101`](../../06-api-reference/variables/models.md#semanticsegmentationfcn_resnet101) | [`PASCAL_VOC_LABELS`](../../06-api-reference/variables/PASCAL_VOC_LABELS.md) (21 classes) | 34.0 MB – 198.1 MB | XNNPACK (CPU), Core ML (Apple) | Fully Convolutional Networks baseline for dense multi-class parsing. |
| **U²-Netp** | [Small](../../06-api-reference/variables/models.md#semanticsegmentationu2netp) | Foreground probability (1 output channel) | 2.4 MB – 5.5 MB | XNNPACK (CPU), Core ML (Apple), Vulkan (Android) | General foreground object masks; selective INT8 for CPU, Core ML/Vulkan acceleration, FP32 for boundary precision. Vulkan FP16 retains one convolution in FP32. |

:::tip Using Custom Models
To use your own fine-tuned semantic segmentation `.pte` model, pass a [`SemanticSegmenterModel`](../../06-api-reference/type-aliases/SemanticSegmenterModel.md) configuration object to [`useSemanticSegmenter`](../../06-api-reference/functions/useSemanticSegmenter.md) or [`createSemanticSegmenter`](../../06-api-reference/functions/createSemanticSegmenter.md):
Expand Down
2 changes: 1 addition & 1 deletion docs/docs/03-core-and-advanced/08-native-libraries.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ Specifying a task under `features` is shorthand: it automatically expands to the
| `depthEstimation` | xnnpack, coreml, vulkan | opencv |
| `objectDetection` | xnnpack, coreml, vulkan | opencv |
| `keypointDetection` | xnnpack, coreml, mlx, vulkan | opencv |
| `semanticSegmentation` | xnnpack, coreml | opencv |
| `semanticSegmentation` | xnnpack, coreml, vulkan | opencv |
| `instanceSegmentation` | xnnpack, coreml, vulkan | opencv |
| `ocr` | xnnpack, coreml, vulkan | opencv |
| `verticalOCR` | xnnpack | opencv |
Expand Down
4 changes: 2 additions & 2 deletions packages/react-native-executorch/scripts/download-libs.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -166,8 +166,8 @@ const FEATURE_MAP = {
},
// Depth Anything V2, V3 and ZipDepth ship xnnpack, coreml and vulkan.
depthEstimation: { backends: ['xnnpack', 'coreml', 'vulkan'], libs: ['opencv'] },
// DeepLab/FCN/LR-ASPP/selfie all ship xnnpack + coreml.
semanticSegmentation: { backends: ['xnnpack', 'coreml'], libs: ['opencv'] },
// DeepLab/FCN/LR-ASPP/selfie ship xnnpack + coreml; U2NetP adds vulkan.
semanticSegmentation: { backends: ['xnnpack', 'coreml', 'vulkan'], libs: ['opencv'] },
// YOLO-seg xnnpack-only, rf_detr-seg/fastsam add coreml, and all three ship
// vulkan Android exports → union.
instanceSegmentation: { backends: ['xnnpack', 'coreml', 'vulkan'], libs: ['opencv'] },
Expand Down
40 changes: 40 additions & 0 deletions packages/react-native-executorch/src/models.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -482,6 +482,34 @@ const SELFIE_SEGMENTATION_LANDSCAPE_COREML_FP16: SemanticSegmenterModel<'backgro
modelOpts: SELFIE_SEGMENTATION_XNNPACK_FP32.modelOpts,
};

const U2NETP_OPTS = {
labels: ['background', 'foreground'] as const,
resizeMode: 'stretch' as const,
interpolation: 'linear' as const,
normalizeOpts: { alpha: 1 / 255, beta: 0 },
outInterpolation: 'linear' as const,
};
const U2NETP_XNNPACK_FP32: SemanticSegmenterModel<'background' | 'foreground'> = {
modelPath: `${BASE_URL}/U2NetP-ExecuTorch/${NEXT_VERSION_TAG}/xnnpack/u2netp_xnnpack_fp32.pte`,
modelOpts: U2NETP_OPTS,
};
const U2NETP_XNNPACK_INT8: SemanticSegmenterModel<'background' | 'foreground'> = {
modelPath: `${BASE_URL}/U2NetP-ExecuTorch/${NEXT_VERSION_TAG}/xnnpack/u2netp_xnnpack_int8.pte`,
modelOpts: U2NETP_OPTS,
};
const U2NETP_VULKAN_FP16: SemanticSegmenterModel<'background' | 'foreground'> = {
modelPath: `${BASE_URL}/U2NetP-ExecuTorch/${NEXT_VERSION_TAG}/vulkan/u2netp_vulkan_fp16.pte`,
modelOpts: U2NETP_OPTS,
};
const U2NETP_COREML_FP16: SemanticSegmenterModel<'background' | 'foreground'> = {
modelPath: `${BASE_URL}/U2NetP-ExecuTorch/${NEXT_VERSION_TAG}/coreml/u2netp_coreml_fp16.pte`,
modelOpts: U2NETP_OPTS,
};
const U2NETP_COREML_FP32: SemanticSegmenterModel<'background' | 'foreground'> = {
modelPath: `${BASE_URL}/U2NetP-ExecuTorch/${NEXT_VERSION_TAG}/coreml/u2netp_coreml_fp32.pte`,
modelOpts: U2NETP_OPTS,
};

const LRASPP_MOBILENET_V3_LARGE_OPTS = {
labels: PASCAL_VOC_LABELS,
resizeMode: 'stretch' as const,
Expand Down Expand Up @@ -2753,6 +2781,18 @@ export const models = {
XNNPACK_FP32: SELFIE_SEGMENTATION_LANDSCAPE_XNNPACK_FP32,
COREML_FP16: SELFIE_SEGMENTATION_LANDSCAPE_COREML_FP16,
}),
/**
* Lightweight U²-Netp foreground segmentation at 320x320 for objects and
* people. Returns a soft grayscale foreground mask for background removal.
* INT8 and Vulkan FP16 retain sensitive operations in FP32.
*/
U2NETP: variants({
XNNPACK_FP32: U2NETP_XNNPACK_FP32,
XNNPACK_INT8: U2NETP_XNNPACK_INT8,
COREML_FP16: U2NETP_COREML_FP16,
COREML_FP32: U2NETP_COREML_FP32,
VULKAN_FP16: U2NETP_VULKAN_FP16,
}),
/**
* Lite R-ASPP semantic segmentation model with MobileNetV3-Large backbone
* (21 classes, see {@link PASCAL_VOC_LABELS}). Optimized for low-latency,
Expand Down
Loading