Skip to content

ENH: Serve SWIG/PCRE archives via direct-file IPFS CIDs (release-5.4)#6547

Merged
hjmjohnson merged 2 commits into
InsightSoftwareConsortium:release-5.4from
thewtex:release-5.4-swig-binaries-ipfs
Jul 4, 2026
Merged

ENH: Serve SWIG/PCRE archives via direct-file IPFS CIDs (release-5.4)#6547
hjmjohnson merged 2 commits into
InsightSoftwareConsortium:release-5.4from
thewtex:release-5.4-swig-binaries-ipfs

Conversation

@thewtex

@thewtex thewtex commented Jul 3, 2026

Copy link
Copy Markdown
Member

Summary

Re-hosts the SWIG and PCRE archives used by the Python-wrapping toolchain as
direct-file IPFS CIDs and refreshes the content-addressed mirror list in
Wrapping/Generators/SwigInterface/CMakeLists.txt.

Previously each *_cid pointed at a UnixFS directory that wrapped the archive,
so every download URL carried a trailing /<filename> path segment. Each archive
is now added to IPFS as a file using the kubo unixfs-v1-2025 import profile
(CIDv1, raw leaves, 1 MiB chunks, 1024 links/node, balanced layout), so the CID
resolves to the archive bytes directly and the URLs reference the CID with no
path suffix.

New CIDs

variable archive CID
swigwin_cid swigwin-amd64-2024-03-26-master.zip bafybeibavyfwb5fenlrjxvhd4pj55brbbvphcgpuwlmbyusrwn44rs2hcq
swiglinux_cid (arm64) swiglinux-arm64-2024-03-26-master.zip bafybeiasuvglbnjji24hiqpzry65kpqszjxyg4bup5ophfel4noykqemwy
swiglinux_cid (amd64) swiglinux-amd64-2024-03-26-master.zip bafybeiemc7gbf6bgh4yopzudhp56dsb36ks6eltq7m5sny4ifr5n2vrulu
swigmacos_cid (arm64) swigmacos-arm64-2024-03-26-master.zip bafybeihhw3wktwsi4evq3sfy4rujb5eia4shzqtw7dvgt6s2zsag2ealmm
swigmacos_cid (amd64) swigmacos-amd64-2024-03-26-master.zip bafybeiegnvdxavxc7gcf5nfkalpej64p6stn54mzv7nfdwtr5k4vkjvbte
swig_cid (source) swig-4.0.2.tar.gz bafybeidevegcr53m6hrfhvel2ahpyla5rfddu5ycsxsjv5ylmzo4czuubu
pcre_cid pcre-10.43.tar.gz bafybeig5jsstzxtwy4g3ap6rplfsnpcuptgi7uxzw2kftjnkly6tbnbiwi

Gateways

The mirror list is updated to the gateways currently serving ITK content
(each URL is now ${gateway}${cid}):

  • https://itk.mypinata.cloud/ipfs/
  • https://ipfs.io/ipfs/
  • https://gateway.pinata.cloud/ipfs/
  • https://insightsoftwareconsortium.github.io/ITKTestingData/CID/
  • https://ipfs.filebase.io/ipfs/

https://data.kitware.com/.../sha512/<hash>/download is retained as the primary
source, and URL_HASH SHA512=... continues to validate whichever mirror responds.

Hosting / pinning

Verification

  • SHA512 of every archive verified against the existing hashes before import.
  • CIDs reproducible with ipfs add --only-hash --cid-version=1.
  • Fetched each new CID through the itk.mypinata.cloud, ipfs.io, and
    ipfs.filebase.io gateways and confirmed the returned bytes match the
    expected SHA512.

Commits

  1. ENH — swap the CIDs and gateway list.
  2. STYLE — format with gersemi 0.19.3 using ITK main's profile (indent 2,
    line length 80, list_expansion: favour-expansion). Whitespace-only: the
    non-whitespace token stream is byte-identical to commit 1, so the file matches
    main's formatting and forward-merges cleanly.

🤖 Generated with Claude Code

thewtex and others added 2 commits July 3, 2026 15:16
Re-add the vendored SWIG prebuilt binary archives (Windows/Linux/macOS),
the SWIG source tarball, and the PCRE source tarball to IPFS using the
kubo `unixfs-v1-2025` import profile (CIDv1, raw leaves, 1 MiB chunks,
1024 links/node, balanced layout). Each archive is now added as a file
so its CID resolves to the archive bytes directly, rather than to a
UnixFS directory that wraps the file. The download URLs therefore drop
the trailing `/<filename>` path segment and reference the CID directly.

Refresh the content-addressed mirror list to the gateways currently
serving the ITK content:

  https://itk.mypinata.cloud/ipfs/
  https://ipfs.io/ipfs/
  https://gateway.pinata.cloud/ipfs/
  https://insightsoftwareconsortium.github.io/ITKTestingData/CID/
  https://ipfs.filebase.io/ipfs/

The data.kitware.com SHA512 endpoint is retained as the primary source,
and URL_HASH continues to validate whichever mirror responds. The
archives are pinned on the itk-filebase and itk-pinata remote pinning
services and mirrored in the ITKTestingData repository under CID/.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Apply the main-branch gersemi profile (0.19.3; indent 2, line length 80,
list_expansion favour-expansion) to Wrapping/Generators/SwigInterface/
CMakeLists.txt. This is a whitespace-only reflow with no change to any
CMake token, so the file matches ITK's main-branch formatting and merges
cleanly forward.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added type:Infrastructure Infrastructure/ecosystem related changes, such as CMake or buildbots type:Enhancement Improvement of existing methods or implementation labels Jul 3, 2026
@greptile-apps

greptile-apps Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR updates the SWIG and PCRE archive mirrors used by the Python wrapping toolchain. The main changes are:

  • Replaces directory-style IPFS archive CIDs with direct-file CIDs.
  • Updates SWIG and PCRE mirror URLs to use ${gateway}${cid} without filename suffixes.
  • Keeps the Kitware SHA512 URL as the primary source and preserves URL_HASH SHA512 validation.
  • Reformats Wrapping/Generators/SwigInterface/CMakeLists.txt without changing the wrapping setup flow.

Confidence Score: 5/5

Safe to merge with minimal risk.

The functional changes are limited to content-addressed mirror CIDs and gateway URLs, and archive downloads remain protected by existing SHA512 hashes.

No files require special attention.

T-Rex T-Rex Logs

What T-Rex did

  • Ran the direct-file IPFS CIDs validation script (trex-artifacts/direct-file-ipfs-cids-validation-script.py) and captured before and after evidence showing CID-based URLs and hash verification results.
  • Executed the CMake harness to generate ExternalProject URLs and captured before and after outputs showing the transition from Kitware-first URLs to direct CID mirror URLs with the same SHA512 and cache behavior.

View all artifacts

T-Rex Ran code and verified through T-Rex

Important Files Changed

Filename Overview
Wrapping/Generators/SwigInterface/CMakeLists.txt Updates SWIG and PCRE ExternalProject mirror URLs to direct-file IPFS CIDs while retaining SHA512 verification; remaining edits are formatting-only.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
participant CMake as CMake configure
participant EP as ExternalProject_Add
participant Kitware as data.kitware.com SHA512 URL
participant IPFS as IPFS/GitHub Pages mirrors
participant Hash as URL_HASH SHA512

CMake->>EP: Select SWIG/PCRE archive for host platform
EP->>Kitware: Try primary archive download
alt Primary unavailable
    EP->>IPFS: Try direct-file CID mirror URLs
end
EP->>Hash: Validate downloaded bytes against SHA512
Hash-->>EP: Accept archive only on hash match
Loading
%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
participant CMake as CMake configure
participant EP as ExternalProject_Add
participant Kitware as data.kitware.com SHA512 URL
participant IPFS as IPFS/GitHub Pages mirrors
participant Hash as URL_HASH SHA512

CMake->>EP: Select SWIG/PCRE archive for host platform
EP->>Kitware: Try primary archive download
alt Primary unavailable
    EP->>IPFS: Try direct-file CID mirror URLs
end
EP->>Hash: Validate downloaded bytes against SHA512
Hash-->>EP: Accept archive only on hash match
Loading

Reviews (1): Last reviewed commit: "STYLE: Format SwigInterface CMakeLists.t..." | Re-trigger Greptile

@dzenanz

dzenanz commented Jul 3, 2026

Copy link
Copy Markdown
Member

Azure Windows failure:

[1305/5903] Performing download step (download, verify and extract) for 'ClangFormat'

FAILED: [code=1] ClangFormat-prefix/src/ClangFormat-stamp/ClangFormat-download D:/a/1/s-build/ClangFormat-prefix/src/ClangFormat-stamp/ClangFormat-download 
C:\Windows\system32\cmd.exe /C "cd /D D:\a\1\s-build\ClangFormat-prefix\src && "C:\Program Files\CMake\bin\cmake.exe" -P D:/a/1/s-build/ClangFormat-prefix/src/ClangFormat-stamp/ClangFormat-download-MinSizeRel.cmake && "C:\Program Files\CMake\bin\cmake.exe" -E touch D:/a/1/s-build/ClangFormat-prefix/src/ClangFormat-stamp/ClangFormat-download"
CMake Error at ClangFormat-stamp/ClangFormat-download-MinSizeRel.cmake:49 (message):
  Command failed: 1

   'C:/Program Files/CMake/bin/cmake.exe' '-Dmake=' '-Dconfig=' '-P' 'D:/a/1/s-build/ClangFormat-prefix/src/ClangFormat-stamp/ClangFormat-download-MinSizeRel-impl.cmake'

  See also

    D:/a/1/s-build/ClangFormat-prefix/src/ClangFormat-stamp/ClangFormat-download-*.log


[1306/5903] Linking CXX shared library bin\ITKStatistics-5.4.dll
[1307/5903] Performing download step (git clone) for 'KWStyle'
ninja: build stopped: subcommand failed.
Command exited with the value: 1
MakeCommand:"C:\Program Files\CMake\bin\cmake.exe" --build . --config "MinSizeRel"
Error(s) when building project
   1 Compiler errors
   1 Compiler warnings
========== BUILD ERRORS (1) ==========
  CMake Error at ClangFormat-stamp/ClangFormat-download-MinSizeRel.cmake:49 (message):
========== BUILD WARNINGS (1) ==========
  *** WARNING non-zero return value in ctest from: C:\Program Files\CMake\bin\cmake.exe
====================================================

@thewtex

thewtex commented Jul 3, 2026

Copy link
Copy Markdown
Member Author

/azp run ITK.Windows

@thewtex

thewtex commented Jul 3, 2026

Copy link
Copy Markdown
Member Author

@hjmjohnson @CavRiley download failures were noted with the conda-forge builds -- increasing robustness with this patch

@hjmjohnson hjmjohnson merged commit e0912ed into InsightSoftwareConsortium:release-5.4 Jul 4, 2026
19 checks passed
@thewtex thewtex deleted the release-5.4-swig-binaries-ipfs branch July 6, 2026 17:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type:Enhancement Improvement of existing methods or implementation type:Infrastructure Infrastructure/ecosystem related changes, such as CMake or buildbots

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants