Add aarch64 support by ludfjig · Pull Request #32 · hyperlight-dev/cargo-hyperlight

ludfjig · 2026-03-10T17:44:20Z

Adds support for compiling aarch64, with some fixes for cross compiling from x86_64 as well

syntactically · 2026-05-26T19:11:22Z

@ludfjig I rebased this PR and added some minor fixes used by the main hyperlight PR.

Looks like CI is passing, so hopefully we can merge that soon.

Copilot

Pull request overview

This PR adds support for building Hyperlight guest binaries targeting aarch64-hyperlight-none, and adjusts argument forwarding / toolchain behavior to improve cross-compilation correctness.

Changes:

Add a new aarch64-hyperlight-none Rust target spec derived from aarch64-unknown-none, and update target-spec generation to pass --target explicitly.
Adjust clang/cflags generation to use an aarch64 --target and make -mno-red-zone x86_64-only.
Improve cargo invocation behavior by avoiding canonicalize() on the cargo path and filtering --target / --target-dir from forwarded CLI args; update docs/CI/justfile for multi-arch support.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
`src/toolchain.rs`	Adds aarch64-aware clang target selection and changes libc header preparation logic.
`src/sysroot.rs`	Introduces `aarch64-hyperlight-none` target-spec customization and adjusts how base target specs are queried.
`src/command.rs`	Filters `--target` / `--target-dir` from forwarded args to avoid overriding resolved env-based target settings.
`src/cargo_cmd.rs`	Stops canonicalizing the cargo binary path to avoid resolving rustup symlinks.
`README.md`	Documents default target selection on ARM and updates output path description.
`justfile`	Uses `arch()` to make the guest path target-arch independent.
`.github/workflows/ci.yml`	Adds concurrency cancellation and gates KVM / guest execution based on runner architecture.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

ludfjig

LGTM but I don't have approval rights to my own pr :)

canonicalize() resolves the cargo -> rustup symlink, causing commands like 'cargo rustc' to be invoked as 'rustup rustc' which doesn't understand cargo-specific arguments. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com> Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

User CLI args like --target aarch64-unknown-none were passed through to the final cargo build command, overriding the resolved CARGO_BUILD_TARGET env var set by populate_from_args. Filter these out since the resolved hyperlight target is set via environment variable. Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com> Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

- Add aarch64-hyperlight-none target spec derived from aarch64-unknown-none with hyperlight customizations - Select correct musl arch headers (aarch64 vs x86_64) - Use appropriate clang --target for aarch64 - Make -mno-red-zone x86_64-only (aarch64 has no red zone) - Fix get_spec to pass --target as CLI arg instead of env var Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com> Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

- Add concurrency group to cancel stale CI runs - Gate KVM setup on X64 runners - Gate run-guest on X64 (no host support on ARM yet) - Update README to mention aarch64 target - Use arch() in justfile for target-independent paths Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com> Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

`Command::exec()` is never used anymore, since its use precludes running on Windows. `CargoCmd::resolve_env()` is not used, because the essentially-identical `Command::resolve_env()` is used instead. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

This started with the observation that we didn't actually build libc headers properly on aarch64, leading to the realisation that we weren't using cargo-hyperlight to build C guests (or anything that depended on most of libc), so all that code was essentially untested. cargo-hyperlight now supports building the guest CAPI library, and can directly output the appropriate cflags/ldflags/libs needed to build and link a C guest. On top of that, it can now build a `hyperlight-config` executable into its produced sysroots which provides the same information, as well as a clang wrapper that uses the same logic for ease of use. This is done in order to allow downstream consumers to build C guests using the same logic as cargo-hyperlight without having to build/install the tool themselves, removing opportunities for e.g. target features to get out of sync. Finally, `cargo-hyperlight build-c-sysroot --sysroot-dir </path/to/sysroot>` will now copy just the final build artifacts (header files, libraries, and the aforementioned hyperlight-config and clang wrapper executables) into a new directory tree, which makes it easy to build a redistributable artifact that bundles everything downstreams need to build C guests (including both build logic and libraries/interfaces themselves). Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Previously, the CI checks that cargo-hyperlight can actually build a guest ran on the GH hosted runner ubuntu-latest, which are poorly-specified and not particularly reflective of the environments where cargo-hyperlight is actually called upon. Switch to using the same images that we do for Hyperlight CI. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

…des) Previously, the sysroot build copied include files from several different directories into the final sysroot include dir, but did not do anything to reset the state of the include directory. If the same sysroot dir was used to build sysroots from multiple different versions of hyperlight-libc, since the headers provided are different, downstream code would get a confusing (and likely to fail to compile) mix of the include files from the two libcs. This changes the way that the include directory (and other copied sysroot directories) are constructed, to ensure that stale files are removed as well as new files being copied. In a somewhat-related issue, when building a C sysroot, cargo-hyperlight needs to build hyperlight-libc (as part of hyperlight-guest-capi), and it turns out that that build was itself getting the output include directory, which meant that rebuidls could also result in stale include files being added. This commit changes the logic involved in building hyperlight-guest-capi to ensure that the output sysroot include directory (which is not yet populated) isn't on the include path. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

syntactically · 2026-06-25T01:57:54Z

@jprendes I think this should be properly ready to merge now as all the odd only-reproducible-in-CI issues seem to be fixed; would appreciate if you took a quick look at it. It more or less follows the plan we talked about earlier.

jprendes

I think I would have preferred the aarch64 support and the clang wrapper to be different PRs, I think most of the changes in the PR stem from the clang wrapper and associated refactors rather than from adding the aarch64 target. But let's keep it this way now.

jprendes · 2026-06-25T08:20:00Z

+        "--target=x86_64-unknown-linux-none"
+    };
+
+    let common_flags: &[&str] = &[


Suggested change

let common_flags: &[&str] = &[

const COMMON_FLAGS: &[&str] = &[

...

];

const X86_64_FLAGS: &[&str] = &[

...

];

const AARCH64_FLAGS: &[&str] = &[

...

];

let all_flags = if args.target.starts_with("aarch64") {

[AARCH64_FLAGS, COMMON_FLAGS]

} else if args.target.starts_with("x86_64") {

[X86_64_FLAGS, COMMON_FLAGS]

};

let mut flags = OsString::new();

for flag in all_flags.iter().copied().flatten() {

flags.push(*flag);

flags.push(" ");

}

Fixed in 14ef629

jprendes · 2026-06-25T09:39:38Z

+            return exe_path;
+        }
+    }
+    panic!("Could not base system implementation of {}", tool_name);


question: is "base" a typo here? "find" maybe?

Thanks, fixed in 1f8553d

jprendes · 2026-06-25T10:04:52Z

+    panic!("Could not base system implementation of {}", tool_name);
+}
+
+fn clang(args: &Args, tool_name: &str, cmd: env::ArgsOs) {


The logic in this function is quite involved, could you add some comments? maybe with example invokations that we are trying to handle in each case?

Does the extra comment in 6bd4e9c look good to you?

jprendes · 2026-06-25T11:01:30Z

+            .ok_or(anyhow!(
+                "Could not find hyperlight-libc or hyperlight-guest-bin package in cargo metadata"
+            ))


Using the anyhow::Context trait you can do

Suggested change

.ok_or(anyhow!(

"Could not find hyperlight-libc or hyperlight-guest-bin package in cargo metadata"

))

.context("Could not find hyperlight-libc or hyperlight-guest-bin package in cargo metadata")

Thanks, 4d1b59b

jprendes · 2026-06-25T12:24:40Z

    if !args.target.starts_with("aarch64") {
-        // Detect which libc variant is present: picolibc or legacy musl
-        let include_dirs: &[&str] = &[
-            // directories for musl
-            "third_party/printf/",
-            "third_party/musl/include",
-            "third_party/musl/arch/generic",
-            "third_party/musl/arch/x86_64",
-            "third_party/musl/src/internal",
-            // directories for picolibc
-            "third_party/picolibc/libc/include",
-            "third_party/picolibc/libc/stdio",
-            "include",
-        ];
-
-        for dir in include_dirs {
-            let include_src_dir = libc_dir.join(dir);
-            let files = glob::glob(&format!("{}/**/*.h", include_src_dir.display()))
-                .context("Failed to read include source directory")?;
-            for file in files {
-                let src = file.context("Failed to read include source file")?;
-                let dst = src.strip_prefix(&include_src_dir).unwrap();
-                let dst = include_dst_dir.join(dst);
-
-                std::fs::create_dir_all(dst.parent().unwrap())
-                    .context("Failed to create include subdirectory")?;
-                std::fs::copy(&src, &dst).context("Failed to copy include file")?;
-            }
-        }
+        include_dirs.push("third_party/musl/arch/x86_64");


I think musl was never supported with aarch64. We never had a release using musl and with aarch64 support simultaneously.

Indeed, that is why there is no code path here for musl aarch64 arch headers.

ah, sorry, I should have been clearer. If the path doesn't exist it's silently ignored (that's how the glob crate does it), so you can just leave that one in the main list, and not worry about the including it for one case or another.
Just a nit, this is fine too.

jprendes · 2026-06-25T12:32:03Z

+*~
+\#*\#
+.\#*
+.*.swp


what are these?

Thanks for catching that, that was unintentional (editor temporary files). Happy to move to my local .git/info/exclude if you would rather not have them in the checked-in .gitignore, although I thought we had equivalent in hyperlight itself.

mainly curious, you can leave them here if you want :-)

Removed in 93ddc62

jprendes · 2026-06-25T12:40:32Z

+    runs-on: ${{ fromJson(
+        format('["self-hosted", "{0}", "X64", "1ES.Pool=hld-{1}-amd", "JobId=cargo-hyperlight-{2}-{3}-{4}"]',
+          matrix.os == 'windows-latest' && 'Windows' || 'Linux',
+          matrix.os == 'windows-latest' && 'win2025' || 'kvm',
+          github.run_id,
+          github.run_number,
+          github.run_attempt)) }}


Was there any failure running on GH runners?
I think it's nice having a different runner than hl-core, as that allows us to not overly specify our dev setup to what's in the CI images.
i.e., I like that GH runners are fairly vanilla and we can still build on them.

I moved it briefly because the GH runners were completely impossible to debug anything on, since we don't have the image for them. We also for example have no idea how llvm is set up on them, which is quite irritating.

Then @simongdavies said he would prefer it to stay on the HL runners to make sure that we catch early if HL CI will fail.

You can use act to reproduce GH CI locally. I've had very good experience with it.
Can we add them as extra runners so that we cover @simongdavies 's concerns? Maybe running a smoke test on the 1ES pools?

jprendes · 2026-06-25T12:56:52Z


-pub(crate) fn copy_glob_with_predicate(
-    src_dir: &Path,
+pub(crate) fn union_glob_with_predicate(


so, unlike copy_* union_* also removes files that are there but weren't copied over? How about calling it "sync" (inspired by rsync)? or is it union because src is now an iterator? maybe "sync_many"? otherwise a comment explaining the expected behaviour would be nice :-)

Yes, that is the behaviour. I used union as the name because it makes dst into the union of the directory trees at the srcs (think e.g. union mounts). I'm happy to change it to sync_several or something like that.

haha, for me union gives me the idea that it will be the union of the src and the dst 😂

Still happy to rename it if you want, but perhaps it is clear enough with the comment?

jprendes · 2026-06-25T13:11:46Z

-        let dst = src.strip_prefix(src_dir).unwrap();
-        if !predicate(dst) {
-            continue;
+    let mut copied = HashSet::new();


Could we create another folder next to the destination folder, populate that one, then rename the old folder, then move the new folder, then delete the old folder? Something like...

cp $FILES /path/to/dest_new mv /path/to/dest /path/to/dest_old mv /path/to/dest_new /path/to/dest rm /path/to/dest_old

When this was done in the builscript, there was an important reason to do it this way instead of say...

rm /path/to/dest cp $FILES /path/to/dest

and it was that the include directory could be used by some C crate using the wrapper while rust was compiling the hl_guest_bin crate which would be writing to the same directory, e.g.:

hl_js -> rquickjs -> quickjs

First we had to build hl_guest_bin so that it would generate the C includes, then we had to build again the whole thing. In that case cargo could have been building hl_guest_bin and rquickjs at the same time (as far as cargo knows there's no dependency between them). So building hl_guest_bin would have deleted the includes folder while rquickjs would have been trying to build quickjs using that include dir (which we told it to through the CFLAGS env var).
With cargo-hyperlight, is there still value in doing it this way? or can we simplify the logic?

I think that approach would work here, subject to some assumptions about non-concurrency that are probably? safe in the cargo-hyperlight approach. I think this is probably slightly more efficient, especially on Windows where file operations are slow (since it is run every time anyone runs a cargo hyperlight command, this might matter), and it doesn't seem like terribly much complexity.

…. includes) Add more documentation on the `union_glob` family of utility functions Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Fix typo Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Add comment describing how the clang wrapper decides which arguments to add. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Address review feedback on structure of cflags computation Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Address review feedback on the use of anyhow! Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Remove gitignore file that was accidentally included Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

jprendes

Just a small question

jprendes · 2026-06-25T14:17:07Z

+    // - `found_non_flag`: If true, there was at least one non-flag
+    //   argument, so the driver will actually be compiling
+    //   something. This is important to catch someone running
+    //   e.g. `clang -v` and avoid inserting the linker arguments,


Would clang -v trigger the found_non_flag = true? I think it wouldn't?
Also, what's with the special handling of "--" and flags_valid?
Can we stop iterating once flags_valid is set?

Yes, clang -v has found_non_flag = false, i.e. there were 0 non-flag arguments, so we do not include the ldflags.

We keep going after flags_valid is set to false because clang -v -- should have that behaviour, but clang -v -- src.c should not. We could bail once both flags_valid is false and found_non_flag is true, but that seemed like an unnecessary optimisation.

ludfjig force-pushed the aarch64 branch 3 times, most recently from 04c7534 to ddedf1c Compare March 10, 2026 18:39

syntactically force-pushed the aarch64 branch from ddedf1c to 28ac7b5 Compare May 26, 2026 18:29

ludfjig requested a review from Copilot May 27, 2026 17:41

Copilot started reviewing on behalf of ludfjig May 27, 2026 17:41 View session

Copilot AI reviewed May 27, 2026

View reviewed changes

Comment thread src/toolchain.rs Outdated

Comment thread src/command.rs

Comment thread src/cargo_cmd.rs

ludfjig marked this pull request as ready for review June 10, 2026 16:16

syntactically force-pushed the aarch64 branch 4 times, most recently from 8506520 to baf9c31 Compare June 23, 2026 05:11

ludfjig commented Jun 23, 2026

View reviewed changes

syntactically force-pushed the aarch64 branch 13 times, most recently from cf06e31 to c1d9c89 Compare June 25, 2026 00:42

ludfjig added 3 commits June 25, 2026 02:44

ludfjig and others added 7 commits June 25, 2026 02:45

Update examples to 0.15.0

d386f03

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

Tweak the search logic for ar for better macOS compatibility

fe8d7f7

Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

syntactically force-pushed the aarch64 branch from de555d3 to db8c8ad Compare June 25, 2026 01:47

syntactically previously approved these changes Jun 25, 2026

View reviewed changes

jprendes reviewed Jun 25, 2026

View reviewed changes

syntactically added 6 commits June 25, 2026 15:04

fixup! Do a better job of cleaning up copied sysroot directories (e.g…

20173cf

…. includes) Add more documentation on the `union_glob` family of utility functions Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

fixup! Better support for building C distributables and guests

1f8553d

Fix typo Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

fixup! Better support for building C distributables and guests

6bd4e9c

Add comment describing how the clang wrapper decides which arguments to add. Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

fixup! Better support for building C distributables and guests

14ef629

Address review feedback on structure of cflags computation Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

fixup! Better support for building C distributables and guests

4d1b59b

Address review feedback on the use of anyhow! Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

fixup! Better support for building C distributables and guests

93ddc62

Remove gitignore file that was accidentally included Signed-off-by: Lucy Menon <168595099+syntactically@users.noreply.github.com>

syntactically dismissed their stale review via 93ddc62 June 25, 2026 14:05

jprendes previously approved these changes Jun 25, 2026

View reviewed changes

syntactically dismissed jprendes’s stale review via 12ced53 June 25, 2026 14:33

syntactically force-pushed the aarch64 branch 2 times, most recently from 7e585d0 to e46f40a Compare June 25, 2026 14:36

syntactically added 2 commits June 25, 2026 15:39

fixup! Switch to using Hyperlight CI runner images

2b100db

fixup! Better support for building C distributables and guests

47f9b9e

syntactically force-pushed the aarch64 branch from e46f40a to 47f9b9e Compare June 25, 2026 14:40

-    let common_flags: &[&str] = &[
+    const COMMON_FLAGS: &[&str] = &[
+       ...
+    ];
+    const X86_64_FLAGS: &[&str] = &[
+       ...
+    ];
+    const AARCH64_FLAGS: &[&str] = &[
+       ...
+    ];
+    let all_flags = if args.target.starts_with("aarch64") {
+        [AARCH64_FLAGS, COMMON_FLAGS]
+    } else if args.target.starts_with("x86_64") {
+        [X86_64_FLAGS, COMMON_FLAGS]
+    };
+    let mut flags = OsString::new();
+    for flag in all_flags.iter().copied().flatten() {
+        flags.push(*flag);
+        flags.push(" ");
+    }

+              *~
+              \#*\#
+              .\#*
+              .*.swp

                
                    No newline at end of file

Uh oh!

Conversation

ludfjig commented Mar 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

syntactically commented May 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ludfjig left a comment

Choose a reason for hiding this comment

Uh oh!

syntactically commented Jun 25, 2026

Uh oh!

jprendes left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jprendes left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

ludfjig commented Mar 10, 2026 •

edited

Loading