[nanvix-unstable] PEB scratch pointers use GVA instead of GPA when guest skips page-table generation #1403

Summary

When the nanvix-unstable feature is enabled, Hyperlight skips guest page-table generation and the guest runs with identity mapping (GVA == GPA). However, the PEB input_stack.ptr and output_stack.ptr fields are always written using scratch_base_gva() (derived from MAX_GVA), while the KVM memory slot for scratch is placed at scratch_base_gpa() (derived from MAX_GPA).

Before PR #1393, MAX_GPA == MAX_GVA == 0xFFFF_FFFF on i686 so this was not observable. After that fix lowered MAX_GPA to 0xFEDF_FFFF (to avoid the KVM APIC access page at 0xFEE00000), scratch_base_gva() and scratch_base_gpa() diverge and the guest reads PEB pointers that target unmapped physical addresses.

Affected code

src/hyperlight_host/src/mem/layout.rs — get_input_data_buffer_gva() and get_output_data_buffer_gva():

fn get_input_data_buffer_gva(&self) -> u64 {
    hyperlight_common::layout::scratch_base_gva(self.scratch_size)
}

pub(crate) fn get_output_data_buffer_gva(&self) -> u64 {
    hyperlight_common::layout::scratch_base_gva(self.scratch_size)
        + self.sandbox_memory_config.get_input_data_size() as u64
}

These are called from write_memory_layout() (lines ~765-782) to populate the PEB pointer fields that the guest reads at boot.

src/hyperlight_host/src/mem/mgr.rs — get_guest_memory_regions() (crashdump, i686-guest path) also uses scratch_base_gva() where it should use scratch_base_gpa() under identity mapping.

Suggested fix

Under #[cfg(feature = "nanvix-unstable")], the PEB pointer helpers should use scratch_base_gpa() instead of scratch_base_gva(), since no page tables translate GVA → GPA for the guest:

#[cfg(feature = "nanvix-unstable")]
fn get_input_data_buffer_gva(&self) -> u64 {
    // nanvix-unstable: guest runs with identity mapping (GVA == GPA),
    // no page tables translate scratch GVA → GPA.
    hyperlight_common::layout::scratch_base_gpa(self.scratch_size)
}

#[cfg(not(feature = "nanvix-unstable"))]
fn get_input_data_buffer_gva(&self) -> u64 {
    hyperlight_common::layout::scratch_base_gva(self.scratch_size)
}

And similarly for get_output_data_buffer_gva() and the crashdump path.

Alternatively, a more general approach would be to add a scratch_base_for_guest(size) helper in hyperlight_common::layout that dispatches to scratch_base_gpa() when nanvix-unstable is enabled and scratch_base_gva() otherwise, so all call sites are correct by construction.

Current workaround

Nanvix currently patches the PEB pointers at boot in the guest kernel (hyperlight_pre_kmain):

let gva_gpa_delta: u64 = (MAX_GVA - MAX_GPA) as u64;
if gva_gpa_delta != 0 {
    unsafe {
        (*peb_ptr).input_stack.ptr -= gva_gpa_delta;
        (*peb_ptr).output_stack.ptr -= gva_gpa_delta;
    }
}

This works but is fragile — any future change that adds more GVA-based pointers to the PEB would silently break the identity-mapped guest.

Context

• Hyperlight PR: don't let scratch region overlap with kvm apic page #1393 (APIC page overlap fix)
• Nanvix PR: [uservm] Model sparse physical memory layout for Hyperlight platform nanvix/nanvix#2082