drm/asahi: Add the Asahi driver for Apple AGX GPUs

drm/asahi: alloc: Support tagging array allocs

It's hard to tell what a given array buffer is just from the type, so
add support for explicitly adding a u32 tag. This can help us
differentiate between allocs in the debug codepaths or when dumping
memory.

To more easily debug GPU/FW-side overreads, use the alloc tag to fill
the padding instead of using a constant.

drm/asahi: buffer,render: Identify and provide layer meta buf

It looks like one of the "heapmeta" pointers is actually a layer
metadata pointer, that macOS just allocates contiguously with the
tilemap headers and heap meta buffers. Size seems to always be 0x100.

Let's allocate it after the heapmeta, which will make debugging easier.

drm/asahi: compute/render: Implement bindless samplers

drm/asahi: fw,queue: Implement helper programs

Also expose no preemption flag (?) separately.

drm/asahi: render: Identify and set Z/S strides for layered rendering

drm/asahi: Add verbose UAPI error reporting

drm/asahi: Identify and allocate clustered layering metadata buf

Turns out multi-cluster machines also need a clustered buffer for
layered rendering.

Fixes layered rendering on G13X with barriers (I guess if you don't
flush memory this stays in some kind of cache and somehow doesn't
matter?).

drm/asahi: Identify and implement helper config register

drm/asahi: alloc: Do not allocate memory to free memory

The existing garbage mechanism could allocate a relatively unbounded vec
when freeing garbage, which was hurting memory exhaustion scenarios.
The only reason we need that buffer is to move garbage out of the lock
so we can drop it without deadlocks. Replace it with a 128-size
pre-allocated garbage buffer, and loop around reusing it.

drm/asahi: Don't lock up when unmapping PTEs fails

If a bug causes PTEs to be unmapped twice, the unmap loop gets stuck
spamming WARNs forever. Just skip a page and try again so we can make
forward progress.

drm/asahi: Convert to GPUVM and implement more VM_BIND ops

drm/asahi: Refactor address types

VAs are u64, PAs and sizes are usize.

drm/asahi: util: Add RangeExt helpers for Range<T>

drm/asahi: mmu: Convert to using Range

drm/asahi: Move the unknown dummy page to the top of the address space

drm/asahi: Convert more ranges to Range<>

drm/asahi: mmu: Fix lockdep issues with GpuVm

drm/asahi: Implement GEM objects sharing a single DMA resv

drm/asahi: queue: Split into Queue and QueueInner

Work around mutability issues when entity.new_job() takes a mutable
reference to the entity by moving all the fields used by the
submit_render() and submit_compute() functions to an inner struct,
eliminating the double-mutable-borrow.

drm/asahi: file: Update to newer VM_BIND API

drm/asahi: Signal soft fault support to userspace

drm/asahi: Fix u32 mult overflow on large tilebufs/TPCs

drm/asahi: Fix event tracking when JobSubmission is dropped

drm/asahi: gpu: Show unknown field in timeouts

drm/asahi: Handle channel errors

drm/asahi: event: Initialize stamps to different values

Makes debugging a bit easier.

drm/asahi: workqueue: Fix "Cannot submit, but queue is empty?" bug

drm/asahi: Clean up jobs in a workqueue

This eliminates a potential deadlock under load and improves the fence
signaling situation (for when we have a shrinker).

drm/asahi: Add robust_isolation kernel parameter

This only allows binding one VM context at once, which serializes GPU
usage between VMs and therefore prevents one faulting VM from affecting
others.

drm/asahi: HACK: Disable compute preemption for now

Possibly because we don't have support in the helper program, this is
broken and causes channel errors. Hack in high priority for now, which
works around it.

Use debug_flags 0x1000000000000 to re-enable for testing.

drm/asahi: Align kernel range to buffer::PAGE_SIZE

We only require alignment to the UAT page size from userspace, but
internally we need more, so just align it if userspace gives us lower
alignment.

drm/asahi: Implement missing ASAHI_BIND_OP_UNBIND

Trivial now that we have GPUVM.

drm/asahi: Implement ASAHI_GET_TIME

drm/asahi: gpu: Force Box move with manual Box<T>::into_inner()

TODO: Investigate why this doesn't work automatically.

drm/asahi: gpu: Collect garbage for private/gpuro together

Avoids double firmware flushes

drm/asahi: alloc: Be more verbose about failures

drm/asahi: gpu: Add a max object count garbage limit

This ensures the garbage Vec does not grow beyond what is reasonable,
and probably reduces jank by doing more smaller GCs instead of big ones.

drm/asahi: Document timestamp ops better, refactor fields

drm/asahi: workqueue: Restrict command objects to only job commands

drm/asahi: gpu: Implement mapping timestamp buffers

drm/asahi: file: Implement ASAHI_GEM_BIND_OBJECT

drm/asahi: fw, queue: Add UserTimestamp object to job structs

drm/asahi: queue: Plumb through objects XArray and add timestamp getter

drm/asahi: fw, queue: Plumb through UserTimestamps -> TimestampPointers

drm/asahi: queue/render,compute: Plumb through timestamps extension

drm/asahi: file: Add user_timestamp_frequency_hz to params

drm/asahi: Set a bit for internal non-render barriers on G14X

drm/asahi: Add the USER_TIMESTAMPS feature

drm/asahi: mmu: Change step_remap() to new api

Fixes deadlock. Also fix missing TLB inval

drm/asahi: file: Reject gem_bind past the end of the object

drm/asahi: mmu: Fix 2x step_remap case

drm/asahi: workqueue: Defer freeing the last completed work item

Maybe helps with firmware crashes?

drm/asahi: mmu: Fix deadlock on remap ops

drm/asahi: mmu: Change step_remap() to new api

drm/asahi: mmu: UAT change for rust page table rewrite

Originally from:

arm64: dts: apple: Remove no-map from pagetables region

This should still be compatible with older kernels, since this region is
always mapped cached.

drm/asahi: debug: Add PgTable debug category

drm/asahi: mmu: Add some barriers

Just being paranoid.

drm/asahi: Implement ASAHI_BIND_SINGLE_PAGE (uapi)

drm/asahi: port to new UAPI

Signed-off-by: Asahi Lina <lina@asahilina.net>
Co-developed-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Signed-off-by: Alyssa Rosenzweig <alyssa@rosenzweig.io>
Signed-off-by: Janne Grunau <j@jannau.net>

Author: hoshinolina

drivers/gpu/drm/Kconfig

@@ -294,6 +294,8 @@ config DRM_VGEM
 
 source "drivers/gpu/drm/vkms/Kconfig"
 
+source "drivers/gpu/drm/asahi/Kconfig"
+
 source "drivers/gpu/drm/exynos/Kconfig"
 
 source "drivers/gpu/drm/rockchip/Kconfig"

drivers/gpu/drm/Makefile

@@ -215,6 +215,7 @@ obj-y			+= tiny/
 obj-$(CONFIG_DRM_PL111) += pl111/
 obj-$(CONFIG_DRM_TVE200) += tve200/
 obj-$(CONFIG_DRM_ADP) += adp/
+obj-$(CONFIG_DRM_ASAHI) += asahi/
 obj-$(CONFIG_DRM_XEN) += xen/
 obj-$(CONFIG_DRM_VBOXVIDEO) += vboxvideo/
 obj-$(CONFIG_DRM_LIMA)  += lima/

drivers/gpu/drm/asahi/Kconfig

@@ -0,0 +1,40 @@
+# SPDX-License-Identifier: GPL-2.0
+
+config RUST_DRM_SCHED
+	bool
+	select DRM_SCHED
+
+config RUST_DRM_GEM_SHMEM_HELPER
+	bool
+	select DRM_GEM_SHMEM_HELPER
+
+config RUST_DRM_GPUVM
+	bool
+	select DRM_GPUVM
+
+config DRM_ASAHI
+	tristate "Asahi (DRM support for Apple AGX GPUs)"
+	depends on RUST
+	depends on DRM=y
+	depends on (ARM64 && ARCH_APPLE) || (COMPILE_TEST && !GENERIC_ATOMIC64)
+	depends on MMU
+	depends on IOMMU_SUPPORT
+	depends on PAGE_SIZE_16KB
+	select RUST_DRM_SCHED
+	select RUST_DRM_GEM_SHMEM_HELPER
+	select RUST_DRM_GPUVM
+	select RUST_APPLE_RTKIT
+	help
+	  DRM driver for Apple AGX GPUs (G13x, found in the M1 SoC family)
+
+config DRM_ASAHI_DEBUG_ALLOCATOR
+	bool "Use debug allocator"
+	depends on DRM_ASAHI
+	help
+	  Use an alternate, simpler allocator which significantly reduces
+	  performance, but can help find firmware- or GPU-side memory safety
+	  issues. However, it can also trigger firmware bugs more easily,
+	  so expect GPU crashes.
+
+	  Say N unless you are debugging firmware structures or porting to a
+	  new firmware version.

drivers/gpu/drm/asahi/Makefile

@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0
+
+obj-$(CONFIG_DRM_ASAHI) += asahi.o