You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: content/blog/2021/08/14-progress-report.md
+10-10Lines changed: 10 additions & 10 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,11 +8,11 @@ author = "marcan"
8
8
9
9
It's been a long time since the last update! In all honesty, the first Progress Report set the bar a little bit too high, and I found it difficult to sit down and put together monthly reports that would do it justice. So, going forward, we're going to be providing shorter-form updates while striving to keep a monthly schedule.
10
10
11
-
That said, a lot has happened in the past few months, so strap on for a bigger update this time!
11
+
That said, a lot has happened in the past few months, so strap in for a bigger update this time!
12
12
13
13
## Core bring-up upstreamed into Linux 5.13
14
14
15
-
The core bring-up work that we detailed in our [our first progress report](/2021/03/progress-report-january-february-2021/) was upstreamed and released on June 27 as part of Linux 5.13! This is not very useful for end-users at this early stage, but it represents months of work laying down a foundation and figuring out how to solve certain difficult problems in a way that is acceptable to the upstream kernel community. This has also gotten members of the kernel community interested in our project; having a good relationship with kernel veterans is critical to ensuring we can work together to keep things upstreamed as development moves forward.
15
+
The core bring-up work that we detailed in our [our first progress report](/2021/03/progress-report-january-february-2021/) was upstreamed and released on June 27 as part of Linux 5.13! This is not very useful for end-users at this early stage, but it represents months of work laying down a foundation and figuring out how to solve certain difficult problems in a way that is acceptable to the upstream kernel community. This has also gotten members of the kernel community interested in our project. This is important, as having a good relationship with kernel veterans is critical to ensuring we can work together to keep things upstreamed as development moves forward.
16
16
17
17
## Hardware Reverse Engineering with the m1n1 Hypervisor
18
18
@@ -28,25 +28,25 @@ This is very different from a typical virtual machine, which is designed to run
28
28
{{< tweet 1397963184959418370 >}}
29
29
{{< /captioned >}}
30
30
31
-
Since the hypervisor is built on m1n1, it works together with Python code running on a separate host machine. Effectively, the Python host can "puppeteer" the M1 and its guest OS remotely, and the hypervisor itself is partially written in Python! This allows us to have a very fast test cycle, even updating parts of the hypervisor itself live during guest execution, without a reboot.
31
+
Since the hypervisor is built on m1n1, it works together with Python code running on a separate host machine. Effectively, the Python host can "puppeteer" the M1 and its guest OS remotely. The hypervisor itself is partially written in Python! This allows us to have a very fast test cycle, and we can even update parts of the hypervisor itself live during guest execution, without a reboot.
32
32
33
33
The hypervisor also includes standard debugging tools (like stopping execution, single-stepping, and getting a backtrace). This makes it not just useful for reverse engineering, but also as a low-level debugging tool for m1n1 itself and Linux, since they can also run on the hypervisor. Yes, you can now run m1n1 on m1n1!
34
34
35
35
On top of the hypervisor, we've built a flexible hardware I/O tracing framework that allows us to seamlessly load and upload tracers that understand how a particular piece of hardware works. For example, the [tracer](https://github.com/AsahiLinux/m1n1/blob/main/proxyclient/m1n1/trace/gpio.py) for the GPIO (General Purpose I/O) hardware can tell us when macOS toggles the state or configuration of each GPIO pin. This allows us to build up our understanding of the hardware, from raw register reads and writes to higher level functions. This was invaluable for the next bit of hardware we tackled: the DCP.
36
36
37
37
## Reverse Engineering DCP
38
38
39
-
One of the biggest challenges for Asahi Linux is making the M1's GPU work. But what most people think of as a "GPU" is actually two completely distinct pieces of hardware: the GPU proper, which is in charge of rendering images, and the display controller, which is in charge of sending those rendered images to the display.
39
+
One of the biggest challenges for Asahi Linux is making the M1's GPU work. But what most people think of as a "GPU" is actually two completely distinct pieces of hardware: the GPU proper, which is in charge of rendering frames in memory, and the display controller, which is in charge of sending those rendered frames from memory to the display.
40
40
41
41
While Alyssa has been hard at work [reverse engineering](https://rosenzweig.io/blog/asahi-gpu-part-4.html) the userspace components of the GPU, from draw calls to shaders, we still haven't looked at the lowest levels of the hardware that handle memory management and submission of commands to the GPU. But before we can use the GPU to render anything, we need a way to put it on the screen! Up until now, we've been using the firmware-provided framebuffer, which is just an area of memory where we can write pixels to be shown on the screen, but this won't cut it for a real desktop. We need features such as displaying new frames without tearing, support for hardware sprites such as the mouse cursor, switching resolutions and configuring multiple outputs, and more. This is the job of the display controller.
42
42
43
-
On most mobile SoCs, the display controller is just a piece of hardware with simple registers. While this is true on the M1 as well, Apple decided to throw a twist into it. They added a coprocessor to the display engine (called DCP), which runs its own firmware (initialized by the system bootloader), and moved most of the display driver into the coprocessor. But instead of doing it at a natural driver boundary... they took half of their macOS C++ driver, moved it into the DCP, and created a remote procedure call interface so that each half can call methods on C++ objects on the other CPU! Talk about overcomplicating things...
43
+
On most mobile SoCs, the display controller is just a piece of hardware with simple registers. While this is true on the M1 as well, Apple decided to give it a twist. They added a coprocessor to the display engine (called DCP), which runs its own firmware (initialized by the system bootloader), and moved most of the display driver into the coprocessor. But instead of doing it at a natural driver boundary... they took half of their macOS C++ driver, moved it into the DCP, and created a remote procedure call interface so that each half can call methods on C++ objects on the other CPU! Talk about overcomplicating things...
44
44
45
45
Reverse engineering this is a huge challenge, but thanks to the hypervisor, we can build up our understanding of how this all works layer by layer. At the lowest layer, DCP is an instance of what apple calls an "ASC", which is their term for these coprocessors (the M1 has about a dozen!). ASC processors run their own firmware and communicate with the main CPU through a mailbox interface, which is a simple message queue where each side can send 64-bit messages to the other, tagged with an "endpoint".
46
46
47
-
Above this simple interface, Apple uses a shared set of endpoints for all ASC processors that run RTKit, Apple's bespoke RTOS. This interface provides features such as sending syslog messages and crash dumps from the ASC to the main CPU, and initalizing endpoints (services). So we built a [tracer](https://github.com/AsahiLinux/m1n1/blob/main/proxyclient/m1n1/trace/asc.py) that can understand these messages, and do things like print the syslog messages directly to the hypervisor console.
47
+
Above this simple interface, Apple uses a shared set of endpoints for all ASC processors that run RTKit, Apple's bespoke RTOS. This interface provides features such as sending syslog messages and crash dumps from the ASC to the main CPU, and initializing endpoints (services). So we built a [tracer](https://github.com/AsahiLinux/m1n1/blob/main/proxyclient/m1n1/trace/asc.py) that can understand these messages, and do things like print the syslog messages directly to the hypervisor console.
48
48
49
-
On top of this, the DCP implements multiple endpoints, one of which serves as the "main" inteface. This interface itself supports making remote method calls in both directions. The DCP often issues synchronous callbacks to the main CPU after it receives a call, and the main CPU can in turn issue more synchronous DCP calls - effectively, the execution call stack extends across the CPU-to-DCP boundary! The interface even supports asynchronous reentrancy, having multiple "channels" so that, for example, the DCP can send asynchronous messages to the main CPU at any time, even during another options.
49
+
On top of this, the DCP implements multiple endpoints, one of which serves as the "main" interface. This interface itself supports making remote method calls in both directions. The DCP often issues synchronous callbacks to the main CPU after it receives a call, and the main CPU can in turn issue more synchronous DCP calls - effectively, the execution call stack extends across the CPU-to-DCP boundary! The interface even supports asynchronous reentrancy, having multiple "channels" so that, for example, the DCP can send asynchronous messages to the main CPU at any time, even during another operation.
50
50
51
51
The method calls themselves send their arguments and return data via buffers in shared memory. These buffers encode simple types like integers; pointers that can pass data in the input, output, or both directions; more complex fixed structures; and even two *different* serialization formats for JSON-like higher-level data structures (blame IOKit)! Our [dcp tracer](https://github.com/AsahiLinux/m1n1/blob/main/proxyclient/hv/trace_dcp.py) takes these buffers and dumps them out to a trace file, so that they can be analyzed offline as we improve our understanding of the protocol.
52
52
@@ -129,11 +129,11 @@ In case you're wondering, no, we can't write our own firmware, as it is loaded b
129
129
130
130
Those of you adventurous enough to try to test out m1n1 and our Linux patches by yourselves will have likely run into our [Developer Quickstart](https://github.com/AsahiLinux/docs/wiki/Developer-Quickstart) guide. It describes a dense and tedious manual installation process.
131
131
132
-
In order for an OS to be bootable on Apple Silicon machines, it has to "look" like a real macOS installation. This means it has to be an APFS container with multiple volumes within it, containing specific directory structuers and files. Until now, the simplest way of doing this was to actually install macOS a second time in a separate partition, and then replace its kernel with m1n1. This is, needless to say, a major pain in the ass, as the installation process is fairly slow. It also wastes around 70GB of disk space, which is how much you need for an upgradable macOS install. It also makes it difficult to install a specific macOS version, which is going to become a problem once we start requiring the usage of specific firmware bundles. This clearly won't cut it for anything beyond early development.
132
+
In order for an OS to be bootable on Apple Silicon machines, it has to "look" like a real macOS installation. This means it has to be an APFS container with multiple volumes within it, containing specific directory structures and files. Until now, the simplest way of doing this was to actually install macOS a second time in a separate partition, and then replace its kernel with m1n1. This is, needless to say, a major pain in the ass, as the installation process is fairly slow. It also wastes around 70GB of disk space, which is how much you need for an upgradable macOS install. It also makes it difficult to install a specific macOS version, which is going to become a problem once we start requiring the usage of specific firmware bundles. This clearly won't cut it for anything beyond early development.
133
133
134
134
What we need is a way to create a "macOS" installation from scratch, containing the required bootloader components and metadata for Apple's tooling to recognize it as a bootable OS. It doesn't have to contain an actual macOS root filesystem, but it does require these components:
135
135
136
-
* Version information and metadata fles
136
+
* Version information and metadata files
137
137
* iBoot2 and its configuration, including the device tree
138
138
* OS-paired firmware (for things like DCP, AGX, etc)
139
139
* The macOS recovery environment (a ~1GB macOS system image)
@@ -267,7 +267,7 @@ This will eventually be available at our short domain `https://alx.sh`, but it i
267
267
268
268
## More kernel drivers
269
269
270
-
Sven has been dutifully working on the Linux driver for DART, the M1's IOMMU (I/O Memory Management Unit). This driver is required to make all kinds of hardware work, like PCIe, USB, DCP, and more. It has just been accepted by upstream and is now on its way to Linux 5.15!
270
+
Sven has been dutifully working on the Linux driver for [DART](https://github.com/AsahiLinux/docs/wiki/Glossary#d), the M1's IOMMU (I/O Memory Management Unit). This driver is required to make all kinds of hardware work, like PCIe, USB, DCP, and more. It has just been accepted by upstream and is now on its way to Linux 5.15!
271
271
272
272
With this driver in, we can now make USB and PCIe work with minimal additional patches and drivers. There are various other dependencies (GPIO for miscellaneous things, I²C for proper USB-PD support, SPI for touchpad/keyboard support on the laptops, and NVMe support patches) that are spread around in various trees that people have been working on. Next we'll direct our focus towards polishing these simpler drivers and putting together a clean, working reference tree that we can use to continue development and provide new developers with a stable foundation. With the current state of things, it's already possible to use Asahi Linux as a development machine with a (non-accelerated) GUI, although things are still rough around the edges. Upstreaming these changes will require a bit more time, as there are some bureaucratic yaks to be shaved around how to properly implement these (technically simple) drivers, but things shouldn't take too long!
0 commit comments