`/dev/fd` lists the elfuse process's descriptors, not the guest's

[jotpalch]

Symptom

Measured on 2c4a8f4, macOS 15.6.1, Apple M1. Three runs of each, identical, and the same with --sysroot on the Alpine rootfs, which carries no /dev/fd:

held: 0 1 2 3 50, plus the descriptor each listing reads through
/dev/fd, read through 4, lists: 0 1 2 3 4 5 6 7 8 9
  of those, not openable as /dev/fd/N: 5 6 7 8 9
/proc/self/fd, read through 128, lists: 50 0 1 3 2
  of those, not openable as /dev/fd/N: none
/dev/fd is a directory

The /dev/fd listing is the descriptor table of the elfuse process on the host. It shares 0 through 4 with the guest's, but it omits 50, which the guest holds, and lists 5 through 9, which the guest cannot open by those names. /proc/self/fd lists the guest's table, less the descriptor it is read through (below).

Linux itself provides no /dev/fd, and the reference boot has none. Userspace creates it as a symlink to /proc/self/fd. With that symlink in place, Linux 6.18.54-0-virt through the qemu reference lane:

held: 0 1 2 3 50, plus the descriptor each listing reads through
/dev/fd, read through 4, lists: 0 1 2 3 4 50
  of those, not openable as /dev/fd/N: none
/proc/self/fd, read through 4, lists: 0 1 2 3 4 50
  of those, not openable as /dev/fd/N: none
/dev/fd is a symlink

Because the listing is host state, it moves with things the guest does not do. With three instances of test-path-fold, as first pushed to #398, running at once, two adjacent listings of /dev/fd differed in 11 of 45 runs. That turned Runtime (Release) red once on that PR: getdents /dev/fd/: 13 entries, want 12 entries.

Reproducer

#include <dirent.h>
#include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/stat.h>
#include <unistd.h>

/* Print what @dir lists, then which of the listed numbers do not open as
 * /dev/fd/N. The listing stays open while the numbers are tried, so its own
 * descriptor is one of the held ones.
 */
static void list(const char *dir) {
  DIR *d = opendir(dir);
  if (!d) {
    perror(dir);
    return;
  }
  int listed[64], n = 0;
  struct dirent *e;
  while ((e = readdir(d)) && n < 64)
    if (e->d_name[0] != '.')
      listed[n++] = atoi(e->d_name);

  printf("%s, read through %d, lists:", dir, dirfd(d));
  for (int i = 0; i < n; i++)
    printf(" %d", listed[i]);
  printf("\n  of those, not openable as /dev/fd/N:");
  int none = 1;
  for (int i = 0; i < n; i++) {
    char path[32];
    snprintf(path, sizeof(path), "/dev/fd/%d", listed[i]);
    int fd = open(path, O_RDONLY | O_NONBLOCK);
    if (fd >= 0) {
      close(fd);
    } else {
      printf(" %d", listed[i]);
      none = 0;
    }
  }
  printf("%s\n", none ? " none" : "");
  closedir(d);
}

int main(void) {
  int a = open("/", O_RDONLY | O_DIRECTORY);
  dup2(1, 50);
  printf("held: 0 1 2 %d 50, plus the descriptor each listing reads through\n",
         a);
  list("/dev/fd");
  list("/proc/self/fd");

  struct stat st;
  lstat("/dev/fd", &st);
  printf("/dev/fd is a %s\n", S_ISLNK(st.st_mode)   ? "symlink"
                              : S_ISDIR(st.st_mode) ? "directory"
                                                    : "?");
  return 0;
}

Mechanism

path_might_use_open_intercept admits every /dev name (src/syscall/path.c:91). proc_intercept_open (src/runtime/procemu.c:3727) hands it to intercept_open_dispatch, which has an arm for /dev/fd/<N> (src/runtime/procemu.c:2382) that dups the guest's descriptor N, and none for /dev/fd itself. The open of the directory falls through to the host, where macOS serves /dev/fd from devfs as a directory of the calling process's descriptors, which are elfuse's. The listing and the lookup therefore answer from two different descriptor tables.

Direction

Serving /dev/fd as the symlink Linux userspace creates, to /proc/self/fd, makes the listing the guest's and keeps it in agreement with /dev/fd/<N>, which already answers from the guest's table.

One difference would carry over, visible above. The synthetic /proc/self/fd does not list the descriptor the listing is read through (128 on elfuse, 4 on Linux). proc_open_fd_scratch (src/runtime/procemu.c:550) snapshots the table when the directory is opened, before that descriptor exists.