docker-in-docker: Feature request - Ability to pin containerd version

[glennschmidt]

On apt-based systems, the feature script installs the moby-engine package, which depends on moby-containerd >= 1.7.0, so it will end up just installing the latest version of containerd available in the repo.

This week, containerd 2.3.0 was released, and it broke my dev containers (containerd/containerd#13346). It would be good if there was a variable we could set to specify what version of moby-containerd to install, as a workaround when these types of issues occur.

[Falco20019]

This is what I did to downgrade (on Ubuntu 24.04), just in case someone else also needs it:
sudo apt-get update && sudo apt-get install -y --allow-downgrades moby-containerd=2.2.3-ubuntu24.04u1

You can use apt list --all-versions moby-containerd | grep 2.2.3 to find your variant.

[mafuba]

+1, hitting this on docker-in-docker Feature with moby-containerd 2.3.0-debian12u1 + moby-engine 29.4.2-debian12u1. Root cause is upstream containerd/containerd#13346 (EROFS-differ init failure cascades into dockerd refusing to start). Workaround we ended up with - a postCreateCommand that idempotently downgrades moby-containerd to 2.2.3-debian12u1:

```bash
PINNED="2.2.3-debian12u1"
if [[ "$(dpkg-query -W -f='${Version}' moby-containerd 2>/dev/null)" != "$PINNED" ]]; then
sudo apt-get update -qq
sudo apt-get install -y --allow-downgrades "moby-containerd=$PINNED"
sudo pkill -9 dockerd containerd 2>/dev/null || true
sudo rm -f /run/docker.pid
sudo /usr/local/share/docker-init.sh &
fi
```

Works, but a containerdVersion knob on the Feature would let us drop the workaround once containerd/containerd#13328 ships.

[silva-renan]

+1 facing this issue. Setting "moby": false is also an workaround.

[Kaniska244]

This PR has been merged to fix the issue with the moby-containerd package and erofs file system module. Please try the latest version of the docker-in-docker feature ghcr.io/devcontainers/features/docker-in-docker:3 and get back in case of any issues.

[JuJup]

Thanks @Kaniska244 for the fix. I tried it, but ran into another problem.

I get this error message:

docker: Error response from daemon: failed to mount /tmp/containerd-mount61222505: mount source: "overlay", target: "/tmp/containerd-mount61222505", fstype: overlay, flags: 0, data: "workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/28/work,upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/28/fs,lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/27/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/19/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/18/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/16/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/15/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/13/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/12/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/10/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/7/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/2/fs,index=off", err: invalid argument

I tried to analyze it with AI support, maybe that helps:

v3 breaks docker-in-docker when the devcontainer itself runs on an overlayfs root (e.g. environments where the host uses containerd to manage the devcontainer).

In v2, dockerd spawned its own built-in containerd child, which stored snapshotter data inside /var/lib/docker — volume-mounted on ext4 by the feature. In v3, containerd is started as a standalone process (to pick up the erofs disable config), but its data lands in /var/lib/containerd on the container's root filesystem. If that root is overlayfs, the overlayfs snapshotter fails with EINVAL (overlay-on-overlay).

The feature's devcontainer-feature.json mounts a volume for /var/lib/docker but not for /var/lib/containerd. Adding a volume mount for /var/lib/containerd fixes it:

"mounts": [
    "source=dind-var-lib-containerd-${devcontainerId},target=/var/lib/containerd,type=volume"
]

This should probably be added to the feature definition alongside the existing /var/lib/docker mount.

The fix with the mount seems to work btw for me.

[Kaniska244]

Hi @JuJup

Thank you for reporting this issue and the detailed analysis. I will look into this and keep you updated.