Skip to content

[docker-in-docker] - Fixing issue with containerd#1653

Open
Kaniska244 wants to merge 3 commits into
devcontainers:mainfrom
Kaniska244:d-in-d-containerd-issues
Open

[docker-in-docker] - Fixing issue with containerd#1653
Kaniska244 wants to merge 3 commits into
devcontainers:mainfrom
Kaniska244:d-in-d-containerd-issues

Conversation

@Kaniska244
Copy link
Copy Markdown
Contributor

@Kaniska244 Kaniska244 commented May 19, 2026

Summary

This PR fixes a docker-in-docker issue where containerd can fail when the dev container root filesystem is itself mounted as overlayfs.

The fix adds a dedicated persisted volume for /var/lib/containerd, ensuring that containerd snapshotter state is stored on non-overlay storage. This prevents overlay-on-overlay mount failures such as invalid argument during image pull or container startup.
Fixes this follow-up issue

Changes included

  • Add persisted volume mount for:
    • dind-var-lib-containerd-${devcontainerId}/var/lib/containerd
  • Bump docker-in-docker feature version from 3.0.0 to 3.0.1
  • Document persisted Docker/containerd state and the overlayfs limitation in src/docker-in-docker/NOTES.md
  • Add regression test:
    • test/docker-in-docker/overlayfs_containerd_root.sh
  • Add test scenario:
    • overlayfs_containerd_root
  • Update workflow coverage to use mcr.microsoft.com/devcontainers/base:noble where needed
  • Extend PR test matrix to cover docker-in-docker on Ubuntu-based images

Why this fix is needed

When /var/lib/containerd lives on an overlay-backed root filesystem, the standalone containerd started by this feature may attempt to place overlayfs snapshotter data on top of another overlay mount. In affected environments, this can cause container/image operations to fail with errors like:

  • failed to mount ... invalid argument

Mounting /var/lib/containerd on its own named volume avoids that storage layout and makes the feature work reliably in containerd-backed and overlayfs-based environments.

Validation

This PR adds regression coverage to verify that:

  • the root filesystem is overlay
  • /var/lib/containerd is not mounted as overlay
  • /var/lib/docker is not mounted as overlay
  • docker run --rm hello-world succeeds
  • the containerd snapshotter code path is actually exercised

Notes

The new persisted volumes are scoped by ${devcontainerId}, so each dev container gets isolated state while preserving images/snapshots across rebuilds. These volumes are not automatically removed when the dev container is deleted.

@Kaniska244 Kaniska244 marked this pull request as ready for review May 19, 2026 09:58
@Kaniska244 Kaniska244 requested a review from a team as a code owner May 19, 2026 09:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

docker-in-docker: Feature request - Ability to pin containerd version

1 participant