devices: drop cilium/ebpf{,link} deps - #64
Conversation
d5f40e9 to
733c596
Compare
…ranch Temporarily point the opencontainers/cgroups dependency at the drop-cilium-ebpf branch (opencontainers/cgroups#64) via a go.mod replace, and re-vendor, so CI can exercise the cilium/ebpf main+link package removal end-to-end in runc. This must not be merged: the replace directive points at a personal fork branch. Once opencontainers/cgroups#64 lands and is tagged, this should be replaced by a normal dependency bump. Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
733c596 to
cf4738d
Compare
Let's drop the second biggest (by size) runc dependency and reduce the binary size by another ~1MB. Draft/DNM until opencontainers/cgroups#64 is merged/released. Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
There was a problem hiding this comment.
Pull request overview
This PR refactors the devices eBPF cgroup-device filter implementation to stop importing github.com/cilium/ebpf’s main and link packages, and instead perform the required operations via direct bpf(2) syscalls while continuing to use github.com/cilium/ebpf/asm for instruction assembly. This aligns with the stated goal of reducing consumer binary size by avoiding heavy transitive dependencies.
Changes:
- Replaced
cilium/ebpfprogram/link usage with thin wrappers aroundBPF_PROG_*commands and raw program fds. - Updated cgroup device filter attach/query logic to operate on fds (with explicit closes) instead of
*ebpf.Program. - Added
nativeEndianselection via build-tagged endian-specific files to satisfyasm.Instructions.Marshalrequirements.
Reviewed changes
Copilot reviewed 3 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| go.sum | Removes now-unused transitive dependencies after dropping cilium/ebpf main/link usage. |
| devices/endian_le.go | Provides nativeEndian = binary.LittleEndian under little-endian arch build tags. |
| devices/endian_be.go | Provides nativeEndian = binary.BigEndian under big-endian arch build tags. |
| devices/ebpf_linux.go | Replaces ebpf/link usage with direct bpf(2) syscall wrappers and fd-based program management. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| err = bpfProgAttach(dirFd, progFd, attachFlags, replaceFd) | ||
| if err != nil { | ||
| return fmt.Errorf("failed to call BPF_PROG_ATTACH (BPF_CGROUP_DEVICE, BPF_F_ALLOW_MULTI): %w", err) | ||
| } |
There was a problem hiding this comment.
ell, since this is the only non-test^ call to bpfProgAttach we can just omit the flags (and BPF_CGROUP_DEVICE, too -- it is kind of expected here).
^ by the test call I mean the one in haveBpfProgReplace
cf4738d to
0fd3600
Compare
Let's drop the second biggest (by size) runc dependency and reduce the binary size by another ~1MB. Draft/DNM until opencontainers/cgroups#64 is merged/released. Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Let's drop the second biggest (by size) runc dependency and reduce the binary size by another ~1MB. Draft/DNM until opencontainers/cgroups#64 is merged/released. Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
0fd3600 to
8decd05
Compare
There was a problem hiding this comment.
@kolyshkin nice, thanks! Did you c&p the definitions? In that case, can you link them so it's simpler to review?
Also, don't we have overlapping functionality in the cgroups package?
Also, I guess no go.mod changes because it is used for the asm part? Nice that the size is decreased anyways :)
Again I'm afraid I fail to understand what you mean here @rata, can you please elaborate or point to whatever you have in mind? |
8decd05 to
ae8789d
Compare
I did not, I just recreated a bare minimum (poor boy) version of functionality that we used from cilium/ebpf and cilium/ebpf/link, basically wiring I have added a separate second commit, linking to the original cilium/ebpf functions. Let me know if you want it or not so I will squash or remove it. |
|
@AkihiroSuda @thaJeztah PTAL (I think I've addressed all of your comments). The second commit is optional and can either be squashed or removed. |
kolyshkin
left a comment
There was a problem hiding this comment.
Looking into kernel's tools/lib/bpf/bpf.c, I see that BPF_PROG_LOAD is retried up to 5 times when EAGAIN is received. Also, cilium/ebpf does that (indefinitely).
Implemented the same.
Let's drop the second biggest (by size) runc dependency and reduce the binary size by another ~1MB. Draft/DNM until opencontainers/cgroups#64 is merged/released. Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
9b49352 to
9da2685
Compare
Let's drop the second biggest (by size) runc dependency and reduce the binary size by another ~1MB. Draft/DNM until opencontainers/cgroups#64 is merged/released. Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
rata
left a comment
There was a problem hiding this comment.
@kolyshkin thanks! Left another round of comments. It's definitely tricky, I haven't looked into all of it yet
| if err == nil { | ||
| return fd, nil | ||
| } |
There was a problem hiding this comment.
Isn't this a rather unexpected thing to happen? This failed before without logs, we are trying to run it again with log, to give a proper error. I'm unsure what we should do if this happens, shall we say all is fine? Or shall we just fail saying we couldn't get a log of the failure?
Sometimes I think the latter is better, so we can't get tricked into something completely unexpected (and have a security bug, maybe?). And if it's a problem, we can always change it and backport it.
What do you think?
There was a problem hiding this comment.
Looking at cilium/ebpf, it does not return an error when something like this happens.
I think we need to print a warning in this case. Something like this maybe:
- if err == nil {
+ if err == nil { // Totally unexpected.
+ logrus.Warnf("BPF_PROG_LOAD retry unexpectedly succeeded after failing with %w earlier", origErr)(here origErr is the one from the initial BPF_PROG_LOAD).
| const minLogSize = 64 * 1024 | ||
| // If the log does not fit into the buffer, the kernel returns ENOSPC, in | ||
| // which case retry with a bigger buffer, up to maxLogSize (the kernel | ||
| // rejects a log_size larger than UINT_MAX >> 8 anyway). |
There was a problem hiding this comment.
It seems newer kernels support a bigger buffer (https://github.com/torvalds/linux/blob/v5.2/kernel/bpf/verifier.c#L8215). I think it's compeltely fine to keep it to 16MB, but I'd amend the comment :D
There was a problem hiding this comment.
Yes, this was changed in linux v5.2 (torvalds/linux@7a9f5c65abcc).
I also found out that kernel can return the actual buffer size (since v6.4, commit https://lore.kernel.org/bpf/20230406234205.323208-13-andrii@kernel.org/) and this is not used by cilium/ebpf.
Comment amended.
|
|
||
| fd, err = bpfFD(unix.BPF_PROG_LOAD, unsafe.Pointer(&attr), unsafe.Sizeof(attr)) | ||
| runtime.KeepAlive(insnsBytes) | ||
| runtime.KeepAlive(licensePtr) |
There was a problem hiding this comment.
Don't we need to keep alive log too?
There was a problem hiding this comment.
It is explicitly referenced below, so I believe technically we don't have to.
There was a problem hiding this comment.
My AI friends telling me that on a code path where continue is called, log is actually unreachable, so it still makes sense to explicitly use runtime.KeepAlive. Re-added (I've removed it yesterday).
| // The code below is copied from | ||
| // github.com/cilium/ebpf/internal/sys/signals.go@v0.22.0. |
There was a problem hiding this comment.
MIT requires some attribution, I think, but not sure if this code is a "significant portion". Shall we add them?
There was a problem hiding this comment.
Yeah I looked into it and decided it's not a "substantial portion" as MIT License says.
|
|
||
| func findAttachedCgroupDeviceFilters(dirFd int) (_ []*ebpf.Program, retErr error) { | ||
| func bpf(cmd uintptr, attr unsafe.Pointer, size uintptr) (uintptr, error) { | ||
| for { |
There was a problem hiding this comment.
Because it is any signal, can't the go runtime trigger this also? It sends SIGUSR to preempt and schedule goroutines.
I think I'd still bound the loop, if it doesn't work in a few retries, we can fail and let the higher level container creation be started again. We don't want an infinite loop on any circumstances.
| uintptr(unsafe.Pointer(&query)), | ||
| unsafe.Sizeof(query)) | ||
| _, err := bpf(unix.BPF_PROG_QUERY, unsafe.Pointer(&query), unsafe.Sizeof(query)) | ||
| runtime.KeepAlive(progIds) |
There was a problem hiding this comment.
Okay, my AI friends caught a weird bug that existed before but we are making slightly more possible to hit now.
The thing is: ProgIds is on the stack and is not tracked as a pointer, as it is uint64 and the pointer it has is gotten via unsafe. If the go runtime decides it needs more memory for this goroutine, it will allocate a bigger stack and move things there. Pointers are re-written, but as ProgIds is not a pointer for the runtime, it is not. And if calling bpf() makes something that needs a bigger stack, we hit this issue.
The way out of this is using https://pkg.go.dev/runtime#Pinner.Pin, it seems. But I'm not very familiar with it.
There was a problem hiding this comment.
progIds is not on the stack because the size is not const and thus Go allocates heap memory for it (using makeslice). It's a pure luck though because if size to be made const, or Go compiler will evolve to stack-allocate variable length slices, we'll have a very curios bug here.
Implemented pinning (and removed runtime.KeepAlive(progIds) as superseded by pinning).
Let's drop the second biggest (by size) runc dependency and reduce the binary size by another ~1MB. Draft/DNM until opencontainers/cgroups#64 is merged/released. Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
|
Replying to the above comment here (for some reason it doesn't allow me doing that in there, i.e. there's no reply box -- I see this from time to time only in comments from @rata for some reason). Implemented 30 retries in a loop. Maybe we can also drop SIGPROF masking, WDYT? |
9da2685 to
d7fd70c
Compare
Let's drop the second biggest (by size) runc dependency and reduce the binary size by another ~1MB. Draft/DNM until opencontainers/cgroups#64 is merged/released. Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
|
@rata I think I've addressed the last round of comments from you (btw thanks for reviewing this!), and updated opencontainers/runc#5340 to test this PR (one fine day we'll have every cgroups PR tested against runc). Can you please take another look? 🙏🏻 Ah, one remaining question I had to you is do you want the second commit in this PR or not? I wrote it as a reaction to your comment at #64 (review), please decide if you want it or not. If not, I'll remove it, otherwise I'll squash it. |
rata
left a comment
There was a problem hiding this comment.
Replying to the above comment here (for some reason it doesn't allow me doing that in there, i.e. there's no reply box -- I see this from time to time only in comments from @rata for some reason).
I think this is just github that shows the post two times when I post a review comment that adds an answer to an existing thread. It shows it once in my review, where you can't answer, and it shows it also in the original thread, that you can answer.
Is that what you are seeing?
Implemented 30 retries in a loop. Maybe we can also drop SIGPROF masking, WDYT?
As this is a generic module, I'd keep it. It doesn't seem to pollute the code much. But no strong opinion, I'm fine if you prefer to drop it.
@rata I think I've addressed the last round of comments from you (btw thanks for reviewing this!), and updated opencontainers/runc#5340 to test this PR (one fine day we'll have every cgroups PR tested against runc). Can you please take another look? 🙏🏻
Done! Here goes the last round of comments. I think this is quite close to done from my POV now. Thanks! :)
Ah, one remaining question I had to you is do you want the second commit in this PR or not? I wrote it as a reaction to your comment at #64 (review), please decide if you want it or not. If not, I'll remove it, otherwise I'll squash it.
Oh, yes, let's squash it. IMHO it is useful and it will be useful if there is a bug and we want to check if cilium added some fix or something.
| err = bpfProgAttach(dirFd, progFd, attachFlags, replaceFd) | ||
| if err != nil { | ||
| return fmt.Errorf("failed to call BPF_PROG_ATTACH (BPF_CGROUP_DEVICE, BPF_F_ALLOW_MULTI): %w", err) | ||
| } |
| if retries < bpfMaxRetries { | ||
| continue | ||
| } | ||
| return r1, fmt.Errorf("bpf: prog load keeps being interrupted by EAGAIN after %d retries", bpfMaxRetries) |
There was a problem hiding this comment.
nit: This is dropping the error, do we want to use %w to preserve it?
There was a problem hiding this comment.
1. It requires more work (because we have syscall.ERRNO here not error, but I guess the cast is easy.
2. It will not be printed as EAGAIN but rather by its string represenation (which is "resource temporarily unavailable"), and I want to see EAGAIN explicitly.
Having said that, guess we can do something like
return r1, fmt.Errorf("bpf: prog load keeps being interrupted by EAGAIN after %d retries: %w", bpfMaxRetries, error(errno))| // The code below is copied from | ||
| // github.com/cilium/ebpf/internal/sys/signals.go@v0.22.0. |
| @@ -0,0 +1,57 @@ | |||
| //go:build !windows | |||
There was a problem hiding this comment.
Maybe singnals_linux.go? We only call it from a linux only file. And I'm not sure if other unix have all the pthread stuff in Go.
|
@kolyshkin are you sure the runc PR tested this? Having a quick look at the SHAs, it seems it was not exactly this? |
d7fd70c to
8cab5a7
Compare
Let's drop the second biggest (by size) runc dependency and reduce the binary size by another ~1MB. Draft/DNM until opencontainers/cgroups#64 is merged/released. NB: to update this: go mod edit -replace github.com/opencontainers/cgroups=github.com/kolyshkin/oc-cgroups@drop-cilium-ebpf make vendor git add vendor git commit --amend -a Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Now it does (only checked post-push, sorry); the sha is taken from my fork, you can check the sha of top commit in https://github.com/kolyshkin/oc-cgroups/tree/drop-cilium-ebpf; currently it's at 8cab5a7 and go.mod in opencontainers/runc#5340 says the same). |
8cab5a7 to
8770082
Compare
Let's drop the second biggest (by size) runc dependency and reduce the binary size by another ~1MB. Draft/DNM until opencontainers/cgroups#64 is merged/released. NB: to update this: go mod edit -replace github.com/opencontainers/cgroups=github.com/kolyshkin/oc-cgroups@drop-cilium-ebpf make vendor git add vendor git commit --amend -a Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
235b46f to
a2957f7
Compare
|
Now that's real fun -- I found a place where the kernel does out of bounds write. More to say, this is the pre-existing code here: Lines 33 to 44 in 79cbc7c We pass the kernel a pointer to a Claude tells this is kernel writing 2 to Looking into the kernel, this bug was introduced in v6.17 (commit 120933984460, Jun 2025) and was recently fixed by commit 21c4b99b27f3, May 2026, which made its way into v7.2-rc1 (7.2 is not out yet). It was also backported to v7.1.5 (I am running v7.1.4 and thus was able to catch this!). Anyway, since it is apparently a well known (and fixed) bug, let me introduce a workaround. |
a2957f7 to
1bb7901
Compare
Let's drop the second biggest (by size) runc dependency and reduce the binary size by another ~1MB. Draft/DNM until opencontainers/cgroups#64 is merged/released. NB: to update this: go mod edit -replace github.com/opencontainers/cgroups=github.com/kolyshkin/oc-cgroups@drop-cilium-ebpf make vendor git add vendor git commit --amend -a Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
bpfAttrQuery only described the query member of union bpf_attr up to
prog_cnt, and that size (32 bytes) is what is passed to bpf(2). Since
Linux 6.16 (kernel commit 120933984460, "bpf: Implement mprog API on top
of existing cgroup progs"), __cgroup_bpf_query copies query.revision
back to userspace unconditionally, without looking at the size we
passed. As revision sits at offset 56, the kernel writes 8 bytes past
the end of the struct on every BPF_PROG_QUERY.
Depending on how the compiler lays out the frame, those 8 bytes either
land on unused stack space, and nothing happens, or on live data. In
runc this showed up as corrupted defer records, crashing "runc restore"
in runtime.copystack, and as garbled strings.
Kernel commit 21c4b99b27f3 ("bpf: fix BPF_PROG_QUERY OOB write and cgroup
backward compat") added the missing size check, but we have to keep
working on kernels that lack it, so describe the query struct in full.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Replace the use of the cilium/ebpf and cilium/ebpf/link with direct bpf(2) syscalls. Keep cilium/ebpf/asm for instruction assembly. Notes: - the eBPF device-filter programs are now tracked by raw file descriptors instead of *ebpf.Program handles; - asm.Instructions.Marshal requires a concrete binary.LittleEndian or binary.BigEndian, so endian.go detects the native byte order at runtime as a workaround. This could be done during compile time but requires maintaining a list of all GOARCHes; - the "removing old filter %d from cgroup" log messages are removed: this always happens when using systemd and the messages are not useful, plus obtaining the details would add more code; This reduces the runc binary size by about ~1M. Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
1bb7901 to
b364e3b
Compare
Let's drop the second biggest (by size) runc dependency and reduce the binary size by another ~1MB. Draft/DNM until opencontainers/cgroups#64 is merged/released. NB: to update this: go mod edit -replace github.com/opencontainers/cgroups=github.com/kolyshkin/oc-cgroups@drop-cilium-ebpf make vendor git add vendor git commit --amend -a Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
This currently contains #69; will rebase once that one is merged.
Replace the use of the cilium/ebpf and cilium/ebpf/link with direct
bpf(2) syscalls. Keep cilium/ebpf/asm for instruction assembly.
Notes:
descriptors instead of *ebpf.Program handles;
binary.BigEndian, thus endian_{le,be}.go are introduced as a
workaround.
This reduces the runc binary size by about ~1M.
NOTE that this is probably limited to runc, because for other users (k8s, cri-o) this was already solved by opencontainers/runc#4248.
Being tested in opencontainers/runc#5340.
For initial discussion about this, see opencontainers/runc#5218.