From 40451799ee7adaed250b34f0d822152a1c0bd20f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ren=C3=A9=20Treffer?= Date: Wed, 26 Aug 2026 12:14:20 +0200 Subject: [PATCH 1/3] Add resctrl collector MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Expose the Linux resctrl monitoring counters (Intel RDT, AMD PQoS, ARM MPAM): per L3 domain memory bandwidth (mbm_total_bytes, mbm_local_bytes) and last level cache occupancy (llc_occupancy). Parsing is done by the procfs resctrlfs package (procfs v0.22.0). The collector is disabled by default because the resctrl filesystem must be mounted first (mount -t resctrl resctrl /sys/fs/resctrl). The mountpoint is configurable via --collector.resctrl.path. Counters a CPU does not support (for example mbm_local_bytes on Arm MPAM) or that read "Unavailable" are skipped per sample. The hardware counters wrap; Prometheus treats the wrap as a counter reset, so no correction is applied. Signed-off-by: René Treffer --- README.md | 1 + collector/fixtures/e2e-64k-page-output.txt | 10 ++ collector/fixtures/e2e-output.txt | 10 ++ .../resctrl/mon_data/mon_L3_00/llc_occupancy | 1 + .../mon_data/mon_L3_00/mbm_local_bytes | 1 + .../mon_data/mon_L3_00/mbm_total_bytes | 1 + .../resctrl/mon_data/mon_L3_01/llc_occupancy | 1 + .../mon_data/mon_L3_01/mbm_local_bytes | 1 + .../mon_data/mon_L3_01/mbm_total_bytes | 1 + collector/resctrl_linux.go | 110 ++++++++++++++++++ collector/resctrl_linux_test.go | 91 +++++++++++++++ end-to-end-test.sh | 2 + 12 files changed, 230 insertions(+) create mode 100644 collector/fixtures/resctrl/mon_data/mon_L3_00/llc_occupancy create mode 100644 collector/fixtures/resctrl/mon_data/mon_L3_00/mbm_local_bytes create mode 100644 collector/fixtures/resctrl/mon_data/mon_L3_00/mbm_total_bytes create mode 100644 collector/fixtures/resctrl/mon_data/mon_L3_01/llc_occupancy create mode 100644 collector/fixtures/resctrl/mon_data/mon_L3_01/mbm_local_bytes create mode 100644 collector/fixtures/resctrl/mon_data/mon_L3_01/mbm_total_bytes create mode 100644 collector/resctrl_linux.go create mode 100644 collector/resctrl_linux_test.go diff --git a/README.md b/README.md index 713663132b..492c1040f1 100644 --- a/README.md +++ b/README.md @@ -213,6 +213,7 @@ pcidevice | Exposes pci devices' information including their link status and par perf | Exposes perf based metrics (Warning: Metrics are dependent on kernel configuration and settings). | Linux processes | Exposes aggregate process statistics from `/proc`. | Linux qdisc | Exposes [queuing discipline](https://en.wikipedia.org/wiki/Network_scheduler#Linux_kernel) statistics | Linux +resctrl | Exposes Intel RDT / AMD PQoS / ARM MPAM memory bandwidth and cache occupancy monitoring counters from the resctrl filesystem (must be mounted). | Linux slabinfo | Exposes slab statistics from `/proc/slabinfo`. Note that permission of `/proc/slabinfo` is usually 0400, so set it appropriately. | Linux softirqs | Exposes detailed softirq statistics from `/proc/softirqs`. | Linux sysctl | Expose sysctl values from `/proc/sys`. Use `--collector.sysctl.include(-info)` to configure. | Linux diff --git a/collector/fixtures/e2e-64k-page-output.txt b/collector/fixtures/e2e-64k-page-output.txt index 5d84335eb1..18a34815ba 100644 --- a/collector/fixtures/e2e-64k-page-output.txt +++ b/collector/fixtures/e2e-64k-page-output.txt @@ -3719,6 +3719,15 @@ node_rapl_core_joules_total{index="0",path="collector/fixtures/sys/class/powerca # HELP node_rapl_package_joules_total Current RAPL package value in joules # TYPE node_rapl_package_joules_total counter node_rapl_package_joules_total{index="0",path="collector/fixtures/sys/class/powercap/intel-rapl:0"} 240422.366267 +# HELP node_resctrl_llc_occupancy_bytes Last level cache bytes occupied in this domain. +# TYPE node_resctrl_llc_occupancy_bytes gauge +node_resctrl_llc_occupancy_bytes{domain="0"} 4.4040192e+07 +node_resctrl_llc_occupancy_bytes{domain="1"} 8.388608e+06 +# HELP node_resctrl_memory_bandwidth_bytes_total Bytes moved between the last level cache and memory. scope=total includes remote sockets, scope=local is memory attached to this domain. +# TYPE node_resctrl_memory_bandwidth_bytes_total counter +node_resctrl_memory_bandwidth_bytes_total{domain="0",scope="local"} 2.10196273664e+11 +node_resctrl_memory_bandwidth_bytes_total{domain="0",scope="total"} 3.15294410752e+11 +node_resctrl_memory_bandwidth_bytes_total{domain="1",scope="total"} 1.05098136832e+11 # HELP node_schedstat_running_seconds_total Number of seconds CPU spent running a process. # TYPE node_schedstat_running_seconds_total counter node_schedstat_running_seconds_total{cpu="0"} 2.045936778163039e+06 @@ -3779,6 +3788,7 @@ node_scrape_collector_success{collector="pressure"} 1 node_scrape_collector_success{collector="processes"} 1 node_scrape_collector_success{collector="qdisc"} 1 node_scrape_collector_success{collector="rapl"} 1 +node_scrape_collector_success{collector="resctrl"} 1 node_scrape_collector_success{collector="schedstat"} 1 node_scrape_collector_success{collector="slabinfo"} 1 node_scrape_collector_success{collector="sockstat"} 1 diff --git a/collector/fixtures/e2e-output.txt b/collector/fixtures/e2e-output.txt index cfdeede26d..0c91a90300 100644 --- a/collector/fixtures/e2e-output.txt +++ b/collector/fixtures/e2e-output.txt @@ -3741,6 +3741,15 @@ node_rapl_core_joules_total{index="0",path="collector/fixtures/sys/class/powerca # HELP node_rapl_package_joules_total Current RAPL package value in joules # TYPE node_rapl_package_joules_total counter node_rapl_package_joules_total{index="0",path="collector/fixtures/sys/class/powercap/intel-rapl:0"} 240422.366267 +# HELP node_resctrl_llc_occupancy_bytes Last level cache bytes occupied in this domain. +# TYPE node_resctrl_llc_occupancy_bytes gauge +node_resctrl_llc_occupancy_bytes{domain="0"} 4.4040192e+07 +node_resctrl_llc_occupancy_bytes{domain="1"} 8.388608e+06 +# HELP node_resctrl_memory_bandwidth_bytes_total Bytes moved between the last level cache and memory. scope=total includes remote sockets, scope=local is memory attached to this domain. +# TYPE node_resctrl_memory_bandwidth_bytes_total counter +node_resctrl_memory_bandwidth_bytes_total{domain="0",scope="local"} 2.10196273664e+11 +node_resctrl_memory_bandwidth_bytes_total{domain="0",scope="total"} 3.15294410752e+11 +node_resctrl_memory_bandwidth_bytes_total{domain="1",scope="total"} 1.05098136832e+11 # HELP node_schedstat_running_seconds_total Number of seconds CPU spent running a process. # TYPE node_schedstat_running_seconds_total counter node_schedstat_running_seconds_total{cpu="0"} 2.045936778163039e+06 @@ -3801,6 +3810,7 @@ node_scrape_collector_success{collector="pressure"} 1 node_scrape_collector_success{collector="processes"} 1 node_scrape_collector_success{collector="qdisc"} 1 node_scrape_collector_success{collector="rapl"} 1 +node_scrape_collector_success{collector="resctrl"} 1 node_scrape_collector_success{collector="schedstat"} 1 node_scrape_collector_success{collector="slabinfo"} 1 node_scrape_collector_success{collector="sockstat"} 1 diff --git a/collector/fixtures/resctrl/mon_data/mon_L3_00/llc_occupancy b/collector/fixtures/resctrl/mon_data/mon_L3_00/llc_occupancy new file mode 100644 index 0000000000..317ea811d9 --- /dev/null +++ b/collector/fixtures/resctrl/mon_data/mon_L3_00/llc_occupancy @@ -0,0 +1 @@ +44040192 diff --git a/collector/fixtures/resctrl/mon_data/mon_L3_00/mbm_local_bytes b/collector/fixtures/resctrl/mon_data/mon_L3_00/mbm_local_bytes new file mode 100644 index 0000000000..5cb8136235 --- /dev/null +++ b/collector/fixtures/resctrl/mon_data/mon_L3_00/mbm_local_bytes @@ -0,0 +1 @@ +210196273664 diff --git a/collector/fixtures/resctrl/mon_data/mon_L3_00/mbm_total_bytes b/collector/fixtures/resctrl/mon_data/mon_L3_00/mbm_total_bytes new file mode 100644 index 0000000000..d91f52ebae --- /dev/null +++ b/collector/fixtures/resctrl/mon_data/mon_L3_00/mbm_total_bytes @@ -0,0 +1 @@ +315294410752 diff --git a/collector/fixtures/resctrl/mon_data/mon_L3_01/llc_occupancy b/collector/fixtures/resctrl/mon_data/mon_L3_01/llc_occupancy new file mode 100644 index 0000000000..16edb4cc19 --- /dev/null +++ b/collector/fixtures/resctrl/mon_data/mon_L3_01/llc_occupancy @@ -0,0 +1 @@ +8388608 diff --git a/collector/fixtures/resctrl/mon_data/mon_L3_01/mbm_local_bytes b/collector/fixtures/resctrl/mon_data/mon_L3_01/mbm_local_bytes new file mode 100644 index 0000000000..ba54161f35 --- /dev/null +++ b/collector/fixtures/resctrl/mon_data/mon_L3_01/mbm_local_bytes @@ -0,0 +1 @@ +Unavailable diff --git a/collector/fixtures/resctrl/mon_data/mon_L3_01/mbm_total_bytes b/collector/fixtures/resctrl/mon_data/mon_L3_01/mbm_total_bytes new file mode 100644 index 0000000000..47cf3d0c1b --- /dev/null +++ b/collector/fixtures/resctrl/mon_data/mon_L3_01/mbm_total_bytes @@ -0,0 +1 @@ +105098136832 diff --git a/collector/resctrl_linux.go b/collector/resctrl_linux.go new file mode 100644 index 0000000000..9995950210 --- /dev/null +++ b/collector/resctrl_linux.go @@ -0,0 +1,110 @@ +// Copyright The Prometheus Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//go:build !noresctrl + +package collector + +import ( + "errors" + "log/slog" + "os" + + "github.com/alecthomas/kingpin/v2" + "github.com/prometheus/client_golang/prometheus" + "github.com/prometheus/procfs/resctrlfs" +) + +const resctrlSubsystem = "resctrl" + +// The resctrl filesystem exposes the cache and memory bandwidth monitoring of +// the CPU (Intel RDT, AMD PQoS, Arm MPAM), documented in +// https://docs.kernel.org/filesystems/resctrl.html and +// https://docs.kernel.org/arch/arm64/mpam.html. +// +// It is not mounted by default, hence this collector is disabled by default. +// Mount it with: +// +// mount -t resctrl resctrl /sys/fs/resctrl +var resctrlPath = kingpin.Flag("collector.resctrl.path", "resctrl filesystem mountpoint.").Default(resctrlfs.DefaultMountPoint).String() + +type resctrlCollector struct{} + +func init() { + registerCollector("resctrl", defaultDisabled, NewResctrlCollector) +} + +var ( + resctrlMemoryBandwidth = prometheus.NewDesc( + prometheus.BuildFQName(namespace, resctrlSubsystem, "memory_bandwidth_bytes_total"), + "Bytes moved between the last level cache and memory. scope=total includes remote sockets, scope=local is memory attached to this domain.", + []string{"domain", "scope"}, nil, + ) + resctrlLLCOccupancy = prometheus.NewDesc( + prometheus.BuildFQName(namespace, resctrlSubsystem, "llc_occupancy_bytes"), + "Last level cache bytes occupied in this domain.", + []string{"domain"}, nil, + ) +) + +// NewResctrlCollector returns a new Collector exposing resctrl monitoring counters. +func NewResctrlCollector(_ *slog.Logger) (Collector, error) { + return &resctrlCollector{}, nil +} + +func (c *resctrlCollector) Update(ch chan<- prometheus.Metric) error { + // The filesystem is opened on every scrape because it can be mounted and + // unmounted while node_exporter runs. + fs, err := resctrlfs.NewFS(*resctrlPath) + if err != nil { + if errors.Is(err, os.ErrNotExist) { + return ErrNoData + } + return err + } + + domains, err := fs.MonData() + if err != nil { + if errors.Is(err, os.ErrNotExist) { + return ErrNoData + } + return err + } + + found := false + for _, domain := range domains { + // The metrics describe the last level cache, so only L3 fits them. + if domain.Resource != "L3" { + continue + } + found = true + + // The bandwidth counters wrap. No correction is applied: Prometheus + // reads a wrap as a counter reset, which is the closest available + // approximation. + if v := domain.MBMTotalBytes; v != nil { + ch <- prometheus.MustNewConstMetric(resctrlMemoryBandwidth, prometheus.CounterValue, float64(*v), domain.ID, "total") + } + if v := domain.MBMLocalBytes; v != nil { + ch <- prometheus.MustNewConstMetric(resctrlMemoryBandwidth, prometheus.CounterValue, float64(*v), domain.ID, "local") + } + if v := domain.LLCOccupancy; v != nil { + ch <- prometheus.MustNewConstMetric(resctrlLLCOccupancy, prometheus.GaugeValue, float64(*v), domain.ID) + } + } + + if !found { + return ErrNoData + } + return nil +} diff --git a/collector/resctrl_linux_test.go b/collector/resctrl_linux_test.go new file mode 100644 index 0000000000..fd5598a92d --- /dev/null +++ b/collector/resctrl_linux_test.go @@ -0,0 +1,91 @@ +// Copyright The Prometheus Authors +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +//go:build !noresctrl + +package collector + +import ( + "io" + "log/slog" + "strings" + "testing" + + "github.com/prometheus/client_golang/prometheus" + "github.com/prometheus/client_golang/prometheus/testutil" +) + +type testResctrlCollector struct { + rc Collector +} + +func (c testResctrlCollector) Collect(ch chan<- prometheus.Metric) { + c.rc.Update(ch) +} + +func (c testResctrlCollector) Describe(ch chan<- *prometheus.Desc) { + prometheus.DescribeByCollect(c, ch) +} + +func newTestResctrlCollector(logger *slog.Logger) (prometheus.Collector, error) { + rc, err := NewResctrlCollector(logger) + if err != nil { + return testResctrlCollector{}, err + } + return testResctrlCollector{rc}, nil +} + +func TestResctrlMetrics(t *testing.T) { + *resctrlPath = "fixtures/resctrl" + + // Domain 1 reports "Unavailable" for mbm_local_bytes, so that sample is + // absent. + testcase := `# HELP node_resctrl_llc_occupancy_bytes Last level cache bytes occupied in this domain. +# TYPE node_resctrl_llc_occupancy_bytes gauge +node_resctrl_llc_occupancy_bytes{domain="0"} 4.4040192e+07 +node_resctrl_llc_occupancy_bytes{domain="1"} 8.388608e+06 +# HELP node_resctrl_memory_bandwidth_bytes_total Bytes moved between the last level cache and memory. scope=total includes remote sockets, scope=local is memory attached to this domain. +# TYPE node_resctrl_memory_bandwidth_bytes_total counter +node_resctrl_memory_bandwidth_bytes_total{domain="0",scope="local"} 2.10196273664e+11 +node_resctrl_memory_bandwidth_bytes_total{domain="0",scope="total"} 3.15294410752e+11 +node_resctrl_memory_bandwidth_bytes_total{domain="1",scope="total"} 1.05098136832e+11 +` + + logger := slog.New(slog.NewTextHandler(io.Discard, nil)) + c, err := newTestResctrlCollector(logger) + if err != nil { + t.Fatal(err) + } + + if err := testutil.CollectAndCompare(c, strings.NewReader(testcase)); err != nil { + t.Fatal(err) + } +} + +func TestResctrlNotMounted(t *testing.T) { + *resctrlPath = t.TempDir() + + logger := slog.New(slog.NewTextHandler(io.Discard, nil)) + c, err := NewResctrlCollector(logger) + if err != nil { + t.Fatal(err) + } + + ch := make(chan prometheus.Metric, 16) + err = c.Update(ch) + close(ch) + + if err != ErrNoData { + t.Fatalf("expected ErrNoData, got %v", err) + } +} diff --git a/end-to-end-test.sh b/end-to-end-test.sh index 661d2d4cc8..cc4aa41c2f 100755 --- a/end-to-end-test.sh +++ b/end-to-end-test.sh @@ -74,6 +74,7 @@ enabled_collectors=$(cat << COLLECTORS processes qdisc rapl + resctrl schedstat slabinfo sockstat @@ -175,6 +176,7 @@ collector_flags=$(cat << FLAGS --collector.netdev.device-include=lo --collector.qdisc.device-include=(wlan0|eth0) --collector.qdisc.fixtures=collector/fixtures/qdisc/ + --collector.resctrl.path=collector/fixtures/resctrl --collector.stat.softirq --collector.sysctl.include-info=kernel.seccomp.actions_avail --collector.sysctl.include=fs.file-nr From 8e6a1e2f1ac83e92e7abd6c3b04934eeb6b03d9f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ren=C3=A9=20Treffer?= Date: Mon, 19 Oct 2026 15:09:59 +0200 Subject: [PATCH 2/3] Describe steps needed to use the resctrl exporter MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: René Treffer --- README.md | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 59 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 492c1040f1..0d8b40a8fb 100644 --- a/README.md +++ b/README.md @@ -213,7 +213,7 @@ pcidevice | Exposes pci devices' information including their link status and par perf | Exposes perf based metrics (Warning: Metrics are dependent on kernel configuration and settings). | Linux processes | Exposes aggregate process statistics from `/proc`. | Linux qdisc | Exposes [queuing discipline](https://en.wikipedia.org/wiki/Network_scheduler#Linux_kernel) statistics | Linux -resctrl | Exposes Intel RDT / AMD PQoS / ARM MPAM memory bandwidth and cache occupancy monitoring counters from the resctrl filesystem (must be mounted). | Linux +resctrl | Exposes Intel RDT / AMD PQoS / ARM MPAM memory bandwidth and cache occupancy monitoring counters from the resctrl filesystem (must be mounted, see [below](#resctrl-collector)). | Linux slabinfo | Exposes slab statistics from `/proc/slabinfo`. Note that permission of `/proc/slabinfo` is usually 0400, so set it appropriately. | Linux softirqs | Exposes detailed softirq statistics from `/proc/softirqs`. | Linux sysctl | Expose sysctl values from `/proc/sys`. Use `--collector.sysctl.include(-info)` to configure. | Linux @@ -271,6 +271,64 @@ found using [`perf list`](http://man7.org/linux/man-pages/man1/perf.1.html) or from debugfs. And example usage of this would be `--collector.perf.tracepoint="sched:sched_process_exec"`. +### Resctrl Collector + +The `resctrl` collector is disabled by default because the filesystem it reads +is usually not mounted. +To enable it, first mount the filesystem and then start `node_exporter` with the collector enabled: +``` +mount -t resctrl resctrl /sys/fs/resctrl +node_exporter --collector.resctrl +``` + +Use `--collector.resctrl.path` if the filesystem is mounted elsewhere. The +collector reports no data, rather than an error, when the path does not exist, +so it is safe to enable on a fleet where only some nodes have it. + +**The hardware has to support monitoring.** Intel RDT, AMD PQoS and ARM MPAM +all expose their counters here, but the set of events varies by CPU: + +``` +cat /sys/fs/resctrl/info/L3_MON/mon_features +``` + +`mbm_total_bytes` and `mbm_local_bytes` back +`node_resctrl_memory_bandwidth_bytes_total`, `llc_occupancy` backs +`node_resctrl_llc_occupancy_bytes`. The collector exports whichever are +present. Only L3 domains are exported, since the metrics describe the last +level cache. Most hypervisors do not pass the monitoring hardware through, so +in a virtual machine the files are usually absent entirely. + +**The counter mode matters on newer kernels.** From Linux 6.19 a CPU with +assignable bandwidth monitoring counters comes up in `mbm_event` mode, where +each monitoring group has to be bound to a hardware counter. On an AMD Zen 5 +part in that mode the root group recorded a value at mount time and then never +advanced, so the collector exports a constant and `rate()` of it is zero. The +classic mode, used by every kernel before 6.19, can be enabled with: + +``` +echo default > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode # fails harmlessly on older kernels +``` + +Check which mode is active with `cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode`; +the active one is in brackets. + +**Only the root group is counted.** The collector reads `mon_data` of the root +group. A task that moves into a control group, created with +`mkdir /sys/fs/resctrl/`, is counted there instead and drops out of these +metrics with no sign that it did. If a container runtime or an RDT orchestrator +manages groups on the node, treat the values as a lower bound. Monitoring +groups below `mon_groups` still roll up into the root and are fine. + +Where the CPU supports Bandwidth Monitoring Event +Configuration, `mbm_total_bytes_config` and `mbm_local_bytes_config` select +which memory transactions each event counts; at anything other than the +defaults the two series stop matching their help text. And the counters are +hardware registers that wrap at a certain value, so query them with `rate()`. + +See the [kernel documentation](https://docs.kernel.org/filesystems/resctrl.html) +for the full picture. + ### Sysctl Collector The `sysctl` collector can be enabled with `--collector.sysctl`. It supports exposing numeric sysctl values From 55212517fd5ab53689ea62f67bb3f216fd0811e9 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Ren=C3=A9=20Treffer?= Date: Mon, 19 Oct 2026 15:43:58 +0200 Subject: [PATCH 3/3] Shorten resctrl documentation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Signed-off-by: René Treffer --- README.md | 50 +++++--------------------------------------------- 1 file changed, 5 insertions(+), 45 deletions(-) diff --git a/README.md b/README.md index 0d8b40a8fb..683116f889 100644 --- a/README.md +++ b/README.md @@ -213,7 +213,7 @@ pcidevice | Exposes pci devices' information including their link status and par perf | Exposes perf based metrics (Warning: Metrics are dependent on kernel configuration and settings). | Linux processes | Exposes aggregate process statistics from `/proc`. | Linux qdisc | Exposes [queuing discipline](https://en.wikipedia.org/wiki/Network_scheduler#Linux_kernel) statistics | Linux -resctrl | Exposes Intel RDT / AMD PQoS / ARM MPAM memory bandwidth and cache occupancy monitoring counters from the resctrl filesystem (must be mounted, see [below](#resctrl-collector)). | Linux +resctrl | Exposes Intel RDT / AMD PQoS / ARM MPAM memory bandwidth and cache occupancy monitoring counters from the resctrl filesystem root group (must be mounted, see [below](#resctrl-collector)). | Linux slabinfo | Exposes slab statistics from `/proc/slabinfo`. Note that permission of `/proc/slabinfo` is usually 0400, so set it appropriately. | Linux softirqs | Exposes detailed softirq statistics from `/proc/softirqs`. | Linux sysctl | Expose sysctl values from `/proc/sys`. Use `--collector.sysctl.include(-info)` to configure. | Linux @@ -273,61 +273,21 @@ from debugfs. And example usage of this would be ### Resctrl Collector -The `resctrl` collector is disabled by default because the filesystem it reads -is usually not mounted. +The `resctrl` collector is disabled by default because the filesystem it reads is usually not mounted. To enable it, first mount the filesystem and then start `node_exporter` with the collector enabled: ``` mount -t resctrl resctrl /sys/fs/resctrl node_exporter --collector.resctrl ``` -Use `--collector.resctrl.path` if the filesystem is mounted elsewhere. The -collector reports no data, rather than an error, when the path does not exist, -so it is safe to enable on a fleet where only some nodes have it. - -**The hardware has to support monitoring.** Intel RDT, AMD PQoS and ARM MPAM -all expose their counters here, but the set of events varies by CPU: - -``` -cat /sys/fs/resctrl/info/L3_MON/mon_features -``` - -`mbm_total_bytes` and `mbm_local_bytes` back -`node_resctrl_memory_bandwidth_bytes_total`, `llc_occupancy` backs -`node_resctrl_llc_occupancy_bytes`. The collector exports whichever are -present. Only L3 domains are exported, since the metrics describe the last -level cache. Most hypervisors do not pass the monitoring hardware through, so -in a virtual machine the files are usually absent entirely. - -**The counter mode matters on newer kernels.** From Linux 6.19 a CPU with -assignable bandwidth monitoring counters comes up in `mbm_event` mode, where -each monitoring group has to be bound to a hardware counter. On an AMD Zen 5 -part in that mode the root group recorded a value at mount time and then never -advanced, so the collector exports a constant and `rate()` of it is zero. The -classic mode, used by every kernel before 6.19, can be enabled with: +From Linux 6.19 a CPU with assignable bandwidth monitoring counters comes up in `mbm_event` mode. +The classic mode (required by this collector) can be enabled with: ``` echo default > /sys/fs/resctrl/info/L3_MON/mbm_assign_mode # fails harmlessly on older kernels ``` -Check which mode is active with `cat /sys/fs/resctrl/info/L3_MON/mbm_assign_mode`; -the active one is in brackets. - -**Only the root group is counted.** The collector reads `mon_data` of the root -group. A task that moves into a control group, created with -`mkdir /sys/fs/resctrl/`, is counted there instead and drops out of these -metrics with no sign that it did. If a container runtime or an RDT orchestrator -manages groups on the node, treat the values as a lower bound. Monitoring -groups below `mon_groups` still roll up into the root and are fine. - -Where the CPU supports Bandwidth Monitoring Event -Configuration, `mbm_total_bytes_config` and `mbm_local_bytes_config` select -which memory transactions each event counts; at anything other than the -defaults the two series stop matching their help text. And the counters are -hardware registers that wrap at a certain value, so query them with `rate()`. - -See the [kernel documentation](https://docs.kernel.org/filesystems/resctrl.html) -for the full picture. +See the [kernel documentation](https://docs.kernel.org/filesystems/resctrl.html) for more details. ### Sysctl Collector