Skip to content

feat: Make containerdebug configurable #860

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
Maleware wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

feat: Make containerdebug configurable #860

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
4cb5112
feat: Make containerdebug configurable
Maleware Nov 11, 2026
eaa3b2d
Add containerdebug config and proper validation
Maleware Nov 13, 2026
25ef329
update trait IntoValidated for operator-rs port
Maleware Nov 13, 2026
0bb6070
change error type to Error
Maleware Nov 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
120 changes: 120 additions & 0 deletions extra/crds.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -223,6 +223,26 @@ spec:
type: object
x-kubernetes-preserve-unknown-fields: true
type: object
containerdebug:
default:
enabled: null
interval: null
description: |-
Configures the `containerdebug` tool, which periodically logs information about the container
environment (such as network interfaces, DNS lookups and disk usage) in the background of the
product container.
properties:
enabled:
description: Whether `containerdebug` runs in the background of the product container. Defaults to `true`.
nullable: true
type: boolean
interval:
description: |-
Time between two runs of `containerdebug`, e.g. `30m`, `1h` or `2d`. Must not be zero.
Defaults to `30m`.
nullable: true
type: string
type: object
gracefulShutdownTimeout:
description: Time period Pods have to gracefully shut down, e.g. `30m`, `1h` or `2d`. Consult the operator documentation for details.
nullable: true
Expand Down Expand Up @@ -790,6 +810,26 @@ spec:
type: object
x-kubernetes-preserve-unknown-fields: true
type: object
containerdebug:
default:
enabled: null
interval: null
description: |-
Configures the `containerdebug` tool, which periodically logs information about the container
environment (such as network interfaces, DNS lookups and disk usage) in the background of the
product container.
properties:
enabled:
description: Whether `containerdebug` runs in the background of the product container. Defaults to `true`.
nullable: true
type: boolean
interval:
description: |-
Time between two runs of `containerdebug`, e.g. `30m`, `1h` or `2d`. Must not be zero.
Defaults to `30m`.
nullable: true
type: string
type: object
gracefulShutdownTimeout:
description: Time period Pods have to gracefully shut down, e.g. `30m`, `1h` or `2d`. Consult the operator documentation for details.
nullable: true
Expand Down Expand Up @@ -1451,6 +1491,26 @@ spec:
type: object
x-kubernetes-preserve-unknown-fields: true
type: object
containerdebug:
default:
enabled: null
interval: null
description: |-
Configures the `containerdebug` tool, which periodically logs information about the container
environment (such as network interfaces, DNS lookups and disk usage) in the background of the
product container.
properties:
enabled:
description: Whether `containerdebug` runs in the background of the product container. Defaults to `true`.
nullable: true
type: boolean
interval:
description: |-
Time between two runs of `containerdebug`, e.g. `30m`, `1h` or `2d`. Must not be zero.
Defaults to `30m`.
nullable: true
type: string
type: object
gracefulShutdownTimeout:
description: Time period Pods have to gracefully shut down, e.g. `30m`, `1h` or `2d`. Consult the operator documentation for details.
nullable: true
Expand Down Expand Up @@ -1918,6 +1978,26 @@ spec:
type: object
x-kubernetes-preserve-unknown-fields: true
type: object
containerdebug:
default:
enabled: null
interval: null
description: |-
Configures the `containerdebug` tool, which periodically logs information about the container
environment (such as network interfaces, DNS lookups and disk usage) in the background of the
product container.
properties:
enabled:
description: Whether `containerdebug` runs in the background of the product container. Defaults to `true`.
nullable: true
type: boolean
interval:
description: |-
Time between two runs of `containerdebug`, e.g. `30m`, `1h` or `2d`. Must not be zero.
Defaults to `30m`.
nullable: true
type: string
type: object
gracefulShutdownTimeout:
description: Time period Pods have to gracefully shut down, e.g. `30m`, `1h` or `2d`. Consult the operator documentation for details.
nullable: true
Expand Down Expand Up @@ -2380,6 +2460,26 @@ spec:
type: object
x-kubernetes-preserve-unknown-fields: true
type: object
containerdebug:
default:
enabled: null
interval: null
description: |-
Configures the `containerdebug` tool, which periodically logs information about the container
environment (such as network interfaces, DNS lookups and disk usage) in the background of the
product container.
properties:
enabled:
description: Whether `containerdebug` runs in the background of the product container. Defaults to `true`.
nullable: true
type: boolean
interval:
description: |-
Time between two runs of `containerdebug`, e.g. `30m`, `1h` or `2d`. Must not be zero.
Defaults to `30m`.
nullable: true
type: string
type: object
gracefulShutdownTimeout:
description: Time period Pods have to gracefully shut down, e.g. `30m`, `1h` or `2d`. Consult the operator documentation for details.
nullable: true
Expand Down Expand Up @@ -3096,6 +3196,26 @@ spec:
type: object
x-kubernetes-preserve-unknown-fields: true
type: object
containerdebug:
default:
enabled: null
interval: null
description: |-
Configures the `containerdebug` tool, which periodically logs information about the container
environment (such as network interfaces, DNS lookups and disk usage) in the background of the
product container.
properties:
enabled:
description: Whether `containerdebug` runs in the background of the product container. Defaults to `true`.
nullable: true
type: boolean
interval:
description: |-
Time between two runs of `containerdebug`, e.g. `30m`, `1h` or `2d`. Must not be zero.
Defaults to `30m`.
nullable: true
type: string
type: object
gracefulShutdownTimeout:
description: Time period Pods have to gracefully shut down, e.g. `30m`, `1h` or `2d`. Consult the operator documentation for details.
nullable: true
Expand Down
25 changes: 18 additions & 7 deletions rust/operator-binary/src/controller/build/container.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,8 @@ use strum::{Display, EnumDiscriminants, IntoStaticStr};

use crate::{
controller::{
ValidatedCluster,
ValidatedCluster, ValidatedDataNodeConfig, ValidatedJournalNodeConfig,
ValidatedNameNodeConfig,
build::{
self,
jvm::{self, construct_global_jvm_args, construct_role_specific_jvm_args},
Expand All @@ -74,7 +75,7 @@ use crate::{
},
},
crd::{
DataNodeConfig, HdfsNodeRole, HdfsPodRef, JournalNodeConfig, NameNodeConfig, UpgradeState,
HdfsNodeRole, HdfsPodRef, UpgradeState,
constants::{
DATANODE_ROOT_DATA_DIR_PREFIX, LISTENER_VOLUME_DIR, LISTENER_VOLUME_NAME,
LIVENESS_PROBE_FAILURE_THRESHOLD, LIVENESS_PROBE_INITIAL_DELAY_SECONDS,
Expand Down Expand Up @@ -337,7 +338,7 @@ impl ContainerConfig {

/// The PVC templates for a namenode role group: one data PVC plus the listener PVC.
pub fn namenode_volume_claim_templates(
config: &NameNodeConfig,
config: &ValidatedNameNodeConfig,
labels: &Labels,
) -> Result<Vec<PersistentVolumeClaim>> {
let listener = ListenerOperatorVolumeSourceBuilder::new(
Expand Down Expand Up @@ -369,7 +370,7 @@ impl ContainerConfig {

/// The PVC template for a journalnode role group: one data PVC.
pub fn journalnode_volume_claim_templates(
config: &JournalNodeConfig,
config: &ValidatedJournalNodeConfig,
) -> Vec<PersistentVolumeClaim> {
vec![config.resources.storage.data.build_pvc(
ContainerConfig::DATA_VOLUME_MOUNT_NAME,
Expand All @@ -378,7 +379,9 @@ impl ContainerConfig {
}

/// The PVC templates for a datanode role group, one per configured data volume.
pub fn datanode_volume_claim_templates(config: &DataNodeConfig) -> Vec<PersistentVolumeClaim> {
pub fn datanode_volume_claim_templates(
config: &ValidatedDataNodeConfig,
) -> Vec<PersistentVolumeClaim> {
DataNodeStorageConfig {
pvcs: config.resources.storage.clone(),
}
Expand All @@ -390,7 +393,10 @@ impl ContainerConfig {
/// Datanodes use an ephemeral listener volume, while namenodes use a persistent volume claim
/// template for stable per-pod identity (see [`Self::namenode_volume_claim_templates`]) and
/// journalnodes have no listener at all.
pub fn datanode_listener_volume(config: &DataNodeConfig, labels: &Labels) -> Result<Volume> {
pub fn datanode_listener_volume(
config: &ValidatedDataNodeConfig,
labels: &Labels,
) -> Result<Volume> {
Ok(VolumeBuilder::new(&*LISTENER_VOLUME_NAME)
.ephemeral(
ListenerOperatorVolumeSourceBuilder::new(
Expand Down Expand Up @@ -574,7 +580,7 @@ impl ContainerConfig {
{remove_vector_shutdown_file_command}
prepare_signal_handlers
mkdir --parents "${file_log_directory}" || true
containerdebug --output={STACKABLE_LOG_DIR}/containerdebug-state.json --loop &
{containerdebug_command}
if [[ -d {LISTENER_VOLUME_DIR} ]]; then
export POD_ADDRESS=$(cat {LISTENER_VOLUME_DIR}/default-address/address)
for i in {LISTENER_VOLUME_DIR}/default-address/ports/*; do
Expand All @@ -588,6 +594,11 @@ impl ContainerConfig {
hadoop_home = Self::HADOOP_HOME,
role = role.as_ref(),
file_log_directory = ENV_FILE_LOG_DIRECTORY,
containerdebug_command = inputs
.common
.containerdebug
.background_command(STACKABLE_LOG_DIR)
.unwrap_or_default(),
remove_vector_shutdown_file_command =
remove_vector_shutdown_file_command(STACKABLE_LOG_DIR),
create_vector_shutdown_file_command =
Expand Down
4 changes: 2 additions & 2 deletions rust/operator-binary/src/controller/build/graceful_shutdown.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
use snafu::{ResultExt, Snafu};
use stackable_operator::builder::pod::PodBuilder;

use crate::crd::CommonNodeConfig;
use crate::controller::ValidatedCommonNodeConfig;

#[derive(Debug, Snafu)]
pub enum Error {
Expand All @@ -12,7 +12,7 @@ pub enum Error {
}

pub fn add_graceful_shutdown_config(
merged_config: &CommonNodeConfig,
merged_config: &ValidatedCommonNodeConfig,
pod_builder: &mut PodBuilder,
) -> Result<(), Error> {
// This must be always set by the merge mechanism, as we provide a default value,
Expand Down
93 changes: 91 additions & 2 deletions rust/operator-binary/src/controller/build/mod.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -493,12 +493,18 @@ pub(crate) fn role_group_selector(

#[cfg(test)]
mod tests {
use stackable_operator::{k8s_openapi::api::core::v1::ConfigMap, kube::Resource};
use stackable_operator::{
crd::containerdebug::ValidatedContainerdebugConfig,
k8s_openapi::api::{apps::v1::StatefulSet, core::v1::ConfigMap},
kube::Resource,
shared::time::Duration,
v2::product_logging::framework::STACKABLE_LOG_DIR,
};

use super::build;
use crate::{
controller::build::properties::test_support::{cluster_info, validated_cluster},
test_support::namenode_listener,
test_support::{deserialize_and_validate_cluster, namenode_listener},
};

/// The sorted `metadata.name`s of a resource collection.
Expand Down Expand Up @@ -781,4 +787,87 @@ mod tests {
);
}
}

/// The `containerdebug` line of the main container's args in the `default` role group of
/// `role`, if any. The main container is named after its role.
fn containerdebug_line(stateful_sets: &[StatefulSet], role: &str) -> Option<String> {
let name = format!("hdfs-{role}-default");
stateful_sets
.iter()
.find(|stateful_set| stateful_set.meta().name.as_deref() == Some(name.as_str()))
.and_then(|stateful_set| stateful_set.spec.as_ref())
.and_then(|spec| spec.template.spec.as_ref())
.expect("the StatefulSet has a pod spec")
.containers
.iter()
.find(|container| container.name == role)
.and_then(|container| container.args.as_ref())
.expect("the main container has args")
.join("\n")
.lines()
.map(str::trim)
.find(|line| line.starts_with("containerdebug "))
.map(str::to_owned)
}

/// Every role starts `containerdebug` in its main container from its own merged config. The
/// command itself is owned (and tested) by operator-rs, so only the wiring is checked here.
#[test]
fn containerdebug_is_started_per_role_from_the_merged_config() {
let cluster = deserialize_and_validate_cluster(
r#"
---
apiVersion: hdfs.stackable.tech/v1alpha1
kind: HdfsCluster
metadata:
name: hdfs
namespace: default
uid: c2c8c5c0-0b5a-4b1e-9f3e-1a2b3c4d5e6f
spec:
image:
productVersion: 3.4.0
clusterConfig:
zookeeperConfigMapName: hdfs-zk
nameNodes:
config:
containerdebug:
interval: 10m
roleGroups:
default:
replicas: 1
journalNodes:
roleGroups:
default:
replicas: 1
dataNodes:
config:
containerdebug:
interval: 10m
roleGroups:
default:
replicas: 1
config:
containerdebug:
enabled: false
"#,
);
let resources = build(&cluster, &cluster_info()).expect("build succeeds");
let enabled = |interval| {
ValidatedContainerdebugConfig::Enabled { interval }
.background_command(STACKABLE_LOG_DIR)
};

assert_eq!(
containerdebug_line(&resources.stateful_sets, "namenode"),
enabled(Duration::from_minutes_unchecked(10))
);
assert_eq!(
containerdebug_line(&resources.stateful_sets, "journalnode"),
enabled(Duration::from_minutes_unchecked(30))
);
assert_eq!(
containerdebug_line(&resources.stateful_sets, "datanode"),
None
);
}
}
Loading
Loading