fix: prefer KUBECONFIG over in-cluster config for k8s client#1012
Open
sradco wants to merge 1 commit into
Open
fix: prefer KUBECONFIG over in-cluster config for k8s client#1012sradco wants to merge 1 commit into
sradco wants to merge 1 commit into
Conversation
1 task
Author
|
/retest |
Contributor
|
/test images |
Contributor
|
/lgtm |
Contributor
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: PeterYurkovich, sradco The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-ci
Bot
added
the
approved
Contributor
|
/retest |
1 similar comment
Contributor
|
/retest |
Contributor
|
New changes are detected. LGTM label has been removed. |
| if bearerToken == "" && config.BearerTokenFile != "" { | ||
| tokenBytes, err := os.ReadFile(config.BearerTokenFile) | ||
| if err != nil { | ||
| return nil, fmt.Errorf("failed to read bearer token file %q: %w", config.BearerTokenFile, err) |
Contributor
There was a problem hiding this comment.
err should already include the path
Suggested change
| return nil, fmt.Errorf("failed to read bearer token file %q: %w", config.BearerTokenFile, err) | |
| return nil, fmt.Errorf("failed to read bearer token file: %w", err) |
Contributor
|
@coderabbitai review |
✅ Action performedReview finished.
|
WalkthroughServer kubeconfig loading was refactored into a new ChangesKubeconfig Loading
E2E Framework Bearer Token
Estimated code review effort: 2 (Simple) | ~10 minutes Sequence Diagram(s)sequenceDiagram
participant createHTTPServer
participant loadKubeConfig
participant clientcmd
participant rest
createHTTPServer->>loadKubeConfig: request kube config
loadKubeConfig->>clientcmd: BuildConfigFromFlags (if KUBECONFIG set)
clientcmd-->>loadKubeConfig: config or error
loadKubeConfig->>rest: InClusterConfig (fallback)
rest-->>loadKubeConfig: config or error
loadKubeConfig-->>createHTTPServer: config or wrapped error
🚥 Pre-merge checks | ✅ 14 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (14 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
🧹 Nitpick comments (1)
pkg/server.go (1)
399-414: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low valueConsider using
ClientConfigLoadingRulesfor standard KUBECONFIG semantics.
clientcmd.BuildConfigFromFlagsonly opensKUBECONFIGas a single file path. Standard kubeconfig tooling (kubectl, CI runners) allowsKUBECONFIGto contain multiple OS-path-separated files that get merged, plus honorscurrent-context. If CI sets a multi-pathKUBECONFIG, this will fail to load wherekubectlwould succeed.♻️ Suggested refactor using deferred loading rules
func loadKubeConfig() (*rest.Config, error) { - if kubeconfig := os.Getenv("KUBECONFIG"); kubeconfig != "" { - cfg, err := clientcmd.BuildConfigFromFlags("", kubeconfig) - if err != nil { - return nil, fmt.Errorf("cannot build config from KUBECONFIG: %w", err) - } - return cfg, nil - } + if kubeconfig := os.Getenv("KUBECONFIG"); kubeconfig != "" { + loadingRules := clientcmd.NewDefaultClientConfigLoadingRules() + cfg, err := clientcmd.NewNonInteractiveDeferredLoadingClientConfig( + loadingRules, &clientcmd.ConfigOverrides{}).ClientConfig() + if err != nil { + return nil, fmt.Errorf("cannot build config from KUBECONFIG: %w", err) + } + return cfg, nil + } cfg, err := rest.InClusterConfig() if err != nil { return nil, fmt.Errorf("cannot get in-cluster config: %w", err) } return cfg, nil }Please verify whether the e2e CI environment ever sets a multi-path
KUBECONFIG; if not, this is low priority.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@pkg/server.go` around lines 399 - 414, The kubeconfig loading in loadKubeConfig currently uses clientcmd.BuildConfigFromFlags with KUBECONFIG, which treats it as a single file and can miss standard merged kubeconfig behavior. Update loadKubeConfig to use clientcmd loading rules (for example, deferred loading with ClientConfigLoadingRules and ConfigOverrides) so multiple OS-path-separated kubeconfig entries and current-context are honored, while keeping the in-cluster rest.InClusterConfig fallback unchanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@pkg/server.go`:
- Around line 399-414: The kubeconfig loading in loadKubeConfig currently uses
clientcmd.BuildConfigFromFlags with KUBECONFIG, which treats it as a single file
and can miss standard merged kubeconfig behavior. Update loadKubeConfig to use
clientcmd loading rules (for example, deferred loading with
ClientConfigLoadingRules and ConfigOverrides) so multiple OS-path-separated
kubeconfig entries and current-context are honored, while keeping the in-cluster
rest.InClusterConfig fallback unchanged.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository YAML (base), Central YAML (inherited)
Review profile: CHILL
Plan: Enterprise
Run ID: 99cb3663-2a8f-4186-ae67-72301f843620
📒 Files selected for processing (2)
pkg/server.gotest/e2e/framework/framework.go
sradco
force-pushed
the
fix-e2e-kubeconfig-fallback
branch
from
75b2454 to
fbe8ddc
Compare
and fix e2e test framework for CI The server previously used rest.InClusterConfig() exclusively, which connects to the build cluster in CI rather than the provisioned test cluster where PrometheusRule CRDs exist. This caused the management API routes to fail to register, resulting in 404 errors during e2e tests. Introduce loadKubeConfig() that prefers KUBECONFIG when set (CI/local dev) and falls back to in-cluster config (production). Additionally fix the e2e test framework and server for CI: - Pass the long-lived server context to the k8s client so informers keep running beyond init. The previous timeout context killed informers when createHTTPServer returned, freezing the relabeled-rules cache. - Create a ServiceAccount token via the TokenRequest API when the kubeconfig uses client-certificate auth (no bearer token available). This is the case in CI where kubeadmin certs are used. - Use unique PromQL expressions per test rule so the spec-equivalence check does not reject rules across tests as duplicates. Signed-off-by: Shirly Radco <sradco@redhat.com> Co-authored-by: AI Assistant <noreply@cursor.com> Co-authored-by: Cursor <cursoragent@cursor.com>
sradco
force-pushed
the
fix-e2e-kubeconfig-fallback
branch
from
fbe8ddc to
52a3798
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
rest.InClusterConfig()which connected to the build cluster instead of the provisioned test cluster. IntroduceloadKubeConfig()that prefers KUBECONFIG when set.createHTTPServerreturned,defer initCancel()cancelled the context and stopped all informers — freezing the relabeled-rules cache permanently. Pass the server-lifetime context toNewClientinstead.config.BearerToken, which is empty when the CI kubeconfig uses client-certificate auth. AddresolveToken()that falls back to creating a ServiceAccount token via the TokenRequest API.vector(1), triggering 409 conflicts across tests. Give each rule a unique PromQL expression.Root Cause
The CI step
monitoring-plugin-tests-management-apiruns on the build cluster (build01) with KUBECONFIG pointing to the provisioned test cluster. Multiple issues prevented the e2e tests from working:rest.InClusterConfig()connecting to the wrong clusterTest plan
go build ./...passesgo test ./pkg/... ./internal/...passesgo vet ./...passes