Fix Missing Default Rules in kube-prometheus-stack

Helm values, manifests, and dry-run output can contain Secrets. Prepare the private path below for backups and renders. Set CHART_VERSION to the installed version; stop if empty. Dry-run success does not prove Operator reconciliation or active rules.

umask 077
DIAG_DIR=$(mktemp -d "${TMPDIR:-/tmp}/prometheus-diag.XXXXXXXX") || exit 1
: "${CHART_VERSION:?Set the installed Chart version first}"

Symptoms and Exact Error

After you set a custom kube-prometheus-stack ruleSelector, Helm can report a successful upgrade while the default rules disappear from Prometheus Rules or Alerts. Start by checking whether the PrometheusRule resources exist and whether their metadata.labels match the deployed spec.ruleSelector; a PromQL error is a separate branch.

This issue can be quickly narrowed down by dividing it into three layers: if Helm did not create defaultRules as a manifest, if the PrometheusRule exists in Kubernetes but the selector cannot pick it up, or if it was included in the selector but the Operator failed to reflect it in the Prometheus configuration. The fact that helm upgrade succeeded does not mean all three steps are normal.

Currently, the default alerts and recording rules of kube-prometheus-stack are generally rendered as PrometheusRule CRs. Therefore, instead of only looking at the number of ConfigMaps, it is more accurate to sequentially compare the Chart rendering results, the PrometheusRule in the cluster, and the active rules in the Prometheus API.

Values to Check First

The following commands are based on an environment where the release name is kube-prometheus-stack and the namespace is monitoring. If these differ, replace only those two values with your actual installation values.

Checkpoints	Command or Setting	Judgment Criteria
defaultRules rendering	`helm get manifest kube-prometheus-stack -n monitoring \| grep -c 'kind: PrometheusRule'`	If 0, check individual rule groups starting from `defaultRules.create` instead of `ruleSelector`.
Deployed PrometheusRule	`kubectl -n monitoring get prometheusrule --show-labels`	If the resource exists, compare whether each `metadata.labels` satisfies the selector conditions.
Actual ruleSelector	`kubectl -n monitoring get prometheus -o yaml`	Judgment is based on the deployed `spec.ruleSelector` value, not the intent in values.yaml.
Chart auto selector	`ruleSelectorNilUsesHelmValues: true`	If the explicit selector is empty, the Chart typically uses the `release` label of the current Helm release.
Rule namespace	`kubectl get prometheusrule -A --show-labels`	PrometheusRules outside the `ruleNamespaceSelector` range will not be selected even if the labels match.

If defaultRules.create: true is true and the result of the first command is 0, it is often because a specific rule group is disabled, another values file overrode the final value, or the Helm release being checked is different. Conversely, if both the rendering and Kubernetes resources exist, you can narrow down the scope by checking the intersection of ruleSelector and metadata.labels.

Cause Branches

Two behaviors of kube-prometheus-stack ruleSelector

PrometheusRule is not created: If defaultRules.create is false or individual groups like defaultRules.rules.general are disabled, rules will not appear even if you change the selector. Distinguish this first using helm get manifest.
Explicit selector differs from default rule labels: If you include ruleSelector.matchLabels.monitoring-role: cluster-rules but the default PrometheusRule.metadata.labels does not contain the same key and value, the Operator will not select the rule.
Judging the meaning of an empty selector based only on values.yaml: While the Prometheus Operator selects all objects with an empty selector, kube-prometheus-stack may render empty values as Helm release label selectors when ruleSelectorNilUsesHelmValues: true is true. You must check the final Prometheus CR.
Mistaking additionalRuleLabels for metadata labels:defaultRules.additionalRuleLabels is used for the labels of individual alert and recording rules. Since ruleSelector compares against PrometheusRule.metadata.labels, you must use defaultRules.labels when matching the explicit selector.
Different namespace selection scopes: Even if the labels are correct, rules will not be loaded if ruleNamespaceSelector does not include the relevant namespace. Record the namespaces of both the Prometheus CR and the PrometheusRule.
Applied values differ from verified values: If multiple -f files or --set values overwrite values later, the values.yaml in the repository will differ from the actual release values. Use helm get values kube-prometheus-stack -n monitoring -a as the final reference.

Fix Sequence

Back up the current state first. Run helm get values kube-prometheus-stack -n monitoring -a > "$DIAG_DIR/kube-prometheus-stack-values-before.yaml" and helm get manifest kube-prometheus-stack -n monitoring > "$DIAG_DIR/kube-prometheus-stack-manifest-before.yaml". This allows you to restore user values that cannot be recovered by the Chart revision alone during a rollback.
Decide whether to let the Chart manage the selector. To use the same release labels as the default rules created by kube-prometheus-stack, leave the explicit selector empty as shown below while maintaining the use of Helm values.
```
defaultRules:
  create: true
prometheus:
  prometheusSpec:
    ruleSelectorNilUsesHelmValues: true
    ruleSelector: {}
```

If you manage the operational labels directly, match the values on both sides. You must include the conditions of ruleSelector in defaultRules.labels as well for the metadata of the default PrometheusRule to be selected.

defaultRules:
  create: true
  labels:
    monitoring-role: cluster-rules
prometheus:
  prometheusSpec:
    ruleSelectorNilUsesHelmValues: false
    ruleSelector:
      matchLabels:
        monitoring-role: cluster-rules

Render using the same Chart version as the installed one. Check the Chart version in helm list -n monitoring --filter '^kube-prometheus-stack$', then run helm template kube-prometheus-stack oci://ghcr.io/prometheus-community/charts/kube-prometheus-stack --version "${CHART_VERSION}" -n monitoring -f values.yaml > "$DIAG_DIR/rendered.yaml". Do not inadvertently mix in the latest version; instead, verify that the selector and rule labels appear together in the current version.
Compare the rendering results and pass the dry-run. Verify the labels on both sides with grep -nE 'kind: PrometheusRule|ruleSelector:|monitoring-role:|release:' "$DIAG_DIR/rendered.yaml", then run helm upgrade kube-prometheus-stack oci://ghcr.io/prometheus-community/charts/kube-prometheus-stack --version "${CHART_VERSION}" -n monitoring -f values.yaml --dry-run > "$DIAG_DIR/dry-run.txt" 2>&1. Only after confirming the intended number of PrometheusRules and the selector should you apply the same command without --dry-run.
Verify the resources and active rules individually. Verify that the targets are visible using kubectl -n monitoring get prometheusrule -l monitoring-role=cluster-rules --show-labels. Next, run kubectl -n monitoring port-forward svc/kube-prometheus-stack-prometheus 9090:9090 in a separate terminal and check if the default alerting and recording groups are reloaded via curl -s http://127.0.0.1:9090/api/v1/rules.

Verification and Rollback

The criterion for a successful modification is not just the fact that the number of PrometheusRule increased. The spec.ruleSelector of the deployed Prometheus CR, the metadata.labels of the targets, and the active rule groups returned by the Prometheus API must all point to the same result.

Verify that the intended ruleSelector was deployed using kubectl -n monitoring get prometheus -o yaml.
Verify that the metadata labels of the default rules match the selector in kubectl -n monitoring get prometheusrule --show-labels.
Verify that alerting rules and recording rules are actually loaded in curl -s http://127.0.0.1:9090/api/v1/rules.
Verify that the default group is visible on the Rules screen of the Prometheus UI and that there are no PrometheusRule reconcile errors in the Operator logs.
Since custom PrometheusRules must also follow the same selector policy, check the combination of separate namespaces and labels as well.

If the number of rules decreases after application or if the Prometheus configuration reload fails, check the previous successful revision using helm history kube-prometheus-stack -n monitoring and roll back using helm rollback kube-prometheus-stack REVISION_NUMBER -n monitoring. Then, compare the backed-up kube-prometheus-stack-values-before.yaml with the current values to isolate the selector changes again.

Local rendering with pinned versions

A separate development environment ran template only with Helm 3.18.6, Chart 65.8.1, and --kube-version 1.30.0. These are reproduction versions, not a current deployment recommendation. Both cases rendered 35 PrometheusRules. Metadata matching monitoring-role: cluster-rules numbered 0 before adding the label and 35 after setting it in defaultRules.labels. Grafana was disabled in both cases. No cluster, Operator, or Prometheus was contacted or installed, so active rule counts were not tested.

helm template lab ./kube-prometheus-stack-65.8.1.tgz --namespace monitoring --kube-version 1.30.0 -f mismatch.json > "$DIAG_DIR/mismatch.yaml"
helm template lab ./kube-prometheus-stack-65.8.1.tgz --namespace monitoring --kube-version 1.30.0 -f matched.json > "$DIAG_DIR/matched.yaml"

mismatch.json

{
  "grafana": {
    "enabled": false
  },
  "prometheus": {
    "prometheusSpec": {
      "ruleSelectorNilUsesHelmValues": false,
      "ruleSelector": {
        "matchLabels": {
          "monitoring-role": "cluster-rules"
        }
      }
    }
  }
}

matched.json

{
  "grafana": {
    "enabled": false
  },
  "prometheus": {
    "prometheusSpec": {
      "ruleSelectorNilUsesHelmValues": false,
      "ruleSelector": {
        "matchLabels": {
          "monitoring-role": "cluster-rules"
        }
      }
    }
  },
  "defaultRules": {
    "labels": {
      "monitoring-role": "cluster-rules"
    }
  }
}

Pinned chart release / 고정 Chart 릴리스

Official documentation and verification criteria

kube-prometheus-stack values reference — ruleSelectorNilUsesHelmValues, ruleSelector, and defaultRules.labels settings in the Chart
Helm template command — How to locally render the values.yaml results and PrometheusRule manifests before applying the Chart
Prometheus Operator API reference — Meaning of empty values, null values, and metadata label selectors for PrometheusRule

Earlier verification notes preserved from this article

Written by: Nole IT Operator

Verification level: Checked against official documentation

This article was reviewed against the documented command and configuration contracts available on the publication date. Prefer the documentation for your installed version when behavior differs, and validate changes with a dry run before production.

Editorial and revision policy · Report an error