Pf9 Kube Prometheus Helm Chart Upgrade

Upgrading kube-prometheus-stack version from v52.1.0 / v52.1.1 (monitoring version - v0.68.0) to v62.7.1 (monitoring version - v0.76.1). Disable prometheus monitoring addon.

Note: Please make sure that promethues monitoring add-on is not enabled on the cluster and in-cluster monitoring is being configured via pf9-kube-prometheus helm chart.

1. Delete the prometheus-node-exporter daemonset.

Bash

root@test-pf9-qbert-bare-os-u20-3375226-630-2:~# k get ds -n pf9-monitoring
NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
proms-prometheus-node-exporter 2 2 2 2 2 kubernetes.io/os=linux 15m

root@test-pf9-qbert-bare-os-u20-3375226-630-2:~# k delete ds proms-prometheus-node-exporter -n pf9-monitoring
daemonset.apps "proms-prometheus-node-exporter" deleted

1. Run following commands to update the CRDs before applying the upgrade.

Bash

kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.76.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagerconfigs.yaml --force-conflicts

kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.76.0/example/prometheus-operator-crd/monitoring.coreos.com_alertmanagers.yaml --force-conflicts

kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.76.0/example/prometheus-operator-crd/monitoring.coreos.com_podmonitors.yaml --force-conflicts

kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.76.0/example/prometheus-operator-crd/monitoring.coreos.com_probes.yaml --force-conflicts

kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.76.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheusagents.yaml --force-conflicts

kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.76.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheuses.yaml --force-conflicts

kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.76.0/example/prometheus-operator-crd/monitoring.coreos.com_prometheusrules.yaml --force-conflicts

kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.76.0/example/prometheus-operator-crd/monitoring.coreos.com_scrapeconfigs.yaml --force-conflicts

kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.76.0/example/prometheus-operator-crd/monitoring.coreos.com_servicemonitors.yaml --force-conflicts

kubectl apply --server-side -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/v0.76.0/example/prometheus-operator-crd/monitoring.coreos.com_thanosrulers.yaml --force-conflicts

 3. Add additional scrape config in the values.yaml file before upgrading the chart.

Requried

Ensure to update the additionalScrapeConfigs key in the values.yaml file being used with the following values.

Bash

additionalScrapeConfigs:
    - job_name: kubelet
      scrape_interval: 5m
      honor_labels: false
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - source_labels:
        - __metrics_path__
        target_label: metrics_path
        replacement: /metrics
    - job_name: cadvisor
      scrape_interval: 2m
      honor_labels: false
      metrics_path: /metrics/cadvisor
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)
      - source_labels:
        - __metrics_path__
        target_label: metrics_path
        replacement: /metrics/cadvisor
    - job_name: apiserver
      scrape_interval: 5m
      honor_labels: false
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        insecure_skip_verify: true
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https

4.   Upgrade the helm chart.<div data-gb-custom-block data-tag="tabs">

helm upgrade <chart-name> pf9-plus/kube-prometheus-stack -f values.yaml