> For the complete documentation index, see [llms.txt](https://docs.platform9.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.platform9.com/private-cloud-director/monitoring-and-observability/alerting-setup-guide.md). # Alerting Setup Guide Private Cloud Director (PCD) ships host- and VM-level metrics into a built-in, Prometheus-backed Grafana instance. This guide lists the metrics we recommend alerting on and walks through configuring those alerts in Grafana end-to-end. By the end, you will have a contact point (email or another channel), an alert rule group for host health, alert rules that notify you when a host is at risk or when VM HA fires, and the know-how to route notifications and silence alerts during maintenance. ## Prerequisites * Access to your PCD Grafana instance with an **Editor** or **Admin** role. * The PCD **Prometheus** data source (the default data source; it is preconfigured; you do not need to add it). * For **email** notifications, **SMTP configured** in Grafana. In most PCD deployments, this is already set up. If email tests fail, contact your PCD administrator to confirm SMTP is enabled. * Familiarity with the `pcd:` metric namespace. See the [PCD Metrics Reference](https://docs.platform9.com/private-cloud-director/monitoring-and-observability/monitoring/pcd-metrics-reference). *** ## Recommended metrics to alert on These are the highest-signal metrics for catching problems before they affect workloads. Start with the host-health metrics (the first four); they protect every VM on a host. The thresholds below are sensible starting points; tune them to your environment and your tolerance for noise.
#AlertPromQL expressionConditionPending periodSeverity
1VMHA event occurredcount_over_time(vmha_events_generated{phase="completed"}[10m])> 00mCritical
2Hypervisor availabilitysum(pcd:hyp_total)< <expected host count>5mCritical
3Host memory utilization high100 * sum by (host) (pcd:hyp_mem_usage) / sum by (host) (pcd:hyp_mem_total)> 855mWarning
4Host CPU utilization high100 * sum by (host) (pcd:hyp_cpu_usage) / sum by (host) (pcd:hyp_cpu_total)> 855mWarning
5Host disk capacity high100 * pcd:hyp_disk_usage / pcd:hyp_disk_space> 8510mWarning
6VM memory pressurepcd:vm_mem_usage_percent> 9010mWarning
7VM disk filling100 * pcd:vm_used_storage / pcd:vm_total_storage> 9010mWarning
8VM CPU throttlingpcd:vm_cpu_throttling_percentage> 2510mWarning
Notes: * Alert 1 covers **High Availability**. It uses the dedicated, purpose-built VMHA metric `vmha_events_generated` (the same metric behind the built-in *VMHA Dashboard*) and fires when HA has actually evacuated a host. Because that metric is **event-driven and absent when the cluster is healthy**, this rule must be configured with **No Data = Normal/OK** (see the HA step), or it will sit in No Data. Host *presence* (a host going offline) is separately covered by alert 2, **Hypervisor availability**. * For **hypervisor availability**, set the threshold to the number of hosts you expect to be reporting. The rule fires if any host stops reporting metrics. * A **pending period** (Grafana's `for` duration) requires the condition to stay true for that long before the alert fires. This suppresses brief spikes. Use a short pending period for availability and a longer one for utilization metrics. * Utilization metrics are computed from raw `usage` and `total` series, so the alert expression divides them. Disk and several other metrics carry extra labels (`fstype`, `device`, `target_device`); aggregate or filter as needed. *** ## Step 1: Create a contact point (notification channel) A contact point defines **where** notifications are sent. Email is the worked example below, but the same flow applies to any of the **23 integration types** your PCD Grafana instance supports (see [Other notification channels](#other-notification-channels)). 1. In the left navigation, go to **Alerting → Contact points**. 2. Click **Add contact point**. 3. Enter a **Name**, for example `Email - Ops`. 4. Under **Integration**, select **Email**. 5. In **Addresses**, enter one or more recipient addresses (comma-separated for multiple). 6. (Recommended) Expand **Optional Email settings** and adjust the message if you like. Then click **Test** to send a test notification and confirm email delivery works. 7. Click **Save contact point**. > If the test email does not arrive, SMTP is most likely not configured in this Grafana instance. Contact your PCD administrator before continuing. ### Other notification channels A contact point can use any of the integration types below, chosen from the **Integration** dropdown. A single contact point can bundle **several** integrations, so one alert can notify (for example) both email and Slack, and each integration has its own **Test** button. **Available integrations in this PCD Grafana build (23):** | Category | Integrations | | --------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | | Email / chat / push | Email, Slack, Microsoft Teams, Discord, Google Chat, Telegram, Cisco Webex Teams, LINE, DingDing, WeCom, Threema Gateway, Pushover | | Incident / on-call | PagerDuty, OpsGenie, VictorOps (Splunk On-Call), Grafana OnCall, Sensu Go | | Generic / integration | Webhook, AWS SNS, Kafka REST Proxy, MQTT, Jira, Alertmanager | Set up specifics for the channels most teams use (all begin at **Alerting → Contact points → Add contact point → Integration**): * **Slack**: use either a Slack **Webhook URL**, or enter a **Bot User OAuth Token** (`xoxb-…`) plus the target **channel ID**. * **PagerDuty**: paste the **Integration Key** from a PagerDuty **Events API v2** integration on the target service. * **Microsoft Teams**: paste a **Webhook URL**. Teams now requires a **Workflow** (Power Automate) that accepts webhook requests; use the built-in template *"Post to a channel when a webhook request is received"* and copy its URL. (The older Office 365 connector URLs are retired.) * **OpsGenie**: enter your **API Key** and the **Alert API URL** for your region. * **Webhook**: enter a target **URL** and HTTP method (plus optional auth and a custom body) to integrate with any system not listed above. For the full field reference of every integration, see Grafana's [Configure contact points](https://grafana.com/docs/grafana/latest/alerting/configure-notifications/manage-contact-points/) documentation. ### Routing: which alert goes to which contact point There are two ways to decide which contact point an alert uses: * **Per-rule (simplest):** in the alert rule's **Notifications** section, select the contact point directly. This is what this guide does; notification policies are bypassed for that rule. * **Notification policies (scalable):** leave rules on the default policy and let **Alerting → Notification policies** route by label. For example, send `severity = critical` to PagerDuty while everything else goes to email, or route `team = pcd` to a specific Slack channel. Policies also control grouping, timing (group wait / interval / repeat), and mute timings. *** ## Step 2: Create the alert rule The steps below use **Host memory utilization high** as the worked example. Repeat them for each metric in the recommended list, substituting the expression, condition, and pending period from the table. 1. Go to **Alerting → Alert rules** and click **New alert rule**. 2. **Name**: enter a clear, specific name, for example `PCD - Host Memory Utilization High`. 3. **Define query and alert condition**: * Set the data source to **Prometheus** (the PCD default). * Switch the query editor to **Code** mode and enter the expression: 
100 * sum by (host) (pcd:hyp_mem_usage) / sum by (host) (pcd:hyp_mem_total)
* Set the query type to **Instant** (you are evaluating the current value). * Grafana adds a **Reduce** expression (`Last`) and a **Threshold** expression by default. In the **Threshold** expression, set the condition to **IS ABOVE `85`**. * Confirm the **Threshold** expression is selected as the **alert condition**. * Click **Preview** to verify the rule evaluates without error and returns the expected value. 4. **Set evaluation behavior**: * **Folder**: select or create a folder to hold PCD alert rules, for example `PCD Alerts`. * **Evaluation group**: create or select a group, for example `PCD Host Health`, and set the **evaluation interval** (how often the rule is checked), for example **1m**. * **Pending period**: set to **5m** (from the recommended table). 5. **Configure labels and notifications**: * Add labels to classify the alert, for example `severity = warning` and `team = pcd`. Labels are useful for routing and filtering. * Under **Notifications**, choose **Select contact point** and pick the contact point you created in Step 1 (`Email - Ops`). * *(Alternative)* If your organization routes by notification policy instead, leave the default and ensure a matching policy directs these labels to the right contact point. 6. **Add annotations** (optional but recommended): * **Summary**: `PCD host memory utilization above 85%`. * **Description**: what the alert means and the first troubleshooting step, for example: *"Hypervisor memory utilization has stayed above 85% for 5 minutes. Investigate VM placement and consider rebalancing or adding capacity."* 7. Click **Save rule and exit**. *** ## Step 3: Verify the alert 1. On **Alerting → Alert rules**, find your rule. Its state will be **Normal** when healthy, **Pending** while the condition is met but the pending period has not elapsed, and **Firing** once it fires. 2. The rule's **health** should be **OK** (not **Error** or **No Data**). An error usually means the query or threshold is misconfigured. 3. To validate the full notification path without waiting for a real incident, temporarily set the threshold so the condition is true now (for example, lower a utilization threshold below the current value, or set the availability threshold above your current host count). Save, confirm the rule moves to **Firing**, and check that the email arrives. **Restore the production threshold afterward.** *** ## Step 4: Configure the High Availability (HA) alert HA is a Tier 1 concern: in an HA-enabled cluster, PCD automatically restarts a failed host's VMs on other healthy hosts. The HA telemetry is the **`vmha_*` metric family** (`vmha_events_generated` and `vmha_task_details`), the same metrics behind the built-in *VMHA Dashboard*. They record that HA actually ran, which host(s) were affected, and which VMs were evacuated. They are **event-driven**: no series exist while the cluster is healthy, which drives the key configuration choice in step 4 below. **Create the VMHA event alert:** 1. **New alert rule** → name it `PCD - VMHA Event`. 2. Data source **Prometheus**, **Code** mode, **Instant** query: ```promql count_over_time(vmha_events_generated{phase="completed"}[10m]) ``` 3. **Threshold**: **IS ABOVE `0`** (any completed VMHA event in the last 10 minutes). 4. **Configure no data and error handling → Alert state if no data = Normal/OK.** This is essential: `vmha_events_generated` has no series during healthy operation, so without this the rule would sit in No Data. You want it to fire only when the metric *appears*. 5. Evaluation: folder `PCD Alerts`, group `PCD HA`, interval `1m`, pending period `0m` (fire on the first evaluation that sees an event). 6. Labels `severity = critical`, `category = ha`. Notifications → your email contact point. Annotations: summary `VM HA evacuated a host, confirm VMs recovered`; surface detail with `vmha_task_details` (labels `vmha_uuid`, `vm_name`, `phase`). 7. Save. Note that this rule **cannot be test-fired** without a real evacuation, since the metric is only emitted by actual events. > **Tip: notification content.** Pair the event count with `vmha_task_details{phase="completed"}` to report how many VMs were evacuated and which ones, matching the VMHA Dashboard's "Total VMs evacuated" / "Evacuated VMs" panels. **What was set up for this exercise:** the HA alert `PCD - VMHA Event` was created in the `PCD HA` group using the query above with **No Data = OK**, routed to the email contact point. It stays in Normal until a real evacuation occurs (it cannot be test-fired without one). > **Pairing with host-down detection.** This event alert tells you HA *ran*. To also be notified the moment a host drops out (before/independent of evacuation), rely on the **Hypervisor availability** alert (`sum(pcd:hyp_total)` below the expected host count) from Step 5, which is the host-presence signal in scope here. *** ## Step 5: Add the remaining alerts Repeat **Step 2** for each remaining metric in the recommended list. Because they share an evaluation group and folder, they will be grouped together in the UI and evaluated on the same interval. For quick setup, here are the expressions and conditions again: | Alert | Expression | Condition | | ------------------------- | ----------------------------------------------------------------------------- | --------------------------------- | | VMHA event occurred | `count_over_time(vmha_events_generated{phase="completed"}[10m])` | IS ABOVE 0 (set No Data = Normal) | | Hypervisor availability | `sum(pcd:hyp_total)` | IS BELOW *expected host count* | | Host CPU utilization high | `100 * sum by (host) (pcd:hyp_cpu_usage) / sum by (host) (pcd:hyp_cpu_total)` | IS ABOVE 85 | | Host disk capacity high | `100 * pcd:hyp_disk_usage / pcd:hyp_disk_space` | IS ABOVE 85 | | VM memory pressure | `pcd:vm_mem_usage_percent` | IS ABOVE 90 | | VM disk filling | `100 * pcd:vm_used_storage / pcd:vm_total_storage` | IS ABOVE 90 | | VM CPU throttling | `pcd:vm_cpu_throttling_percentage` | IS ABOVE 25 | *** ## Silencing alerts A **silence** temporarily stops notifications for alerts matching a set of labels, without disabling or deleting the alert rule. The rule keeps evaluating and still shows its state in the UI; it just does not notify. Use silences for planned maintenance, a known issue you are already working on, or to mute a noisy rule (such as a validation/test threshold) until you tune it. Silences are time-bound and **expire automatically**, so you cannot accidentally mute an alert forever. ### Create a silence There are two entry points: * **From a notification:** click **Silence** in the alert email (or in the alert detail view). The label matchers are pre-filled from that alert. * **From scratch:** go to **Alerting → Silences** and click **Add Silence**. Confirm the Alertmanager selector (top right) is set to **Grafana**. Then: 1. **Silence start and end / Duration.** Set the start and end times, or just type a **Duration** (for example `2h`) and Grafana fills in the end time. 2. **Refine affected alerts: add label matchers.** Add one matcher per row: a **Label**, an **Operator** (`=`, `!=`, `=~` regex, `!~`), and a **Value**. Matchers are ANDed together. For example, to silence only the critical PCD host alerts: | Label | Operator | Value | | ---------- | -------- | ---------- | | `severity` | `=` | `critical` | | `team` | `=` | `pcd` | Use **+ Add matcher** for more rows. Match on `alertname` to target one specific rule (for example `alertname = PCD - Hypervisor Availability`). 3. **Comment** (required). Note why the silence exists and, ideally, a ticket or change reference. 4. **Check Affected alert instances.** The preview lists every instance the matchers will suppress (state shown as **Suppressed**), so you can confirm the scope before saving. If it is too broad or too narrow, adjust the matchers. 5. Click **Save silence**. ### Manage silences **Alerting → Silences** lists every silence with its **State** (Active, Pending, or Expired), the **alert rule targeted**, the **matching labels**, how many **alerts silenced**, and the **schedule**. From the **Actions** column, you can: * **Unsilence**: end an active silence immediately (for example, when maintenance finishes early). * **Edit**: adjust the time window, matchers, or comment. Use the **Search by matchers** box to find silences by label. > **Tip:** scope matchers as tightly as the situation needs. A broad matcher like `team = pcd` silences *every* PCD alert; adding `alertname` or `severity` limits the blast radius so you do not miss unrelated incidents during the window. *** ## Tuning and best practices * **Tune thresholds to your environment.** The values here are starting points. Watch the metrics for a week, then set thresholds above normal peaks to avoid alert fatigue. * **Use pending periods.** Most utilization alerts should require the condition to hold for several minutes before firing, so momentary spikes do not page you. * **Set `No data` and `Error` behavior deliberately.** For availability-style alerts, treating **No Data** as **Alerting** can be useful (no data may mean a host is down). For utilization alerts, **No Data → Normal/OK** is usually safer. * **Group related alerts** in the same folder and evaluation group for easier management and consistent intervals. * **Silence during planned maintenance** so expected alerts do not page the team. See [Silencing alerts](#silencing-alerts) above. * **Note on disk and per-device metrics.** `pcd:hyp_disk_usage`, `pcd:hyp_disk_space`, and the VM storage/IO metrics carry labels such as `fstype`, `device`, and `target_device`. Decide whether you want a single worst-case alert (`max(...)`) or per-device alerts, and aggregate accordingly. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.platform9.com/private-cloud-director/monitoring-and-observability/alerting-setup-guide.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.