Alerting Setup Guide

Private Cloud Director (PCD) ships host- and VM-level metrics into a built-in, Prometheus-backed Grafana instance. This guide lists the metrics we recommend alerting on and walks through configuring those alerts in Grafana end-to-end.

By the end, you will have a contact point (email or another channel), an alert rule group for host health, alert rules that notify you when a host is at risk or when VM HA fires, and the know-how to route notifications and silence alerts during maintenance.

Prerequisites

• Access to your PCD Grafana instance with an Editor or Admin role.
• The PCD Prometheus data source (the default data source; it is preconfigured; you do not need to add it).
• For email notifications, SMTP configured in Grafana. In most PCD deployments, this is already set up. If email tests fail, contact your PCD administrator to confirm SMTP is enabled.
• Familiarity with the pcd: metric namespace. See the PCD Metrics Reference.

---

Recommended metrics to alert on

These are the highest-signal metrics for catching problems before they affect workloads. Start with the host-health metrics (the first four); they protect every VM on a host. The thresholds below are sensible starting points; tune them to your environment and your tolerance for noise.

#	Alert	PromQL expression	Condition	Pending period	Severity
1	VMHA event occurred	count_over_time(vmha_events_generated{phase="completed"}[10m])	> 0	0m	Critical
2	Hypervisor availability	sum(pcd:hyp_total)	< <expected host count>	5m	Critical
3	Host memory utilization high	100 * sum by (host) (pcd:hyp_mem_usage) / sum by (host) (pcd:hyp_mem_total)	> 85	5m	Warning
4	Host CPU utilization high	100 * sum by (host) (pcd:hyp_cpu_usage) / sum by (host) (pcd:hyp_cpu_total)	> 85	5m	Warning
5	Host disk capacity high	100 * pcd:hyp_disk_usage / pcd:hyp_disk_space	> 85	10m	Warning
6	VM memory pressure	pcd:vm_mem_usage_percent	> 90	10m	Warning
7	VM disk filling	100 * pcd:vm_used_storage / pcd:vm_total_storage	> 90	10m	Warning
8	VM CPU throttling	pcd:vm_cpu_throttling_percentage	> 25	10m	Warning

Notes:

• Alert 1 covers High Availability. It uses the dedicated, purpose-built VMHA metric vmha_events_generated (the same metric behind the built-in VMHA Dashboard) and fires when HA has actually evacuated a host. Because that metric is event-driven and absent when the cluster is healthy, this rule must be configured with No Data = Normal/OK (see the HA step), or it will sit in No Data. Host presence (a host going offline) is separately covered by alert 2, Hypervisor availability.
• For hypervisor availability, set the threshold to the number of hosts you expect to be reporting. The rule fires if any host stops reporting metrics.
• A pending period (Grafana's for duration) requires the condition to stay true for that long before the alert fires. This suppresses brief spikes. Use a short pending period for availability and a longer one for utilization metrics.
• Utilization metrics are computed from raw usage and total series, so the alert expression divides them. Disk and several other metrics carry extra labels (fstype, device, target_device); aggregate or filter as needed.

---

Step 1: Create a contact point (notification channel)

A contact point defines where notifications are sent. Email is the worked example below, but the same flow applies to any of the 23 integration types your PCD Grafana instance supports (see Other notification channels).

1. In the left navigation, go to Alerting → Contact points.
2. Click Add contact point.
3. Enter a Name, for example Email - Ops.
4. Under Integration, select Email.
5. In Addresses, enter one or more recipient addresses (comma-separated for multiple).
6. (Recommended) Expand Optional Email settings and adjust the message if you like. Then click Test to send a test notification and confirm email delivery works.
7. Click Save contact point.

If the test email does not arrive, SMTP is most likely not configured in this Grafana instance. Contact your PCD administrator before continuing.

Other notification channels

A contact point can use any of the integration types below, chosen from the Integration dropdown. A single contact point can bundle several integrations, so one alert can notify (for example) both email and Slack, and each integration has its own Test button.

Available integrations in this PCD Grafana build (23):

Category	Integrations
Email / chat / push	Email, Slack, Microsoft Teams, Discord, Google Chat, Telegram, Cisco Webex Teams, LINE, DingDing, WeCom, Threema Gateway, Pushover
Incident / on-call	PagerDuty, OpsGenie, VictorOps (Splunk On-Call), Grafana OnCall, Sensu Go
Generic / integration	Webhook, AWS SNS, Kafka REST Proxy, MQTT, Jira, Alertmanager

Set up specifics for the channels most teams use (all begin at Alerting → Contact points → Add contact point → Integration):

• Slack: use either a Slack Webhook URL, or enter a Bot User OAuth Token (xoxb-…) plus the target channel ID.
• PagerDuty: paste the Integration Key from a PagerDuty Events API v2 integration on the target service.
• Microsoft Teams: paste a Webhook URL. Teams now requires a Workflow (Power Automate) that accepts webhook requests; use the built-in template "Post to a channel when a webhook request is received" and copy its URL. (The older Office 365 connector URLs are retired.)
• OpsGenie: enter your API Key and the Alert API URL for your region.
• Webhook: enter a target URL and HTTP method (plus optional auth and a custom body) to integrate with any system not listed above.

For the full field reference of every integration, see Grafana's Configure contact points documentation.

Routing: which alert goes to which contact point

There are two ways to decide which contact point an alert uses:

• Per-rule (simplest): in the alert rule's Notifications section, select the contact point directly. This is what this guide does; notification policies are bypassed for that rule.
• Notification policies (scalable): leave rules on the default policy and let Alerting → Notification policies route by label. For example, send severity = critical to PagerDuty while everything else goes to email, or route team = pcd to a specific Slack channel. Policies also control grouping, timing (group wait / interval / repeat), and mute timings.

---

Step 2: Create the alert rule

The steps below use Host memory utilization high as the worked example. Repeat them for each metric in the recommended list, substituting the expression, condition, and pending period from the table.

1. Go to Alerting → Alert rules and click New alert rule.
2. Name: enter a clear, specific name, for example PCD - Host Memory Utilization High.
3. Define query and alert condition:

   • Set the data source to Prometheus (the PCD default).
   • Switch the query editor to Code mode and enter the expression:

     100 * sum by (host) (pcd:hyp_mem_usage) / sum by (host) (pcd:hyp_mem_total)

   • Set the query type to Instant (you are evaluating the current value).
   • Grafana adds a Reduce expression (Last) and a Threshold expression by default. In the Threshold expression, set the condition to IS ABOVE 85.
   • Confirm the Threshold expression is selected as the alert condition.
   • Click Preview to verify the rule evaluates without error and returns the expected value.
4. Set evaluation behavior:

   • Folder: select or create a folder to hold PCD alert rules, for example PCD Alerts.
   • Evaluation group: create or select a group, for example PCD Host Health, and set the evaluation interval (how often the rule is checked), for example 1m.
   • Pending period: set to 5m (from the recommended table).
5. Configure labels and notifications:

   • Add labels to classify the alert, for example severity = warning and team = pcd. Labels are useful for routing and filtering.
   • Under Notifications, choose Select contact point and pick the contact point you created in Step 1 (Email - Ops).
   • (Alternative) If your organization routes by notification policy instead, leave the default and ensure a matching policy directs these labels to the right contact point.
6. Add annotations (optional but recommended):

   • Summary: PCD host memory utilization above 85%.
   • Description: what the alert means and the first troubleshooting step, for example: "Hypervisor memory utilization has stayed above 85% for 5 minutes. Investigate VM placement and consider rebalancing or adding capacity."
7. Click Save rule and exit.

---

Step 3: Verify the alert

1. On Alerting → Alert rules, find your rule. Its state will be Normal when healthy, Pending while the condition is met but the pending period has not elapsed, and Firing once it fires.
2. The rule's health should be OK (not Error or No Data). An error usually means the query or threshold is misconfigured.
3. To validate the full notification path without waiting for a real incident, temporarily set the threshold so the condition is true now (for example, lower a utilization threshold below the current value, or set the availability threshold above your current host count). Save, confirm the rule moves to Firing, and check that the email arrives. Restore the production threshold afterward.

---

Step 4: Configure the High Availability (HA) alert

HA is a Tier 1 concern: in an HA-enabled cluster, PCD automatically restarts a failed host's VMs on other healthy hosts. The HA telemetry is the vmha_* metric family (vmha_events_generated and vmha_task_details), the same metrics behind the built-in VMHA Dashboard. They record that HA actually ran, which host(s) were affected, and which VMs were evacuated. They are event-driven: no series exist while the cluster is healthy, which drives the key configuration choice in step 4 below.

Create the VMHA event alert:

1. New alert rule → name it PCD - VMHA Event.
2. Data source Prometheus, Code mode, Instant query:

   count_over_time(vmha_events_generated{phase="completed"}[10m])

3. Threshold: IS ABOVE 0 (any completed VMHA event in the last 10 minutes).
4. Configure no data and error handling → Alert state if no data = Normal/OK. This is essential: vmha_events_generated has no series during healthy operation, so without this the rule would sit in No Data. You want it to fire only when the metric appears.
5. Evaluation: folder PCD Alerts, group PCD HA, interval 1m, pending period 0m (fire on the first evaluation that sees an event).
6. Labels severity = critical, category = ha. Notifications → your email contact point. Annotations: summary VM HA evacuated a host, confirm VMs recovered; surface detail with vmha_task_details (labels vmha_uuid, vm_name, phase).
7. Save. Note that this rule cannot be test-fired without a real evacuation, since the metric is only emitted by actual events.

Tip: notification content. Pair the event count with vmha_task_details{phase="completed"} to report how many VMs were evacuated and which ones, matching the VMHA Dashboard's "Total VMs evacuated" / "Evacuated VMs" panels.

What was set up for this exercise: the HA alert PCD - VMHA Event was created in the PCD HA group using the query above with No Data = OK, routed to the email contact point. It stays in Normal until a real evacuation occurs (it cannot be test-fired without one).

Pairing with host-down detection. This event alert tells you HA ran. To also be notified the moment a host drops out (before/independent of evacuation), rely on the Hypervisor availability alert (sum(pcd:hyp_total) below the expected host count) from Step 5, which is the host-presence signal in scope here.

---

Step 5: Add the remaining alerts

Repeat Step 2 for each remaining metric in the recommended list. Because they share an evaluation group and folder, they will be grouped together in the UI and evaluated on the same interval.

For quick setup, here are the expressions and conditions again:

Alert	Expression	Condition
VMHA event occurred	count_over_time(vmha_events_generated{phase="completed"}[10m])	IS ABOVE 0 (set No Data = Normal)
Hypervisor availability	sum(pcd:hyp_total)	IS BELOW expected host count
Host CPU utilization high	100 * sum by (host) (pcd:hyp_cpu_usage) / sum by (host) (pcd:hyp_cpu_total)	IS ABOVE 85
Host disk capacity high	100 * pcd:hyp_disk_usage / pcd:hyp_disk_space	IS ABOVE 85
VM memory pressure	pcd:vm_mem_usage_percent	IS ABOVE 90
VM disk filling	100 * pcd:vm_used_storage / pcd:vm_total_storage	IS ABOVE 90
VM CPU throttling	pcd:vm_cpu_throttling_percentage	IS ABOVE 25

---

Silencing alerts

A silence temporarily stops notifications for alerts matching a set of labels, without disabling or deleting the alert rule. The rule keeps evaluating and still shows its state in the UI; it just does not notify. Use silences for planned maintenance, a known issue you are already working on, or to mute a noisy rule (such as a validation/test threshold) until you tune it. Silences are time-bound and expire automatically, so you cannot accidentally mute an alert forever.

Create a silence

There are two entry points:

• From a notification: click Silence in the alert email (or in the alert detail view). The label matchers are pre-filled from that alert.
• From scratch: go to Alerting → Silences and click Add Silence. Confirm the Alertmanager selector (top right) is set to Grafana.

Then:

1. Silence start and end / Duration. Set the start and end times, or just type a Duration (for example 2h) and Grafana fills in the end time.
2. Refine affected alerts: add label matchers. Add one matcher per row: a Label, an Operator (=, !=, =~ regex, !~), and a Value. Matchers are ANDed together. For example, to silence only the critical PCD host alerts:

   Label	Operator	Value
   severity	=	critical
   team	=	pcd

   Use + Add matcher for more rows. Match on alertname to target one specific rule (for example alertname = PCD - Hypervisor Availability).
3. Comment (required). Note why the silence exists and, ideally, a ticket or change reference.
4. Check Affected alert instances. The preview lists every instance the matchers will suppress (state shown as Suppressed), so you can confirm the scope before saving. If it is too broad or too narrow, adjust the matchers.
5. Click Save silence.

Manage silences

Alerting → Silences lists every silence with its State (Active, Pending, or Expired), the alert rule targeted, the matching labels, how many alerts silenced, and the schedule. From the Actions column, you can:

• Unsilence: end an active silence immediately (for example, when maintenance finishes early).
• Edit: adjust the time window, matchers, or comment.

Use the Search by matchers box to find silences by label.

Tip: scope matchers as tightly as the situation needs. A broad matcher like team = pcd silences every PCD alert; adding alertname or severity limits the blast radius so you do not miss unrelated incidents during the window.

---

Tuning and best practices

• Tune thresholds to your environment. The values here are starting points. Watch the metrics for a week, then set thresholds above normal peaks to avoid alert fatigue.
• Use pending periods. Most utilization alerts should require the condition to hold for several minutes before firing, so momentary spikes do not page you.
• Set No data and Error behavior deliberately. For availability-style alerts, treating No Data as Alerting can be useful (no data may mean a host is down). For utilization alerts, No Data → Normal/OK is usually safer.
• Group related alerts in the same folder and evaluation group for easier management and consistent intervals.
• Silence during planned maintenance so expected alerts do not page the team. See Silencing alerts above.
• Note on disk and per-device metrics. pcd:hyp_disk_usage, pcd:hyp_disk_space, and the VM storage/IO metrics carry labels such as fstype, device, and target_device. Decide whether you want a single worst-case alert (max(...)) or per-device alerts, and aggregate accordingly.