# Troubleshooting Cluster Issues

### Cluster Creation <a href="#cluster-creation" id="cluster-creation"></a>

#### Public Cloud Provider <a href="#public-cloud-provider" id="public-cloud-provider"></a>

* Make sure the permissions for the account you provided to PMK as part of cloud provider creation has all the required privileges. See the AWS pre-requisites under Getting Started section for more details

#### Cluster Creation Fails for BareOS <a href="#cluster-creation-fails-for-bareos" id="cluster-creation-fails-for-bareos"></a>

* Navigate to Infrastructure -> Clusters tab.
* Click on the cluster name. This will take you to the cluster details page.
* Click on the “Node Health” tab

Here you should see detailed breakdown of which nodes failed to install and which specific steps failed. Next, check [Troubleshooting Node Issues](https://docs.platform9.com/managed-kubernetes/support/troubleshooting/troubleshooting-node-issues).

### Etcd <a href="#etcd" id="etcd"></a>

#### Heartbeat/Election Timeout Interval <a href="#heartbeatelection-timeout-interval" id="heartbeatelection-timeout-interval"></a>

```
2021-02-04 18:36:31.380207 W | etcdserver: failed to send out heartbeat on time (exceeded the 100ms timeout for 124.999498ms, to 92d6e239c543436)
2021-02-04 18:36:31.380220 W | etcdserver: server is likely overloaded
2021-02-04 18:36:31.382208 W | etcdserver: read-only range request "key:\"/registry/mutatingwebhookconfigurations/vault-agent-injector-cfg\" " with result "range_response_count:1 size:2723" took too long (264.355727ms) to execute
```

**ETCD\_HEARTBEAT\_INTERVAL -** This is the frequency with which the leader will notify followers that it is still the leader.

**ETCD\_ELECTION\_TIMEOUT -** This timeout is how long a follower node will go without hearing a heartbeat before attempting to become a leader itself.

By default, etcd uses a`100ms`heartbeat interval and`1000ms`election timeout.

```
# cat /etc/pf9/kube.env | grep -i etcd
export ETCD_HEARTBEAT_INTERVAL="1000"
export ETCD_ELECTION_TIMEOUT="10000"
```

#### Database  <a href="#database-size-exceeded" id="database-size-exceeded"></a>

#### Size Exceeded <a href="#database-size-exceeded" id="database-size-exceeded"></a>

<figure><img src="https://content.gitbook.com/content/MgloA6PW3VtptV1H2a6z/blobs/yGw4DouTmLdA2KBdexJK/image.png" alt=""><figcaption></figcaption></figure>

```
etcdserver: failed to apply request,took 2.429<C2><B5>s,request header:<ID:1920634987875929770 > txn:<compare:<target:MOD key:"/registry/services/endpoints/kube-system/kube-controller-manager" mod_revision:287319046 > success:<request_put:<key:"/registry/services/endpoints/kube-system/kube-controller-manager" value_size:473 >> failure:<>>,resp ,err is etcdserver: no space
```

* Stop the pf9-hostagent and nodeletd services on the master node(s).

```
sudo systemctl stop pf9-{hostagent,nodeletd}
```

* Issue a `stop` for the Nodelet phases.

```
/opt/pf9/nodelet/nodeletd phases stop
```

* In `/opt/pf9/pf9-kube/master_utils.sh` , modify the function `ensure_etcd__r_unning()`to add the following environment variable.

{% tabs %}
{% tab title="/opt/pf9/pf9-kube/master\_utils.sh" %}

```javascript
--volume ${ETCD_DATA_DIR}:/var/etcd/data \
        -e ETCD_DEBUG=${DEBUG}
        -e ETCD_QUOTA_BACKEND_BYTES=<size_in_bytes>"
```

{% endtab %}
{% endtabs %}

* Start the `pf9-hostagent` service.

```
sudo systemctl start pf9-hostagent
```

* Verify the size was correctly set by scraping the etcd metrics endpoint.

```
curl -L http://localhost:2379/metrics | grep etcd_server_quota_backend_bytes
```


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.platform9.com/managed-kubernetes/support/troubleshooting/troubleshooting-cluster-issues.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.