Registry Mirror For Container Images

What is a Container Registry

A container registry is a catalogue of storage locations where you can push and pull container images.

However, the actual physical locations where images are stored are known as repositories. Each repository stores a collection of related images with the same name.

Each image within a repository represents a different version of the same container deployment. For example, on the popular container registry Docker Hub, nginx is the name of the repository that contains different versions of the Docker image for open-source web server installation NGINX.

A specific image is identified by either its tag or its own unique reference.

What is a Registry Mirror

A registry mirror is a local copy of a public registry that copies or mirrors the file structure of the public registry. For example, let's say that the path to your registry mirror is 10.101.101.100:443. When containerd encounters a request to pull an image such as gcr.io/etcd-development/etcd:v3.5.5, containerd attempts to pull that image, not from gcr.io, but from your registry at the following path: 10.101.101.100:443/etcd-development/etcd:v3.5.5. If the image isn't in this registry path, the image is pulled automatically from the public registry gcr.io.

Using a registry mirror provides the following benefits:

• Can be used as custom registry for images.

• Protects from public registry outages.

• Speeds up pod creation.

• Allows to conduct image vulnerability scanning.

• Avoids limits imposed by public registries on how frequently you can issue commands to them.

Using Registry Mirror with PMK clusters

Pre-requisites

• Container registry server set up in your network.

• If your registry server runs a private TLS certificate, you must have the certificate authority (CA) file.

• If your registry server needs authentication, you must have the login credentials.

• Must use containerd as a runtime.

• Supported only with PMK Qbert API and must be used to configure clusters.

• PMK support only the following repositories with this feature. Images with any other registry URL will be directly downloaded from the mentioned registry (See in figure - anyOtherRegistry).

  • platform9.io

  • docker.io

  • gcr.io

  • quay.io

  • registry.k8s.io

Understanding the API Body changes in cluster API

API spec located at https://platform9.com/docs/qbert/ref

Example:

Bash

curl --request POST \
 --url https://<account>.platform9.net/qbert/v4/{projectId}/clusters \
 --header "X-Auth-Token: {X-Auth-Token}" \
 --data '{
"customRegistryUrl": "https://10.101.101.100:443",
"customRegistryRepoPath": "infra/k8s"
"customRegistryUsername": "username",
"customRegistryPassword": "password",
"customRegistrySkipTls": false,
"customRegistrySelfSignedCerts": true,
"customRegistryCertPath": "/usr/local/share/reg-ca.crt",
.
.
.

API Parameter	Type	Description	default	Example
customRegistryUrl	string	Endpoint URL for the registry.	null	Eg: https://10.101.101.100:443, https://user.jfrog.io
customRegistryRepoPath		Path where all images are cached.	null	- Can be empty, meaning all images are cached at root. - Can be like infra where all images will be cached in infra repository. - Can be like infra/k8s where all images will be cached under infra/k8s
customRegistryUsername	string	Username for the registry.	null
customRegistryPassword	string	Password for the registry.	null
customRegistrySkipTls	boolean	To be used to skip TLS.	false
customRegistrySelfSignedCerts	boolean	To be used if the the registry has a self-signed certificate.	false
customRegistryCertPath	string	Path to the cert file on the nodes or the endpoint URL from where it can be fetched.	null	Can be of the following formats: - https://example.com/path/to/cert - http://example.com/path/to/cert - file:///path/to/cert - /path/to/cert

Note: Before using the API

• Add the CA certificate file (ca.cert file) at specified location "customRegistryCertPath": "/usr/local/share/reg-ca.crt" on the node. (this path should be readable by pf9 user) OR

• Store the CA certificate file at some endpoint say https://example.com/cacert and set this field to "customRegistryCertPath": "https://example.com/cacert" so that the nodelet can fetch the cert from this endpoint and place it on nodes and configure it into containerd.

Example Registry

As a user if you have setvup a private Jfrog registry like the below image:

API request body will have following parameters to use this feature:

Bash

"customRegistryUrl": "https://schoudhari.jfrog.io",
"customRegistryPath": "infra/k8s"
"customRegistryUsername": "username",
"customRegistryPassword": "password",

How this feature affects containerd?

Containerd configurations for registries are stored at /etc/containerd/certs.d on the nodes.

For example, if 10.101.101.100:443 is used as the registry mirror and in PMK there are multiple images from various registries, the /etc/containerd/certs.d will look like:

Bash

[root@test-pf9-qbert-bare-os-c7-cntrd-2855088-142 certs.d]# pwd
/etc/containerd/certs.d
[root@test-pf9-qbert-bare-os-c7-cntrd-2855088-142 certs.d]# tree
.
|-- 10.101.101.100:443
|   `-- hosts.toml
|-- docker.io
|   `-- hosts.toml
|-- gcr.io
|   `-- hosts.toml
|-- platform9.io
|   `-- hosts.toml
|-- quay.io
|   `-- hosts.toml
`-- registry.k8s.io
    `-- hosts.toml

PMK currently has above 5 registries. Along with that, we configure the custom registry 10.101.101.100:443 . The pull requests to all the registries will go through config in their hosts.toml.

Listed are examples of all registry hosts.toml files for this case:

• 10.101.101.100:443 ← custom registry

Bash

[root@test-pf9-qbert-bare-os-c7-cntrd-2855088-142 certs.d]# cat 10.101.101.100\:443/hosts.toml 
server = "https://10.101.101.100:443"

[host."https://10.101.101.100:443/v2/infra/k8s"]
  capabilities = ["pull", "resolve"]
  ca = "/usr/local/share/reg-ca.crt"
  override_path = true
  [host."https://10.101.101.100:443/v2/infra/k8s".header]
  authorization = "Basic d1h1YmhhbTpwZjck"

• docker.io

Bash

[root@test-pf9-qbert-bare-os-c7-cntrd-2855088-142 certs.d]# cat docker.io/hosts.toml 
server = "https://docker.io"

[host."https://10.101.101.100:443/v2/infra/k8s"]
  capabilities = ["pull", "resolve"]
  ca = "/usr/local/share/reg-ca.crt"
  override_path = true
  [host."https://10.101.101.100:443/v2/infra/k8s".header]
  authorization = "Basic d1h1YmhhbTpwZjck"

[host."https://dockermirror.platform9.io"]
  capabilities = ["pull", "resolve"]

[host."https://registry-1.docker.io"]
  capabilities = ["pull", "resolve"]

• platform9.io

Bash

[root@test-pf9-qbert-bare-os-c7-cntrd-2855088-142 certs.d]# cat platform9.io/hosts.toml 
server = "https://platform9.io"

[host."https://10.101.101.100:443/v2/infra/k8s"]
  capabilities = ["pull", "resolve"]
  ca = "/usr/local/share/reg-ca.crt"
  override_path = true
  [host."https://10.101.101.100:443/v2/infra/k8s".header]
  authorization = "Basic d1h1YmhhbTpwZjck"

[host."https://dockermirror.platform9.io"]
  capabilities = ["pull", "resolve"]

• quay.io

Bash

[root@test-pf9-qbert-bare-os-c7-cntrd-2855088-142 certs.d]# cat quay.io/hosts.toml 
server = "https://quay.io"

[host."https://10.101.101.100:443/v2/infra/k8s"]
  capabilities = ["pull", "resolve"]
  ca = "/usr/local/share/reg-ca.crt"
  override_path = true
  [host."https://10.101.101.100:443/v2/infra/k8s".header]
  authorization = "Basic d1h1YmhhbTpwZjck"
  
[host."https://dockermirror.platform9.io"]
  capabilities = ["pull", "resolve"]

• gcr.io

Bash

[root@test-pf9-qbert-bare-os-c7-cntrd-2855088-142 certs.d]# cat gcr.io/hosts.toml 
server = "https://gcr.io"

[host."https://10.101.101.100:443/v2/infra/k8s"]
  capabilities = ["pull", "resolve"]
  ca = "/usr/local/share/reg-ca.crt"
  override_path = true
  [host."https://10.101.101.100:443/v2/infra/k8s".header]
  authorization = "Basic d1h1YmhhbTpwZjck"
  
[host."https://dockermirror.platform9.io"]
  capabilities = ["pull", "resolve"]

• registry.k8s.io

Bash

[root@test-pf9-qbert-bare-os-c7-cntrd-2855088-142 certs.d]# cat registry.k8s.io/hosts.toml 
server = "https://registry.k8s.io"

[host."https://10.101.101.100:443/v2/infra/k8s"]
  capabilities = ["pull", "resolve"]
  ca = "/usr/local/share/reg-ca.crt"
  override_path = true
  [host."https://10.101.101.100:443/v2/infra/k8s".header]
  authorization = "Basic d1h1YmhhbTpwZjck"
  
[host."https://dockermirror.platform9.io"]
  capabilities = ["pull", "resolve"]

Using this feature during cluster creation

Cluster Create (API Reference)

• Use the Cluster POST API with the following parameters to use this feature. Example:

Bash

curl --request POST \
 --url https://<account>.platform9.net/qbert/v4/{projectId}/clusters \
 --header "X-Auth-Token: {X-Auth-Token}" \
 --data '{
  "customRegistryUrl": "https://10.101.101.100:443",
  "customRegistryUsername": "user",
  "customRegistryRepoPath": "infra/k8s",
  "customRegistryPassword": "password",
  "customRegistrySkipTls": false,
  "customRegistrySelfSignedCerts": true,
  "customRegistryCertPath": "http://10.101.101.100:8080/domain.crt",
  .
  .
  .

Using this feature with an existing cluster with cluster Kubernetes version upgrade

Cluster Create (API Reference)

• Use the Cluster POST API with the following parameters to use this feature. Example:

Bash

curl --request POST \
 --url https://<account>.platform9.net/qbert/v4/{project_uuid}/clusters/{uuid}/upgrade?type={type}&force={force} \
 --header "X-Auth-Token: {X-Auth-Token}" \
 --data '{
  "customRegistryUrl": "{string}",
  "customRegistryRepoPath": "{string}",
  "customRegistryUsername": "{string}",
  "customRegistryPassword": "{string}",
  "customRegistrySkipTls": "{boolean}",
  "customRegistrySelfSignedCerts": "{boolean}",
  "customRegistryCertPath": "{string}",
  .
  .
  .

This shall update all the the nodes with new containerd configuration.

Using this feature with an existing cluster with cluster update (no Kubernetes version upgrade)

Cluster Create (API Reference)

• Use the Cluster PUT API with the following parameters to use this feature. Example:

Bash

curl --request PUT \
 --url https://<account>.platform9.net/qbert/v4/{project_uuid}/clusters/{uuid} \
 --header "X-Auth-Token: {X-Auth-Token}" \
 --data '{
  "customRegistryUrl": "{string}",
  "customRegistryRepoPath": "{string}",
  "customRegistryUsername": "{string}",
  "customRegistryPassword": "{string}",
  "customRegistrySkipTls": "{boolean}",
  "customRegistrySelfSignedCerts": "{boolean}",
  "customRegistryCertPath": "{string}",
  .
  .
  .

Note: With cluster update, following commands have to be run on each node of the cluster to restart the nodelet phases. With cluster version upgrade, following takes places automatically during node upgrades.

Example:

Bash

root@test-pf9-bare-os-u20-3074234-211-1:/etc/containerd/certs.d# systemctl stop pf9-hostagent
root@test-pf9-bare-os-u20-3074234-211-1:/etc/containerd/certs.d# systemctl stop pf9-nodeletd
root@test-pf9-bare-os-u20-3074234-211-1:/etc/containerd/certs.d# su pf9 -s /bin/bash
pf9@test-pf9-bare-os-u20-3074234-211-1:/etc/containerd/certs.d$ /opt/pf9/nodelet/nodeletd phases stop
root@test-pf9-bare-os-u20-3074234-211-1:/etc/containerd/certs.d# systemctl start pf9-hostagent