• OPENSHIFT

Automatically Update Red Hat Container Images on OpenShift 3.11

#sync , #Red Hat , #container , #imagestream

Automatically Update Red Hat Container Images on OpenShift 3.11

OpenShift manages container images using a registry. This is the place where it caches upstream container images and stores the images from your own builds as well. Each build or container image correlates to an ImageStream, which is an object that defines any number of related images by tags. For example, one specific version of a Ruby container might be v2.5-22, but you can have one ImageStream definition that holds ruby tags and correlating images for v2.5, v2.4, v2.3 and so on.

At Red Hat’s Online Container Catalog, searching for Ruby brings up the following for the 2.5 release:

ruby list

You can also see that the Container Catalog tracks the latest release history and details:

ruby list details

When using the OpenShift Ansible (Advanced) Installer, you will find that by default the installer adds some useful ImageStreams to your openshift project namespace from the openshift_examples [1] folder. Examples include the all-powerful Apache httpd, a useful Jenkins CI/CD server, a few scripting languages, and a few databases among many others. These are just a sampling of what you can find in the full Red Hat Container Catalog online [2].

When you deploy the Red Hat Container Registry, either as a part of your OpenShift cluster or external to it, you will get a web console where you can take a peak at what ImageStreams and container images you have in your registry. Here is an example of the corresponding Ruby ImageStream. Note that it has several tags for different major.minor versions. As a part of normal security and development, any one of those major.minor versions might get a new release at any time. You can see when each was last updated in the far right column.

registry [3]

From the console, you can pull some details as well.

$ oc get is -n openshift
NAME                                               DOCKER REPO                                                                      TAGS                           UPDATED
imagestreams/httpd                                 docker-registry.default.svc:5000/openshift/httpd                                 2.4,latest                     4 months ago
imagestreams/jenkins                               docker-registry.default.svc:5000/openshift/jenkins                               v3.5,v3.6,v3.7 + 2 more...     4 months ago
imagestreams/mariadb                               docker-registry.default.svc:5000/openshift/mariadb                               10.1,latest                    4 months ago
imagestreams/mongodb                               docker-registry.default.svc:5000/openshift/mongodb                               3.2,latest,2.4 + 1 more...     4 months ago
imagestreams/mysql                                 docker-registry.default.svc:5000/openshift/mysql                                 5.5,5.6,5.7 + 1 more...        4 months ago
imagestreams/nodejs                                docker-registry.default.svc:5000/openshift/nodejs                                0.10,4,6 + 1 more...           4 months ago
imagestreams/perl                                  docker-registry.default.svc:5000/openshift/perl                                  5.16,5.20,5.24 + 1 more...     4 months ago
imagestreams/php                                   docker-registry.default.svc:5000/openshift/php                                   7.0,latest,5.5 + 1 more...     18 hours ago
imagestreams/postgresql                            docker-registry.default.svc:5000/openshift/postgresql                            latest,9.2,9.4 + 1 more...     4 months ago
imagestreams/python                                docker-registry.default.svc:5000/openshift/python                                3.4,3.5,latest + 2 more...     19 hours ago
imagestreams/redis                                 docker-registry.default.svc:5000/openshift/redis                                 3.2,latest                     21 hours ago
imagestreams/ruby                                  docker-registry.default.svc:5000/openshift/ruby                                  latest,2.2,2.3 + 2 more...     21 hours ago

Note: This list is truncated from the default.

Along the way during your use of OpenShift, you may notice that those images never update. Months go by, and you realize that you are running old containers. Note on the right hand side of the output above that most of these images have not been updated in 3-4 months! A couple of them have been updated manually in a more recent time frame. One day you ask yourself, why? Well, by default we don’t want to push people into making changes they aren’t thinking about. So this is the default behavior. There are also concerns over supported and tested configurations [4].

The good news is that OpenShift can handle the management of checking for latest updates to upstream images and allowing such events to trigger new builds of your own custom images, to keep them up-to-date as well. This is the container equivalent to doing a yum update on your system to get the latest patches of your underlying infrastructure or middleware, such as upgrading your apache, your ruby, or your tomcat. You should be aware when implementing automatic image updates that some existing users may already have build triggers in place that kick off new builds and deployments when these images are updated. They will need to know you plan on changing this behavior. This is a great and powerful feature, but a change like this should always be tested and communicated with your users knowledge.

Here are a few steps to get you going toward automatic container image updates on OpenShift.

Configure imagePolicyConfig in the Master Config to Run a Scheduled Import Process

First, you need to set your ImagePolicyConfig on your masters to handle an update schedule. Again, note that the default setting does not have the scheduled import on!

Set these variables in your OpenShift Ansible Installer inventory file [5]:

openshift_master_image_policy_config={"MaxImagesBulkImportedPerRepository": "3", "DisableScheduledImport": "false", "MaxScheduledImageImportsPerMinute": "10", "ScheduledImageImportMinimumIntervalSeconds": "1800"}

Then run the playbook to just update your master configs:

# ansible-playbook -i /etc/ansible/hosts /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-master/config.yml

To do this manually without the ansible playbook, on each master node, add or edit the following in your /etc/origin/master/master-config.yaml:

imagePolicyConfig:
  MaxScheduledImageImportsPerMinute: 10
  ScheduledImageImportMinimumIntervalSeconds: 1800
  disableScheduledImport: false
  maxImagesBulkImportedPerRepository: 3

Be sure to restart services on each master after your update:

On OpenShift < = 3.9:

# systemctl restart atomic-openshift-master-api

On OpenShift > = 3.10:

# master-restart api
# master-restart controllers

Configure Existing ImageStreams to Update Automatically

Second, you need to update all your ImageStreams to automatically update. Note that the openshift_examples installed by the OpenShift Ansible Installer only install into your OpenShift environment ONCE. They never get updated. If new templates arrive in subsequent updates, they do get added, and some get removed. But they never get updated! [1] Here is how to change that.

To perform this operation on all ImageStreams in the openshift namespace using a CLI json editor tool, jq:

$ oc get is -n openshift -o json > openshift-is.json
$ jq '.items[].spec.tags[]? |= if .from.kind == "DockerImage" then .importPolicy.scheduled |= true else . end' openshift-is.json > openshift-is-scheduled.json
$ oc apply -f openshift-is-scheduled.json -n openshift

To perform this operation on just one imagestream, in this example the Redis 3.2 image:

$ oc patch is redis -p '{"spec":{"tags":[{"name":"3.2","importPolicy":{"scheduled":true}}]}}'

You can run a describe operation on any ImageStream and should now see a comment that informs us the image will update automatically from its upstream resource URL:

$ oc describe is redis | grep "updates automatically"
  updates automatically from registry registry.redhat.io/rhscl/redis-32-rhel7:latest

Notes

  • The tag "name" must be of kind "DockerImage" and not "ImageStreamTag"!

  • Once you apply the scheduled update to one tag, it will update all tags on the same object.

  • The final oc apply command above may output some errors about yaml formats. Ignore them, it gets the job done.

Configure New ImageStreams to Update Automatically

Third, when working with ImageStreams, performing an import-image or oc tag into any namespace, be sure to specify the flag --scheduled=true. Let’s test it out on a sample project space:

$ oc new-project test
Now using project "test" on server "https://openshift.example.com:8443".

You can add applications to this project with the 'new-app' command. For example, try:

    oc new-app centos/ruby-22-centos7~https://github.com/openshift/ruby-ex.git

to build a new example application in Ruby.

The latest version as of this writing, 3.11, uses an authenticated Red Hat registry at registry.redhat.io [6].

Copy your auth token from the openshift namespace just for this test. OpenShift versions prior to 3.11 don’t need this (yet).

$ oc get secret imagestreamsecret -n openshift --export -o yaml | oc create -f- -n test
secret/imagestreamsecret created

Now import an image with the -scheduled=true flag and notice the output below indicate it will update automatically (output truncated).

$ oc import-image ruby --from=registry.redhat.io/rhscl/ruby-25-rhel7 --confirm --scheduled=true
imagestream.image.openshift.io/ruby imported

Name:			ruby
Namespace:		test
Created:		12 minutes ago
Labels:			<none>
Annotations:		openshift.io/image.dockerRepositoryCheck=2018-11-12T21:36:36Z
Docker Pull Spec:	docker-registry.default.svc:5000/test/ruby
Image Lookup:		local=false
Unique Images:		1
Tags:			1

latest
  updates automatically from registry registry.redhat.io/rhscl/ruby-25-rhel7

  * registry.redhat.io/rhscl/ruby-25-rhel7@sha256:88b5a4ae11075034ef05eed69b17a5527eb44ae1352e660d02df96394eb258d7
      Less than a second ago

For further reading:

Image Configuration Parameters

See --scheduled=true flag on

For officially supported configurations: