OpenShift uses Hawkular Metrics as a metrics engine which stores the data persistently in a Cassandra database. When this is configured, CPU, memory and network-based metrics are viewable from the OpenShift Container Platform web console and are available for use by horizontal pod autoscalers. Chillax. To set this up it take just one command line switch. Sponsored by Ichibaby. This lab is another in the OpenShift MiniLabs series.
Objective
Let’s get up you up and running with metric collections using Hawkular, Cassandra and Heapster in just a few minutes.
Setup
Adding the –metrics parameter to the oc cluster up launch command will set up the Hawkular metrics infrastructure. Bump up the memory allocated to Docker and away you go.
$ oc cluster down $ oc cluster up --metrics
Instructions
Inspect the Hawkular Metrics Services
Command Line
From the command line, inspect the openshift-infra project as the cluster admin and verify that all the required containers (heapster, hawkular-cassandra and hawkular-metrics) are running. This may take some time.
$ oc login -u system:admin $ oc project openshift-infra $ oc get pods
Console
Alternatively, you can login in to the Console using credentials: admin/admin and inspect the Hawkular metrics related services at: https://127.0.0.1:8443/console/project/openshift-infra/overview
Verify Metrics URL is Accessible
Login to the Console as developer/developer and then point your Browser to: https://hawkular-metrics-openshift-infra.127.0.0.1.nip.io/hawkular/metrics. This is the metrics URL that would have been reported upon launching the cluster instance.
Generate Traffic
We need some container workload traffic. These instructions assume that you have completed the A/B Deployment MiniLab.
$ while true; do curl -s http://ab-cotd.127.0.0.1.nip.io/item.php | grep "data/" | cut -c 63-80; sleep 2; done
Verify Success
Command Line
Check out https://github.com/openshift/origin-metrics and https://github.com/openshift/origin-metrics/blob/master/docs/hawkular_metrics.adoc for command line, REST and python client techniques for accessing metric content. The Bearer string to pass to the REST invocation is described at http://www.hawkular.org/blog/2016/10/24/hawkular-metrics-openshift-and-grafana.html . In the example that follows, extract the Bearer ID from issuing an “oc whoami”. Then inject your pod ID noting demarcation using %2F in the query string.
$ oc login -u developer -p developer
$ oc project cotd
$ oc whoami --token
$ curl -H "Authorization: Bearer XXXXXXXXXXXXX" \
-X GET -H 'Content-Type: application/json;charset=UTF-8' \
--insecure \
-H "Hawkular-Tenant: cotd" \
-X GET https://hawkular-metrics.13.70.146.253.nip.io/hawkular/metrics | \
python -m json.tool
curl -H "Authorization: Bearer EfnH5vSRAPZJwY8y62s0J_G2C-NCPwwCFSr8ZigIwwM" \
-X GET -H 'Content-Type: application/json;charset=UTF-8' \
--insecure \
-H "Hawkular-Tenant: dev" \
-X GET https://hawkular-metrics.13.70.146.253.nip.io/hawkular/metrics/counters/cotd%2F492da9a4-83bc-11e7-a323-000d3ae0fb29%2Fuptime/data | python -m json.tool
Console
Point your Browser at the metrics Tab that now appears against each running pod instance and confirm as per screenshot below. For example, you can find this by lick inside the pod circle at: https://127.0.0.1:8443/console/project/cotd/overview.
Trivia
Read all about metrics at https://docs.openshift.com/enterprise/3.1/install_config/cluster_metrics.html and https://access.redhat.com/documentation/en/openshift-container-platform/3.3/paged/installation-and-configuration/chapter-28-enabling-cluster-metrics.
There is some weirdness when using the –metrics switch with oc cluster up such that the previous profile system state is not recovered. Nothing has been lost. Restarting without the –metrics will recover your previous changes. This is a known issue and is being attended to.