Capturing Metrics from Docker

In this step, we will configure Docker to expose metrics and then use a receiver with the OpenTelemetry Collector to pull the metrics and export them to Cloud Observability.

Configure Docker

Open Docker Desktop
In Settings, click on Docker Engine

Add "metrics-addr": "127.0.0.1:9100" to the JSON configuration. The full configuration should look similar to this:

{
  "builder": {
    "gc": {
      "defaultKeepStorage": "20GB",
      "enabled": true
    }
  },
  "experimental": false,
  "metrics-addr": "127.0.0.1:9100"
}

Click Apply & restart

This configures Docker to expose metrics at the given address.

If you are not using Docker Desktop, you can update your Docker configuration by editing /etc/docker/daemon.json. Then restart the Docker daemon for the changes to take effect.

Configure the OpenTelemetry Collector

Open the collector configuration file (/opentelemetry/conf/config.yaml)

In the receivers section, add the following:

# Receiver to scrape metrics from Docker in Prometheus format
prometheus/docker:
  config:
    scrape_configs:
      - job_name: docker-otel-eg
        static_configs:
          - targets: ["host.docker.internal:9100"]

This configures a Prometheus receiver to scrape/pull metrics in Prometheus format from the given endpoint. This endpoint is the one we configured Docker to expose the metrics at.

In the pipelines section, add the following metrics pipeline:
```
metrics:
  receivers: [prometheus/docker]
  processors: [batch]
  exporters: [debug, otlp/lightstep]
```
This defines the pipeline for metrics which receives the metrics using the Prometheus receiver, batches them and then exports them to the logs as well as to Cloud Observability.

Get Your Cloud Observability Access Token

If you are continuing from a previous workshop and/or have already updated the access token in your collector configuration you can skip this step

Cloud Observability uses access tokens to determine which project to associate your telemetry data with when the data is ingested. When you create a project in Cloud Observability, an access token is created automatically. To get your access token, follow these steps:

Log in to your Cloud Observability account
Click on the Settings icon in the navigation bar
Click on Access tokens under TOKENS AND KEYS
Click the icon button next to your access token to copy the token to your clipboard
In your config.yaml file, replace {LIGHTSTEP_ACCESS_TOKEN} with the access token you just copied and save the file

Deploy and Test

Now we’re ready to test our configuration and see our metric data in Cloud Observability.

If your containers are still running, press Ctrl+C in the terminal and wait for them to gracefully stop
Run the following command to restart the containers
```
docker-compose up
```

Once the services are up and running you should see some metrics appears in the log output like this:

otel-collector  | Metric #1
otel-collector  | Descriptor:
otel-collector  |      -> Name: http.client.duration
otel-collector  |      -> Description: Measures the duration of outbound HTTP requests.
otel-collector  |      -> Unit: ms
otel-collector  |      -> DataType: Histogram
otel-collector  |      -> AggregationTemporality: Cumulative
otel-collector  | HistogramDataPoints #0
otel-collector  | Data point attributes:
otel-collector  |      -> http.method: Str(POST)
otel-collector  |      -> net.peer.name: Str(otel-collector)
otel-collector  |      -> net.peer.port: Int(4318)
otel-collector  |      -> http.status_code: Int(200)
otel-collector  |      -> http.flavor: Str(1.1)
otel-collector  | StartTimestamp: 2024-01-08 18:23:49.523 +0000 UTC
otel-collector  | Timestamp: 2024-01-08 18:39:02.726 +0000 UTC
otel-collector  | Count: 304
otel-collector  | Sum: 1143.508620
otel-collector  | Min: 0.654375
otel-collector  | Max: 16.337542
otel-collector  | ExplicitBounds #0: 0.000000
otel-collector  | ExplicitBounds #1: 5.000000
otel-collector  | ExplicitBounds #2: 10.000000
...

View the Metrics in Cloud Observability

Let’s take a look at a couple of the metrics coming from Docker.

It can sometimes take a few minutes for the metrics to make it to Cloud Observability. If you don’t see any metrics initially, wait a few moments and try again. If you still don’t see any metrics after that, double check your collector configuration and review the previous steps.

If you are not still logged into Cloud Observability, login again
Click on Notebooks in the navigation bar
In the query editor, type engine_daemon_container_states_containers and hit Enter
In the Group by textbox, type state and hit Enter. This will group the metrics by the unique state values.
Change the View as option to Table. Now you should see a list showing how many containers you have in each state (i.e. running, stopped, paused)

Continue querying the metric data and see what insights you can derive. To find out which metrics have been reported to Cloud Observability, go to Project Settings then Metric Details. This will provide a searchable list of metric names that are currently reporting to your project.

You now have useful metrics about Docker and your containers being ingested into Cloud Observability. Next, we will generate custom metrics from our services and explore these on the platform.

next: Custom Metrics