Sending Custom Logs

Consuming the existing logs produced by your applications and infrastructure is a great start and in many cases may satisfy your needs. You can enhance this solution though by creating and sending custom logs that provide additional insights or context about what is happening in your system. In systems that are also instrumented for tracing you can attach the trace context to the logs. This correlates the logs and traces and provides powerful investigative options in Cloud Observability. Let’s add a custom log to our web service.

Add the following packages to the dependencies section of the package.json file in both services (/users/package.json and /web/package.json)

"@opentelemetry/exporter-logs-otlp-grpc": "^0.46.0",
"@opentelemetry/sdk-logs": "^0.46.0",
"@opentelemetry/api-logs": "^0.46.0",

Edit /opentelemetry/src/instrumentation.js

Add the following statements at the top of the file with the other require statements

const { logs } = require("@opentelemetry/api-logs");
const {
  OTLPLogExporter,
} = require("@opentelemetry/exporter-logs-otlp-grpc");
const {
  LoggerProvider,
  BatchLogRecordProcessor,
} = require("@opentelemetry/sdk-logs");

Add the following after the require statements

const loggerExporter = new OTLPLogExporter({
  url: "http://otel-collector:4317",
});
const loggerProvider = new LoggerProvider();
loggerProvider.addLogRecordProcessor(
  new BatchLogRecordProcessor(loggerExporter)
);

logs.setGlobalLoggerProvider(loggerProvider);

This configures the logger provider with batch processing and exporting to your OpenTelemetry Collector

Edit the web service (/web/src/index.js)

Add this line to the require statements at the top

const { logs, SeverityNumber } = require("@opentelemetry/api-logs");

Add the following code right after the const meter = api.metrics.getMeter("default"); line
```
const logger = logs.getLogger("default");
```
This acquires a logger which you will use to emit your custom log

To emit the custom log, add this code before the res.json(response.data) line in the app.get("/api/data", ...) block

// log the response status and set the span and trace ID
logger.emit({
  severityNumber: SeverityNumber.INFO,
  severityText: "INFO",
  body: `Response from ${USERS_SERVICE_URL}/api/data was ${response.status} ${response.statusText}`,
  attributes: {
    span_id: activeSpan.spanContext().spanId,
    trace_id: activeSpan.spanContext().traceId,
  },
});

This will create and send the custom log with the response status from the API call. In the attributes section we set the trace context values (span and trace IDs) by getting them from the active span context.

Edit the collector configuration (/opentelemetry/conf/config.yaml)
Add otlp to the list of receivers in the logs pipeline. The logs pipeline configuration should look like this
```
logs:
  receivers: [otlp, filelog]
  processors: [batch]
  exporters: [otlp/lightstep]
```
Save your changes

Deploy and Test

Now let’s deploy and test your changes with the custom log.

If your containers are still running, press Ctrl+C in the terminal and wait for them to gracefully stop
Run the following command to restart the containers
```
docker-compose up --build
```
Once your services are up and running, make some requests to http://localhost:4000/api/data

View the Logs in Cloud Observability

If you are not still logged into Cloud Observability, login again
Click on the Logs icon the side navigation
In the search bar at the top, type Response from and hit Enter. You should see a few entries like this one
Notice in the log data that there are values from span_id and trace_id and there is an active button on the right to Open linked trace. Click on the buttom
This opens the trace view for the trace when this log was generated. In the trace view, the span that was active when the log was generated is selected

By adding the trace context to the log when we emitted it you are now able to jump from the log to the trace in Cloud Observability. This is powerful because you can better understand the context of what was happening in your system at the time of the log. The trace view gives you additional attributes about the request as well as data like the latency, error state and other calls that were upstream and/or downstream.

Before we wrap up this workshop, let’s go ahead and shut down your services by pressing Ctrl+C in the terminal.

next: Conclusion