Cheatsheet

Monitoring and Logging

Metrics, alerts, logs and traces: PromQL, rule files, the log commands you reach for, and the SLO arithmetic.

PromQL

```
rate(http_requests_total[5m])
```
Requests per second over 5 minutes (always rate() a counter)

sum by (job) (rate(http_requests_total[5m]))

Add the rates up per job

sum(rate(http_requests_total{code=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))

The share of requests that fail

histogram_quantile(0.95, sum by (le) (rate(request_duration_seconds_bucket[5m])))

The 95th percentile latency

```
increase(jobs_processed_total[1h])
```
How many in the last hour

topk(5, sum by (instance) (rate(node_cpu_seconds_total{mode!="idle"}[5m])))

The five busiest instances

```
up == 0
```
Targets Prometheus could not scrape

avg_over_time(queue_depth[10m])  |  max_over_time(...)

Smooth or peak a gauge over a window

predict_linear(node_filesystem_avail_bytes[6h], 24 * 3600) < 0

Will the disk be full within a day?

The metrics text format

# HELP http_requests_total Total requests.
# TYPE http_requests_total counter
http_requests_total{method="GET",code="200"} 1027

A counter with labels: names end in _total, label values are quoted

```
# TYPE queue_depth gauge
queue_depth 12
```
A gauge goes up and down

request_duration_seconds_bucket{le="0.5"} 80
request_duration_seconds_bucket{le="+Inf"} 100
request_duration_seconds_sum 41.7
request_duration_seconds_count 100

A histogram: cumulative buckets, a sum and a count

```
Never use user ids or request ids as labels
```
Every distinct label value is a new time series; keep high-cardinality data in logs

Prometheus and Alertmanager files

global:
  scrape_interval: 15s
scrape_configs:
  - job_name: app
    metrics_path: /metrics
    static_configs:
      - targets: ["app:8080"]

A scrape job

rule_files: [alerts.yml]
alerting:
  alertmanagers:
    - static_configs:
        - targets: ["alertmanager:9093"]

Load rules, and say where to send alerts

- alert: InstanceDown
  expr: up == 0
  for: 5m
  labels: { severity: page }
  annotations:
    summary: "{{ $labels.instance }} is down"

An alert rule: the condition must hold for 5 minutes

- record: job:http_requests:rate5m
  expr: sum by (job) (rate(http_requests_total[5m]))

A recording rule: save a query under a new name

route:
  receiver: team-mail
  group_by: [alertname, job]
  routes:
    - matchers: [severity="page"]
      receiver: team-pager

Alertmanager: group alerts, send pages to the pager

promtool check config prometheus.yml  |  promtool check rules alerts.yml

Validate the files before you load them

Logs on the command line

journalctl -u nginx --since "1 hour ago" -p err

A service's errors from the last hour

```
journalctl -f
```
Follow the system log

jq -r 'select(.level=="error") | .service' app.log | sort | uniq -c | sort -rn

Count errors per service in JSON logs

jq -c 'select(.request_id=="req-0042")' app.log

Everything one request did, across services

jq -r 'select(.level=="error") | .ts[11:16]' app.log | sort | uniq -c

Errors per minute

```
grep -c " 5[0-9][0-9] " access.log
```
How many 5xx responses in an nginx access log

/var/log/app/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    create 0640 app app
}

A logrotate rule: a week of compressed logs, no error if empty or missing

```
sudo logrotate -d /etc/logrotate.conf
```
Dry run: show what would rotate

Grafana and OpenTelemetry

apiVersion: 1
datasources:
  - { name: Prometheus, type: prometheus, access: proxy, url: "http://prometheus:9090", isDefault: true }

Provision a data source from a file

```
label_values(up, job)  ->  $job
```
A dashboard variable, used in queries as job="$job"

"fieldConfig": { "defaults": { "unit": "percentunit", "thresholds": { "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 0.05 }] } } }

A panel unit and colour thresholds (dashboard JSON)

receivers: { otlp: { protocols: { grpc: {}, http: {} } } }
processors: { batch: {} }
exporters: { otlphttp: { endpoint: "http://tempo:4318" } }
service:
  pipelines:
    traces: { receivers: [otlp], processors: [batch], exporters: [otlphttp] }

A collector: receive, batch, export, and connect them in a pipeline

```
traceparent: 00-<trace id>-<parent span id>-01
```
The header that carries trace context from one service to the next

SLOs and error budgets

```
availability = 1 - errors / total requests
```
The SLI: the share of requests that succeed
```
error budget = 1 - SLO  (99.9% -> 0.1%)
```
What the target allows to fail
```
0.1% of 30 days = 43.2 minutes
```
The downtime a 99.9% SLO allows in a month
```
burn rate = error ratio / (1 - SLO)
```
1 spends the budget exactly over the window; 14.4 spends it in two days
```
(ratio[1h] > 14.4 * 0.001) and (ratio[5m] > 14.4 * 0.001)
```
A fast-burn page: both a long and a short window must agree

99% 3.65 days  |  99.9% 8.8 hours  |  99.99% 52.6 min  (per year)

What each "nine" of availability allows

Cheatsheet for the Monitoring and Logging path · Deboistech · All cheatsheets