Infrastructure and custom metrics
Watch hosts and containers with the OpenTelemetry Collector, Prometheus, Grafana Alloy or Vector, chart any metric your app sends, and alert on thresholds.
Watch your servers and containers next to your traffic, errors and traces: CPU, memory, disks, network and every container on each box, plus any metric your own app counts. Little Friend has no agent of its own. You run a standard one you may already know, the OpenTelemetry Collector, Prometheus, Grafana Alloy or Vector, and point it at Little Friend with a server key. The configs on this page are complete and ready to copy.
Endpoints and key
- OTLP:
https://in.littlefriend.io/v1/metrics, OTLP/HTTP metrics in protobuf or JSON, plain, gzip or deflate. - Prometheus remote write:
https://in.littlefriend.io/v1/prom/write, remote write 1.0 with snappy, the format Prometheus, Grafana Alloy and Vector send by default. - Header:
Authorization: Bearer lfs_..., your project's secret server key, the same kind that sends traces and logs. Remote write also takes HTTP basic auth with the key as the password.
Create a server key under Settings, Keys, or from the terminal. With --write-env .env it goes straight into the file Docker Compose reads, and is never printed. Keep it on the server.
littlefriend keys create --kind server --label "Metrics" \
--write-env .env --env-name LF_SERVER_KEYOpenTelemetry Collector
The OpenTelemetry Collector (the contrib build) reads the host with its hostmetrics receiver and every Docker container with docker_stats, and sends both every 30 seconds. This is the setup we suggest for a server running Docker. Put these two files in one folder on the box, with LF_SERVER_KEY in .env next to them, and start it.
# One box's metrics agent: the OpenTelemetry Collector with host metrics and Docker containers.
# LF_SERVER_KEY=lfs_... docker compose up -d
# The host's root is mounted read-only for host metrics; the Docker socket is read for container stats.
# Host networking: the network scraper reads the interfaces of the collector's own network namespace,
# so in a bridged container it would report the container's eth0, not the box's. Nothing listens
# except the collector's own telemetry on 127.0.0.1:8888.
# `uts: host` gives the collector the box's own hostname, which names the host on the Infrastructure page.
# Set OTEL_RESOURCE_ATTRIBUTES=host.name=<name> (in .env) to name it otherwise.
# The limits keep the agent from starving the box's apps.
services:
otel-collector:
image: otel/opentelemetry-collector-contrib:0.163.2
restart: unless-stopped
command: ["--config=/etc/otelcol/config.yaml"]
user: "0:0"
environment:
LF_SERVER_KEY: ${LF_SERVER_KEY:?set LF_SERVER_KEY to the project's server key}
OTEL_RESOURCE_ATTRIBUTES: ${OTEL_RESOURCE_ATTRIBUTES:-}
GOMEMLIMIT: 160MiB
volumes:
- ./otelcol.yaml:/etc/otelcol/config.yaml:ro
- /:/hostfs:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
network_mode: host
pid: host
uts: host
mem_limit: 256m
memswap_limit: 256m
cpus: 0.5# OpenTelemetry Collector (contrib) for one server: host metrics and Docker containers to Little Friend.
# Needs LF_SERVER_KEY (the project's lfs_ server key) and, inside a container, the host's root
# mounted at /hostfs and the Docker socket. See docker-compose.yml next to this file.
receivers:
hostmetrics:
collection_interval: 30s
root_path: /hostfs
scrapers:
cpu: {}
load: {}
memory:
metrics:
system.memory.utilization:
enabled: true
paging: {}
disk: {}
filesystem:
metrics:
system.filesystem.utilization:
enabled: true
exclude_mount_points:
match_type: regexp
mount_points: ["/var/lib/docker/.*", "/run/docker/.*", "/snap/.*", "/proc/.*", "/sys/.*"]
exclude_fs_types:
match_type: strict
fs_types: [overlay, nsfs, squashfs, tmpfs, devtmpfs, autofs, tracefs]
network:
exclude:
match_type: regexp
interfaces: ["^veth.*", "^br-.*", "^docker.*", "^lo$"]
processes: {}
system: {}
docker_stats:
endpoint: unix:///var/run/docker.sock
collection_interval: 30s
metrics:
container.restarts:
enabled: true
container.uptime:
enabled: true
processors:
memory_limiter:
check_interval: 1s
limit_mib: 150
spike_limit_mib: 30
resourcedetection:
detectors: [env, system]
system:
hostname_sources: [os]
batch:
timeout: 10s
exporters:
otlphttp/littlefriend:
endpoint: https://in.littlefriend.io
headers:
Authorization: Bearer ${env:LF_SERVER_KEY}
compression: gzip
service:
pipelines:
metrics:
receivers: [hostmetrics, docker_stats]
processors: [memory_limiter, resourcedetection, batch]
exporters: [otlphttp/littlefriend]
telemetry:
metrics:
readers:
- pull:
exporter:
prometheus:
host: 127.0.0.1
port: 8888# docker-compose.yml and otelcol.yaml side by side, LF_SERVER_KEY in .env
docker compose up -d
docker compose logs -f otel-collectorThe host's root is mounted read-only so the collector can read the host and not just its own container, and the Docker socket is read for container stats. Container filesystems, virtual network interfaces and loop disks are left out, by the config and again by Little Friend. The box shows up on the Infrastructure page within a minute.
How hosts are named
A host is named by the host.name resource attribute, which the resourcedetection processor fills in. Its env detector reads OTEL_RESOURCE_ATTRIBUTES first, then its system detector uses the hostname the collector sees. The compose file shares the box's hostname and network with the collector (the network scraper reads the interfaces it can see, so a bridged container would report only its own), so the box's own name is used. To name it otherwise, set OTEL_RESOURCE_ATTRIBUTES in .env:
# .env, next to docker-compose.yml
OTEL_RESOURCE_ATTRIBUTES=host.name=app-1Keep names stable: a new name is a new host. The OS, architecture, container id and image are kept on the host or container, never on a series, so a redeploy does not start new series.
Prometheus with node_exporter and cAdvisor
Already run Prometheus? Add a remote_write block and it sends what it scrapes. node_exporter covers the host and cAdvisor the containers. The host external label names the box on the Infrastructure page; without it, Little Friend reads hostname, nodename or instance without its port.
# Prometheus with node_exporter and cAdvisor, writing to Little Friend. Set `host` to the box's
# name: it names the host on the Infrastructure page. The key file holds the lfs_ server key.
global:
scrape_interval: 30s
external_labels:
host: app-1
scrape_configs:
- job_name: node
static_configs:
- targets: ["node-exporter:9100"]
- job_name: cadvisor
static_configs:
- targets: ["cadvisor:8080"]
remote_write:
- url: https://in.littlefriend.io/v1/prom/write
authorization:
credentials_file: /etc/prometheus/lf_server_keyFor one box with nothing yet, this runs all three. The key goes in a file, never in the config:
# One box with Prometheus, node_exporter and cAdvisor, writing to Little Friend.
# echo -n lfs_... > lf_server_key && docker compose -f docker-compose.prometheus.yml up -d
services:
prometheus:
image: prom/prometheus:v3.15.1
restart: unless-stopped
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
- ./lf_server_key:/etc/prometheus/lf_server_key:ro
node-exporter:
image: prom/node-exporter:v1.12.1
restart: unless-stopped
command: ["--path.rootfs=/host"]
pid: host
volumes:
- /:/host:ro,rslave
cadvisor:
image: gcr.io/cadvisor/cadvisor:v0.55.1
restart: unless-stopped
privileged: true
devices:
- /dev/kmsg
volumes:
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:roIf your setup prefers basic auth, the key is the password and the username can be anything:
remote_write:
- url: https://in.littlefriend.io/v1/prom/write
basic_auth:
username: littlefriend
password_file: /etc/prometheus/lf_server_keyA body Little Friend cannot decode is answered 400, which Prometheus does not retry. Remote write 2.0 is answered 415, so leave Prometheus on its default 1.0 format.
Grafana Alloy
Grafana Alloy has node_exporter and cAdvisor built in, so one process covers the host and its containers. Set LF_SERVER_KEY in its environment. The box is named by constants.hostname; when Alloy runs in a container, give the container the box's name or write the name in place of it.
// Grafana Alloy: node_exporter and cAdvisor built in, written to Little Friend.
// LF_SERVER_KEY holds the project's lfs_ server key. `host` names the box on the Infrastructure page:
// in a container, run Alloy with the host's UTS namespace (`--uts=host`) so the hostname is the box's.
prometheus.exporter.unix "host" {
rootfs_path = "/host"
}
prometheus.exporter.cadvisor "docker" {
docker_host = "unix:///var/run/docker.sock"
}
prometheus.scrape "box" {
targets = array.concat(prometheus.exporter.unix.host.targets, prometheus.exporter.cadvisor.docker.targets)
scrape_interval = "30s"
forward_to = [prometheus.remote_write.littlefriend.receiver]
}
prometheus.remote_write "littlefriend" {
external_labels = { host = constants.hostname }
endpoint {
url = "https://in.littlefriend.io/v1/prom/write"
bearer_token = sys.env("LF_SERVER_KEY")
}
}Vector
Vector's host_metrics source sends the host's CPU, memory, disks, filesystems, load and network over remote write, each metric tagged with host. It does not read containers: add cAdvisor and Prometheus, or the OpenTelemetry Collector, for those. Put the server key in a file at /etc/vector/secrets/lf_server_key: Vector reads environment variables in its config only with a flag it marks dangerous, so the config reads the key as a secret. Start it with vector --config vector.toml.
# Vector: host metrics written to Little Friend over Prometheus remote write.
# The project's lfs_ server key goes in /etc/vector/secrets/lf_server_key: Vector reads
# environment variables in its config only with a flag it calls dangerous, so the key is a
# secret file. Each metric carries a `host` tag.
# Run: vector --config vector.toml (the file name is not Vector's default).
[secret.lf]
type = "directory"
path = "/etc/vector/secrets"
[sources.host]
type = "host_metrics"
scrape_interval_secs = 30
collectors = ["cpu", "disk", "filesystem", "load", "host", "memory", "network"]
[sinks.littlefriend]
type = "prometheus_remote_write"
inputs = ["host"]
endpoint = "https://in.littlefriend.io/v1/prom/write"
healthcheck.enabled = false
auth.strategy = "bearer"
auth.token = "SECRET[lf.lf_server_key]"Your app's own metrics
Anything your app counts, such as orders placed, queue depth or job duration, goes to the same endpoint. Point any OpenTelemetry SDK's OTLP/HTTP metric exporter at https://in.littlefriend.io/v1/metrics with the server key. Every SDK reads the standard environment variables, so often no code changes. In the headers variable the space after Bearer is written %20.
export OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=https://in.littlefriend.io/v1/metrics
export OTEL_EXPORTER_OTLP_METRICS_HEADERS="Authorization=Bearer%20lfs_YOUR_SERVER_KEY"
export OTEL_EXPORTER_OTLP_METRICS_PROTOCOL=http/protobuf
export OTEL_SERVICE_NAME=api// npm install @opentelemetry/api @opentelemetry/sdk-metrics @opentelemetry/resources @opentelemetry/exporter-metrics-otlp-proto
import { metrics } from '@opentelemetry/api';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-proto';
import { resourceFromAttributes } from '@opentelemetry/resources';
import { MeterProvider, PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
const provider = new MeterProvider({
resource: resourceFromAttributes({ 'service.name': 'api' }),
readers: [
new PeriodicExportingMetricReader({
exporter: new OTLPMetricExporter({
url: 'https://in.littlefriend.io/v1/metrics',
headers: { Authorization: `Bearer ${process.env.LF_SERVER_KEY}` },
}),
exportIntervalMillis: 30_000,
}),
],
});
metrics.setGlobalMeterProvider(provider);
const meter = metrics.getMeter('checkout');
const orders = meter.createCounter('orders.placed');
const duration = meter.createHistogram('checkout.duration', { unit: 's' });
orders.add(1, { plan: 'pro' });
duration.record(0.42, { plan: 'pro' });# pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http
import os
from opentelemetry import metrics
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.resources import Resource
exporter = OTLPMetricExporter(
endpoint="https://in.littlefriend.io/v1/metrics",
headers={"Authorization": f"Bearer {os.environ['LF_SERVER_KEY']}"},
)
reader = PeriodicExportingMetricReader(exporter, export_interval_millis=30_000)
metrics.set_meter_provider(
MeterProvider(resource=Resource.create({"service.name": "api"}), metric_readers=[reader])
)
orders = metrics.get_meter("checkout").create_counter("orders.placed")
orders.add(1, {"plan": "pro"})Gauges, counters and histograms are read, cumulative or delta. Summaries are refused. An app that already exposes Prometheus metrics needs no change: add it as a scrape job and Prometheus forwards it with the rest.
scrape_configs:
- job_name: api
static_configs:
- targets: ["api:9464"]Keep attributes and labels to things with a few values, such as a plan, a route template or a status. A value shaped like an id is dropped, so a label per user or per request never becomes a new series.
What the Infrastructure page shows
Every host that reported in the last 72 hours, the ones in trouble first, with CPU, memory, its fullest disk, load, cores, uptime and containers over the last 5 minutes. A host is a warning at 85% CPU, memory or disk, critical at 95%, and not reporting after 5 silent minutes. Open a host for the standard views:
| View | What it draws |
|---|---|
| CPU | Total and per core, by state: user, system, iowait, steal and the rest. |
| Load | The 1, 5 and 15 minute load averages, next to the number of cores. |
| Memory | Used and available, and swap used. |
| Disk | Usage and inodes per mount, IO bytes and operations, and latency per operation. |
| Network | Bytes, packets and errors in and out, per interface. |
| Processes | Running and blocked processes, and uptime. |
| Containers | CPU, memory against its limit, restarts and network in and out, per container. |
The same views draw from any of the agents above. A chart the host sends nothing for says so instead of showing zero. A server you took down can be forgotten from its page; it comes back if it reports again.
Metrics explorer and dashboards
The explorer charts any metric: filter by attribute, group by up to 3 keys such as host.name or container.name, and pick how to read it. Gauges read as average, minimum, maximum, sum, last or count, counters as a rate per second or an increase, and histograms as p50, p95, p99, average, rate or count. A chart shows up to 20 groups, largest first. A metric keeps the names it was sent under, so searching for node_cpu_seconds_total finds system.cpu.time.
Save a chart as a line, area, bar or single number, and put up to 24 of them on a dashboard. A project can keep 100 saved charts and 20 dashboards. Editors and owners save them; anyone in the workspace can read them.
Alerts
Two alert kinds watch metrics, checked every minute:
- Metric threshold (
metric_threshold): a metric above or below a value for 5 minutes in a row by default (1 to 1,440), read the way the explorer reads it at one minute. It can watch every series together or each host, container or attribute value on its own. Each minute is decided 90 seconds after it ends, so late samples count, and a minute with no data never counts as past the threshold. It is over after 2 minutes in a row back on the right side. - Host not reporting (
host_not_reporting): one host, or every host that reported in the last 72 hours, silent for 5 minutes by default (2 to 1,440). It is over once every host it named reports again or is forgotten.
They email and post to Slack like every other alert, naming up to 5 hosts or groups with their values and linking to the host or the metric.
littlefriend alerts create --kind metric_threshold --metric system.cpu.load_average.5m \
--agg max --above 4 --for 10 --group-by host.name
littlefriend alerts create --kind host_not_reporting --for 5Limits and retention
- Retention: raw samples 2 days, 1-minute rollups 15 days, and hourly rollups as long as the project keeps its other totals. Charts read raw samples up to an hour, minute rollups up to 15 days and hourly ones beyond.
- Series: a series is a metric and its attribute values, and counts while it reported in the last 24 hours. A project may have 10,000 by default (100 to 200,000), one metric 1,000 (10 to 50,000), and the project 2,000 metric names. A new series past a limit is refused and counted; series already reporting keep reporting.
- Daily cap: 20,000,000 samples stored per UTC day by default (100,000 to 500,000,000). Past it, the rest of the day's samples are refused until midnight UTC, with a partial success on OTLP and a
400on remote write, neither of which agents retry. Samples never count against your event quota. - Exporter metrics:
standard, the default, keeps the node_exporter, cAdvisor and docker_stats metrics the standard views draw and leaves out the rest of what exporters send, such as per-process metrics and exporter runtimes likego_*andpromhttp_*.allkeeps every metric. Your app's own metrics are always kept. - Requests: up to 8 MB and 20,000 data points each. A series keeps at most 16 attributes, of which 10 your own, with values up to 128 characters.
Change the cap, the series limits and exporter metrics in the dashboard, or with the settings API. Editors and owners change them, and each change is audited.
curl -X PUT https://api.littlefriend.io/api/projects/prj_YOUR_PROJECT/metrics/settings \
-H "Authorization: Bearer $LITTLEFRIEND_TOKEN" -H "Content-Type: application/json" \
-d '{"exporterMetrics": "all", "maxSeries": 20000}'What is never kept
Agents send more than a chart needs. Little Friend drops the rest when it arrives, whatever your agent sent:
- Process command lines, arguments, paths, owners and ids.
- Environment variables, and any attribute that looks like one.
- Credentials, tokens and anything the log rules call a secret.
- Addresses: client and peer IPs, MAC addresses, URLs and queries. A host's own address is kept only as its name, since a Prometheus target is often its address.
- Emails, user names and user agents.
- Values shaped like an id: UUIDs, long hex, base64 or digit runs. Container labels and SDK instance ids go too, so series do not grow with every deploy.
See data we never collect for the same rules across the product.
One model
Every metric is stored under one name per meaning, OpenTelemetry's semantic convention names, so the same views draw from any agent. node_exporter, cAdvisor, Vector and docker_stats names are renamed on the way in, labels and values included: node_cpu_seconds_total{mode="iowait"} is stored as system.cpu.time{state="wait"}. Any other metric keeps its own name.
| View | OpenTelemetry Collector | node_exporter and cAdvisor | Stored as |
|---|---|---|---|
| CPU | system.cpu.time | node_cpu_seconds_total | system.cpu.time{cpu, state} |
| Load | system.cpu.load_average.1m, 5m, 15m | node_load1, node_load5, node_load15 | system.cpu.load_average.1m, 5m, 15m |
| Memory | system.memory.usage, system.memory.limit | node_memory_MemTotal_bytes, MemAvailable_bytes and others | system.memory.limit, .available, .usage{state} |
| Swap | system.paging.usage | node_memory_SwapTotal_bytes, SwapFree_bytes | system.paging.limit, .usage{state} |
| Disk usage and inodes | system.filesystem.usage, .inodes.usage | node_filesystem_size_bytes, avail_bytes, files, files_free | system.filesystem.limit, .usage, .inodes.limit, .inodes.usage |
| Disk IO and latency | system.disk.io, .operations, .operation_time | node_disk_read_bytes_total, written_bytes_total and others | the same, with direction read or write |
| Network | system.network.io, .packets, .errors, .dropped | node_network_receive_bytes_total and the other receive and transmit counters | the same, with direction receive or transmit |
| Processes | system.processes.count | node_procs_running, node_procs_blocked | system.processes.count{status} |
| Uptime | system.uptime | node_boot_time_seconds | system.uptime, system.boot_time |
| Container CPU | container.cpu.usage.total | container_cpu_usage_seconds_total | container.cpu.time, in seconds |
| Container memory | container.memory.usage.total, .usage.limit | container_memory_working_set_bytes, container_spec_memory_limit_bytes | container.memory.usage, container.memory.limit |
| Container network | container.network.io.usage.rx_bytes, tx_bytes | container_network_receive_bytes_total, transmit_bytes_total | container.network.io{direction} |
| Container restarts | container.restarts | changes of container_start_time_seconds | container.restarts, container.start_time |
From the terminal and AI assistants
The CLI lists hosts and containers and queries any metric. An AI assistant connected over MCP reads the same with get_hosts, get_host_metrics, list_metrics and query_metrics.
littlefriend infra hosts
littlefriend infra host app-1 --since 6h
littlefriend infra containers --host app-1
littlefriend metrics list --q http
littlefriend metrics query system.cpu.time --filter state=wait --group-by host.name --since 1h
littlefriend metrics query http.server.request.duration --filter service.name=api --agg p95