Agents, log drains and edge events

Search crawlers, AI crawlers and AI agents read your site every day. Google Analytics filters known bots out and doesn't show them. Little Friend gives them their own report, and says how sure it is about each one. With the door, you also decide which of them get in.

Why a browser script can't see them

Most crawlers fetch your HTML and never run JavaScript, so lf.js never loads for them. To see them, Little Friend needs the request itself: from a Vercel log drain, or from your CDN, edge functions or server middleware with @littlefriend/edge. Without one, the Agents report says it's showing browser activity only.

What kind of visitor

How we know who runs it

Every verdict carries its evidence. An operator name, like OpenAI or Anthropic, is only as strong as the evidence behind it.

Verdicts store the version of the rules that made them, so a rule change never rewrites history.

Sites on Vercel: add a log drain

A log drain makes Vercel report every request your site serves, crawlers and AI agents included, with no code change. It works for static sites too. Drains need a Vercel Pro or Enterprise team.

1. In the dashboard, open your site's Agents page and choose Add a Vercel log drain. Copy the endpoint, the header and the signing secret. The secret is shown once.
2. In Vercel, add a log drain in your team's settings. Pick this project only, all sources, production and 100% sampling. Paste the endpoint, the header and the signing secret from step 1.
3. Within minutes of traffic, the Coverage card on the Agents page shows the drain as Live.

Refused deliveries are listed under the drain. bad_signature means the secret in Vercel doesn't match.

Any host: add @littlefriend/edge

The package records each request your code serves, with the status the visitor got, and sends it with an edge key. Use it when your site isn't on Vercel, or alongside a drain: a request both of them report is counted once. It runs on Cloudflare Workers, Vercel, Deno, Bun and Node 18 or later.

pnpm add @littlefriend/edge

Create an edge key in Settings, Keys. It starts with lfe_. Make one observer per process or isolate, at module scope, and reuse it for every request. Observing never delays or fails a response.

import { withLittleFriend } from '@littlefriend/edge';

interface Env {
  LF_EDGE_KEY: string;
}

const worker = {
  async fetch(request: Request, env: Env) {
    return fetch(request);
  },
};

export default withLittleFriend(worker, (env: Env) => ({
  key: env.LF_EDGE_KEY,
  ipHeader: 'cf-connecting-ip',
}));

import express from 'express';
import { createEdge, nodeMiddleware } from '@littlefriend/edge';

const edge = createEdge({
  key: process.env.LF_EDGE_KEY!,
  ipHeader: 'x-forwarded-for', // only a header your own proxy sets
});

const app = express();
app.use(nodeMiddleware(edge)); // first, so it sees every response

In Next.js, observe where the response is known. The proxy (proxy.ts, or middleware.ts before Next.js 16) runs before the page renders, so it never sees the status the visitor gets. Wrap route handlers instead (on Vercel, pass waitUntil from @vercel/functions), use nodeMiddleware in a custom Node server, or put a Cloudflare Worker in front of the site. The door is the one piece that belongs in the proxy, because it decides before the page renders.

Ignored routes: leave out your own API

On an app, most requests a drain or the door sees are the app calling its own API: session checks, polling, presence. None of them is a crawler or an agent reading your site, so every project starts with /api as an ignored route. Requests to an ignored route and everything below it are counted, and nothing else about them is kept. /api covers /api/session, not /apiary.

Change the list in Settings, Project, or with littlefriend projects update --add-ignored-route /trpc. Remove /api to see who calls your API. The door still decides on these requests, and a request a door rule matches is still reported. The Coverage card on the Agents page shows how many requests each source ignored.

Send observations yourself

From another language, post what your edge saw to /v1/edge with the same edge key. Either way, the IP address and full user agent are used to classify and verify the visitor, and never stored.

{
  "v": 1,
  "observations": [
    {
      "at": 1790346600000,
      "method": "GET",
      "host": "example.com",
      "path": "/docs/install",
      "status": 200,
      "bytes": 18432,
      "durationMs": 41,
      "cache": "hit",
      "ua": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)",
      "ip": "203.0.113.7",
      "country": "US"
    }
  ]
}

// A Cloudflare Worker in front of your site
export default {
  async fetch(request, env, ctx) {
    const started = Date.now();
    const response = await fetch(request);
    const url = new URL(request.url);

    ctx.waitUntil(fetch('https://in.littlefriend.io/v1/edge', {
      method: 'POST',
      headers: {
        authorization: `Bearer ${env.LF_EDGE_KEY}`,
        'content-type': 'application/json',
      },
      body: JSON.stringify({
        v: 1,
        observations: [{
          at: started,
          method: request.method,
          host: url.hostname,
          path: url.pathname,
          status: response.status,
          durationMs: Date.now() - started,
          ua: request.headers.get('user-agent') ?? '',
          ip: request.headers.get('cf-connecting-ip') ?? undefined,
          country: request.cf?.country,
          asn: request.cf?.asn,
          signature: request.headers.get('signature') ?? undefined,
          signatureInput: request.headers.get('signature-input') ?? undefined,
          signatureAgent: request.headers.get('signature-agent') ?? undefined,
        }],
      }),
    }));

    return response;
  },
};

On a busy site, batch them: one call carries up to 50 observations. Signature headers pass through untouched, so signed agents can be verified.

The door: choose which agents get in

The door decides on each request before your app runs, inside your Worker, proxy or server. It names the agent, checks its signature and its operator's addresses, then lets it in, slows it down, blocks it or asks it to sign, following rules you set in the dashboard.

Did your agent get a 401 from a site? The site asks agents to sign their requests. Sign with Web Bot Auth.

The door comes with @littlefriend/edge 0.2.0 and later, and uses the same edge key as the observer. It decides with a copy of your rules that it keeps fresh in the background, so your requests don't wait on Little Friend.

Cloudflare Workers

Add door: createDoor to the Worker above:

import { createDoor, withLittleFriend } from '@littlefriend/edge';

export default withLittleFriend(worker, (env: Env) => ({
  key: env.LF_EDGE_KEY,
  ipHeader: 'cf-connecting-ip',
  door: createDoor,
}));

The door uses the same key, endpoint and headers as the observer. Passing createDoor, rather than a flag, keeps the door out of the bundle of a Worker that only observes.

Next.js

Add it to proxy.ts on Next.js 16. On Next.js 15, name the file middleware.ts and export middleware instead of proxy.

import { createDoor, createEdge, nextProxy } from '@littlefriend/edge';
import { NextResponse } from 'next/server';

const lf = { key: process.env.LF_EDGE_KEY!, ipHeader: 'x-forwarded-for' };
export const proxy = nextProxy(createDoor(lf), NextResponse, createEdge(lf));
export const config = { matcher: ['/((?!_next/static|_next/image|favicon.ico).*)'] };

Pages and route handlers get the request with the door's x-lf-agent header. A proxy never sees the status your app answers with, so for each agent it lets through, nextProxy sends the door's decision alone. Those decisions fill the door's report, dry run included. The other Agents pages need a response, so they count these requests from a Vercel log drain or from observers where the response is known: route handlers, a custom server, or a Worker in front.

Next.js route handlers

Route handlers can run the door themselves, and record every response exactly. Create the door and the observer once, in a shared module, and leave these routes out of the proxy's matcher so each request meets the door once.

import { createDoor, createEdge } from '@littlefriend/edge';

const lf = { key: process.env.LF_EDGE_KEY!, ipHeader: 'x-forwarded-for' };
export const edge = createEdge(lf);
export const door = createDoor(lf);

import { wrapFetch } from '@littlefriend/edge';
import { waitUntil } from '@vercel/functions';
import { door, edge } from '@/lib/lf';

export const GET = wrapFetch(edge, handler, { door, waitUntil });

Node, Express and Connect

import { createServer } from 'node:http';
import { createDoor, createEdge, nodeDoor } from '@littlefriend/edge';

const lf = { key: process.env.LF_EDGE_KEY!, ipHeader: 'x-forwarded-for' };
const gate = nodeDoor(createDoor(lf), createEdge(lf));
createServer((req, res) => gate(req, res, () => handler(req, res))).listen(3000);

With Express or Connect, use app.use(nodeDoor(createDoor(lf), createEdge(lf))) in place of nodeMiddleware. It records every response exactly, with the door's decision, and your app reads x-lf-agent from req.headers.

Bun, Deno and other fetch handlers

Pass the door to wrapFetch(edge, handler, { door }), or run it yourself inside your handler:

const admission = await door.admit(request);
if (admission.response) return admission.response;
const response = await app(new Request(request, { headers: admission.headers() }));
edge.observe(request, response, { door: admission });

admit never throws. Its answer has the action, the ruleId that decided, the mode, the policy version, what it knows about the agent and its principal, and in live mode the response to send instead of running your app.

Dry run, then live

Every project starts in dry run. The door makes every decision and records what it would have done, and never changes a response.

1. Open the door. In the dashboard, open the Agents page and choose The door.
2. Pick a preset or write rules. The preview shows what they would have done to the last 7 days of agent requests.
3. Save and watch. The door's report shows what each rule decided, still in dry run.
4. Go live when the rules do what you want. Your servers pick up saved rules within about a minute.

Rules and presets

A rule can match an agent's class, purpose, operator or product, how well it proved who it is, who it acts for, a failed operator check, path prefixes and methods. The first rule that matches decides. A request no rule matches gets in.

A preset replaces your rules in the editor. Nothing changes at the door until you save.

What each action does

Every answer from the door is plain text with Cache-Control: no-store, so a shared cache never serves it to anyone else. In dry run the door sends none of them.

The x-lf-agent header

Every request the door lets through reaches your app with one header, an RFC 8941 dictionary:

x-lf-agent: class=ai_agent, purpose=ai_agent, operator="OpenAI", product="ChatGPT agent", evidence=signature_verified, principal=signed_user, spoofed=?0

A browser reads class=browser, purpose=human, evidence=none, principal=none, spoofed=?0. The door first removes any x-lf-agent or x-lf-principal header the client sent, so your app can trust it on every route the door runs on. Use it to answer agents differently, such as with a lighter page. readAgentHeader(value) parses it.

When Little Friend can't be reached

The door reads two files with your edge key: the trust bundle (user agent rules, operator addresses and vetted signing keys) and your project's rules. It refreshes them in the background, your rules every minute and the trust bundle every 15 minutes, and answers from the copy it has meanwhile.

• Only the first requests on a new instance wait, for at most 300 ms. Call door.ready() at startup to load them early, or change the wait with bootTimeoutMs.
• Without both files, every request gets in, and its observation carries no decision.
• A file that can't be read is ignored, and the last good copy is kept.
• Rules this version of the SDK can't read are refused as a whole, so an older SDK never runs part of your policy.

For agents: sign your requests

A 401 from the door means the site asks agents to prove who they are with Web Bot Auth, the IETF standard for signed agent requests. Web Bot Auth working group

• Publish your Ed25519 public keys at https://your-domain/.well-known/http-message-signatures-directory, and name that origin in a Signature-Agent header.
• Sign each request (RFC 9421) over @authority, @method, @path and signature-agent, with keyid, created, expires and a fresh nonce. The door accepts a signature created in the last 5 minutes, allows 60 seconds of clock skew, and refuses a nonce it has already seen on another request.
• The door trusts keys Little Friend has fetched and vetted. A new signer reads as unverified until a later trust bundle carries its keys.
• Some sites ask for an agent acting for a signed-in user, for example at checkout. Sign those requests with a Visa Trusted Agent Protocol tag: agent-browser-auth or agent-payer-auth.
• A 429 carries Retry-After in seconds, so wait that long before trying again. A 403 names the site's rule that turned you away.

Privacy

• The door never stops a person. A browser is always let in, whatever the rules say.
• Who an agent acts for stays on your site. The door sends Little Friend only its decision: the action, the rule, the mode and the policy version.
• Decisions happen in your runtime. Requests are never sent to Little Friend to be decided. The door only downloads its two files.
• No visitor keys or fingerprints. The client address and user agent are used in memory for the decision. Rate limits are kept per rule and agent, never per visitor.

Limits

• AI citations aren't tracked. GPTBot reading a page doesn't mean an AI answer used it.
• Unknown stays unknown. We never infer identity from an IP address and device traits.
• Automated requests have their own count. Views come only from pages a script or SDK reports, so a crawler's requests never add to them.
• Agents that pass as browsers get in. An agent that sends a browser's user agent from ordinary addresses looks like a person, and the door never stops a person.
• The door needs code that runs per request. Static hosting alone has none, so put a Worker or a function in front. A Vercel log drain sees requests after they're served.