# wpe-log-parser

Finds AI crawler traffic in WP Engine nginx logs and writes it to S3 as JSON.

Reads `s3://bsd-wpe-logs-private/logs/nginx/*.apachestyle.log.gz`, keeps only
lines whose User-Agent matches a known AI crawler, and writes them to
`s3://bsd-wpe-logs-parsed`. Runs nightly at 09:00 UTC via EventBridge.

## Where the logs come from

This parser is the last stage of a pipeline:

```
WP Engine  ->  s3://bsd-wpe-logs          (public - deliberately, see below)
           ->  wpe-log-sweep Lambda
                 - S3 ObjectCreated event  -> moves in ~2s   [the fast path]
                 - rate(15 minutes)        -> backstop sweep [safety net]
           ->  s3://bsd-wpe-logs-private
           ->  wpe-log-parser Lambda, nightly 09:00 UTC [this repo]
           ->  s3://bsd-wpe-logs-parsed
```

The sweep runs long before 09:00, so the nightly parse always sees a complete
previous day.

## bsd-wpe-logs is public, by decision

**The `bsd-wpe-logs` bucket grants `s3:GetObject` to `Principal: "*"` -- anyone
on the internet.** `get-bucket-policy-status` reports `IsPublic: true`. All
four bucket-level public access blocks are off; there is no account-level
block.

**This is intentional and must stay.** WP Engine support confirmed their log
delivery requires the bucket to be public. Do **not** remove the
`PublicReadObjectsOnly` statement or enable Block Public Access -- it breaks
log ingestion. `fix-public-bucket.sh` in this repo implements that removal and
is **deliberately not run**; it is kept only as a rollback artifact and a
record of what was considered.

The files transiting this bucket contain visitor IP addresses and full browsed
URLs for law-firm sites (buckfirelaw.com, protectingpatientrights.com,
leifertlaw.com, and 350-odd others -- 355 installs deliver here as of
2026-07-17). IP + browsed-URL on a legal or medical site is PII with GDPR/CCPA
weight. Since the bucket must stay open, the mitigation is to keep files in it
for as short a time as possible -- hence the S3 event trigger below.

Limits on the exposure, for the record:

- Anonymous `ListBucket` is **not** granted, so the bucket cannot be
  enumerated over S3. But keys are predictable --
  `logs/nginx/YYYYMMDD-HHMM-<sitename>.apachestyle.log.gz` and timestamps
  cluster at 0014-0017. `<sitename>` is the WP Engine **install** name, not the
  domain: it is capped at 14 characters and hand-chosen, so it is not simply
  derivable (`amdmedicalpllc` serves `amdmedicalgroup.com`). Do not read that as
  protection. Measured against the real set, 59% of installs are the domain's
  first label or a prefix of it, so guessing still works for most of the estate
  where enumeration does not.
- The exposure window is now ~2 seconds rather than ~15 minutes.

One loose end worth revisiting with WP Engine: the bucket policy also grants
`arn:aws:iam::902500896138:user/wpengine-remote-logs` an explicit
`ListBucket`/`GetObject`/`PutObject`. That named principal appears sufficient
for delivery on its own, which does not obviously square with needing public
read. Worth a specific follow-up question to support -- ideally "does delivery
use the IAM principal or anonymous access?" -- rather than a general one. Until
then the bucket stays public.

## Never put a lifecycle expiry on bsd-wpe-logs

**WP Engine keeps `.largefs` validation files at the bucket root. They must
never be moved or deleted -- WP Engine needs them there to keep delivering
logs, and losing them silently stops ingestion.**

The sweep already protects them: `_skip()` returns True for any key ending in
`.largefs` at any depth, and the backstop only lists `logs/` so it never even
sees the root. That is covered by tests in `sweep/test_sweep.py`.

**A lifecycle rule bypasses all of that.** On 2026-07-16 the bucket had an
out-of-band rule (`public-backstop-expire`, `Expiration: Days: 1`, `Filter:
{}` -- i.e. *every object in the bucket*) which deleted the `.largefs` files.
Lifecycle expiration is executed by S3 itself: it does not consult the sweep's
skip rules, and it cannot be blocked by IAM or a bucket policy Deny. Because
`.largefs` files are exactly the objects the sweep deliberately never moves,
they were the only things that lived long enough to hit the 1-day expiry --
the rule's sole routine effect was destroying them. Versioning is **off** on
this bucket, so those deletes left no delete markers and are unrecoverable.

The rule was deleted on 2026-07-16; `bsd-wpe-logs-lifecycle.backup.json` is
the snapshot. **Do not re-add it.** S3 lifecycle filters match on prefix, tag,
and object size -- there is *no suffix filter*, so "expire everything except
`*.largefs`" is not expressible. Any bucket-wide expiry here will eventually
eat the `.largefs` files again. The exposure window is already bounded by the
sweep (~2 s typical, <=15 min worst case), which is far tighter than a 1-day
expiry could ever be -- and the sweep *moves* files, where lifecycle only
destroys them.

If the `.largefs` files are missing, they must be restored by WP Engine; they
are not in either of our buckets and there is no version history to recover.

## WP Engine delivers to BOTH `logs/` and `wpe_logs/`

Up to 2026-07-16 delivery landed under `logs/`. When the `.largefs` files were
restored on 2026-07-17, WP Engine resumed under **`wpe_logs/`** instead. The
sweep watched only `logs/`, so it moved nothing -- and logged `moved=0`, which
this README documented as *the healthy steady state*, every 15 minutes while
**~331MB of visitor logs sat world-readable in the public bucket** for 45
minutes. An anonymous `curl` returned them, no credentials.

Both prefixes are swept now, and `dest_key()` normalises `wpe_logs/...` to
`logs/...` on the way into the private bucket. That keeps the delivery prefix a
concern of the sweep alone: the parser's `SOURCE_PREFIX`, its IAM policy (which
only grants read on `logs/nginx/*`) and the viewer all still see `logs/`.

**The lesson is the monitoring, not the prefix.** `moved=0` only ever meant
"nothing to move *where I looked*". `full_sweep()` now also calls
`unswept_prefixes()` and logs a WARNING for any top-level prefix no `PREFIXES`
entry covers -- that is the alarm that would have caught this on the first run
instead of never. Objects at the root are not reported, since `.largefs` lives
there deliberately.

It currently warns about `/` (ten 51-byte `*_upload_test.txt` probes WP Engine
writes, content: "WP Engine upload test. This file is safe to delete."). They
carry no PII. Left in place rather than allowlisted, because an alarm you have
taught yourself to ignore is how this happened in the first place.

If delivery ever moves again: add the prefix to `PREFIXES` in
`sweep/lambda_function.py`, and add a matching `ObjectCreated` notification on
the bucket (the two prefixes must not overlap or S3 rejects the config).

## Exposure window: ~15 min -> ~2 s

The sweep is triggered two ways, both handled by `sweep/lambda_function.py`:

| Trigger | Role | Latency |
|---|---|---|
| S3 `ObjectCreated:*` under `logs/` | fast path, does ~all the work | ~2 s |
| `wpe-log-sweep-15min` EventBridge rule | **backstop** for dropped events | <=15 min |

**The backstop is load-bearing.** S3 -> Lambda is an *asynchronous* invoke:
Lambda retries a failing event twice and then **silently discards** it, and
this function has no DLQ. Without the scheduled sweep, one dropped event
strands a log file in a public bucket forever -- unbounded exposure, strictly
worse than the 15-minute window it replaced. Keeping it bounds the worst case
at no-worse-than-before while the typical case drops to seconds. It costs
~800ms of a 256MB Lambda every 15 minutes.

Healthy steady state: `event: moved=1` lines at ~00:17 UTC, and scheduled runs
logging `sweep: moved=0`. **If the scheduled sweep is consistently reporting
`moved>0`, the event trigger is not firing** -- investigate rather than shrug,
because that means you are silently back to 15-minute exposure.

Reserved concurrency is set to **50**. WP Engine drops ~120 files at once and
this account's *total* concurrency limit is **400** (not the usual 1000), so an
unbounded burst would consume 30% of the account budget and contend with
`wpe-log-parser`. Throttled S3 events are retried automatically by Lambda's
async path, so bounding it loses nothing.

Verified end-to-end on 2026-07-16: a test object PUT at 11:23:24 UTC was in
the private bucket by 11:23:26 (including a 605ms cold start).

## Files

| File | What it does |
|---|---|
| `crawlers.py` | **The crawler list.** Edit this to add/remove crawlers. |
| `logparser.py` | Parses one log line into an event dict. |
| `handler.py` | Lambda entry point: S3 listing, reading, writing, manifest. |
| `test_parser.py` | Unit tests. `python3 test_parser.py` |
| `test_local.py` | Dry run: reads real S3 logs, writes to a local folder. |
| `deploy.sh` | Creates/updates all AWS resources. Idempotent. |
| `iam-policy.json` | Least-privilege S3 policy for the Lambda role. |
| `sweep/lambda_function.py` | The `wpe-log-sweep` function: S3-event fast path + backstop sweep. |
| `sweep/lambda_function.original.py` | The pre-2026-07-16 sweep code, for rollback. |
| `sweep/test_sweep.py` | Sweep tests against a fake S3. `python3 test_sweep.py` |
| `sweep/deploy-sweep.sh` | Wires up the S3 trigger. Dry-run by default; `--rollback` undoes it. |
| `fix-public-bucket.sh` | **Do not run.** Removes public access; would break WP Engine delivery. Kept as a record. |
| `bsd-wpe-logs-policy.backup.json` | The live (public) policy, as of 2026-07-16. |
| `bsd-wpe-logs-lifecycle.backup.json` | The deleted `public-backstop-expire` rule. **Do not restore** — it deleted the `.largefs` files. |
| `viewer/run.sh` | **Start the dev server.** `cd viewer && ./run.sh` (local only) |
| `viewer/gunicorn.conf.py` | Production server config. `gunicorn -c gunicorn.conf.py app:app` |
| `viewer/app.py` | The viewer: login, overview, detail table. |
| `viewer/categories.py` | **The bot → group mapping**, incl. the `ai_search` split. |
| `viewer/sites.py` | **Site identity**: WP Engine install ↔ domain mapping. |
| `viewer/s3data.py` | Read-only S3 access + the caching and shared fetch pool. |
| `viewer/test_viewer.py` | Viewer tests, no AWS needed. `python3 test_viewer.py` |

## The viewer

A small Flask app for browsing the parsed data. It never writes to S3, creates
no AWS resources, and changes no bucket settings. Run it locally with the Flask
dev server, or in production behind gunicorn -- see [Deploying the
viewer](#deploying-the-viewer-digitalocean) for the latter.

```bash
cd viewer && ./run.sh     # dev server on http://127.0.0.1:8765 (local only)
```

`run.sh` binds to `127.0.0.1`, so nothing outside your machine can reach the dev
server. First run installs dependencies and copies `.env.example` to `.env`; set
`VIEWER_USER` / `VIEWER_PASSWORD` there. `.env` is gitignored. Port 8765 is
used because macOS AirPlay already squats on 5000.

The events contain visitor IPs and browsed URLs, so every page sits behind a
login. S3 is read server-side with your AWS credentials; the browser never
talks to AWS.

**Overview** reads only the daily summaries. **Detail** is a filterable,
paginated table that fetches just the shards matching your filters -- a date
narrows it to that day, a site to that install's shards (usually one file, more
if it serves several vhosts) -- in parallel, and caches them. **Export CSV**
downloads every row matching the current filters, not just the page you are
looking at.

The date filter has presets (Latest Day, This Week, This Month, Last Month, Last
90 Days, Custom). They anchor on **the newest day with data, not the wall
clock** -- the parser runs nightly, so a preset anchored on today would show an
empty page every morning. A small "Data current through YYYY-MM-DD" line under
the filter says which day that is, so a range ending days before today does not
look like a bug.

It lands on **Latest Day** (a single day) by default, but the range no longer
governs how heavy the landing page is. **The overview reads only the daily
summaries** — nine to fifteen small files, one per day — for *every* preset,
content filter and all. It used to fall back to scanning every event shard when
"Content pages only" was on (a month was ~6,600 shards and minutes of hang, and
the browser gave up before it finished); now the parser precomputes the
content-only counts into each summary (`content_hits` / `by_platform_content`),
so the widest range is still just a handful of files. Latest Day stays the
default as a sensible first scope; the wider ranges are one dropdown away and now
open just as fast. On startup a background thread pre-loads the default day.

The **detail table** (individual log rows) is the one page that still reads the
event shards, because it shows per-request URLs the summaries do not carry.
That is cheap when scoped by site or bot and heavy for a wide unfiltered range —
so three things keep it quick and keep it from taking the server down:

- **The reads run wide.** Fetching shards is I/O-bound (each thread just waits on
  an S3 GET), so the fetch pool runs `VIEWER_FETCH_WORKERS` readers at once,
  default **48**. On the latest day that is the difference between ~4.9s and
  ~2.1s; past ~48 it is S3 latency, not concurrency, that sets the floor. Dial it
  down with the env var if a small box feels the load.
- **One shared fetch pool, not one per request.** All shard reads across all
  requests go through that single bounded pool (`_executor` in `s3data.py`).
  Concurrent "View hits" clicks used to each spawn their own pool against a
  connection pool sized for it -- a handful of clicks oversubscribed it and the
  whole server stalled. Now that worker count is the *total* S3 concurrency no
  matter how many requests are in flight; the excess just queues.
- **The shard cache is sized above a week's working set** (`_MAX_CACHED_SHARDS`),
  so two overlapping scans do not evict each other's shards and re-fetch in a
  loop, and paging within a range you have already opened is instant.

### "Content pages only" (default ON)

A crawler fetches three different things and only one is your writing: the page
itself, the furniture it needs (assets, APIs), and — if it is not really a
crawler — whatever it can find. Counting all three overstates how much of your
*content* is being read. The toggle hides the latter two; untick it to see
every request. **Nothing is filtered out of S3** — this only changes what the
viewer counts. The patterns live in `content.py` at the repo root: the parser
imports it to precompute each summary's content counts, and the viewer imports
it (via the sys.path insert in `viewer/categories.py`) to classify the detail
rows. One file, so the two can never disagree — it ships in the Lambda zip.

Matching is on **path segments, not substrings**, which matters more than it
sounds: `"secret" in url` also matches a real article at
`/blog/the-secret-to-winning`, and `".env" in url` matches
`/uploads/flyer.envelope.pdf`. Substring matching would have quietly deleted
real traffic from the numbers.

### One WP Engine install is one site

The "Sites" box reads **237 / 355**: sites an AI crawler hit in the range, out
of every site we are capturing logs for. The table lists all 355 — a site with a
log file and no AI traffic is a row with `0`, sorted after the ones with hits.

The total comes from the **log filenames**, via the parser's
`manifest/processed.json` (its keys *are* the source filenames). Filenames are
the only source that can see a site with no traffic: an empty log produces no
events, and therefore no vhost to read. The viewer reads the manifest out of the
parsed bucket rather than listing `bsd-wpe-logs-private` itself — the raw logs
are IP + browsed URL, and the viewer has no business being able to read them.

**The filename is not the domain.** The filename carries the WP Engine *install*
name; the events inside carry the *vhost* from each request line. They differ,
and one install serves several vhosts:

    buckfirelaw     -> buckfirelaw.com + 8 others, and buckfirelaw.wpengine.com
    allmandlawfirm  -> allmandlaw.com,             and allmandlawfirm.wpengine.com

So the site identity is the **install**, not the vhost. Keying on the vhost lists
one site twice — once under its real domain, once under the WP Engine holding
domain — and can never name a site whose log is empty.

Nothing guesses the mapping. The install name is truncated to 14 characters and
chosen by hand, so it drifts from the domain entirely (`amdmedicalpllc` ->
`amdmedicalgroup.com`, `bermanandrusso` -> `brslaw.com`); matching the two by
string similarity gets **41% of installs wrong**. Instead the pairing is read
straight out of the event key, which already contains both halves:

    events/date=2026-07-08/site=amdmedicalgroup.com/20260709-0011-amdmedicalpllc.jsonl
                                \_ vhost                          \_ install

A row is labelled with the install's real domain (the holding domain only if
that is all we have seen), and `+N` means the install serves N other real
domains whose hits are included in that number. See `viewer/sites.py`.

**What the total does and does not mean.** It counts sites whose logs reach us,
not every production site BSD runs. A site that has never had WP Engine log
export switched on has no file, no manifest entry, and cannot appear here or in
the total — it is invisible to this tool rather than reported as zero. A real
site inventory would have to come from a master list fed in separately; that is
deliberately not solved here.

### The viewer regroups `search`, deliberately

`crawlers.py` files every indexing bot under `search`, which buries OpenAI's
and Perplexity's crawlers alongside Bingbot. The viewer splits that in two:

| Group | Meaning | Default |
|---|---|---|
| `ai_training` | collects a corpus to train on | shown |
| `ai_assistant` | fetches a page live for a user's question | shown |
| `ai_search` | **an AI company building an index** (OAI-SearchBot, PerplexityBot, YouBot, Claude-SearchBot) | shown |
| `search` | a conventional search engine (Bingbot, Amazonbot, Applebot) | **hidden** |

So the default view is all AI activity (175,418 hits over 2026-07-08..16, 50.3%)
and hides only conventional search engines (173,287, 49.7%). A checkbox
includes them.

**The viewer derives the group from `ai_platform`, ignoring the `category`
stored in the event.** Every event already in S3 predates this split and says
`"category": "search"` for OAI-SearchBot. Deriving on read applies the grouping
to all existing data with no reprocessing. `viewer/categories.py` imports
`AI_CRAWLERS` from `crawlers.py`, so a crawler added there flows through
automatically; only the `ai_search` overrides live in the viewer.

Two things that surprise people, both confirmed on 2026-07-16:

- **Googlebot is not in this data at all.** The parser only matches AI
  crawlers, and Googlebot is not one, so ordinary Google search crawling never
  appears. `GoogleOther` and `Google-Extended` are different bots and are both
  `ai_training`. Same for DuckDuckGo: `DuckAssistBot` (their AI assistant) is
  tracked, their index crawler is not.
- **`Applebot` and `Applebot-Extended` are different bots.** The first is
  Siri/Spotlight indexing (`search`, hidden); the second is Apple's AI training
  crawler (`ai_training`, shown).

A follow-up, not done: `crawlers.py` still emits the old three categories, so
newly parsed data keeps the old labels. The viewer doesn't care. Making the raw
data match would mean editing `crawlers.py` and reprocessing the whole bucket.

## Deploying the viewer (DigitalOcean)

`run.sh` is the Flask dev server -- fine for your laptop, not for a public box.
In production the viewer runs behind **gunicorn**, with **nginx** terminating
TLS in front of it. The pages expose visitor IPs and browsed URLs, so it must
never be served over plain HTTP or with the app port open to the internet.

> **Run gunicorn on Linux, not macOS.** Gunicorn forks its workers, and
> fork + boto3 is unsafe on macOS -- the worker crashes (you get "Python quit
> unexpectedly" dialogs). The steps below are for the Linux server. For local
> work on a Mac, use `./run.sh`, which threads instead of forking. The config
> has a darwin-only guard that makes `gunicorn` *survivable* on a Mac if you
> must, but it is best-effort; `run.sh` is the supported local path.

**Why one gunicorn worker.** The app keeps a large in-memory shard cache and a
single bounded fetch pool. A second worker process would hold a second, cold
copy of the cache and split traffic across them. So `gunicorn.conf.py` pins
`workers = 1` and gets concurrency from threads (`gthread`) instead -- correct
here because the work is I/O-bound on S3. Don't raise `workers` without first
moving the cache out of process; see the note at the top of `gunicorn.conf.py`.

**1. AWS credentials (read-only, parsed bucket only).** The viewer only reads
`bsd-wpe-logs-parsed`; it needs neither the source bucket nor any write. Create
an IAM user with exactly this policy and nothing more:

```json
{
  "Version": "2012-10-17",
  "Statement": [
    { "Effect": "Allow", "Action": ["s3:GetObject"],
      "Resource": "arn:aws:s3:::bsd-wpe-logs-parsed/*" },
    { "Effect": "Allow", "Action": ["s3:ListBucket"],
      "Resource": "arn:aws:s3:::bsd-wpe-logs-parsed" }
  ]
}
```

Put its keys where boto finds them -- `~/.aws/credentials` for the service user,
or `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` in the `.env` below (the app
loads `.env` itself, so either works).

**2. Code and dependencies.**

```bash
sudo adduser --system --group viewer
sudo git clone <repo> /opt/bsd-crawler-parser
cd /opt/bsd-crawler-parser
python3 -m venv .venv
.venv/bin/pip install -r viewer/requirements.txt   # includes gunicorn
```

**3. Config.** Create `viewer/.env` (gitignored) and lock it down:

```bash
VIEWER_USER=<pick one>
VIEWER_PASSWORD=<a long random string>
# REQUIRED in production: without it every restart invalidates all sessions
# (a fresh random key is generated per start). Generate once and keep it.
SECRET_KEY=<python3 -c 'import secrets; print(secrets.token_hex(32))'>
PARSED_BUCKET=bsd-wpe-logs-parsed
AWS_REGION=us-east-1
# AWS_ACCESS_KEY_ID=...        # if not using ~/.aws/credentials
# AWS_SECRET_ACCESS_KEY=...
```

```bash
sudo chown viewer:viewer viewer/.env && sudo chmod 600 viewer/.env
```

**4. systemd unit** at `/etc/systemd/system/crawler-viewer.service`:

```ini
[Unit]
Description=AI crawler viewer
After=network-online.target
Wants=network-online.target

[Service]
User=viewer
WorkingDirectory=/opt/bsd-crawler-parser/viewer
ExecStart=/opt/bsd-crawler-parser/.venv/bin/gunicorn -c gunicorn.conf.py app:app
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target
```

```bash
sudo systemctl enable --now crawler-viewer
sudo systemctl status crawler-viewer      # expect one worker, "cached ... ready"
```

gunicorn binds `127.0.0.1:8765` by default (override with `BIND`), so it is not
reachable from outside the box until nginx proxies it.

**5. nginx + TLS.** Reverse-proxy 443 to gunicorn on loopback:

```nginx
server {
    listen 80;
    server_name crawlers.example.com;
    location / {
        proxy_pass http://127.0.0.1:8765;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        # Must be >= gunicorn's timeout (300s) or a cold wide-range scan 504s.
        proxy_read_timeout 300s;
    }
}
```

```bash
sudo certbot --nginx -d crawlers.example.com   # adds the 443 block + HTTP->HTTPS redirect
sudo ufw allow 'Nginx Full' && sudo ufw allow OpenSSH && sudo ufw enable
```

Do **not** open port 8765 in the firewall -- only nginx (80/443) faces the
internet; gunicorn stays on loopback. gunicorn already trusts `X-Forwarded-*`
only from `127.0.0.1` (`forwarded_allow_ips` in the config), so the login and
logs see the real client IP via nginx, not `127.0.0.1`.

**Tunables** (env vars, all optional): `BIND` (default `127.0.0.1:8765`),
`WEB_THREADS` (default 8), `WEB_TIMEOUT` seconds (default 300). Raise
`WEB_THREADS` if many people browse at once; the S3 fan-out stays bounded by the
shared pool regardless.

## Output layout

```
events/date=2026-07-08/site=buckfirelaw.com/20260709-0017-buckfirelaw.jsonl
summary/date=2026-07-08.json
manifest/processed.json
```

One JSONL shard per (date, site, source file). One object per hit:

```json
{
  "site": "buckfirelaw.com",
  "ai_platform": "GPTBot",
  "category": "ai_training",
  "user_agent": "Mozilla/5.0 ... GPTBot/1.4; +https://openai.com/gptbot)",
  "url": "/case-types/dog-bite/",
  "status_code": 200,
  "bytes": 7354,
  "timestamp": "2026-07-08T00:17:29Z",
  "source_file": "logs/nginx/20260709-0017-buckfirelaw.apachestyle.log.gz"
}
```

The daily summary has `totals`, `by_site` (with a per-site platform breakdown),
and `by_platform`. `totals`, `by_site` and `by_platform` each carry a
**content-only** count alongside the all-hits one — `content_hits`, plus
`by_platform_content` on each site — so the viewer's "Content pages only"
overview reads these rather than re-scanning events. A summary written before
those fields existed simply lacks them, and the viewer falls back to a live scan
for that day until it is rebuilt.

The summary deliberately does **not** carry a per-URL breakdown. It once had a
`by_url` map, but nothing read it (the detail table gets URLs from the event
shards) and it was ~99% of the file — a high-traffic day's summary was ~8 MB,
which made the overview slow to load once it started reading summaries for every
range. Dropping it took each summary back down to ~0.1 MB.

## A user-agent is a claim, not an identity — we do not verify it

**Everything here attributes traffic on the strength of a User-Agent string the
client chose to send.** Anyone can send any string, and scanners routinely
borrow a reputable bot's name to look boring in a log file. `crawlers.py`
substring-matches that header and takes it at face value, so **every number in
this pipeline is "traffic claiming to be X", not "traffic from X"**.

Observed 2026-07-17: requests labelled `OAI-SearchBot` fetching `/secrets.yml`
and `/.ssh/id_ed25519`. OpenAI's crawler indexes pages for ChatGPT; it does not
go looking for private keys. That is a scanner wearing OpenAI's name, and we
counted it as `ai_search`.

The real thing is verifiable, and the major operators publish the means:

| Operator | How to verify |
|---|---|
| OpenAI (GPTBot, OAI-SearchBot, ChatGPT-User) | published IP ranges (`openai.com/*-ranges.txt`) |
| Anthropic (ClaudeBot) | published IP ranges |
| Google (Googlebot, Google-Extended) | reverse DNS -> `*.googlebot.com` / `*.google.com`, then forward-confirm |
| Bing (Bingbot) | reverse DNS -> `*.search.msn.com`, then forward-confirm |
| Apple (Applebot) | reverse DNS -> `*.applebot.apple.com`, then forward-confirm |

The standard check is a **forward-confirmed reverse DNS lookup**: reverse the
client IP, confirm the hostname belongs to the operator, then resolve that
hostname forward and confirm it returns the original IP. Reverse DNS alone is
forgeable by whoever controls the PTR record.

**Not implemented, deliberately.** The parser reads the log line and never sees
the client IP as a verified fact, and doing this properly means an IP-range
feed or DNS lookups per unique IP. The viewer's "Content pages only" filter
hides the most obvious impostor traffic (`/.env`, `/.ssh/...`) from the counts,
but that is cosmetic: it drops the requests that *look* like probing, not the
ones that lie convincingly. **Treat per-bot numbers as an upper bound.** If
these figures ever back a business decision or an external claim, verify first.

## Two things that are easy to get wrong

**The date comes from the log line, not the filename.** WP Engine rotates logs
at 00:17, so `20260709-0017-foo.log.gz` mostly contains *July 8* traffic. A
single source file therefore writes into two date partitions. Partitioning on
the filename would mislabel ~99% of traffic by one day.

**`category` exists so you can filter out search engines.** Bingbot alone is
~25% of all matched hits and Amazonbot/Applebot add another ~25%. If you want
"AI traffic" in the sense of model training and assistants, filter to
`category != "search"`. The three categories are `ai_training`,
`ai_assistant`, and `search`.

## Re-runs never double-count

`manifest/processed.json` records each source file by ETag and skips it next
time. But the manifest is only a speed optimization, not a correctness crutch:
each source file writes to a shard named after itself, so reprocessing
*overwrites* rather than appends. If the manifest is ever lost or corrupted,
delete it and re-run — the output converges to identical numbers. Verified:
reprocessing 80 files with no manifest produced 54,658 hits both times.

Summaries are rebuilt from the event shards on disk rather than accumulated
in memory, so they are always consistent with the events, even on partial runs.

## Common tasks

```bash
# Add a crawler: edit AI_CRAWLERS in crawlers.py, then
python3 test_parser.py && ./deploy.sh

# Dry run against real logs, output to ./local_out (no AWS writes)
python3 test_local.py --files 20

# Run it now
aws lambda invoke --function-name wpe-log-parser --region us-east-1 /dev/stdout

# Pause / resume the nightly schedule
aws events disable-rule --name wpe-log-parser-nightly --region us-east-1
aws events enable-rule  --name wpe-log-parser-nightly --region us-east-1

# Logs
aws logs tail /aws/lambda/wpe-log-parser --region us-east-1 --follow
```

Reprocess everything from scratch (safe — it converges to the same numbers):

```bash
aws s3 rm s3://bsd-wpe-logs-parsed/manifest/processed.json --region us-east-1
aws lambda invoke --function-name wpe-log-parser --region us-east-1 /dev/stdout
```

## AWS resources

| Resource | Name |
|---|---|
| S3 bucket | `bsd-wpe-logs-parsed` (private, encrypted, versioned) |
| Lambda | `wpe-log-parser` (python3.12, arm64, 1024 MB, 900 s) |
| IAM role | `wpe-log-parser-role` |
| IAM policy | `wpe-log-parser-policy` |
| EventBridge rule | `wpe-log-parser-nightly` — `cron(0 9 * * ? *)` |

The role can read `logs/nginx/*` on the source bucket and read/write the parsed
bucket. It has **no delete permission anywhere** — verified with
`aws iam simulate-principal-policy`. It cannot touch `logs/error/`.

`MAX_FILES_PER_RUN` (default 200) caps files per invocation so a large backlog
can't hit the 15-minute timeout; leftovers are picked up on the next run.
A 200-file run takes ~2 min and 118 MB of the 1024 MB limit.