| Server IP : 138.197.107.151 / Your IP : 216.73.217.10 Web Server : Apache/2.4.58 (Ubuntu) System : Linux BloxBy-Builder 6.8.0-71-generic #71-Ubuntu SMP PREEMPT_DYNAMIC Tue Jul 22 16:52:38 UTC 2025 x86_64 User : wpbetasites_mrakzqskir ( 1022) PHP Version : 8.3.6 Disable Function : NONE MySQL : OFF | cURL : ON | WGET : ON | Perl : ON | Python : OFF | Sudo : ON | Pkexec : OFF Directory : /var/www/bsd-crawler-parser/__pycache__/ |
Upload File : |
�
= ^j�* � �� � d Z ddlZddlZddlZddlZddlZddlmZ ddlmZm Z ddl
Z
ej j dd� Z
ej j dd� Zej j d d
� ZdZdZ eej j d
d� � Z ej( d� Zd� Zd� Z G d� de� Z G d� de� Zd� Zd� Zd� Zd� Zd� Zd� Z d� Z!e"fd�Z#d� Z$y)a% AWS Lambda: parse WP Engine nginx logs for AI crawler traffic.
Reads s3://bsd-wpe-logs-private/logs/nginx/*.apachestyle.log.gz
Writes s3://bsd-wpe-logs-parsed/events/date=YYYY-MM-DD/site=<site>/<stem>.jsonl
s3://bsd-wpe-logs-parsed/summary/date=YYYY-MM-DD.json
s3://bsd-wpe-logs-parsed/manifest/processed.json
The date partition comes from each log LINE's timestamp, not the filename:
WP Engine rotates at 00:17, so 20260709-0017-foo.log.gz mostly holds Jul 8
traffic. One source file can therefore land in two date partitions.
� N)�defaultdict)�datetime�timezone�
SOURCE_BUCKETzbsd-wpe-logs-private�DEST_BUCKETzbsd-wpe-logs-parsed�
SOURCE_PREFIXzlogs/nginx/zmanifest/processed.jsonz.apachestyle.log.gz�MAX_FILES_PER_RUN�200z
[^a-z0-9._-]+c �� � | j dd� d }|j t � r|dt t � }t j d|j
� � S )zDlogs/nginx/20260709-0017-foo.apachestyle.log.gz -> 20260709-0017-foo�/� ���N�_)�rsplit�endswith�
SOURCE_SUFFIX�len�
_SAFE_SITE_RE�sub�lower)�key�names �&/var/www/bsd-crawler-parser/handler.py�stem_forr '