| Server IP : 138.197.107.151 / Your IP : 216.73.217.7 Web Server : Apache/2.4.58 (Ubuntu) System : Linux BloxBy-Builder 6.8.0-71-generic #71-Ubuntu SMP PREEMPT_DYNAMIC Tue Jul 22 16:52:38 UTC 2025 x86_64 User : wpbetasites_mrakzqskir ( 1022) PHP Version : 8.3.6 Disable Function : NONE MySQL : OFF | cURL : ON | WGET : ON | Perl : ON | Python : OFF | Sudo : ON | Pkexec : OFF Directory : /proc/698955/cwd/__pycache__/ |
Upload File : |
�
K ^j�F � �d � d Z ddlZddlZddlZddlZddlZddlZddlmZ ddl m
Z
mZ ddlZddl
mZ ddlmZ ddlmZ ddlmZ dd lmZmZ ej0 j3 d
d� Zej0 j3 dd
� ZdZdZdZ ed� \ Z Z!Z"Z#Z$Z%dZ& ejN � Z( ejR � Z*i Z+i Z, ejZ � Z.da/ ejN � Z0d� Z1d� Z2d� Z3d� Z4d*d�Z5d� Z6d� Z7d� Z8d� Z9d� Z:d� Z;d� Z<d� Z=d � Z>d!� Z?d*d"�Z@d#� ZAd$� ZBd%� ZCd&� ZD d+d'�ZEd,d(�ZFd)� ZGy)-a# Read-only access to the parsed bucket, plus the caching that keeps it quick.
Everything here is GetObject/ListObjectsV2 against bsd-wpe-logs-parsed. Nothing
in this app writes to S3, creates AWS resources, or touches bucket settings.
The shape we read:
summary/date=YYYY-MM-DD.json small, one per day
events/date=YYYY-MM-DD/site=<domain>/<f>.jsonl one JSON object per line
HOW THIS STAYS FAST
The overview only ever reads the summary files (9 small objects), so it is
effectively instant.
The detail table works out which shards can possibly match the filters and
fetches only those. A date narrows it to that day's ~110 shards; a site narrows
it to one. Shards are fetched in parallel and cached in memory, so paging
through results costs nothing after the first load. The worst case -- every day,
no site filter -- is ~160MB and takes a while the first time, then is cached.
Events are cached as plain tuples rather than dicts, and the repeated strings
are interned, which keeps the whole dataset in memory at a sane size.
� N)�ThreadPoolExecutor)�date� timedelta)�Config)�ClientError)�category_for)�
is_content)�install_from_source_key�install_from_stem�
PARSED_BUCKETzbsd-wpe-logs-parsed�
AWS_REGIONz us-east-1zmanifest/processed.json� i� � i, c � � t �,t 5 t �t t d�� a ddd� t S t S # 1 sw Y t S xY w)zAThe process-wide fetch pool, built on first use (after any fork).N�s3fetch)�max_workers�thread_name_prefix)�_pool�
_pool_lockr �_WORKERS� � �,/var/www/bsd-crawler-parser/viewer/s3data.py� _executorr Y sF � �
�}�
� I��}�*�x�>G�I�� I� �L�5�L� I� �L�s �9�Ac � � t j � }t 5 t j | � }|r|d |kD r|d cddd� S ddd� |� }t 5 ||z |ft | <