403Webshell
Server IP : 138.197.107.151  /  Your IP : 216.73.217.10
Web Server : Apache/2.4.58 (Ubuntu)
System : Linux BloxBy-Builder 6.8.0-71-generic #71-Ubuntu SMP PREEMPT_DYNAMIC Tue Jul 22 16:52:38 UTC 2025 x86_64
User : wpbetasites_mrakzqskir ( 1022)
PHP Version : 8.3.6
Disable Function : NONE
MySQL : OFF  |  cURL : ON  |  WGET : ON  |  Perl : ON  |  Python : OFF  |  Sudo : ON  |  Pkexec : OFF
Directory :  /var/www/bsd-crawler-parser/__pycache__/

Upload File :
current_dir [ Writeable ] document_root [ Writeable ]

 

Command :


[ Back ]     

Current File : /var/www/bsd-crawler-parser/__pycache__/handler.cpython-312.pyc
�

= ^j�*����dZddlZddlZddlZddlZddlZddlmZddlmZm	Z	ddl
Z
ejjdd�Z
ejjdd�Zejjd	d
�ZdZdZeejjd
d��Zej(d�Zd�Zd�ZGd�de�ZGd�de�Zd�Zd�Zd�Zd�Zd�Zd�Z d�Z!e"fd�Z#d�Z$y)a%AWS Lambda: parse WP Engine nginx logs for AI crawler traffic.

Reads   s3://bsd-wpe-logs-private/logs/nginx/*.apachestyle.log.gz
Writes  s3://bsd-wpe-logs-parsed/events/date=YYYY-MM-DD/site=<site>/<stem>.jsonl
        s3://bsd-wpe-logs-parsed/summary/date=YYYY-MM-DD.json
        s3://bsd-wpe-logs-parsed/manifest/processed.json

The date partition comes from each log LINE's timestamp, not the filename:
WP Engine rotates at 00:17, so 20260709-0017-foo.log.gz mostly holds Jul 8
traffic. One source file can therefore land in two date partitions.
�N)�defaultdict)�datetime�timezone�
SOURCE_BUCKETzbsd-wpe-logs-private�DEST_BUCKETzbsd-wpe-logs-parsed�
SOURCE_PREFIXzlogs/nginx/zmanifest/processed.jsonz.apachestyle.log.gz�MAX_FILES_PER_RUN�200z
[^a-z0-9._-]+c���|jdd�d}|jt�r|dtt�}tjd|j
��S)zDlogs/nginx/20260709-0017-foo.apachestyle.log.gz -> 20260709-0017-foo�/����N�_)�rsplit�endswith�
SOURCE_SUFFIX�len�
_SAFE_SITE_RE�sub�lower)�key�names  �&/var/www/bsd-crawler-parser/handler.py�stem_forr'sQ���:�:�c�1��b�!�D��}�}�]�#��)�s�=�)�)�*�����S�$�*�*�,�/�/�c�t�tjd|j��jd�}|xsdS)z+Make a vhost safe for use inside an S3 key.rz._-�unknown)rrr�strip)�site�cleaneds  r�	safe_siter!/s0������T�Z�Z�\�2�8�8��?�G���i�rc�$�eZdZd�Zd�Zd�Zd�Zy)�S3Sinkc� �||_||_y�N)�bucket�s3)�selfr&�clients   r�__init__zS3Sink.__init__;s�������rc�t�|jj|j||jd�d��y)N�utf-8zapplication/json)�Bucket�Key�Body�ContentType)r'�
put_objectr&�encode)r(r�bodys   r�putz
S3Sink.put?s1��������;�;�C����W�%�*�	�	
rc���	|jj|j|��}|dj�j	d�S#|jj
j$rYywxYw)N�r-r.r/r,)r'�
get_objectr&�read�decode�
exceptions�	NoSuchKey)r(r�resps   r�getz
S3Sink.getFsd��	��7�7�%�%�T�[�[�c�%�B�D���<�$�$�&�-�-�g�6�6���w�w�!�!�+�+�	��	�s�AA�#A1�0A1c���g}|jjd�}|j|j|��D]-}|j	dg�D]}|j|d���/|S)N�list_objects_v2�r-�Prefix�Contentsr.)r'�
get_paginator�paginater&r=�append)r(�prefix�keys�	paginator�page�objs      r�	list_keyszS3Sink.list_keysMsp�����G�G�)�)�*;�<�	��&�&�d�k�k�&�&�I�	(�D��x�x�
�B�/�
(�����C��J�'�
(�	(��rN)�__name__�
__module__�__qualname__r*r4r=rK�rrr#r#:s���
��rr#c�.�eZdZdZd�Zd�Zd�Zd�Zd�Zy)�	LocalSinkzGMirrors S3Sink onto the filesystem so we can test without touching AWS.c��||_yr%)�root)r(rSs  rr*zLocalSink.__init__Ys	����	rc�V�tjj|j|�Sr%)�os�path�joinrS)r(rs  r�_pathzLocalSink._path\s���w�w�|�|�D�I�I�s�+�+rc���|j|�}tjtjj	|�d��t|d�5}|j
|�ddd�y#1swYyxYw)NT)�exist_ok�w)rXrU�makedirsrV�dirname�open�write)r(rr3rV�fhs     rr4z
LocalSink.put_sU���z�z�#���
���B�G�G�O�O�D�)�D�9�
�$��_�	���H�H�T�N�	�	�	�s�A-�-A6c���|j|�}tjj|�syt	|�5}|j�cddd�S#1swYyxYwr%)rXrUrV�existsr^r8)r(rrVr`s    rr=z
LocalSink.getesI���z�z�#����w�w�~�~�d�#��
�$�Z�	�2��7�7�9�	�	�	�s�A�A c�(�|j|�}g}tj|�D]f\}}}|D][}tjj	||�}|jtjj
||j���]�h|Sr%)rXrU�walkrVrWrE�relpathrS)	r(rF�baserG�dirpath�_dirs�filesr�fulls	         rrKzLocalSink.list_keysls|���z�z�&�!����%'�W�W�T�]�	>�!�G�U�E��
>���w�w�|�|�G�T�2�����B�G�G�O�O�D�$�)�)�<�=�
>�	>��rN)	rLrMrN�__doc__r*rXr4r=rKrOrrrQrQVs��Q��,���rrQc�8�ddlm}tt�}dddd�}|D][}|j	�s�|dxxdz
cc<	|||�}|��.|dxxdz
cc<|d	dd
}|||dfj
|��]||fS#t
$r|dxxdz
cc<Y�zwxYw)zBGroup AI-crawler events by (date, site). Returns (grouped, stats).r)�
parse_line)�lines�hits�unparsedrnr
rpNro�	timestamp�
r)�	logparserrmr�listr�	ExceptionrE)rn�
source_keyrm�grouped�stats�line�event�dates        r�
process_linesr|{s���$��$��G����2�E��
5���z�z�|��
�g��!���	��t�Z�0�E��=��
�f�
���
��[�!�#�2�&����u�V�}�%�&�-�-�e�4�
5��E�>����	��*���"���	�s�	B�B�Bc��t|�}t�}|j�D][\\}}}d|�dt|��d|�d�}dj	d�|D��dz}	|j||	�|j
|��]|S)zBWrite one JSONL shard per (date, site). Returns the dates touched.zevents/date=z/site=r�.jsonl�
c3�JK�|]}tj|d�����y�w)T)�	sort_keysN)�json�dumps)�.0�es  r�	<genexpr>zwrite_events.<locals>.<genexpr>�s����G�1����A��6�6�G�s�!#)r�set�itemsr!rWr4�add)
�sinkrwrv�stem�datesr{r�eventsrr3s
          r�write_eventsr��sz���J��D��E�E� '�
�
������t�f�37��4��$�O���y�y�G��G�G�$�N������d��
�	�	�$��	�
�Lrc���d|z}td��}td��}td��}d}d}t|j|��D�]}|jd�s�|j	|�}	|	s�*|	j�D]�}
|
j
�s�tj|
�}|d}|d}
|j	d	d�}|d
z
}||z
}||dxxd
z
cc<||d	xx|z
cc<||d|
xxd
z
cc<||
dxxd
z
cc<||
d	xx|z
cc<|j	d
�||
d
<||
dj|�|||dxxd
z
cc<����!|tjtj�jd�||t|�t|�d�t|j!�d���D��cic]8\}}||d|d	t#t|dj!�d����d���:c}}t|j!�d���D��cic]"\}}||d|d	|d
t|d�d���$c}}|j!�D��cic]+\}}|t#t|j!�d������-c}}d�}|j%d|ztj&|d���|dScc}}wcc}}wcc}}w)z�Recompute summary/date=X.json from every event shard for that date.

    Reading the shards back (rather than summing only this run's events) keeps
    the summary correct on incremental runs and idempotent on re-runs.
    zevents/date=%s/c�(�ddtt�d�S)Nr�ro�bytes�by_platform�r�intrOrr�<lambda>z!rebuild_summary.<locals>.<lambda>�s��1�q��UX�IY�"Z�rc� �dddt�d�S)Nr�ror��category�sites)r�rOrrr�z!rebuild_summary.<locals>.<lambda>�s��q�1�$�Y\�Y^�&_�rc� �tt�Sr%r�rOrrr�z!rebuild_summary.<locals>.<lambda>�s���S�!1�rrr~r�ai_platformr�r
ror�r�r��url�%Y-%m-%dT%H:%M:%SZ)ror�r��	platformsc��|ddS�Nr
rorO��kvs rr�z!rebuild_summary.<locals>.<lambda>�s��2�a�5��=�.�r)rc��|dS�Nr
rOr�s rr�z!rebuild_summary.<locals>.<lambda>�s��UW�XY�UZ�TZ�rr�c��|ddSr�rOr�s rr�z!rebuild_summary.<locals>.<lambda>�s��B�q�E�&�M�>�rr�c��|dSr�rOr�s rr�z!rebuild_summary.<locals>.<lambda>�s��B�q�E�6�r)r{�generated_at�totals�by_siter��by_urlzsummary/date=%s.json�)�indentr�)r�sortedrKrr=�
splitlinesrr��loadsr�r�nowr�utc�strftimerr��dictr4r�)r�r{rFr�r�r��
total_hits�total_bytesrr3ryr�r�platform�nbytes�v�p�urls�summarys                   r�rebuild_summaryr��s����
%�F��Z�[�G��_�`�K�
�1�
2�F��J��K��d�n�n�V�,�-�(���|�|�H�%���x�x��}�����O�O�%�	(�D��:�:�<���
�
�4� �A��V�9�D���'�H��U�U�7�A�&�F��!�O�J��6�!�K��D�M�&�!�Q�&�!��D�M�'�"�f�,�"��D�M�-�(��2�a�7�2���!�&�)�Q�.�)���!�'�*�f�4�*�01���j�0A�K��!�*�-���!�'�*�.�.�t�4��4�L��5��"�a�'�"�#	(�
(�4� ���X�\�\�2�;�;�<P�Q�� ���\��[�)�	
�"�'�-�-�/�7P�Q�

���a�
��&�	��7��#�F�1�]�+;�+A�+A�+C�IZ�$[�\��
�
��{�0�0�2�8Q�R�
���1�

��&�	��7���j�M��Q�w�Z��	�
�
�%�l�l�n�
���d�
�$�v�d�j�j�l�0A�B�C�C�
�5�G�>	�H�H�
#�d�
*�D�J�J�w�q�,I�J��8����/
��
��
s�=K�2'K�/0K c��|jt�}|sdiiS	tj|�}|jdi�|S#t$rdiicYSwxYw)N�	processed)r=�MANIFEST_KEYr�r��
ValueError�
setdefault)r�r3�datas   r�
load_manifestr��sb���8�8�L�!�D���R� � �!��z�z�$���	�O�O�K��$��K���!��R� � �!�s�A�
A�Ac���tjtj�j	d�|d<|jttj|dd���y)Nr��
updated_atr�T)r�r�)	rr�rr�r�r4r�r�r�)r��manifests  r�
save_manifestr��s@��%�\�\�(�,�,�7�@�@�AU�V�H�\���H�H�\�4�:�:�h�q�D�I�Jrc�4�g}|jd�}|j||��D]^}|jdg�D]G}|djt�s�|j|d|dj
d�|df��I�`|j�|S)zCReturn [(key, etag, size)] for apachestyle logs, oldest name first.r?r@rBr.�ETag�"�Size)rCrDr=rrrEr�sort)r'r&rF�outrHrIrJs       r�list_source_filesr��s���
�C�� � �!2�3�I��"�"�&��"�@�J���8�8�J��+�	J�C��u�:�&�&�}�5���J�J��E�
�C��K�$5�$5�c�$:�C��K�H�I�	J�J�
�H�H�J��Jrc#�K�|j||��}tjtj|dj����5}tj|dd��D]}|���	ddd�y#1swYyxYw�w)z8Stream a gzipped object from S3 and yield decoded lines.r6r/)�fileobjr,�replace)�encoding�errorsN)r7�gzip�GzipFile�io�BytesIOr8�
TextIOWrapper)r'r&rr<�gz�raws      r�read_gzip_linesr�ss����
�=�=��C�=�0�D�	
���r�z�z�$�v�,�*;�*;�*=�>�	?��2��#�#�B���K�	�C��I�	����s�AB�"A<�3	B�<B�Bc���t|�}|d}t|||�}|D�	�
�cgc]/\}	}
}|j|	i�jd�|
k7r|	|
|f��1}}
}	}|dt|�t|�fz�|d|}
t	�}ddddd�}|
D]�\}	}
}tt
|||	�|	�\}}t|||	�}||z}|
||d|d|dt|�tjtj�jd	�d
�||	<|dxxdz
cc<|dxx|dz
cc<|dxx|dz
cc<|dxx|dz
cc<|d
|	|d|d|dt|�xsdfz���t|�D]"}t||�}|d||d|dfz��$t||�t!|�}t|�|d<t|�t|
�z
|d<|Scc}}
}	w)Nr��etagz"source files: %d total, %d pendingr)rirnrorprnrorpr�)r��sizernrorpr��processed_atrir
z*  %s: %d lines, %d hits, %d unparsed -> %s�-z(summary date=%s: %d hits across %d sitesr�r��	remaining)r�r�r=rr�r|r�r�r�rr�rr�r�r�r�r�)r'r��
source_bucket�
source_prefix�	max_files�logr�r��	all_filesrr�r��pending�batch�
touched_datesr�rwrxr�r{�t�results                      r�runr�sL���T�"�H���%�I�!�"�m�]�C�I� )����C��t��=�=��b�!�%�%�f�-��5�

�d�D���G��
�,��I���G��/M�M�N��J�Y��E��E�M��1�a�Q�
?�F� �]���T�4�&��r�=�#�'N�PS�T�����T�7�C�0�����
����7�^��&�M��j�)��E�]�$�L�L����6�?�?�@T�U�
�	�#��	�w��1����w��5��>�)���v��%��-�'���z��e�J�/�/���8��E�'�N�E�&�M�5��3D�f�U�m�FZ�WZ�[�
\�	]�%]�*�}�%�X���D�$�'���6�$��&�	�1�W�:�9V�V�W�X��$��!�
�&�\�F��]�+�F�7�O��g�,��U��3�F�;���M��Ss�4G+c���tjd�}tt|�}t	||t
tt�}ttj|��|S)Nr')�boto3r)r#rr�rrr	�printr�r�)rz�contextr'r�r�s     r�lambda_handlerr�@sE��	���d�	�B��+�r�"�D�
��T�=�-�9J�
K�F�	�$�*�*�V�
���Mr)%rkr�r�r�rU�re�collectionsrrrr��environr=rrrr�rr�r	�compilerrr!�objectr#rQr|r�r�r�r�r�r�r�r�r�rOrr�<module>r�s���
��	��	�	�#�'���
�
����0F�G�
��j�j�n�n�]�,A�B���
�
����
�>�
�(��&�
���
�
���':�E�B�C����
�
�+�,�
�0� ��V��8���J�2	�F�R	�K�

��@E�.�br

Youez - 2016 - github.com/yon3zu
LinuXploit