| Server IP : 138.197.107.151 / Your IP : 216.73.217.7 Web Server : Apache/2.4.58 (Ubuntu) System : Linux BloxBy-Builder 6.8.0-71-generic #71-Ubuntu SMP PREEMPT_DYNAMIC Tue Jul 22 16:52:38 UTC 2025 x86_64 User : wpbetasites_mrakzqskir ( 1022) PHP Version : 8.3.6 Disable Function : NONE MySQL : OFF | cURL : ON | WGET : ON | Perl : ON | Python : OFF | Sudo : ON | Pkexec : OFF Directory : /var/www/bsd-crawler-parser/__pycache__/ |
Upload File : |
�
> ^j� �
�� � d Z ddlZddlmZmZ ddlmZ dZ ej dez dz ez dz ez d z � Zd
d d d
ddddddddd� Z ej d� Z
d� Z d� Z d� Z
d� Zd � Zd � Zy) z�Parsing for WP Engine nginx logs in Apache combined format + vhost.
Line shape:
IP vhost - [timestamp] "METHOD path HTTP/x" status bytes "referrer" "user-agent"
� N)�datetime�timezone)�
match_crawlerz(?:[^"\\]|\\.)*zZ^(?P<ip>\S+)\s+(?P<vhost>\S+)\s+(?P<ident>\S+)\s+\[(?P<timestamp>[^\]]+)\]\s+"(?P<request>z:)"\s+(?P<status>\d{3})\s+(?P<bytes>-|\d+)\s+"(?P<referrer>z)"\s+"(?P<user_agent>z)"� � � � � � � � � �
� � ) �Jan�Feb�Mar�Apr�May�Jun�Jul�Aug�Sep�Oct�Nov�DeczN^(\d{2})/([A-Za-z]{3})/(\d{4}):(\d{2}):(\d{2}):(\d{2})\s+([+-])(\d{2})(\d{2})$c
�� � t j | j � � }|sy|j � \ }}}}}}}} }
t j |j
� � } | �yt | � dz t |
� z } |dk( r| } t t |� | t |� t |� t |� t |� t j �� }
|
t | � z
S # t $ r Y yw xY w)z?'08/Jul/2026:00:17:35 +0000' -> timezone-aware datetime in UTC.N�< �-)�tzinfo)
�_TS_RE�match�strip�groups�_MONTHS�get�title�intr r �utc�
ValueError�_minutes)�raw�m�day�mon�year�hour�minute�sec�sign�oh�om�month�offset_minutes�dts �(/var/www/bsd-crawler-parser/logparser.py�parse_timestampr< ' s� � �� � �S�Y�Y�[�!�A� ��67�h�h�j�3�C��d�D�&�#�t�R�� �K�K�� � � � $�E� �}����W�r�\�C��G�+�N� �s�{�(� ���
� ��I�u�c�#�h��D� �3�v�;��C���<�<�
�� �� �(� (�(�� � ���s �
A
C% �% C1�0C1c � � ddl m} || �� S )Nr )� timedelta)�minutes)r r> )�nr> s r; r, r, > s � �"� �Q� � � c �
� | j d� }t |� dk\ r1|d j d� r |d dj |dd � |d fS t |� dk( r |d |d dfS t |� dk( rd|d dfS d| dfS ) zF'GET /path HTTP/1.1' -> ('GET', '/path', 'HTTP/1.1'), tolerating junk.� r ���zHTTP/r r r � )�split�len�
startswith�join)�request�partss r; �
split_requestrL C s� � � �M�M�#� �E�
�5�z�Q��5��9�/�/��8��Q�x����%��"�+�.��b� �9�9�
�5�z�Q���Q�x��q��2�%�%�
�5�z�Q���5��8�R� � �
�w��?�rA c � � | j � j � j d� }|j d� r|dd }|S )z<Fold www. into the bare domain; leave everything else alone.�.zwww.r N)r$ �lower�rstriprH )�vhost�sites r; �normalize_siterS O s= � � �;�;�=� � � � '� '�� ,�D� ���v� ��A�B�x�� �KrA c
�� � t j | j � � }|sy|j d� }t |� }|�y|\ }}t |j d� � }|�yt
|j d� � \ }} }
|j d� } t |j d� � |||| t |j d� � | dk( rd n
t | � |j d
� |d � S ) z�Parse one log line; return an event dict only for AI crawler hits.
Returns None for non-matching lines and for lines we cannot parse.
N�
user_agent� timestamprJ �bytesrQ �statusr r z%Y-%m-%dT%H:%M:%SZ) rR � ai_platform�categoryrU �url� status_coderW rV � source_file)
�LINE_REr# r$ �groupr r< rL rS r) �strftime) �line�
source_keyr. rU �matched�labelrZ r: �_method�path�_proto� raw_bytess r; �
parse_lineri W s� � �
�
�
�d�j�j�l�#�A� ����� �&�J��J�'�G���� �O�E�8� �����-� .�B� �z�� )�!�'�'�)�*<� =��G�T�6����� �I� �q�w�w�w� /�0� � � ���1�7�7�8�,�-� �3�&��C� �N��[�[�!5�6�!�
�
rA c # �b K � | D ]& }|j � s�t ||� }|��#|�� �( y�w)zIYield (event, stats_key) for AI hits; stats_key tracks why lines dropped.N)r$ ri )�linesrb ra �events r; � iter_eventsrm z s8 � �� �� ���z�z�|� ��4��,�� � ��K� �s �%/�/)�__doc__�rer r �crawlersr �_QUOTED�compiler^ r& r" r< r, rL rS ri rm � rA r; �<module>rt s� ���
� '� "� ��
�"�*�*�� � �"� � � �#�� "�"� %*�*�
��
�Q�q��1�Q� �Q�q��B�r� ��
����U�
��
)�. �
�� �FrA