| Server IP : 138.197.107.151 / Your IP : 216.73.217.7 Web Server : Apache/2.4.58 (Ubuntu) System : Linux BloxBy-Builder 6.8.0-71-generic #71-Ubuntu SMP PREEMPT_DYNAMIC Tue Jul 22 16:52:38 UTC 2025 x86_64 User : wpbetasites_mrakzqskir ( 1022) PHP Version : 8.3.6 Disable Function : NONE MySQL : OFF | cURL : ON | WGET : ON | Perl : ON | Python : OFF | Sudo : ON | Pkexec : OFF Directory : /var/www/bsd-crawler-parser/__pycache__/ |
Upload File : |
�
< ^j3 �
� � d Z i dd�dd�dd�dd�d d
�d d �d
d�dd�dd�dd�dd�dd�dd�dd �d d �d d �d!d"�d#d$d%d&d'd(d)d*d+��Zd,� Z e� Zd-� Zy.)/a� Registry of AI crawler user-agents we care about.
To add a crawler: add one row to AI_CRAWLERS. The key is the case-insensitive
substring to look for in the User-Agent; `label` is what shows up in reports;
`category` is one of ai_training / ai_assistant / search.
ai_training collects a corpus to train models on
ai_assistant fetches a page live, because a user asked an assistant about it
search builds a search index (including AI-flavoured search indexes)
Longer patterns win over shorter ones, so "Applebot-Extended" is matched
before the "Applebot" that is a substring of it. That ordering is handled
automatically in _compile_patterns() -- you do not need to sort this dict.
�GPTBot)r � ai_training�
OAI-SearchBot)r �search� ChatGPT-User)r � ai_assistant� ClaudeBot)r r � Claude-User)r r �Claude-SearchBot)r
r � anthropic-ai)r r �
PerplexityBot)r r �Perplexity-User)r
r �Google-Extended)r r � GoogleOther)r r �Applebot-Extended)r r �Applebot)r r � Amazonbot)r r �
Bytespider)r r �meta-externalagent)r r � FacebookBot)r r )�CCBotr )� cohere-air )�Diffbotr )� ImagesiftBotr )�Timpibotr )�YouBotr )�
DuckAssistBotr )�Bingbotr )r r r r r r r r c � � t j � D � ��cg c] \ } \ }}| j � ||f�� }}} }|j d� d�� |S c c}}} w )z=Lowercased (pattern, label, category), longest pattern first.c � � t | d � S )N� )�len)�rows �'/var/www/bsd-crawler-parser/crawlers.py�<lambda>z#_compile_patterns.<locals>.<lambda>= s � �c�#�a�&�k� � T)�key�reverse)� AI_CRAWLERS�items�lower�sort)�pattern�label�category�rowss r# �_compile_patternsr0 7 sb � � +6�*;�*;�*=� � � &�G�&�e�X�
� � � �%��*� �D� � �I�I�)�4�I�8� �K�� s � Ac �\ � | sy| j � }t D ] \ }}}||v s� ||fc S y)z�Return (label, category) for the first AI crawler in `user_agent`.
Returns None when the user-agent is not a known AI crawler.
N)r* � _PATTERNS)�
user_agent�haystackr, r- r. s r# �
match_crawlerr5 D sF � �
��� � �!�H�$-� %� ���� �h� ��8�$� $�%� r% N)�__doc__r( r0 r2 r5 � r% r# �<module>r8 s! ��
� $� �'�$� �0�$� �4� $� �-�
$� �2�$� �6�$� �3�$� �0�$� �:�$� �9� $� �1� $�" �=�#$�$ �&�%$�( �(�)$�, �/�-$�0 �?�1$�2 �1�3$�6 &�-�)�3�+�"�6�$�G$� �N�
� � � r%