| Server IP : 138.197.107.151 / Your IP : 216.73.217.7 Web Server : Apache/2.4.58 (Ubuntu) System : Linux BloxBy-Builder 6.8.0-71-generic #71-Ubuntu SMP PREEMPT_DYNAMIC Tue Jul 22 16:52:38 UTC 2025 x86_64 User : wpbetasites_mrakzqskir ( 1022) PHP Version : 8.3.6 Disable Function : NONE MySQL : OFF | cURL : ON | WGET : ON | Perl : ON | Python : OFF | Sudo : ON | Pkexec : OFF Directory : /proc/698955/cwd/__pycache__/ |
Upload File : |
�
I ^j ! �
�> � d Z ddlZddlZej j dej j ej j ej j
e� � � � ddlm Z h d�Z
dZ e dz Z ddd d
d �Z
d d
ddd �Zi dd�dd�dd�dd�dd�dd�d d �d d �d d!�d"d#�d$d%�d&d'�d(d)�d*d+�d,d-�d.d/�d0d1�d2d3d4d5d6d7d8d9d:��Zd;� Z e j" � D � �ci c] \ } }| |��
c}} Z G d<� d=e� Zd>� Z e� dBd?�Zd@� ZdA� Zyc c}} w )Ca� The bot -> group mapping the viewer displays.
Derived from the parser's own AI_CRAWLERS registry, with one change: the
parser's `search` bucket conflated two different things, so it is split in two.
ai_training collects a corpus to train models on
ai_assistant fetches a page live, because a user asked an assistant about it
ai_search an AI company building an index <- split out of `search`
search a conventional search engine indexer <- the only hidden group
Only the last group is hidden by default; the other three are "AI activity".
WHY WE RE-DERIVE THIS INSTEAD OF READING THE EVENT'S OWN `category` FIELD
The ~350k events already in S3 have the OLD three-way category baked in at
parse time -- OAI-SearchBot is literally stored as "search". Deriving the group
from `ai_platform` on read applies this grouping to all existing data with no
reprocessing and no writes to the bucket. It also means old and newly parsed
data always agree, even though crawlers.py still emits the old categories.
Adding a crawler to crawlers.py picks it up here automatically. Only add a name
to AI_SEARCH_PLATFORMS below if it is an AI company's *indexing* bot.
� N)� AI_CRAWLERS> �YouBot�
PerplexityBot�
OAI-SearchBot�Claude-SearchBot)� ai_training� ai_assistant� ai_search)�searchz AI trainingz AI assistantz AI searchzSearch engines)r r r
r zmBots that copy your pages to help train an AI model. Your content becomes part of what the model learns from.zlBots that fetch one of your pages right now, because a person just asked an AI assistant something about it.zeAI companies indexing your pages so their AI can find you and link to you when it answers a question.zoOrdinary search engines building their normal search index. Hidden by default, because this is not AI activity.�GPTBotzEOpenAI's collector. Copies pages to help train future ChatGPT models.r zHOpenAI's indexer. Lists your pages so ChatGPT can find and link to them.z ChatGPT-Userz<Visits one page because someone just asked ChatGPT about it.� ClaudeBotz9Anthropic's collector. Copies pages to help train Claude.z Claude-Userz;Visits one page because someone just asked Claude about it.r zGAnthropic's indexer. Lists your pages so Claude can find and cite them.z anthropic-aiz=An older Anthropic collector that gathers pages for training.r zGPerplexity's indexer. Lists your pages so its AI answers can cite them.zPerplexity-Userz?Visits one page because someone just asked Perplexity about it.zGoogle-ExtendedzTNot a visitor as such: Google's switch for whether Gemini may learn from your pages.� GoogleOtherzWA general-purpose Google crawler used by its internal teams. This is not Google Search.zApplebot-ExtendedzDApple's AI collector. Copies pages to help train Apple Intelligence.�ApplebotzCApple's ordinary search crawler, behind Siri and Spotlight results.� Amazonbotz1Amazon's crawler, mostly feeding Alexa's answers.�
BytespiderzEByteDance's collector (TikTok's owner). Copies pages for AI training.zmeta-externalagentz]Meta's collector. Copies pages to help train Meta AI across Facebook, Instagram and WhatsApp.� FacebookBotz?Meta's collector, gathering pages to train its language models.zzCommon Crawl, a non-profit that archives the web. Many AI companies train on its archive, so this feeds others downstream.zFCohere's collector. Copies pages to help train its business AI models.zEDiffbot's crawler. Turns pages into a knowledge database it sells on.z<Collects images from your pages, used for AI image training.z7Timpi's crawler, building a community-run search index.zCYou.com's indexer. Lists your pages so its AI search can cite them.zEVisits one page because someone just asked DuckDuckGo's AI assistant.zCMicrosoft's ordinary search crawler -- the one behind Bing results.)�CCBotz cohere-ai�Diffbot� ImagesiftBot�Timpibotr �
DuckAssistBot�Bingbotc �b � | t v r t | S t j t | � d� S )z;One-line explanation of a bot, falling back to its group's.zAn automated crawler.)�BOT_INFO�
GROUP_INFO�get� category_for)�platforms �0/var/www/bsd-crawler-parser/viewer/categories.py�bot_infor l s, � ��8����!�!� �>�>�,�x�0�2I� J�J� c � � e Zd ZdZy)�UncategorisedBotaj A bot does not resolve to exactly one of the four display groups.
Raised loudly rather than letting the hit vanish. Every hit belongs to
exactly one group, so the category boxes must always sum to total Hits; a
bot that resolves to something else would be counted in the total but shown
in no box, and the page would quietly stop adding up.
N)�__name__�
__module__� __qualname__�__doc__� r! r r# r# v s � �r! r# c � � t t j � D � �ch c] \ } }|t vr| |f�� c}} � }|r;t ddj d� |D � � �ddj t � �d�� �yc c}} w )a1 Fail at startup if crawlers.py has a category we cannot display.
The parser's registry is the source of truth for which bots exist, so it can
grow a category (say "ai_agent") that this viewer has no box for. That must
be a loud error here, not two hits silently missing from a dashboard.
z7crawlers.py has categories this viewer cannot display: �, c 3 �& K � | ] }d |z �� � y�w)z%s=%rNr( )�.0�rs r � <genexpr>z"_check_registry.<locals>.<genexpr>� s � �� �4���1��4�s �z .
Every bot must map to one of z`. Either add a group here (and a stat box in overview.html), or fix the category in crawlers.py.N)�sortedr �values�
ALL_GROUPSr# �join)�label�cat�rogues r �_check_registryr6 � s| � �
�;�3E�3E�3G� .�Z�U�C��*�,� �C�L� .�
/�E� � � �y�y�4�e�4�4�d�i�i�
�6K�
M�N� N�
��.s �A9
c � � | t v ryt j | � }|�0|t vr&t | �d|�ddj t � ��� �|S |t v r|S y)a0 The display group for a bot name. Always exactly one of ALL_GROUPS.
`stored` is the category recorded in the event, used only as a fallback for
a bot this viewer does not know about (i.e. one added to the parser after
this file was last read). Unknown bots are never silently called AI.
r
z has category z, which is not one of r* r )�AI_SEARCH_PLATFORMS�_PARSER_CATEGORYr r1 r# r2 )r �stored�knowns r r r � sg � � �&�&�� � � �� *�E� �� �
� "�"��U�D�I�I�j�$9�;�<�
<� � �
����
� r! c � � | t v S )z6True for anything except a conventional search engine.)� AI_GROUPS)�groups r �is_air? � s
� � �I� � r! c �: � t D � ci c] } | g �� }} t D ] }|t |� j |� �! t D ] }|t vs� |d j |� �! |j � D � �ci c] \ } }| t
|� �� c}} S c c} w c c}} w )zC{group: [platform, ...]} over every configured crawler, for the UI.r
)r1 r9 r �appendr8 �itemsr/ )r> �outr3 �labelss r �platforms_by_grouprE � s� � �",�
-��5�"�9�
-�C�
-�!� /�� �L�� � �'�'��.�/�$� +�� �(� (� � � � #� #�E� *�+� 8;�y�y�{� C�m�e�V�E�6�&�>� !� C�C��
.�� Ds �
B�8B)N)r' �os�sys�path�insert�dirname�abspath�__file__�crawlersr r8 r= r1 � GROUP_LABELSr r r r0 r9 � Exceptionr# r6 r r? rE )r3 �categorys 00r �<module>rQ s ���0
�
� ������2�7�7�?�?�2�7�7�?�?�2�7�7�?�?�8�3L�#M�N� O� �
� �
9� �
��
$�
� !�"� � � � �N�L�?�D� �
�% � �U�% � �_�% � �R� % � �L�
% � �P�% � �a�% � �S�% � �^�% � �X�% � � 0� % � � 0�!% �& �_�'% �( �U�)% �, �D�-% �0 �Y�1% �2 � >�3% �6 �T�7% �8F�Y�V�R�I�S�\�T�I% ��PK� <N�;�;M�;M�;O�P���x�E�8�O�P� ��y� �N�" � ��* �
D��w Qs �(
D