| Server IP : 138.197.107.151 / Your IP : 216.73.217.120 Web Server : Apache/2.4.58 (Ubuntu) System : Linux BloxBy-Builder 6.8.0-71-generic #71-Ubuntu SMP PREEMPT_DYNAMIC Tue Jul 22 16:52:38 UTC 2025 x86_64 User : wpbetasites_mrakzqskir ( 1022) PHP Version : 8.3.6 Disable Function : NONE MySQL : OFF | cURL : ON | WGET : ON | Perl : ON | Python : OFF | Sudo : ON | Pkexec : OFF Directory : /var/www/bsd-crawler-parser/viewer/__pycache__/ |
Upload File : |
�
H ^j( � �< � d Z dZdZdZdZdZdZdZd� Zd � Z d
� Z
d � Z y )
a� What counts as a "content page", for the Content-pages-only toggle.
A crawler hitting a site fetches three quite different things, and only one of
them is your writing:
1. content the page itself /blog/dog-bite-claims/
2. furniture assets and APIs the page needs /js/main.js, /wp-json/...
3. probing someone looking for a way in /.env, /.ssh/id_ed25519
Counting all three together overstates how much of your *content* is being
read. The toggle hides 2 and 3; turning it off shows everything. Nothing is
filtered out of S3 -- this only affects what the viewer counts and displays.
MATCHING IS ON PATH SEGMENTS, NOT SUBSTRINGS. `".env" in url` also matches
"/uploads/flyer.envelope.pdf", and `"secret" in url` matches a perfectly real
law-firm article at "/blog/the-secret-to-winning". Splitting the path and
testing whole segments avoids quietly deleting real traffic from the numbers.
)z.jsz.mjsz.cjsz.cssz.mapz.pngz.jpgz.jpegz.gifz.svgz.icoz.webpz.avifz.bmpz.tifz.tiffz.woffz.woff2z.ttfz.otfz.eotz.mp4z.webmz.movz.mp3z.wavz.ogg)z.ymlz.yamlz.iniz.bakz.sql�.env)zwp-json)z
robots.txt)r z.gitz wp-configzservice-account) z.sshz.awsz.svnz.vscodez.idea�secrets�secret� credentials�id_rsa�id_dsa�id_ecdsa�
id_ed25519)r �
credentialc �r � | xs dj d� d j d� d j � S )zBThe path part of a request, lowercased, without query or fragment.� �?� �#)�split�lower��urls �-/var/www/bsd-crawler-parser/viewer/content.py�_pathr : s5 � � �I�2� � �S� !�!� $� *� *�3� /�� 2� 8� 8� :�:� c �z � t | � }|j d� D �cg c] }|s�|�� }}|r|d nd}|t v ry|D ]- }|t v r y|j t
� r y|t v s�- y d|v rB|j t � ry|j t � ry|j t � ryy c c}w )
zcWhy a request is not content: 'asset' | 'api' | 'robots' | 'probe',
or None when it IS content.�/���r �robots�probe�api�.�assetN) r r �NON_CONTENT_FILES�PROBE_SEGMENTS_EXACT�
startswith�PROBE_SEGMENT_PREFIXES� API_SEGMENTS�endswith�CONFIG_EXTENSIONS�PROBE_FILENAME_PREFIXES�STATIC_EXTENSIONS)r �path�s�segments�last�segs r �classifyr- ? s� � � ��:�D� �:�:�c�?�0�a�a��0�H�0�#�8�B�<��D� � � ��� �� �&� &�� �>�>�0� 1�� �,� ��
� �d�{� �=�=�*� +�� �?�?�2� 3�� �=�=�*� +�� ��- 1s
�B8�B8c � � t | � du S )z-True when this request is for an actual page.N)r- r s r �
is_contentr/ \ s � � �C�=�D� � r c
�( � ddt t � fddt t � fddt D � cg c] } d| z �� c} fddt t � fd d
t t
� fd d t t � fd d t D �cg c] }|d
z �� c}fgS c c} w c c}w )zGThe filter, as data, so the UI and the docs cannot drift from the code.z
Static assetszfile extensionzConfig / secret filesz
API endpointsz path segmentz/%s/zRobots instructions�filenamezScanner probeszpath segment starts withzpath segment equalszfilename starts withz.*)�listr' r% r# r r" r r& )r) �ps r �pattern_summaryr4 a s� � �
�*�D�1B�,C�D� �"2�D�9J�4K�L� �.�|�*L�!�6�A�:�*L�M� �
�D�1B�,C�D� �5�
�$� %� '� �0�$�7K�2L�M� �1� 3� 4�q�!�d�(� 4� 6�
�
�� +M��
5s � B
�9 BN) �__doc__r' r% r# r r" r r& r r- r/ r4 � r r �<module>r7 sV ���(� � F� � � � $� �� �� � 3� �;�
�:!�
r