޺i5,%SrSSKrSSKJrJrJr SrSrSrSr Sr S r S r S \-r \\S 'S \S\S\4SjrS \S\S-4SjrS \S\S-4SjrS \S\S-4SjrS\S\4SjrSS\S\S\4Sjjrg)aStage 1a+: UTF-16/UTF-32 detection for data without BOM. This stage runs after BOM detection but before binary detection. UTF-16 and UTF-32 encoded text contains characteristic null-byte patterns that would otherwise cause binary detection to reject the data. Note: ``from __future__ import annotations`` is intentionally omitted because this module is compiled with mypyc, which does not support PEP 563 string annotations. N)ASCII_TEXT_BYTESDETERMINISTIC_CONFIDENCEDetectionResulti gQ??gffffff?g333333?_NULL_SEPARATOR_ALLOWEDdata null_fracreturncNU[:agURS[5(+$)ucReturn True if the data looks like ASCII with null byte separators. :param data: The raw byte sample to examine. :param null_frac: The positional null fraction for this UTF-16 candidate (i.e. fraction of null bytes in even positions for BE, or odd positions for LE) — not the total null fraction across all bytes. Checks two conditions: 1. The positional null fraction is below ``_NULL_SEPARATOR_MAX_FRACTION`` 2. Every non-null byte is printable ASCII or common whitespace When both conditions are met, the nulls are likely field separators (e.g. ``find -print0``), not UTF-16 encoding artifacts. FN)_NULL_SEPARATOR_MAX_FRACTION translater )r r s W/var/www/piano.thomer.com/venv/lib/python3.13/site-packages/chardet/pipeline/utf1632.py_is_null_separator_patternr6s%00~~d$;< <<crUS[n[U5[:ag[U5nUbU$[ U5$)aDetect UTF-32 or UTF-16 encoding from null-byte patterns. UTF-32 is checked before UTF-16 since UTF-32 patterns are more specific. :param data: The raw byte data to examine. :returns: A :class:`DetectionResult` if a strong pattern is found, or ``None``. N) _SAMPLE_SIZElen_MIN_BYTES_UTF16 _check_utf32 _check_utf16)r sampleresults rdetect_utf1632_patternsrJsD-< F 6{%%& !F    rc ^[T5[T5S-- nU[:agTSUmUS-n[U4Sj[S[T5S555n[U4Sj[S[T5S555nX2:Xa:XB- S:a2TR S5n[ U5(a[ S[SS9$[U4S j[S [T5S555n[U4S j[S [T5S555nXb:Xa;Xr- S:a3TR S 5n[ U5(a[ S [SS9$gg![a Nf=f![a gf=f)avCheck for UTF-32 encoding based on 4-byte unit structure. For valid Unicode (U+0000 to U+10FFFF = 0x0010FFFF): - UTF-32-BE: the first byte of each 4-byte unit is always 0x00 - UTF-32-LE: the last byte of each 4-byte unit is always 0x00 For BMP characters (U+0000 to U+FFFF), additionally: - UTF-32-BE: the second byte is also 0x00 - UTF-32-LE: the third byte is also 0x00 Nc3>># UHnTUS:XdMSv M g7frN.0ir s r _check_utf32..tJ#9aT!W\#9  rc3D># UHnTUS-S:XdMSv M g7f)r!rNr"r#s rr&r'vs$O$:qd1q5kQ>N$:s  rz utf-32-beencoding confidencelanguagec3>># UHnTUS:XdMSv M g7fr r"r#s rr&r'sI"8QDGqLqq"8r)c3>># UHnTUS:XdMSv M g7fr r"r#s rr&r'r(r)z utf-32-le) r_MIN_BYTES_UTF32sumrangedecode_looks_like_textrrUnicodeDecodeError)r trimmed_len num_units be_first_nullbe_second_nulltext le_last_null le_third_nulls` rrr`svd)s4y1}-K%%   Dq IJ5CIq#9JJMOE!SY$:OON!n&@3&F ;;{+D%%&(7!&I%3t9a"8IILJ5CIq#9JJM ]%>%D ;;{+D%%&(7!& )"   ""    s$/E"/E% E"!E"% E21E2cV^[[T5[5nXS--nU[:agUS-n[ U4Sj[ SUS555n[ U4Sj[ SUS555nX2- nXB- n/nU[ :a'[TSUU5(dURSU45 U[ :a'[TSUU5(dURSU45 U(dg[U5S:Xa>USSnTSURU5n [U 5(a[U[SS 9$gSn S n UH0upTSURU5n [U 5n X:dM,U n Un M2 U bU [:a[U [SS 9$g![a gf=f![a Mnf=f) aCheck for UTF-16 via null-byte patterns in alternating positions. UTF-16 encodes each BMP character as two bytes. For characters whose code-point high byte is 0x00 (Latin, digits, basic punctuation, many control structures), one of the two bytes in each unit will be a null. Even for non-Latin scripts (Arabic, CJK, Cyrillic, etc.) a significant fraction of code units still contain at least one null byte. Non-UTF-16 single-byte encodings never contain null bytes, so even a small null-byte fraction in alternating positions is a strong signal. When both endiannesses show null-byte patterns (e.g., Latin text where every other byte is null), we disambiguate by decoding both ways and comparing text-quality scores. r2Nc3>># UHnTUS:XdMSv M g7fr r"r#s rr&_check_utf16..K#:ad1gl#:r)rc3>># UHnTUS:XdMSv M g7fr r"r#s rr&rBrCr)r!z utf-16-lez utf-16-ber+)minrrrr4r5_UTF16_MIN_NULL_FRACTIONrappendr6r7rrr8 _text_quality_MIN_TEXT_QUALITY)r sample_lenr: be_null_count le_null_countbe_fracle_frac candidatesr,r= best_encoding best_quality_qualitys` rrrs SY -Jq. J$$aIK5J#:KKMK5J#:KKM'G'G*,J**3M [j744 ;01**3M [j744 ;01  :!a=#  $++H5D%%&%7!&!%ML!   $++H5D %  !"L$M" \5F%F"/  5"   "   s$>2F ?F FF F('F(r=cjU(dgUSSn[SU55nU[U5- [:$)z9Quick check: is decoded text mostly printable characters.FNc3^# UH#oR5(dUS;dMSv M% g7f) r!N) isprintable)r$cs rr&#_looks_like_text..s Jv!AMAAvs- -)r4r_MIN_PRINTABLE_FRACTION)r=r printables rr7r7s8  $3ZFJvJJI s6{ "%< <10% control characters or >20% combining marks (category ``M*``). NrrELr!MZsrXCg?g?r)r unicodedatacategoryord) r=r^rnlettersmarksspacescontrols ascii_lettersrZcatscores rrIrIs(&5\F F AAvG E FHM ""1% q6S= qLG1v|"  Vs] QJE D[AM aKF Vs] MH|c y3 KE a3 &&E2v&1*   Lr)rV)__doc__rfchardet.pipelinerrrrr3rrGrJr\rr bytes__annotations__floatboolrrrrstrr7intrIr"rrrys XX   $")+;!;;=U=u==( % Od,B ,5u54!75pQuQ4!7Qh=3=4=99C9%9r