޺ic %SrSSKJr SSKrSSKJr SSKJr SSKJ r J r J r J r SSK JrJrJrJr SSKJr SS KJr SS KJr SS KJr SS KJr SS KJr SSKJr SSK J!r!J"r"J#r# SSK$J%r% SSK&J'r' SSK(J)r) SSK*J+r+J,r, \"S\SS9r-\"SSSS9r.Sr/\0"1Sk5r1S\2S'\0"1Sk5r3S\2S'\0"1Sk5r4S\2S'\0"1Sk5r5S\2S '\3\4\5S!.r6S"\2S#'\0"1S$k5r7S\2S%'S:S&jr8S;S'jr9S(r:S)r;S*rSS/jrAS>S0jrBS1rCS?S2jrDS>S3jrES>S4jrF\4SSS5S6S7.S@S8jjjrG\4SSS5S6S7.S@S9jjjrHg)Au@Pipeline orchestrator — runs all detection stages in sequence.) annotationsN)DEFAULT_MAX_BYTES) EncodingEra) BigramProfilehas_model_variantsinfer_languagescore_best_language)DETERMINISTIC_CONFIDENCE HIGH_BYTESDetectionResultPipelineContext) detect_ascii) is_binary) detect_bom)resolve_confusion_groups)detect_escape_encoding)detect_markup_charset)score_candidates)compute_lead_byte_diversitycompute_multibyte_byte_coveragecompute_structural_score) detect_utf8)detect_utf1632_patterns)filter_by_validity) EncodingInfoget_candidatesencoding confidencelanguageg333333?>cp1252 iso8859-1 iso8859-15frozenset[str]_COMMON_LATIN_ENCODINGS>.zfrozenset[int]_ISO_8859_10_DISTINGUISHING>r'r(r*r+r,r-r/r0r1r3r4r5r6r7r8r:r;r<r=r>r?r@rArIrR_ISO_8859_14_DISTINGUISHING>rXrYrZr[_WINDOWS_1254_DISTINGUISHING)z iso8859-10z iso8859-14cp1254zdict[str, frozenset[int]]_DEMOTION_CANDIDATES> r'r(r+r8_KOI8_T_DISTINGUISHINGctX;a([R"USU<S3[SS9 [/$[ USSS9/$)zReturn a low-confidence result for *encoding*, or ``encoding=None`` if filtered out. ``stacklevel=5`` targets the public caller: detect() -> run_pipeline() -> _run_pipeline_core() -> _make_fallback_or_none().  zL is excluded by include_encodings/exclude_encodings; returning encoding=None) stacklevelg?Nr)warningswarn UserWarning _NONE_RESULTr )rallowed param_names \/var/www/piano.thomer.com/venv/lib/python3.13/site-packages/chardet/pipeline/orchestrator.py_make_fallback_or_nonervsO l!H<(K L   ~ X$ N OOcj^[RU5mTcg[U4SjU55(+$)aaReturn True if encoding is a demotion candidate with no distinguishing bytes. Checks whether any non-ASCII byte in *data* falls in the set of byte values that decode differently under the given encoding vs iso-8859-1. If none do, the data is equally valid under both encodings and there is no byte-level evidence for preferring the candidate encoding. Fc3<># UHoS:dM UT;v M g7fN).0bdistinguishings ru !_should_demote..sA1D&1&s  )ragetany)rdatars @ru_should_demoters2*--h7NAAA AArwg?gffffff?cD/nUGH nUR(a[XU5nXRRUR'U[:aMEUR c0[ U5[ URS[55- UlUR [:aM[XX"R S9nXbRUR'U[:aMUR [:a[XU5nU[:aMUR!U5 GM [#U5$)aEliminate CJK multi-byte candidates that lack genuine multi-byte structure. Four checks are applied in order to each multi-byte candidate: 1. **Structural pair ratio** (valid_pairs / lead_bytes) must be >= ``_CJK_MIN_MB_RATIO``. Catches files with many orphan lead bytes. 2. **Minimum non-ASCII byte count**: the data must contain at least ``_CJK_MIN_NON_ASCII`` bytes > 0x7F. Tiny files with 1-5 high bytes can accidentally form perfect pairs and score 1.0 structurally. 3. **Byte coverage** (non-ASCII bytes in valid multi-byte sequences / total non-ASCII bytes) must be >= ``_CJK_MIN_BYTE_COVERAGE``. Latin text has many high bytes that are NOT consumed by multi-byte pairs; genuine CJK text has nearly all high bytes accounted for. 4. **Lead byte diversity**: the number of distinct lead byte values in valid pairs must be >= ``_CJK_MIN_LEAD_DIVERSITY``. Genuine CJK text draws from a wide repertoire of lead bytes; European false positives cluster in a narrow band (e.g. 0xC0-0xDF for accented Latin). Returns the filtered candidate list. Structural scores are cached in ``ctx.mb_scores`` for reuse in Stage 2b. N)non_ascii_count) is_multibyter mb_scoresname_CJK_MIN_MB_RATIOrlen translater _CJK_MIN_NON_ASCIIr mb_coverage_CJK_MIN_BYTE_COVERAGE_CJK_DIVERSITY_MIN_NON_ASCIIr_CJK_MIN_LEAD_DIVERSITYappendtuple)rvalid_candidatesctxgatedencmb_score byte_coveragelead_diversitys ru_gate_cjk_candidatesrs:!#E   /3?H&.MM#(( #++""*&)$i#dnnT:6V2W&W#""%77;30C0CM)6OOCHH %55""&BB!= 0.95) receive a confidence boost proportional to coverage. When nearly all non-ASCII bytes form valid multi-byte pairs, the structural evidence is strong and should increase the candidate's ranking relative to single-byte alternatives whose bigram models may score higher on small samples. Note: boosted confidence values may exceed 1.0 and are used only for relative ranking among candidates. ``run_pipeline`` clamps all confidence values to [0.0, 1.0] before returning to callers. c3B># UHupUT;dM TUv M g7fNr|)r}r_sc enc_lookups rur/_score_structural_candidates..Gs&*;YTtz?Q 4*;s  c3J# UHoR(aMUv M g7fr)r)r}es rurrJsJ#3a>>#3s# #r!gffffff?rcUR$r)rxs ru._score_structural_candidates..[sq||rwTkeyreverse) rrrlistrrrrrr rr sort) rstructural_scoresrrrvalid_mb single_byteresultsboostedrcoveragers @ru_score_structural_candidatesr-s0,++a~~  ++J*;HJ#3JJK#D*CH*C{*CDEG&(G ;<::3??&&qzz373 t  NNZZ ||q8|<ZZ  NN1  LL+TL: N1+s D!D!c[U5S:aUSRb[USRU5(aUSRnUSSHlnUR[;dMUVs/sHoDRU:wdMXCLdMUPM nnUVs/sHoDRU:XdMUPM nnU/UQUQs $ U$s snfs snf)aDemote niche Latin encodings when no distinguishing bytes are present. Some bigram models (e.g. iso-8859-10, iso-8859-14, windows-1254) can win on data that contains only bytes shared with common Western Latin encodings. When there is no byte-level evidence for the winning encoding, promote the first common Western Latin candidate to the top and push the demoted encoding to last. rrN)rrrr&)rrdemoted_encodingrrothersdemoted_entriess ru_demote_niche_latinr_s G q AJ   + 71:.. 5 5"1:..Azz44&&!**8H*HAQZAw/6"XgGW9W1g"X5F5_55  N #Ys/CC CC /C cU(aUSRS:waU$[S[U55S5nUcU$[SU55(a/Xn[U5VVs/sHupEXB:wdM UPM nnnU/UQ$U$s snnf)agPromote KOI8-T over KOI8-R when Tajik-specific bytes are present. KOI8-T and KOI8-R share the entire 0xC0-0xFF Cyrillic letter block, making statistical discrimination difficult. However, KOI8-T maps 12 bytes in 0x80-0xBF to Tajik-specific Cyrillic letters where KOI8-R has box-drawing characters. If any of these bytes appear, KOI8-T is the better match. rzkoi8-rc3N# UHupURS:XdMUv M g7f)zkoi8-tN)r)r}irs rur!_promote_koi8t..s!Q$6DA!**:Paa$6s% %Nc3B# UHoS:dM U[;v M g7frz)rj)r}r~s rurrs A1D &1& &s )rnext enumerater)rr koi8t_idx koi8t_resultrrrs ru_promote_koi8tr{s gaj))X5QIg$6QSWXI A AAA) )' 2E 2an! 2E&v&& NFs ' B6Bic|US:XaU$URUSS9RSSS9$![[4a gf=f)a<Decode data from encoding and re-encode as UTF-8 for language scoring. Returns None if the encoding is unknown. For UTF-8, returns data as-is. Uses ``errors="ignore"`` because the data already passed byte-validity filtering for the detected encoding; any residual invalid bytes are irrelevant for language scoring. utf-8ignore)errors surrogatepassN)decodeencode LookupError TypeError)rrs ru_to_utf8rsY7 {{8H{5<< O=    #s (;;c b/nSnSnUGH!nURcURb[UR5nUcEU(a>[UR5(a$Uc [ U5n[ XRUS9upvUc^U(aW[S5(aG[ XR5nU(a+UbURS:wa [ U5n[ USUS9upvUb1UR[URURUS95 GMURU5 GM$ U$)aFill in language for results missing it. Tier 1: single-language encodings via hardcoded map (instant). Tier 2: multi-language encodings via statistical bigram scoring (lazy). Tier 3: decode to UTF-8, score against UTF-8 language models (universal fallback). N)profilerr) r rrrrr rrr r) rrfilledr utf8_profileresultlang_ utf8_datas ru_fill_languagers %'F$(G)-L ?? "v'B!&//2D|);FOO)L)L?+D1G-dOOWU|);G)D)D$T??; #+v'/I'4Y'? 1!7LGA #!'#)#4#4!% f78 MrwcD[X5n[X5n[X5$)zGApply confusion resolution, niche Latin demotion, and KOI8-T promotion.)rrr)rrs ru_postprocess_resultsrs# 't5G!$0G $ ((rwr"rinclude_encodingsexclude_encodingsno_match_encodingempty_input_encodingc[5nUSUn[XU5n[SU55n U(d [XiS5$[ U5n U bU R U ;aU /$[ U5n U bU R U ;aU /$[U5n U b U R bU R U ;aU /$[U5n [U5nU cUc[XS9(a[/$[U5nUbUR U ;aU/$UbUR U ;aU/$U bU R U ;aU /$[X5nU(d [XYS5$[UUU5nU(d [XYS5$/nUHpnUR(dMUR R#UR$5nUc ['UUU5nUS:dMSUR)UR$U45 Mr U(a=UR+SSS 9 US unnU[,:a[/UUUU5n[1UU5$[3[5U[7U555nU(d [XYS5$[1UU5$) zBCore pipeline logic. Returns list of results sorted by confidence.Nc38# UHoRv M g7fr)r)r}rs rur%_run_pipeline_core..s'GJSJsr) max_bytesrr!c US$)Nrr|rs rur$_run_pipeline_core..TsQqTrwTrr)r r frozensetrvrrrrrrr_BINARY_RESULTrrrrrrrrrr _STRUCTURAL_CONFIDENCE_THRESHOLDrrrrr)r encoding_erarrrrrr candidatesrs bom_resultutf1632_result escape_result utf8_precheckascii_precheck markup_resultrrrscorer best_scorers ru_run_pipeline_corers  C   D  ARSJ''GJ'GGG % +A  D!J*"5"5"@| -T2N!n&=&=&H +40M!  " " .  " "g - %M"$'N   " d 0 *$/M ]%;%;w%F!n&=&=&H ]%;%;w%F*$; %&7BUVV,D2BCH %&7BUVV24    MM%%chh/E}0sC@s{!((#((E):;  >4@)!, : 9 92')93G(g6 6#D%0@*ABCG %&7BUVV g ..rwc ([UUUUUUUS9n[US[U5nU(d Sn[U5eUV s/sHJn U RS:a5[ U R [U RS5U R5OU PML sn $s sn f)a1Run the full detection pipeline. :param data: The raw byte data to analyze. :param encoding_era: Filter candidates to a specific era of encodings. :param max_bytes: Maximum number of bytes to process. :param include_encodings: If not ``None``, only return these encodings. :param exclude_encodings: If not ``None``, never return these encodings. :param no_match_encoding: Encoding returned when no candidate survives. :param empty_input_encoding: Encoding returned for empty input. :returns: A list of :class:`DetectionResult` sorted by confidence descending. rNz/pipeline must always return at least one resultg?) rr_LANG_SCORE_MAX_BYTES RuntimeErrorrr rminr ) rrrrrrrrmsgrs ru run_pipelinerds*! +++1GT"8#897CG ?3  A <<#   C c$:AJJG     sAB)rstrrsr%rtrreturnlist[DetectionResult])rrrbytesrbool)rrrtuple[EncodingInfo, ...]rr rr) rrrzlist[tuple[str, float]]rrrr rr)rrrrrr)rrrrrz bytes | None)rrrrrintrfrozenset[str] | Nonerrrrrrrr)I__doc__ __future__rrochardet._utilsr chardet.enumsrchardet.modelsrrrr chardet.pipeliner r r r chardet.pipeline.asciirchardet.pipeline.binaryrchardet.pipeline.bomrchardet.pipeline.confusionrchardet.pipeline.escaperchardet.pipeline.markuprchardet.pipeline.statisticalrchardet.pipeline.structuralrrrchardet.pipeline.utf8rchardet.pipeline.utf1632rchardet.pipeline.validityrchardet.registryrrrrrrrr&__annotations__rUr\r_rarjrvrrrrrrrrrrrrrrrrr|rwrurs6F",%  0-+?:99 .<89 6tL $( +4+/8/1/^1n/8 "/^"T09(0n.-*3/*3L* PP PP P* B$   "3 3.3 3 3l/ /.///  /  /d  "8  ":$( (/((V) ) "))'/ 04/3% '/ /// - / - ////J', 04/3% ', ,,, - , - ,,,,rw