޺i%%SrSSKJr SSKrSSKrSSKrSSKrSSKJ r J r J r J r SSK Jr SSKJr \\\\4\\\\\\\\4444r0SS_SS _S S _S S _SS_SS_SS_SS_SS_SS_SS_SS_SS_S S!_S"S#_S$S%_S&S'_S(S)S*S+S,S-S.S/S0S1S2S3S4S5. ErS6\S7'\R35VVs0sHupX_M snnrS8\S9'SCS:jr\R8SDS;j5r0SS_S S_S S_S S_SS_SS_SS_SS_SS_SS_S!S_S#S_S%S_S'S_S(S_S*S_S)S_SSS S S S SSSSSSSS<. ErS8\S='SES>jrSFS?jr SGS@jr!SHSAjr"SISBjr#gs snnf)JayConfusion group resolution for similar single-byte encodings. At runtime, loads pre-computed distinguishing byte maps from confusion.bin and uses them to resolve statistical scoring ties between similar encodings. Build-time computation (``compute_confusion_groups``, ``compute_distinguishing_maps``, ``serialize_confusion_data``) lives in ``scripts/confusion_training.py``. ) annotationsN)NON_ASCII_BIGRAM_WEIGHT BigramProfile get_enc_indexscore_with_profile)DetectionResult)lookup_encodingLuLlLtLmLoMnMcMeNd Nl No Pc Pd PsPePiPfPoSmScSkSoZsZlZpCcCfCsCoCn) zdict[int, str]_INT_TO_CATEGORYzdict[str, int]_CATEGORY_TO_INTc0nSn[R"SX5unUS- n[U5GHn[R"SX5unUS- nXX%-RS5nX%- n[R"SX5unUS- nXX'-RS5nX'- n[R"SX5un US- n/n 0n [U 5Han[R"SX5upnUS- nU R U 5 [ R U S 5[ R US 54X'Mc [U 5U 4XU4'GM U$) zLoad confusion group data from raw bytes. :param data: The raw binary content of a confusion.bin file. :returns: A :data:`DistinguishingMaps` dictionary keyed by encoding pairs. rz!Hr z!Br zutf-8z!BBBrr7)struct unpack_fromrangedecodeappendrEget frozenset)dataresultoffset num_pairs_ name_a_lenname_a name_b_lenname_b num_diffsdiff_bytes_list categoriesbv cat_a_int cat_b_ints Y/var/www/piano.thomer.com/venv/lib/python3.13/site-packages/chardet/pipeline/confusion.py%deserialize_confusion_data_from_bytesr_Hsa "$F F%%dD9LY aKF 9 **4> ! v23::7C**4> ! v23::7C))$= ! %'13 y!A'-'9'9&$'O $B9 aKF  " "2 & $$Y5 $$Y5JN "%.o$> #K 14 Mc[RRS5RS5nUR 5nU(d[ R "S[SS9 0$[U5n0nUR5H5uupgn[U5=(d Un [U5=(d Un XX4'M7 U$![R[4anSU3n[U5UeSnAff=f)zLoad confusion group data from the bundled confusion.bin file. :returns: A :data:`DistinguishingMaps` dictionary keyed by encoding pairs. zchardet.modelsz confusion.binuZchardet confusion.bin is empty — confusion resolution disabled; reinstall chardet to fixr ) stacklevelzcorrupt confusion.bin: N) importlib resourcesfilesjoinpath read_byteswarningswarnRuntimeWarningr_rHerrorUnicodeDecodeError ValueErroritemsr ) refrawraw_mapsemsg normalizedabvaluenorm_anorm_bs r^load_confusion_datarzps    # #$4 5 > > OC .. C   '    %8= &(J!)  #(q #(q',F#$*  LL, -%'s+o1$%s# B;;C+C&&C+) r.r/r0r1r2r4r3r6r5r7rrr_CATEGORY_PREFERENCEcSnSn[U5U-nU(dgUHQnXHup[RU S5n [RU S5n X:a X[U - - nMCX:dMJXlU - - nMS XV:aU$Xe:aU$g)atResolve between two encodings using Unicode category voting. For each distinguishing byte present in the data, compare the Unicode general category under each encoding. The encoding whose interpretation has the higher category preference score gets a vote. The encoding with more votes wins. :param data: The raw byte data to examine. :param enc_a: First encoding name. :param enc_b: Second encoding name. :param diff_bytes: Byte values where the two encodings differ. :param categories: Mapping of byte value to ``(cat_a, cat_b)`` Unicode general category pairs. :returns: The winning encoding name, or ``None`` if tied. rN)rNr{rM) rOenc_aenc_b diff_bytesrZvotes_avotes_brelevantr[cat_acat_bpref_apref_bs r^resolve_by_category_votingrs,GG+H !~ %))%3%))%3 ?  &G _  &G  r`c`^URU5nU(dg[U4SjU55$)zDReturn the best bigram score across all language variants for *enc*.gc3B># UHupn[TX#5v M g7f)N)r).0rSmodel model_keyprofiles r^ &_best_variant_score..s'#+ Ai 7E55#+s)rMmax)rindexencvariantss` r^_best_variant_scorers2 yy~H  #+ r`c[U5S:ag0n[[U5S- 5HMnXnXS-nXc;aXs;aMUS-U-nUS:dUS:a[OSn URUS5U -XH'MO U(dg[R "U5n [ 5n [XU5n [XU5n X:aU$X:aU$g)aResolve between two encodings by re-scoring only distinguishing bigrams. Builds a focused bigram profile containing only bigrams where at least one byte is a distinguishing byte, then scores both encodings against their best language model. :param data: The raw byte data to examine. :param enc_a: First encoding name. :param enc_b: Second encoding name. :param diff_bytes: Byte values where the two encodings differ. :returns: The winning encoding name, or ``None`` if tied. r Nr rr)lenrJrrMrfrom_weighted_freqrr)rOr}r~rfreqib1b2idxweightrrbest_abest_bs r^resolve_by_bigram_rescorers$ 4y1}D 3t9q= ! W a%[ B$8 Qw"n-/$Y"t)(!HHS!$v- " ..t4G OE  7F  7F    r`c,X4U;aX4$X!4U;aX!4$g)zEFind the canonical key for a pair of encodings in the confusion maps.N)mapsr}r~s r^_find_pair_keyrs+ ~~ ~~ r`cZ[U5S:aU$USnUSnURb URcU$[5n[XBRUR5nUcU$XEupgUup[ XXU5n [ XX5n U bU OU n U bXR:XaU$X2/USSQ$)aResolve confusion between similar encodings in the top results. Compares the top two results. If they form a known confusion pair, it determines which encoding should win by checking the resolve_by_bigram_rescore and resolve_by_category_voting tie-breakers and giving precedence to bigram re-scoring when they disagree. :param data: The raw byte data to examine. :param results: Detection results sorted by confidence descending. :returns: A reordered list of :class:`DetectionResult` with the winner first. r rr N)rencodingrzrrr) rOresultstopsecondrpair_keyrrZr}r~ cat_winner bigram_winnerwinners r^resolve_confusion_groupsr's 7|a !*C QZF ||v6  DdLL&//BH!^JLE+DJWJ-d5MM+7]ZF ~<</  &'!"+ &&r`)rObytesreturnDistinguishingMaps)rr) rOrr}strr~rrfrozenset[int]rZzdict[int, tuple[str, str]]r str | None)rrrz2dict[str, list[tuple[str | None, bytearray, str]]]rrrfloat) rOrr}rr~rrrrr)rrr}rr~rrztuple[str, str] | None)rOrrlist[DetectionResult]rr)$__doc__ __future__r functoolsimportlib.resourcesrcrHrhchardet.modelsrrrrchardet.pipelinerchardet.registryr dicttuplerrNintrrE__annotations__rnrFr_cacherzr{rrrrr)kvs00r^rs#  -, #s(O )C.$sE#s(O34 457$t$t$t$t $ t $ t $t$t$t$t$$$$$$ !$"#$$             =$.D6F5K5K5M#N5MTQAD5M#N.N%P >("("( "( ! (  ! (  ! ( !( !( !( !( !( !( !( !( !(  !!(" !#($ =(nD' ' ' ' ' + '  'T    =    , , , , ,  ,^        &' &' "&'&'E$Os-E-