޺iP' :%SrSSKrSSKrSSKrSSKrSSKrSSKJrJ r \R"S5Rr \R"S5RrSr\\S'0r\\\4\S'\R*"5H7r\"\R05S :XdM\R0S\\R2'M9 S \S \\\\4\\\444S jr\R>S \\\\4\\\444S j5r S \\\44Sjr!S\\\4S \\\"\\S-\\444Sjr#\R>S \\\"\\S-\\444Sj5r$S\S \S-4Sjr%S\S \&4Sjr'S \\\44Sjr("SS5r)SS\)S\S\S \4Sjjr*SS \S\S\)S-S \\\S-44Sjjr+g)zModel loading and bigram scoring utilities. Note: ``from __future__ import annotations`` is intentionally omitted because this module is compiled with mypyc, which does not support PEP 563 string annotations. N)REGISTRYlookup_encodingz>Iz>BBBNON_ASCII_BIGRAM_WEIGHT_SINGLE_LANG_MAPdatareturnc0n0n[Rn[n[nSnU"X5unUS- nUS:aSUS3n[ U5e[ U5Hn U"X5un US- nU S:aSU S3n[ U5eXXj-R S 5n Xj- nU"X5un US- nU S :aS U S 3n[ U5e[S 5n SnU S -nXXo-n[U5U:waSU <3n[ U5eXo- nU"U5HunnnUU US-U-'UUU-- nM XU 'U"U5X+'M X4$![R[4anSU3n[ U5UeSnAff=f)zParse the binary models.bin format into model tables and L2 norms. :param data: Raw bytes of models.bin (must be non-empty). :returns: A ``(models, norms)`` tuple. :raises ValueError: If the data is corrupt or truncated. ri'z"corrupt models.bin: num_encodings=z exceeds limitzcorrupt models.bin: name_len=z exceeds 256zutf-8z corrupt models.bin: num_entries=z exceeds 65536z-corrupt models.bin: truncated entry data for rzcorrupt models.bin: N) mathsqrt_unpack_uint32 _iter_3bytes ValueErrorrangedecode bytearraylenstructerrorUnicodeDecodeError)r modelsnorms_sqrt _unpack_u32 _iter_bbboffset num_encodingsmsg_name_lenname num_entriestablesq_sumexpected_byteschunkb1b2weightes V/var/www/piano.thomer.com/venv/lib/python3.13/site-packages/chardet/models/__init__.py_parse_models_binr1s$&F E IIE KI&%&t4!  6 !6}o^TCS/ !}%A%d3KX aKF#~5hZ|L o%!23::7CD  F(6N[ aKFU"8 ^T o%e$EF(1_N&"9:E5z^+EdXN o%  $F"+E"2B(.rQw"n%&6/)#3!4L-EK5&> = LL, -%$QC(o1$%sDD44E$EE$c[RRS5RS5nUR 5nU(d[ R "S[SS9 004$[U5$)zcLoad and parse models.bin, returning (models, norms). Cached: only reads from disk on first call. zchardet.modelsz models.binuXchardet models.bin is empty — statistical detection disabled; reinstall chardet to fix) stacklevel) importlib resourcesfilesjoinpath read_byteswarningswarnRuntimeWarningr1)refr s r0_load_models_datar>Wsb    # #$4 5 > >| LC >> D   '   2v T ""c[5S$)zLoad all bigram models from the bundled models.bin file. Each model is a bytearray of length 65536 (256*256). Index: (b1 << 8) | b2 -> weight (0-255). :returns: A dict mapping model key strings to 65536-byte lookup tables. rr>r?r0 load_modelsrCls  q !!r?rc0nUR5H<up#URSS5upEURU/5RXCU45 M> [ U5H n[ U5nUcMXq;dMXX'M" U$)zBuild a grouped index from a models dict. :param models: Mapping of ``"lang/encoding"`` keys to 65536-byte tables. :returns: Mapping of encoding name to ``[(lang, model, model_key), ...]``. /r)itemssplit setdefaultappendlistr)rindexkeymodellangencenc_name canonicals r0_build_enc_indexrRwsACElln IIc1%  b!(($s);<% K#H-  Y%;$E  Lr?c([[55$)zTReturn a pre-grouped index mapping encoding name -> [(lang, model, model_key), ...].)rRrCrBr?r0 get_enc_indexrTs KM **r?encodingc,[RU5$)zReturn the language for a single-language encoding, or None. :param encoding: The canonical encoding name. :returns: An ISO 639-1 language code, or ``None`` if the encoding is multi-language. )rgetrUs r0infer_languagerYs    ))r?cU[5;$)zReturn True if the encoding has language variants in the model index. :param encoding: The canonical encoding name. :returns: ``True`` if bigram models exist for this encoding. )rTrXs r0has_model_variantsr[s } &&r?c[5S$)zAReturn cached L2 norms for all models, keyed by model key string.rrArBr?r0_get_model_normsr]s  q !!r?cT\rSrSrSrSrS\SS4Sjr\S\ \ \ 4SS4S j5r Sr g) BigramProfileaPre-computed bigram frequency distribution for a data sample. Computing this once and reusing it across all models reduces per-model scoring from O(n) to O(distinct_bigrams). Stores a single ``weighted_freq`` dict mapping bigram index to *count * weight* (weight is 8 for non-ASCII bigrams, 1 otherwise). This pre-multiplies the weight during construction so the scoring inner loop only needs a single dict traversal with no branching. ) input_norm weight_sum weighted_freqr r Nc[U5S- nUS::a0UlSUlSUlg0nSn[nUR n[ U5HInXnXS-n US-U -n US:dU S:aU"U S5U-X:'XE- nM6U"U S5S-X:'US- nMK X0lX@l[R"[SUR5555Ulg)zbCompute the bigram frequency distribution for *data*. :param data: The raw byte data to profile. rrNrc3*# UH oU-v M g7fNrB.0vs r0 )BigramProfile.__init__..s'E}!A}) rrcrbrarrWrrrsumvalues) selfr total_bigramsfreqw_sumhi_w_getir,r-idxs r0__init__BigramProfile.__init__s D A A 13D #$DO%(DO !&xx}%AB!eB7b.CDyBI aL4/   aL1,  &"))C'Et{{}'E$EFr?rccU"S5nXl[UR55Ul[R "[SUR5555UlU$)a#Create a BigramProfile from pre-computed weighted frequencies. Computes ``weight_sum`` and ``input_norm`` from *weighted_freq* to ensure consistency between the three fields. :param weighted_freq: Mapping of bigram index to weighted count. :returns: A new :class:`BigramProfile` instance. r?c3*# UH oU-v M g7frhrBris r0rl3BigramProfile.from_weighted_freq..s*Q:PQq5:Prn)rcrorprbrrra)clsrcprofiles r0from_weighted_freq BigramProfile.from_weighted_freqsSc( - !5!5!78!YYs*Q-:N:N:P*Q'QRr?) __name__ __module__ __qualname____firstlineno____doc__ __slots__bytesry classmethoddictintr__static_attributes__rBr?r0r_r_sN >IGUGtG< tCH~ /  r?r_rrM model_keyc|URS:Xag[5nU(aURU5OSnUc>Sn[S5HnXnU(dMXWU-- nM [R "U5nUS:XagSnUR R5HupXU U -- nM XUR-- $)zSScore a pre-computed bigram profile against a single model using cosine similarity.reNrr)rar]rWrrrrcrF) rrMrr model_normr)rwrkdotrxwcounts r0score_with_profilersS   E)29%JuAAqa%YYv& S C,,224  SzF""5 w111 22r?cU(dUcg[5nURU5nUcgUc [U5nSnSnUHupxn [X(U 5n X:dMU nUnM XV4$)ahScore data against all language variants of an encoding. Returns (best_score, best_language). Uses a pre-grouped index for O(L) lookup where L is the number of language variants for the encoding. If *profile* is provided, it is reused instead of recomputing the bigram frequency distribution from *data*. :param data: The raw byte data to score. :param encoding: The canonical encoding name to match against. :param profile: Optional pre-computed :class:`BigramProfile` to reuse. :returns: A ``(score, language)`` tuple with the best cosine-similarity score and the corresponding language code (or ``None``). N)reNre)rTrWr_r) r rUrrKvariants best_score best_langrNrMrss r0score_best_languagersz& GO OEyy"H%J I"*Y wy 9 >JI #+   r?)rh),r functoolsimportlib.resourcesr5rrr:chardet.registryrrStruct unpack_fromr iter_unpackrrr__annotations__rrstrrp_encr languagesr&rtuplerfloatr1cacher>rCrJrRrTrYboolr[r]r_rrrBr?r0rsV 6t$00}}V$00  ! $&$sCx.% OO D 4>>a&*nnQ&7#  6 6 4Y c5j!1 126r #5c9n!5tCJ7G!GH##("T#y.)" i  #tE#*i456 67. +tCeC$J 3,F&G!HHI++ *S*S4Z*'''"$sEz*" ::|@B3 3#,39<3 34%)&! &!&!T !&! 5#*  &!r?