i$1 ddlZddlmZddlmZddlmZmZmZm Z m Z ddl Z ddl m Z ddlmZddlmZdd lmZmZdd lmZdd lmZdd lmZdd lmZddlmZmZddl m!Z!ddlm"Z"m#Z#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddl+m,Z,ddl-m.Z.ddl/m0Z0dZ1d7de e2fdZ3Gdde"Z4GddeZ5Gdd eZ6e6jod!d"d#ged$d$d%d&id'd(d(dd)*d+ed,e d-e2d.e8d/e8d0e ef d1Z9ed2gd3Z:d8d4Z;d5Zy):N) namedtuple)Path)AnyCallableDictOptionalUnion)Model)util)Errors) BaseDefaultsLanguage) Morphologizer)DEFAULT_MORPH_MODEL)Scorer)POS)Doc MorphAnalysis)validate_examples)DummyTokenizerload_config_from_strregistry)Vocab) STOP_WORDS)SYNTAX_ITERATORS)TAG_BIGRAM_MAP)TAG_MAP) TAG_ORTH_MAPzU [nlp] [nlp.tokenizer] @tokenizers = "spacy.ja.JapaneseTokenizer" split_mode = null split_modecfd}|S)Nc2t|jS)Nr!)JapaneseTokenizervocab)nlpr!s g/var/www/vps2.regionflexible.com/Desarrollo/venv/lib/python3.12/site-packages/spacy/lang/ja/__init__.pyjapanese_tokenizer_factoryz4create_tokenizer..japanese_tokenizer_factory$s zBB)r!r)s` r(create_tokenizerr,#sC &%r*ceZdZddedeeddfdZdZdedefdZ dd e fd Z d Z d Z deeeffd ZifdeeefddfdZdefdZdeddfdZdeeefddfdZdeeefddfdZy)r%Nr&r!returncv||_||_t|j|_|duxs|dk( |_y)NA)r&r!try_sudachi_import tokenizerneed_subtokens)selfr&r!s r(__init__zJapaneseTokenizer.__init__+s: $+DOO<#-#5#Js9JKr*c>t|j|jffSN)r%r&r!r4s r( __reduce__zJapaneseTokenizer.__reduce__2s 4::t"???r*textc r|jj|}|j|}t||\}}|rt |nggdz\}}}}} } } t | } t |j||} d} tt | |D] \}\}}|j|_ | r | |_ d} nBt|j|j|dzt|kr||dznd\|_ } |jr |jn |j |_i}|j$r|j$|d<|j&|_|j*r$t-j.dd|j*|d<t1|j||_ |j4r| | j6d<| S) N)wordsspacesr Inflectionz[=|]_Reading sub_tokens)r2tokenize _get_dtokensget_dtokens_and_spacesziplistrr& enumeratetagtag_pos resolve_posorth_lenlemmasurfacelemma_infnormnorm_readingresubrmorphr3 user_data)r4r:sudachipy_tokensdtokensr>r=tags inflectionslemmasnormsreadingssub_tokens_listdocnext_posidxtokendtokenrXs r(__call__zJapaneseTokenizer.__call__5s>>2248##$450$?%CM2$( Kt[&%?/$**E&9$-c#w.?$@ ; C%EJ$ &1KKJJ%(1Ws4y%8DqMd'# 8 ,2<<6<|r|j|nd}t|Dcgc]\}}t|jdkDrt |jdj |j ddDcgc] }|dk7s | c}dj |j ddDcgc] }|dk7s | c}|j|j|j|r||nd}}}}t|Dcgc]j\}}|dk(s^|jjrD|jdk7s5||dz jjr||dz jdk7r|lc}}Scc}wcc}wcc}}}wcc}}w)Nr-*;空白r) _get_sub_tokensrHrNrP DetailedTokenjoinpart_of_speechdictionary_formnormalized_form reading_formisspacerI) r4rZrhrardrexxr[ts r(rDzJapaneseTokenizer._get_dtokens]s6ED !1 24 ((89  U5==?#a'  u';';'=bq'AOR3Y"OPu';';'=ab'AOR3Y"OP%%'%%'""$,;OC(   ($G, Qax99$$&uu 37#++335sQw##x/   PO  $ s8AF= FF &F2 F =F AFA/F Fcv|jsyg}|D]#}|j|jjj}t |dk(r|j dS|jdk(r#|j |j|dg|j|jjj}t |t |k(r&|j|d}|j ||g|j |j|d|j|dg&|S)NrBF) r3splitr2 SplitModer0rNappendr!rDrz)r4rZraresub_asub_br[s r(roz!JapaneseTokenizer._get_sub_tokens}s""% EKK 8 8 : :;E5zQ&&t,C'&&(9(9%(G'HI DNN$<$<$>$>?u:U+"//u=G#**GW+=>#** --eU; --eU; $r*cDt|dtj|S)NzJapaneseTokenizer.score)rrscore_tokenization)r4exampless r(scorezJapaneseTokenizer.scores($=>((22r*cd|jiSNr!r$r8s r( _get_configzJapaneseTokenizer._get_configsdoo..r*configc2|jdd|_yr)getr!)r4rs r( _set_configzJapaneseTokenizer._set_configs **\48r*c >dfdi}tj|gS)NcfgcJtjjSr7)srsly json_dumpsrr8sr(z,JapaneseTokenizer.to_bytes..se&6&6t7G7G7I&Jr*)r to_bytes)r4kwargs serializerss` r(rzJapaneseTokenizer.to_bytessJK }}["--r*datac xdfdi}tj||gtj_S)NrcLjtj|Sr7)rr json_loads)br4s r(rz.JapaneseTokenizer.from_bytes..s$*:*:5;K;KA;N*Or*)r from_bytesr1r!r2)r4rr deserializerss` r(rzJapaneseTokenizer.from_bytess3 OP  mR0+DOO< r*pathc ltj|}dfdi}tj||gy)NrcLtj|jSr7)r write_jsonrpr4s r(rz+JapaneseTokenizer.to_disk..s(8(8D.s(8(89K(Lr*)r r from_diskr1r!r2rs` r(rzJapaneseTokenizer.from_disksC%LM  t["-+DOO< r*r7)T)__name__ __module__ __qualname__rrstrr5r9rrgboolrDrorrrrrbytesrrr rrrr+r*r(r%r%*sLeL#L$L@&S&S&P d @43/T#s(^/469$sCx.9$9.E.u3F ,E#t),,4, eCI.=Pr*r%c0eZdZeeZeZeZ ddddZ y)JapaneseDefaultsltrF) directionhas_case has_lettersN) rrrrDEFAULT_CONFIGrr stop_wordsrsyntax_iteratorswriting_systemr+r*r(rrs# !. 1FJ'#(eERNr*rceZdZdZeZy)JapanesejaN)rrrlangrDefaultsr+r*r(rrs DHr*r morphologizerz token.morphz token.posTz@scorerszspacy.morphologizer_scorer.v1)model overwriteextendscorerg?)pos_acc morph_micro_fmorph_per_feat)assignsdefault_configdefault_score_weightsr'rnamerrrc8t|j|||||S)N)rrr)rr&)r'rrrrrs r(make_morphologizerrs".  5$)F6 r*rp)rPrIrRrOrSrUrBc ddlm}m}|jjj |jjj |jjj |jjjd|}|jj|}|S#t$r tddwxYw)zSudachiPy is required for Japanese support, so check for it. It it's not available blow up and explain how to fix it. split_mode should be one of these values: "A", "B", "C", None->"A".r) dictionaryr2)Nr0rzC)modezJapanese support requires SudachiPy and SudachiDict-core (https://github.com/WorksApplications/SudachiPy). Install with `pip install sudachipy sudachidict_core` or install spaCy with `pip install spacy[ja]`.N) sudachipyrr2 Tokenizerr|r0rzr Dictionarycreate ImportError)r!rr2toks r(r1r1s3%%//11$$..00$$..00$$..00    ##%,,*,=  :   s B+B..Cc|tvrt|}||vr||dfS|r0||f}|tvr$t|\}}|t|t|fS||fSt|tdfS)a2If necessary, add a field to the POS tag for UD mapping. Under Universal Dependencies, sometimes the same Unidic POS tag can be mapped differently depending on the literal token or its context in the sentence. This function returns resolved POSs for both token and next_token by tuple. N)r rrr)orthrInext_tagorth_map tag_bigram current_posrcs r(rLrLs l$ 8 D>4' '(]  '$2:$> !K"CL% #H,, 3< d ""r*c |Dcgc]}|j}}djdj|jdj|jk7r*ttj j ||g}g}d}t|dk(r||fSt|Dcgc]}|jr|c}dk(r+|jsJt||d||ddg}dg}||fStt||D]\} \}} |jr ||dj|} | dkDr>|||| z} |jt| |d| | dd|jd|| z }|j| |jd|t|z }| dzt|ks|| dzjdk(sd|d<|dz }|t|kr6||d} |jt| |d| | dd|jd||fScc}wcc}w#t$r,ttj j ||dwxYw) N)r:r=rFr T)rPrqr{ ValueErrorr E194formatrNrvrprHrFindexr}) r[r:gap_tagxr= text_dtokens text_spacestext_poswordirf word_startws r(rErE sy ' (1QYY (E ( wwrwwu~##%&"''$**,*??++U+CDDLKH 5zQ[(( u ;tDLLNd ; < A||~~%dGRtT4PQ g [(('s5'':;>D& <<>  Shi..t4J >X: 56A    a"aD$ O P   u %  "H F#5!CI q53w< GAEN$:$:c$A"KO MH/4#d) OM!Wb!QdKL5!  $$a )< SV[[//T/GHd R SsH53H: H:2H??5I4r7)r0)rn)?rV collectionsrpathlibrtypingrrrrr r thinc.apir rr errorsr languagerrpipelinerpipeline.morphologizerrrrsymbolsrtokensrrtrainingrrrrr&rrrrrtag_bigram_maprtag_mapr tag_orth_mapr rrr,r%rrfactoryrrrpr1rLrE__all__r+r*r(rsL "77 .%9()BB".*&&#&KK\S|S x   K ($>?            X   X 0#D2%j ,r*