i.ddlZddlZddlmZddlmZmZmZmZm Z m Z m Z ddl Z ddl mZddlmZmZddlmZmZddlmZd d lmZd d lmZerdd lmZd Zej<j?d d(de ede de!de!de ededgeeff dZ"ej<j?d d)de e e#efde!de!de!dedgeeff dZ$ej<j?dddde e#efde fdZ%ej<j?d d*de ede!de!dedgeeffd Z&de e#efde efd!Z'Gd"d#Z(Gd$d%Z)Gd&d'Z*y)+N)Path) TYPE_CHECKINGCallableIterableIteratorListOptionalUnion)util)ErrorsWarnings)DocDocBin)Vocab) dont_augment)Example)Languagez.spacyzspacy.Corpus.v1path gold_preproc max_lengthlimit augmenterreturnrc|ttjtjj d|t |||||S)NzLoading corpus from path: %s)rrrr) ValueErrorr E913r loggerdebugCorpus)rrrrrs f/var/www/vps2.regionflexible.com/Desarrollo/venv/lib/python3.12/site-packages/spacy/training/corpus.pycreate_docbin_readerr#sI |%%KK4d;  !  zspacy.JsonlCorpus.v1 min_lengthc t||||S)N)r%rr) JsonlCorpus)rr%rrs r"create_jsonl_readerr(*s t zQV WWr$zspacy.read_labels.v1F)requirer)cht|}|s|jsytj|SN)rexistssrsly read_json)rr)s r" read_labelsr/4s* :D 4;;= ??4  r$zspacy.PlainTextCorpus.v1cT|ttjt|||S)Iterate Example objects from a file or directory of plain text UTF-8 files with one line per doc. path (Path): The directory or filename to read from. min_length (int): Minimum document length (in tokens). Shorter documents will be skipped. Defaults to 0, which indicates no limit. max_length (int): Maximum document length (in tokens). Longer documents will be skipped. Defaults to 0, which indicates no limit. DOCS: https://spacy.io/api/corpus#plaintextcorpus r%r)rr rPlainTextCorpus)rr%rs r"create_plain_text_readerr4>s'" |%% 4J: NNr$ctj|}|js!|jdj |r|gS|}|g}g}t }|D]}t ||vr|jt ||jr|jdjdrV|jr |j|j|jdj |s|j|t|dk(r4tjtj j#|||j%|S)N.r)rformat)r ensure_pathis_dirpartsendswithsetstradd startswithextenditerdirappendlenwarningswarnrW090r8sort)r file_type orig_pathpathslocsseens r" walk_corpusrNTs   D !D ;;=TZZ^44Y?v I FE D 5D  t9   T ::$**R.33C8  [[] LL ( ZZ^ $ $Y / KK   4yA~ hmm** )*LMIIK Kr$ceZdZdZdddddddeeefdeded ed e e d ed dfd Z ddd e e fdZdddeded e fdZdddeed e e fdZdddeed e e fdZdedeeeefd e efdZy)r!a6Iterate Example objects from a file or directory of DocBin (.spacy) formatted data files. path (Path): The directory or filename to read from. gold_preproc (bool): Whether to set up the Example object with gold-standard sentences and tokens for the predictions. Gold preprocessing helps the annotations align to the tokenization, and may result in sequences of more consistent length. However, it may reduce run-time accuracy due to train/test skew. Defaults to False. max_length (int): Maximum document length. Longer documents will be split into sentences, if sentence boundaries are available. Defaults to 0, which indicates no limit. limit (int): Limit corpus to a subset of examples, e.g. for debugging. Defaults to 0, which indicates no limit. augment (Callable[Example, Iterable[Example]]): Optional data augmentation function, to extrapolate additional examples from your annotations. shuffle (bool): Whether to shuffle the examples. DOCS: https://spacy.io/api/corpus rFN)rrrrshufflerrrrrrPrctj||_||_||_||_||nt |_||_yr+) r r9rrrrrrrP)selfrrrrrrPs r"__init__zCorpus.__init__sA$$T* ($ &/&; r$nlprc#rK|j|jt|jt}|j r t |}tj ||jr|j||}n|j||}|D]}|j||D]}|yw)zYield examples from the data. nlp (Language): The current nlp object. YIELDS (Example): The examples. DOCS: https://spacy.io/api/corpus#call N) read_docbinvocabrNr FILE_TYPErPlistrandomrmake_examples_gold_preproc make_examplesr)rRrTref_docsexamplesreal_eg augmented_egs r"__call__zCorpus.__call__s##CII{499i/PQ <<H~H NN8 $   66sHEH))#x8H #G $sG < # "" # #sB5B7 referencec 2|s |jr[tt|j|Dcgc]}|jc}|Dcgc]}t |j c}|St|j|j|Scc}wcc}w)Nwordsspaces)has_unknown_spacesrrrWtextbool whitespace_make_doc)rRrTrbrwords r" _make_examplezCorpus._make_examples 977II1:;499;?HItD!1!12I   3<< 7C C <Is BBreference_docsc#K|D]}t|dk(r|jdk(st||jkr|j||dO|jdsa|jD]\}t|dk(r|jdk(st||jks:|j||j d^yw)NrF SENT_START)rDrrmhas_annotationsentsas_doc)rRrTrnrbref_sents r"r\zCorpus.make_exampless( PI9~"A%Y$//)I((i??)),7 )PH8}) A-X1P"00hoo6GOO P PsA#C&AC,(Cc#K|D]r}|jdr(|jDcgc]}|j}}n|g}|D]/}|j||d}t |j s,|1tycc}ww)NrpT)rqrrrsrmrDx)rRrTrnrbsent ref_sentsrtegs r"r[z!Corpus.make_examples_gold_preprocs( I'' 57@GtT[[]G G&K % ''Xt<rtt9H   Gs%BA;3B2BrWrLc#\Kd}|D]}tj|}|jdjts;t j |}|j|}|D]8}t|s||dz }|jdk\s(||jk\s8yw)z(Yield training examples as example dictsrr6rN) r r9r;r<rXr from_diskget_docsrDr)rRrWrLilocdoc_bindocsdocs r"rVzCorpus.read_docbins  "C""3'Cyy}%%i0 (,,S1''."C3x! Q::?qDJJ! " "s?B,:B,=B,B,&B,)__name__ __module__ __qualname____doc__r r>rintrir rrSrrrarrmrr\r[rrVr$r"r!r!ms02"(,CI     H% "#J#8G+<#* D D*- D=A D  D P P/7} P '  P  /7}  '  """*5d+;"<" #"r$r!c `eZdZdZdZdddddeeeefde de de d d f d Z d d d e e fdZ y )r'acIterate Example objects from a file or directory of jsonl formatted raw text files. path (Path): The directory or filename to read from. min_length (int): Minimum document length (in tokens). Shorter documents will be skipped. Defaults to 0, which indicates no limit. max_length (int): Maximum document length (in tokens). Longer documents will be skipped. Defaults to 0, which indicates no limit. limit (int): Limit corpus to a subset of examples, e.g. for debugging. Defaults to 0, which indicates no limit. DOCS: https://spacy.io/api/corpus#jsonlcorpus jsonlr)rr%rrrr%rrNcbtj||_||_||_||_yr+)r r9rr%rr)rRrrr%rs r"rSzJsonlCorpus.__init__s+$$T* $$ r$rTrc #Kt|jdD]}tj|}|D]}|j |d}|j dk\rt ||j kr?|jdk\rt ||jk\rg|Dcgc]}|j}}|Dcgc]}t|j}}t|t|j||ycc}wcc}ww)zYield examples from the data. nlp (Language): The current nlp object. YIELDS (Example): The example objects. DOCS: https://spacy.io/api/corpus#jsonlcorpus-call z.jsonlrhrrdN)rNrr- read_jsonlrkr%rDrrhrirjrrrW) rRrTr~recordsrecordrwrerfs r"razJsonlCorpus.__call__styy(3 SC&&s+G! Sll6&>2??a'CHt,F__)c#h$//.I-01QVV1E1;>?ad1==1?F?"#s399E&'QRR S S2?sBDC>1D7D5DrrrrrIr r r>rrrSrrrarr$r"r'r'sw I  uS$Y'(        SJS8G+<Sr$r'c ZeZdZdZdZddddeeeefde de dd fd Z d d de e fd Z y )r3r1txtrr2rr%rrNcTtj||_||_||_yr+)r r9rr%r)rRrr%rs r"rSzPlainTextCorpus.__init__,s$$$T* $$r$rTrc#Kt|jdD]}t|d5}|D]}|jd}t |s |j |}|j dk\rt ||j krY|jdk\rt ||jkDrt||j dddy#1swYxYww)zYield examples from the data. nlp (Language): The current nlp object. YIELDS (Example): The example objects. DOCS: https://spacy.io/api/corpus#plaintextcorpus-call z.txtzutf-8)encodingz rN) rNropenrstriprDrkr%rrcopy)rRrTr~frhrs r"razPlainTextCorpus.__call__7styy&1 7CcG, 7 7D;;v.D4y!ll40??a/CHt4N$!__1c#h6P$&c388:66 7 7 7 7 7 7s#'C#"C A?C C#C C#rrr$r"r3r3sg I  %uS$Y'( % %  %  %7J78G+<7r$r3)rrN)rrr)rr)+rZrEpathlibrtypingrrrrrr r r-r errorsr rtokensrrrWraugmentrexamplerlanguagerrXregistryreadersrirr#r>r(r/r4rNr!r'r3rr$r"rs( UUU % !#  ()$(  4.   !  zlHW--. *&-. X 5d# $XXX  X zlHW--. X/X-.;@!eCI&!D!/!12O 4.OOOzlHW--. O3O*eCI&d4j2v"v"r4S4Sn.7.7r$