idZddlZddlZddlZddlmZddlmZmZmZddl m Z ddl m Z ddlmZmZmZdd lmZdd lmZmZdd lmZdd lmZmZmZmZmZmZm Z dd l!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'm(Z(ddl)m*Z*e'jVe,Z-Gddej\Z/ ddl0m1Z1e1Z/e-jedGddej\Z6Gddej\Z7Gddej\Z8Gddej\Z9Gddej\Z:Gd d!ej\Z;Gd"d#eZ<Gd$d%ej\Z=e&Gd&d'e"Z>Gd(d)e>Z?e&Gd*d+e>Z@e&d,-Gd.d/e>eZAe&Gd0d1e>ZBe&d2-Gd3d4e>ZCe&Gd5d6e>ZDe&Gd7d8e>ZEgd9ZFy#e3$rY5e4$re-jkdYMwxYw):zPyTorch T5 model.N)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCacheEncoderDecoderCache)GenerationMixin)create_bidirectional_maskcreate_causal_mask)GradientCheckpointingLayer)BaseModelOutput)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput#Seq2SeqQuestionAnsweringModelOutputSeq2SeqSequenceClassifierOutputTokenClassifierOutput)PreTrainedModel) DUMMY_INPUTS DUMMY_MASKauto_docstringloggingtorch_compilable_check)T5Configc&eZdZdfd ZdZxZS) T5LayerNormct|tjt j ||_||_y)zc Construct a layernorm module in the T5 style. No bias and no subtraction of mean. N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizeeps __class__s s/var/www/vps2.regionflexible.com/Desarrollo/venv/lib/python3.12/site-packages/transformers/models/t5/modeling_t5.pyr$zT5LayerNorm.__init__/s1 ll5::k#:; #c|jtjjdj dd}|tj ||j zz}|jjtjtjfvr%|j|jj}|j|zS)NT)keepdim) tor&float32powmeanrsqrtr)r(dtypefloat16bfloat16)r* hidden_statesvariances r.forwardzT5LayerNorm.forward7s !##EMM266q9>>r4>P% Ht?T?T4T(UU  ;;   ? ?),,T[[->->?M{{]**r/)gư>)__name__ __module__ __qualname__r$r> __classcell__r-s@r.r!r!.s $ +r/r!) FusedRMSNormzODiscovered apex.normalization.FusedRMSNorm - will use it instead of T5LayerNormzBdiscovered apex but it failed to load, falling back to T5LayerNormc*eZdZdeffd ZdZxZS)T5DenseActDenseconfigc^t|tj|j|j d|_tj|j |jd|_tj|j|_ t|j|_ yNFbias)r#r$rLineard_modeld_ffwiwoDropout dropout_ratedropoutr dense_act_fnactr*rGr-s r.r$zT5DenseActDense.__init__Usn ))FNNFKKeD))FKKeDzz&"5"56 &--.r/c|j|}|j|}|j|}t|jj t jr|j|jj jk7r`|jj jt jk7r/|j|jj j}|j |}|SN) rOrUrS isinstancerPr(r&Tensorr9int8r4r*r<s r.r>zT5DenseActDense.forward\s . /  ]3 tww~~u|| 4##tww~~';';;$$ 2),,TWW^^-A-ABM . r/r?r@rArr$r>rBrCs@r.rFrFTs/x/ r/rFc*eZdZdeffd ZdZxZS)T5DenseGatedActDenserGct|tj|j|j d|_tj|j|j d|_tj|j |jd|_tj|j|_ t|j|_yrI)r#r$rrLrMrNwi_0wi_1rPrQrRrSr rTrUrVs r.r$zT5DenseGatedActDense.__init__ks IIfnnfkkF IIfnnfkkF ))FKKeDzz&"5"56 &--.r/c,|j|j|}|j|}||z}|j|}t |j j tjr|j|j j jk7r`|j j jtjk7r/|j|j j j}|j |}|SrX) rUrarbrSrYrPr(r&rZr9r[r4)r*r< hidden_gelu hidden_linears r.r>zT5DenseGatedActDense.forwardsshhtyy78  -0 #m3  ]3 tww~~u|| 4##tww~~';';;$$ 2),,TWW^^-A-ABM . r/r]rCs@r.r_r_js/x/r/r_c*eZdZdeffd ZdZxZS) T5LayerFFrGct||jrt||_nt ||_t |j|j|_ tj|j|_ y)Nr,)r#r$ is_gated_actr_DenseReluDenserFr!rMlayer_norm_epsilon layer_normrrQrRrSrVs r.r$zT5LayerFF.__init__s_    "6v">D "1&"9D %fnn&:S:STzz&"5"56 r/cr|j|}|j|}||j|z}|SrX)rmrkrS)r*r<forwarded_statess r.r>zT5LayerFF.forwards=??=9../?@% 5E(FF r/r]rCs@r.rgrgs7x7r/rgc`eZdZ ddededzffd Zed dZd dZ d dZ xZ S) T5AttentionNrG layer_idxct||j|_||_|j|_|j |_|j |_|j|_|j|_ |j|_ |j|jz|_ ||_|9|jr-tj!d|j"j$dt'j(|j |jd|_t'j(|j |jd|_t'j(|j |jd|_t'j(|j|j d|_|jr/t'j2|j|j|_d|_y)NzInstantiating a decoder z without passing `layer_idx` is not recommended and will to errors during the forward call, if caching is used. Please make sure to provide a `layer_idx` when creating this class.FrJ)r#r$ is_decoderhas_relative_attention_biasrelative_attention_num_bucketsrelative_attention_max_distancerMd_kvkey_value_proj_dim num_headsn_headsrRrS inner_dimrrlogger warning_oncer-r?rrLqkvo Embeddingrelative_attention_biasgradient_checkpointingr*rGrurrr-s r.r$zT5Attention.__init__si  +++F(.4.S.S+/5/U/U,~~ "(++'' ** (?(??"     *4>>+B+B*CD,,  4<<eD4<<eD4<<eD4>>4<=max_distance map to the same bucket. All relative positions <=-max_distance map to the same bucket. This should allow for more graceful generalization to longer sequences than the model has been trained on Args: relative_position: an int32 Tensor bidirectional: a boolean - whether the attention is bidirectional num_buckets: an integer max_distance: an integer Returns: a Tensor with the same shape as relative_position, containing int32 values in the range [0, num_buckets) rr1r) r4r&longabsmin zeros_likelogfloatmath full_likewhere)relative_position bidirectional num_buckets max_distancerelative_buckets max_exactis_smallrelative_position_if_larges r._relative_position_bucketz%T5Attention._relative_position_buckets(,  A K !2Q!6 : :5:: F T T  % *; < !&+N>NO`>a!b b  1$ $y0&/ II'--/); <hh|i/0 1Y& ( "UZZ. &" &+YY &8RT_bcTc(d& " EKK2CE_``r/c| |jjj}tj|tj |dddf|z}tj|tj |dddf}||z }|j ||j |j|j}|j|} | jgdjd} | S)z%Compute binned relative position biasN)r9device)rrr)r1rrr) rr(rr&arangerrrtrvrwpermute unsqueeze) r* query_length key_lengthrpast_seen_tokenscontext_positionmemory_positionrrelative_position_bucketvaluess r. compute_biaszT5Attention.compute_biass >1188??F << EJJvVWXZ^W^_brr,,zFSTXZ[T[\+.>>#'#A#A #.;;== $B$  --.FG *44Q7 r/c |jdd}g|d|j} ||j|jnd} t | t j r| jn| } |du} |j|j| jdd} d} t |trA|jj|j} | r |j}n|j}n|}| r|n|}| rK|I| rG|j |jj"}|j |jj$}ng|jddd|j}|j'|j|jdd}|j)|j|jdd}|K|j+|||j\}}| r)t |trd|j|j<t j,| |jdd}||jd }|j.s`t j0d| jd|d|f|j2|j4 }|j6r6|j8r*d|_n"|j=|d||j2| }|#|ddddddd|jd f}||z}|}||z }t>j@jC|jEd jG|}t>j@jI||jH|j8 }t j,||}|jddjK}|jLg|d}|jO|}||f}|r||fz}|S)z Self-attention (if key_value_states is None) or attention over source sentence (provided by key_value_states). Nr2rrr1FTr)rr9)rrdim)ptraining)(shaperyget_seq_lengthrrrYr&rZclonerview transposer is_updatedgetcross_attention_cacheself_attention_cachelayerskeysrrrupdatematmulruzerosrr9rr requires_gradrr functionalsoftmaxrtype_asrS contiguousreshaper)r*r<maskkey_value_states position_biaspast_key_valuesoutput_attentionskwargs input_shape hidden_shaperis_cross_attention query_statesrcurr_past_key_valuescurrent_states key_states value_stateskv_shapescoresr causal_maskposition_bias_masked attn_weights attn_outputoutputss r.r>zT5Attention.forwards$))#2. BBbB$*A*AB M\Mh?99$..Ino7ABRTYT`T`7a+113gw.T9vvm,11,?II!QO  o': ;(3377GJ!'6'L'L$'6'K'K$#2 -?)] /"=*-44T^^DIIJ/66t~~FMMLP--cr2PBP8O8OPH/44X>HHANJ66.166x@JJ1aPL*+?+F+FzS_aeaoao+p( L%*_FY*ZAEO..t~~>lJ,@,@A,FG  #))"-J33 %  **1-{1~zJSYS`S`hnhtht! ..4==26M/ $ 1 1NJv}}Wg!2! "1a,Bj.>.>r.B,B#BC - ; ,&&}},,V\\^,DLLVT }},,\T\\TXTaTa,b ll<> !++Aq1<<> )k));;;; ff[)  . /Gr/FN)T )Nr)NNNNF) r?r@rArintr$ staticmethodrrr>rBrCs@r.rqrqsX%* $ , ,: ,D- - ^([r/rqc>eZdZddedzffd Z ddZxZS)T5LayerSelfAttentionNrrct|t||||_t |j |j |_tj|j|_ y)Nrurrri) r#r$rq SelfAttentionr!rMrlrmrrQrRrSrs r.r$zT5LayerSelfAttention.__init__\sT ( 0KW` &fnn&:S:STzz&"5"56 r/c |j|}|j||||||} ||j| dz}|f| ddz} | S)N)rrr use_cacherrr)rmrrS) r*r<attention_maskrrrrrnormed_hidden_statesattention_outputrs r.r>zT5LayerSelfAttention.forwarddsp $}=-- '+/ . & 5Ea5H(II  "%5ab%99r/r)NNNFFr?r@rArr$r>rBrCs@r.rr[s*7SSWZ7r/rc<eZdZddedzffd Z ddZxZS)T5LayerCrossAttentionNrrct|t|d||_t |j |j |_tj|j|_ y)NFrri) r#r$rqEncDecAttentionr!rMrlrmrrQrRrS)r*rGrrr-s r.r$zT5LayerCrossAttention.__init__}sN *6u`ij%fnn&:S:STzz&"5"56 r/c |j|}|j||||||} ||j| dz} | f| ddz} | S)N)rrrrrrr)rmrrS) r*r<rrrrrrrr layer_outputrs r.r>zT5LayerCrossAttention.forwardso $}=// -'+/ 0 %t||4DQ4G'HH /$4QR$88r/rX)NNNFrrCs@r.rr|s&7#*7r/rcFeZdZddedzffd Z ddZxZS)T5BlockNrrcpt||j|_tj|_|j j t||||jr&|j j t|||j j t|y)Nr)rr) r#r$rtr ModuleListlayerappendrrrgrs r.r$zT5Block.__init__s  ++]]_   E`lu v  ?? JJ  3FiP Q )F+,r/c  x|jd|||||| } | d}| dd} |jtjk(rtjtj |j tj|jjdz tj|jj}tj|| |}|jxr|du}|r|jd|||||| }|d}|jtjk(rtjtj |j tj|jjdz tj|jj}tj|| |}| |ddz} |jd|}|jtjk(rtjtj |j tj|jjdz tj|jj}tj|| |}|f}|| zS)Nr)rrrrrri)rmax)rrrrrr2) rr9r&r:risinfanyfinforclamprt)r*r<rrencoder_hidden_statesencoder_attention_maskencoder_decoder_position_biasrrr return_dictrself_attention_outputsattention_outputs clamp_valuedo_cross_attentioncross_attention_outputsrs r.r>zT5Block.forwardsY"/A )'+/ " /q1 2126   %-- /++ M*..0 M//044t; M//044K "KK K<[YM!__R1Fd1R &3djjm!65; /"3 ' #4A6M""emm3#kkKK .224KK 3 34884?KK 3 3488 !& M |Q\ ] !24KAB4O O ' 2}5    %-- /++ M*..0 M//044t; M//044K "KK K<[YM " ' ' r/r) NNNNNNFFTrrCs@r.rrs7 -SSWZ -"#&*J r/rc`eZdZdZdeffd ZdejdejfdZxZ S)T5ClassificationHeadz-Head for sentence-level classification tasks.rGc(t|tj|j|j|_tj |j|_tj|j|j|_ y)N)r) r#r$rrLrMdenserQclassifier_dropoutrS num_labelsout_projrVs r.r$zT5ClassificationHead.__init__sZ YYv~~v~~> zzF$=$=>  &..&2C2CD r/r<returnc|j|}|j|}tj|}|j|}|j |}|SrX)rSrr&tanhrr\s r.r>zT5ClassificationHead.forwardsN ]3  =1  =1  ]3  m4 r/) r?r@rA__doc__rr$r&rZr>rBrCs@r.rrs/7ExE U\\ellr/rcteZdZUeed<dZdZdZdgZdgZ e dZ e jdZdZy ) T5PreTrainedModelrG transformerTrrPcvtjt}tjt}|||d}|S)N)decoder_input_ids input_idsdecoder_attention_mask)r&tensorrr)r*r input_mask dummy_inputss r.rzT5PreTrainedModel.dummy_inputss6LL. \\*- !*"&0 r/cr|jj}t|tr$t j |j |dzy t|ttttfrt j|jj d|dzt|drE|jjs/t j|jj d|dzt|drpt j|j j d||jj"dzzt j$|j j&y y t|t(rft|drYt j|j*j d|dzt j$|j*j&y y t|t,r9t j|j.j d||jj"dzzt|j.dr?|j.j&)t j$|j.j&t j|j0j d||jj"dzzt|j0drA|j0j&*t j$|j0j&y y y t|t2r9t j|j4j d||jj"dzzt|j4dr?|j4j&)t j$|j4j&t j|j6j d||jj8dzzt|j6drA|j6j&*t j$|j6j&y y y t|t:rt j|j<j d||jj"dzzt|j<dr?|j<j&)t j$|j<j&t j|j>j d||jj"dzzt|j>dr?|j>j&)t j$|j>j&t j|j6j d||jj8dzzt|j6drA|j6j&*t j$|j6j&y y y t|t@rP|jj"}|jjB}|jjD}t j|jFj d|||zdzzt j|jHj d||dzzt j|jJj d||dzzt j|jLj d|||zdzz|jNr3t j|jPj d||dzzy y y ) zInitialize the weightsg?g)r7stdlm_head qa_outputs classifierrKN))rGinitializer_factorrYr!init constant_r(T5ModelT5ForConditionalGenerationT5EncoderModelT5ForQuestionAnsweringnormal_sharedhasattrtie_word_embeddingsrrrMzeros_rKT5ForTokenClassificationrrrrrFrOrPrNr_rarbrqrxrzrrrrrur)r*modulefactorrMryr{s r. _init_weightszT5PreTrainedModel._init_weightss6// fk * NN6==&3, 7   0.BX Y  LL--CVc\ Jvy)$++2Q2Q V^^22&3,Ov|, V..55CVPTP[P[PcPchlOlEmn F--223- 8 9v|, V..55CVc\R F--223- 4 5 LL,,3Ft{{GZGZ_cFc%>%@#qr'"$:&!  PQ Q&&'8D'@,O  r/N)r?r@rAr__annotations__base_model_prefixsupports_gradient_checkpointing_can_compile_fullgraph_no_split_modules_keep_in_fp32_modulespropertyrr&no_gradr+r4r/r.r r s` %&*#!" !F U]]_4p4pl!r/r c@eZdZfdZdZ ddZxZS)T5Stackc t||tj|j|j |_|j|_tjt|jDcgc]}t|t|dk(|c}|_ t|j |j|_tj"|j$|_|j)d|_ycc}w)NrrriF)r#r$rr vocab_sizerM embed_tokensrtrrange num_layersrboolblockr!rlfinal_layer_normrQrRrS post_initr)r*rGir-s r.r$zT5Stack.__init__js  LL):):FNNK ++]]]bcictct]u vXYWVa1fQR S v  !,FNN@Y@Y Zzz&"5"56  &+# ws7!Dc||_yrX)rBr*new_embeddingss r.set_input_embeddingszT5Stack.set_input_embeddingszs *r/c ||n|jj}||n|jj}| | n|jj} | | n|jj} |$|"|j rdnd} t d| d| d|&|j} |jd| d}n8||jdd} n"|j rdnd} t d| d| d |jr%|jr|rtjd d }|(|j t d |j|}| \}}|d ur|j st d|d|j rf|rr|p|jjr5tt!|jt!|j}n%t!|j}n|j sd}|jj r6t#|j||t%|tr |j&n|}nt)|j||}d}|j r|t)|j|||}| rdnd}|rdnd}|r|j rdnd}d}d}|j+|}|j,D]_}| r||fz}|||||||||||  }|d}|d}|j r | ||rdnd}|sA||dfz}|j sW||dfz}a|j/|}|j+|}| r||fz}| st1d|||||fDSt3|||||S)Ndecoder_zYou cannot specify both zinput_ids and zinputs_embeds at the same timer2zYou have to specify either z input_ids or inputs_embedszZ`use_cache=True` is incompatible with gradient checkpointing. Setting `use_cache=False`...Fzz"T5Stack.forward..s = s)last_hidden_staterr< attentionscross_attentions)rGrroutput_hidden_statesrrtr0sizerrrr}r~rBis_encoder_decoderr r rrYrrrSrFrGtupler)r*rrrrrQrrrrYrrerr_msg_prefixr batch_size seq_lengthencoder_extended_attention_maskall_hidden_statesall_attentionsall_cross_attentionsrrr< layer_module layer_outputss r.r>zT5Stack.forward}s"+!6IDKK+/??ZN*>*:.HXXvw  "#..*K!r;r?;I  &',,.s3K+/??ZN:>:J-XfWggtuv v  & &4==##p"    ( !_`` --i8M!, J  ?? #LTFRg!hii ??_4;;11&9$DKK8,dkk:Z'O'3$++&FO#O ;; ! !/{{+-o/BC!0 D D$ N7{{+-N +/' ??4@.G{{+5&; / +#7BD0d&7DOOrRV (,% ]3  JJ VL#$58H$H!(%/- /#"3' M*!,M *!,M#8#D0=CTaZ[0\- !/=3C2E!E??+?=QRCSBU+U(; V>--m<  ]3  1]4D D  "#%"(    9+++%1   r/) NNNNNNNNNN)r?r@rAr$rMr>rBrCs@r.r?r?is3, + "#![ r/r?ceZdZdgZdddZdeffd ZdZdZe dd e jdzd e jdzd e jdzd e jdzd eee jdzdedzde j dzde j dzdedzdedzdedzdedzdee jezfdZxZS)rFdecoder.block.0.layer.1.EncDecAttention.relative_attention_bias.weight shared.weightencoder.embed_tokens.weightdecoder.embed_tokens.weightrGct||tj|j|j |_tj|}d|_ d|_ t||_ tj|}d|_ |j|_t||_|j!yNFT)r#r$rrrArMr$copydeepcopyrtrr?encodernum_decoder_layersrDdecoderrHr*rGencoder_configdecoder_configr-s r.r$zT5Model.__init__%s  ll6#4#4fnnE v.$)!#( ~. v.$(!$*$=$=!~.  r/c|jSrXr$r*s r.get_input_embeddingszT5Model.get_input_embeddings6 {{r/c~||_|jj||jj|yrXr$rprMrrrKs r.rMzT5Model.set_input_embeddings9-$  )).9 )).9r/Nrrrrencoder_outputsrrQdecoder_inputs_embedsrrrYrrc D| | n|jj} | | n|jj} ||j|||| | | }nI| rGt |t s7t |dt |dkDr|dndt |dkDr|dnd}|d}|j||||||| | | |  }| s||zSt|j|j|j|j|j|j|j|jS) aV input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) T5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [T5 Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. Example: ```python >>> from transformers import AutoTokenizer, T5Model >>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small") >>> model = T5Model.from_pretrained("google-t5/t5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> decoder_input_ids = tokenizer("Studies show that", return_tensors="pt").input_ids # Batch size 1 >>> # preprocess: Prepend decoder_input_ids with start token which is pad token for T5Model. >>> # This is not needed for torch's T5ForConditionalGeneration as it does this internally using labels arg. >>> decoder_input_ids = model._shift_right(decoder_input_ids) >>> # forward pass >>> outputs = model(input_ids=input_ids, decoder_input_ids=decoder_input_ids) >>> last_hidden_states = outputs.last_hidden_state ```NrrrQrrYrrrr1rVr<rW rrrQrrrrrrYr)rVrdecoder_hidden_statesdecoder_attentionsrXencoder_last_hidden_staterencoder_attentions)rGrrrprYrlenrrrrVrr<rWrX)r*rrrrr~rrQrrrrYrrr<decoder_outputss r.r>zT5Model.forward>sOD"+!6IDKKrBrCs@r.rrsx Q*&(7'6 x": .23759:>BF(,-159!%)-,0#'q ##d*q ))D0q !++d2 q !& 0 04 7 q uU%6%6784? q q ||d*q  %||d2q $;q  $;q #Tkq D[q  u  !$6 6q q r/rz: T5 Model with a `language modeling` head on top. ) custom_introceZdZdgZddddZdeffd ZdZdZe dd e jdzd e jdzd e jdzd e jdzd eee jdzdedzde jdzde jdzde jdzdedzdedzdedzdedzdee jezfdZde jfdZxZS)r rgrh)zlm_head.weightrjrkrGct|||j|_t j |j |j|_tj|}d|_ d|_ t||_ tj|}d|_ |j|_t||_t j"|j|j d|_|j'y)NFTrJ)r#r$rM model_dimrrrAr$rnrortrr?rprqrDrrrLrrHrss r.r$z#T5ForConditionalGeneration.__init__s  ll6#4#4fnnE v.$)!#( ~. v.$(!$*$=$=!~. yy1B1BO  r/c|jSrXrwrxs r.ryz/T5ForConditionalGeneration.get_input_embeddingsrzr/c~||_|jj||jj|yrXr|rKs r.rMz/T5ForConditionalGeneration.set_input_embeddingsr}r/Nrrrrr~rrQrlabelsrrrYrrc | | n|jj} | | n|jj} ||j|||| | | }nI| rGt |t s7t |dt |dkDr|dndt |dkDr|dnd}|d}| |||j| }|j||||||| | | |  }|d}|jjr||jdzz}|j|}d}| ^td }| j|j} ||jd |j!d | jd }| s|f|ddz|z}||f|zS|St#|||j$|j&|j(|j*|j,|j&|j( S) a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) T5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [T5 Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[-100, 0, ..., config.vocab_size - 1]`. All labels set to `-100` are ignored (masked), the loss is only computed for labels in `[0, ..., config.vocab_size]` Examples: ```python >>> from transformers import AutoTokenizer, T5ForConditionalGeneration >>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small") >>> model = T5ForConditionalGeneration.from_pretrained("google-t5/t5-small") >>> # training >>> input_ids = tokenizer("The walks in park", return_tensors="pt").input_ids >>> labels = tokenizer(" cute dog the ", return_tensors="pt").input_ids >>> outputs = model(input_ids=input_ids, labels=labels) >>> loss = outputs.loss >>> logits = outputs.logits >>> # inference >>> input_ids = tokenizer( ... "summarize: studies have shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model.generate(input_ids) >>> print(tokenizer.decode(outputs[0], skip_special_tokens=True)) >>> # studies have shown that owning a dog is good for you. ```Nrrrr1rrrr- ignore_indexr2 losslogitsrrrrXrrr)rGrrrprYrrr4rrscale_decoder_outputsrrrr4rrrZrrr<rWrXrV)r*rrrrr~rrQrrrrrYrrr<rsequence_output lm_logitsrloss_fctoutputs r.r>z"T5ForConditionalGeneration.forwardsR"+!6IDKKrrBrCs@r.r r s Q*&*'6'6 x*: .23759:>=A(,26:>*.!%)-,0#'M ##d*M ))D0M !++d2 M !& 0 04 7 M uU\\23d: M M ((4/M  %0047M   4'M $;M  $;M #TkM D[M u  !O 3!M M ^)ELL)r/r ceZdZddiZdgZdeffd ZdZdZe dd e jdzd e jdzd e jdzd e dzd e dzde dzdee jezfdZxZS)r!rjrhrrrGct||tj|j|j |_|}d|_d|_t||_ |jy)NF) r#r$rrrArMr$rr[r?rprH)r*rGrtr-s r.r$zT5EncoderModel.__init__xsY  ll6#4#4fnnE #( ,1)~.  r/c|jSrXrwrxs r.ryz#T5EncoderModel.get_input_embeddingsrzr/cH||_|jj|yrX)r$rprMrKs r.rMz#T5EncoderModel.set_input_embeddingss$  )).9r/NrrrQrrYrrc h||n|jj}|j||||||}|S)aI input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). Example: ```python >>> from transformers import AutoTokenizer, T5EncoderModel >>> tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small") >>> model = T5EncoderModel.from_pretrained("google-t5/t5-small") >>> input_ids = tokenizer( ... "Studies have been shown that owning a dog is good for you", return_tensors="pt" ... ).input_ids # Batch size 1 >>> outputs = model(input_ids=input_ids) >>> last_hidden_states = outputs.last_hidden_state ```r)rGrrp) r*rrrQrrYrrr~s r.r>zT5EncoderModel.forwardsHD&1%rBrCs@r.r!r!ss7I*4& x :.23726)-,0#',##d*,))D0,((4/ ,  $; , #Tk ,D[, u  !O 3,,r/r!z T5 model with a sequence classification/head on top (a linear layer on top of the pooled output) e.g. for GLUE tasks. c~eZdZdgZdeffd Ze ddejdzdejdzdejdzdejdzd e ejdzd ejdzd ejdzd ejdzd e dzde dzde dzde dzde ezfdZxZS)T5ForSequenceClassificationrgrGct||t||_t ||_|j yrX)r#r$rrrclassification_headrHrVs r.r$z$T5ForSequenceClassification.__init__s5  "6?#7#?  r/Nrrrrr~rQrrrrrYrrc | | n|jj} |d} |$|"td|jj| || t d|j |}|j|||||||| | | |  }|d}|j|jjj|j}ttj|jdj!dk(d|j"\}}}||ddfj%|d |ddd ddf}|j'|}d}||j|j}|jj(|jj*dk(rd |j_nv|jj*dkDrL|j,tj.k(s|j,tj0k(rd |j_nd |j_|jj(d k(rSt3}|jj*dk(r&||j5|j5}n|||}n|jj(d k(rGt7}||j%d |jj*|j%d }n,|jj(d k(rt9}|||}| s|f|ddz}||f|zS|St;|||j<|j>|j@|jB|jD|jF|jH S)a input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) T5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [T5 Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). NFz8Passing input embeddings is currently not supported for If no `decoder_input_ids` or `decoder_inputs_embeds` are passed, `input_ids` cannot be `None`. Please pass either `input_ids` or `decoder_input_ids` or `decoder_inputs_embeds`.) rrrr~rQrrrrYrrrz7All examples must have the same number of tokens.r2 regressionsingle_label_classificationmulti_label_classificationr)%rGrNotImplementedErrorr-r?r0r4req eos_token_idr4rrr&unique_consecutivesumnumelrrr problem_typerr9rrrsqueezerrrrrrrXrrr)r*rrrrr~rQrrrrrYrrrreos_maskr^_r+sentence_representationrrrrs r.r>z#T5ForSequenceClassification.forwardsN`&1%>KbKbJcd   $)>)F  U !% 1 1) < "" )/#9+'"7/!5##  "!*<< 8 89<<_=S=ST  $ $X\\!_ 5 ; ; = B E &5%:%:" A{"1(A+">"C"CJPRT_"`abdfhiai"j))*AB  YYv}}-F{{''/;;))Q./;DKK,[[++a/V\\UZZ5OSYS_S_chclclSl/LDKK,/KDKK,{{''<7"9;;))Q.#FNN$4fnn6FGD#FF3D))-JJ+- B 0F0F GUWY))-II,./Y,F)-)9TGf$ Ev E.#33")"?"?&99$55&-&G&G")"?"?&99  r/r)r?r@rArrr$rr&rrZlistrrEr\rr>rBrCs@r.rrsU+s)s&x.2.259:>:>26:>*.!%)-,0#'A ##d*A  t+A !++d2 A !& 0 04 7 A e//047 A ((4/A  %0047A   4'A $;A  $;A #TkA D[A  0 0A A r/rceZdZdeffd Ze d dejdzdejdzdejdzdejdzdedzd edzd edzd e eje zfd Z xZ S)r(rGc,t|||j|_t||_t j |j|_t j|j|j|_ |jyrX) r#r$rr!rrrQrrSrLr+rrHrVs r.r$z!T5ForTokenClassification.__init__Ssj   ++)&1zz&";";< ))F$6$68I8IJ r/NrrrQrrrYrrc ||n|jj}|j||||||} | d} |j| } |j | } d} |z T5ForTokenClassification.forward^s4&1%rBrCs@r.r(r(Qs x *..2-1&*)-,0#'6 <<$&6  t+6 ||d* 6  t# 6  $; 6 #Tk6 D[6  u|| 4 46 6 r/r(ceZdZdgZdddZdeffd ZdZdZe dd e jdzd e jdzd e jdzd e jdzd eee jdzde jdzde jdzde jdzde jdzdedzdedzdedzdedzdee jezfdZxZS)r"rgrhrirGc$t|||j|_t j |j |j|_tj|}d|_ d|_ t||_ tj|}d|_ |j|_t||_|j"|_t j$|j&|j"|_|j+yrm)r#r$rMrrrrAr$rnrortrr?rprqrDrrrrLr+rrHrss r.r$zT5ForQuestionAnswering.__init__s  ll6#4#4fnnE v.$)!#( ~. v.$(!$*$=$=!~.  ++))F$6$68I8IJ r/c|jSrXrwrxs r.ryz+T5ForQuestionAnswering.get_input_embeddingsrzr/c~||_|jj||jj|yrXr|rKs r.rMz+T5ForQuestionAnswering.set_input_embeddingsr}r/Nrrrrr~start_positions end_positionsrQrrrrYrrc | | n|jj} | | n|jj} ||d} | | | td|j |}| | n|jj} | | n|jj} ||j |||| | | }nI| rGt |ts7t|dt|dkDr|dndt|dkDr|dnd}|d}|j||| d||| | | | }|d}|j|}|jdd \}}|jd j}|jd j}d}||t|jdkDr*|jd j|j }t|jdkDr*|jd j|j }|jd}|j#d|}|j#d|}t%| }|||}|||}||zdz }| s||f|ddz|z}||f|zS|St'||||j(|j*|j,|j.|j0|j*|j, S)az input_ids (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Indices of input sequence tokens in the vocabulary. T5 is a model with relative position embeddings so you should be able to pad the inputs on both the right and the left. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for detail. [What are input IDs?](../glossary#input-ids) To know more on how to prepare `input_ids` for pretraining take a look a [T5 Training](./t5#training). decoder_input_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Indices of decoder input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are decoder input IDs?](../glossary#decoder-input-ids) T5 uses the `pad_token_id` as the starting token for `decoder_input_ids` generation. If `past_key_values` is used, optionally only the last `decoder_input_ids` have to be input (see `past_key_values`). To know more on how to prepare `decoder_input_ids` for pretraining take a look at [T5 Training](./t5#training). decoder_attention_mask (`torch.BoolTensor` of shape `(batch_size, target_sequence_length)`, *optional*): Default behavior: generate a tensor that ignores pad tokens in `decoder_input_ids`. Causal mask will also be used by default. NFrrrrr1rrr2rr) r start_logits end_logitsrrrrXrrr)rGrrr0r4rprYrrrrrsplitrrrZr4rrrrrr<rWrXrV)r*rrrrr~rrrQrrrrYrrr<rrrrr total_loss ignored_indexr start_lossend_lossrs r.r>zT5ForQuestionAnswering.forwards0\&1%)F  U !% 1 1) < !*!6IDKKrBrCs@r.r"r"s*r)s&'6'6 x,: .23759:>=A371526:>!%)-,0#'I ##d*I ))D0I !++d2 I !& 0 04 7 I uU\\23d: I ))D0I ''$.I ((4/I  %0047I $;I  $;I #TkI D[I u  !$G G!I I r/r")r!r rr r"rr()Gr rnrr&rtorch.nnrrrrPrr activationsr cache_utilsr r r generationr masking_utilsrrmodeling_layersrmodeling_outputsrrrrrrrmodeling_utilsrutilsrrrrrconfiguration_t5r get_loggerr?r}Moduler!apex.normalizationrDinfo ImportError ExceptionwarningrFr_rgrqrrrrr r?rr r!rr(r"__all__r=r/r.rsI AA&!CC)J9.^^&   H %+"))+2 Y/K KKabbii,299: $"))D299BBII>W (W t299$^!^!^!Bo o dT T T n x)!2Ox)  x)vD&DDN M "3M  M `C 0C C Lo .o o d }/ Y NNWXYsG##H+HH