i3dZddlZddlmZddlZddlmZddlmZmZm Z ddl m Z ddl mZdd lmZmZdd lmZdd lmZdd lmZdd lmZddlmZmZmZddlmZm Z ddl!m"Z"ddl#m$Z$m%Z%m&Z&m'Z'ddl(m)Z)ddl*m+Z+ddl,m-Z-ddl.m/Z/e'j`e1Z2GddejfZ4dejjde6dejjfdZ7 dGd/d0eZ?e%Gd1d2e Z@e%Gd3d4e@ZAGd5d6e@eZBe%d78Gd9d:e@ZCgd;ZDy)=zPyTorch Zamba model.N)Callable)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)initialization)ACT2FN)Cache DynamicCache)GenerationMixin)lazy_load_kernel)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputWithPastCausalLMOutputWithPast SequenceClassifierOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)merge_with_config_defaults)resolve_internal_import)capture_outputs) ZambaConfigcheZdZddeddffd Zdej dej fdZdZxZ S) ZambaRMSNormepsreturnNct|tjt j ||_||_y)z; ZambaRMSNorm is equivalent to T5LayerNorm N)super__init__r Parametertorchonesweightvariance_epsilon)self hidden_sizer" __class__s y/var/www/vps2.regionflexible.com/Desarrollo/venv/lib/python3.12/site-packages/transformers/models/zamba/modeling_zamba.pyr&zZambaRMSNorm.__init__2s1 ll5::k#:; # hidden_statesc"|j}|jtj}|j dj dd}|tj ||jzz}|j|j|zS)NT)keepdim) dtypetor(float32powmeanrsqrtr+r*)r,r1 input_dtypevariances r/forwardzZambaRMSNorm.forward:sy#)) %((7  $$Q',,R,>% Ht?T?T4T(UU {{]--k:::r0c^t|jjd|jS)Nz, eps=)tupler*shaper+)r,s r/ extra_reprzZambaRMSNorm.extra_reprAs* ))*+6$2G2G1HIIr0)gư>) __name__ __module__ __qualname__floatr&r(Tensorr>rB __classcell__r.s@r/r!r!1s7$$$$;U\\;ell;Jr0r!r1n_repr#c|j\}}}}|dk(r|S|dddddddddfj|||||}|j|||z||S)z This is the equivalent of torch.repeat_interleave(x, dim=1, repeats=n_rep). The hidden states go from (batch, num_key_value_heads, seqlen, head_dim) to (batch, num_attention_heads, seqlen, head_dim) rN)rAexpandreshape)r1rJbatchnum_key_value_headsslenhead_dims r/ repeat_kvrRFso 2?1D1D.E h z!!Qa"23::5BUW\^bdlmM  (;e(CT8 TTr0modulequerykeyvalueattention_maskscalingdropoutc t||j}t||j} tj||j dd|z} || |z} t j j| dtjj|j} t j j| ||j} tj| | } | j ddj} | | fS)Nr3rr4)dimr6)ptrainingr)rRnum_key_value_groupsr(matmul transposer functionalsoftmaxr8r7r6rYr] contiguous) rSrTrUrVrWrXrYkwargs key_states value_states attn_weights attn_outputs r/eager_attention_forwardriRs3 ; ;$>&B\B\$\!'-'E'E$ )d2 !'!9!9ii < X>X[_[h[h>hotu ii < X>X[_[h[h>hotu ii < X>X[_[h[h>hotu ii : :T]] JFL^L^ejk r0Nr1rWpast_key_valuesrdr#c |jdd}g|d|j}|j|j|j dd}|j |j|j dd} |j |j|j dd} ||j| | |\} } tj|jjt} | ||| | |f|jsdn |j|jd|\} } | j g|dj#} |j%| } | | fS)Nr4rr3)rYrX)rArQryviewr`rzr{updater get_interfacerl_attn_implementationrir]rwrXrMrcr|)r,r1rmrWr~rd input_shape hidden_shape query_statesrerfattention_interfacerhrgs r/r>zZambaAttention.forwardsk$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST  &'6'='=j,Xa'b $J (?(M(M KK , ,.E) %8      % $}}C$2H2HLL %  % ! \*k));;;;FFH kk+. L((r0N)rCrDrE__doc__rintr&r(rGr rrr@r>rHrIs@r/rkrkks l{lsl.)- #)||#)#) t+ #)  #) +, #) u||U\\D0% 2E2LL M#)r0rkc~eZdZdZdeffd Z d dejdedzfdZ d dedzfdZ d dedzfd Z xZ S) ZambaMambaMixeruE Compute ∆, A, B, C, and D the state space parameters and compute the `contextualized_states`. A, D are input independent (see Mamba paper [1] Section 3.5.2 "Interpretation of A" for why A isn't selective) ∆, B, C are input-dependent (this is a key difference between Mamba and the linear time invariant S4, and is why Mamba is called **selective** state spaces) This module differs from `transformers.models.mamba.modeling_mamba.MambaMixer` in two ways: - Added multi-head: the output of `self.in_proj` is split into `self.n_mamba_heads` heads, and each head undergoes an independent forward pass, identical to the original `MambaMixer`, up until the pre-activations of `self.out_proj`. The pre-activations, coming from different mamba heads, are then concatenated and fed into `self.out_proj`. rlc  t|||_||_|j|_|j |_|j|_|j|jz|_ |j|_ |j|_ |j|jz|_|j|_|j"|_t'j(|j|j|j |j|j|jdz |_|j,|_t0|j,|_|j4|_t'j8|j|jdz|j$|_t'j<t?j@|j|j|j dzz|j|_!t'j<t?j@|j|j|jdz dz|jdzz |_"t'j<t?j@|j|j|_#t?jHd|j dzt>jJdddf}|jM|jdjO}t'j<t?jP|jS|j|jd|_*t'j<t?jV|j|j|_,t'j8|j|j|j$|_-t]da/tat^d da1tat^d da2t]d a3titfd a5tatfdda6tatfdda7tqtjtltdtbtnfa9trsttjwdyy)Nr) in_channels out_channelsrq kernel_sizegroupspaddingr3rpg?r6r4z causal-conv1dcausal_conv1d_updatecausal_conv1d_fnz mamba-ssmz8ops.triton.selective_state_update.selective_state_update) chained_pathselective_scan_fnmamba_inner_fnaqThe fast path is not available because one of `(selective_state_update, selective_scan_fn, causal_conv1d_fn, causal_conv1d_update, mamba_inner_fn)` is None. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d. If you want to use the naive implementation, set `use_mamba_kernels=False` in the model config)|j@|j@gd\} }}|jBdddf| jddz}t)jD|jFjI }|jJ|jJjInd}t)jL|d|f|jN|jP}|rtS|jD]}tU|j|jjVdd|f||d df||d df||||dddf||dddf|jX|| |d df||d j'd}t)jZ||fd}n5t)jL|d|j8|j@f|jN|jP}tS|jD]}t]||||||||jdd||jdd|jX|jI| |||d d \}}t)jZ||fdj}t)jZ||j'dfd}|||j_||j|ja|jdd}|S) Nrr3r4r[r)rdevicer6.T) dt_softplus)delta_softplusreturn_last_state)1rAhas_previous_staterr`rchunksqueezercrMrrr*sizerlayersrm conv_statesrqr unsqueezer(rrrapadrupdate_conv_staterrrsplitrrrexprrFremptyrr6rangerrecurrent_statesrcatrupdate_recurrent_stater)r,r1rrW batch_sizeseq_len_use_precomputed_statesprojected_statesgate conv_weightsrssm_parameters time_stepBCdiscrete_time_steprtime_proj_bias scan_outputsn scan_outputs_ ssm_state ssm_state_contextualized_statess r/cuda_kernels_forwardz$ZambaMambaMixer.cuda_kernels_forward s"/!4!4 GQ!-T!9!nl>]>]!nbimnbn << 6@@AF.33JAwOUUVW]^U_ t%--a0;;= ||A||J(:(:BHRRSTVWX{{))..t{{/A/A/F/Fq/I4;;K]K]KbKbcdKef !0%%b)##DNN3??    M*33B7M)%))Na7a<);;M4>>Z % *t{{7I7I!QPQ'7R*RXZ [ %%!T[[%5%55M $ 7 : :5 A K KB O !-$1N1}GZGZ[]G^F^F`C`4a4k4klm4n$nM]]..}t?T?TWdWjWjkmWn?npq>rs );;JW $])CC'M)R S !-$1N1}GZGZ[]G^F^F`C`4a4k4klm4n$nM) -0H0H0K K  HHT[[%?XgX %NOM) -0H0H0K K &--b$2D2DdFYFY[bcmmnoqrs ,,Qa];mKVVWY[]^++ T00$2E2EtGZGZ[ac  1a#11!T':Y=P=PQSUW=XX\`\m\m tQ ]   ]]334FGYYtzz'') * *YYqD!T1!458J1aQRTUW[K[8\\] '1aD(89AaD!Q>N ? E E GG w 9A"1aAq=1;;AqAIMPXYZ\]_`bcefYfPgPqPqrsuvPwwI,,y':':1a'@'C'CE'JAaQRTUWXjMLcLcdfLghK    Aq!QJ 7 8 9kk,B7 !]TVVAtQ> J!%  ! !!Q ' / / B H R RSTVW X! %$r0c tttttt f}|j rC|r"d|jjjvr td|j|||S|j|||S)NcudazFast Mamba kernels are not available. Make sure to they are installed and that the mamba module is on a CUDA device. lease run 'pip install causal-conv1d>=1.2.0' and 'pip install mamba-ssm', or set use_mamba_kernels=False in the model's config.)rW) rrrrrrrrrtype ValueErrorrr)r,r1rrWrdrs r/r>zZambaMambaMixer.forwards!$ #%68HJ^`n o"   )V4;M;M;T;T;Y;Y-Y i ,,]LYg,h h   ^ \\r0)NN) rCrDrErrr&r(rGr rrr>rHrIs@r/rrsd M{M`^b_%"\\_%9>_%BP%ut|P%d ]54< ]r0rc$eZdZfdZdZxZS)ZambaMLPct|||_|j|_|j|_t j |j|jd|_t j |j|jd|_t j |j|jd|_ t|j|_ yNFrp) r%r&rlr-rrrx gate_projup_proj down_projr hidden_actact_fnr,rlr.s r/r&zZambaMLP.__init__s  !--!'!9!94#3#3T5K5KRWXyy!1!143I3IPUV 4#9#94;K;KRWXV../ r0c|j|j|j||j|z}|Sr)rr rr)r,xrs r/r>zZambaMLP.forwards6NN4;;t~~a/@#ADLLQRO#ST r0)rCrDrEr&r>rHrIs@r/rrs 0r0rc eZdZd dededzffd Z ddejdejdedejdzdedzd e dzd e e d e eje ejejfdzffd ZxZS)ZambaAttentionDecoderLayerNrlrmct|t|||_t ||_t |j|j|_ t |j|j|_ y)Nr") r%r&rk self_attnr feed_forwardr!rr rms_norm_epsinput_layernormr-pre_ff_layernormr}s r/r&z#ZambaAttentionDecoderLayer.__init__s_ ' :$V,+F,H,HfNaNab ,V-?-?VEXEX Yr0r1original_hidden_statesrWr~ use_cacherdr#c tj||gd}|j|}|jd|||||d|\}}|j |}|j |}|S)ac Args: hidden_states (`torch.FloatTensor`): output of previous Mamba layer of shape `(batch, seq_len, embed_dim)` original_hidden_states (`torch.FloatTensor`): word embedding output of shape `(batch, seq_len, embed_dim)`. This is concatenated with `hidden_states` (which is the output of the previous (mamba) layer). The concatenated tensor is then used as input of the pre-attention RMSNorm (see fig. 2 in https://huggingface.co/papers/2405.16712). layer_idx (`int`): layer_idx in the forward pass. Used to distinguish Zamba's tied transformer layers. attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`Cache`, *optional*): cached past key and value projection states use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). r4r)r1rmrWr~r)r( concatenaterrrr) r,r1rrmrWr~rrdrs r/r>z"ZambaAttentionDecoderLayer.forwards2))=:P*QWYZ ,,]; )4>> ')+     q--m< ))-8 r0r)NNF)rCrDrErrr&r(rGr boolrrr@ FloatTensorr>rHrIs@r/rrsZ{ZsTzZ/3(,!&'||'!& ' '  t+ '  '$;'+,' u  %(9(95;L;L(L"MPT"TT U'r0rcreZdZdedeffd Z ddejdejdzdedzdejdzdejdzd edzd e dzd ejdzd ejdzd e e de eje ejejfdzffdZxZS)ZambaMambaDecoderLayerrlrmct|t|||_t |j |j |_||_y)N)rlrmr) r%r&rmambar!r-rrrmr}s r/r&zZambaMambaDecoderLayer.__init__s> $FiH +F,>,>FDWDWX"r0Nr1rrW causal_maskr~r position_idstransformer_hidden_statesrdr#c  t|} | || zn|}|j|}|jd|||d| }| |z}|S)aX Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`Cache`, *optional*): cached past key and value projection states use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). )r1rrWr)rr ) r,r1rrmrWr!r~rr"r#rdresiduals r/r>zZambaMambaDecoderLayer.forwardsn0! :S9^M5 5dq ,,]; "  '()   !=0 r0)NNNNNFNN)rCrDrErrr&r(rGr r LongTensorrrr@rr>rHrIs@r/rrs#{#s#7; $.2+/(,!&049=*||*!& t 3*: *  t+ * \\D( **$;*&&-*$)<<$#6*+,* u  %(9(95;L;L(L"MPT"TT U*r0rcJeZdZdedej deffd Z ddejdejdzde dzd ejdzd ejdzd e dzd e dzd e edeej eej ej fdzffdZxZS)ZambaHybridLayer shared_transflinearr cLt|||_||_||_yr)r%r&r)r* mamba_decoder)r,r)r*r r.s r/r&zZambaHybridLayer.__init__Is% * "r0Nr1rrmrWr!r~rrdr#c |j|f|||||d|} |j| } |j|f| |||d|}|S)aF Args: hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)` original_hidden_states (`torch.FloatTensor`): word embedding output that will be concatenated with hidden activations to form the input of the shared transformer layer. layer_idx (`int`): layer number. attention_mask (`torch.FloatTensor`, *optional*): attention mask of size `(batch, sequence_length)` where padding elements are indicated by 0. past_key_values (`Cache`, *optional*): cached past key and value projection states use_cache (`bool`, *optional*): If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding (see `past_key_values`). )rrmrWr~r)r#rWr~r)r)r*r,) r,r1rrmrWr!r~rrdr#s r/r>zZambaHybridLayer.forwardOs2%7D$6$6 % #9&+ % % !%)KK0I$J!***  &?)+    r0)NNNNNF)rCrDrErrrxrr&r(rGrr rrrr@rr>rHrIs@r/r(r(Hs#&@#"))#\r#7; $.2+/(,!&-||-!& t 3-: -  t+ - \\D( --$;-+,- u  %(9(95;L;L(L"MPT"TT U-r0r(c|eZdZUeed<dZdZddgZdZdZ dZ dZ e e dZej fd ZxZS) ZambaPreTrainedModelrlmodelTr(rr~F)r1 attentionsc|jj}t| |t |t rt j|jd||jjdz}t j|j| ||jj|jjz|jjz}tj tj"|jj|t%j&|jj(t%j&|jj*z zt%j&|jj*zj-|jj.}|tj&tj0|  z}t j2|j4|tj6d|j8dztj:dddf}|j=|j>djA}t j2|jBtj&|jE|j|jFdt jH|jJyy)Nr)r:stdro)minrrr4)&rlinitializer_ranger% _init_weights isinstancerinitnormal_rruniform_rrr-rr(rrandmathr time_step_max time_step_minclamptime_step_floorexpm1copy_rrrr8rLrrcrrMrones_r) r,rSr3 dt_init_stdrdtinv_dtrr.s r/r6z"ZambaPreTrainedModel._init_weightsskk++ f% fo . LL--CS A++33T9K MM&//+{ K![[55 8O8OOSWS^S^SlSllN 4;;44nE88DKK556$++B[B[9\\^((4;;4456e 33e4  %))U[["%5$566F JJv**F 3 Q 5 5 9OPTVWPWXA1126AACA JJv||UYYq\%9%9&:N:NPVPePegi%j k JJvxx % /r0)rCrDrEr__annotations__base_model_prefixsupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_supports_flash_attn_supports_sdpa _is_statefulrrk_can_record_outputsr(no_gradr6rHrIs@r/r/r/sa &*#+-EF"3 NL/$ U]]_!!r0r/ceZdZdZdeffd Zeee dde jdzde jdzde jdzde dzd e jdzd edzd eed eezfd ZxZS) ZambaModelz Transformer decoder consisting of *config.num_hidden_layers* layers. Each layer is a [`ZambaDecoderLayer`] Args: config: ZambaConfig rlcFt|||j|_|j|_t j |j|j|j|_|j|_ g}d|_ t|jD]\}}t||}|dk(rt j|jj|jjd}|jt!t#||||jd|dd|di|_ |j|t j$||_t)|j|j* |_d|_|j1y) N)rmhybridFrpz layers.(?!z\.)\d+.shared_transfzlayers.z.shared_transfr)r%r& pad_token_id padding_idx vocab_sizer Embeddingr- embed_tokenslayers_block_type_tied_weights_keys enumeraterrxrlrr(r ModuleListrr!rfinal_layernormgradient_checkpointing post_init)r,rlrlayer_id layer_typer r*r.s r/r&zZambaModel.__init__sc  !.. ++LL):):F,>FDWDWX&+# r0N input_idsrWr"r~ inputs_embedsrrdr#c >|du|duz r td||j|}|}tj|} |r|t |j }|V||j nd} tj|jd|j| z}|jd}t|j ||||} t|jD]\} } | || | || f||d|}|j|}t||r|SdS) NzaYou cannot specify both input_ids and inputs_embeds at the same time, and must specify either one)rlrrr)rlrdrWr~r")r~r)last_hidden_stater~)rrYr(rr rlget_seq_lengthrrArrrr\rr^r)r,rcrWr"r~rdrrdr1rpast_seen_tokensr!rmlayers r/r>zZambaModel.forwards^  -t"; <s    --i8M% !&]!; 0*$++>O  CRC^==?de  << (;(;A(>}G[G[\_ooL'11!4L(;;')+%  !*$++ 6  Iu!&  !0#  M ,,]; &+/8O  >B  r0)NNNNNN)rCrDrErrr&rrrr(r&rGr rrrrr@rr>rHrIs@r/rRrRs{8 .2.204(,26!%8 ##d*8  t+8 &&- 8  8 ((4/ 8 $;8 +,8  ( (8  8 r0rRcNeZdZddiZdeffd Zee ddejdzdejdzdejdzd e dzd ejdzd ejdzd e dzd eejzdeedeezfdZ dfd ZxZS)ZambaForCausalLMzlm_head.weightzmodel.embed_tokens.weightrlct||t||_|j|_t j |j|jd|_|jyr) r%r&rRr0rWrrxr-lm_headr`r s r/r&zZambaForCausalLM.__init__sU  '  ++yy!3!3V5F5FUS  r0NrcrWr"r~rdlabelsrlogits_to_keeprdr#c b|jd||||||d| } | j} t|tr t | dn|} |j | dd| ddf} d}||j | ||jfi| }t|| | j| j| jS)ah labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, ZambaForCausalLM >>> model = ZambaForCausalLM.from_pretrained("Zyphra/Zamba-7B-v1") >>> tokenizer = AutoTokenizer.from_pretrained("Zyphra/Zamba-7B-v1") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```)rcrWr"r~rdrNlosslogitsr~r1r1r) r0rgr7rslicern loss_functionrWrr~r1r1)r,rcrWr"r~rdrorrprdoutputsr1 slice_indicesrtrss r/r>zZambaForCausalLM.forwardsH,64::, )%+' , ,  11 8B>SV8W~ot4]k mA}a,?@A  %4%% D&#33!//))   r0c h|jj|d<t | |f||||||d|} | S)Nrp)r~rWrdr"ris_first_iteration)rlnum_logits_to_keepr%prepare_inputs_for_generation) r,rcr~rWrdr"rrzrd model_inputsr.s r/r|z.ZambaForCausalLM.prepare_inputs_for_generation[sU$(;;#A#A w<   +)'%1    r0)NNNNNNNr)NNNNTF)rCrDrEr[rr&rrr(r&rGr rrrrrr@rr>r|rHrIs@r/rlrl s0*,GH{.2.204(,26*.!%-.@ ##d*@  t+@ &&- @  @ ((4/ @   4'@ $;@ ell*@ +,@  ' '@ @ J r0rla The Zamba Model with a sequence classification head on top (linear layer). [`ZambaForSequenceClassification`] uses the last token in order to do the classification, as other causal models (e.g. GPT-2) do. Since it does classification on the last token, it requires to know the position of the last token. If a `pad_token_id` is defined in the configuration, it finds the last token that is not a padding token in each row. If no `pad_token_id` is defined, it simply takes the last value in each row of the batch. Since it cannot guess the padding tokens when `inputs_embeds` are passed instead of `input_ids`, it does the same (take the last value in each row of the batch). ) custom_introceZdZfdZee d dejdzdejdzdejdzde dzdejdzdejdzd e dzd e e d eezfd ZxZS)ZambaForSequenceClassificationct|||j|_t||_t j |j|jd|_|jyr) r%r& num_labelsrRr0rrxr-scorer`r s r/r&z'ZambaForSequenceClassification.__init__sS   ++' YYv114??O  r0NrcrWr"r~rdrorrdr#c |j|f|||||d|} | j} |j| } ||jd} n|jd} |jj | dk7r t d|jj d} n|||jj k7j| jtj}tj|jd| jtj}||zjd} n.d} tj|jj d| tj| | j | f}d}||j| j}|jj"|j$dk(rd |j_nl|j$dkDrL|j&tj(k(s|j&tj*k(rd |j_nd |j_|jj"d k(rIt-}|j$dk(r&||j/|j/}n|||}n|jj"d k(r=t1}||j3d|j$|j3d}n,|jj"d k(rt5}|||}t7||| j8| j:| j< S)a labels (`torch.LongTensor` of shape `(batch_size,)`, *optional*): Labels for computing the sequence classification/regression loss. Indices should be in `[0, ..., config.num_labels - 1]`. If `config.num_labels == 1` a regression loss is computed (Mean-Square loss), If `config.num_labels > 1` a classification loss is computed (Cross-Entropy). )rWr"r~rdrNrrz=Cannot handle batch sizes > 1 if no padding token is defined.r4rz will not detect padding tokens in `inputs_embeds`. Results may be unexpected if using padding tokens in conjunction with `inputs_embeds.`rf regressionsingle_label_classificationmulti_label_classificationrr)r0rgrrArlrUrr7rr(int32rargmaxrrr.rC problem_typerr6longrrrrrrrr~r1r1)r,rcrWr"r~rdrorrdtransformer_outputsr1rtrlast_non_pad_token non_pad_mask token_indices pooled_logitsrsloss_fcts r/r>z&ZambaForSequenceClassification.forwards&8Btzz 8 )%+' 8 8 ,== M*  "+J&,,Q/J ;; # # + a\] ] ;; # # +!#   "%)A)AAEEfmmUZU`U`aL!LL)!F!Fr!J !#    >>**+,ZZ  u||Jv}}MOaab   YYv}}-F{{''/??a'/;DKK,__q(fllejj.HFLL\a\e\eLe/LDKK,/KDKK,{{''<7"9??a'#M$9$9$;V^^=MND#M6:D))-JJ+- 2 22t GUWY))-II,. v6/ /??-;;*55   r0)NNNNNNN)rCrDrEr&rrr(r&rGr rrrrr@rr>rHrIs@r/rrus.2.204(,26*.!%R ##d*R  t+R &&- R  R ((4/ R   4'R $;R +,R  1 1R R r0r)rlrrRr/)r)Err<collections.abcrr(rtorch.nnrrrr r8 activationsr cache_utilsr r generationr integrations.hub_kernelsr masking_utilsrmodeling_layersrmodeling_outputsrrrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrutils.import_utilsrutils.output_capturingrconfiguration_zambar get_loggerrCrModuler!rGrrRrFrirkrrrrr(r/rRrlr__all__rr0r/rs& $ AA&!.)8/9qqF&RR795,   H %J299J* UU\\ U# U%,, U&% II% <<% % << % LL4' %  %%2C)RYYC)L\]bii\]@ ryy  00f171h414n$!?$!$!N_ %_ _ Fg+_gT  ^ %9^  ^ B gr0