iBrddlmZddlZddlmZddlmZddlmZddl m Z m Z ddl m Z dd lmZdd lmZmZdd lmZmZdd lmZmZdd lmZddlmZmZmZddlm Z ddl!m"Z"ddl#m$Z$m%Z%ddl&m'Z'm(Z(m)Z)ddl*m+Z+ddl,m-Z-m.Z.ddl/m0Z0ejbe2Z3Gdde(Z4Gdde'Z5GddejlZ7Gdde+Z8Gd d!e-Z9Gd"d#ejlZ:Gd$d%eZ;Gd&d'eZeGd+d,e>Z?Gd-d.e.Z@Gd/d0ee>ZAgd1ZBy)2)CallableN)nn)initialization)ACT2FN)Cache DynamicCache)lazy_load_kernel)create_causal_mask) GenericForSequenceClassificationGradientCheckpointingLayer)MoeCausalLMOutputWithPastMoeModelOutputWithPast)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)merge_with_config_defaults)resolve_internal_import)OutputRecordercapture_outputs)LlamaAttention LlamaRMSNormeager_attention_forward) MistralMLP)MixtralExpertsMixtralForCausalLM) JambaConfigc eZdZy) JambaRMSNormN__name__ __module__ __qualname__x/var/www/vps2.regionflexible.com/Desarrollo/venv/lib/python3.12/site-packages/transformers/models/jamba/modular_jamba.pyr$r$.r*r$ceZdZdedeffd Z d dejdejdzdedzde e d e ejejdzff d Z xZ S) JambaAttentionconfig layer_idxct|||tj|j|j |j zd|_tj|j|j|j zd|_ tj|j|j|j zd|_ tj|j |j z|jd|_ yNFbias) super__init__rLinear hidden_sizenum_attention_headshead_dimq_projnum_key_value_headsk_projv_projo_proj)selfr/r0 __class__s r+r6zJambaAttention.__init__3s +ii 2 2F4N4NQUQ^Q^4^ejk ii 2 2F4N4NQUQ^Q^4^ejk ii 2 2F4N4NQUQ^Q^4^ejk ii : :T]] JFL^L^ejk r*N hidden_statesattention_maskpast_key_valueskwargsreturnc |jdd}g|d|j}|j|j|j dd}|j |j|j dd}|j |j|j dd} | |j|| |j\}} tj|jjt} | |||| |f|jsdn |j|j d|\} } | j"g|dj%} |j'| } | | fS)Nr!r)dropoutscaling)shaper:r;view transposer=r>updater0r get_interfacer/_attn_implementationrtrainingattention_dropoutrKreshape contiguousr?) r@rBrCrDrE input_shape hidden_shape query_states key_states value_statesattention_interface attn_output attn_weightss r+forwardzJambaAttention.forward:sq$))#2. 88b8$--8 {{=166|DNNqRST [[/44\BLLQPQR {{=166|DNNqRST  &'6'='=j,X\XfXf'g $J (?(M(M KK , ,.E) %8      % $}}C$2H2HLL %  % ! \*k));;;;FFH kk+. L((r*NN)r&r'r(r"intr6torchTensorrrrtupler^ __classcell__rAs@r+r.r.2sl{lsl/3(, ")||") t+") ") +, ") u||U\\D00 1 ")r*r.ceZdZdZdeffd Z d dejdedzdejdzfdZ d dedzdejdzfd Z d dedzdejdzfd Z xZ S) JambaMambaMixeru Compute ∆, A, B, C, and D the state space parameters and compute the `contextualized_states`. A, D are input independent (see Mamba paper [1] Section 3.5.2 "Interpretation of A" for why A isn't selective) ∆, B, C are input-dependent (this is a key difference between Mamba and the linear time invariant S4, and is why Mamba is called **selective** state spaces) r/c t|||_||_|j|_|j |_|j|_|j|jz|_ |j|_ |j|_|j|_t#j$|j|j|j|j|j|jdz |_|j(|_t,|j(|_t#j0|j|jdz|j |_t#j0|j|j|j dzzd|_t#j0|j|jd|_t9j:d|j dzdddf}|j=|jdj?}t#j@t9jB||_"t#j@t9jF|j|_$t#j0|j|j|j |_%tM|j|jN|_(tM|j |jN|_)tM|j |jN|_*tWd }tY|d da-tY|d da.tWd }t_|d a0tY|dda1tY|dda2tgt`tbt\tZtdfa4thstjjmdyy)Nr!) in_channels out_channelsr4 kernel_sizegroupspaddingrr3FTrHepsz causal-conv1dcausal_conv1d_updatecausal_conv1d_fnz mamba-ssmz8ops.triton.selective_state_update.selective_state_update) chained_pathselective_scan_fnmamba_inner_fnaThe fast path is not available because on of `(selective_state_update, selective_scan_fn, causal_conv1d_fn, causal_conv1d_update, mamba_inner_fn)` is None. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d.)7r5r6r/r0r8 mamba_d_statessm_state_size mamba_d_convconv_kernel_size mamba_expandintermediate_size mamba_dt_ranktime_step_rankmamba_conv_bias use_conv_biasmamba_proj_biasuse_biasrConv1dconv1d hidden_act activationractr7in_projx_projdt_projraarangeexpandrU ParameterlogA_logonesDout_projr$ rms_norm_eps dt_layernorm b_layernorm c_layernormr getattrrprqrselective_state_updatersrtallis_fast_path_availablelogger warning_once)r@r/r0A causal_conv1d mamba_ssmrAs r+r6zJambaMambaMixer.__init__gs  "!--$22 & 3 3!'!4!4v7I7I!I$22#33.. ii..//##--))))A-   !++&++,yy!1!143I3IA3MTXTaTab ii 6 68K8KdNaNadeNe8elqr yy!4!4d6L6LSWX  LLD//!3 4T1W = HHT++R 0 ; ; =\\%))A,/ ejj)?)?@A $"8"8$:J:JQUQ^Q^_ ()<)<&BUBUV'(;(;ATATU'(;(;ATATU)9 &}6LdS"=2DdK%[1 !8 $^" $I/BDI ,jj@}t1jB5t1jD|j>jj@|j>j_ ddd|j?| j dd}t1jB5||j>j_ dddt1jF|jHjK }||jKnd}|rgtM|j|jjN|d|d||dddf|dddf|jP| d|d j d}nptS||||j dd|j dd|jPjK| |dd \}}|||jU||j|jW|j dd}|S#1swYxYw#1swYUxYw) Nr!rdimrrH)r).rT) dt_softplus)delta_softplusreturn_last_state),rLhas_previous_stater0rrNchunk unsqueezerweightrMsizerpsqueezelayers conv_statesr4rr functionalpadrxupdate_conv_staterqrrasplitr|rvrrrrdatano_grad zeros_likeexprfloatrrecurrent_statesrrsupdate_recurrent_stater)r@rBrrC batch_sizeseq_len_use_precomputed_statesprojected_statesgate conv_weightsrssm_parameters time_stepBCtime_proj_biasdiscrete_time_stepr scan_outputs ssm_statecontextualized_statess r+cuda_kernels_forwardz$JambaMambaMixer.cuda_kernels_forwards8 "/!4!4 GQ  $ i)H)H)X i]dhi]i  << 6@@AF/44QA4> t  %)N,D,DQ,GGM{{))..t{{/A/A/F/Fq/I4;;K]K]KbKbcdKef !0%%b)##DNN3??    M*33B7M' mm// @U@UXeXkXklnXo@oqr?st ..{DNNK,]L$++JZJZgkgvgvwM  %)N,D,DQ,GGM]%<%>q!D ]]_ 4%3DLL   " 4YYtzz'') * *3A3M--/SW !1##DNN3DDf%"6*!Q$!Q$V   im '8" Aq! Aq! #"& ' #L)$)A33It~~N!% l.D.DQ.J K$$S N N 4 4sA R1R>1R;>Sc  |j\}}}|j}|j|jdd}|j dd\} } || |j dz} |M|j |jr2|j|jjj} n9tj||j|jf| j|} |I|j |jr|dk(r|j!| |j} tj"| |j$j&dddddfzd} |j(r| |j$j*z } |j-| j/|j d} nt0j2j5| |j6| jdz df} |j!| |j} |j-|j%| dd|f} n'|j-|j%| dd|f} || |j dz} |j9| jdd} tj:| |j<|j|jgd\}}}|j?|}|jA|}|jC|}|jE|}t0j2jG|jdd}tjH|jJjM }tjH|ddddddf|dddddddfz}|dddddddf|dddddddfjMz}|| dddddddfjMz}g}tO|D]}}|dddd|ddf| z|dddd|ddfz} tjP| j/||dd|ddfj d}|jS|dddddftjT|d}|| |jVddddfzz}||j-| z}||jY| |j|j[|jdd}|S)Nr!rr)devicedtyperrH.).rLrrrNrrrr0rrclonerazerosrzrvrrsumrrr~r4rtorrrrxrrr|rrrrsoftplusrrrrangematmulappendstackrrr)r@ input_statesrrCrrrrrrBrr conv_staterrrrrr discrete_A discrete_BdeltaB_uri scan_outputrs r+ slow_forwardzJambaMambaMixer.slow_forwards!-!3!3 GQ""<< 5??1E.44QA4> t  %)N,D,DQ,GGM  # (G(G(W$++DNN;LLRRTI T33T5H5HI$++5I  #..t~~>7a<);;M4>>Z % *t{{7I7I!QPQ'7R*RXZ [ %%!T[[%5%55M $ 7 : :5 A K KB O ]]..!**]-@-@-DDaH *;;JW $])CC'M)R S  HHT[[%?XgX %NOM  %)N,D,DQ,GGM]%<%> J!% k.C.CAq.I J$$r*cV|jjrXtr,d|jjj j vr&tjdd|j_|jjr|j|||S|j|||S)NcudazFast Mamba kernels are not available. Make sure that they are installed and that the mamba module is on a CUDA device. Turning off the fast path `config.use_mamba_kernels=False` and falling back to the slow path.F) r/use_mamba_kernelsrrrrtyperrrr)r@rBrrCs r+r^zJambaMambaMixer.forward]s ;; ( (&& 8J8J8Q8Q8V8V*V   V  -2DKK ) ;; ( (,,]L.Y Y   nMMr*r_)r&r'r(__doc__r"r6rarbr LongTensorrrr^rdres@r+rgrg_sA{AL&*26 c%||c%dlc%((4/ c%LJ%ut|J%\a\l\los\sJ%`&*26 NdlN((4/ Nr*rgc eZdZy)JambaMLPNr%r)r*r+rrrr,r*rc eZdZy) JambaExpertsNr%r)r*r+rrvr,r*rcfeZdZdZdeffd ZdZdejdejfdZ xZ S)JambaSparseMoeBlocka This implementation is strictly equivalent to standard MoE with full capacity (no dropped tokens). It's faster since it formulates MoE operations in terms of block-sparse operations to accommodate imbalanced assignments of tokens to experts, whereas standard MoE either (1) drop tokens at the cost of reduced performance or (2) set capacity factor to number of experts and thus waste computation and memory on padding. r/c,t||j|_|j|_|j |_|j|_tj|j|j d|_ t||_ yr2)r5r6r8 hidden_dimrzffn_dim num_expertsnum_experts_per_toktop_krr7routerrexpertsr@r/rAs r+r6zJambaSparseMoeBlock.__init__sm  ,,// !--// ii1A1AN #F+ r*ctjjj|dtj}tj ||j d\}}||j|jfS)NrH)rrr) rarrsoftmaxrtopkrrr)r@rB router_logitsrouting_weights top_k_weights top_k_indexs r+route_tokens_to_expertsz+JambaSparseMoeBlock.route_tokens_to_expertssb((--55mSXS^S^5_%*ZZQS%T" {M,,]-@-@AAAr*rBrFc|j\}}}|jd|}|j|}|j||\}}|j |||}|j |||}|S)NrH)rLrMrrrrT)r@rBrsequence_lengthrrrrs r+r^zJambaSparseMoeBlock.forwardsx2?2E2E/ OZ%**2z:  M2 %)%A%A-Q^%_" ] ]KO %--j/:V r*) r&r'r(rr"r6rrarbr^rdres@r+rrzs5 ,{,B U\\ellr*rceZdZdedeffd Z d dejdejdzdejdzde dzd e dzd e e d ejfd ZxZS)JambaAttentionDecoderLayerr/r0cRt||jr|j|nd}t|||_|dkDrt nt }|||_t|j|j|_ t|j|j|_ y)Nr!rn) r5r6layers_num_expertsr. self_attnrr feed_forwardr$r8rinput_layernormpre_ff_layernormr@r/r0rffn_layer_classrAs r+r6z#JambaAttentionDecoderLayer.__init__s >D>W>Wf// :]^ ' :1,>FDWDWX ,V-?-?VEXEX Yr*NrBrC position_idsrD use_cacherErFc |}|j|}|jd|||||d|\}}||z}|}|j|}|j|}||z}|S)N)rBrCrrDrr))rrrr) r@rBrCrrDrrEresidualrs r+r^z"JambaAttentionDecoderLayer.forwards!,,]; )4>> ')%+     q!=0  --m< ))-8  =0 r*)NNNF)r&r'r(r"r`r6rarbrrboolrr FloatTensorr^rdres@r+rrsZ{ZsZ/304(,!& || t+&&-    $; +,   r*rc eZdZdedeffd Z d dejdejdzdejdzde dzd e e d ejf d Z xZS) JambaMambaDecoderLayerr/r0cTt||jr|j|nd}t|||_|dkDrt nt }|||_t|j|j|_ t|j|j|_ y)Nr!)r/r0rn) r5r6rrgmambarrrr$r8rrrrs r+r6zJambaMambaDecoderLayer.__init__s >D>W>Wf// :]^ $FiH 1,>FDWDWX ,V-?-?VEXEX Yr*NrBrCrrDrErFc |}|j|}|j|||}||z}|}|j|}|j|}||z}|S)N)rBrrC)rrrr)r@rBrCrrDrEr s r+r^zJambaMambaDecoderLayer.forwardsv!,,];  '()# !=0  --m< ))-8  =0 r*)NNN)r&r'r(r"r`r6rarbrrrrr r^rdres@r+r r sZ{ZsZ/304(, || t+&&-    +,    r*r ) attentionrceZdZUeed<dZdZddgZdZdZ dZ dZ e e geeej"dd Zej(fd ZxZS) JambaPreTrainedModelr/modelTrr rDr) layer_name)rB attentionsrct||t|trt j d|j dzdddf}|j|jdj}tj|jt j|tj|jyt|t rmtj"|j$d|j&j(tj"|j*d|j&j(yy)Nr!rHrI)meanstd)r5 _init_weights isinstancergrarrvrrzrUinitcopy_rrones_rrnormal_ gate_up_projr/initializer_range down_proj)r@modulerrAs r+rz"JambaPreTrainedModel._init_weightss f% fo . Q 5 5 9:47CA1126AACA JJv||UYYq\ 2 JJvxx  - LL,,3DKK,>FDWDWX&+# r*N input_idsrCrrD inputs_embedsrrErFc 2|du|duz r td||j|}|r|t|j}|V||j nd}t j |jd|j|z}|jd}t|j||||} |j||} |} |jD]$} t| tr| n| } | | f| |||d|} &|j| } t!| |S) Nz:You must specify exactly one of input_ids or inputs_embeds)r/rr!)r)r/r?rCrDr)rCrrDr)last_hidden_staterD) ValueErrorr4r r/get_seq_lengthrarrLrrr _update_mamba_maskrrr r9r)r@r>rCrrDr?rrEpast_seen_tokens causal_mask mamba_maskrB decoder_layer layer_masks r+r^zJambaModel.forwardsJ  -t"; <YZ Z   --i8M 0*$++>O  CRC^==?de  << (;(;A(>}G[G[\_ooL'11!4L(;;')+%  ,,^_M % ![[ M'1-AW'X^iJ))) /#   M ,,]; %++  r*cf|}||js|tj|dk(rd}|S)zv No need for zeroing states when 1. Cached forward 2. Attending to all inputs Nr!)rrar)r@rCrDrGs r+rDzJambaModel._update_mamba_maskPs; $  'O,N,N,P  &599^q5H+IJr*)NNNNNN)r&r'r(r"r6rrrrarrbrr r rrrr^rDrdres@r+r.r.s{$ .2.204(,26!%2 ##d*2  t+2 &&- 2  2 ((4/ 2 $;2 +,2  2  2 h r*r.c$eZdZdeffd Z ddej dzdejdzdej dzdedzdejdzd ej dzd e dzd e dzd e ejzd e e deffd ZxZS)JambaForCausalLMr/cFt|||j|_y)N)r5r6rrs r+r6zJambaForCausalLM.__init___s  !--r*Nr>rCrrDr?labelsroutput_router_logitslogits_to_keeprErFc 2t ||||||||| fi| S)aj labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.vocab_size]`. Example: ```python >>> from transformers import AutoTokenizer, JambaForCausalLM >>> model = JambaForCausalLM.from_pretrained("ai21labs/Jamba-v0.1") >>> tokenizer = AutoTokenizer.from_pretrained("ai21labs/Jamba-v0.1") >>> prompt = "Hey, are you conscious? Can you talk to me?" >>> inputs = tokenizer(prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(inputs.input_ids, max_length=30) >>> tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Hey, are you conscious? Can you talk to me?\nI'm not conscious, but I can talk to you." ```)r5r^) r@r>rCrrDr?rNrrOrPrErAs r+r^zJambaForCausalLM.forwardcs9Fw            r*) NNNNNNNNr)r&r'r(r"r6rarrbrr r r`rrrr^rdres@r+rLrL^s.{. .2.204(,26*.!%,0-.- ##d*-  t+- &&- -  - ((4/ -   4'- $;- #Tk- ell*- +,-  #- - r*rLc eZdZy)JambaForSequenceClassificationNr%r)r*r+rSrSr,r*rS)rLrSr.r)Ccollections.abcrrarrr activationsr cache_utilsrr integrationsr masking_utilsr modeling_layersr r modeling_outputsrrmodeling_utilsrrprocessing_utilsrutilsrrr utils.genericrutils.import_utilsrutils.output_capturingrrllama.modeling_llamarrrmistral.modeling_mistralrmixtral.modeling_mixtralrr configuration_jambar" get_loggerr&rr$r.Modulergrrrrr r6rr.rLrS__all__r)r*r+ris3&% &!.,/[QF&@@79EXX1I,   H % < *)^*)ZPNbiiPNf z  > ""))"J#!;#L7B)CMcdX?X8U%UUp2 )2 j %EG[  gr*