i&ddlmZddlmZddlZddlmZddlmZddl m Z ddl m Z m Z mZddlmZdd lmZdd lmZmZdd lmZdd lmZdd lmZmZmZmZmZddl m!Z!ddl"m#Z#m$Z$ddl%m&Z&ddl'm(Z(m)Z)m*Z*m+Z+ddl,m-Z-ddl.m/Z/m0Z0ddl1m2Z2m3Z3m4Z4ddl5m6Z6m7Z7m8Z8ddl9m:Z:m;Z;e+jxe=Z>e)deGddeZ?ee)dGddeZ@Gd d!ejZBGd"d#ejZCGd$d%e3ZDGd&d'e2ZEGd(d)e6ZFGd*d+eZGe)Gd,d-e$ZHGd.d/eHZIGd0d1e7ZJGd2d3e:ZKe)d4Gd5d6eHeZLgd7ZMy)8)Callable) dataclassN)strict)ACT2FN)Cache DynamicCacheEncoderDecoderCache)PreTrainedConfig)GenerationMixin)create_bidirectional_maskcreate_causal_mask)FlashAttentionKwargs)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPast)BaseModelOutputWithPastAndCrossAttentionsSeq2SeqLMOutputSeq2SeqModelOutput)RopeParameters)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringcan_return_tuplelogging)merge_with_config_defaults)OutputRecordercapture_outputs) GlmAttentionGlmRotaryEmbeddingapply_rotary_pos_emb)LlamaDecoderLayer LlamaModeleager_attention_forward) WhisperModelshift_tokens_rightzUsefulSensors/moonshine-tiny) checkpointceZdZUdZdZdgZdddddZd Zee d <d Z ee d <d Z ee d<dZ ee d<dZ ee d<dZee d<dZee d<dZedze d<dZedze d<dZedze d<dZee d<dZee d<dZee d<dZee d<dZee d<dZee d <dZeezdze d!<dZee d"<d#Z ee d$<d%Z!eeze d&<dZ"edze d'<d(Z#ee$ezdze d)<dZ%edze d*<dZ&ee d+<fd,Z'xZ(S)-MoonshineConfiga encoder_num_key_value_heads (`int`, *optional*): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `encoder_num_key_value_heads=encoder_num_attention_heads`, the model will use Multi Head Attention (MHA), if `encoder_num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out [this paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `num_attention_heads`. decoder_num_key_value_heads (`int`, *optional*): This is the number of key_value heads that should be used to implement Grouped Query Attention. If `decoder_num_key_value_heads=decoder_num_attention_heads`, the model will use Multi Head Attention (MHA), if `decoder_num_key_value_heads=1` the model will use Multi Query Attention (MQA) otherwise GQA is used. When converting a multi-head checkpoint to a GQA checkpoint, each group key and value head should be constructed by meanpooling all the original heads within that group. For more details, check out [this paper](https://huggingface.co/papers/2305.13245). If it is not specified, will default to `decoder_num_attention_heads`. pad_head_dim_to_multiple_of (`int`, *optional*): Pad head dimension in encoder and decoder to the next multiple of this value. Necessary for using certain optimized attention implementations. encoder_hidden_act (`str` or `function`, *optional*, defaults to `"gelu"`): The non-linear activation function (function or string) in the encoder. decoder_hidden_act (`str` or `function`, *optional*, defaults to `"silu"`): The non-linear activation function (function or string) in the decoder. Example: ```python >>> from transformers import MoonshineModel, MoonshineConfig >>> # Initializing a Moonshine style configuration >>> configuration = MoonshineConfig().from_pretrained("UsefulSensors/moonshine-tiny") >>> # Initializing a model from the configuration >>> model = MoonshineModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ``` moonshinepast_key_valuesdecoder_num_key_value_headsdecoder_num_attention_headsdecoder_num_hidden_layersdecoder_hidden_act)num_key_value_headsnum_attention_headsnum_hidden_layers hidden_acti vocab_sizei  hidden_sizeiintermediate_sizeencoder_num_hidden_layersencoder_num_attention_headsNencoder_num_key_value_headspad_head_dim_to_multiple_ofgeluencoder_hidden_actsiluimax_position_embeddingsg{Gz?initializer_rangedecoder_start_token_idT use_cacherope_parametersis_encoder_decoderFattention_biasattention_dropout bos_token_idr! eos_token_id pad_token_idtie_word_embeddingsc |j|j|_|j|j|_|j ddt |di|y)Npartial_rotary_factorg?)r>r=r/r0 setdefaultsuper __post_init__)selfkwargs __class__s /var/www/vps2.regionflexible.com/Desarrollo/venv/lib/python3.12/site-packages/transformers/models/moonshine/modular_moonshine.pyrVzMoonshineConfig.__post_init__sX  + + 3/3/O/OD ,  + + 3/3/O/OD ,137 ''))__name__ __module__ __qualname____doc__ model_typekeys_to_ignore_at_inference attribute_mapr7int__annotations__r8r9r;r1r=r0r>r/r?rAstrr2rCrDfloatrFrGboolrHrdictrIrJrLrMrNlistrOrPrV __classcell__rYs@rZr,r,2sg& PJ#4"5<<8* MJK!s!%&s&%&s&'(('((.2t2.2t2.2t2$$$$#&S&#u#"#C#It48O^d*T18## ND %(us{( L#* +,L#S /D(,#L#*# $$((r[r,z Extends [~modeling_outputs.BaseModelOutput] to include the output attention mask since sequence length is not preserved in the model's forward. ) custom_introc6eZdZUdZej dzed<y)MoonshineEncoderModelOutputNattention_mask)r\r]r^rotorchTensorrdrSr[rZrnrns+/NELL4'.r[rncVeZdZfdZdej dej fdZxZS)MoonshineEncoderMLPct|||_t||_t j |j|j|_ t j |j|j|_ yN rU__init__configr activation_fnnnLinearr8r9fc1fc2rWrxr6rYs rZrwzMoonshineEncoderMLP.__init__s^  #J/99V//1I1IJ99V55v7I7IJr[ hidden_statesreturncl|j|}|j|}|j|}|Sru)r|ryr})rWrs rZforwardzMoonshineEncoderMLP.forwards4/ **=9 / r[r\r]r^rwrprqrrjrks@rZrsrss$KU\\ellr[rscVeZdZfdZdej dej fdZxZS)MoonshineDecoderMLPct|||_t||_t j |j|jdz|_ t j |j|j|_ y)Nr!rvr~s rZrwzMoonshineDecoderMLP.__init__sc  #J/99V//1I1IA1MN99V55v7I7IJr[rrc|j|}|jdd\}}|j||z}|j|}|S)Nr!)dim)r|chunkryr})rWrgates rZrzMoonshineDecoderMLP.forwardsS/ +11!1< t**40=@ / r[rrks@rZrrs$KU\\ellr[rc eZdZy)MoonshineRotaryEmbeddingN)r\r]r^rSr[rZrrsr[rcHeZdZdededededef fd Z ddejd e ejejfdzd ejdzd e dzd ejdzd e e de ejejdze ejdzffdZ xZS)MoonshineAttentionrx layer_idx is_causalr4r3cn|j||dt| ||||_t |d|j |j z|_|jjC|jj}||j|zdz |zz}||jz |_ yd|_ y)N)r4r3head_dimrEr) updaterUrwrgetattrr8r4rrxr?head_dim_padding) rWrxrrr4r3target_multipletarget_head_dimrYs rZrwzMoonshineAttention.__init__s  .AZmno +" F4F4F&JdJd4de  ;; 2 2 >"kkEEO-$--/2QTU2UZi1ijO$3dmm$CD !$%D !r[Nrposition_embeddingsror.key_value_statesrXrc :|jdd\}}|j|j|||jj|j j dd} |du} |Y|jj|j} | r&d|j|j<|j}n |j}||n|} | rK|rI rG|j|jj} |j|jj}n|j| j|d|jj|j j dd} |j!| j|d|jj|j j dd}| r"| |j#| ||j\} }| s8|\}}t%| | ||\} } | |j#| ||j\} }t'j(|jj*t,}|j.xr |duxr|dkD}|j0dkDrt2j4j6j9| d|j0f} t2j4j6j9| d|j0f} t2j4j6j9|d|j0f}||| | ||f|j:sdn |j<|j>|d|\}}|j0dkDr|dd|j0 f}|jA||djC}|jE|}||fS) NrrEr!TrrK)dropoutscalingr.)#shapeq_projviewrxr3r transpose is_updatedgetrcross_attention_cacheself_attention_cachelayerskeysvaluesk_projv_projrr$r get_interface_attn_implementationr'rrrprz functionalpadtrainingrLrreshape contiguouso_proj)rWrrror.rrXbszq_len query_statesis_cross_attentionrcurrent_states key_states value_statescossinattention_interfacer attn_output attn_weightss rZrzMoonshineAttention.forwards]#(("- U KK & + +C 8W8WY]YfYf g q qrsuv w .T9  &(3377GJ!=A**4>>:"1"G"G"1"F"F.>-I)} /j(//?DDJ*11$..AHHL N+c2t{{>> N1a   N+c2t{{>> N1a  "o&A+:+A+A*l\`\j\j+k( L!*HC';L*VY[^'_ $L**+:+A+A*l\`\j\j+k( L(?(M(M KK , ,.E) NNK~'=K%!)  1 $ 88..22EO) t+ O)  O)  ,,- O)-.O) u||U\\D0% 2E2LL MO)r[rc(eZdZdedeffd ZxZS)MoonshineEncoderLayerrxrcFt|||t||d|j|j|_t ||j|_tj|jd|_ tj|jd|_ y)NFrxrrr4r3bias)rUrwrr=r> self_attnrsrAmlprz LayerNormr8input_layernormpost_attention_layernormrWrxrrYs rZrwzMoonshineEncoderLayer.__init__ s ++ & B B & B B  'vv/H/HI!||F,>,>UK(* V5G5Ge(T%r[)r\r]r^r,rcrwrjrks@rZrrs U U3 U Ur[rceZdZddededzffd Z ddejdejdzdejdzdejdzd ejdzd ejdzd e dzd e dzd e ejejfdzde ejejfdzde e de eje ejejfdzffdZxZS)MoonshineDecoderLayerNrxrct||j|_t||d|j|j |_t||d|j|j |_t||j|_ tj|jd|_ tj|jd|_tj|jd|_y)NTrFr)rUrwr8rr4r3r encoder_attnrr6rrzrrrfinal_layernormrs rZrwzMoonshineDecoderLayer.__init__1s !--+ & : : & : :  / & : : & : :  'vv/@/@A!||F,>,>UK(* V5G5Ge(T%!||F,>,>UKr[rroencoder_hidden_statesencoder_attention_mask position_idsencoder_position_idsr.rGrencoder_position_embeddingsrXrc &|} |j|}|jd|||||| d| \}} | |z}|1|} |j|}|j|||||\}} | |z}|} |j |}|j |}| |z}|S)N)rrorr.rGr)rrror.rGrS)rrrrrr)rWrrorrrrr.rGrrrXresidual_s rZrzMoonshineDecoderLayer.forwardIs!,,]; )4>> ')%+ 3    q!=0 ,$H 99-HM#00+!65 /# 1  M1%}4M ,,]; /  =0 r[ru) NNNNNNFNN)r\r]r^r,rcrwrprq LongTensorrrgrrr FloatTensorrrjrks@rZrr0sRLL3:L6/3596:048<(,!&HLPT,||, t+, %||d2 , !& t 3 , &&- ,$..5,,$;,#5<<#=>E,&+5<<+E%F%M,+,, u  %(9(95;L;L(L"MPT"TT U,r[rc\eZdZUeed<dZdZdZdZddgZ dZ dZ dZ de jfd Zy ) MoonshinePreTrainedModelrxmodel input_valuesaudioTrr input_lengthsc~t|dz dz dz}t|dz dz dz}t|dz dz dz}|S)zH Computes the output length of the convolutional layers @rErr!)rc)rWroutput_conv1_lengthoutput_conv2_lengthoutput_conv3_lengths rZ _get_feat_extract_output_lengthsz9MoonshinePreTrainedModel._get_feat_extract_output_lengthssZ"=3#6""N>N#r[rc eZdZdZdZeedZdeffd Z de jfdZ de jfd Z ee ddej"d ej$d zd eedeezfd ZxZS)MoonshineEncoderz Transformer encoder consisting of *config.num_hidden_layers* layers. Each layer is a [`MoonshineEncoderLayer`] Args: config: MoonshineConfig r) attentionsrrxc bt||||_|j}t j d|ddd|_t j |d|zdd |_t j d|z|dd |_t jd|d |_ t jt|jDcgc]}t||c}|_t j |d |_t%| |_d|_|j+ycc}w)NrErrF) kernel_sizestriderr!rr)rrgh㈵>) num_groups num_channelsepsrrx)rUrwrxr8rzConv1dconv1conv2conv3 GroupNorm groupnorm ModuleListranger;rrr layer_normr rotary_embgradient_checkpointing post_init)rWrx embed_dimidxrYs rZrwzMoonshineEncoder.__init__s   && YYq)ReT YYy!i-QqQ YYq9}iQqQ PTUmm;@AaAa;b cC "63 / c  ,,yu=2&A&+#  dsD,rc|jSrurrWs rZget_input_embeddingsz%MoonshineEncoder.get_input_embeddingss zzr[valuec||_yrur )rWr s rZset_input_embeddingsz%MoonshineEncoder.set_input_embeddingss  r[NrorXc h|jd}tjj|j |}|j |}tjj |j|}tjj |j|}|jddd}d}|3|j|jd}d}|ddd|fdd|f}|}t|j|||}tjd|jd|j jd}|j#|| } |j$D]} | |f||| d |}|j'|}t)|||j+ Sd S) a. Args: input_values (`torch.FloatTensor` of shape `(batch_size, audio_length)`): Float values of the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoFeatureExtractor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding indices in `input_values`. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) rErr!Nri.rx inputs_embedsrordevicer)rorr)last_hidden_statero) unsqueezerzrtanhrrr@rrpermuterrr rxrparangerrrrrnrc) rWrrorXroutput_attention_maskmask_lendownsample_striderr encoder_layers rZrzMoonshineEncoder.forwards.$--a0  **4::l+CD }5  **4::m+DE  **4::m+DE %--aA6 !%  %<<^=Q=QRT=UVH * +C1D3D1D,DEc9H9nUN$2 !2;;')"/  ||A}':':1'=mFZFZ[eefgh "oom,oW![[ M)-)$7   M  6 *+:O:[0446  ae  r[ru)r\r]r^r_rrr_can_record_outputsr,rwrzModuler rrr rprrqrrrrrrjrks@rZrrs%O(. $bii")) /3< ''<  t+< +, < ( ( <  < r[rcbeZdZdZeeddeeedddZdeffd Z e e dde jd zd e jd zd e jd zd ed zd e j d zded zde j d zde jd zdeedeezfdZxZS)MoonshineDecoder input_idsrEr)index layer_namer)rrcross_attentionsrxc t||tj|jd|_tj t|jDcgc]}t||c}|_ ycc}wNFr) rUrwrzrr8normrrr5rr)rWrxrrYs rZrwzMoonshineDecoder.__init__s[  LL!3!3%@ mmSXY_YqYqSr$sC%:63%G$st $ssA=Nrorr.rrGrrrXrc |du|duz r td||j|}|r6|4tt|jt|j}|V||j nd} t j|jd|j| z}|jd}t|j||||} t|j|||}|} |j| | } |jD]}|| | |f||||| d | } |j| } t!| |r| Sd S) a encoder_hidden_states (`torch.FloatTensor` of shape `(batch_size, encoder_sequence_length, hidden_size)`, *optional*): Sequence of hidden-states at the output of the last layer of the encoder. Used in the cross-attention of the decoder. encoder_attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding indices in `encoder_hidden_states`. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) Nz:You must specify exactly one of input_ids or inputs_embedsrrrEr)rxrror.rrr)rrr.rGr)rr.) ValueError embed_tokensr r rxget_seq_lengthrprrrrrr rrr)r)rWr#rorr.rrGrrrXpast_seen_tokens causal_maskrr decoder_layers rZrzMoonshineDecoder.forwards0  -t"; <YZ Z   --i8M 01,dkk2RT`hlhshsTtuO  CRC^==?de  << (;(;A(>}G[G[\_ooL'11!4L(;;')+%  ";;;'1"7 " & "oom,oW![[ M)% (>) /#$7  M  -0 8+/8O  >B  r[)NNNNNNNN)r\r]r^rrrrrr,rwrr rprrqrrrgrrrrrrjrks@rZr"r"s2!O$%7q[Y.*+=QSab uu  .2.204(,26!%:>6:G ##d*G  t+G &&- G  G ((4/ G $;G  %0047G !& t 3G +,G  ( (G  G r[r"cNeZdZdZee ddejdzdejdzdejdzdejdzde e ejdzde dzd e ejdzd e ejdzd e dzd e e d efdZy)MoonshineModelctd)NzNot needed for Moonshine)AttributeErrorr s rZ_mask_input_featuresz#MoonshineModel._mask_input_featuresRs788r[Nrrodecoder_input_idsdecoder_attention_maskencoder_outputsr.decoder_inputs_embedsdecoder_position_idsrGrXrc T||j|fd|i| }|jd|||j|j|||| d| } t | j| j | j | j| j|j|j |jS)a  input_values (`torch.FloatTensor` of shape `(batch_size, audio_length)`): Float values of the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoFeatureExtractor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. decoder_position_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`): Indices of positions of each input sequence tokens in the position embeddings. Used to calculate the position embeddings up to `config.decoder_config.max_position_embeddings` Example: ```python >>> import torch >>> from transformers import AutoFeatureExtractor, MoonshineModel >>> from datasets import load_dataset >>> model = MoonshineModel.from_pretrained("UsefulSensors/moonshine-tiny") >>> feature_extractor = AutoFeatureExtractor.from_pretrained("UsefulSensors/moonshine-tiny") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> inputs = feature_extractor(ds[0]["audio"]["array"], return_tensors="pt") >>> input_values = inputs.input_values >>> decoder_input_ids = torch.tensor([[1, 1]]) * model.config.decoder_start_token_id >>> last_hidden_state = model(input_values, decoder_input_ids=decoder_input_ids).last_hidden_state >>> list(last_hidden_state.shape) [1, 2, 288] ``` ro)r#rorrr.rrrG)rr.decoder_hidden_statesdecoder_attentionsr&encoder_last_hidden_staterencoder_attentionsrS) encoderdecoderrrorr.rrr&) rWrror6r7r8r.r9r:rGrXdecoder_outputss rZrzMoonshineModel.forwardUsZ  "/;t||L/rYg/rkq/rOEQT\\ F '1"1"C"C#2#A#A+/- F  F "-??+;;"1"?"?.99,==&5&G&G"1"?"?.99  r[) NNNNNNNNN)r\r]r^r5rrrprrrr rgrrrrrSr[rZr2r2Qs)9262659:>BF6:AE?C!%C ''$.C ((4/C !++d2 C !& 0 04 7 C uU%6%6784? C -t3C  %U%6%67$>C $E$4$45<C $;C +,C  C C r[r2zj The Moonshine Model with a language modeling head. Can be used for automatic speech recognition. ceZdZddiZdeffd ZdZdZdejfdZ e e dd e jd zd e jd zd e jd zd e jd zdeee jd zded zdee jd zdee jd zded zde jd zdeedefdZxZS)!MoonshineForConditionalGenerationzproj_out.weightz!model.decoder.embed_tokens.weightrxct||t||_t j |j |jd|_|jyr() rUrwr2rrzr{r8r7proj_outr)rWrxrYs rZrwz*MoonshineForConditionalGeneration.__init__sH  #F+  &"4"4f6G6GeT  r[c|jSrurFr s rZget_output_embeddingsz7MoonshineForConditionalGeneration.get_output_embeddingss }}r[c||_yrurH)rWnew_embeddingss rZset_output_embeddingsz7MoonshineForConditionalGeneration.set_output_embeddingss & r[rc6|jjSru)rr r s rZr z6MoonshineForConditionalGeneration.get_input_embeddingsszz..00r[Nrror6r7r8r.r9r:rGlabelsrXc | 9|7|5t| |jj|jj}|j|f|||||||| d| } |j | j } d}| (|j| | |jj}t|| | j| j| j| j| j| j| j  S)a0 input_values (`torch.FloatTensor` of shape `(batch_size, audio_length)`): Float values of the raw speech waveform. Raw speech waveform can be obtained by loading a `.flac` or `.wav` audio file into an array of type `list[float]`, a `numpy.ndarray` or a `torch.Tensor`, *e.g.* via the torchcodec library (`pip install torchcodec`) or the soundfile library (`pip install soundfile`). To prepare the array into `input_values`, the [`AutoFeatureExtractor`] should be used for padding and conversion into a tensor of type `torch.FloatTensor`. decoder_position_ids (`torch.LongTensor` of shape `(batch_size, target_sequence_length)`): Indices of positions of each input sequence tokens in the position embeddings. Used to calculate the position embeddings up to `config.decoder_config.max_position_embeddings` Example: ```python >>> import torch >>> from transformers import AutoProcessor, MoonshineForConditionalGeneration >>> from datasets import load_dataset >>> processor = AutoProcessor.from_pretrained("UsefulSensors/moonshine-tiny") >>> model = MoonshineForConditionalGeneration.from_pretrained("UsefulSensors/moonshine-tiny") >>> ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation") >>> inputs = processor(ds[0]["audio"]["array"], return_tensors="pt") >>> input_values = inputs.input_values >>> generated_ids = model.generate(input_values, max_new_tokens=100) >>> transcription = processor.batch_decode(generated_ids, skip_special_tokens=True)[0] >>> transcription 'Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel.' ```N)ror6r8r7r.r9r:rG)logitsrNr7) lossrPr.r<r=r&r>rr?)r)rxrOrFrrFr loss_functionr7rr.r<r=r&r>rr?)rWrror6r7r8r.r9r:rGrNrXoutputsrPrQs rZrz)MoonshineForConditionalGeneration.forwards d   (-B-J$6DKK44dkk6X6X%!'1djj  ' )/+#9+"7!5 '  ' w889  %%VFt{{OeOe%fD#33")"?"?&99$55&-&G&G")"?"?&99  r[) NNNNNNNNNN)r\r]r^_tied_weights_keysr,rwrIrLrzr r rrrprrrr rgrrrrrjrks@rZrDrDsr ,-PQ'1bii1262659:>BF6:AE?C!%*.R ''$.R ((4/R !++d2 R !& 0 04 7 R uU%6%6784? R -t3R  %U%6%67$>R $E$4$45<R $;R   4'R +,R  R R r[rD)r,r2rrD)Ncollections.abcr dataclassesrrptorch.nnrzhuggingface_hub.dataclassesr activationsr cache_utilsrr r configuration_utilsr generationr masking_utilsr rmodeling_flash_attention_utilsrmodeling_layersrmodeling_outputsrrrrrmodeling_rope_utilsrmodeling_utilsrrprocessing_utilsrutilsrrrr utils.genericrutils.output_capturingrr glm.modeling_glmr"r#r$llama.modeling_llamar%r&r'whisper.modeling_whisperr(r) get_loggerr\loggerr,rnr rsrrrrrrrr"r2rD__all__rSr[rZrms%! .!CC3)JB92F&RR7EUUYYG   H %9:S(&S(;S(l  ///  / "))  "))   1 e)e)PU-U"E6EP###0d /d NV zV rI \I X h (@/h  h V r[