iCddlmZddlZddlmZddlmZddlmZ ddl m Z ddl m Z mZddlmZmZdd lmZdd lmZmZmZd d lmZmZmZmZmZmZmZd d l m!Z!d dl"m#Z#m$Z$ejJe&Z'edeGdde Z(GddeZ)Gdde#Z*GddeZ+GddeZ,GddeZ-GddeZ.eGdd eZ/Gd!d"eZ0Gd#d$eZ1gd%Z2y)&)CallableN)strict)initialization)PreTrainedConfig)BaseModelOutputBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack)TransformersKwargsauto_docstringlogging)CLIPMLP CLIPAttention CLIPEncoderCLIPEncoderLayerCLIPVisionEmbeddingsCLIPVisionModelCLIPVisionTransformer)eager_attention_forward)VisionRotaryEmbeddingapply_rotary_pos_emb_visionz&DeepGlint-AI/mlcd-vit-bigG-patch14-336) checkpointceZdZUdZdZdZdZeed<dZ eed<dZ eed <d Z eed <d Z eed <dZ eed<dZeeezeeefzed<dZeeezeeefzed<dZeed<dZeed<dZeezed<dZeed<dZeed<y)MLCDVisionConfigav num_key_value_groups (`int`, *optional*, defaults to 1): Number of key-value groups used in Attention. Example: ```python >>> from transformers import MLCDVisionConfig, MLCDVisionModel >>> # Initializing a MLCDVisionConfig with DeepGlint-AI/mlcd-vit-bigG-patch14-336 style configuration >>> configuration = MLCDVisionConfig() >>> # Initializing a MLCDVisionModel (with random weights) from the DeepGlint-AI/mlcd-vit-bigG-patch14-336 style configuration >>> model = MLCDVisionModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```mlcd_vision_model vision_configi hidden_sizei intermediate_size0num_hidden_layersnum_attention_headsnum_key_value_groupsr num_channelsiP image_size patch_sizegelu hidden_actgh㈵>layer_norm_epsattention_dropoutg{Gz?initializer_range?initializer_factorN)__name__ __module__ __qualname____doc__ model_typebase_config_keyr int__annotations__r!r#r%r'r(r)listtupler+r-strr.floatr0r1r3v/var/www/vps2.regionflexible.com/Desarrollo/venv/lib/python3.12/site-packages/transformers/models/mlcd/modular_mlcd.pyrr*s &%J%OK!s!s!! !#!L#47Jd3i%S/1746Jd3i%S/16J NE %(us{(#u# ##rArc eZdZy)MLCDMLPN)r4r5r6r@rArBrDrDRsrArDc4eZdZdededej fdZy)MLCDRotaryEmbeddingnum_patches_heightnum_patches_widthreturnctj||jjj dj d|}tj||jjj dj |d}tj |j|jgd}t||}tj||jj|jj}tj||j}||jd} | S)a} Calculate the Rotary Position Embedding (RoPE) for MLCDVisionModel based on the grid size. Args: num_patches_height (int): Number of patches in the height dimension. num_patches_width (int): Number of patches in the width dimension. Returns: torch.Tensor: Rotary positional embeddings for the given grid size. )devicer&rdim)rKdtype) torcharangeinv_freqrK unsqueezeexpandstackflattenmaxrOouter) selfrGrHhpos_idswpos_idspos_ids max_grid_sizeseqrotary_pos_emb_fullrotary_pos_embs rBforwardzMLCDRotaryEmbedding.forwardWs  LL+DMM4H4H I S STU V ] ]^`bs t  LL*4==3G3G H R RST U \ \]oqs t  ++x//183C3C3EFBO.0AB ll=1E1ET]]M`M`a#kk#t}}=-W5==a@rAN)r4r5r6r:rPTensorrar@rArBrFrFVs ##%,,rArFc\eZdZdeffd Zdej dejfdZxZ S)MLCDVisionEmbeddingsconfigc(t|||`yN)super__init__position_embeddingrYre __class__s rBrizMLCDVisionEmbeddings.__init__ys   #rA pixel_valuesrIcT|jd}|jjj}|j|j |}|j dj dd}|jj|dd}tj||gd}|S)NrrOrr&rLrM) shapepatch_embeddingweightrOtorV transposeclass_embeddingrTrPcat)rYrm batch_size target_dtype patch_embeds class_embeds embeddingss rBrazMLCDVisionEmbeddings.forward}s!''* ++2288 ++LOO,O,OP #++A.88A> ++22:q"E YY l;C rA) r4r5r6rrirP FloatTensorrbra __classcell__rls@rBrdrdxs-$/$ E$5$5 %,, rArdceZdZdZdeffd Z d dejdeejejfdejdzde e d eejejdzff d Z xZ S) MLCDAttentionzMulti-headed attention with RoPE. Refer to papers: - Attention is all you need: https://huggingface.co/papers/1706.03762 - RoFormer: Enhanced Transformer with Rotary Position Embedding: https://huggingface.co/papers/2104.09864 recTt|||j|_d|_y)NF)rhrir' is_causalrks rBrizMLCDAttention.__init__s%  $*$?$?!rAN hidden_statesposition_embeddingsattention_maskkwargsrIc |jdd\}}|j|j|||j|jf}|j |j|||j|jf}|j |j|||j|jf} |djdj} |djdj} t||| | \}}|jddddj}|jddddj}| jddddj} tj|jjt } | |||| |f|j"sdn |j$|j&|j(d|\} }| jddddj} | j+||d} |j-| } | jdddj} | |fS)NrLrr&rrr/)dropoutscalingr)rpq_projreshape num_headshead_dimk_projv_projrSr?rpermute contiguousr get_interfacere_attn_implementationrtrainingrscalerviewout_proj)rYrrrrrw seq_length query_states key_states value_statescossinattention_interface attn_output attn_weightss rBrazMLCDAttention.forwards;"/!4!4Sb!9 J{{=199:zSWSaSacgcpcp:qr [[/77ZQUQ_Q_aeanan8op {{=199:zSWSaSacgcpcp:qr "!$..q1779!!$..q1779#>|ZY\^a#b j$++Aq!Q7BBD ''1a3>>@ #++Aq!Q7BBD (?(M(M KK , ,.E) %8      % $}}C$,,JJnn %  % ! \"))!Q15@@B !&&z:rB mmK0 !))!Q2==? L((rArg) r4r5r6r7rrirPrbr=r r rar}r~s@rBrrs//3 ,)||,)#5<<#=>,) t+ ,) +, ,) u||U\\D00 1 ,)rArc eZdZdeffd Z d dej deej ej fdej dzdee deejf d Z xZ S) MLCDEncoderLayerrecDt||t||_yrg)rhrir self_attnrks rBrizMLCDEncoderLayer.__init__s  &v.rANrrrrrIc |}|j|}|jd|||d|\}}||z}|}|j|}|j|}||z}|S)a Args: hidden_states (`torch.FloatTensor`): Input to the layer of shape `(batch, seq_len, embed_dim)`. Represents the hidden states from the previous layer or the input embeddings. position_embeddings (`tuple[torch.Tensor, torch.Tensor]`): A tuple of two tensors, each of shape `(batch, seq_len, embed_dim)`. Represents absolute positional embeddings for the query and key in the attention mechanism. attention_mask (`torch.FloatTensor`): Attention mask of shape `(batch, 1, q_len, k_v_seq_len)` where padding elements are indicated by very large negative values. )rrrr@) layer_norm1r layer_norm2mlp)rYrrrrresidual_s rBrazMLCDEncoderLayer.forwards$!((7 )4>> ' 3)    q !=0  ((7 /  =0 rArg) r4r5r6rrirPrbr=r r r|rar}r~s@rBrrsz////3 "||"#5<<#=>" t+ " +, " u  ! "rArc eZdZdZdeffd Z d dejdeejejfdejdzde e d ee zf d Z xZS) MLCDEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`MLCDEncoderLayer`]. Args: config: MLCDVisionConfig rec$t||y)z3Overwrite dummy `MLCDConfig` to `MLCDVisionConfig`.N)rhrirks rBrizMLCDEncoder.__init__s  rAN inputs_embedsrrrrIc V|}|jD] }||||fi|}t|S)a= Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. position_embeddings (`tuple[torch.Tensor, torch.Tensor]`): A tuple of two tensors, each of shape `(batch, seq_len, embed_dim)`. Represents absolute positional embeddings for the query and key in the attention mechanism. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) )last_hidden_state)layersr)rYrrrrr encoder_layers rBrazMLCDEncoder.forwardsK,& ![[ M)# M +  rArg)r4r5r6r7rrirPr|r=rbr r rrar}r~s@rBrrs{!/!/3 ! ((! #5<<#=>!  t+ ! +, !  ! rArcleZdZUeed<dZdZdZdZdZ dZ dZ e e dZej dZy)MLCDPreTrainedModelremlcdTF)r attentionsc& |jj}t|tr|jj}t j |j d|jdz|zt j |jj|jj|zt j|jtj|jjdj!dyt|t"r|jj}|jdzd|jj$zdzz|z}|jdz|z}t j |j&j|t j |j(j|t j |j*j|t j |j,j|yt|t.r|jj}|jj0dzd|jj$zdzz|z}d|jj0zdz|z}t j |j2j|t j |j4j|yt|t6ro|jj}|jj0|jj8zdzdz|z}t j |j:d|yt|t<j>r?t j@|jBt jD|jyt|t<jFr,|jB t j@|jByt|tHrod |jJtjd |jLdtjN |jLz zz }t j|jP|yy) zInitialize the weightsr/g)meanstd)rrL)r&rLrNr2rro))rer3 isinstancerdinitnormal_ru embed_dimrqrrr1copy_ position_idsrPrQrprTrr#rrrrrDr fc1fc2MLCDVisionTransformerr% class_pos_embnn LayerNormzeros_biasones_LinearrFthetarNr?rR)rYmodulefactor in_proj_std out_proj_stdfc_std pos_emb_stdrRs rB _init_weightsz!MLCDPreTrainedModel._init_weights0s#// f2 3[[33F LL//cv?O?OQU?UX^?^ _ LL//66FMM<[<[^d>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> from transformers import AutoProcessor, MLCDVisionModel >>> model = MLCDVisionModel.from_pretrained("DeepGlint-AI/mlcd-vit-bigG-patch14-448") >>> processor = AutoProcessor.from_pretrained("DeepGlint-AI/mlcd-vit-bigG-patch14-448") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.no_grad(): ... outputs = model(**inputs, output_attentions=True) >>> features = outputs.last_hidden_state >>> print(f"Extracted features shape: {features.shape}") >>> print(f"Number of attention layers: {len(outputs.attentions)}") >>> print(f"Attention shape: {outputs.attentions[0].shape}") ```rmr@) vision_model)rYrmrs rBrazMLCDVisionModel.forwards):!t   %   rArg) r4r5r6rPr|r r r=r rar@rArBrr~s?26  ''$.  +,   + +  rAr)rrr)3collections.abcrrPtorch.nnrhuggingface_hub.dataclassesrrrconfiguration_utilsrmodeling_outputsrr modeling_utilsr r processing_utilsr utilsr rrclip.modeling_cliprrrrrrrllama.modeling_llamarqwen2_vl.modeling_qwen2_vlrr get_loggerr4loggerrrDrFrdrrrrrr__all__r@rArBrs % .&3KF&@@;[   H %CD#$'#$E#$L g /D/$9)M9)x'''T. +. b02/0202f& 1& R! o! H rA