i dZddlmZddlmZddlmZddlZddlmZddl m Z dd l m Z dd lmZdd lmZdd lmZmZmZdd lmZmZddlmZddlmZmZmZmZm Z ddl!m"Z"m#Z#ddl$m%Z%ddl&m'Z'm(Z(m)Z)ejTe+Z,dejZdejZfdZ.dejZdejZfdZ/dejZdejZfdZ0eedGddeZ1eedGd d!eZ2eeGd"d#eZ3Gd$d%ejhZ5Gd&d'ejhZ6 dOd(ejhd)ejZd*ejZd+ejZd,ejZdzd-e7d.e7d/eefd0Z8Gd1d2ejhZ9Gd3d4ejhZ:Gd5d6eZ;eGd7d8eZ<Gd9d:ejhZ=Gd;ded=Gd>d?e<Z?Gd@dAe<Z@edBGdCdDe<ZAeGdEdFe<ZBeGdGdHe<ZCeGdIdJe<ZDedKGdLdMe<ZEgdNZFy)PzPyTorch CLIP model.)Callable) dataclass)AnyN)nn)initialization)ACT2FN)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPoolingImageClassifierOutput)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringlogging torch_int)can_return_tuplemerge_with_config_defaults)capture_outputs) CLIPConfigCLIPTextConfigCLIPVisionConfiglogitsreturnctjj|tjt ||j S)Ndevice)r functional cross_entropytorcharangelenr")rs w/var/www/vps2.regionflexible.com/Desarrollo/venv/lib/python3.12/site-packages/transformers/models/clip/modeling_clip.pycontrastive_lossr)/s/ == & &vu||CKPVP]P]/^ __ similaritycZt|}t|j}||zdz S)Ng@)r)t)r+ caption_loss image_losss r( clip_lossr03s,#J/L!*,,.1J : % ,,r*tensorctj|d}tj|dd}tj|d}|S)z This method is equivalent to tensor.norm(p=2, dim=-1, keepdim=True) and used to make model `executorch` exportable. See issue https://github.com/pytorch/executorch/issues/3566 T)dimkeepdim?)r%powsum)r1 square_tensor sum_tensor normed_tensors r(_get_vector_normr=9s< IIfa(M=b$?JIIj#.M r*z} Base class for vision model's outputs that also contains image embeddings of the pooling of the last hidden states. ) custom_introceZdZUdZdZej dzed<dZej dzed<dZ e ej dfdzed<dZ e ej dfdzed<y)CLIPVisionModelOutputz image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The image embeddings obtained by applying the projection layer to the pooler_output. N image_embedslast_hidden_state. hidden_states attentions) __name__ __module__ __qualname____doc__rAr% FloatTensor__annotations__rBrCtuplerDr*r(r@r@Dsr .2L%##d*126u((4/6:>M5**C/047>7;Je'',-4;r*r@ze Base class for text model's outputs that also contains a pooling of the last hidden states. ceZdZUdZdZej dzed<dZej dzed<dZ e ej dfdzed<dZ e ej dfdzed<y)CLIPTextModelOutputz text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim)` *optional* returned when model is initialized with `with_projection=True`): The text embeddings obtained by applying the projection layer to the pooler_output. N text_embedsrB.rCrD) rErFrGrHrOr%rIrJrBrCrKrDrLr*r(rNrNVsr -1K""T)026u((4/6:>M5**C/047>7;Je'',-4;r*rNceZdZUdZdZej dzed<dZej dzed<dZ ej dzed<dZ ej dzed<dZ ej dzed<dZ e ed<dZe ed <d eefd Zy) CLIPOutputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`CLIPTextModel`]. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`CLIPVisionModel`]. text_model_output (`BaseModelOutputWithPooling`): The output of the [`CLIPTextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`CLIPVisionModel`]. Nlosslogits_per_imagelogits_per_textrOrAtext_model_outputvision_model_outputrcBtd|jDS)Nc3`K|]&}t|tr|jn|(ywN) isinstancerto_tuple).0vs r( z&CLIPOutput.to_tuple..s$^1Z;%?QZZ\QF^s,.)rKvaluesselfs r(r[zCLIPOutput.to_tuples^PTP[P[P]^^^r*)rErFrGrHrRr%rIrJrSrTrOrArUr rVrKrr[rLr*r(rQrQhs&&*D%  d ")15e''$.504OU&&-4,0K""T)0-1L%##d*148186:3:_%*_r*rQceZdZdeffd Zdej dededej fdZd dejdej fd Z xZ S) CLIPVisionEmbeddingsconfigct|||_|j|_|j |_|j |_tjtj|j|_ tj|j|j|j |j d|_|j |j zdz|_|jdz|_tj"|j |j|_|j'dtj(|j j+ddy)NF) in_channels out_channels kernel_sizestridebiasr3r position_idsrr4 persistent)super__init__rd hidden_size embed_dim image_size patch_sizer Parameterr%randnclass_embeddingConv2d num_channelspatch_embedding num_patches num_positions Embeddingposition_embeddingregister_bufferr&expandrard __class__s r(rpzCLIPVisionEmbeddings.__init__s   ++ ++ ++!||EKK,GH!yy++??  !OOt>1D!--1"$,,t/A/A4>>"R ^U\\$:L:L-M-T-TU\-]jopr* embeddingsheightwidthrc|jddz }|jjjd}|jddz }tj j s%||k(r ||k(r|j|jS|ddddf}|ddddf}|jd} ||jz} ||jz} t|dz} |jd| | | }|jdddd}tjj|| | fdd }|jddddjdd| }t j ||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNr4r7rr3bicubicF)sizemode align_cornersr5)shaper~weight unsqueezer%jit is_tracingrkrtrreshapepermuterr# interpolateviewcat) rarrrr{r~r|class_pos_embedpatch_pos_embedr5 new_height new_widthsqrt_num_positionss r(interpolate_pos_encodingz-CLIPVisionEmbeddings.interpolate_pos_encodingsv!&&q)A- !44;;EEaH*003a7 yy##%+*F6UZ?**4+<+<= =,QU3,QU3r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy/?;CCr* pixel_valuesc `|j\}}}}|sJ||jk7s||jk7r,td|d|d|jd|jd |jjj }|j|j |}|jdjdd}|jj|dd} tj| |gd } |r| |j| ||z} | S| |j|jz} | S) NzInput image size (*z) doesn't match model (z).)dtyper3rr4r)rrs ValueErrorrzrrtoflatten transposerwrr%rrr~rk) rarr batch_size_rr target_dtype patch_embeds class_embedsrs r(forwardzCLIPVisionEmbeddings.forwards6'3'9'9$ Avu'Vt-F%SWSbSbJb$VHAeW4KDOOK\\]^b^m^m]nnpq ++2288 ++LOO,O,OP #++A.88A> ++22:q"E YY l;C ##d&C&CJPVX]&^^J$d&=&=d>O>O&PPJr*F) rErFrGrrpr%TensorintrrIr __classcell__rs@r(rcrcseq/q,'D5<<'D'DUX'D]b]i]i'DRE$5$5Z_ZfZfr*rcc eZdZdeffd Z d dej dzdej dzdejdzdejfdZ xZ S) CLIPTextEmbeddingsrdcNt||j}tj|j ||_tj|j||_|jdtj|jjddy)NrkrlFrm) rorprqrr} vocab_sizetoken_embeddingmax_position_embeddingsr~rr%r&rrardrrrs r(rpzCLIPTextEmbeddings.__init__s && !||F,=,=yI"$,,v/M/My"Y  ELL)G)GHOOPWXej  r*N input_idsrk inputs_embedsrc8||jdn|jd}|jjjd}||kDrtd|d|||jddd|f}||j |}|j|}||z}|S)Nr4rzRSequence length must be less than max_position_embeddings (got `sequence length`: z and max_position_embeddings: )rr~rrrkr)rarrkr seq_lengthmax_position_embeddingposition_embeddingsrs r(rzCLIPTextEmbeddings.forwards -6,AY__R(}GZGZ[]G^ !%!8!8!?!?!E!Ea!H . .d,<=S> ')   q !=0  ((7 /  =0 r*) rErFrGrrrpr%rrrrIrrrs@r(rrcsVS/.@S|| +,     r*rcleZdZUeed<dZdZdZdZdZ dZ dZ e e dZej dZy)CLIPPreTrainedModelrdclip)imagetextT)rCrDc |jj}t|trt j |j jd|dzt j |jjd|dzt j|jtj|jjdjdnt|tr|jj}t j |j d|j"dz|zt j |j$j|jj&|zt j |jj|jj&|zt j|jtj|j(jdnt|t*r|jj}|j"dzd|jj,zdzz|z}|j"dz|z}t j |j.j|t j |j0j|t j |j2j|t j |j4j|nt|t6r|jj}|jj8dzd|jj,zdzz|z}d|jj8zdz|z}t j |j:j|t j |j<j|nt|t>rt j |j@j|jBdz|jjzt j |jDj|jFdz|jjznGt|tHrZt j |jDj|jj8dz|jjznt|tJrZt j |j@j|jj8dz|jjznst|tLrct j |jNj|jjPj8dz|jjzt|tRjTr>t jV|jXt jZ|jt|tRj\r-|jX t jV|jXy y y ) zInitialize the weightsrg{Gz?)meanstdr4rlr)rr3N)/rdinitializer_factorrZrinitnormal_rrr~copy_rkr%r&rrrcrwrrrzinitializer_ranger|rnum_hidden_layersrrrrrrqrr CLIPModeltext_projectiontext_embed_dimvisual_projectionvision_embed_dimCLIPVisionModelWithProjectionCLIPTextModelWithProjectionCLIPForImageClassification classifier vision_configrrzeros_rjones_r)rarfactor in_proj_std out_proj_stdfc_stds r( _init_weightsz!CLIPPreTrainedModel._init_weightssu// f0 1 LL//66Sftm T LL2299RV W JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h  4 5[[33F LL//cv?O?OQU?UX^?^ _ LL//66FMM<[<[^d LL((//KK++T1DKK4R4RR  ; < LL&&--KK++T1DKK4R4RR  : ; LL!!((KK--994?$++B`B``  fbll + KK $ JJv}} % fbii (V[[-D KK $.E (r*N)rErFrGrrJbase_model_prefixinput_modalitiessupports_gradient_checkpointing_supports_sdpa_supports_flash_attn_supports_flex_attn_supports_attention_backendrr_can_record_outputsr%no_gradrrLr*r(rrs[ (&*#N"&)# U]]_8%8%r*rc`eZdZdZdeffd Z d dejdzdee de fdZ xZ S) CLIPEncoderz Transformer encoder consisting of `config.num_hidden_layers` self attention layers. Each layer is a [`CLIPEncoderLayer`]. Args: config: CLIPConfig rdct|||_tjt |j Dcgc] }t|c}|_d|_ ycc}wNF) rorprdr ModuleListranger rlayersgradient_checkpointing)rardrrs r(rpzCLIPEncoder.__init__sO  mmuVMeMeGf$g!%5f%=$gh &+#%hsA#Nrrrc T|}|jD] }|||fi|}t|S)a7 Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) )rB)r-r )rarrrrC encoder_layers r(rzCLIPEncoder.forwardsH(& ![[ M)M +  r*rY) rErFrGrHrrpr%rrrr rrrs@r(r(r(sK,z,/3  t+ +,    r*r(ceZdZUeed<dZddgZdeffd Zee de dd e jdzd e jdzd e jdzd e ed ef dZxZS)CLIPTextTransformerrdrrrct||||_|j}t ||_t ||_tj||j|_ |j|_ |jyr)rorprdrqrrr(encoderrrrfinal_layer_norm eos_token_id post_initrs r(rpzCLIPTextTransformer.__init__sm   && ,V4"6* " YF>> from transformers import AutoTokenizer, CLIPTextModel >>> model = CLIPTextModel.from_pretrained("openai/clip-vit-base-patch32") >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled (EOS token) states ```rrrkrL)rG)rarrrkrs r(rzCLIPTextModel.forwardhs/0t )%    r*r)rErFrGrrJrrCrprModulerJrNrr%rrrr rrrs@r(rErEQs  -/AB~ :bii:;*..2,0  <<$&  t+ llT)  +,  $   r*rEc eZdZUeed<dZdZdgZdeffd Ze e de d de jdzd edzd eed efd ZxZS)CLIPVisionTransformerrdrrrcBt||||_|j}t ||_t j||j|_ t||_ t j||j|_ |jyr)rorprdrqrcrrrr pre_layrnormr(r5post_layernormr8rs r(rpzCLIPVisionTransformer.__init__sv   && .v6LL8M8MN"6*  ll9&:O:OP r*Fr9Nrrrc | td|j||}|j|}|jdd|i|}|j}|dddddf}|j |}t ||S)Nz You have to specify pixel_values)rrrr=rL)rrrWr5rBrXr )rarrrrCrArBrBs r(rzCLIPVisionTransformer.forwards  ?@ @ Ogh ))-8 +74<<, ', ,  ,==)!Q'2 ++M: )/'  r*r*)rErFrGrrJmain_input_namerrCrprrrr%rIboolrrr rrrs@r(rTrTs $O!+, /  E22605 ''$. #'+ +,  $  3  r*rTzK The vision model from CLIP without any head or projection on top. c eZdZUeed<dZdZdgZdeffd Zde jfdZ e d de jdzd ed eedefd ZxZS) CLIPVisionModelrdrrUrcdt||t||_|j yrY)rorprT vision_modelr8rs r(rpzCLIPVisionModel.__init__s'  1&9 r*rcB|jjjSrYr_rrzr`s r(rJz$CLIPVisionModel.get_input_embeddings  ++;;;r*Nrrc ,|jd||d|S)a( Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, CLIPVisionModel >>> model = CLIPVisionModel.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```rrrL)r_)rarrrs r(rzCLIPVisionModel.forwards.<!t   %%=   r*r*)rErFrGrrJrZrrCrprrRrJrr%rIr[rrr rrrs@r(r]r]s $O!+,/ Q>QX]^<< T[[5W5W(XY r*Nrrrkrrc x|jd|||dd|}|j}|j||_|S)a Examples: ```python >>> import torch >>> from transformers import AutoTokenizer, CLIPModel >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> with torch.inference_mode(): ... text_features = model.get_text_features(**inputs) ```T)rrrk return_dictrL)rGr>r)rarrrkr text_outputsrBs r(get_text_featureszCLIPModel.get_text_featuressV04C4??4 )% 4  4 %22 %)%9%9-%H "r*rrc v|jd||dd|}|j}|j||_|S)a Examples: ```python >>> import torch >>> from transformers import AutoProcessor, CLIPModel >>> from transformers.image_utils import load_image >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = load_image(url) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.inference_mode(): ... image_features = model.get_image_features(**inputs) ```T)rrrqrL)r_r>r)rarrrvision_outputsrBs r(get_image_featureszCLIPModel.get_image_features<sU66GT5F5F6 %%=6  6  '44 '+'='=m'L$r* return_lossc L|jd||d|}|jd|||d|} |j} |j| } | j} |j | } | t | z } | t | z } t j| | jj| j} | |jjj| jz} | j} d}|r t| }t|| | | | | |S)a return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. Examples: ```python >>> import torch >>> from transformers import AutoProcessor, CLIPModel >>> from transformers.image_utils import load_image >>> model = CLIPModel.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = load_image(url) >>> inputs = processor( ... text=["a photo of a cat", "a photo of a dog"], images=image, return_tensors="pt", padding=True ... ) >>> with torch.inference_mode(): ... outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```rdrQN)rRrSrTrOrArUrVrL)r_rGr>rrr=r%rr-rr"roexpr0rQ)rarrrrkrwrrrurrrArOrTrSrRs r(rzCLIPModel.forwardbsIL6GT5F5F6 %%=6 6  4C4??4 )%4  4 &33 --l; "00 **;7 $&6|&DD !$4[$AA  ,,{LNN4D4G4G HZHZ4[\)D,<,<,@,@,B,E,EkFXFX,YY*,,. _-D-+#%* .  r*NNr)NNNNNF)rErFrGrrJrCrprrr%rrrrKr rsrIr[rvrrQrrrs@r(r r s Z!z!F/3,0 <<  t+ llT)  +,  + +  D*/"''"#'"+, " + + ""H.215.204#').M ##d*M ''$.M  t+ M &&- M D[ M #'M +,M  M M r*r c eZdZUeed<dZddgZdeffd ZdejfdZ dZ e e dd ejd zd ejd zd ejd zd eedef dZxZS)rrdr3rrct||tj|}|j|_t j |j|jd|_ |jyNFrh) rorprErmrGrrrqrlrr8)rardrGrs r(rpz$CLIPTextModelWithProjection.__init__s[  "//7 $//!yy););V=R=RY^_ r*rcB|jjjSrYrIr`s r(rJz0CLIPTextModelWithProjection.get_input_embeddingsrKr*c:||jj_yrYrIrMs r(rNz0CLIPTextModelWithProjection.set_input_embeddingsrOr*Nrrrkrc |jd|||d|}|j}|j|}t||j|j |j S)a@ Examples: ```python >>> import torch >>> from transformers import AutoTokenizer, CLIPTextModelWithProjection >>> model = CLIPTextModelWithProjection.from_pretrained("openai/clip-vit-base-patch32") >>> tokenizer = AutoTokenizer.from_pretrained("openai/clip-vit-base-patch32") >>> inputs = tokenizer(["a photo of a cat", "a photo of a dog"], padding=True, return_tensors="pt") >>> with torch.inference_mode(): ... outputs = model(**inputs) >>> text_embeds = outputs.text_embeds ```rQ)rOrBrCrDrL)rGr>rrNrBrCrD)rarrrkrrrrBrOs r(rz#CLIPTextModelWithProjection.forwardsu44C4??4 )%4  4 %22 **=9 "#*<<&44#..   r*r)rErFrGrrJrrCrprrRrJrNrrr%rrrrNrrrs@r(rrs  -/AB ~ :bii:;*..2,0 & <<$&&  t+& llT) & +, &  & & r*rc eZdZUeed<dZdZdeffd ZdejfdZ e e d de jdzded eedefd ZxZS) rrdrrUct||tj|}|j|_t j |j|jd|_ |jyr}) rorpr]rmr_rrrqrlrr8rardr_rs r(rpz&CLIPVisionModelWithProjection.__init__s\  &33F; (55!#6+=+=v?T?T[`!a r*rcB|jjjSrYrar`s r(rJz2CLIPVisionModelWithProjection.get_input_embeddingsrbr*Nrrc |jd||d|}|j}|j|}t||j|j |j S)a Examples: ```python >>> import torch >>> from transformers import AutoProcessor, CLIPVisionModelWithProjection >>> from transformers.image_utils import load_image >>> model = CLIPVisionModelWithProjection.from_pretrained("openai/clip-vit-base-patch32") >>> processor = AutoProcessor.from_pretrained("openai/clip-vit-base-patch32") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = load_image(url) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.inference_mode(): ... outputs = model(**inputs) >>> image_embeds = outputs.image_embeds ```rd)rArBrCrDrL)r_r>rr@rBrCrD)rarrrrurBrAs r(rz%CLIPVisionModelWithProjection.forward st:6GT5F5F6 %%=6 6  '44 --m< $%,>>(66%00   r*r*)rErFrGrrJrZrrprrRrJrrr%rIr[rrr@rrrs@r(rrs $O! /