i^ dZddlmZddlmZddlmZddlZddlmZm Z ddl m Z dd l mZdd lmZdd lmZdd lmZmZdd lmZmZddlmZddlmZmZmZmZm Z m!Z!ddl"m#Z#m$Z$ddl%m&Z&ddl'm(Z(m)Z)m*Z*erddl+m,Z,e jZe.Z/dejdejfdZ0dejdejfdZ1eeGddeZ2dedefdZ3dedefdZ4d Z5d!Z6eed"#Gd$d%eZ7eed&#Gd'd(eZ8Gd)d*e jrZ:Gd+d,e jrZ; dQd-e jrd.ejd/ejd0ejd1ejdzd2eGd8d9e jrZ?Gd:d;eZ@eGd<d=eZAGd>d?e jrZBGd@dAeAZCGdBdCeAZDGdDdEeAZEGdFdGeAZFeGdHdIeAZGGdJdKe jrZHGdLdMe jrZIGdNdOeAZJgdPZKy)RzPyTorch OWLv2 model.)Callable) dataclass)AnyN)Tensornn)initialization)ACT2FN)create_causal_mask)GradientCheckpointingLayer)BaseModelOutputBaseModelOutputWithPooling)ALL_ATTENTION_FUNCTIONSPreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringis_vision_availablelogging torch_int)can_return_tuplemerge_with_config_defaults)capture_outputs) Owlv2ConfigOwlv2TextConfigOwlv2VisionConfig)center_to_corners_formatlogitsreturnctjj|tjt ||j S)Ndevice)r functional cross_entropytorcharangelenr$)r s y/var/www/vps2.regionflexible.com/Desarrollo/venv/lib/python3.12/site-packages/transformers/models/owlv2/modeling_owlv2.pycontrastive_lossr+6s/ == & &vu||CKPVP]P]/^ __ similaritycZt|}t|j}||zdz S)Ng@)r+t)r- caption_loss image_losss r* owlv2_lossr2;s,#J/L!*,,.1J : % ,,r,ceZdZUdZdZej dzed<dZej dzed<dZ ej dzed<dZ ej dzed<dZ ej dzed<dZ e ed<dZe ed <d eefd Zy) Owlv2Outputa loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `return_loss` is `True`): Contrastive loss for image-text similarity. logits_per_image (`torch.FloatTensor` of shape `(image_batch_size, text_batch_size)`): The scaled dot product scores between `image_embeds` and `text_embeds`. This represents the image-text similarity scores. logits_per_text (`torch.FloatTensor` of shape `(text_batch_size, image_batch_size)`): The scaled dot product scores between `text_embeds` and `image_embeds`. This represents the text-image similarity scores. text_embeds (`torch.FloatTensor` of shape `(batch_size * num_max_text_queries, output_dim`): The text embeddings obtained by applying the projection layer to the pooled output of [`Owlv2TextModel`]. image_embeds (`torch.FloatTensor` of shape `(batch_size, output_dim`): The image embeddings obtained by applying the projection layer to the pooled output of [`Owlv2VisionModel`]. text_model_output (tuple[`BaseModelOutputWithPooling`]): The output of the [`Owlv2TextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`Owlv2VisionModel`]. Nlosslogits_per_imagelogits_per_text text_embeds image_embedstext_model_outputvision_model_outputr!cHtfdjDS)Nc3dK|]'}|dvr|nt|j)yw)r:r;Ngetattrto_tuple.0kselfs r* z'Owlv2Output.to_tuple..a=  LLDGRYZ^`aRbRkRkRm m -0tuplekeysrEs`r*rAzOwlv2Output.to_tuple`# YY[   r,)__name__ __module__ __qualname____doc__r5r' FloatTensor__annotations__r6r7r8r9r:rr;rJrrAr,r*r4r4As(&*D%  d ")15e''$.504OU&&-4,0K""T)0-1L%##d*148186:3: %* r,r4r/c|jr>|jtjtjfvr|S|j S|jtj tjfvr|S|jSN) is_floating_pointdtyper'float32float64floatint32int64int)r/s r*_upcastr_hs`GG u}}==qL1779LGG U[[99qFquuwFr,boxescft|}|dddf|dddfz |dddf|dddfz zS)a Computes the area of a set of bounding boxes, which are specified by its (x1, y1, x2, y2) coordinates. Args: boxes (`torch.FloatTensor` of shape `(number_of_boxes, 4)`): Boxes for which the area will be computed. They are expected to be in (x1, y1, x2, y2) format with `0 <= x1 < x2` and `0 <= y1 < y2`. Returns: `torch.FloatTensor`: a tensor containing the area for each box. Nrrr)r_)r`s r*box_arearcqsB ENE !Q$K%1+ %%1+ad *C DDr,c^t|}t|}tj|dddddf|ddddf}tj|dddddf|ddddf}||z j d}|dddddf|dddddfz}|dddf|z|z }||z } | |fS)Nrbrminr)rcr'maxrfclamp) boxes1boxes2area1area2left_top right_bottom width_heightinterunionious r*box_iourss V E V Eyy4! ,fQUmr?rBs r*rFz6Owlv2ObjectDetectionOutput.to_tuple..rGrHrIrLs`r*rAz#Owlv2ObjectDetectionOutput.to_tuplerMr,)rNrOrPrQr5r'rRrSr~dictr rrr8r9rr:rr;rJrrArTr,r*r}r}s>&*D%  d ")!Itd{!'+FE   $+26u((4/6+/J!!D(/,0K""T)0-1L%##d*1-1L%##d*148186:3: %* r,r}zL Output type of [`Owlv2ForObjectDetection.image_guided_detection`]. c0eZdZUdZdZej dzed<dZej dzed<dZ ej dzed<dZ ej dzed<dZ ej dzed<dZ ej dzed<dZ eed <dZeed <d eefd Zy) %Owlv2ImageGuidedObjectDetectionOutputa logits (`torch.FloatTensor` of shape `(batch_size, num_patches, num_queries)`): Classification logits (including no-object) for all queries. image_embeds (`torch.FloatTensor` of shape `(batch_size, patch_size, patch_size, output_dim`): Pooled output of [`Owlv2VisionModel`]. OWLv2 represents images as a set of image patches and computes image embeddings for each patch. query_image_embeds (`torch.FloatTensor` of shape `(batch_size, patch_size, patch_size, output_dim`): Pooled output of [`Owlv2VisionModel`]. OWLv2 represents images as a set of image patches and computes image embeddings for each patch. target_pred_boxes (`torch.FloatTensor` of shape `(batch_size, num_patches, 4)`): Normalized boxes coordinates for all queries, represented as (center_x, center_y, width, height). These values are normalized in [0, 1], relative to the size of each individual target image in the batch (disregarding possible padding). You can use [`~Owlv2ImageProcessor.post_process_object_detection`] to retrieve the unnormalized bounding boxes. query_pred_boxes (`torch.FloatTensor` of shape `(batch_size, num_patches, 4)`): Normalized boxes coordinates for all queries, represented as (center_x, center_y, width, height). These values are normalized in [0, 1], relative to the size of each individual query image in the batch (disregarding possible padding). You can use [`~Owlv2ImageProcessor.post_process_object_detection`] to retrieve the unnormalized bounding boxes. class_embeds (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`): Class embeddings of all image patches. OWLv2 represents images as a set of image patches where the total number of patches is (image_size / patch_size)**2. text_model_output (tuple[`BaseModelOutputWithPooling`]): The output of the [`Owlv2TextModel`]. vision_model_output (`BaseModelOutputWithPooling`): The output of the [`Owlv2VisionModel`]. Nr r9query_image_embedstarget_pred_boxesquery_pred_boxesrr:r;r!cHtfdjDS)Nc3dK|]'}|dvr|nt|j)ywr>r?rBs r*rFzAOwlv2ImageGuidedObjectDetectionOutput.to_tuple..rGrHrIrLs`r*rAz.Owlv2ImageGuidedObjectDetectionOutput.to_tuplerMr,)rNrOrPrQr r'rRrSr9rrrrr:rr;rJrrArTr,r*rrs8(,FE   $+-1L%##d*137))D0726u((4/615e''$.5-1L%##d*148186:3: %* r,rceZdZdeffd Zdej dededej fdZd dejd e dej fd Z xZ S) Owlv2VisionEmbeddingsconfigct||j|_||_|j|_t jtj|j|_ t j|j|j |j|jd|_ |j|jzdz|_|jdz|_t j"|j |j |_|j'dtj(|j j+ddy)NF) in_channels out_channels kernel_sizestridebiasrbr position_idsr persistent)super__init__ patch_sizer hidden_size embed_dimr Parameterr'randnclass_embeddingConv2d num_channelspatch_embedding image_size num_patches num_positions Embeddingposition_embeddingregister_bufferr(expandrEr __class__s r*rzOwlv2VisionEmbeddings.__init__s  ++ ++!||EKK8J8J,KL!yy++))$$  #--1B1BBqH!--1"$,,t/A/A4>>"R ^U\\$:L:L-M-T-TU\-]jopr, embeddingsheightwidthr!c|jddz }|jjjd}|jddz }tj j s%||k(r ||k(r|j|jS|ddddf}|ddddf}|jd} ||jz} ||jz} t|dz} |jd| | | }|jdddd}tjj|| | fdd }|jddddjdd| }t j ||fd S) a This method allows to interpolate the pre-trained position encodings, to be able to use the model on higher resolution images. This method is also adapted to support torch.jit tracing. Adapted from: - https://github.com/facebookresearch/dino/blob/de9ee3df6cf39fac952ab558447af1fa1365362a/vision_transformer.py#L174-L194, and - https://github.com/facebookresearch/dinov2/blob/e1277af2ba9496fbadf7aec6eba56e8d882d1e35/dinov2/models/vision_transformer.py#L179-L211 rrNrg?rrbbicubicF)sizemode align_cornersdim)shaperweight unsqueezer'jit is_tracingrrrreshapepermuterr% interpolateviewcat) rErrrrrrclass_pos_embedpatch_pos_embedr new_height new_widthsqrt_num_positionss r*interpolate_pos_encodingz.Owlv2VisionEmbeddings.interpolate_pos_encoding.sv!&&q)A- !44;;EEaH*003a7 yy##%+*F6UZ?**4+<+<= =,QU3,QU3r"t. T__, &}c'9:)11!5GI[]`a)11!Q1=--33 i( 4 *11!Q1=BB1b#Nyy/?;CCr, pixel_valuesrch|j\}}}}|j|}|jdjdd}|jj |dd}t j||gd} |r| |j| ||z} | S| |j|jz} | S)Nrbrrr) rrflatten transposerrr'rrrr) rErr batch_size_rr patch_embedsrrs r*forwardzOwlv2VisionEmbeddings.forwardWs'3'9'9$ Avu++L9 #++A.88A> ++22:q"E YY l;C ##d&C&CJPVX]&^^J$d&=&=d>O>O&PPJr,F) rNrOrPrrr'rr^rrRboolr __classcell__rs@r*rrsmq0q*'D5<<'D'DUX'D]b]i]i'DR E$5$5 QU bgbnbn r,rc eZdZdeffd Z d dej dzdej dzdejdzdejfdZ xZ S) Owlv2TextEmbeddingsrc^t|tj|j|j |_tj|j|j |_|jdtj|jjddy)NrrFr) rrrr vocab_sizertoken_embeddingmax_position_embeddingsrrr'r(rrs r*rzOwlv2TextEmbeddings.__init__gs !||F,=,=v?Q?QR"$,,v/M/MvOaOa"b  ELL)G)GHOOPWXej  r,N input_idsr inputs_embedsr!c||jdn|jd}||jddd|f}||j|}|j|}||z}|S)Nr)rrrr)rErrr seq_lengthposition_embeddingsrs r*rzOwlv2TextEmbeddings.forwardqs{ -6,AY__R(}GZGZ[]G^  ,,Q  ^L==((6??([L,,|U3K''1-88:K  $$r,c eZdZdZfdZ d dej dej dzdeede ej ej dzffdZ xZ S) Owlv2Attentionz=Multi-headed attention from 'Attention Is All You Need' paperct|||_|j|_|j |_|j|j z|_|j|j z|jk7r&td|jd|j d|jdz|_ |j|_ d|_ tj|j|j|_tj|j|j|_tj|j|j|_tj|j|j|_y)Nz;embed_dim must be divisible by num_heads (got `embed_dim`: z and `num_heads`: z).rF)rrrrrnum_attention_heads num_headshead_dimrvscaleattention_dropoutr is_causalrLineark_projv_projq_projout_projrs r*rzOwlv2Attention.__init__s  ++33$..8 ==4>> )T^^ ;MdnnM]^NN#2' ]]D( // ii? ii? ii?  $..$..A r,N hidden_statesrrr!c |jdd}g|d|j}|j|j|j dd}|j |j|j dd}|j |j|j dd}tj|jjt} | |||||f|j|jsdn |jd|\} } | jg|dj!} |j#| } | | fS)Nrrrb)rr)rrrrrrrr get_interfacer_attn_implementationrrrrrrr) rErrr input_shape hidden_shape query_states key_states value_statesattention_interfacerrs r*rzOwlv2Attention.forwardsQ $))#2. 88b8$--8 6t{{=166 EOOPQSTU 4T[[/44lCMMaQRS 6t{{=166 EOOPQSTU (?(M(M KK , ,.E) %8      % JJ#}}C$,, %  % ! \*k));;;;FFH mmK0 L((r,rV) rNrOrPrQrr'rrrrJrrrs@r*rrsfGB./3)||) t+)+, ) u||U\\D00 1 )r,rcVeZdZfdZdej dej fdZxZS)Owlv2MLPct|||_t|j|_t j|j|j|_ t j|j|j|_ yrV) rrrr hidden_act activation_fnrrrintermediate_sizefc1fc2rs r*rzOwlv2MLP.__init__sd  #F$5$5699V//1I1IJ99V55v7I7IJr,rr!cl|j|}|j|}|j|}|SrV)rrr)rErs r*rzOwlv2MLP.forwards4/ **=9 / r,)rNrOrPrr'rrrrs@r*rrs$KU\\ellr,rceZdZdeezffd Zdejdejdee dejfdZ xZ S)Owlv2EncoderLayerrcDt||j|_t ||_t j|j|j|_ t||_ t j|j|j|_ yNeps) rrrrr self_attnr LayerNormlayer_norm_eps layer_norm1rmlp layer_norm2rs r*rzOwlv2EncoderLayer.__init__sm ++'/<<F> ')   q !=0  ((7 /  =0 r,) rNrOrPrrrr'rrrrRrrrs@r*rrsVS0?BS|| +,     r,rceZdZUeed<dZdZdZdZdZ dZ dZ dgZ e edZddgZej$d ej(fd Zy ) Owlv2PreTrainedModelrowlv2)imagetextTr)r attentionsz&.*text_model\.embeddings\.position_idsz(.*vision_model\.embeddings\.position_idsrc$ |jj}t|trt j |j jd|dzt j |jjd|dzt j|jtj|jjdjdnt|trt j |j d|j"dz|zt j |j$j|jj&|zt j |jj|jj&|zt j|jtj|jjdjdnt|t(r|j"dzd|jj*zdzz|z}|j"dz|z}t j |j,j|t j |j.j|t j |j0j|t j |j2j|nt|t4r|jj6dzd|jj*zdzz|z}d|jj6zdz|z}t j |j8j|t j |j:j|nt|t<rt j |j>j|j@dz|zt j |jBj|jDdz|zt jF|jH|jjJnTt|tLrDt j|jN|jQ|jR|jTt|tVjXr>t jZ|j\t j^|jt|tVj`rOt j |jd||j\ t jZ|j\y y y ) zInitialize the weightsrg{Gz?)meanstdrrr)r,rbN)1rinitializer_factor isinstancerinitnormal_rrrcopy_rr'r(rrrrrrinitializer_rangernum_hidden_layersrrrrrrrr Owlv2Modeltext_projectiontext_embed_dimvisual_projectionvision_embed_dim constant_ logit_scalelogit_scale_init_valueOwlv2ForObjectDetectionbox_biascompute_box_biasnum_patches_heightnum_patches_widthrrzeros_rones_r)rErfactor in_proj_std out_proj_stdfc_stds r* _init_weightsz"Owlv2PreTrainedModel._init_weights#s// f1 2 LL//66Sftm T LL2299RV W JJv**ELL9L9L9R9RSU9V,W,^,^_f,g h  5 6 LL//cv?O?OQU?UX^?^ _ LL//66FMM<[<[^d$hi &+#%isA#Nrrr!c T|}|jD] }|||fi|}t|S)a7 Args: inputs_embeds (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Optionally, instead of passing `input_ids` you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert `input_ids` indices into associated vectors than the model's internal embedding lookup matrix. attention_mask (`torch.Tensor` of shape `(batch_size, sequence_length)`, *optional*): Mask to avoid performing attention on padding token indices. Mask values selected in `[0, 1]`: - 1 for tokens that are **not masked**, - 0 for tokens that are **masked**. [What are attention masks?](../glossary#attention-mask) )last_hidden_state)rZr )rErrrr encoder_layers r*rzOwlv2Encoder.forwardasH(& ![[ M)M +  r,rV) rNrOrPrQrrr'rrrr rrrs@r*rUrURsK,{,/3  t+ +,    r,rUceZdZdeffd Zeede d dejdzdejdzdejdzd e e d e e zf d ZxZS) Owlv2TextTransformerrct|||j}t||_t ||_tj||j|_ |jyr) rrrrrrUencoderrrrfinal_layer_norm post_init)rErrrs r*rzOwlv2TextTransformer.__init__sX  && -f5#F+ " YF>> from transformers import AutoProcessor, Owlv2TextModel >>> model = Owlv2TextModel.from_pretrained("google/owlv2-base-patch16") >>> processor = AutoProcessor.from_pretrained("google/owlv2-base-patch16") >>> inputs = processor( ... text=[["a photo of a cat", "a photo of a dog"], ["photo of a astranaut"]], return_tensors="pt" ... ) >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled (EOS token) states ```rrrT)rs)rErrrs r*rzOwlv2TextModel.forwards,6t )   r,NN)rNrOrPrrSrIrrrSrvrxrr'rrrrJrrrrs@r*rqrqs   :bii:;*..2 <<$&  t+ +,  + +   r,rqc eZdZdeffd Zeede d dejde dzde e d e ezfd ZxZS) Owlv2VisionTransformerrcDt||t||_t j |j |j|_t||_ t j |j |j|_ |jyr) rrrrrrrr pre_layernormrUrbpost_layernormrdrs r*rzOwlv2VisionTransformer.__init__sr  /7\\&*<*<&BWBWX#F+  ll6+=+=6CXCXY r,FrerrNrr!c T|jjjj}|j |}|j||}|j |}|j dd|i|}|j}|dddddf}|j|}t||S)N)rrrrirT) rrrrXrlrrbr]rr) rErrrexpected_input_dtyperrnr]ros r*rzOwlv2VisionTransformer.forwards $>>EEKK#';<  Ogh **=9 +74<<, ', ,  ,==)!Q'2 ++M: )/'  r,r)rNrOrPrrrrrr'rRrrrrJrrrrs@r*r}r}sz 0  E216 '' #'+ +,  + +  3  r,r}c eZdZUeed<dZdZdeffd ZdejfdZ e d de jdzded eedefd ZxZS) Owlv2VisionModelrr)r'cdt||t||_|j yrV)rrr} vision_modelrdrs r*rzOwlv2VisionModel.__init__'s'  26: r,r!cB|jjjSrV)rrrrLs r*rvz%Owlv2VisionModel.get_input_embeddings-s  ++;;;r,Nrrc ,|jd||d|S)a# Examples: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, Owlv2VisionModel >>> model = Owlv2VisionModel.from_pretrained("google/owlv2-base-patch16") >>> processor = AutoProcessor.from_pretrained("google/owlv2-base-patch16") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> last_hidden_state = outputs.last_hidden_state >>> pooled_output = outputs.pooler_output # pooled CLS states ```rrrT)r)rErrrs r*rzOwlv2VisionModel.forward0s.8!t   %%=   r,rW)rNrOrPrrSmain_input_namerIrrrSrvrr'rRrrrrrrrs@r*rr"s $O!0 Q>QX]^<< V5R5R(ST r,Nrrrr!c t|jd||d|}|j}|j||_|S)a input_ids (`torch.LongTensor` of shape `(batch_size * num_max_text_queries, sequence_length)`): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids) Examples: ```python >>> import torch >>> from transformers import AutoProcessor, Owlv2Model >>> model = Owlv2Model.from_pretrained("google/owlv2-base-patch16-ensemble") >>> processor = AutoProcessor.from_pretrained("google/owlv2-base-patch16-ensemble") >>> inputs = processor( ... text=[["a photo of a cat", "a photo of a dog"], ["photo of a astranaut"]], return_tensors="pt" ... ) >>> with torch.inference_mode(): ... text_features = model.get_text_features(**inputs) ```rzrT)rsrjr5)rErrr text_outputsros r*get_text_featureszOwlv2Model.get_text_featureslsP64C4??4 )4 4 %22 %)%9%9-%H "r,rrc p|jd||d|}|j|j|_|S)a Examples: ```python >>> import torch >>> from transformers.image_utils import load_image >>> from transformers import AutoProcessor, Owlv2Model >>> model = Owlv2Model.from_pretrained("google/owlv2-base-patch16-ensemble") >>> processor = AutoProcessor.from_pretrained("google/owlv2-base-patch16-ensemble") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> image = load_image(url) >>> inputs = processor(images=image, return_tensors="pt") >>> with torch.inference_mode(): ... image_features = model.get_image_features(**inputs) ```rrT)rr7rj)rErrrvision_outputss r*get_image_featureszOwlv2Model.get_image_featuressM26GT5F5F6 %%=6 6  (,'='=n>Z>Z'[$r, return_lossreturn_base_image_embedsc ||jd ||d|}|jd ||d|} | j} |j| } |j} |j | } | t j j| dddz } | t j j| dddz } |jjj| j} t j| | j| z}|j}d}|r t|}| } t|||| | | |S) a return_loss (`bool`, *optional*): Whether or not to return the contrastive loss. return_base_image_embeds (`bool`, *optional*): Whether or not to return the base image embeddings. Examples: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, Owlv2Model >>> model = Owlv2Model.from_pretrained("google/owlv2-base-patch16-ensemble") >>> processor = AutoProcessor.from_pretrained("google/owlv2-base-patch16-ensemble") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(text=[["a photo of a cat", "a photo of a dog"]], images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> logits_per_image = outputs.logits_per_image # this is the image-text similarity score >>> probs = logits_per_image.softmax(dim=1) # we can take the softmax to get the label probabilities ```rrzrbrT)ordrkeepdimN)r5r6r7r8r9r:r;rT)rrsrjr5r7r'linalgnormr:exprlr$rr/r2r4)rErrrrrrrrrr8r9text_embeds_normr:r7r6r5s r*rzOwlv2Model.forwardsdF6GT5F5F6 %%=6 6 4C4??4 )4 4 #00 **;7 %33 --l; $ell&7&7 !QS]a&7&bb &):):;ASU_c):)dd&&**,// 0C0CD ,,'79IJ[X*,,. o.D& -+#%* .  r,rVr)NNNNFN)rNrOrPrrSrrrr'rrrrJrrrrrrRr4rrrs@r*r4r4Ss {(/3!<<! t+!+, ! + + !!F*/ll#'+,  + + @.215.2#').04K ##d*K ''$.K  t+ K D[ K #' K #'+K +,K   K K r,r4cbeZdZddedeffd ZdejdejfdZ xZ S)Owlv2BoxPredictionHeadrout_dimc"t||jj}t j |||_t j |||_t j|_ t j |||_ yrV) rrrrrrdense0dense1GELUgeludense2)rErrrrs r*rzOwlv2BoxPredictionHead.__init__sb $$00iiu- iiu- GGI iiw/ r,image_featuresr!c|j|}|j|}|j|}|j|}|j|}|SrV)rrrr)rEroutputs r*rzOwlv2BoxPredictionHead.forwardsM^,6"V$6"V$ r,)) rNrOrPrr^rr'rrRrrrs@r*rrs30{0S0ellu7H7Hr,rc eZdZdeffd Zdej dej dzdejdzdeej fdZ xZ S) Owlv2ClassPredictionHeadrct||jj}|jj|_t j|j ||_t j|j d|_ t j|j d|_ t j|_ y)Nr) rrrrr query_dimrrr logit_shiftr:ELUelu)rErrrs r*rz!Owlv2ClassPredictionHead.__init__s $$00--99ii8 99T^^Q799T^^Q7668r,r9 query_embedsN query_maskr!c0|j|}|S|j}|jdd\}}tj|||j fj |}||fS|tjj|dddzz }|tjj|dddzz }tjd||}|j|} |j|} |j| dz} || z| z}||jdkDrtj|d }tj|d k(tj |j"j$|}|j tj&}||fS) NrbrT)rrgư>z...pd,...qd->...pqrrrr)rr$rr'zerosrrlrreinsumrr:rndimrwherefinforXrfrY) rEr9rrimage_class_embedsr$rr pred_logitsrr:s r*rz Owlv2ClassPredictionHead.forward$s "[[6  '..F&8&>&>r&B #J ++z;&OPSSTZ[K!34 405<<3D3DEW]_im3D3nqu3uv#u||'8'82W['8'\_c'cd ll#79K\Z &&|4 &&|4 hh{+a/ "[0K?  !""__ZR@ ++jAou{{;CTCT7U7Y7Y[fgK%..7K/00r,) rNrOrPrrr'rRrrJrrrs@r*rrs_ { !1''!1''$.!1LL4' !1 u  ! !1r,rceZdZUeed<deffd ZedededejfdZ dejdejfdZ dededejfd Z dd ejd ejd edejfd Z dd ejdejdzdejdzdeejfdZ ddejdejdejd edeedeejf dZ ddejd edeedeejfdZ ddejdejd edejfdZee d dejdejdzd edeedef dZee d dejdejdejdzd edeedef dZxZS)!r<rct||t||_t ||_t ||_t |d|_tj|jj|jj|_tj|_||_|j"jj$|j"jj&z|_|j"jj$|j"jj&z|_|j-d|j/|j(|j*d|j1y)Nr)rrr=Fr)rrr4r&r class_headrbox_headobjectness_headrrrrr layer_normSigmoidsigmoidrrrr?r@rr>rdrs r*rz Owlv2ForObjectDetection.__init__Ks  ' 26:.v6 5faH,,v';';'G'GVMaMaMpMpqzz|  "&++";";"F"F$++JcJcJnJn"n!%!:!:!E!EIbIbImIm!m  --d.E.EtG]G]^kp  r,r?r@r!cjtjd|dztj}tjd|dztj}tj||d\}}tj||fd}|dxx|zcc<|dxx|zcc<|j dd }|S) Nr)rXxy)indexingrr.r.rrb)r'r(rYmeshgridstackr)r?r@ x_coordinates y_coordinatesxxyybox_coordinatess r*!normalize_grid_corner_coordinatesz9Owlv2ForObjectDetection.normalize_grid_corner_coordinates_s Q(9A(=U]]S  Q(:Q(>emmT  }tLB ++r2hB7#44#55*..r15r,rcR|j}|j|}|d}|S)a#Predicts the probability that each image feature token is an object. Args: image_features (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_dim)`)): Features extracted from the image. Returns: Objectness scores. r)detachr)rErrs r*objectness_predictorz,Owlv2ForObjectDetection.objectness_predictorqs4(..0 00@-f5  r,c|j||}tj|dd}tj|dztj| dzz }tj |d}|dxx|zcc<|dxx|zcc<tj|dztj| dzz }tj ||gd}|S)Nrg?g-C6?rrrr)rr'cliploglog1p full_liker)rEr?r@rbox_coord_biasbox_size box_size_biasr=s r*r>z(Owlv2ForObjectDetection.compute_box_biass@@ASUfg**_c3??T#9:U[[/IY\`I`=aa??>37--.. (T/2U[[(TAQ5RR 99nm<"Er, image_feats feature_maprc|j|}|r$|j\}}}}|j||}n |j}|j |j }||z }|j |}|S)a Args: image_feats: Features extracted from the image, returned by the `image_text_embedder` method. feature_map: A spatial re-arrangement of image_features, also returned by the `image_text_embedder` method. interpolate_pos_encoding: Whether to interpolate the pre-trained position encodings. Returns: pred_boxes: List of predicted boxes (cxcywh normalized to 0, 1) nested within a dictionary. )rrr>r=rlr$r) rErrrrrr?r@r=s r* box_predictorz%Owlv2ForObjectDetection.box_predictors|&]];/  $:E:K:K 7A!#4a,,-?ARSH}}H;;{112h \\*- r,Nrrc6|j|||\}}||fS)a8 Args: image_feats: Features extracted from the `image_text_embedder`. query_embeds: Text query embeddings. query_mask: Must be provided with query_embeddings. A mask indicating which query embeddings are valid. )r)rErrrrrs r*class_predictorz'Owlv2ForObjectDetection.class_predictors)-1OOKWa,b)(/00r,rrrrc |jd||||d|}|rX|j\}}}} ||jjjz} | |jjjz} n|j } |j } |jd} |jjj| } tj| ddddddf| ddddfj}| ddddddf|z} |j| } | jd| | | jdf}| j|} |d}|| |fS)N)rrrrrrrrT)r&rrrrr?r@r;rrr' broadcast_torr)rErrrrroutputsrrrr?r@r]r9class_token_outnew_sizer8s r*image_text_embedderz+Owlv2ForObjectDetection.image_text_embeddersy$** %)%=     $"."4"4 Aq&%!'4;;+D+D+O+O!O  %)B)B)M)M M !%!8!8  $ 6 6 $77:zz..==>OP  ,,\!RaR(-C\RSUXVXUXRXEYE_E_`$Aqr1H-? |4    q !     r "   $++H5 bk \733r,c |jjd||d|}|rX|j\}}}}||jjj z}||jjj z} n|j }|j} |d} |jjj| } tj| ddddddf| ddddfj} | ddddddf| z} |j| } | jd|| | jdf} | j| } | |fS)NrrrrrT) r&rrrrrr?r@rr'rrr)rErrrrrrrr?r@r]r9rrs r*image_embedderz&Owlv2ForObjectDetection.image_embeddersg6MTZZ5L5L6 %@X6 \b6  $"."4"4 Aq&%!'4;;+D+D+O+O!O  %)B)B)M)M M !%!8!8  $ 6 6 +1-zz..==>OP  ,,\!RaR(-C\RSUXVXUXRXEYE_E_`$Aqr1H-? |4    q !     r "   $++H5 n--r,query_image_featuresquery_feature_mapcj|j|\}}|j|||}t|}g}g} |j} t |j dD]} t jgdg| } || } t| | \}}t j|ddk(r t| | }t j|dz}|d|k\j}|js|| |jd}t j|| d}t j d||}|t j"|}|j%|| || j%|"|r+t j&|}t j&| }nd \}}|||fS) Nr)rrrrr#rg?r)axiszd,id->ir{)rrrr$rYrr'rrsrurzrgnonzeronumelsqueezer+rargminappendr)rErrrrrrpred_boxes_as_cornersbest_class_embedsbest_box_indicespred_boxes_deviceieach_query_boxeach_query_pred_boxesious iou_threshold selected_indsselected_embeddings mean_embedsmean_sim best_box_indr box_indicess r*embed_image_queryz)Owlv2ForObjectDetection.embed_image_query#s ../CD<''(<>OQij 8 D188+11!45 6A"\\<.ARSN$9!$< !n.CDGD!yyaC(*>;PQ"IIdOc1M!!W 5>>@M""$&21om6K6KA6N&O##jjaqA  << ;@ST,U\\(-CD !((a)FG '' 5' 6*  ;;'89L++&67K(2 %L+[*44r,query_pixel_valuesc |j||d}|jd||d|\}}|j\}} } } tj||| | z| f} |j\}} } } tj||| | z| f} |j | ||\}}}|j | |\}}|j | ||}t||||||d|S)a query_pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Pixel values of query image(s) to be detected. Pass in one query image per target image. Examples: ```python >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> import torch >>> from transformers import AutoProcessor, Owlv2ForObjectDetection >>> processor = AutoProcessor.from_pretrained("google/owlv2-base-patch16-ensemble") >>> model = Owlv2ForObjectDetection.from_pretrained("google/owlv2-base-patch16-ensemble") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> query_url = "http://images.cocodataset.org/val2017/000000001675.jpg" >>> with httpx.stream("GET", query_url) as response: ... query_image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, query_images=query_image, return_tensors="pt") >>> # forward pass >>> with torch.no_grad(): ... outputs = model.image_guided_detection(**inputs) >>> target_sizes = torch.Tensor([image.size[::-1]]) >>> # Convert outputs (bounding boxes and class logits) to Pascal VOC format (xmin, ymin, xmax, ymax) >>> results = processor.post_process_image_guided_detection( ... outputs=outputs, threshold=0.9, nms_threshold=0.3, target_sizes=target_sizes ... ) >>> i = 0 # Retrieve predictions for the first image >>> boxes, scores = results[i]["boxes"], results[i]["scores"] >>> for box, score in zip(boxes, scores): ... box = [round(i, 2) for i in box.tolist()] ... print(f"Detected similar object with confidence {round(score.item(), 3)} at location {box}") Detected similar object with confidence 0.938 at location [327.31, 54.94, 547.39, 268.06] Detected similar object with confidence 0.959 at location [5.78, 360.65, 619.12, 366.39] Detected similar object with confidence 0.902 at location [2.85, 360.01, 627.63, 380.8] Detected similar object with confidence 0.985 at location [176.98, -29.45, 672.69, 182.83] Detected similar object with confidence 1.0 at location [6.53, 14.35, 624.87, 470.82] Detected similar object with confidence 0.998 at location [579.98, 29.14, 615.49, 489.05] Detected similar object with confidence 0.985 at location [206.15, 10.53, 247.74, 466.01] Detected similar object with confidence 0.947 at location [18.62, 429.72, 646.5, 457.72] Detected similar object with confidence 0.996 at location [523.88, 20.69, 586.84, 483.18] Detected similar object with confidence 0.998 at location [3.39, 360.59, 617.29, 499.21] Detected similar object with confidence 0.969 at location [4.47, 449.05, 614.5, 474.76] Detected similar object with confidence 0.966 at location [31.44, 463.65, 654.66, 471.07] Detected similar object with confidence 0.924 at location [30.93, 468.07, 635.35, 475.39] ```rr)rrN)r9rrrr rr:r;rT)rrr'rrrrr)rErrrrrrrrr?r@ hidden_dimrquery_image_featsrrrrrrs r*image_guided_detectionz.Owlv2ForObjectDetection.image_guided_detectionOsH|!//+F^0  ':d&9&9' %%=' ' # ^ ITHYHYE &(9:mmK*>PSd>dfp1qr HYH_H_E &(9:!MM  ,>AR,RT^_ <@;Q;Q 02J< 8 &(8 '+&:&:{am&:&n#l!..{KIab4$0/-%" .  r,c |jd||||d|\}}}|j} |j} |j\} } } }t j || | | z|f}|jd| z}|j | ||jd}|j | ||jd}|ddkD}|j |||\}}|j|}|j|||}t||||||| | S)a input_ids (`torch.LongTensor` of shape `(batch_size * num_max_text_queries, sequence_length)`, *optional*): Indices of input sequence tokens in the vocabulary. Indices can be obtained using [`AutoTokenizer`]. See [`PreTrainedTokenizer.encode`] and [`PreTrainedTokenizer.__call__`] for details. [What are input IDs?](../glossary#input-ids). Examples: ```python >>> import httpx >>> from io import BytesIO >>> from PIL import Image >>> import torch >>> from transformers import Owlv2Processor, Owlv2ForObjectDetection >>> processor = Owlv2Processor.from_pretrained("google/owlv2-base-patch16-ensemble") >>> model = Owlv2ForObjectDetection.from_pretrained("google/owlv2-base-patch16-ensemble") >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> text_labels = [["a photo of a cat", "a photo of a dog"]] >>> inputs = processor(text=text_labels, images=image, return_tensors="pt") >>> outputs = model(**inputs) >>> # Target image sizes (height, width) to rescale box predictions [batch_size, 2] >>> target_sizes = torch.tensor([(image.height, image.width)]) >>> # Convert outputs (bounding boxes and class logits) to Pascal VOC format (xmin, ymin, xmax, ymax) >>> results = processor.post_process_grounded_object_detection( ... outputs=outputs, target_sizes=target_sizes, threshold=0.1, text_labels=text_labels ... ) >>> # Retrieve predictions for the first image for the corresponding text queries >>> result = results[0] >>> boxes, scores, text_labels = result["boxes"], result["scores"], result["text_labels"] >>> for box, score, text_label in zip(boxes, scores, text_labels): ... box = [round(i, 2) for i in box.tolist()] ... print(f"Detected {text_label} with confidence {round(score.item(), 3)} at location {box}") Detected a photo of a cat with confidence 0.614 at location [341.67, 23.39, 642.32, 371.35] Detected a photo of a cat with confidence 0.665 at location [6.75, 51.96, 326.62, 473.13] ```)rrrrrrr)r9r8rr rrr:r;rT) rr:r;rr'rrrrr})rErrrrrrrrrrrr?r@rrmax_text_queriesrrrrrs r*rzOwlv2ForObjectDetection.forwardsYf.FT-E-E. %)%= .  . * k700  44HSHYHYE &(9:mmK*>PSd>dfp1qr %??1-;#++J8H,J\J\]_J`a %%j2BIOOTVDWX v&* '+&:&:; V`&a#l!55kB'' [BZ[ )$$!/%* .  r,rr{rW)rNrOrPrrSr staticmethodr^r'rrrRrr>rrrJrrrrrrrrrr r}rrrs@r*r<r<Hs {(cVY^c^j^j !53D3D !IZIZ !33SXS_S_.*/ &&&&#'     J26*. 1&&1''$.1LL4' 1 u  ! 12*/ -4<<-4''-4 -4 #' -4 +, -4 u  !-4f*/'.'''.#''.+, '. u  ! '.\*/ *5#//*5!,,*5#' *5    *5X8<). ` ''` "--4` #' ` +, ` / ` ` D /3). Z <<Z ''Z  t+ Z #' Z +, Z  $Z Z r,r<)r4r%rqrr<)Nr)LrQcollections.abcr dataclassesrtypingrr'rrr r/ activationsr masking_utilsr modeling_layersr modeling_outputsr rmodeling_utilsrrprocessing_utilsrutilsrrrrrr utils.genericrrutils.output_capturingrconfiguration_owlv2rrrtransformers.image_transformsr get_loggerrNloggerr+r2r4r_rcrsrzr}rrSrrr[rrrrr%rUr`rqr}rr4rrr<__all__rTr,r*r s$! &!/9KF&J5PPF   H %`U\\`ell` -5<<-ELL-  ! +!  ! JGvG&G EF Ev E" "'0  / /  / d  * K*  * \JBIIJ\"))L!% II% <<% % << % LL4' % T\ %%'(%:6)RYY6)t ryy  2B>)?>)>)D- 299- b< /< @/ )/ f( 1( X. +. bk %k k ^RYY(-1ryy-1`G 2G T rr,