i<:dZddlZddlmZddlmZddlmZddlmZm Z ddl m Z dd l m Z dd lmZdd lmZmZmZmZmZd d lmZej0eZeGdde ZedGddeZedGddeeZgdZy)zPyTorch Fuyu model.N)nn)Cache)GenerationMixin)BaseModelOutputWithPoolingCausalLMOutputWithPast)PreTrainedModel) AutoModel)Unpack)TransformersKwargsauto_docstringcan_return_tupleloggingtorch_compilable_check) FuyuConfigc<eZdZUeed<dZdZdZdZdZ dZ dZ gZ dZ y)FuyuPreTrainedModelconfigmodel)imagetextTpast_key_valuesN)__name__ __module__ __qualname__r__annotations__base_model_prefixinput_modalitiessupports_gradient_checkpointing_supports_attention_backend_supports_flash_attn_supports_sdpa_supports_flex_attn_no_split_modules_skip_keys_device_placementw/var/www/vps2.regionflexible.com/Desarrollo/venv/lib/python3.12/site-packages/transformers/models/fuyu/modeling_fuyu.pyrr s= (&*#"&N"3r(rzt The Fuyu model which consists of a vision backbone and a language model, without a language modeling head. ) custom_introc@eZdZdeffd ZdZdZdejde ejdejdejfd Z e e d ejd eedeezfd Zd ej&dejdejfdZe e dd ej&dzdejdzdejdzdejdzdej&dzdedzdejdzdedzd eedeezfdZxZS) FuyuModelrct|||j|_|jj |_t j|j|_tj|j|jz|jz|j|_d|_|j!y)NF)super__init__ pad_token_id padding_idx text_config vocab_sizer from_configlanguage_modelrLinear patch_size num_channels hidden_sizevision_embed_tokensgradient_checkpointing post_initselfr __class__s r)r/zFuyuModel.__init__4s  !.. ,,77'33F4F4FG#%99    1 1 1F4G4G GI[I[$  ',# r(c6|jjSN)r5get_input_embeddingsr>s r)rBzFuyuModel.get_input_embeddingsAs""7799r(c:|jj|yrA)r5set_input_embeddingsr>values r)rEzFuyuModel.set_input_embeddingsDs 007r(word_embeddingscontinuous_embeddingsimage_patch_input_indicesreturnc |jdt|k(s'tdt|d|jd|j}t |jdD]}t j ||dk\dd}|||}|jd||jdkDr,td||jd|jd|d |||j|j|||f<|S) aThis function places the continuous_embeddings into the word_embeddings at the locations indicated by image_patch_input_indices. Different batch elements can have different numbers of continuous embeddings. Args: word_embeddings (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Tensor of word embeddings. continuous_embeddings (`torch.FloatTensor` of shape `(batch_size, num_patches, hidden_size)`): Tensor of continuous embeddings. The length of the list is the batch size. Each entry is shape [num_image_embeddings, hidden], and num_image_embeddings needs to match the number of non-negative indices in image_patch_input_indices for that batch element. image_patch_input_indices (`torch.LongTensor` of shape `(batch_size, sequence_length)`): Tensor of indices of the image patches in the input_ids tensor. rz7Batch sizes must match! Got len(continuous_embeddings)=z and word_embeddings.shape[0]=T)as_tuplezGNumber of continuous embeddings continuous_embeddings[batch_idx].shape=zA does not match number of continuous token ids src_indices.shape=z in batch element .) shapelen ValueErrorclonerangetorchnonzerotodevice)r>rHrIrJoutput_embeddings batch_idx dst_indices src_indicess r)gather_continuous_embeddingsz&FuyuModel.gather_continuous_embeddingsGsZ( %%a(C0E,FFJs3H/I.KKjQ`QfQfghQiPkl ,11344Q78 I --(A)(LPQ(Q\`abcdK4I>{KK  #&;I&F&L&LQ&OO ^7LY7W7]7]6_`I6A6G6G5II[\e[ffgi9Ni8XYd8e8h8h!((9 i4 5  ! r( pixel_valueskwargsc <|j|}t|S)z pixel_values (`torch.FloatTensor` of shape `(batch_size, num_channels, image_size, image_size)`): The tensors corresponding to the input images. )last_hidden_state)r:r)r>r]r^patch_embeddingss r)get_image_featureszFuyuModel.get_image_featuresss! 33LA)rcrdrespecial_image_maskn_image_tokensn_image_featuress r)get_placeholder_maskzFuyuModel.get_placeholder_masks  !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*dkk.H.H!H +//1)//2^5I5I!5LL/99"=GG VYYZgZnZno , - 3 3 59M9M9O OD^DTT`aq`r s "!r(N image_patchesimage_patches_indicesattention_mask position_idsr use_cachec l|du|duz r td| |jj|}|jd} |i| |jn |j} ||j nd} t j| | | zt j| }|jd}|i|j|dj} | j|j|j} |j||| }|j|| }|jd |||||d | }|S) a image_patches (`torch.FloatTensor` of shape `(batch_size, num_total_patches, patch_size_ x patch_size x num_channels)`, *optional*): Image patches to be used as continuous embeddings. The patches are flattened and then projected to the hidden size of the model. image_patches_indices (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Tensor of indices of the image patches in the input_ids tensor. Nz:You must specify exactly one of input_ids or inputs_embedsrrrgT) return_dict)rdre)rdrxryrrzr')rQr5rBrOrWget_seq_lengthrTarangerlrorbr`rVrhrumasked_scatter)r>rcrvrwrxryrrdrzr^seq_lenrWpast_key_values_lengthrarroutputss r)forwardzFuyuModel.forwardsl,  -t"; <YZ Z  FD//DDFyQM%%a(  )2)>Y%%MDXDXFIXId_%C%C%Ejk " <<&2H(HPUPZPZciL(11!4L  $#66}RV6Wii /22=3G3GI\I\] !%!:!:GW";" *889KM]^M%$%% ')%+    r()NNNNNNNN)rrrrr/rBrErTTensorlistr\rr FloatTensorr r tuplerrb LongTensorrurboolrr __classcell__r?s@r)r,r,.s z :8*!*! $ELL1*!$)<< *!  *!XN!--N9?@R9SN + +NN"))":?:K:K"]b]n]n"0.2-159.204(,26!%5##d*5||d* 5 %||d2 5  t+ 5&&-55((4/5$;5+,5 ' '55r(r,zz Fuyu Model with a language modeling head on top for causal language model conditioned on image patches and text. ceZdZddiZdeffd ZdZdZee dde jdzd e jdzd e jdzd e jdzd e jdzd e dzde jdzdedzde jdzdedzdeedeezfdZ dfd ZxZS)FuyuForCausalLMzlm_head.weightz(model.language_model.embed_tokens.weightrct||t||_t j |j j|j jd|_ |jy)NF)bias) r.r/r,rrr6r2r9r3lm_headr<r=s r)r/zFuyuForCausalLM.__init__sS  v& yy!3!3!?!?ASASA^A^ejk  r(c6|jjSrA)rrBrCs r)rBz$FuyuForCausalLM.get_input_embeddingsszz..00r(c:|jj|yrA)rrErFs r)rEz$FuyuForCausalLM.set_input_embeddingss ''.r(Nrcrvrwrxryrrdrzlabelslogits_to_keepr^rKc |jd||||||||d| } | d} t| tr t| dn| }|j | dd|ddf}d}| 4|j d|| |j jjd| }t||| j| j| jS)a image_patches (`torch.FloatTensor` of shape `(batch_size, num_total_patches, patch_size_ x patch_size x num_channels)`, *optional*): Image patches to be used as continuous embeddings. The patches are flattened and then projected to the hidden size of the model. image_patches_indices (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Tensor of indices of the image patches in the input_ids tensor. labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.text_config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.text_config.vocab_size]`. Examples: ```python >>> from transformers import FuyuProcessor, FuyuForCausalLM >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> processor = FuyuProcessor.from_pretrained("adept/fuyu-8b") >>> model = FuyuForCausalLM.from_pretrained("adept/fuyu-8b") >>> url = "https://huggingface.co/datasets/hf-internal-testing/fixtures-captioning/resolve/main/bus.png" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> prompt = "Generate a coco-style caption.\n" >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> outputs = model(**inputs) >>> generated_ids = model.generate(**inputs, max_new_tokens=7) >>> generation_text = processor.batch_decode(generated_ids[:, -7:], skip_special_tokens=True) >>> print(generation_text[0]) A blue bus parked on the side of a road. ```)rcrvrwrdrxryrrzrN)logitsrr3)lossrr hidden_states attentionsr') r isinstanceintslicer loss_functionrr2r3rrrr)r>rcrvrwrxryrrdrzrrr^rr slice_indicesrrs r)rzFuyuForCausalLM.forwardsj$**  '"7')%+      8B>SV8W~ot4]k mA}a,?@A  %4%%f9P9P9[9[_eD&#33!//))   r(c rt ||f||||||d|} |s|jddr d| d<d| d<| S)N)rrxrdrvrwis_first_iterationrzTrwrv)r.prepare_inputs_for_generationget) r>rcrrxrdrvrwrr^ model_inputsr?s r)rz-FuyuForCausalLM.prepare_inputs_for_generation9sew<   +)''"71    "fjjd&C48L0 1,0L )r() NNNNNNNNNr)NNNNNF)rrr_tied_weights_keysrr/rBrErr rTrrrrrrr r rrrrrrs@r)rrsc +,VWz 1/.2-159.204(,26!%&*%&P ##d*P ||d* P %||d2 P  t+ P &&-P P ((4/P $;P  t#P d P +,P  ' 'P P j" r(r)rrr,)__doc__rTr cache_utilsr generationrmodeling_outputsrrmodeling_utilsr models.auto.modeling_autor processing_utilsr utilsr r rrrconfiguration_fuyur get_loggerrloggerrr,r__all__r'r(r)rs  )R-2&jj*   H % 4/ 4 4 [#[  [| @)?@  @F Br(