i_(dZddlmZddlmZddlZddlmZddlmZddl m Z dd l m Z dd l mZdd lmZdd lmZmZdd lmZddlmZddlmZmZmZmZmZmZddlm Z ddl!m"Z"ddl#m$Z$ejJe&Z'eedGddeZ(eedGddeZ)GddejTZ+dejXdefd Z-e d!d"d#$ d6d%e d#ejXd&ejXdzd'edzd(ejXdzd)ejXdzd*ej\dzd+e/dzd,e/dzde0fd-Z1eGd.d/eZ2ed0Gd1d2e2Z3ed0Gd3d4e2e Z4gd5Z5y)7zPyTorch PaliGemmamodel.)Callable) dataclassN)nn)Cache)PreTrainedConfig)GenerationMixin)create_masks_for_generate)FlashAttentionKwargs)BaseModelOutputWithPastBaseModelOutputWithPooling)PreTrainedModel)Unpack) ModelOutputTransformersKwargsauto_docstringcan_return_tupleloggingtorch_compilable_check)deprecate_kwarg) AutoModel)PaliGemmaConfigzN Base class for Paligemma outputs, with hidden states and attentions.  custom_introc:eZdZUdZdZej dzed<y)PaligemmaModelOutputWithPasta image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder and after projecting the last hidden state. Nimage_hidden_states)__name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__/var/www/vps2.regionflexible.com/Desarrollo/venv/lib/python3.12/site-packages/transformers/models/paligemma/modeling_paligemma.pyrr.s 59**T18r(rzU Base class for PaliGemma causal language model (or autoregressive) outputs. ceZdZUdZdZej dzed<dZej dzed<dZ e dzed<dZ e ej dzed<dZ e ej dzed<dZej dzed<y) PaliGemmaCausalLMOutputWithPasta8 loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Language modeling loss (for next-token prediction). logits (`torch.FloatTensor` of shape `(batch_size, sequence_length, config.text_config.vocab_size)`): Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). past_key_values (`Cache`, *optional*, returned when `use_cache=True` is passed or when `config.use_cache=True`): It is a [`~cache_utils.Cache`] instance. For more details, see our [kv cache guide](https://huggingface.co/docs/transformers/en/kv_cache). Contains pre-computed hidden-states (key and values in the self-attention blocks) that can be used (see `past_key_values` input) to speed up sequential decoding. image_hidden_states (`torch.FloatTensor`, *optional*): A `torch.FloatTensor` of size `(batch_size, num_images, sequence_length, hidden_size)`. image_hidden_states of the model produced by the vision encoder after projecting last hidden state. Nlosslogitspast_key_values hidden_states attentionsr)r r!r"r#r,r$r%r&r-r.rr/tupler0rr'r(r)r+r+>s &*D%  d ")'+FE   $+$(OUT\(59M5**+d2926Je''(4/648**T18r(r+c*eZdZdeffd ZdZxZS)PaliGemmaMultiModalProjectorconfigct|tj|jj |jj d|_y)NTbias)super__init__rLinear vision_config hidden_sizeprojection_dimlinearselfr4 __class__s r)r9z%PaliGemmaMultiModalProjector.__init__]s; ii 4 4 @ @&BVBVBeBelpq r(c(|j|}|SN)r>)r@image_featuresr/s r)forwardz$PaliGemmaMultiModalProjector.forwardas N3 r()r r!r"rr9rE __classcell__rAs@r)r3r3\srrr(r3 group_idsreturnc Pdtdtdtdtdtf fd }|S)au This function adds the correct offsets to the `q_idx` and `kv_idx` as the torch API can only accept lengths, not start and end indices. Args: group_ids (`torch.Tensor`): A tensor of shape `(bs, len)` assigning each token to a vision group. Tokens with the same group come from the same input image. Text is denoted by `-1`. batch_idxhead_idxq_idxkv_idxrIc jd}|j|dz }|j|dz } ||f} ||f}tj||k|d}tj||k|d}||k(|dk\zS)Nr)maxr)shapeclampr$where) rKrLrMrN seq_length q_idx_clampedkv_idx_clampedq_groupkv_grouprHs r) inner_maskz0token_type_ids_mask_function..inner_maskqs__R(  Q 7 *q.9I}45Y67++ej0'2>;;v 2HbA8#1 55r()intbool)rHrZs` r)token_type_ids_mask_functionr]gs3 6c 6S 6 6c 6d 6 r( input_embeds5.6.0 inputs_embedsversionnew_namer4attention_maskr. position_idstoken_type_ids pixel_values is_trainingis_first_iterationc  |r | td|j||||d} |r|n|duxs|j xs|du}|s| jdds<|d|z }n4tj dt j|dddddf}||r|dk(j|j} tjj| d d dddd f} | | z} t j| jd dz }t j| |t j |d }t#|| d <t%di| S)a" Overwrites the base `create_masks_for_generate` with `token_type_ids` masking to create the causal mask mapping for all kinds of forward passes. Paligemma uses a bidirectional mask on the prompt tokens. Uses `pixel_values` as an optional input to disambiguate edge cases. Nz;`token_type_ids` is required as a model input when training)r4r`rdr.re use_cacheTrzIt is a prefill stage but The `token_type_ids` is not provided. We recommend passing `token_type_ids` to the model to prevent bad attention masking.r)rr)valuerP)dimor_mask_functionr') ValueErrorget_text_configis_initializedgetlogger warning_oncer$ ones_liketodevicer functionalpadcumsumr[rT full_liker]r )r4r`rdr.rerfrgrhrikwargs mask_kwargsis_imageis_previous_imagenew_image_startrHs r)create_causal_mask_mappingrsu&~-VWW((*&(*$ K  %g_-K-K)Kg|cgOg K!>  %/N   Z  #__];Aq!GDN !&8 #a'++M,@,@AMM--ha-HCRCP"&7%77LL!4!4!6A>B KK)U__^UW5XY *Fy*Q &' $ 3{ 33r(cBeZdZUeed<dZdZdZdgZdZ dZ dZ dZ dZ dZy) PaliGemmaPreTrainedModelr4model)imagetextTr3r.FN)r r!r"rr&base_model_prefixinput_modalitiessupports_gradient_checkpointing_no_split_modules_skip_keys_device_placement_can_compile_fullgraph_supports_flash_attn_supports_sdpa_supports_flex_attn_supports_attention_backendr'r(r)rrsF (&*#78"3"N"&r(rz| The Base Paligemma model which consists of a vision backbone and a language model without language modeling head., ceZdZdZdeffd ZdZdZee dde jd e e d eezfd Zd e j"d e jde jfdZee dd e j"dzde jdzde j&dzde j"dzdedzde j"dzd e jdzde j"dzdedzd e ed eezfdZxZS)PaliGemmaModelFr4ct||tj|j|_t ||_|jj|_ tj|j}||_ |jjjxs |j|_|jy)N)r4)r8r9r from_configr; vision_towerr3multi_modal_projector text_config vocab_sizelanguage_modelr4rpdtypetext_config_dtype post_init)r@r4rrAs r)r9zPaliGemmaModel.__init__s  %119M9MN%A&%I" ,,77"..f6H6HI,!%!!D!D!R  r(c6|jjSrC)rget_input_embeddingsr@s r)rz#PaliGemmaModel.get_input_embeddingss""7799r(c:|jj|yrC)rset_input_embeddingsr@rls r)rz#PaliGemmaModel.set_input_embeddingss 007r(zWObtains image last hidden states from the vision tower and apply multimodal projection.rrgr|rIc t|j|fi|}|j}|j|}||_|SrC)rlast_hidden_stater pooler_output)r@rgr| image_outputsselected_image_featurerDs r)get_image_featuresz!PaliGemmaModel.get_image_featuressF*)),A&A !.!@!@334JK&4 #r( input_idsr`rDcN|m||jtj|jjtj |j k(}|jd}n||jjk(}|j}|jd|jdz}|jdj|j|j }t||j|jk(d|d||S)z Obtains multimodal placeholder mask from `input_ids` or `inputs_embeds`, and checks that the placeholder token count is equal to the length of multimodal features. If the lengths are different, an error is raised. )rrwrPrrz6Image features and image tokens do not match, tokens: z , features: )rr$tensorr4image_token_idlongrwallsumrR unsqueeze expand_asrvrnumel)r@rr`rDspecial_image_maskn_image_tokensn_image_featuress r)get_placeholder_maskz#PaliGemmaModel.get_placeholder_masks  !.2M$2K2K2M T[[77uzzR_RfRfg3" "4!7!7!; !*dkk.H.H!H +//1)//2^5I5I!5LL/99"=GG VYYZgZnZno , - 3 3 59M9M9O OD^DTT`aq`r s "!r(Nrdrer.rflabelsrkc |du|duz r td|R|jj|jk\r/||jjk(} |j } d| | <n|} ||j | }|Y||j nd} tj|jd|j| z}|jddz}|g|j|j}|j|j|j}|j!|||} |j#| |}t%|x}t&s't)|j|||||||j*}|j,d ||||| d| }t/|j0|j2|j4|j6| Sd S)  labels (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): Labels for computing the masked language modeling loss. Indices should either be in `[0, ..., config.text_config.vocab_size]` or -100 (see `input_ids` docstring). Tokens with indices set to `-100` are ignored (masked), the loss is only computed for the tokens with labels in `[0, ..., config.text_config.vocab_size]`. Example: ```python >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> from transformers import AutoProcessor, PaliGemmaForConditionalGeneration >>> model = PaliGemmaForConditionalGeneration.from_pretrained("google/paligemma2-3b-mix-224") >>> processor = AutoProcessor.from_pretrained("google/paligemma2-3b-mix-224") >>> prompt = "Where is the cat standing?" >>> url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> inputs = processor(images=image, text=prompt, return_tensors="pt") >>> # Generate >>> generate_ids = model.generate(**inputs,) >>> processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0] "Where is the cat standing?\nsnow" ```Nz:You must specify exactly one of input_ids or inputs_embedsrr)rw)r`rD)rh)rdrer.r`rk)rr.r/r0rr')ror4rrclonerget_seq_lengthr$arangerRrwrrrrvrrmasked_scatter isinstancedictrtrainingrrrr.r/r0)r@rrgrdrer.rfr`rrkr|r llm_input_idspast_seen_tokensrDcausal_mask_mappingoutputss r)rEzPaliGemmaModel.forwardsZ  -t"; <YZ Z  T[[%?%?4??%R!*dkk.H.H!H %OO-M01M, -%M  7D557 FM  CRC^==?de  << (;(;A(>}G[G[\_ooL'11!4q8L  #!44\BPPN+..}/C/C]EXEXYN!%!:!:~";" *889K^\M?-F"<  MM # &$%% .%+'    ,%77#33!//))2>2J   QU   r() NNNNNNNNN)r r!r"accepts_loss_kwargsrr9rrrrr$r%rrr1r r LongTensorrTensorrr\r rrErFrGs@r)rrs   :8n!--9?@R9S + +"))":?:K:K"]b]n]n"0.215.204(,2626*.!%c ##d*c ''$.c  t+ c &&- c  c ((4/c ((4/c   4'c $;c -.c  - -c c r(rceZdZddiZdeffd ZdZdZede jde e fd Z ee dd e jd zde jd zd e j d zd e jd zded zde jd zde jd zde jd zded zdee j zde e deezfdZ dfd Zeeddd ddede j d e j d zded zd e j d zde j d zded zdefdZxZS)!PaliGemmaForConditionalGenerationzlm_head.weightz(model.language_model.embed_tokens.weightr4ct||t||_t j |j j|j jd|_ |jy)NFr6) r8r9rrrr:rr<rlm_headrr?s r)r9z*PaliGemmaForConditionalGeneration.__init__sS  #F+ yy!3!3!?!?ASASA^A^ejk  r(c6|jjSrC)rrrs r)rz6PaliGemmaForConditionalGeneration.get_input_embeddingsszz..00r(c:|jj|yrC)rrrs r)rz6PaliGemmaForConditionalGeneration.set_input_embeddingss ''.r(rgr|c <|jj|fi|SrC)rr)r@rgr|s r)rz4PaliGemmaForConditionalGeneration.get_image_featuress,tzz,,\DVDDr(Nrrdrer.rfr`rrklogits_to_keeprIc |jd|||||||| |d | } | d} t| tr t| dn| }|j | dd|ddf}d}|4|j d|||j jjd| }t||| j| j| j| jS)r) rrgrfrdrer.r`rkrrN)r-rr)r,r-r.r/r0rr')rrr[slicer loss_functionr4rrr+r.r/r0r)r@rrgrdrer.rfr`rrkrr|rr/ slice_indicesr-r,s r)rEz)PaliGemmaForConditionalGeneration.forwardsZ$**  %))%+'      8B>SV8W~ot4]k mA}a,?@A  %4%%f9P9P9[9[_eD/#33!//)) ' ; ;   r(c t||f|||||| || d| } | jd | ddz| d<| s|s|| d<| S)N)r.r`rdrerkrrfrirerrg)r8prepare_inputs_for_generationrr)r@rr.r`rergrdrfrkrrrir| model_inputsrAs r)rz?PaliGemmaForConditionalGeneration.prepare_inputs_for_generations| w<   +')%))1       N + 7+7+G!+KL ( Y+7L (r(r^r_raric t||||||fd|i|jD cic]\}} |dk7s || c} }Scc} }w)Nrirg)ritems) r4r`rdr.rerfrir|kvs r)r z;PaliGemmaForConditionalGeneration.create_masks_for_generates]*         2  !' F1!~2Eq!tF  Gs <<) NNNNNNNNNr) NNNNNNTNNF)NF)r r!r"_tied_weights_keysrr9rrrr$r%rrrrrrrr\r[r1r+rEr staticmethodrrrr rFrGs@r)rrsI +,VW 1/Eu/@/@EFSeLfEE.215.204(,2626*.!%-.J ##d*J ''$.J  t+ J &&- J  J ((4/J ((4/J   4'J $;J ell*J +,J  0 0J J ^ )V^WO/3*/   ||  t+   llT)   t+  !4K   P r(r)rrr)NNFN)6r#collections.abcr dataclassesrr$r cache_utilsrconfiguration_utilsr generationr masking_utilsr modeling_flash_attention_utilsr modeling_outputsr r modeling_utilsrprocessing_utilsrutilsrrrrrrutils.deprecationrautorconfiguration_paligemmar get_loggerr rsrr+Moduler3rr]r%r\rrrrr__all__r'r(r)rs!$!  3)6BS-&14   H %  9#:9  9  9k9  90299ELLX6?K+/-1$&*C4 C4<<C4LL4'C4T\ C4 ,,% C4 LL4' C4##d*C4C4t C4 C4LC4L ' ' ' c -c  c L b (@/b  b J ^r(