iddlZddlmZddlmZej j dejjdejdejdejd ejdzd ed ejd eje e ejfzd e de e e e fzdejdzde ejdffdZy)N)PagedAttentionCache)!lazy_import_paged_flash_attentionmoduleqkvattention_maskcache cu_seq_lens_q cu_seq_lens_k max_seqlen_q max_seqlen_k block_tablereturnc 2t|jj\} } t|ddsdn|jdz df}|dk(rdnd}t |t r ||}| |} |  |j|||j| d | d  \}}d | vrd | jd ini}| |jdd jdj|j|j|jtj|jtjj!|| f|j"d|d|}t |t$r|d}|dfS|j&|j\}}|j(|j+d|j,|j.|j0}|j2|j+d|j,|j.|j0}|j5d dddj}|j5d dddj}|j5d dddj}|j7d}|d|dz|d|z dz jtj}|j9| | |i}d | vr| d |d <| d|||||||j"d|d |}t |t$r|d}|jd}|dfS)apPerforms the forward pass of attention with paged key-value cache. This function handles the cache updates and performs the attention computation. For decode-only batches (when block_table is provided), uses `flash_attn_with_kvcache` for fused attention + cache update. Otherwise uses `flash_attn_varlen_func`. See the [paged attention guide](https://huggingface.co/docs/transformers/en/paged_attention) for more details. Args: q: (1, nheads, total_q, headdim), where total_q = total number of query tokens in the batch. k: (1, nheads_k, total_k, headdim), where total_k = total number of key tokens in the batch. v: (1, nheads_k, total_k, headdim), where total_k = total number of key tokens in the batch. cu_seq_lens_q: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into q. cu_seq_lens_k: (batch_size + 1,), dtype torch.int32. The cumulative sequence lengths of the sequences in the batch, used to index into kv. max_seqlen_q: int. Maximum query sequence length in the batch. max_seqlen_k: int. Maximum key sequence length in the batch. block_table: (num_groups, batch_size, max_blocks_per_seq), dtype int32. Block table for paged KV cache. If provided, uses flash_attn_with_kvcache for fused attention + cache update. For each request, the block table is a vector of size (max_blocks_per_seq,) with indices indicating the physical location of the cache to read from and write to. The kernel, using the cache_seqlens for that request, knows how much cache to read and dispatches the read using the block table. Same for the write. If a request has fewer than max_blocks_per_seq blocks, the block table is padded with -1s to indicate that the block is not allocated. sliding_windowF)rrfull_attentionsliding_attentionN read_index write_index) key_states value_states layer_idxrrs_auxrT) softmax_scalecausal window_sizer) rk_cachev_cacherr cache_seqlensrrr )rconfig_attn_implementationgetattrr isinstancedictupdaterget transposesqueeze contiguoustotorchint32clonescalingtuplelayer_index_to_group_indices key_cacheview block_sizenum_key_value_headshead_dim value_cachepermutesizeget_block_table_key)rrrr r r r r rrrkwargsflash_attn_varlen_funcflash_attn_with_kvcacher layer_type custom_kwargs attn_output group_idxlayer_idx_in_groupr"r# batch_sizer$ flash_kwargss v/var/www/vps2.regionflexible.com/Desarrollo/venv/lib/python3.12/site-packages/transformers/integrations/flash_paged.pypaged_attention_forwardrKsIL7X **733 &-V5Eu%MXTZTiTilmTmopSqN%3x%?!EXJ-&%j1 #J/ ||&&l+}-  1;BV:K&**W"56QS , KK1  % %a ( 3 3 5 LLN LLN   U[[ )   U[[ ) / / 1    !..&     k5 )%a.KT  K).(J(J6K[K[(\% %//"45::   %";";U^^ ##$67<<   %";";U^^  IIaAq ! , , . IIaAq ! , , . IIaAq ! , , .VVAY &q:>:];J=WWZ[[__`e`k`kl 112IJKXaLbc f $*7OL !-  ' ..&     k5 )%a.K!))!,  )r1generation.continuous_batchingrmodeling_flash_attention_utilsrcompilerdisablennModuleTensorr*strintr5rKr%rLrJrVs  @Ns HHOOs ||s ||s || s LL4' s  s<<s<<$sELL'8"99ssS#X&s$s 5<< ssrL