i ddlmZddlmZddlmZddlZddlmZddl m Z m Z m Z m Z Gdd Zd ed efd Zd ededed efdZdededed efdZ d dej(deedeeded df dZdede dededed ee f dZy)!) OrderedDict)ceil)AnyN)PretrainedConfig)FutureRequestState RequestState RequestStatusloggerceZdZdZdeddfdZddZdededejjdzfd Z dd e ddfd Z deded ejjddfd Z y)CudaGraphBufferz>A fixed-size dict for CUDA graphs with LRU eviction when full.max_sizereturnNcV|dkrtd|||_t|_y)Nrz#max_size must be positive, but got ) ValueErrorrr_storage)selfrs /var/www/vps2.regionflexible.com/Desarrollo/venv/lib/python3.12/site-packages/transformers/generation/continuous_batching/utils.py__init__zCudaGraphBuffer.__init__s, q=B8*MN N  LWM c\|j}d|_|jd||_y)NrT)silent)rplan_for_new_graph)roriginal_max_sizes r__del__zCudaGraphBuffer.__del__"s+ MM  t,) rq_lenkv_lenc~|jj||f}||jj||f|SN)rget move_to_endrrrgraphs r get_graphzCudaGraphBuffer.get_graph(s; !!5&/2   MM % %ufo 6 rrc"t|j|jk\rm|jjd\}}|st j d||j t|j|jk\rlyy)NF)lastz!Evicting graph for evicted_key = )lenrrpopitemr inforeset)rr evicted_key evicted_graphs rrz"CudaGraphBuffer.plan_for_new_graph.sn$-- DMM1)-)>)>E)>)J &K @+1ABC    ! $-- DMM1rr#c||jtjd|d|||j||f<y)NzSetting graph for q_len = z , kv_len = )rr r)rr"s r set_graphzCudaGraphBuffer.set_graph5s: ! 1 ,VK@A). ufo&r)rN)F)__name__ __module__ __qualname____doc__intrrtorchcuda CUDAGraphr$boolrr.rrr r sHZZZ * sCEJJ4H4H44O ""$"/s/C/ 8L8L/QU/rr configrc|jdvS)z:Checks if attention mask is needed for the given (config).)z paged|eagerz paged|sdpa)_attn_implementation)r9s rattn_mask_is_neededr<<s  & &*G GGrsize interval_size max_valuecX|dkr|S|dkDrt||z |zn|}t||S)zQReturn the smallest multiple of (interval_size) >= (size), capped at (max_value).r)rmin)r=r>r?paddeds rpad_to_intervalrCAs9;?!8T$& '- 7F vy !!rx divide_byalign_tocTtt||z }||zr ||||zz z }|Sr)r3r)rDrErFs raligned_dividerHIs4 DY  A8| XX && Hrattention_maskcumulative_seqlens_qcumulative_seqlens_ksliding_windowcJtj|jj}t t |dz D]}||dz||z }||dz||z }||kr|dk\r ||z dz}nd}t ||||dz} t ||||dz} tj|d| | fj||j|j} tj| |} |dkDr"||z |z } | tj| | z } | |d| | f<y)uBuilds an attention mask inplace using the cumulative seqlens of the query and key. If given a sliding window, it will also apply a sliding window mask on top. The attention mask is not boolean, it uses zeroes and -inf (or its equivalent) so it's more of an attention score bias tensor. The attention mask is a block-diagonal matrix, with each block an attention mask for a single query-key pair. Each of those block is built from a causal mask and, if there is a sliding window, a sliding window mask. An example is represented below, with seqlen_k = 8, seqlen_q = 4 and sliding_window = 6: CAUSAL MASK: █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ ░ ░ █ █ █ █ █ █ █ ░ █ █ █ █ █ █ █ █ SLIDING WINDOW MASK: ┌──────────────────────── seqlen_k - seqlen_q - sliding_window = 8 - 4 - 6 = -2 offset to the left <─┴─> ░ █ | █ █ █ █ █ █ █ █ ░ ░ | █ █ █ █ █ █ █ █ ░ ░ | ░ █ █ █ █ █ █ █ ░ ░ | ░ ░ █ █ █ █ █ █ ATTENTION MASK (sum of causal and sliding window masks): █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ ░ ░ ░ █ █ █ █ █ █ Another example with seqlen_k = 5, seqlen_q = 3 and sliding_window = 2: CAUSAL MASK: █ █ █ ░ ░ █ █ █ █ ░ █ █ █ █ █ SLIDING WINDOW MASK: ┌──────────────────────── seqlen_k - seqlen_q - sliding_window = 5 - 3 - 2 = 0 offset to the left <┴> | ░ █ █ █ █ | ░ ░ █ █ █ | ░ ░ ░ █ █ ATTENTION MASK (sum of causal and sliding window masks): ░ █ █ ░ ░ ░ ░ █ █ ░ ░ ░ ░ █ █ r.)dtypedevice)diagonalN) r4finforNrAranger'slicefullshaperOtriutril)rIrJrKrL min_valueiseqlen_qseqlen_kcausal_diagonal query_range key_range minus_infmaskedsliding_diagonals rbuild_attention_maskrbPsSt N00155I 3+,q0 1='A.1Ea1HH'A.1Ea1HH h 8q=&1A5OO035I!a%5PQ .q13GA3NO JJ 3 Y6 7 = =  &&!((  I@ A '(2^C  ejj5EF FF6<sK23-=rnumstatusnum_query_tokensnum_cache_tokenscachect|Dcgc]}d|jd|d}}||z}t||jz }g} |D]m} t | dg|zd} || _dg|z| _|| _|j|| jd} | | cS| jt| ddo| Scc}w) zQAn utility function to create a list of FutureRequestStates for the warmup of CB. __warmup____rr) request_idinitial_tokensmax_new_tokensT) has_new_tokencomplete_blocks) rRnamer block_sizer _statustokens_to_processposition_offsetallocate_blocksrlappendr) rcrdrerfrgrY request_ids total_tokens blocks_needed future_statesreq_idstate allocateds rcreate_warmup_future_statesrs=B#JGqYv{{m1QCr2GKG#&66L(8(889MM _s\?Qbcd #$#(8"8 0))-9I9I1M   /T[\]^ _ HsB=)r) collectionsrmathrtypingrr4 transformers.configuration_utilsrrequestsrr r r r r7r<r3rCrHTensorlistrbrr8rrrs$ =MM / /FH 0HTH "#"c"c"c" c c S S  Q=LLQ=s)Q=s)Q= Q=  Q=h        r