i o dZddlZddlZddlZddlZddlmZmZddlm Z ddl m Z ddl mZmZddlmZmZmZmZdd lmZdd lmZmZdd lmZd d lmZej<eZ d7de!e!e"ffdZ#GddejHZ%GddejLZ'GddejPZ)GddejTZ+GddejPZ,d8dej de-de"dej fdZ.GddejPZ/d9d Z0Gd!d"ejPZ1Gd#d$ejPZ2Gd%d&ejPZ3Gd'd(ejPZ4Gd)d*ejPZ5eGd+d,eZ6eGd-d.e6Z7ed/0Gd1d2e6Z8ed30Gd4d5ee6Z9gd6Z:y):z9PyTorch BiT model. Also supports backbone for ViT hybrid.N)Tensornn)initialization)ACT2FN) BackboneMixinfilter_output_hidden_states)BackboneOutputBaseModelOutputWithNoAttention(BaseModelOutputWithPoolingAndNoAttention$ImageClassifierOutputWithNoAttention)PreTrainedModel)auto_docstringlogging)can_return_tuple) BitConfigreturncd}||dz ||dz zzdz}||fSt|tra|j}|dk(r0|dk(r#||dz zdzdk(r|dz ||dz zzdz}||fSd}d}||fS|dk(rd}||fS|dz ||dz zzdz}||fS)al Utility function to get the tuple padding value given the kernel_size and padding. Args: padding (Union[`str`, `int`], *optional*): Padding value, can be either `"same"`, `"valid"`. If a different value is provided the default padding from PyTorch is used. kernel_size (`int`, *optional*, defaults to 7): Kernel size of the convolution layers. stride (`int`, *optional*, defaults to 1): Stride value of the convolution layers. dilation (`int`, *optional*, defaults to 1): Dilation value of the convolution layers. FrsamerTvalid) isinstancestrlower)padding kernel_sizestridedilationdynamics u/var/www/vps2.regionflexible.com/Desarrollo/venv/lib/python3.12/site-packages/transformers/models/bit/modeling_bit.pyget_padding_valuer")sGQJ(kAo">>1D'3--/ f {K!O <AQF"QJ(kAo*FF1L G  G   G G  h+/&BBqHG G c6eZdZdZ dfd ZdZxZS)WeightStandardizedConv2dzConv2d with Weight Standardization. Used for ViT Hybrid model. Paper: [Micro-Batch Training with Batch-Channel Normalization and Weight Standardization](https://huggingface.co/papers/1903.10520) c t||||\}} t | ||||||||| rt||||_| |_yd|_| |_y)N)rr)rrrgroupsbias)r"super__init__ DynamicPad2dpadeps) self in_channel out_channelsrrrrr'r(r- is_dynamic __class__s r!r*z!WeightStandardizedConv2d.__init__Yss0V^fg       #KBDHDHr#c |j|j|}tjj|jj d|j ddddd|jj|j}tjj|||j|j|j|j|j}|S)NrT)trainingmomentumr-)r,r functional batch_normweightreshaper0r- reshape_asconv2dr(rrrr')r. hidden_stater:s r!forwardz WeightStandardizedConv2d.forwardvs 88 88L1L)) KK  4#4#4b 94PT_bhlhphp* *T[[ ! }}++ &$))T[[$,, W[WbWb r#)rSAMErrFgư>__name__ __module__ __qualname____doc__r*r? __classcell__r2s@r!r%r%Rs&  : r#r%c*eZdZdZdfd ZdZxZS)BitGroupNormActivationzQ A module that combines group normalization with an activation function. ct||j||||rt|j|_yt j|_y)N)r-affine)r)r* num_groupsr hidden_act activationrIdentity)r.config num_channelsr-rKapply_activationr2s r!r*zBitGroupNormActivation.__init__sA **Lc&Q $V%6%67DO kkmDOr#ctjj||j|j|j |j }|j|}|SN)rr8 group_normrLr:r(r-rN)r.r>s r!r?zBitGroupNormActivation.forwardsH}}// doot{{\`\e\egkgogop |4 r#)gh㈵>TTrArGs@r!rIrIs,r#rIc*eZdZdZdfd ZdZxZS)r+z A module that wraps dynamic padding of any input, given the parameters of the convolutional layer and the input hidden states. ct|t|tr||f}t|tr||f}t|tr||f}||_||_||_||_d}||_y)Ncpttj||z dz |z|dz |zzdz|z dS)Nrr)maxmathceil)xrrrs r!compute_paddingz.DynamicPad2d.__init__..compute_paddingsB !f*-1V;{QRZ>ZZ]^^abbdef fr#) r)r*rintrrrvaluer])r.rrrr_r]r2s r!r*zDynamicPad2d.__init__sw  k3 '& 4K fc "f%F h $ (+H&     g /r#c |jdd\}}|j||jd|jd|jd}|j||jd|jd|jd}|dkDs|dkDrBt j j||dz||dzz |dz||dzz g|j}|S)Nrrr)r_) sizer]rrrrr8r,r_)r.input input_height input_widthpadding_height padding_widths r!r?zDynamicPad2d.forwards$)JJL$5! k--lD)rrr-rr)rrrpr@)rrrrr5)rr_ preactivationrQ)r)r*r%rQembedding_sizeglobal_padding convolutionriembedding_dynamic_paddingpoolerupperrrOr, ConstantPad2d layer_typerInormr.rPr2s r!r*zBitEmbeddings.__init__s 3     ! !))  #qPVPpPpq   ,1F1F1L1L1NRX1X{{}DH'' CHDH    /.vFDYDYZDI DI"//r# pixel_valuesrc|jd}||jk7r td|j|}|j |}|j |}|j |}|S)NrzeMake sure that the channel dimension of the pixel values match with the one set in the configuration.)shaperQ ValueErrorr~r,rr)r.rrQ embeddings r!r?zBitEmbeddings.forwardsr#))!, 4,, ,w $$\2 HHY' IIi( KK * r#) rBrCrDrErr*rr?rFrGs@r!rvrvs'0y06Fvr#rvrc drop_probr6c|dk(s|s|Sd|z }|jdfd|jdz zz}|tj||j|j z}|j |j||z}|S)zc Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). r5rr)r)dtypedevice)rndimtorchrandrrfloor_div)rcrr6 keep_probr random_tensoroutputs r! drop_pathrs Cx I I [[^  Q 7 7E 5 ELL YYM YYy !M 1F Mr#cxeZdZdZd dedzddffd ZdejdejfdZde fdZ xZ S) BitDropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).Nrrc0t|||_yrT)r)r*r)r.rr2s r!r*zBitDropPath.__init__(s "r#rtcDt||j|jSrT)rrr6rss r!r?zBitDropPath.forward,s FFr#c d|jS)Nzp=)r)r.s r! extra_reprzBitDropPath.extra_repr/sDNN#$$r#rT) rBrCrDrEfloatr*rrr?rrrFrGs@r!rr%sGb#%$,#$#GU\\GellG%C%r#rcf|}t|t||dz z|z|z}|d|zkr||z }|S)Nrg?)rYr^)r_divisor min_value new_values r!make_divr3sGIIs57Q;#677BWLMI3;W r#c:eZdZdZ dfd ZdZxZS)BitPreActivationBottleneckLayeraPre-activation (v2) bottleneck block. Follows the implementation of "Identity Mappings in Deep Residual Networks": https://github.com/KaimingHe/resnet-1k-layers/blob/master/resnet-pre-act.lua Except it puts the stride on 3x3 conv when available. c t ||xs|}|xs|}t||z} | rt||||d|_nd|_t |||_t|| dd|j|_ t || |_ t| | d||d|j|_ t || |_ t| |dd|j|_ | d kDrt| |_ytj |_y) NTrpreactrryr-rr{r)rr'r-rr)r)r*rBitDownsampleConv downsamplerInorm1r%r}conv1norm2conv2norm3conv3rrrOr) r.rP in_channelsr0 bottle_ratiorrfirst_dilationr'drop_path_rateis_first_layer mid_channelsr2s r!r*z(BitPreActivationBottleneckLayer.__init__Cs '38#2{  | ;< / DO#DO+FK@ -k<PT^d^s^st +FN - ,&T[a[p[p  ,FLA -lL!QU_e_t_tu 8F8J^4PRP[P[P]r#c0|j|}|}|j|j|}|j|}|j|j |}|j |j |}|j|}||zSrT)rrrrrrrr)r.rthidden_states_preactshortcuts r!r?z'BitPreActivationBottleneckLayer.forwardos#zz-8! ?? &';eZdZdZ dfd ZdZdedefdZxZS)BitStagez7 A ResNet v2 stage composed by stacked layers. c ^t||dvrdnd} |jdk(rt} nt} |} t j |_t|D]Q} |j| ||\}} }|jjt| | || ||||| | | |} |} Sy)N)rrrr bottleneck)rrrrrr) r)r*rrrr Sequentiallayersrange_get_updated_hyperparameters add_moduler)r.rPrr0rrdepthr layer_dropoutr layer_clsprev_chs layer_idxrrr2s r!r*zBitStage.__init__s &&0a    ,*I7Immo u &I595V5V6=6 2FNN KK " "I !%!-#1#1#1  $H%N+ &r#c8|r||}nd}|dk7rd}|dk(}|||fS)zt Get the new hyper-parameters with respect to the previous ones and the index of the current layer. r5rr)r.rrrrrs r!rz%BitStage._get_updated_hyperparameterss8 *95N N >F"a~~55r#rcrcT|}t|jD] \}}||}|SrT) enumerater)r.rcr>_layers r!r?zBitStage.forward"s3 !$++. /HAu .L /r#)rN) rBrCrDrEr*rrr?rFrGs@r!rrs.,&\6 Vr#rc FeZdZdeffd ZdZ d dedededefdZ xZ S) BitEncoderrPc t|tjg|_|j }d}d}t jtjd|jt|jj|jDcgc]}|j}}tt!|j|j"|D]`\}\}} } |j%||| ||\} } }t'||| | ||| } | }|| z}|jj)t+|| bycc}w)Nrr)rrrr)r)r*r ModuleListstagesr|rrnplinspacersumdepthssplittolistrzip hidden_sizesrrrr)r.rPrcurrent_striderr\layer_dropouts stage_idx current_depthcurrent_hidden_sizerr0rstager2s r!r*zBitEncoder.__init__*sA mmB' ((\\"++a1F1FFMMHZ"[\bbcicpcpq  HHJ  OX  v22N CO  : JIJ ':M.2-N-N>+>&. *L&(!#+E$H f $N KK " "3y>5 9+ :  sEczt||jz}|dk(rdnd}||jk\r||z}d}|||fS)Nrrr)r width_factor output_stride)r.rrrrrPr0rs r!rz'BitEncoder._get_updated_hyperparametersPsO 3f6I6I IJ 1n! V11 1  HFVX--r#r>output_hidden_states return_dictrc|rdnd}|jD]}|r||fz}||}|r||fz}|std||fDSt||S)Nrc3&K|] }|| ywrTr).0vs r! z%BitEncoder.forward..gsSqQ]Ss)last_hidden_statert)rtupler )r.r>rrrt stage_modules r!r?zBitEncoder.forwardXsv3  KK 6L# - ? ' 5L  6 )\O;MS\=$ASS S-*'  r#)FT) rBrCrDrr*rrboolr r?rFrGs@r!rr)sA$:y$:L.]a " :> UY ' r#rcXeZdZUeed<dZdZdZdgZe jdZ y)BitPreTrainedModelrPbit)imagerrvct|tjr#tj|j ddyt|tj rtj|j tjd|jxtjjj|j \}}|dkDrdtj|z nd}tj|j| |yyt|tjtjfrtj |j dtj |jdt#|dd^tj$|j&tj(|j*tj$|j,yyy) Nfan_outrelu)mode nonlinearity)arr running_mean)rrConv2dinitkaiming_normal_r:Linearkaiming_uniform_rZsqrtr(r_calculate_fan_in_and_fan_outuniform_ BatchNorm2d GroupNorm constant_getattrzeros_rones_ running_varnum_batches_tracked)r.modulefan_inrbounds r! _init_weightsz BitPreTrainedModel._init_weightsws> fbii (  YV T  *  ! !&--499Q< @{{&!HHMMGG V 17!DIIf-- fkkE659' > ? NN6==! , NN6;; *v~t4@ F//0 6--. F667A@r#N) rBrCrDr__annotations__base_model_prefixinput_modalitiesmain_input_name_no_split_modulesrno_gradrrr#r!rros< !$O()U]]_88r#rc ReZdZfdZe ddededzdedzdefdZxZ S) BitModelcJt||||_t||_t ||_|jdk(rt||jdntj|_ tjd|_|jy)Nrzr4r{)rr)r)r*rPrvembedderrencoderrrIrrrOrAdaptiveAvgPool2dr post_initrs r!r*zBitModel.__init__s   %f- !&)   O3 #68K8KB8O P **62  r#Nrrrrc J||n|jj}||n|jj}|j|}|j |||}|d}|j |}|j |}|s ||f|ddzSt|||jS)Nrrrr)r pooler_outputrt) rPrrr&r'rrr rt) r.rrrkwargsembedding_outputencoder_outputsr pooled_outputs r!r?zBitModel.forwards%9$D $++JjJj &1% 1` a classification loss is computed (Cross-Entropy). Nr+rr)losslogitsrt)rPrrr,r8 loss_functionr rt) r.rr9rrr-outputsr0r<r;rs r!r?z!BitForImageClassification.forwards&1%R`k(l1<--'!* /  %%ffdkkBDY,F'+'7D7V# CV C3f\c\q\qrrr#)NNNN) rBrCrDr*rr FloatTensor LongTensorrr r?rFrGs@r!r4r4s 26*.,0#' s''$.s  4's#Tk s D[ s .ssr#r4zL BiT backbone, to be used with frameworks like DETR and MaskFormer. c jeZdZdZfdZeee d dede dzde dzde fdZ xZ S) BitBackboneFct||t||_|jg|j z|_|jyrT)r)r*r$rr|r num_featuresr)rs r!r*zBitBackbone.__init__sD  F##223f6I6II r#Nrrrrc ||n|jj}||n|jj}|j|dd}|j}d}t |j D]\}} | |jvs|||fz }|s|f} |r| |jfz } | St||r|jdSddS)a Examples: ```python >>> from transformers import AutoImageProcessor, AutoBackbone >>> import torch >>> from PIL import Image >>> import httpx >>> from io import BytesIO >>> url = "http://images.cocodataset.org/val2017/000000039769.jpg" >>> with httpx.stream("GET", url) as response: ... image = Image.open(BytesIO(response.read())) >>> processor = AutoImageProcessor.from_pretrained("google/bit-50") >>> model = AutoBackbone.from_pretrained("google/bit-50") >>> inputs = processor(image, return_tensors="pt") >>> outputs = model(**inputs) ```NTr+r) feature_mapsrt attentions) rPrrrrtr stage_names out_featuresr ) r.rrrr-r>rtrFidxrrs r!r?zBitBackbone.forwards<&1%r[s@  &!H .,-(   H %&ERWY]R]L^&R-ryy-`R\\$02990f 2<< 6/BII/f U\\ e T V[VbVb  %")) %A(biiA(HFFR' '.GryyGTC C L88862 !2 2 j ,s 2,s  ,s^ B -!3B  B J Yr#