iPddlmZmZmZmZmZddlmZmZddl m Z GddZ y))DictIterableListUnioncast) has_torch_amptorch)is_torch_arrayc eZdZdZ ddededededef dZdZ dd e d e d fd e d e d ffd Z d d de ddefdZdZedZdZdZy)PyTorchGradScalera Gradient scaler for the PyTorch shim. Gradients with small magnitudes are not representable in half-precision and will underflow to zero. A gradient scaler counters this issue by scaling up the loss before backpropagation, increasing the gradients by the same magnitude. A large enough scale will avoid that the gradients underflow. The gradients are unscaled in single precision after backpropagation, to provide the unscaled gradients to the optimizer. enabled init_scalebackoff_factor growth_factorgrowth_intervalc||_||_||_||_t j ddtj |_t j d||_d|_ y)a Construct a gradient scaler for the PyTorch shim. enabled (bool): Sets whether the gradient scalar is enabled. If it is disabled, the methods of the grad scaler are no-ops. init_scale (float): The initial scale used to increase the gradient magnitude. backoff_factor (float): The scale will be multiplied by this factor if any of the gradients overflows. growth_factor (float): The scale will be multiplied by this factor when none of the gradients overflowed for "growth_interval" steps. growth_interval (int): When no overflows were found for this number of steps, the scale will be multiplied by "growth_factor". r)dtypeFN) _enabled_growth_factor_backoff_factor_growth_intervalr fullint_growth_tracker_scale _found_inf)selfrrrrrs p/var/www/vps2.regionflexible.com/Desarrollo/venv/lib/python3.12/site-packages/thinc/shims/pytorch_grad_scaler.py__init__zPyTorchGradScaler.__init__sU<  +- /$zz$Cjjz2 c|jj||_|jj||_yN)rtor)r devices r!to_zPyTorchGradScaler.to_:s/#3366v>kknnV, r#tensors torch.TensorreturncH|js td|Std}t}t |rtd|}|j |||St |tr:g}|D]1}t |s||j|j |||3|S|)z)Scale up the values in the given tensors.r*z>Input to gradient scaling must be a Tensor or Iterable[Tensor]) rr ValueErrordictr _scale_tensor isinstancerappend)r r)inplaceincorrect_typescale_per_devicetensorscaled_tensorss r!scalezPyTorchGradScaler.scale>s}}0 0# L  BF ' ".'2F%%f.>H H  *N! %f-((%%&&v/?I  " !r#r5r4)z torch.devicer*r2cts td|js d}t||j}||vr|jj |||<||}|r|j |S||zS)NzHGradient scaling is not supported, requires capable GPU and torch>=1.9.0zGradient scaling is only supported for CUDA tensors. If you are using PyTorch models, you can avoid this error by disabling mixed-precision support.r')r r-is_cudar'rr&mul_)r r5r4r2msgr'r7s r!r/zPyTorchGradScaler._scale_tensor^s Z ~~>  S/ ! ) )'+{{~~V~'D V $ ( ;;u% %E> !r#ct}|D]/}|j|jg}|j|1|Sr%)r. setdefaultr'r1)r r)tensors_per_devicer5device_tensorss r!_tensors_per_devicez%PyTorchGradScaler._tensors_per_device|sE!V *F/::6=="MN  ! !& ) *"!r#c|jSr%)r)r s r! found_infzPyTorchGradScaler.found_infs r#c|jsy|jjjj }|j |}|j D]\\}}tjdd|}|j|}tj|||t|dk7sVd|_ ^|jS)zNUnscale the given tensors. Returns True if any of the gradients were infinite.Frr9rT) rrdouble reciprocalfloatrAitemsr rr&*_amp_foreach_non_finite_check_and_unscale_boolr)r r) inv_scaler?r'r@found_inf_deviceinv_scale_devices r!unscalezPyTorchGradScaler.unscales}}KK&&(335;;= "55g>&8&>&>&@ ' "FN$zz$FC (||6|:   < < 02B $)*"& 'r#c8|jsytjd|jrdnd|jj }tj |j|j||j|j|jd|_y)z Update the scale factor and clear information about infinities. This method should be called after each optimization step. Nrg?rEr9F) rr rrrr'_amp_update_scale_rrrr)r rMs r!updatezPyTorchGradScaler.updates} }}  :: #c$++:L:L    KK        ! !   r#N)Fg@g?g@i)F)__name__ __module__ __qualname____doc__rKrHrr"r(rrrr7rr/rApropertyrCrOrRr#r!r r s # #"# % % %  %  %  % N- QV^Xn-EEF ~tN33 4@""=>" "<". r#r N) typingrrrrrcompatr r utilr r rXr#r!r\s44)!n n r#