irdZddlZddlmZddlmZddlmZedeGd d eZd gZ y) zMAMBA configurationN)strict)PreTrainedConfig)auto_docstringzstate-spaces/mamba-2.8b) checkpointceZdZUdZdZdZeed<dZeed<dZ eed<d Z eed <d Z e ed <d Z edzed<d Zedzed<d Zeeezdzed<dZeed<dZeed<dZeed<dZeed<dZeed<dZe ed<dZeed<dZeezed <d!Ze ed"<d#Ze ed$<dZe ed%<d&Zeed'<d(Ze ed)<dZ eed*<dZ!eed+<dZ"eed,<dZ#eed-<dZ$eed.<fd/Z%e&d0Z'xZ(S)1 MambaConfiga: layer_norm_epsilon (`float`, *optional*, defaults to 1e-05): The epsilon to use in the layer normalization layers. expand (`int`, *optional*, defaults to 2): Expanding factor used to determine the intermediate size. use_bias (`bool`, *optional*, defaults to `False`): Whether or not to use bias in ["in_proj", "out_proj"] of the mixer block use_conv_bias (`bool`, *optional*, defaults to `True`): Whether or not to use bias in the convolution layer of the mixer block. residual_in_fp32 (`bool`, *optional*, defaults to `True`): Whether or not residuals should be in `float32`. If set to `False` residuals will keep the same `dtype` as the rest of the model rescale_prenorm_residual (`bool`, *optional*, defaults to `False`): Whether or not to rescale `out_proj` weights when initializing. use_mambapy (`bool`, *optional*, defaults to `False`): Determines the fallback strategy during training if the CUDA-based official implementation of Mamba is not available. If `True`, the mamba.py implementation is used. If `False`, the naive and slower implementation is used. Consider switching to the naive version if memory is limited. use_associative_scan (`bool`, *optional*, defaults to `True`): Whether to use PyTorch's `torch._higher_order_ops.associative_scan` for the parallel scan instead of the naive sequential implementation. The associative scan is only active during `torch.compile` tracing and requires torch >= 2.9.0. Both paths are tested to produce numerically identical results (see `test_associative_scan_matches_sequential`). Set to `False` to fall back to the sequential loop. Example: ```python >>> from transformers import MambaConfig, MambaModel >>> # Initializing a Mamba configuration >>> configuration = MambaConfig() >>> # Initializing a model (with random weights) from the configuration >>> model = MambaModel(configuration) >>> # Accessing the model configuration >>> configuration = model.config ```mambaih vocab_sizei hidden_size state_size num_hidden_layersgh㈵>layer_norm_epsilonrN pad_token_id bos_token_id eos_token_idexpand conv_kernelFuse_biasT use_conv_biassilu hidden_actg?initializer_rangeresidual_in_fp32autotime_step_rankg?time_step_scalegMbP? time_step_min time_step_maxrandomtime_step_init_schemeg-C6?time_step_floorrescale_prenorm_residual use_cache use_mambapyuse_associative_scantie_word_embeddingsc t|j|jz|_|jdk(r"t j |jdz n |j|_t| di|y)Nrr ) intrr intermediate_sizer mathceilsuper __post_init__)selfkwargs __class__s ~/var/www/vps2.regionflexible.com/Desarrollo/venv/lib/python3.12/site-packages/transformers/models/mamba/configuration_mamba.pyr3zMambaConfig.__post_init__^sf!$T[[43C3C%C!D040C0Cv0MDIId&&+ ,SWSfSf  ''c"dg|jzS)Nr )r)r4s r7 layer_typeszMambaConfig.layer_typesesy41111r8))__name__ __module__ __qualname____doc__ model_typer r.__annotations__r rrrfloatrrrlistrrrboolrrstrrrr r!r"r#r%r&r'r(r)r*r+r3propertyr: __classcell__)r6s@r7r r sM$ LJJKJs $$ L#*  L#* +,L#S /D(,FCOKHdM4J"u"!d! &NC#I& OU  M5 M5!)3)!OU!%*d*ItK!%$% $$(22r8r ) r>r0huggingface_hub.dataclassesrconfiguration_utilsrutilsrr __all__r-r8r7rKsM .3#45M2"M26M2` /r8