#idžddlmZddlZddlZddlmZmZddlmZm Z ddl m Z ddl m Z mZddlmZddlZddlZddlmZmZmZdd lmZ ddlZdd lmZerdd lmZej@e!Z"d Z#d Z$dhZ%GddeZ&GddeZ'GddZ(Gdde(eZ)Gdde)Z*d$dZ+ d%dZ,Gdde)Z-Gdde(eeZ.Gdd e.Z/Gd!d"e.Z0gd#Z1y#e$rdZYwxYw)&) annotationsN)ABCabstractmethod) defaultdictdeque)Iterator) accumulatecycle)Any) BatchSampler ConcatDatasetSubsetRandomSampler) ExplicitEnum)is_datasets_available)Datasetll dataset_namec eZdZdZdZdZdZdZy) BatchSamplersa  Stores the acceptable string identifiers for batch samplers. The batch sampler is responsible for determining how samples are grouped into batches during training. Valid options are: - ``BatchSamplers.BATCH_SAMPLER``: **[default]** Uses :class:`~sentence_transformers.base.sampler.DefaultBatchSampler`, the default PyTorch batch sampler. - ``BatchSamplers.NO_DUPLICATES``: Uses :class:`~sentence_transformers.sampler.NoDuplicatesBatchSampler`, ensuring no duplicate samples in a batch. - ``BatchSamplers.NO_DUPLICATES_HASHED``: Uses :class:`~sentence_transformers.sampler.NoDuplicatesBatchSampler` with ``precompute_hashes=True``, a variant that precomputes hashes for faster duplicate checks at a small memory cost. Requires the ``xxhash`` library to be installed. Both are recommended for losses that use in-batch negatives, such as: - :class:`~sentence_transformers.sentence_transformer.losses.MultipleNegativesRankingLoss` - :class:`~sentence_transformers.sentence_transformer.losses.CachedMultipleNegativesRankingLoss` - :class:`~sentence_transformers.sentence_transformer.losses.MultipleNegativesSymmetricRankingLoss` - :class:`~sentence_transformers.sentence_transformer.losses.CachedMultipleNegativesSymmetricRankingLoss` - :class:`~sentence_transformers.sentence_transformer.losses.MegaBatchMarginLoss` - :class:`~sentence_transformers.sentence_transformer.losses.GISTEmbedLoss` - :class:`~sentence_transformers.sentence_transformer.losses.CachedGISTEmbedLoss` - ``BatchSamplers.GROUP_BY_LABEL``: Uses :class:`~sentence_transformers.sampler.GroupByLabelBatchSampler`, which constructs each batch by drawing at least 2 samples from each of at least 2 distinct labels. This guarantees every batch contains multiple classes, which is required for in-batch triplet mining. Recommended for: - :class:`~sentence_transformers.sentence_transformer.losses.BatchAllTripletLoss` - :class:`~sentence_transformers.sentence_transformer.losses.BatchHardSoftMarginTripletLoss` - :class:`~sentence_transformers.sentence_transformer.losses.BatchHardTripletLoss` - :class:`~sentence_transformers.sentence_transformer.losses.BatchSemiHardTripletLoss` If you want to use a custom batch sampler, then you can subclass :class:`~sentence_transformers.base.sampler.DefaultBatchSampler` and pass the class (not an instance) to the ``batch_sampler`` argument in :class:`~sentence_transformers.sentence_transformer.training_args.SentenceTransformerTrainingArguments` (or :class:`~sentence_transformers.cross_encoder.training_args.CrossEncoderTrainingArguments`, etc.). Alternatively, you can pass a function that accepts ``dataset``, ``batch_size``, ``drop_last``, ``valid_label_columns``, ``generator``, and ``seed`` and returns a :class:`~sentence_transformers.base.sampler.DefaultBatchSampler` instance. Usage: :: from sentence_transformers import SentenceTransformer, SentenceTransformerTrainer, SentenceTransformerTrainingArguments from sentence_transformers.sentence_transformer.training_args import BatchSamplers from sentence_transformers.sentence_transformer.losses import MultipleNegativesRankingLoss from datasets import Dataset model = SentenceTransformer("microsoft/mpnet-base") train_dataset = Dataset.from_dict({ "anchor": ["It's nice weather outside today.", "He drove to work."], "positive": ["It's so sunny.", "He took the car to the office."], }) loss = MultipleNegativesRankingLoss(model) args = SentenceTransformerTrainingArguments( output_dir="checkpoints", batch_sampler=BatchSamplers.NO_DUPLICATES, ) trainer = SentenceTransformerTrainer( model=model, args=args, train_dataset=train_dataset, loss=loss, ) trainer.train() batch_sampler no_duplicatesno_duplicates_hashedgroup_by_labelN)__name__ __module__ __qualname____doc__ BATCH_SAMPLER NO_DUPLICATESNO_DUPLICATES_HASHEDGROUP_BY_LABELs/var/www/vps2.regionflexible.com/Desarrollo/venv/lib/python3.12/site-packages/sentence_transformers/base/sampler.pyrr!s BH$M#M1%Nr"rceZdZdZdZdZy)MultiDatasetBatchSamplersa Stores the acceptable string identifiers for multi-dataset batch samplers. The multi-dataset batch sampler is responsible for determining in what order batches are sampled from multiple datasets during training. Valid options are: - ``MultiDatasetBatchSamplers.ROUND_ROBIN``: Uses :class:`~sentence_transformers.base.sampler.RoundRobinBatchSampler`, which uses round-robin sampling from each dataset until one is exhausted. With this strategy, it's likely that not all samples from each dataset are used, but each dataset is sampled from equally. - ``MultiDatasetBatchSamplers.PROPORTIONAL``: **[default]** Uses :class:`~sentence_transformers.base.sampler.ProportionalBatchSampler`, which samples from each dataset in proportion to its size. With this strategy, all samples from each dataset are used and larger datasets are sampled from more frequently. If you want to use a custom multi-dataset batch sampler, then you can subclass :class:`~sentence_transformers.base.sampler.MultiDatasetDefaultBatchSampler` and pass the class (not an instance) to the ``multi_dataset_batch_sampler`` argument in :class:`~sentence_transformers.sentence_transformer.training_args.SentenceTransformerTrainingArguments`. (or :class:`~sentence_transformers.cross_encoder.training_args.CrossEncoderTrainingArguments`, etc.). Alternatively, you can pass a function that accepts ``dataset`` (a :class:`~torch.utils.data.ConcatDataset`), ``batch_samplers`` (i.e. a list of batch sampler for each of the datasets in the :class:`~torch.utils.data.ConcatDataset`), ``generator``, and ``seed`` and returns a :class:`~sentence_transformers.base.sampler.MultiDatasetDefaultBatchSampler` instance. Usage: :: from sentence_transformers import SentenceTransformer, SentenceTransformerTrainer, SentenceTransformerTrainingArguments from sentence_transformers.sentence_transformer.training_args import MultiDatasetBatchSamplers from sentence_transformers.sentence_transformer.losses import CoSENTLoss from datasets import Dataset, DatasetDict model = SentenceTransformer("microsoft/mpnet-base") train_general = Dataset.from_dict({ "sentence_A": ["It's nice weather outside today.", "He drove to work."], "sentence_B": ["It's so sunny.", "He took the car to the bank."], "score": [0.9, 0.4], }) train_medical = Dataset.from_dict({ "sentence_A": ["The patient has a fever.", "The doctor prescribed medication.", "The patient is sweating."], "sentence_B": ["The patient feels hot.", "The medication was given to the patient.", "The patient is perspiring."], "score": [0.8, 0.6, 0.7], }) train_legal = Dataset.from_dict({ "sentence_A": ["This contract is legally binding.", "The parties agree to the terms and conditions."], "sentence_B": ["Both parties acknowledge their obligations.", "By signing this agreement, the parties enter into a legal relationship."], "score": [0.7, 0.8], }) train_dataset = DatasetDict({ "general": train_general, "medical": train_medical, "legal": train_legal, }) loss = CoSENTLoss(model) args = SentenceTransformerTrainingArguments( output_dir="checkpoints", multi_dataset_batch_sampler=MultiDatasetBatchSamplers.PROPORTIONAL, ) trainer = SentenceTransformerTrainer( model=model, args=args, train_dataset=train_dataset, loss=loss, ) trainer.train() round_robin proportionalN)rrrr ROUND_ROBIN PROPORTIONALr!r"r#r%r%ls@D K!Lr"r%c,eZdZdZdfd ZddZxZS) SetEpochMixinz Required for a BatchSampler as the Trainer will call set_epoch on the BatchSampler at the beginning of each epoch. The BatchSampler can then set the generator seed accordingly. c2t||i|d|_yNr)super__init__epoch)selfargskwargs __class__s r#r/zSetEpochMixin.__init__s $)&) r"c||_yN)r0)r1r0s r# set_epochzSetEpochMixin.set_epochs  r"returnNone)r0intr9r:)rrrrr/r7 __classcell__r4s@r#r+r+s r"r+cFeZdZdZ d dfd ZxZS)DefaultBatchSampleraF This sampler is the default batch sampler used in the SentenceTransformer library. It is equivalent to the PyTorch BatchSampler. Args: dataset (Dataset): The dataset to sample from. batch_size (int): Number of samples per batch. drop_last (bool): If True, drop the last incomplete batch if the dataset size is not divisible by the batch size. valid_label_columns (List[str], optional): List of column names to check for labels. The first column name from ``valid_label_columns`` found in the dataset will be used as the label column. generator (torch.Generator, optional): Optional random number generator for shuffling the indices. seed (int): Seed for the random number generator to ensure reproducibility. Defaults to 0. cTt||||||_||_||_y)N batch_size drop_last)r.r/valid_label_columns generatorseed)r1datasetrBrCrDrErFr4s r#r/zDefaultBatchSampler.__init__s/ Z9M#6 " r"NNrrGrrBr;rCboolrDlist[str] | NonerEtorch.Generator | NonerFr;r9r:)rrrrr/r<r=s@r#r?r?sa,15,0     .  *     r"r?cheZdZdZ d dfd ZeddZd dZd dZxZ S) GroupByLabelBatchSamplera Batch sampler that groups samples by label for in-batch triplet mining. Samples are shuffled within each label, then interleaved in round-robin fashion to produce a stream where labels are well-mixed. This stream is chunked into batches of exactly ``batch_size``. Every batch is guaranteed to contain multiple distinct labels, each with at least 2 samples. Labels take turns emitting 2 samples each. The stream stops when fewer than 2 labels remain, so the dominant label's tail ends up in the remainder. Produces excellent per-batch balance. Recommended for: - :class:`~sentence_transformers.sentence_transformer.losses.BatchAllTripletLoss` - :class:`~sentence_transformers.sentence_transformer.losses.BatchHardSoftMarginTripletLoss` - :class:`~sentence_transformers.sentence_transformer.losses.BatchHardTripletLoss` - :class:`~sentence_transformers.sentence_transformer.losses.BatchSemiHardTripletLoss` Args: dataset (Dataset): The dataset to sample from. batch_size (int): Number of samples per batch. Must be an even number >= 4. drop_last (bool): If True, drop the last incomplete batch if the dataset size is not divisible by the batch size. valid_label_columns (List[str], optional): List of column names to check for labels. The first column name from ``valid_label_columns`` found in the dataset will be used as the label column. generator (torch.Generator, optional): Optional random number generator for shuffling the indices. seed (int): Seed for the random number generator to ensure reproducibility. Defaults to 0. c  t|||||||||_|jdks|jdzdk(rt d|jd|j ||j }tt}t|D]\} } || j| |jD cic])\} } t| dk\s| | dt| dzdz+c} } |_ t|jdkr"t dt|jdtd |jjDd } | d dt! fd | Dz|_ycc} } w) NrBrCrDrErFz0batch_size must be an even number >= 4, but got .z^GroupByLabelBatchSampler requires at least 2 distinct labels with >= 2 samples each, but only z label(s) qualified.c38K|]}t|dzyw)rRNlen).0idxs r# z4GroupByLabelBatchSampler.__init__..)sF#CA FsT)reversec36K|]}t|ywr6)min)rXpcaps r#rZz4GroupByLabelBatchSampler.__init__..+s%Aac!Sk%As)r.r/rGrB ValueError_determine_labels_to_userDrlist enumerateappenditemsrWgroupssortedvaluessum_stream_length)r1rGrBrCrDrErFlabelsrf sample_idxlabelindicespairsr_r4s @r#r/z!GroupByLabelBatchSampler.__init__s  ! 3   ??Q $//A"5":OPTP_P_O``abc c..w8P8PQ'24'8!*6!2 - J 5M  , - KQ,,. 8Fw\_`g\hlm\mE72S\Q.23 3   t{{ a  ,--AC  F1C1C1EFPTUAh#%A5%A"AA s >E=E=cz|xsgD]}||jvs||cStd|d|jd)Nz None of the valid_label_columns z3 are in the dataset, which only has these columns: rT) column_namesr`)rGrD column_names r#raz1GroupByLabelBatchSampler._determine_labels_to_use-sa.4" ,Kg222{++ ,./B.CD--4-A-A,B! E  r"c#K|jr>|j2|jj|j|jzi}|jj D]F\}t jt|j}tfd|D||<Ht|}g}t|dk\rt jt||jDcgc]}|| }}|D]}|j||j|j||jt||jk\s`|d|j||jd}|Dcgc] }||s | }}t|dk\r|jst|dk\r|yyycc}wcc}ww)NrEc3(K|] }| ywr6r!)rXirns r#rZz4GroupByLabelBatchSampler.__iter__..?s!;'!*!;srRrQ)rErF manual_seedr0rfretorchrandpermrWrrbrdpopleftrBrC)r1queuesrmpermremaining_labelsbatchrvrns @r#__iter__z!GroupByLabelBatchSampler.__iter__7s >>dii3 NN & &tyy4::'= >)+"kk//1 >#g,$..ID!!;d!;;F5M <  <"#q(-2^^C@P!,-.c$tTYT`T`TbOcegJhFiJ z):;AR;<< F$" " $ 9F&M'*E   mCJ7 8  9  j!"  U"?+:( 0 "r"c |jr |jyt|jj}|jj t d|j|j||tdd}ddl } |jjd}t||jr|j}t||j |j"fs t%dt'|}|dk(r-t)j*dt(j, |_ ~y|j.j1d }t3|d |dz }|dks+t)j4t)j6||k(s t%d |j8j1d j;t(j,d }|j<||zk7r t%d|j?||f|_~y#~wxYw)NT)rrzHashing dataset values)batchedrBnum_procr fn_kwargsdescrrz)Expected a list column for hashed values.)rrdtypeF)zero_copy_onlyrSz!Hashed rows have varying lengths.)copyz$Unexpected hashed value buffer size.) rrrbrGrqmaprrr_EXCLUDE_DATASET_COLUMNSpyarrowdatar isinstance ChunkedArraycombine_chunks ListArrayLargeListArrayr`rWnpzerosint64offsetsto_numpyr;alldiffrhastypesizereshape) r1rhash_dspar row_countrrow_sizerhs r# _build_hashesz&NoDuplicatesBatchSampler._build_hashess%%)9)9)E t||001,,"" 11--"")>VW)#   \\((4F&"//2..0fr||R5F5F&GH !LMMF IA~#%88F"((#C nn--U-CG71: 23H!|266"'''*:h*F#G !DEE]]++5+AHHX]H^F{{i(22 !GHH%~~y(.CDD s?B(H*CHH c#Kjr>j2jjjjzjr#j j d fd }nd fd }d d}tj}|dk(ry|tjtjjkrtjntj}tj|j|j!}|dztjtjjkrtjntj}tj"d|dz|}d |d <d}|d k7rt%} g} |} d } d } | d k7rt'|| }t'|| }||}||| r| } |} 7| j)|| d k(r|}n||| <t| j*k(rd } | n| j-||} | d k7r| sj.s| |d k7ryyw)a5 Iterate over the remaining non-yielded indices. For each index, check if the sample values are already in the batch. If not, add the sample values to the batch keep going until the batch is full. If the batch is full, yield the batch indices and continue with the next batch. Nc|Sr6r!)indexrs r#get_sample_valuesz.get_sample_valuess!%((r"cj|jDchc]\}}|tvst|c}}Scc}}wr6)rGrerr)rkeyrr1s r#rz.get_sample_valuessC15e1D1J1J1L#-3PS[sPsCJs A Acpt|tr|j Stfd|DS)Nc3&K|]}|v ywr6r!)rXr batch_valuess r#rZzJNoDuplicatesBatchSampler.__iter__.._has_overlap..sHu ,Hs)rr isdisjointany) sample_valuesrs `r# _has_overlapz7NoDuplicatesBatchSampler.__iter__.._has_overlap s2--(33LAAAH-HH Hr"r)rErrSrFT)rr;r9set[str] | np.ndarray)rrrzset[str | np.int64]r9rJ)rErFrwr0rrrrWrGriinfoint32rrxrrynumpyarangerr;rdrBupdaterC)r1rrnum_rows index_dtyperemaining_indicesposition_dtypenext_positions head_positionr batch_indicescurrent_positionprevious_position full_batch next_positionrrrs` @r#rz!NoDuplicatesBatchSampler.__iter__s >>dii3 NN & &tyy4::'= >  ! !    %)%5%5J )    It||$ q= &."((1C1G1G%GekkU[[ !NN8t~~U`aggi&.\RXXbhh5G5K5K%KQSQYQY1hl.Ir r!03L')M,  " J"b( #N3C$D E -.>?@ 1% 8  |<(8%'4$$$U+$*$1M8EN#45}%8!%J''##M2#0 -#b(0~~''Cr!sH7I<IIc|jr"t|j|jzSt|j|jzdz |jzS)a_Return the approximate number of batches. .. note:: This is an upper-bound estimate. The actual number of batches yielded by :meth:`__iter__` may be smaller when the dataset contains many duplicate values, because those samples are deferred or skipped rather than placed into a batch. rS)rCrWrGrBr1s r#rz NoDuplicatesBatchSampler.__len__CsI >>t||$7 7 %7!;O Or")NNrFNi)rGrrBr;rCrJrDrKrErLrFr;rrJrz int | Nonerr;r9r:r8rr)rrrr/rrrr<r=s@r#rrys 15,0"'*.%)I;I;I; I; . I; * I;I; I;(I; #I; I;V,\N(` Pr"rc`eZdZdZ d dfd ZeddZeddZxZS) MultiDatasetDefaultBatchSampleraT Abstract base batch sampler that yields batches from multiple batch samplers. This class must be subclassed to implement specific sampling strategies, and cannot be used directly. Args: dataset (ConcatDataset): A concatenation of multiple datasets. batch_samplers (List[BatchSampler]): A list of batch samplers, one for each dataset in the ConcatDataset. generator (torch.Generator, optional): A generator for reproducible sampling. Defaults to None. seed (int): Seed for the random number generator to ensure reproducibility. Defaults to 0. ct|jt|k7r tdt|||dj |dj ||_||_||_ ||_ y)NzTThe number of batch samplers must match the number of datasets in the ConcatDataset.rrA) rWdatasetsr`r.r/rBrCrGbatch_samplersrErF)r1rGrrErFr4s r#r/z(MultiDatasetDefaultBatchSampler.__init__`sr w C$7 7st t ^A->-I-IUcdeUfUpUpq ," r"cy)z?Yield batches from the underlying datasets in a specific order.Nr!rs r#rz(MultiDatasetDefaultBatchSampler.__iter__o r"cy)z,Return the number of batches in the sampler.Nr!rs r#rz'MultiDatasetDefaultBatchSampler.__len__trr"r-) rGr rzlist[BatchSampler]rErLrFr;r9r:rr) rrrrr/rrrr<r=s@r#rrSsm  -1  + *         r"rc eZdZdZddZddZy)RoundRobinBatchSamplera Batch sampler that yields batches in a round-robin fashion from multiple batch samplers, until one is exhausted. With this sampler, it's unlikely that all samples from each dataset are used, but we do ensure that each dataset is sampled from equally. Args: dataset (ConcatDataset): A concatenation of multiple datasets. batch_samplers (List[BatchSampler]): A list of batch samplers, one for each dataset in the ConcatDataset. generator (torch.Generator, optional): A generator for reproducible sampling. Defaults to None. seed (int): Seed for the random number generator to ensure reproducibility. Defaults to 0. c#<K|jr>|j2|jj|j|jz|jj Dcgc] }t |}}dgtt|z}|jDcgc] }t|}}ttt |D])}||} t||Dcgc]}||z c}+ycc}wcc}wcc}w#t$rYywxYwwr-)rErFrwr0rGrrWrbr rrr rr StopIteration) r1rG num_samplessample_offsetssamplerr dataset_idx sample_offsetrYs r#rzRoundRobinBatchSampler.__iter__s >>dii3 NN & &tyy4::'= >37<<3H3HIs7|I ItJ{$;<<7;7J7JKG$w-KK s>':!;< K*;7M 6:>+;V6WXss]*XX  JLY   sTA#D%C=7(DD1'DD ) D5D : DD  DDDDcftd|jDt|jzS)Nc32K|]}t|ywr6rV)rXrs r#rZz1RoundRobinBatchSampler.__len__..sCG3w<Cs)r]rrWrs r#rzRoundRobinBatchSampler.__len__s)Ct/B/BCCc$J]J]F^^^r"Nrrrrrrrrr!r"r#rrzs  _r"rc eZdZdZddZddZy)ProportionalBatchSamplera| Batch sampler that samples from each dataset in proportion to its size, until all are exhausted simultaneously. With this sampler, all samples from each dataset are used and larger datasets are sampled from more frequently. Args: dataset (ConcatDataset): A concatenation of multiple datasets. batch_samplers (List[BatchSampler]): A list of batch samplers, one for each dataset in the ConcatDataset. generator (torch.Generator, optional): A generator for reproducible sampling. Defaults to None. seed (int): Seed for the random number generator to ensure reproducibility. Defaults to 0. c#K|jr>|j2|jj|j|jz|jj Dcgc] }t |}}dgtt|z}|jDcgc] }t |}}t|Dcgc]\}}t|D]}|} }}}t| |j} |jDcgc] }t|} }| D])} || } t| | Dcgc]}|| z c}+ycc}wcc}wcc}}}wcc}wcc}w#t$rYSwxYww)Nrrt)rErFrwr0rGrrWrbr rrcrrrrr)r1rGrrr num_batchesrYlengthrdataset_indicesdataset_idx_samplerrrrs r#rz!ProportionalBatchSampler.__iter__sS >>dii3 NN & &tyy4::'= >37<<3H3HIs7|I ItJ{$;<<373F3FGs7|G G2;K2H``;3RWX^R_`Q3`3``1/T^^\7;7J7JKG$w-KK. K*;7M 6:>+;V6WXss]*XX JH`LY   slA#E4%E 7(E4E1E4E)E4E E4&E%6 E E%E4 E%% E1.E40E11E4cdt|jDcgc] }t|c}Scc}wr6)rirrW)r1rs r#rz ProportionalBatchSampler.__len__s%0C0CDWCLDEEDs-Nrrrr!r"r#rrs &Fr"r)r?rNrrrrrr%)rrr9r;)r~zdict[str, list[Any]]rz list[str]rzset[str]r9zdict[str, list[list[int]]])2 __future__rloggingrabcrr collectionsrrcollections.abcr itertoolsr r typingr rrrxtorch.utils.datar r rtransformers.utilsrrrsentence_transformers.utilrrr getLoggerrloggerrrrrr%r+r?rNrrrrrr__all__r!r"r#rs;" #*$' MM+=   8 $*+H&LH&VD" D"N  -Bw2wt  *3 FN  *WP2WPt$ m\3$ N_<_B F> FF Y Fs C66D?D