:i/*SSKJr /SQrSrSrSrg))safe_isinstance)ztransformers.MarianTokenizerztransformers.T5Tokenizerztransformers.XLNetTokenizerztransformers.AlbertTokenizerc[US5(d[US5(aGSSKJnJn [ U5UR 5;=(d [ U5UR 5;$g)zMCheck if the given model object is a huggingface transformers language model.ztransformers.PreTrainedModelztransformers.TFPreTrainedModel)MODEL_FOR_CAUSAL_LM_MAPPING&MODEL_FOR_SEQ_TO_SEQ_CAUSAL_LM_MAPPINGF)r transformersrrtypevalues)modelrrs Z/srv/projetos/modelo_ml_acdoc/venv/lib/python3.13/site-packages/shap/utils/transformers.pyis_transformers_lmr sgu<== /BB KAHHJ J CE{9@@BB  cU"S5SnSup#[U5S:XaUSn[US5(av[US5(aeURnSU;d S U;dS 5eSU;aURU5US:XaSnSnOcS U;aURU5US :XaSnSnO@[ S 5e[U5S -S:XdS 5e[U5S -n[U5S -nUUUS.$)zSet prefix and suffix tokens based on null tokens. Example for distillgpt2: null_tokens=[], for BART: null_tokens = [0,2] and for MarianMT: null_tokens=[0] used to slice tokens belonging to sentence after passing through tokenizer.encode(). input_ids)NNrrspecial_tokens_mapdecode eos_token bos_tokenz-No eos token or bos token found in tokenizer!zThe given tokenizer produces one token when applied to the empty string, but does not have a .special_tokens_map['eos_token'] or .special_tokens_map['bos_token'] property (and .decode) to specify if it is an eos (end) of bos (beginning) token!z>An odd number of boundary tokens are added to the null string!) keep_prefix keep_suffix null_tokens)lenhasattrrr Exception) tokenizerrrr null_tokenst_maps r !parse_prefix_suffix_for_tokenizerr s1 B- ,K)K ;1 ^ 92 3 3 88T8T11F6){f/D vGv vEv%I,<,.>z.JfU`Na.a  d  ;!#q(j*jj(+&!+ +&!+ #"#  rcSn[USS5(a SnSUl[XS5nU(aSUlUS:XaSnU$)zThis turns of verbose logging of missing attributes for huggingface transformers. This is motivated by huggingface transformers objects that print error warnings when we access unset properties. FverboseTNNone)getattrr")objattr reset_verbosevals r getattr_silentr)JsM MsIu%%  #T "C  f} JrN)_generalrSENTENCEPIECE_TOKENIZERSr r r)rr r-s %"+\r