u:i]xSrSSKJrJrJrJrJr SSKrSSK J r SSK J r J r SSKJrJr SSKJr SS KJrJrJr S S KJrJrJr S S KJr S \\S\\SS4Sjr"SS\5r"SS\5r\"\\5 "SS\\\5r "SS\5r!\"\ \!5 "SS\5r""SS\5r#\"\"\#5 g)z8Xgboost pyspark integration submodule for estimator API.)AnyListOptionalTypeUnionN) keyword_only)ParamParams)HasProbabilityColHasRawPredictionCol)Config) XGBClassifier XGBRanker XGBRegressor)_ClassificationModel_SparkXGBEstimator_SparkXGBModel)get_class_name estimatormodelreturnc^^^ TR5nS[S[4U 4Sjjm S[S[SS4UUU 4SjjnUR 5HGnS[ TR 55S U3n[[R"5XES 9nU"XF5 MI TR5nUR 5HRnS[ TR 55S U3nUS :XaUS - n[[R"5XES 9nU"XF5 MT TR5nUR 5HGnS[ TR 55SU3n[[R"5XES 9nU"XF5 MI g)zuThis function automatically infer to xgboost parameters and set them into corresponding pyspark estimators and modelsvrc>[U[R5(a$[R"U5R 5$[U[ 5(a,UR 5VVs0sHupUT"U5_M snn$[U[5(aUVs/sH nT"U5PM sn$U$s snnfs snfN) isinstancenpgenericarrayitemdictitemslist)rknvparam_value_converters Z/srv/projetos/modelo_ml_acdoc/venv/lib/python3.13/site-packages/xgboost/spark/estimator.pyr(?_set_pyspark_xgb_cls_param_attrs..param_value_converters a $ $88A;##% % a  >?ggiHiUQA,R00iH H a  89:")"-: :I:s -B6B< attr_nameparamNcD>TUl[TX5 [TX5 gr) typeConvertersetattr)r+r,rrr(s r)set_param_attrs9_set_pyspark_xgb_cls_param_attrs..set_param_attrs(s3 9,y(zRefer to XGBoost doc of z for this param )namedocz.fit() for this param callbackszThe callbacks can be arbitrary functions. It is saved using cloudpickle which is not a fully self-contained format. It may fail to load with different versions of dependencies.z.predict() for this param ) _get_xgb_params_defaultrstrr keysr_xgb_clsr _dummy_get_fit_params_default_get_predict_params_default) rr params_dictr0r3r4 param_objfit_params_dictpredict_params_dictr(s `` @r) _set_pyspark_xgb_cls_param_attrsrAs 335K)3)u)))   "&i002344DTF L !tE (# 779O$$&&~i6H6H6J'K&L$TF ,  ;  6 C &--/> ('$??A#((*&~i6H6H6J'K&L( 0 &--/> ( +r2c"^\rSrSrSr\SSSSSSSSSSSSS SS .S \\\\4S \S \S\ \S\ \S\ \S\ \S\ S\ \S\ S\ S\ S\ S\ \ S\ SS4 U4Sjjj5r\S\\4Sj5r\S\S4Sj5rS!U4SjjrS rU=r$)"SparkXGBRegressorOaSparkXGBRegressor is a PySpark ML estimator. It implements the XGBoost regression algorithm based on XGBoost python library, and it can be used in PySpark Pipeline and PySpark ML meta algorithms like - :py:class:`~pyspark.ml.tuning.CrossValidator`/ - :py:class:`~pyspark.ml.tuning.TrainValidationSplit`/ - :py:class:`~pyspark.ml.classification.OneVsRest` SparkXGBRegressor automatically supports most of the parameters in :py:class:`xgboost.XGBRegressor` constructor and most of the parameters used in :py:meth:`xgboost.XGBRegressor.fit` and :py:meth:`xgboost.XGBRegressor.predict` method. To enable GPU support, set `device` to `cuda` or `gpu`. SparkXGBRegressor doesn't support setting `base_margin` explicitly as well, but support another param called `base_margin_col`. see doc below for more details. SparkXGBRegressor doesn't support `validate_features` and `output_margin` param. SparkXGBRegressor doesn't support setting `nthread` xgboost param, instead, the `nthread` param for each xgboost worker will be set equal to `spark.task.cpus` config value. Parameters ---------- features_col: When the value is string, it requires the features column name to be vector type. When the value is a list of string, it requires all the feature columns to be numeric types. label_col: Label column name. Default to "label". prediction_col: Prediction column name. Default to "prediction" pred_contrib_col: Contribution prediction column name. validation_indicator_col: For params related to `xgboost.XGBRegressor` training with evaluation dataset's supervision, set :py:attr:`xgboost.spark.SparkXGBRegressor.validation_indicator_col` parameter instead of setting the `eval_set` parameter in `xgboost.XGBRegressor` fit method. weight_col: To specify the weight of the training and validation dataset, set :py:attr:`xgboost.spark.SparkXGBRegressor.weight_col` parameter instead of setting `sample_weight` and `sample_weight_eval_set` parameter in `xgboost.XGBRegressor` fit method. base_margin_col: To specify the base margins of the training and validation dataset, set :py:attr:`xgboost.spark.SparkXGBRegressor.base_margin_col` parameter instead of setting `base_margin` and `base_margin_eval_set` in the `xgboost.XGBRegressor` fit method. num_workers: How many XGBoost workers to be used to train. Each XGBoost worker corresponds to one spark task. device: .. versionadded:: 2.0.0 Device for XGBoost workers, available options are `cpu`, `cuda`, and `gpu`. force_repartition: Boolean value to specify if forcing the input dataset to be repartitioned before XGBoost training. repartition_random_shuffle: Boolean value to specify if randomly shuffling the dataset when repartitioning is required. enable_sparse_data_optim: Boolean value to specify if enabling sparse data optimization, if True, Xgboost DMatrix object will be constructed from sparse matrix instead of dense matrix. launch_tracker_on_driver: Boolean value to indicate whether the tracker should be launched on the driver side or the executor side. coll_cfg: The collective configuration. See :py:class:`~xgboost.collective.Config` kwargs: A dictionary of xgboost parameters, please refer to https://xgboost.readthedocs.io/en/stable/parameter.html Note ---- The Parameters chart above contains parameters that need special handling. For a full list of parameters, see entries with `Param(parent=...` below. This API is experimental. Examples -------- >>> from xgboost.spark import SparkXGBRegressor >>> from pyspark.ml.linalg import Vectors >>> df_train = spark.createDataFrame([ ... (Vectors.dense(1.0, 2.0, 3.0), 0, False, 1.0), ... (Vectors.sparse(3, {1: 1.0, 2: 5.5}), 1, False, 2.0), ... (Vectors.dense(4.0, 5.0, 6.0), 2, True, 1.0), ... (Vectors.sparse(3, {1: 6.0, 2: 7.5}), 3, True, 2.0), ... ], ["features", "label", "isVal", "weight"]) >>> df_test = spark.createDataFrame([ ... (Vectors.dense(1.0, 2.0, 3.0), ), ... (Vectors.sparse(3, {1: 1.0, 2: 5.5}), ) ... ], ["features"]) >>> xgb_regressor = SparkXGBRegressor(max_depth=5, missing=0.0, ... validation_indicator_col='isVal', weight_col='weight', ... early_stopping_rounds=1, eval_metric='rmse') >>> xgb_reg_model = xgb_regressor.fit(df_train) >>> xgb_reg_model.transform(df_test) featureslabel predictionNrFT) features_col label_colprediction_colpred_contrib_colvalidation_indicator_col weight_colbase_margin_col num_workersdeviceforce_repartitionrepartition_random_shuffleenable_sparse_data_optimlaunch_tracker_on_drivercoll_cfgrHrIrJrKrLrMrNrOrPrQrRrSrTrUkwargsrc ^>[TU]5 URnUR"S0UD6 gNsuper__init__ _input_kwargs setParams)selfrHrIrJrKrLrMrNrOrPrQrRrSrTrUrV input_kwargs __class__s r)r\SparkXGBRegressor.__init__s*( ))  &&r2c[$rrclss r)r9SparkXGBRegressor._xgb_clsr2SparkXGBRegressorModelc[$r)rires r)_pyspark_model_cls$SparkXGBRegressor._pyspark_model_clss%%r2cx>[TU]5 URUR5(a [ S5eg)NzCSpark Xgboost regressor estimator does not support `qid_col` param.r[_validate_params isDefinedqid_col ValueErrorr_ras r)ro"SparkXGBRegressor._validate_paramss6  " >>$,, ' 'U  (r2rYrN)__name__ __module__ __qualname____firstlineno____doc__rrr7rrintboolrrr\ classmethodrrr9rkro__static_attributes__ __classcell__ras@r)rCrCOs[ob/9 **.26$()- $"'+0).)-%)!'CcN+' '  ' #3- '#+3-'SM'"#'' ' '%)'#''#'' 6"!'"#'$ %''.l+&4(@#A&&r2rCc6\rSrSrSr\S\\4Sj5rSr g)rizh The model returned by :func:`xgboost.spark.SparkXGBRegressor.fit` .. Note:: This API is experimental. rc[$rrdres r)r9SparkXGBRegressorModel._xgb_clsrhr2rYN) rvrwrxryrzr}rrr9r~rYr2r)riris' l+r2ric&(^\rSrSrSr\SSSSSSSSSS SS S S S SS .S \\\\4S\S\S\S\S\ \S\ \S\ \S\ \S\ S\ \S\ S\ S\ S\ S\ \ S\ SS4$U4Sjjj5r\S\\4S j5r\S\S!4S"j5rS%U4S#jjrS$rU=r$)&SparkXGBClassifieraSparkXGBClassifier is a PySpark ML estimator. It implements the XGBoost classification algorithm based on XGBoost python library, and it can be used in PySpark Pipeline and PySpark ML meta algorithms like - :py:class:`~pyspark.ml.tuning.CrossValidator`/ - :py:class:`~pyspark.ml.tuning.TrainValidationSplit`/ - :py:class:`~pyspark.ml.classification.OneVsRest` SparkXGBClassifier automatically supports most of the parameters in :py:class:`xgboost.XGBClassifier` constructor and most of the parameters used in :py:meth:`xgboost.XGBClassifier.fit` and :py:meth:`xgboost.XGBClassifier.predict` method. To enable GPU support, set `device` to `cuda` or `gpu`. SparkXGBClassifier doesn't support setting `base_margin` explicitly as well, but support another param called `base_margin_col`. see doc below for more details. SparkXGBClassifier doesn't support setting `output_margin`, but we can get output margin from the raw prediction column. See `raw_prediction_col` param doc below for more details. SparkXGBClassifier doesn't support `validate_features` and `output_margin` param. SparkXGBClassifier doesn't support setting `nthread` xgboost param, instead, the `nthread` param for each xgboost worker will be set equal to `spark.task.cpus` config value. Parameters ---------- features_col: When the value is string, it requires the features column name to be vector type. When the value is a list of string, it requires all the feature columns to be numeric types. label_col: Label column name. Default to "label". prediction_col: Prediction column name. Default to "prediction" probability_col: Column name for predicted class conditional probabilities. Default to probabilityCol raw_prediction_col: The `output_margin=True` is implicitly supported by the `rawPredictionCol` output column, which is always returned with the predicted margin values. pred_contrib_col: Contribution prediction column name. validation_indicator_col: For params related to `xgboost.XGBClassifier` training with evaluation dataset's supervision, set :py:attr:`xgboost.spark.SparkXGBClassifier.validation_indicator_col` parameter instead of setting the `eval_set` parameter in `xgboost.XGBClassifier` fit method. weight_col: To specify the weight of the training and validation dataset, set :py:attr:`xgboost.spark.SparkXGBClassifier.weight_col` parameter instead of setting `sample_weight` and `sample_weight_eval_set` parameter in `xgboost.XGBClassifier` fit method. base_margin_col: To specify the base margins of the training and validation dataset, set :py:attr:`xgboost.spark.SparkXGBClassifier.base_margin_col` parameter instead of setting `base_margin` and `base_margin_eval_set` in the `xgboost.XGBClassifier` fit method. num_workers: How many XGBoost workers to be used to train. Each XGBoost worker corresponds to one spark task. device: .. versionadded:: 2.0.0 Device for XGBoost workers, available options are `cpu`, `cuda`, and `gpu`. force_repartition: Boolean value to specify if forcing the input dataset to be repartitioned before XGBoost training. repartition_random_shuffle: Boolean value to specify if randomly shuffling the dataset when repartitioning is required. enable_sparse_data_optim: Boolean value to specify if enabling sparse data optimization, if True, Xgboost DMatrix object will be constructed from sparse matrix instead of dense matrix. launch_tracker_on_driver: Boolean value to indicate whether the tracker should be launched on the driver side or the executor side. coll_cfg: The collective configuration. See :py:class:`~xgboost.collective.Config` kwargs: A dictionary of xgboost parameters, please refer to https://xgboost.readthedocs.io/en/stable/parameter.html Note ---- The Parameters chart above contains parameters that need special handling. For a full list of parameters, see entries with `Param(parent=...` below. This API is experimental. Examples -------- >>> from xgboost.spark import SparkXGBClassifier >>> from pyspark.ml.linalg import Vectors >>> df_train = spark.createDataFrame([ ... (Vectors.dense(1.0, 2.0, 3.0), 0, False, 1.0), ... (Vectors.sparse(3, {1: 1.0, 2: 5.5}), 1, False, 2.0), ... (Vectors.dense(4.0, 5.0, 6.0), 0, True, 1.0), ... (Vectors.sparse(3, {1: 6.0, 2: 7.5}), 1, True, 2.0), ... ], ["features", "label", "isVal", "weight"]) >>> df_test = spark.createDataFrame([ ... (Vectors.dense(1.0, 2.0, 3.0), ), ... ], ["features"]) >>> xgb_classifier = SparkXGBClassifier(max_depth=5, missing=0.0, ... validation_indicator_col='isVal', weight_col='weight', ... early_stopping_rounds=1, eval_metric='logloss') >>> xgb_clf_model = xgb_classifier.fit(df_train) >>> xgb_clf_model.transform(df_test).show() rErFrG probability rawPredictionNrFT)rHrIrJprobability_colraw_prediction_colrKrLrMrNrOrPrQrRrSrTrUrHrIrJrrrKrLrMrNrOrPrQrRrSrTrUrVrc |>[TU]5 URnUR"S0UD6 UR SS9 g)N) objectiverY)r[r\r]r^ _setDefault)r_rHrIrJrrrKrLrMrNrOrPrQrRrSrTrUrVr`ras r)r\SparkXGBClassifier.__init__rs=,  ))  && 4(r2c[$rrres r)r9SparkXGBClassifier._xgb_clsr2SparkXGBClassifierModelc[$r)rres r)rk%SparkXGBClassifier._pyspark_model_clss&&r2c>[TU]5 URUR5(a [ S5eUR S5(a [ S5eg)NzDSpark Xgboost classifier estimator does not support `qid_col` param.rzHSetting custom 'objective' param is not allowed in 'SparkXGBClassifier'.)r[rorprqrr getOrDefaultrss r)ro#SparkXGBClassifier._validate_paramssY  " >>$,, ' 'V    [ ) )Z  *r2rYru)rvrwrxryrzrrr7rrr{r|rrr\r}rrr9rkror~rrs@r)rrsuwr/9 *,"1*.26$()- $"'+0).)-%)%)CcN+) )  )  ) )#3-)#+3-)SM)"#)) ) )%)) #'!)"#'#)$6"%)&')( )))<m,'4(A#B''  r2rc6\rSrSrSr\S\\4Sj5rSr g)rizi The model returned by :func:`xgboost.spark.SparkXGBClassifier.fit` .. Note:: This API is experimental. rc[$rrres r)r9 SparkXGBClassifierModel._xgb_clsrr2rYN) rvrwrxryrzr}rrr9r~rYr2r)rrs' m,r2rc$(^\rSrSrSr\SSSSSSSSSSSSSS SS .S \\\\4S \S \S\ \S\ \S\ \S\ \S\ \S\ S\ \S\ S\ S\ S\ S\ \ S\ SS4"U4Sjjj5r\S\\4Sj5r\S\S4Sj5rS"U4S jjrS!rU=r$)#SparkXGBRankeriaSparkXGBRanker is a PySpark ML estimator. It implements the XGBoost ranking algorithm based on XGBoost python library, and it can be used in PySpark Pipeline and PySpark ML meta algorithms like :py:class:`~pyspark.ml.tuning.CrossValidator`/ :py:class:`~pyspark.ml.tuning.TrainValidationSplit`/ :py:class:`~pyspark.ml.classification.OneVsRest` SparkXGBRanker automatically supports most of the parameters in :py:class:`xgboost.XGBRanker` constructor and most of the parameters used in :py:meth:`xgboost.XGBRanker.fit` and :py:meth:`xgboost.XGBRanker.predict` method. To enable GPU support, set `device` to `cuda` or `gpu`. SparkXGBRanker doesn't support setting `base_margin` explicitly as well, but support another param called `base_margin_col`. see doc below for more details. SparkXGBRanker doesn't support setting `output_margin`, but we can get output margin from the raw prediction column. See `raw_prediction_col` param doc below for more details. SparkXGBRanker doesn't support `validate_features` and `output_margin` param. SparkXGBRanker doesn't support setting `nthread` xgboost param, instead, the `nthread` param for each xgboost worker will be set equal to `spark.task.cpus` config value. Parameters ---------- features_col: When the value is string, it requires the features column name to be vector type. When the value is a list of string, it requires all the feature columns to be numeric types. label_col: Label column name. Default to "label". prediction_col: Prediction column name. Default to "prediction" pred_contrib_col: Contribution prediction column name. validation_indicator_col: For params related to `xgboost.XGBRanker` training with evaluation dataset's supervision, set :py:attr:`xgboost.spark.SparkXGBRanker.validation_indicator_col` parameter instead of setting the `eval_set` parameter in :py:class:`xgboost.XGBRanker` fit method. weight_col: To specify the weight of the training and validation dataset, set :py:attr:`xgboost.spark.SparkXGBRanker.weight_col` parameter instead of setting `sample_weight` and `sample_weight_eval_set` parameter in :py:class:`xgboost.XGBRanker` fit method. base_margin_col: To specify the base margins of the training and validation dataset, set :py:attr:`xgboost.spark.SparkXGBRanker.base_margin_col` parameter instead of setting `base_margin` and `base_margin_eval_set` in the :py:class:`xgboost.XGBRanker` fit method. qid_col: Query id column name. num_workers: How many XGBoost workers to be used to train. Each XGBoost worker corresponds to one spark task. device: .. versionadded:: 2.0.0 Device for XGBoost workers, available options are `cpu`, `cuda`, and `gpu`. force_repartition: Boolean value to specify if forcing the input dataset to be repartitioned before XGBoost training. repartition_random_shuffle: Boolean value to specify if randomly shuffling the dataset when repartitioning is required. enable_sparse_data_optim: Boolean value to specify if enabling sparse data optimization, if True, Xgboost DMatrix object will be constructed from sparse matrix instead of dense matrix. launch_tracker_on_driver: Boolean value to indicate whether the tracker should be launched on the driver side or the executor side. coll_cfg: The collective configuration. See :py:class:`~xgboost.collective.Config` kwargs: A dictionary of xgboost parameters, please refer to https://xgboost.readthedocs.io/en/stable/parameter.html .. Note:: The Parameters chart above contains parameters that need special handling. For a full list of parameters, see entries with `Param(parent=...` below. .. Note:: This API is experimental. Examples -------- >>> from xgboost.spark import SparkXGBRanker >>> from pyspark.ml.linalg import Vectors >>> ranker = SparkXGBRanker(qid_col="qid") >>> df_train = spark.createDataFrame( ... [ ... (Vectors.dense(1.0, 2.0, 3.0), 0, 0), ... (Vectors.dense(4.0, 5.0, 6.0), 1, 0), ... (Vectors.dense(9.0, 4.0, 8.0), 2, 0), ... (Vectors.sparse(3, {1: 1.0, 2: 5.5}), 0, 1), ... (Vectors.sparse(3, {1: 6.0, 2: 7.5}), 1, 1), ... (Vectors.sparse(3, {1: 8.0, 2: 9.5}), 2, 1), ... ], ... ["features", "label", "qid"], ... ) >>> df_test = spark.createDataFrame( ... [ ... (Vectors.dense(1.5, 2.0, 3.0), 0), ... (Vectors.dense(4.5, 5.0, 6.0), 0), ... (Vectors.dense(9.0, 4.5, 8.0), 0), ... (Vectors.sparse(3, {1: 1.0, 2: 6.0}), 1), ... (Vectors.sparse(3, {1: 6.0, 2: 7.0}), 1), ... (Vectors.sparse(3, {1: 8.0, 2: 10.5}), 1), ... ], ... ["features", "qid"], ... ) >>> model = ranker.fit(df_train) >>> model.transform(df_test).show() rErFrGNrFT)rHrIrJrKrLrMrNrqrOrPrQrRrSrTrUrHrIrJrKrLrMrNrqrOrPrQrRrSrTrUrVrc ^>[TU]5 URnUR"S0UD6 grXrZ)r_rHrIrJrKrLrMrNrqrOrPrQrRrSrTrUrVr`ras r)r\SparkXGBRanker.__init__/s** ))  &&r2c[$rrres r)r9SparkXGBRanker._xgb_clsHr2SparkXGBRankerModelc[$r)rres r)rk!SparkXGBRanker._pyspark_model_clsLs""r2cx>[TU]5 URUR5(d [ S5eg)Nz@Spark Xgboost ranker estimator requires setting `qid_col` param.rnrss r)roSparkXGBRanker._validate_paramsPs6  "~~dll++R ,r2rYru)rvrwrxryrzrrr7rrr{r|rrr\r}rrr9rkror~rrs@r)rrskxt/9 **.26$()-!% $"'+0).)-%)#'CcN+' '  ' #3- '#+3-'SM'"#'#'' ' '%)'#'' #'!'"6"#'$%'& '''0i#4(=#>##r2rc6\rSrSrSr\S\\4Sj5rSr g)riXze The model returned by :func:`xgboost.spark.SparkXGBRanker.fit` .. Note:: This API is experimental. rc[$rrres r)r9SparkXGBRankerModel._xgb_cls_rr2rYN) rvrwrxryrzr}rrr9r~rYr2r)rrXs& ir2r)$rztypingrrrrrnumpyrpysparkrpyspark.ml.paramr r pyspark.ml.param.sharedr r collectiversklearnrrrcorerrrutilsrrArCrirrrrrYr2r)rs> 43 *J<< "5)&'5)04^0D5) 5)pW*Wt ^ !!24JKj+->@SjZ 2 !!35LMa'aH . !1DEr2