DATABRICKS-MACHINE-LEARNING-ASSOCIATE Online Practice Questions and Answers

Correct Answer: C

TheOneHotEncoderin Spark ML requires numerical indices as inputs rather than string labels. Therefore, you need to first convert the string columns to numerical indices usingStringIndexer. After that, you can applyOneHotEncoderto these

indices.

Corrected code:

frompyspark.ml.featureimportStringIndexer, OneHotEncoder# Convert string column to indexindexers = [StringIndexer(inputCol=col,

outputCol=col+"_index")forcolininput_columns] indexer_model = Pipeline(stages=indexers).fit(features_df) indexed_features_df = indexer_model.transform(features_df)# One-hot encode the indexed columnsohe = OneHotEncoder

(inputCols=[col+"_index"forcolininput_columns], outputCols=output_columns) ohe_model = ohe.fit(indexed_features_df) ohe_features_df = ohe_model.transform(indexed_features_df)

References:

PySpark ML Documentation

Correct Answer: D

In the context of Spark MLlib, an estimator refers to an algorithm which can be "fit" on a DataFrame to produce a model (referred to as a Transformer), which can then be used to transform one DataFrame into another, typically adding

predictions or model scores. This is a fundamental concept in machine learning pipelines in Spark, where the workflow includes fitting estimators to data to produce transformers.

References:

Spark MLlib Documentation:https://spark.apache.org/docs/latest/ml-pipeline.html#estimators

Correct Answer: A

To calculate the overall cross-validation root-mean-squared error (RMSE), you average the RMSE values obtained from each validation fold. Given the RMSE values of 10.0, 12.0, and 17.0 for the three folds, the overall cross-validation

RMSE is calculated as the average of these three values:

Overall CV RMSE=10.0+12.0+17.03=39.03=13.0Overall CV RMSE=310.0+12.0+17.0 =339.0=13.0

Thus, the correct answer is 13.0, which accurately represents the average RMSE across all folds.References:

Cross-validation in Regression (Understanding Cross-Validation Metrics).

Correct Answer: D

To determine the number of machine learning models that can be trained in parallel, we need to calculate the total number of combinations of hyperparameters. The given hyperparameter grid includes:

Hyperparameter 1: [2, 5, 10] (3 values)

Hyperparameter 2: [50, 100] (2 values)

The total number of combinations is the product of the number of values for each hyperparameter:3 (values of Hyperparameter 1)? (values of Hyperparameter 2)=63 (value s of Hyperparameter 1)? (values of Hyperparameter 2)=6 With 3-fold

cross-validation, each combination of hyperparameters will be evaluated 3 times. Thus, the total number of models trained will

be:6 (combinations)? (folds)=186 (combinations)? (folds)=18 However, the number of models that can be trained in parallel is equal to the number of hyperparameter combinations, not the total number of models considering cross-validation.

Therefore, 6 models can be trained in parallel.

References:

Databricks documentation on hyperparameter tuning: Hyperparameter Tuning

Correct Answer: D

In the provided code block, theImputerobject is created but not fitted on the data to generate anImputerModel. Thetransformmethod is being called directly on the Imputerobject, which does not yet contain the fitted median values needed for imputation. The correct approach is to fit the imputer on the dataset first. Corrected code: imputer = Imputer( strategy="median", inputCols=input_columns, outputCols=output_columns ) imputer_model = imputer.fit(features_df)# Fit the imputer to the dataimputed_features_df = imputer_model.transform(features_df)# Transform the data using the fitted imputer References: PySpark ML Documentation

Correct Answer: B

To filter rows in a Spark DataFrame based on a condition, you use thefilter method along with a column condition. The correct syntax in PySpark to accomplish this task isspark_df.filter(col("price") > 0), which filters the DataFrame to include

only those rows where the value in the "price" column is greater than 0. Thecolfunction is used to specify column-based operations. The other options provided either do not use correct Spark DataFrame syntax or are intended for different

types of data manipulation frameworks like pandas.References:

PySpark DataFrame API documentation (Filtering DataFrames).

Correct Answer: E

To display visual histograms and summaries of the numeric features in a Spark DataFrame, the Databricks utility functiondbutils.data.summarizecan be used. This function provides a comprehensive summary, including visual histograms.

Correct code:

dbutils.data.summarize(spark_df)

Other options likespark_df.describe()andspark_df.summary()provide textual statistical summaries but do not include visual histograms.

References:

Databricks Utilities Documentation

Correct Answer: A

The suggestion not to one-hot encode categorical feature variables within the feature repository is justified because one-hot encoding can be problematic for some machine learning algorithms. Specifically, one-hot encoding increases the dimensionality of the data, which can be computationally expensive and may lead to issues such as multicollinearity and overfitting. Additionally, some algorithms, such as tree-based methods, can handle categorical variables directly without requiring one-hot encoding. References: Databricks documentation on feature engineering: Feature Engineering

Correct Answer: C

For large datasets, Spark ML uses iterative optimization methods to distribute the training of a linear regression model. Specifically, Spark MLlib employs techniques like Stochastic Gradient Descent (SGD) and Limited-memory Broyden

Correct Answer: D

If the new solution requires that each of the three models computes a prediction for every record, the time efficiency during inference will be reduced. This is because the inference process now involves running multiple models instead of a single model, thereby increasing the overall computation time for each record. In scenarios where inference must be done by multiple models for each record, the latency accumulates, making the process less time efficient compared to using a single model. References: Model Ensemble Techniques

Correct Answer: C

Random Forest is a machine learning algorithm that typically uses bagging (Bootstrap Aggregating). Bagging involves training multiple models independently on different random subsets of the data and then combining their predictions.

Random Forests consist of many decision trees trained on random subsets of the training data and features, and their predictions are averaged to improve accuracy and control overfitting. This method enhances model robustness and

predictive performance.References:

Ensemble Methods in Machine Learning (Understanding Bagging and Random Forests).

Correct Answer: A

To use the pandas API on Spark, the data scientist can run the following code block:

importpyspark.pandasasps df = ps.DataFrame(spark_df) This code imports the pandas API on Spark and converts the Spark DataFramespark_df into a pandas-on-Spark DataFrame, allowing the data scientist to use familiar pandas functions

for further feature engineering.

References:

Databricks documentation on pandas API on Spark: pandas API on Spark

Correct Answer: C

To compute the root mean-squared-error (RMSE) of a linear regression model using Spark ML, theRegressionEvaluatorclass is used. TheRegressionEvaluator is specifically designed for regression tasks and can calculate various metrics,

including RMSE, based on the columns containing predictions and actual values. The correct code block to compute RMSE from thepreds_dfDataFrame is:

regression_evaluator = RegressionEvaluator( predictionCol="prediction", labelCol="actual", metricName="rmse") rmse = regression_evaluator.evaluate(preds_df) This code creates an instance ofRegressionEvaluator, specifying the prediction

and label columns, as well as the metric to be computed ("rmse"). It then evaluates the predictions in preds_dfand assigns the resulting RMSE value to thermsevariable. Options A and B incorrectly useBinaryClassificationEvaluator, which is

not suitable for regression tasks. Option D also incorrectly usesBinaryClassificationEvaluator.

References:

PySpark ML Documentation

Correct Answer: A

The code block provided by the machine learning engineer will perform the desired inference when the Feature Store feature set was logged with the model at model_uri. This ensures that all necessary feature transformations and metadata

are available for the model to make predictions. The Feature Store in Databricks allows for seamless integration of features and models, ensuring that the required features are correctly used during inference.

References:

Databricks documentation on Feature Store: Feature Store in Databricks

Correct Answer: C

The F1 score is the harmonic mean of precision and recall and is particularly useful in situations where there is a significant imbalance between positive and negative classes. When there is a class imbalance, accuracy can be misleading

because a model can achieve high accuracy by simply predicting the majority class. The F1 score, however, provides a better measure of the test's accuracy in terms of both false positives and false negatives.

Specifically, the F1 score should be used over accuracy when:

There is a significant imbalance between positive and negative classes. Avoiding false negatives is a priority, meaning recall (the ability to detect all positive instances) is crucial.

In this scenario, the F1 score balances both precision (the ability to avoid false positives) and recall, providing a more meaningful measure of a model's performance under these conditions.

References:

Databricks documentation on classification metrics: Classification Metrics