 # XGBoost for Regression

## XGBoost for Regression

The results of the regression problems are continuous or real values. Some commonly used regression algorithms are Linear Regression and Decision Trees. There are several metrics involved in regression like root-mean-squared error (RMSE) and mean-squared-error (MAE). These are some key members of XGBoost models, each plays an important role.

• RMSE: It is the square root of mean squared error (MSE).
• MAE: It is an absolute sum of actual and predicted differences, but it lacks mathematically, that’s why it is rarely used, as compared to other metrics.

XGBoost is a powerful approach for building supervised regression models. The validity of this statement can be inferred by knowing about its (XGBoost) objective function and base learners.

The objective function contains loss function and a regularization term. It tells about the difference between actual values and predicted values, i.e how far the model results are from the real values. The most common loss functions in XGBoost for regression problems is `reg:linear`, and that for binary classification is `reg:logistics`.

Ensemble learning involves training and combining individual models (known as base learners) to get a single prediction, and XGBoost is one of the ensemble learning methods. XGBoost expects to have the base learners which are uniformly bad at the remainder so that when all the predictions are combined, bad predictions cancels out and better one sums up to form final good predictions.

Code:

 `# Necessary imports` `import` `numpy as np` `import` `pandas as pd` `import` `xgboost as xg` `from` `sklearn.model_selection ``import` `train_test_split` `from` `sklearn.metrics ``import` `mean_squared_error as MSE` ` ` `# Load the data` `dataset ``=` `pd.read_csv(``"boston_house.csv"``)` `X, y ``=` `dataset.iloc[:, :``-``1``], dataset.iloc[:, ``-``1``]` ` ` `# Splitting` `train_X, test_X, train_y, test_y ``=` `train_test_split(X, y,` `                      ``test_size ``=` `0.3``, random_state ``=` `123``)` ` ` `# Instantiation` `xgb_r ``=` `xg.XGBRegressor(objective ``=``'reg:linear'``,` `                  ``n_estimators ``=` `10``, seed ``=` `123``)` ` ` `# Fitting the model` `xgb_r.fit(train_X, train_y)` ` ` `# Predict the model` `pred ``=` `xgb_r.predict(test_X)` ` ` `# RMSE Computation` `rmse ``=` `np.sqrt(MSE(test_y, pred))` `print``(``"RMSE : % f"` `%``(rmse))`

Output:

```129043.2314
```

Code: Linear base learner

 `# Necessary imports` `import` `numpy as np` `import` `pandas as pd` `import` `xgboost as xg` `from` `sklearn.model_selection ``import` `train_test_split` `from` `sklearn.metrics ``import` `mean_squared_error as MSE` ` ` `# Load the data` `dataset ``=` `pd.read_csv(``"boston_house.csv"``)` `X, y ``=` `dataset.iloc[:, :``-``1``], dataset.iloc[:, ``-``1``]` ` ` `# Splitting` `train_X, test_X, train_y, test_y ``=` `train_test_split(X, y, ` `                      ``test_size ``=` `0.3``, random_state ``=` `123``)` ` ` `# Train and test set are converted to DMatrix objects,` `# as it is required by learning API.` `train_dmatrix ``=` `xg.DMatrix(data ``=` `train_X, label ``=` `train_y)` `test_dmatrix ``=` `xg.DMatrix(data ``=` `test_X, label ``=` `test_y)` ` ` `# Parameter dictionary specifying base learner` `param ``=` `{``"booster"``:``"gblinear"``, ``"objective"``:``"reg:linear"``}` ` ` `xgb_r ``=` `xg.train(params ``=` `param, dtrain ``=` `train_dmatrix, num_boost_round ``=` `10``)` `pred ``=` `xgb_r.predict(test_dmatrix)` ` ` `# RMSE Computation` `rmse ``=` `np.sqrt(MSE(test_y, pred))` `print``(``"RMSE : % f"` `%``(rmse))`

Output:

``` 124326.24465
```

Note: The dataset needs to be converted into DMatrix. It is an optimized data structure that the creators of XGBoost made. It gives the package its performance and efficiency gains.

The loss function is also responsible for analyzing the complexity of the model, and if the model becomes more complex there becomes a need to penalize it and this can be done using Regularization. It penalizes more complex models through both LASSO (L1) and Ridge (L2) regularization to prevent overfitting. The ultimate goal is to find simple and accurate models.

Regularization parameters are as follows:

• gamma: minimum reduction of loss allowed for a split to occur. Higher the gamma, fewer the splits.
alpha: L1 regularization on leaf weights, larger the value, more will be the regularization, which causes many leaf weights in the base learner to go to 0.
• lamba: L2 regularization on leaf weights, this is smoother than L1 nd causes leaf weights to smoothly decrease, unlike L1, which enforces strong constraints on leaf weights.

Below are the formulas which help in building the XGBoost tree for Regression.

Step 1: Calculate the similarity scores, it helps in growing the tree.

```Similarity Score = (Sum of residuals)^2 / Number of residuals + lambda
```

Step 2: Calculate the gain to determine how to split the data.

```Gain = Left tree (similarity score) + Right (similarity score) - Root (similarity score)
```

Step 3: Prune the tree by calculating the difference between Gain and gamma (user-defined tree-complexity parameter)

```Gain - gamma
```

If the result is a positive number then do not prune and if the result is negative, then prune and again subtract gamma from the next Gain value way up the tree.

Step 4: Calculate output value for the remaining leaves

```Output value = Sum of residuals / Number of residuals + lambda
```

Note: If the value of lambda is greater than 0, it results in more pruning by shrinking the similarity scores and it results in smaller output values for the leaves.

Let’s see a part of mathematics involved in finding the suitable output value to minimize the loss function

For classification and regression, XGBoost starts with an initial prediction usually 0.5, as shown in the below diagram. To find how good the prediction is, calculate the Loss function, by using the formula, For the given example, it came out to be 196.5. Later, we can apply this loss function and compare the results, and check if predictions are improving or not.

XGBoost uses those loss function to build trees by minimizing the below equation: The first part of the equation is the loss function and the second part of the equation is the regularization term and the ultimate goal is to minimize the whole equation.

For optimizing output value for the first tree, we write the equation as follows, replace p(i) with the initial predictions and output value and let lambda = 0 for simpler calculations. Now the equation looks like, The loss function for initial prediction was calculated before, which came out to be 196.5.

So, for output value = 0loss function = 196.5. Similarly, if we plot the point for output value = -1, loss function = 203.5 and for output value = +1, loss function = 193.5, and so on for other output values and, if we plot this in the graph. we get a parabola like structure. This is the plot for the equation as a function of output values. If lambda = 0, the optimal output value is at the bottom of the parabola where the derivative is zero. XGBoost uses Second-Order Taylor Approximation for both classification and regression. The loss function containing output values can be approximated as follows: The first part is Loss Function, the second part includes the first derivative of the loss function and the third part includes the second derivative of the loss function. The first derivative is related o Gradient Descent, so here XGBoost uses ‘g’ to represent the first derivative and the second derivative is related to Hessian, so it is represented by ‘h’ in XGBoost. Plugging the same in the equation: Remove the terms that do not contain the output value term, now minimize the remaining function by following steps:

• Take the derivative w.r.t output value.
• Set derivative equals 0 (solving for the lowest point in parabola)
• Solve for the output value.
• g(i) = negative residuals
• h(i) = number of residuals This is the output value formula for XGBoost in Regression. It gives the x-axis coordinate for the lowest point in the parabola.

Last Updated on March 1, 2022 by admin

## iconphoto() method in Tkinter | Pythoniconphoto() method in Tkinter | Python

iconphoto() method in Tkinter | Python iconphoto() method is used to set the titlebar icon of any tkinter/toplevel window. But to set any image as the icon of titlebar, image should be the object of PhotoImage class. Syntax:  iconphoto(self, default = False, *args)

## Python Dictionary ComprehensionPython Dictionary Comprehension

Python Dictionary Comprehension Like List Comprehension, Python allows dictionary comprehensions. We can create dictionaries using simple expressions. A dictionary comprehension takes the form {key: value for (key, value) in iterable} Let’s see a example,lets assume we have two lists named keys and

## How to Filter DataFrame Rows Based on the Date in Pandas?How to Filter DataFrame Rows Based on the Date in Pandas?

How to Filter DataFrame Rows Based on the Date in Pandas? Different regions follow different date conventions (YYYY-MM-DD, YYYY-DD-MM, DD/MM/YY, etc.).  It is difficult to work with such strings in the data. Pandas to_datetime() function allows converting the date and time in string

## Pooling Layer for Convolutional Neural Networks (CNN)Pooling Layer for Convolutional Neural Networks (CNN)

CNN | Introduction to Pooling Layer The pooling operation involves sliding a two-dimensional filter over each channel of feature map and summarising the features lying within the region covered by the filter. For a feature map having dimensions nh x nw x nc,

## How to compare two NumPy arrays?How to compare two NumPy arrays?

How to compare two NumPy arrays? This article focuses on the comparison done using NumPy on arrays. Comparing two NumPy arrays determines whether they are equivalent by checking if every element at each corresponding index are the same. Method 1:

## Function Annotations in PythonFunction Annotations in Python

Function Annotations in Python PEP: PEP stands for Python Enhancement Proposal. It is a design document that describes new features for Python or its processes or environment. It also provides information to the python community. PEP is a primary mechanism for

## Beautifulsoup Installation – PythonBeautifulsoup Installation – Python

Beautifulsoup Installation – Python Beautiful Soup is a Python library for pulling data out of HTML and XML files. It works with your favorite parser to provide idiomatic ways of navigating, searching, and modifying the parse tree. It commonly saves

## Python | Linear Regression using sklearnPython | Linear Regression using sklearn

Python | Linear Regression using sklearn Linear Regression is a machine learning algorithm based on supervised learning. It performs a regression task. Regression models a target prediction value based on independent variables. It is mostly used for finding out the