🔍 What is Backward Elimination in Machine Learning?

Backward Elimination in Machine Learning

In the world of Machine Learning, building a model is not just about feeding data into an algorithm and getting results. A significant part of model building involves feature selection, where we identify the most relevant variables that impact the model’s performance.

One such efficient and widely-used technique is Backward Elimination. This method helps in refining models by removing less significant features, making them simpler, faster, and more accurate.

In this blog by Updategadh, we’ll explore what Backward Elimination is, why it’s important, how to implement it step-by-step, and how it optimizes a Multiple Linear Regression (MLR) model.

Complete Python Course with Advance topics:-Click Here
SQL Tutorial :-Click Here

✨ Why Feature Selection Matters?

Machine Learning models perform better when they use only the most influential features. Including irrelevant or weak predictors can:

• Add noise to the model
• Increase computational complexity
• Lead to overfitting
• Make the model hard to interpret

Thus, feature elimination techniques like Backward Elimination become essential tools for data scientists and engineers.

✅ Backward Elimination: The Concept

Backward elimination is a feature selection strategy that uses statistical tests to remove the least significant variables from a model.. It starts with all features and eliminates the ones that don’t have a meaningful impact on the output.

Other Feature Selection Methods:

• All-in
• Forward Selection
• Backward Elimination ✅
• Bidirectional Elimination
• Score Comparison

Among these, Backward Elimination is often preferred because it is fast, reliable, and data-driven.

🪜 Steps to Apply Backward Elimination

Let’s walk through the step-by-step procedure to implement backward elimination.

Step 1: Choose a significance level (SL)

Typically, SL = 0.05. This means any feature with a p-value > 0.05 is considered statistically insignificant.

Step 2: Fit the model with all independent variables.

Step 3: Check the p-values of each variable.

• If the highest p-value > SL, remove that variable.
• Otherwise, stop! The model is optimized.

Step 4: Repeat steps 2 and 3 until all variables in the model have p-values less than SL.

💡 Let’s Understand with an Example

Imagine you are working with a dataset of 50 companies. You want to predict Profit based on:

• R&D Spend
• Administration Spend
• Marketing Spend
• State (Dummy Variables)

We’ll first build a Multiple Linear Regression (MLR) model with all features and then apply Backward Elimination to optimize it.

📌 Step 1: Build the Full MLR Model

import numpy as np  
import matplotlib.pyplot as plt  
import pandas as pd  

# Load dataset
dataset = pd.read_csv('50_CompList.csv')  
X = dataset.iloc[:, :-1].values  
y = dataset.iloc[:, 4].values  

# Encode Categorical Data (State)
from sklearn.preprocessing import LabelEncoder, OneHotEncoder  
labelencoder = LabelEncoder()  
X[:, 3] = labelencoder.fit_transform(X[:, 3])  
from sklearn.compose import ColumnTransformer  
ct = ColumnTransformer([("State", OneHotEncoder(), [3])], remainder='passthrough')  
X = ct.fit_transform(X)

# Avoid dummy variable trap
X = X[:, 1:]

# Split dataset
from sklearn.model_selection import train_test_split  
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)  

# Train model
from sklearn.linear_model import LinearRegression  
regressor = LinearRegression()  
regressor.fit(X_train, y_train)  

# Predict and score
y_pred = regressor.predict(X_test)  
print("Train Score:", regressor.score(X_train, y_train))  
print("Test Score:", regressor.score(X_test, y_test))  

Output:

Train Score: 0.95018
Test Score: 0.93470

🧮 Step 2: Apply Backward Elimination

Add a constant column:

import statsmodels.api as sm  
X = np.append(arr = np.ones((50,1)).astype(int), values = X, axis = 1)

Start Elimination:

X_opt = X[:, [0,1,2,3,4,5]]  
regressor_OLS = sm.OLS(endog = y, exog = X_opt).fit()  
print(regressor_OLS.summary())

Iteratively Remove Variables with High p-value:

# Remove variable with p > 0.05 and repeat
X_opt = X[:, [0,2,3,4,5]]  
regressor_OLS = sm.OLS(endog = y, exog = X_opt).fit()

X_opt = X[:, [0,3,4,5]]  
regressor_OLS = sm.OLS(endog = y, exog = X_opt).fit()

X_opt = X[:, [0,3,5]]  
regressor_OLS = sm.OLS(endog = y, exog = X_opt).fit()

X_opt = X[:, [0,3]]  
regressor_OLS = sm.OLS(endog = y, exog = X_opt).fit()

After running these, you’ll find that R&D Spend is the only statistically significant variable left.

🎯 Final Optimized Model

Let’s now use only R&D Spend for our final model:

# Load optimized dataset
dataset = pd.read_csv('50_CompList1.csv')  
X_BE = dataset.iloc[:, :-1].values  
y_BE = dataset.iloc[:, 1].values  

# Split dataset
from sklearn.model_selection import train_test_split  
X_BE_train, X_BE_test, y_BE_train, y_BE_test = train_test_split(X_BE, y_BE, test_size=0.2, random_state=0)  

# Train optimized model
from sklearn.linear_model import LinearRegression  
regressor = LinearRegression()  
regressor.fit(np.array(X_BE_train).reshape(-1,1), y_BE_train)  

# Predict and score
y_pred = regressor.predict(X_BE_test)  
print("Train Score:", regressor.score(X_BE_train, y_BE_train))  
print("Test Score:", regressor.score(X_BE_test, y_BE_test))  

Output:

Train Score: 0.94495
Test Score: 0.94645

🎉 Result: Our simplified model using only R&D Spend is almost as accurate as the original complex model using four features. The difference in score is minimal, and the model is now cleaner and more efficient.

Download New Real Time Projects :-Click here
Complete Advance AI topics:- CLICK HERE

📌 Conclusion

Backward Elimination helps in building optimized, high-performing models by removing less useful features. It’s especially useful in regression models where interpretability and performance go hand-in-hand.

By applying this technique, we realized that R&D Spend alone could predict the profit of a company quite accurately — making the model simpler without compromising its predictive power.

✅ Tip from Updategadh:
Always perform feature analysis before finalizing your model. More features don’t always mean better results — sometimes, less is more!

If you found this blog helpful, share it with your fellow data enthusiasts. For more insightful tutorials and guides, keep visiting Updategadh — your trusted tech companion. 🚀

Written by Updategadh Team | Professional Guides on Data Science & Machine Learning

backward elimination in machine learning with example
backward elimination python
backward elimination in machine learning python
backward elimination method
backward elimination in machine learning geeksforgeeks
backward elimination algorithm
backward elimination vs forward selection
backward elimination regression