Decision Tree Classification Algorithm in Machine Learning

Decision Tree Classification Algorithm

In machine learning, Decision Tree Classification is one of the most popular and easy-to-understand algorithms used for making predictions from data. It provides a simple way to represent decisions and their possible outcomes in a tree-like structure.

Decision Trees are especially useful for classification problems because their decision-making process is easy to follow. Instead of working like a black box, a decision tree shows how different features are used step by step to arrive at a final prediction.

Complete Advance AI Topics: Click Here
SQL Tutorial: Click Here

What is a Decision Tree?

A Decision Tree is a Supervised Learning algorithm that can be used for both classification and regression tasks. However, it is widely used for classification problems.

The algorithm works in a way that is similar to human decision-making. It uses a tree-like structure consisting of decision nodes, branches, and leaf nodes to make predictions.

• Decision Nodes: Represent features or attributes used to make decisions.
• Branches: Represent the decision rules or possible outcomes of a decision.
• Leaf Nodes: Represent the final predicted class or result.

For example, imagine making a decision by answering a series of yes-or-no questions. Each answer takes you to another question until you finally reach an outcome. This is essentially how a Decision Tree works.

Structure of a Decision Tree

A Decision Tree begins with a root node, which represents the complete dataset. The data is then divided into smaller groups according to selected conditions. These divisions continue until the algorithm reaches suitable leaf nodes.

• Root Node: The topmost node of the tree where the first decision is made.
• Splitting: The process of dividing a node into smaller groups based on a feature or condition.
• Sub Tree: A smaller tree that forms part of the main decision tree.
• Pruning: The process of removing unnecessary branches to reduce overfitting.
• Parent/Child Node: A parent node is connected to one or more child nodes in the tree hierarchy.

Why Use Decision Trees?

Decision Trees have several characteristics that make them useful for machine learning and data analysis.

• Easy to understand: The decision-making process is straightforward and resembles human reasoning.
• Visual representation: The tree structure makes predictions and decisions easier to interpret.
• No feature scaling required: Decision Trees generally do not require normalization or standardization of features.
• Handles numerical and categorical data: Decision Trees can work with different types of input features.

How Does the Decision Tree Algorithm Work?

A Decision Tree classifies a record by starting from the root node and following the branches according to the values of its features.

1. Start at the root node.
2. Compare the relevant attribute with the decision condition.
3. Follow the branch corresponding to the result.
4. Continue this process until a leaf node is reached.
5. The class represented by the leaf node becomes the prediction.

Steps Involved in Building a Decision Tree

1. Start with the complete dataset S.
2. Use an Attribute Selection Measure (ASM) to identify the most suitable attribute for splitting.
3. Split S according to the selected attribute.
4. Create a decision tree node for each split.
5. Continue splitting the resulting subsets until a suitable leaf node is reached and no further useful splitting is possible.

Attribute Selection Measures (ASM)

Selecting the right feature for each split is an important part of building an effective Decision Tree. Two commonly used Attribute Selection Measures are Information Gain and Gini Index.

1. Information Gain

Information Gain measures how much the entropy of a dataset changes after it is split using a particular attribute.

A feature with a higher information gain is generally considered a better choice for splitting the dataset.

Formula:

Information Gain = Entropy(S) - Σ [(Weighted Avg) × Entropy(subset)]

Entropy:

Entropy(S) = -P(yes) log2 P(yes) - P(no) log2 P(no)

2. Gini Index

The Gini Index measures the impurity of a dataset. It indicates how mixed the classes are within a particular node.

A lower Gini Index indicates a better split because it represents a more pure group of data.

Formula:

Gini Index = 1 - Σ (Pj)2

Machine Learning Tutorial:-

Pruning: Creating an Optimal Tree

If a Decision Tree continues growing without proper control, it can become very large and may start learning noise from the training data. This can lead to overfitting.

On the other hand, a tree that is too small may not capture enough information from the dataset and can result in underfitting.

Pruning helps maintain a suitable balance by removing unnecessary branches from the tree.

• Cost Complexity Pruning
• Reduced Error Pruning

Advantages of Decision Tree

• Easy to understand and interpret.
• Works with both numerical and categorical data.
• Requires minimal data preprocessing.
• Useful for exploratory data analysis.

Disadvantages of Decision Tree

• Can be prone to overfitting, particularly when working with noisy datasets.
• Performance may decrease when there are many class labels.
• Small changes in the training data can sometimes result in a significantly different tree.
• Can produce biased trees when some classes are much more dominant than others.

Python Implementation of Decision Tree Classifier

Let’s implement a Decision Tree Classifier using the user_data.csv dataset. In this example, Age and Estimated Salary are used as features, while Purchased is the target variable.

Step 1: Data Preprocessing

import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

# Load dataset
dataset = pd.read_csv('user_data.csv')

X = dataset.iloc[:, [2, 3]].values  # Features: Age and EstimatedSalary
y = dataset.iloc[:, 4].values       # Target: Purchased

# Split data
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.25, random_state=0
)

# Feature scaling
from sklearn.preprocessing import StandardScaler

sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

Step 2: Fitting the Decision Tree Classifier

from sklearn.tree import DecisionTreeClassifier

classifier = DecisionTreeClassifier(
    criterion='entropy',
    random_state=0
)

classifier.fit(X_train, y_train)

Step 3: Predicting the Test Set Results

After training the classifier, we can use it to predict the outcomes for the test dataset.

y_pred = classifier.predict(X_test)

Step 4: Evaluating Using Confusion Matrix

A confusion matrix can be used to evaluate the classification results by comparing the actual values with the predicted values.

from sklearn.metrics import confusion_matrix

cm = confusion_matrix(y_test, y_pred)
print(cm)

Step 5: Visualizing the Training Set Result

The following code visualizes the classification regions created by the Decision Tree for the training dataset.

from matplotlib.colors import ListedColormap

X_set, y_set = X_train, y_train

X1, X2 = np.meshgrid(
    np.arange(
        start=X_set[:, 0].min() - 1,
        stop=X_set[:, 0].max() + 1,
        step=0.01
    ),
    np.arange(
        start=X_set[:, 1].min() - 1,
        stop=X_set[:, 1].max() + 1,
        step=0.01
    )
)

plt.contourf(
    X1,
    X2,
    classifier.predict(
        np.array([X1.ravel(), X2.ravel()]).T
    ).reshape(X1.shape),
    alpha=0.75,
    cmap=ListedColormap(('red', 'green'))
)

plt.xlim(X1.min(), X1.max())
plt.ylim(X2.min(), X2.max())

# Plot the points
for i, j in enumerate(np.unique(y_set)):
    plt.scatter(
        X_set[y_set == j, 0],
        X_set[y_set == j, 1],
        c=ListedColormap(('red', 'green'))(i),
        label=j
    )

plt.title('Decision Tree Classifier (Training set)')
plt.xlabel('Age')
plt.ylabel('Estimated Salary')
plt.legend()
plt.show()

Conclusion

Decision Tree Classification is one of the most intuitive and widely used classification techniques in machine learning. Its tree-based structure makes the decision-making process easy to understand and visualize.

Decision Trees can work with numerical as well as categorical data and generally require minimal preprocessing. However, proper control of tree growth and pruning is important to reduce the risk of overfitting.

Download New Real Time Projects :- Click here

---

Keywords: Decision Tree Classification Algorithm, decision tree classification algorithm in machine learning, decision tree classification algorithm in data mining, decision tree examples with solutions, decision tree classification algorithm example, decision tree regression, decision tree classification algorithm Python Decision Tree Classification Algorithm in Machine Learning Decision Tree Classification Algorithm in Machine Learning