Example: Compute the Impurity using Entropy and Gini Index.

Tip: This article is the continuation of Tree Models. Therefore, I recommend that you read this carefully.

The example that we will see next is taken from the book: Machine Learning: “The Art and Science of Algorithms that make Sense of Data”, Flach Peter.

Suppose you come across a number of sea animals that you suspect belong to the same species. You observe their length in metres, whether they have gills, whether they have a prominent beak and whether they have few or many teeth. After, you have classified in two groups; the group of the positives and the group of the negatives:

Five positive examples:

• p1: Length = 3 ∧ Gills = no ∧ Beak = yes ∧ Teeth = many
• p2: Length = 4 ∧ Gills = no ∧ Beak = yes ∧ Teeth = many
• p3: Length = 3 ∧ Gills = no ∧ Beak = yes ∧ Teeth = few
• p4: Length = 5 ∧ Gills = no ∧ Beak = yes∧Teeth = many
• p5: Length = 5 ∧ Gills = no ∧ Beak = yes ∧ Teeth = few

Five negative examples:

• n1: Length = 5 ∧ Gills = yes ∧ Beak = yes ∧ Teeth = many
• n2: Length = 4 ∧ Gills = yes ∧ Beak = yes ∧ Teeth = many
• n3: Length = 5 ∧ Gills = yes ∧ Beak = no ∧ Teeth = many
• n4: Length = 4 ∧ Gills = yes ∧ Beak = no ∧…