Introduction

In the previous article, we looked at content-based filtering. The idea was simple: understand what a user already likes, look at the characteristics of those items, and recommend other items with similar features.

It works well, but it also has an important limitation.

If you’ve watched several science-fiction movies, a content-based system may keep recommending more science fiction. If you frequently buy running gear, you may continue seeing products that look very similar to things you’ve already purchased.

The recommendations can be highly relevant, but they can also become predictable.

This is known as overspecialization. By relying heavily on a user’s existing preferences, the system may keep reinforcing what it already knows instead of helping the user discover something new.

That’s a problem for the user, but it can also become a problem for the business.

A retailer may have thousands of products that a customer would potentially love but never see because they don’t resemble previous purchases. A streaming platform may have an entire catalog of content that remains undiscovered because recommendations stay within a narrow set of genres.

Good recommendations shouldn’t just help users find more of the same. They should also create opportunities for discovery.

So how can we recommend something outside a user’s usual preferences without simply guessing?

One answer is to learn from other users.

Imagine you and another user have enjoyed many of the same movies. You both liked Interstellar, Inception, and The Martian. That overlap suggests that your tastes may be similar.

Now suppose that user also loved a movie you’ve never watched.

Even if the movie doesn’t look particularly similar to your previous favorites, their behavior gives us a new signal:

People who like many of the same things you do also liked this. Maybe you will too.

This is the intuition behind collaborative filtering.

Instead of relying primarily on the characteristics of the items, collaborative filtering learns from patterns in how users interact with them. Those interactions might include ratings, purchases, clicks, views, listens, or likes.

This allows the system to uncover relationships that aren’t obvious from item features alone. Two movies don’t need to share a genre to be related. Two products don’t need to belong to the same category. If similar users consistently interact with them, the system can learn that connection.

In this article, we’ll look at how collaborative filtering works, explore user-based and item-based approaches, and see how collective user behavior can turn into personalized recommendations.

What Is Collaborative Filtering?

Collaborative filtering makes recommendations by learning from patterns in user behavior.

Unlike content-based filtering, it doesn’t necessarily need to know anything about the items themselves. We don’t need a movie’s genre, actors, director, or description. Instead, we look at how users have interacted with different movies.

The underlying assumption is simple:

Users who behaved similarly in the past may have similar preferences in the future.

The User-Item Interaction Matrix

A common way to represent this information is with a user-item interaction matrix.

Imagine four users and five movies:

Each row represents a user, while each column represents an item. The values capture interactions—in this case, movie ratings from 1 to 5.

The question marks are where things become interesting. User 1 hasn’t rated Movies C or E, so we don’t know whether they would enjoy them.

But look at User 1 and User 2.

Both gave Movie A a rating of 5, Movie B a 4, and Movie D a 1. Their preferences appear to be quite similar.

User 2 also gave Movie C a rating of 5.

That gives us evidence that Movie C could be a good recommendation for User 1—even without knowing anything about what Movie C is actually about.

Learning From the Crowd

This is where the word collaborative comes from. The system uses the collective behavior of many users to learn which recommendations might be relevant to an individual.

The same idea works with more than ratings. An e-commerce platform could use purchases, a music platform could use listens, and a news platform could use clicks or reading history.

These are all different ways of building the interaction matrix.

In practice, most of these matrices are extremely sparse. A user may interact with only a tiny fraction of the millions of items available, leaving most user-item combinations empty.

The challenge is to use the interactions we do have to make useful predictions about the ones we don’t.

There are two classic ways to approach this problem:

User-based collaborative filtering asks:

Which users are similar to this user?

Item-based collaborative filtering asks:

Which items tend to receive similar interactions from users?

We’ll start with the first approach.

User-Based Collaborative Filtering

User-based collaborative filtering starts with a simple idea:

Find users with similar preferences and use their behavior to recommend something new.

Let’s return to our movie example.

Suppose User 1 has rated:

Movie A: 5
Movie B: 4
Movie D: 1

User 2 has rated those same movies:

Movie A: 5
Movie B: 4
Movie D: 1

Their preferences look very similar. They both liked Movies A and B and disliked Movie D.

But User 2 has also watched Movie C and rated it 5.

Since User 1 and User 2 have shown similar preferences in the past, Movie C becomes a strong candidate to recommend to User 1.

How Do We Find Similar Users?

With millions of users, we need a mathematical way to measure similarity.

One option is cosine similarity, the same concept we used in content-based filtering. Instead of comparing item-feature vectors, however, we’re now comparing user interaction vectors.

For example:

User 1: [5, 4, ?, 1]
User 2: [5, 4, 5, 1]

We compare their ratings on the items they have both interacted with. The more similar their interaction patterns are, the more similar we consider the users.

Other measures, such as Pearson correlation, can also be used. Pearson correlation can be particularly useful with ratings because different users may use rating scales differently.

From Similar Users to Recommendations

In practice, we don’t usually rely on just one similar user. We find a group of the target user’s closest neighbors and look at the items they liked.

Users who are more similar can be given more influence when estimating how much the target user might like an unseen item.

The process is essentially:

Find similar users → Look at what they liked → Score unseen items → Rank → Recommend

This is often called user-user collaborative filtering or user-based nearest-neighbor collaborative filtering.

The important idea is that we still don’t need to know anything about Movie C itself. We recommend it because people whose preferences resemble User 1’s preferences liked it.

Item-Based Collaborative Filtering

User-based collaborative filtering finds people with similar preferences. Item-based collaborative filtering flips the idea around and looks for items that users tend to interact with in similar ways.

Suppose many users who gave Movie A a high rating also gave Movie C a high rating. Over time, the system learns that these two movies have similar interaction patterns.

Now imagine User 1 has already enjoyed Movie A but hasn’t watched Movie C. Because other users tend to like both movies, Movie C becomes a good recommendation.

The important point is that we don’t need to know anything about the movies themselves.

How Are Similar Items Identified?

Just as users can be represented by their interactions with items, items can be represented by the users who interacted with them.

For example:

Movie A: [5, 5, 1, 2]
Movie C: [?, 5, 1, 2]

Each position represents a different user’s rating. Since Movies A and C receive similar ratings from the users who have seen both, the system may consider them similar.

Once again, measures such as cosine similarity can be used to compare these interaction vectors.

If a user likes Movie A, we can find movies with similar interaction patterns and use them as candidates for recommendation.

Isn’t This the Same as Content-Based Filtering?

They may sound similar, but there’s an important difference.

Content-based filtering might decide that two movies are similar because they share the same genre, actors, director, or themes.

Item-based collaborative filtering might decide that those same movies are similar because the same types of users tend to like them.

This means two movies could have completely different characteristics and still be considered similar by collaborative filtering.

A science-fiction movie and a historical drama may not look similar based on their content. But if users who enjoy one consistently enjoy the other, the system can discover that relationship.

That’s one of the strengths of collaborative filtering: it can uncover connections that aren’t obvious from the item features alone.

Subscribe now

From Similarity to Recommendations

Finding similar users or items is useful, but similarity alone doesn’t give us a recommendation. We still need to decide which unseen items are most likely to interest the user.

This is where the neighbors come in.

Using Similar Users

Suppose we want to estimate whether User 1 would like Movie C.

We first find the users most similar to User 1 who have already rated Movie C. Instead of treating all of their ratings equally, we can give more weight to users whose preferences are more similar to User 1.

For example:

User 2: Similarity = 0.90 → Movie C rating = 5
User 3: Similarity = 0.70 → Movie C rating = 4
User 4: Similarity = 0.20 → Movie C rating = 2

User 2’s rating should influence our prediction more than User 4’s because User 2 has a much more similar interaction history.

By combining these ratings and weighting them by similarity, we can estimate a preference score for User 1.

Using Similar Items

The same idea works with item-based collaborative filtering.

Suppose User 1 hasn’t watched Movie C but has given high ratings to Movies A and B. If Movie C has interaction patterns similar to Movies A and B, that’s evidence that User 1 may also enjoy it.

Again, the most similar items contribute more to the estimated score than weakly related ones.

Turning Scores Into Recommendations

We repeat this process for items the user hasn’t interacted with and estimate a preference score for each one.

Those scores can then be ranked:

Movie C → 4.7
Movie F → 4.3
Movie E → 3.8
Movie G → 2.1

The highest-scoring unseen items become the top recommendations.

The overall process is:

Find neighbors → Estimate preference scores → Rank unseen items → Recommend

This is the core idea behind neighborhood-based collaborative filtering. We use the behavior of similar users—or the interaction patterns of similar items—to fill in some of the missing information in the user-item matrix.

Advantages and Limitations of Collaborative Filtering

Collaborative filtering is powerful because it doesn’t need to understand what an item actually is. It learns from the patterns created by users interacting with items.

That gives it some important advantages over content-based filtering, but it also introduces a different set of challenges.

It Can Help Users Discover Something New

One of the biggest advantages of collaborative filtering is discovery.

With content-based filtering, someone who watches science-fiction movies may continue receiving more science-fiction recommendations because the system looks for items with similar features.

Collaborative filtering can move beyond those obvious similarities. If people with similar tastes also enjoy a historical drama, the system can recommend that movie even though it looks very different from the user’s previous choices.

This can benefit both the user and the business. Users discover more of the catalog, while businesses have an opportunity to surface relevant items that might otherwise receive very little exposure.

It Doesn’t Require Detailed Item Features

Collaborative filtering can also work without detailed information about the items.

We don’t necessarily need to know a movie’s genre or actors, or a product’s category and description. If we have enough user-item interactions, the system can learn useful relationships directly from behavior.

This is particularly valuable when item features are difficult to define or don’t fully capture why people like something.

The Cold-Start Problem

Cold start is one of the biggest challenges for collaborative filtering.

A new user has no interaction history, so the system doesn’t yet know which existing users have similar preferences. Until the user starts clicking, watching, purchasing, or rating items, personalized recommendations are difficult.

A new item creates a similar problem. If nobody has interacted with it yet, collaborative filtering has no behavioral pattern from which to learn who might like it.

This is different from content-based filtering, which can often recommend a new item immediately using its features.

In practice, platforms may use popular items, onboarding preferences, content-based recommendations, or other signals until enough interaction data becomes available.

Data Sparsity

Even established users interact with only a tiny fraction of the available catalog.

Imagine a platform with one million users and 100,000 items. The user-item matrix contains an enormous number of possible interactions, but most of those cells will be empty.

This sparsity makes it harder to find reliable similarities, especially for users or items with relatively few interactions.

Scalability

Neighborhood-based collaborative filtering can also become expensive as the platform grows.

Finding similar users among millions of users—or comparing large numbers of items—requires significant computation. Techniques such as approximate nearest-neighbor search can help, but at very large scale, other approaches may become more practical.

This is one reason recommendation systems often move beyond traditional neighborhood methods to techniques such as matrix factorization and embeddings.

The Trade-Off

Collaborative filtering can uncover relationships that item features alone would never reveal. It is especially useful for discovery because recommendations are informed by the collective behavior of many users.

But that strength depends on having enough interaction data. New users, new items, sparse matrices, and large-scale similarity calculations can all make the approach more difficult.

There is no universally better choice between content-based and collaborative filtering. In practice, many recommendation systems combine both because their strengths and weaknesses complement each other.

Subscribe now

Key Takeaways

Collaborative filtering makes recommendations by learning from patterns in user-item interactions. Instead of relying on detailed information about the items themselves, it uses the collective behavior of users to discover relationships.

Here are the main ideas to remember:

• The user-item interaction matrix is the foundation. It can contain explicit feedback such as ratings or implicit signals such as clicks, purchases, views, or likes.

• User-based collaborative filtering finds similar users. If people with preferences similar to yours liked something you haven’t seen, that item may be a good recommendation for you.

• Item-based collaborative filtering finds similar items based on interaction patterns. Two items can be considered similar even if their actual characteristics are completely different.

• Similarity can be measured in different ways. Cosine similarity is common, but the right measure depends on the type of interaction data.

• Neighbor preferences can be used to estimate scores for unseen items. Those scores are then ranked to generate the final recommendations.

• Collaborative filtering can improve discovery. Because it isn’t limited to item features, it can surface relevant items outside a user’s usual preferences.

• Cold start and sparsity are major challenges. New users and new items have little interaction history, while most users interact with only a small fraction of the available catalog.

The central idea is simple:

Don’t just look at what an item is. Learn from how people interact with it.

That shift—from understanding item characteristics to learning from collective behavior—is what makes collaborative filtering so powerful.

What’s Next?

Collaborative filtering gives us a powerful way to generate recommendations without needing to understand the content of every item. By learning from user-item interactions, we can discover relationships that may not be obvious from item features alone.

But as the number of users and items grows, the interaction matrix becomes enormous—and mostly empty. Comparing users or items directly can become difficult and expensive.

What if we could take that large, sparse matrix and represent every user and item using a much smaller set of hidden characteristics?

That’s the idea behind matrix factorization.

Instead of working directly with millions of interactions, matrix factorization learns compact representations—or latent factors—for users and items. Those representations can then be used to estimate preferences and generate recommendations.

In the next article, we’ll explore what those latent factors actually mean, how matrix factorization learns them, and why this approach became such an important part of modern recommendation systems.

Until next time, keep learning, keep building!

A few weeks ago, Dataford sent me some data that surprised me. For years, “Data Scientist” was the top job in data. The best title, the best pay, the one everyone wanted. That’s no longer true. Two newer job titles now pay more: Research Engineer and Machine Learning Engineer.

Here’s what the data shows, and what it means if you’re building a career in data.

The pay gap is bigger than you’d think

Dataford looked at median pay across the US, using over 85,000 real job listings. They also scored how hard each job is to interview for, using more than 600,000 real interview reports, on a scale of 1 to 5.

The best-paid data job, Research Engineer, has a median pay of $200,000 a year. The lowest-paid, Research Analyst, pays under $93,000. That’s more than double, even though both jobs have “research” in the name.

The lesson here: a job title alone doesn’t tell you much about the pay. Two people can both call themselves an “Analyst” and earn very different salaries.

Data jobs are splitting into two groups

A clear pattern shows up in the data. A small group of jobs pays very well: Research Engineer, Machine Learning Engineer, Quantitative Analyst, and Data Scientist. These all pay over $170,000.

Below that is a much bigger group of jobs, all packed into a narrower pay range, roughly $93,000 to $141,000. This group includes Data Analyst, Business Intelligence Analyst, Analytics Engineer, and several others. Even though these jobs use different skills, like writing SQL, building dashboards, or running statistics, they all land in a similar, lower pay range.

In plain terms: jobs that build and ship machine learning systems are pulling far ahead in pay. Jobs based on analysis and reporting are staying behind.

If you’re early in your data career, this is the biggest thing to take away. The fastest way to a higher salary is through machine learning and engineering skills, not analytics tools alone.

The hardest job to get isn’t the best paid

Here’s the most surprising finding in the data. Quantitative Analyst has the hardest interview of any job Dataford measured, out of 53 different roles. But it doesn’t pay the most. It actually pays less than both Research Engineer and Machine Learning Engineer.

So Quantitative Analysts go through the toughest interviews in tech and still end up earning less than people in some easier-to-get roles. If you’re deciding where to focus your job search energy, Machine Learning Engineer looks like a better trade: strong pay, for a somewhat easier interview.

The company matters more than the job title

One more pattern stood out. How hard a company’s interview is barely changes based on the job you’re applying for. But it changes a lot from company to company.

A few examples:

Anthropic and OpenAI both pay very well for data and machine learning roles, over $338,000 a year on average. Their interviews are also some of the hardest in the dataset.

Netflix pays the most of any company Dataford measured, at $473,000 a year. But its interview difficulty is only average, not the hardest.

Google, on the other hand, has one of the toughest interview processes of any company, but its pay for the same kind of role is less than half of what Netflix offers.

The lesson: two companies can pay very differently for the same job, at a similar interview difficulty. It pays to research the company, not just the job title.

What this means for you

If you’re planning your next move in data, a few things are worth remembering:

The top of the pay scale has shifted from Data Scientist to Machine Learning Engineer and Research Engineer. If higher pay is your goal, that’s the direction to build your skills in.

The middle of the market, roughly $93,000 to $141,000, is crowded. Moving from one analyst role to another is usually a small pay bump. The bigger jump in pay comes from moving into machine learning or engineering work.

A hard interview doesn’t mean a higher salary. It often says more about the company than about the job. It’s worth researching companies as carefully as you prepare for their interviews.

About Dataford

That last point, about interviews telling you more about the company than the job, is exactly what Dataford is built to help with. Founded in 2023 by Amney Mounir, a former Lead Analyst at Meta, Dataford is an interview prep platform built specifically for tech, AI, and data roles. Rather than generic study guides, it uses real candidate interview reports to build prep that’s tailored to the specific company and role you’re interviewing for — a question bank of 19,000+ practice questions, over 40,000 company-specific guides across 50+ roles, AI-run mock interviews with automated feedback, and built-in SQL and Python practice spaces. The 613,000+ interview reports behind this article’s difficulty scores are the same kind of data that powers Dataford’s prep tools. You can check it out at dataford.io.

Data shared by Dataford. Pay figures are the midpoint of each job listing’s stated salary range, based on US job postings. Interview difficulty is scored from real interview reports, from 1 (very easy) to 5 (very hard).

Introduction

In my previous article, I explored how content-based recommendation systems work—how they use the characteristics of items to identify and recommend similar products.

But understanding the idea is only the first step.

What does it actually take to build one end-to-end?

In this article, we’ll move from theory to implementation by building a content-based recommendation system using the H&M Personalized Fashion Recommendations dataset. We’ll start with raw product metadata and work our way through creating product profiles, converting them into numerical representations, measuring similarity, and generating recommendations.

But we won’t stop once the algorithm produces a list of similar products.

We’ll inspect the recommendations, identify where the first version falls short, improve the recommendation logic, and finally evaluate the system using actual customer transaction data.

By the end, we’ll have worked through the complete pipeline:

Product Data → Product Profiles → TF-IDF Vectors → Cosine Similarity → Recommendations → Refinement → Evaluation

The goal isn’t just to build a recommender that works technically. It’s to understand the decisions that turn a similarity algorithm into a more useful recommendation experience.

1. Understanding the Dataset

For this project, we’ll use the H&M Personalized Fashion Recommendations dataset from Kaggle. It contains information about H&M’s product catalog, customers, and historical transactions.

The dataset includes three main files:

• articles.csv — product information and attributes

• customers.csv — customer-level information

• transactions_train.csv — historical customer purchases

We’ll start with articles.csv, since a content-based recommender relies primarily on information about the items themselves.

articles = pd.read_csv("data/articles.csv")

print(articles.shape)
articles.head()

The dataset contains more than 100,000 products, with attributes describing different aspects of each article.

Some of the most useful fields for our recommender are:

• product_type_name — dress, trousers, shirt, etc.

• product_group_name — broader product category

• graphical_appearance_name — solid, striped, patterned, etc.

• colour_group_name — product color

• department_name — department the product belongs to

• section_name — product section

• garment_group_name — garment category

• detail_desc — natural-language product description

For example, a product in the catalog might look something like:

These attributes give us the raw material for describing what each product is.

That’s important because our recommender won’t initially know anything about customer preferences or what other shoppers purchased. It will make recommendations entirely by comparing the characteristics of one product with another.

Later in the article, we’ll bring in transactions_train.csv—not to build the recommendations, but to evaluate them against actual customer behavior.

Dataset: H&M Personalized Fashion Recommendations — Kaggle

With the product catalog loaded, the next step is deciding which attributes should define similarity between two products.

2. Preparing the Product Data

Not every column in the product catalog needs to be part of the recommendation system. The goal is to select attributes that meaningfully describe a product and can help us determine whether two items are similar.

For this project, we’ll use a combination of product category, appearance, color, department, and description.

content_features = [
    "product_type_name",
    "product_group_name",
    "graphical_appearance_name",
    "colour_group_name",
    "department_name",
    "section_name",
    "garment_group_name",
    "detail_desc"
]

We also retain article_id, product_code, and prod_name. The article_id uniquely identifies an article, while product_code will become useful later when we deal with multiple variants of the same underlying product.

Handling Missing Values

Before combining these attributes, we need to handle missing values. Since we’re going to represent the product information as text, we can replace missing values with an empty string.

products[content_features] = (
    products[content_features].fillna("")
)

Preparing Categorical Features

There is one more small but important preprocessing step.

Consider the color Light Blue. If we treat all our attributes as ordinary text, the vectorizer will see light and blue as two separate terms. Similarly, Garment Upper Body would become three independent words.

For categorical attributes, we want to preserve the complete category instead:

Light Blue → light_blue

Garment Upper Body → garment_upper_body

We can do that with a simple transformation:

for col in categorical_features:
    products[col] = (
        products[col]
        .str.lower()
        .str.replace(" ", "_")
    )

Product descriptions are different. They contain natural language, so individual words can carry useful information. We therefore keep the description as regular text and apply only light cleaning.

products["detail_desc"] = (
    products["detail_desc"]
    .str.lower()
    .str.strip()
)

At this point, we have a clean set of attributes describing each product.

Next, we’ll combine them to create a content profile for every item in the catalog—the representation our recommendation system will use to compare products.

Subscribe now

3. Building Product Content Profiles

Now that the product attributes are clean, we need to bring them together into a single representation for each item.

Think of this as creating a profile that describes each product using everything we know about it.

First, we combine the categorical attributes:

products["categorical_content"] = (
    products[categorical_features]
    .agg(" ".join, axis=1)
)

Then we add the product description:

products["content_profile"] = (
    products["categorical_content"]
    + " "
    + products["detail_desc"]
)

A simplified content profile might now look something like:

dress garment_full_body solid black dark
divided_basics jersey_basic
short fitted dress with narrow shoulder straps

Instead of knowing the product only as an article_id, our system now has information about what the product is, what it looks like, where it belongs in the catalog, and how it is described.

Converting Product Profiles into Vectors

The content profiles are still text, so we need to convert them into a numerical form before we can compare products.

We’ll use TF-IDF (Term Frequency–Inverse Document Frequency).

TF-IDF gives more importance to terms that help distinguish one product from another while reducing the influence of terms that occur across many products.

tfidf = TfidfVectorizer(
    stop_words="english",
    max_features=10000
)

tfidf_matrix = tfidf.fit_transform(
    products["content_profile"]
)

The result is a sparse matrix where:

Rows → Products

Columns → Product attributes and descriptive terms

Each product is now represented as a numerical vector containing information about its content.

Why Use TF-IDF?

We could simply one-hot encode all the categorical attributes. But our product profiles contain both structured attributes and natural-language descriptions.

TF-IDF gives us a simple way to represent both in the same feature space without introducing a much more complex modeling approach.

With every product represented as a vector, we’re ready for the key step: measuring the similarity between products and generating recommendations.

4. Generating Recommendations with Cosine Similarity

Now that every product has a numerical representation, we can start comparing them.

We’ll use cosine similarity, which measures how similar two vectors are based on the angle between them. A score closer to 1 indicates greater similarity, while a score closer to 0 indicates that the products share fewer characteristics.

Instead of calculating a similarity matrix between every pair of products—which would be unnecessarily large for a catalog of more than 100,000 items—we calculate similarity only when a recommendation is requested.

similarity_scores = cosine_similarity(
    tfidf_matrix[idx],
    tfidf_matrix
).flatten()

We then rank the products from most to least similar and return the top results.

similar_indices = similarity_scores.argsort()[::-1]
top_indices = similar_indices[:5]

The basic recommendation process is therefore quite simple:

Select a Product → Find its TF-IDF Vector → Calculate Cosine Similarity → Rank Products → Return Top Recommendations

Our First Recommendations

Let’s test the recommender using the Alcazar Strap Dress, a black dress from the Divided Basics section.

Our initial Top-5 recommendations looked like this:

At first glance, these are excellent similarity scores. The recommender clearly understands that these products are closely related to the selected dress.

But there is also an obvious problem.

Every recommendation is essentially the same dress in a different color.

We observed the same behavior when testing a pair of jogger bottoms—the recommendations were dominated by the same or nearly identical Jerry Jogger products.

Technically, the recommender is doing exactly what we asked it to do. Different variants of the same product have almost identical metadata and descriptions, so cosine similarity naturally ranks them at the top.

From a recommendation-experience perspective, however, we need to distinguish between two different use cases:

Other Colors / Variants

Show me another version of the product I’m already viewing.

versus

You May Also Like

Show me different products that are similar to the one I’m viewing.

For this project, we’re trying to build the second experience.

This is an important reminder that a mathematically correct recommendation isn’t necessarily a useful recommendation.

So before evaluating the system, we need to make one more improvement to our recommendation logic.

Subscribe now

5. Improving Recommendation Diversity

Our first recommender was good at finding similar products—but too good at finding near-duplicates.

To make the results more useful for product discovery, we need to prevent multiple variants of essentially the same product from occupying our recommendation slots.

Fortunately, the H&M dataset gives us a useful field: product_code.

While article_id identifies a specific article or variant, product_code helps us identify products belonging to the same underlying product family.

We can use this information as a post-ranking rule. After ranking products by cosine similarity, we:

1. Exclude the product family of the selected item.

2. Keep only one recommendation from each product family.

3. Avoid repeating the same product name.

The core filtering logic looks like this:

seen_codes = {selected_product_code}
seen_names = {selected_name.lower()}

for i in similar_indices:
    code = products.loc[i, "product_code"]
    name = products.loc[i, "prod_name"].lower()

    if code not in seen_codes and name not in seen_names:
        filtered_indices.append(i)
        seen_codes.add(code)
        seen_names.add(name)

This doesn’t change the underlying similarity model. Instead, it changes how we use its rankings to create the final recommendation experience.

Did It Improve the Recommendations?

Let’s return to the black Alcazar Strap Dress.

After applying the diversity rules, the recommendations changed from different colors of the same dress to products such as:

The similarity scores are lower than before—and that’s perfectly reasonable.

Previously, scores above 0.90 came from products that were nearly identical. Now the system is finding different products that still share important characteristics with the selected item.

We observed the same improvement with the jogger example. Instead of filling the results with Jerry Jogger variants, the recommender began surfacing related products such as shorts and sweatpants.

This highlights an important aspect of building recommendation systems:

The highest similarity score does not always produce the best recommendation experience.

The model provides a ranking, but business rules and post-ranking logic can determine whether those results are actually useful to the customer.

Now that we’re satisfied with the recommendation behavior, the next question is harder:

How do we know whether the recommender is actually good?

6. Evaluating the Recommender

Generating recommendations is relatively easy. Evaluating whether those recommendations are actually good is much harder.

So far, we have evaluated our system qualitatively by looking at the recommendations and asking whether they make sense. To add a quantitative perspective, we’ll use the historical purchases available in transactions_train.csv.

The basic idea is:

Past Purchase → Generate Top-10 Recommendations → Compare Against Future Purchases

To avoid using future information, we create a time-based holdout rather than randomly splitting transactions.

cutoff_date = pd.Timestamp("2020-09-01")

history = transactions[transactions["t_dat"] < cutoff_date]
future = transactions[transactions["t_dat"] >= cutoff_date]

For each customer, we use their most recent purchase before the cutoff as the seed product. Products purchased after the cutoff become our relevant items.

Because the transaction dataset is large, we evaluate the recommender on a sample of 500 customers who made purchases both before and after the cutoff.

Choosing the Metrics

We’ll look at four common recommendation metrics:

• Precision@10 — How many of the Top-10 recommendations were later purchased?

• Recall@10 — How many of the customer’s future purchases appeared in the Top 10?

• NDCG@10 — Did relevant products appear near the top of the ranking?

• Hit Rate@10 — For how many customers did at least one recommendation match a future purchase?

Our first evaluation produced:

These numbers are extremely low.

Does that mean our recommender is bad?

Not necessarily.

Consider a customer whose most recent purchase was a black dress. Our content-based recommender might return ten highly relevant dresses. But if the customer’s next purchase is a handbag or a pair of trousers, every recommendation receives zero credit.

The problem is that our recommender answers:

“What products are similar to this product?”

Our evaluation, however, is asking:

“Can this one product predict anything the customer will purchase next?”

Those are fundamentally different objectives.

This brings us to an important lesson in evaluating recommendation systems:

Before interpreting a metric, make sure the evaluation task matches the recommendation experience you’re trying to build.

For our “You May Also Like” recommender, we therefore need an evaluation that is better aligned with its purpose.

Subscribe now

7. Aligning Evaluation with the Recommendation Use Case

Since our system is designed to recommend similar products, a more appropriate evaluation is to compare recommendations against future purchases within the same product category.

For example, if the seed product is a dress, we evaluate whether the Top-10 recommendations include another dress that the customer later purchased.

Conceptually, the evaluation becomes:

Past Purchase (Dress) → Recommend Similar Products → Compare Against Future Dress Purchases

We keep the same time-based holdout and the same ranking metrics. The only difference is how we define a relevant item.

relevant = customer_future[
    customer_future["product_type_name"] == seed_type
]["article_id"].unique()

Of the original 500 customers, 103 had at least one future purchase in the same product category, giving us a smaller but more relevant evaluation set.

The results changed considerably:

Every metric improves when the evaluation is better aligned with the intended recommendation experience. Recall@10, for example, increases from approximately 0.03% to 1.46%.

The absolute numbers are still low, and that’s worth acknowledging.

Exact-purchase evaluation is a very strict test. If we recommend a highly relevant black dress but the customer purchases a different black dress outside our Top 10, the recommendation receives no credit. We’re also selecting just 10 items from a catalog containing more than 100,000 products.

More importantly, content similarity alone cannot capture everything that influences a purchase. Price, availability, individual taste, previous behavior, trends, and many other factors can affect what a customer ultimately chooses.

The goal here isn’t to make the metrics look impressive. It’s to evaluate the recommender in a way that reflects what it was actually designed to do.

A recommendation metric is only meaningful when the definition of relevance matches the experience you’re trying to build.

This is why offline recommendation metrics should always be interpreted alongside the business objective and the type of recommendation system being evaluated.

8. Limitations and Key Takeaways

We now have an end-to-end content-based recommendation system: starting with raw product metadata, creating product representations, generating and refining recommendations, and finally evaluating them against actual customer behavior.

But like any recommendation approach, content-based filtering has limitations.

Where Does Content-Based Filtering Fall Short?

It is only as good as the product metadata.
Our model understands products through attributes such as type, color, department, garment group, and description. If an important characteristic isn’t captured in the data, the recommender can’t use it.

Similarity doesn’t necessarily mean preference.
Knowing that two dresses are similar doesn’t tell us whether a particular customer will like both. Our model understands products, but it doesn’t really understand the customer.

It can become repetitive.
We saw this firsthand when our initial recommendations were dominated by different variants of the same product. Post-ranking rules helped, but this highlights the tendency of content-based systems to stay close to what they already know.

Offline metrics don’t tell the whole story.
A recommendation can be useful even if the customer doesn’t eventually purchase that exact product. Metrics such as Precision, Recall, and NDCG are valuable, but they need to be interpreted in the context of the recommendation experience.

Key Takeaways

There are a few lessons from this project that extend beyond content-based filtering:

• Item representation matters. Better product features lead to better similarity comparisons.

• The highest similarity score isn’t always the best recommendation. Ranking often needs additional business logic.

• Inspect your recommendations. Our first model worked mathematically, but qualitative inspection exposed a clear product-experience problem.

• Evaluation should match the objective. Changing how we defined relevance substantially changed how we interpreted the same recommender.

• A model is only one part of a recommendation system. Feature engineering, catalog structure, ranking rules, evaluation, and business context all contribute to the final experience.

Perhaps the biggest takeaway is that building a recommender doesn’t end when cosine_similarity() returns a score.

The real work begins when we ask whether those recommendations are actually useful.

Final Thoughts

In my previous article, we looked at how content-based recommendation systems work. In this one, we took the next step and built one end-to-end using a real-world product catalog.

What I enjoyed most about this project was that the interesting part wasn’t getting cosine similarity to work—it was everything that came after. We had to inspect the recommendations, deal with repetitive product variants, think about what a useful recommendation actually means, and make sure our evaluation matched the experience we were trying to build.

If you’d like to explore the complete implementation or experiment with it yourself, I’ve included the full notebook and code in the GitHub repository.

I’ll continue sharing practical applications of data science and breaking down how these systems work beyond just the algorithms.

Until next time, keep learning, keep building, and keep solving real problems with data.

Introduction

You watch a movie on a streaming platform and enjoy it. When you come back the next day, you see several recommendations that feel similar—maybe they belong to the same genre, have similar themes, or feature some of the same actors.

How did the system decide what to show you?

One way is to look at the movies you’ve already enjoyed, understand what they have in common, and find other movies with similar characteristics.

This is the basic idea behind content-based filtering.

Unlike approaches that rely on what other users are watching or buying, content-based filtering focuses primarily on you and the items you’ve interacted with. If you frequently read articles about machine learning, for example, the system might recommend other articles covering similar topics.

The idea sounds simple, but it raises a few interesting questions. How does a machine understand that two movies or products are similar? Which characteristics should it compare? And how do we turn those similarities into a ranked list of recommendations?

That’s what we’ll explore in this article.

We’ll look at how items are represented using features, how similarity is measured, and how those pieces come together to generate personalized recommendations. We’ll also discuss where content-based filtering works well and where it starts to struggle.

By the end, you’ll understand the intuition behind one of the simplest and most useful approaches to building recommendation systems.

What Is Content-Based Filtering?

Content-based filtering recommends items based on the characteristics of things a user has liked in the past. The idea is fairly intuitive. If we know what you liked before, we can look for other items that share similar characteristics.

Suppose you watched and enjoyed several science-fiction movies. Those movies might share features such as science fiction, space, adventure, futuristic themes, or certain actors and directors.

A content-based recommendation system uses this information to build an understanding of your preferences. It can then search the catalog for other movies with similar features and rank them based on how closely they match your interests.

The same idea works outside of movies. An e-commerce platform might use product category, brand, price range, or product description. A news platform could use topics, keywords, or article content.

The important point is that the system is learning from the content of the items themselves.

A Simple Example

Imagine you’ve liked these three movies:

• Movie A: Sci-Fi, Space, Adventure

• Movie B: Sci-Fi, Space, Thriller

• Movie C: Sci-Fi, Adventure, Futuristic

From this history, the system might learn that you have a strong preference for science fiction and space-related content.

Now suppose there are two movies you haven’t watched:

• Movie D: Sci-Fi, Space, Futuristic

• Movie E: Romance, Comedy, Drama

Movie D is much more similar to the movies you’ve enjoyed before, so it would likely receive a higher recommendation score.

At a high level, the process looks like this:

Items you liked → Identify their features → Find similar items → Recommend

This also highlights what makes content-based filtering different from collaborative filtering. We don’t need to know whether thousands of other users liked Movie D. The recommendation can be made based on the relationship between your preferences and the characteristics of the movie.

Of course, a machine needs a mathematical way to represent those characteristics before it can decide that two items are similar.

That’s where item features and vectors come in.

How Do We Represent Items?

For content-based filtering to work, the system needs a way to describe each item.

As humans, we can look at two movies and recognize that they’re similar. A machine needs that information represented in a form it can compare.

This is where item features come in.

For a movie, useful features might include genre, actors, director, language, or keywords from the description. For a product, they could include category, brand, price range, color, or information from the product description.

The choice of features matters. If the features don’t capture what makes two items meaningfully similar, the recommendations won’t be very useful.

Turning Features Into Vectors

Once we’ve identified the features, we need to convert them into numbers.

Suppose we describe movies using four genres:

Sci-Fi | Adventure | Romance | Comedy

A science-fiction adventure movie could be represented as:

[1, 1, 0, 0]

A romantic comedy might look like:

[0, 0, 1, 1]

These numerical representations are called item vectors. Once every movie is represented as a vector, the system has a consistent way to compare them.

Real recommendation systems can have hundreds or thousands of features, but the basic idea remains the same.

What About Text?

Sometimes the most useful information isn’t stored in neat categories. It might be hidden inside a movie synopsis, product description, or article.

One common approach is TF-IDF, which converts text into numerical features based on the words that are important within each document. Items that contain similar important words will end up with similar representations.

More modern systems often use embeddings instead. Embeddings can capture meaning beyond exact word matches, allowing two items to be considered similar even when they use different words to describe the same idea.

For now, the important idea isn’t which technique we use. It’s that we need to turn each item into a numerical representation that captures something meaningful about it.

Once we have those vectors, the next question becomes:

How do we measure how similar two vectors are?

That’s where cosine similarity comes in.

How Do We Measure Similarity?

Once every item has been represented as a vector, we need a way to compare those vectors.

In other words, we need to answer a simple question: How similar are these two items?

One of the most common ways to do this is cosine similarity.

The Intuition Behind Cosine Similarity

Cosine similarity looks at the angle between two vectors rather than simply measuring the distance between them.

If two vectors point in roughly the same direction, the items they represent are considered similar. If they point in very different directions, the items are less similar.

The score usually ranges from 0 to 1 when the features are non-negative. A score close to 1 means the items are very similar, while a score closer to 0 means they have little in common.

A Simple Example

Let’s go back to our movie example.

Suppose a user enjoyed a movie represented by:

Movie A: [1, 1, 0, 0]
Sci-Fi, Adventure

Now we have two possible recommendations:

Movie B: [1, 1, 1, 0]
Sci-Fi, Adventure, Romance

Movie C: [0, 0, 1, 1]
Romance, Comedy

Movie B points in a similar direction to Movie A because they share two important features. Movie C shares none of those features, so its cosine similarity with Movie A would be much lower.

The system can calculate this similarity across many items and use the scores to identify the closest matches.

Why Cosine Similarity?

Another option would be to calculate the straight-line distance between vectors. That can work, but cosine similarity is often useful when we care more about the pattern of features than their absolute magnitude.

This is especially helpful with text representations such as TF-IDF, where one document may contain many more words than another. Cosine similarity focuses on whether the two documents emphasize similar features rather than simply comparing their size.

Cosine similarity isn’t the only way to measure similarity. Depending on the problem, we could also use Euclidean distance, Jaccard similarity, or other measures. But for content-based recommendation systems, cosine similarity is a common and intuitive place to start.

Now we have the two pieces we need: a way to represent items and a way to measure how similar they are.

The next step is putting them together to generate recommendations.

How Content-Based Recommendations Are Generated

So far, we’ve seen how to represent items as vectors and measure the similarity between them. Now we can put those pieces together to generate recommendations for a user.

Let’s continue with our movie example.

Suppose we’re representing movies using these features:

[Sci-Fi, Adventure, Romance, Comedy, Thriller, Space, Drama]

Step 1: Look at the User’s History

We start with movies the user has already interacted with. These could be movies they watched, liked, or rated highly.

Suppose the user liked these two movies:

Movie A: [1, 1, 0, 0, 0, 1, 0]
Sci-Fi, Adventure, Space

Movie B: [1, 0, 0, 0, 1, 1, 0]
Sci-Fi, Thriller, Space

Looking at these movies, we can already see some patterns. Both contain Sci-Fi and Space, while Adventure and Thriller appear once each.

Step 2: Build a User Profile

Next, we combine the vectors of the movies the user liked to create a representation of their preferences.

One simple approach is to average the item vectors.

From Movie A and Movie B, we get:

User Profile: [1, 0.5, 0, 0, 0.5, 1, 0]

This tells us that Sci-Fi and Space are the strongest signals in the user’s history, while Adventure and Thriller also contribute to their profile.

In a real system, the calculation could be more sophisticated. A movie the user rated five stars, for example, could receive more weight than one they simply watched.

Step 3: Compare the Profile With New Items

Now suppose we have three unseen movies:

Movie C: [1, 1, 0, 0, 0, 1, 0]
Sci-Fi, Adventure, Space

Movie D: [1, 0, 0, 0, 1, 0, 0]
Sci-Fi, Thriller

Movie E: [0, 0, 1, 1, 0, 0, 1]
Romance, Comedy, Drama

We can compare each movie vector with the user’s profile using cosine similarity.

Movie C matches several of the user’s strongest preferences, so we would expect a high similarity score. Movie D also has some overlap, while Movie E looks very different from the user’s history.

Step 4: Rank the Items

Suppose the similarity scores are:

Movie C → 0.94
Movie D → 0.67
Movie E → 0.00

The system can now rank the movies:

1. Movie C
2. Movie D
3. Movie E

Movie C would be the strongest recommendation because its feature vector most closely matches the user’s preference vector.

The entire process can be summarized as:

Item vectors → User profile vector → Calculate similarity → Rank items → Recommend

This is the core idea behind content-based filtering. Instead of relying on what other users liked, the system builds a representation of your preferences and looks for items that are mathematically similar to that profile.

Subscribe now

Advantages and Limitations of Content-Based Filtering

Content-based filtering is relatively simple to understand and can produce highly personalized recommendations. But like any recommendation approach, it comes with trade-offs.

Understanding those trade-offs helps us decide when content-based filtering is a good fit and when another approach might work better.

Why Content-Based Filtering Works Well

One major advantage is that the system doesn’t need information about other users. Recommendations are based on an individual’s preferences and the characteristics of the available items.

This also makes the recommendations relatively easy to explain. If a user likes science-fiction movies and receives another movie with similar themes, genres, or actors, there is a clear reason behind the recommendation.

Content-based filtering can also work well for new items. A newly added movie may have no views or ratings, but if we know its genre, actors, description, and other features, the system can still determine which users might find it relevant.

The Cold-Start Problem

A cold-start problem occurs when the system doesn’t have enough information to make useful recommendations. This can happen with either a new user or a new item.

New-user cold start is a challenge for content-based filtering. If someone has just joined the platform, they haven’t watched, purchased, liked, or rated anything yet. Without that history, there isn’t enough information to build a meaningful user profile.

Platforms can address this in several ways. A streaming service might ask new users to choose a few movies or genres they like during onboarding. Until enough interactions are collected, the system might also rely on popular or trending items.

New-item cold start is different. Content-based filtering can often handle new items better because it doesn’t need other users to interact with them first.

Suppose a new movie has this feature vector:

New Movie: [1, 1, 0, 0, 0, 1, 0]
Sci-Fi, Adventure, Space

Even if nobody has watched it yet, we can compare it with an existing user profile:

User Profile: [1, 0.5, 0, 0, 0.5, 1, 0]

Because the vectors share several features, the movie could still become a strong recommendation for that user.

This is an important distinction: content-based filtering struggles with new users, but can often handle new items as long as useful item features are available.

Overspecialization

Another limitation is that content-based filtering tends to recommend more of what the user already likes.

If someone frequently watches science-fiction movies, the system may continue recommending similar science-fiction titles. Those recommendations may be relevant, but they don’t necessarily help the user discover something different.

This is known as overspecialization. The system can become so focused on existing preferences that it limits novelty and discovery.

A user who loves science fiction might also enjoy a historical drama, but a purely content-based system may never discover that connection if the items don’t share enough features.

Dependence on Item Features

The quality of the recommendations also depends heavily on how well the items are represented.

If important characteristics are missing, the system may conclude that two genuinely similar items are unrelated. Adding more features doesn’t automatically solve the problem either; those features need to capture what actually matters to users.

This can be particularly challenging for products where preferences are difficult to describe using structured attributes alone. Two books, songs, or movies may feel similar to a person even when their obvious metadata looks quite different.

The Trade-Off

Content-based filtering works particularly well when items have rich, meaningful features and users have enough history to reveal their preferences. It can personalize recommendations without needing behavior from a large community of users and can surface new items before they accumulate interactions.

Its biggest weakness is also closely connected to its strength: it learns from what the user already likes. That makes recommendations personalized, but it can also make them predictable.

This is where collaborative filtering takes a very different approach. Instead of asking “What items are similar to the things you already like?”, it asks what we can learn from the preferences of other users.

Key Takeaways

Content-based filtering is one of the most intuitive ways to build a recommendation system. It learns what a user prefers from the items they’ve interacted with and then searches for other items with similar characteristics.

Here are the main ideas to remember:

• Items need to be represented using meaningful features. These could be categories, genres, product attributes, TF-IDF features, or embeddings.

• Item features are converted into vectors, giving the system a numerical representation that it can compare.

• A user profile can be built from previously liked items. A simple approach is to average their item vectors, although interactions can also be weighted based on ratings or other signals.

• Cosine similarity is commonly used to compare vectors. Items that are more similar to the user’s preference vector receive higher recommendation scores.

• Content-based filtering doesn’t depend on other users’ behavior. This makes it possible to recommend new items as long as useful item features are available.

• Cold start affects users and items differently. New users are difficult because there isn’t enough history to understand their preferences. New items are easier to handle because their features can still be compared with existing user profiles.

• Overspecialization is an important limitation. Recommending only items similar to what a user already likes can reduce diversity and discovery.

The central idea is simple:

Understand what the user likes, represent those preferences as features, and find other items that look similar.

That simplicity is what makes content-based filtering a useful starting point for understanding recommendation systems.

What’s Next?

So far, we’ve focused on understanding how content-based filtering works—from representing items as vectors to building user profiles and using similarity to find relevant recommendations.

But understanding the idea is only the first step. The best way to make it stick is to actually build one.

In the next article, we’ll create a content-based recommendation system end to end using Python.

We’ll start with a real dataset, prepare the item features, convert them into numerical representations, calculate similarity, and generate top recommendations for a user. Along the way, we’ll connect each piece of code back to the concepts we covered in this article.

The goal isn’t just to get a recommender running.

It’s to understand exactly how raw data turns into a personalized list of recommendations.

After that, we’ll move on to collaborative filtering and see how recommendations change when we start learning from the behavior of other users.

Until then, keep building, keep learning—wishing you the best in your data journey.

Introduction

Open Netflix, Spotify, YouTube, or your favorite shopping app, and you’ll notice something interesting: your experience is uniquely yours.

The movies on your homepage aren’t the same ones your friend sees. Spotify introduces you to artists you may never have searched for. An online retailer suggests a product you didn’t know you needed—yet somehow, it feels relevant.

None of this happens by accident.

Behind these experiences are recommendation systems, designed to solve a simple but challenging problem:

Out of thousands—or even millions—of options, what should we show this user next?

Think about an online retailer with millions of products. Showing every customer the same bestsellers would be easy, but not particularly useful. Someone shopping for running gear has very different interests from someone furnishing a new apartment.

The challenge is figuring out those preferences from the signals users leave behind.

What did they view? What did they buy? What did they skip? Which products did they return to? What do people with similar interests seem to like?

Those signals help recommendation systems narrow an enormous catalog down to a small set of options that might actually matter to an individual user.

But getting recommendations right involves more than predicting what someone is likely to click.

What do you recommend to a brand-new user with no history? How do new products get discovered when nobody has interacted with them yet? Should you keep showing users more of what they already like, or occasionally introduce something different?

And perhaps most importantly:

How do you know whether your recommendations are actually working?

These questions are what make recommendation systems such an interesting machine learning problem. They sit at the intersection of algorithms, user behavior, experimentation, and business outcomes.

In this article, we’ll build the foundation: what recommendation systems are, the major approaches behind them, how recommendations are generated and ranked, and how we evaluate whether they’re delivering value.

Because ultimately, a good recommendation system isn’t about showing users more options.

It’s about finding the right item for the right user at the right time.

What Is a Recommendation System?

At its simplest, a recommendation system helps users find relevant items from a large set of choices.

Those “items” could be almost anything: movies, songs, products, articles, restaurants, jobs, or even people to follow. Most recommendation problems involve three basic pieces:

Users — the people receiving recommendations
Items — the things that could be recommended
Interactions — the signals that connect users and items

Interactions can be explicit, such as giving a movie five stars or liking a song. But often, they’re implicit. A user clicks on a product, watches a video until the end, skips a song after ten seconds, adds something to a cart, or makes a purchase.

Each action tells us something about preference—although not always as clearly as we might think. For example, purchasing a product usually indicates interest. But what about viewing the same product five times without buying it? Or watching an entire movie and never rating it?

Recommendation systems have to make sense of these imperfect signals.

From Prediction to Ranking

There’s another important distinction between recommendation systems and many traditional machine learning problems.

Suppose a streaming platform has 50,000 movies available. The goal isn’t simply to predict whether you would like one particular movie.

The more useful question is:

Which movies should appear at the top of your recommendations?

That makes recommendation systems fundamentally a ranking problem in many real-world applications. The system might estimate your preference for thousands of items, narrow those possibilities down, and then rank the strongest candidates so that the most relevant ones appear first.

And relevance isn’t necessarily the same as popularity.

The most popular movie on a platform may not interest you at all. Meanwhile, a lesser-known movie that matches your viewing history could be a much better recommendation.

That’s what personalization is trying to achieve: moving from

“What is popular?”

to

“What is relevant to this user?”

Of course, we still need a way to determine that relevance. Some recommendation systems look at the characteristics of the items you’ve liked before. Others learn from the behavior of users who seem similar to you. Many real-world systems combine several approaches.

We’ll look at those approaches next.

The Three Main Approaches to Recommendation Systems

There isn’t one universal way to build a recommendation system.

The right approach depends on the data you have, the type of recommendations you’re making, and the problem you’re trying to solve.

Most recommendation systems are built around three broad approaches: content-based filtering, collaborative filtering, and hybrid systems.

1. Content-Based Filtering

Content-based filtering starts with the items a user has already shown interest in.

It looks at the characteristics of those items and searches for others that are similar.

Imagine you’ve watched several science-fiction movies. A content-based system might look at features such as genre, director, actors, or themes and recommend other movies with similar characteristics.

The basic idea is:

You liked this, so you might like something similar.

This approach can work well even when users have very different tastes because recommendations are based on each person’s individual history.

However, it can also become repetitive. If the system only recommends things similar to what you’ve already consumed, discovering something genuinely different becomes harder.

2. Collaborative Filtering

Collaborative filtering takes a different approach.

Instead of asking “What items are similar?”, it looks for patterns across users and their interactions.

Suppose you and another user have watched and enjoyed many of the same movies. If that person loved a movie you haven’t seen yet, it could be a good recommendation for you.

The intuition is:

People with similar preferences may enjoy similar things.

One advantage is that the system doesn’t necessarily need detailed information about the items themselves. Patterns in user behavior can be enough to uncover useful recommendations.

The challenge is that collaborative filtering needs interaction data. New users and new items have very little history, creating what is commonly known as the cold-start problem.

3. Hybrid Recommendation Systems

In practice, recommendation systems don’t always fit neatly into one category.

A hybrid system combines multiple approaches to take advantage of their strengths while reducing some of their weaknesses.

For example, a system might use collaborative filtering to learn from behavior across users while also incorporating information about the products themselves. It may even bring in additional signals such as popularity, recency, location, context, or business rules.

This is why real-world recommendation systems are often much more than a single machine learning algorithm.

The important thing at this stage isn’t to memorize every technique. It’s to understand the different sources of information each approach relies on:

Content-based filtering learns from the items you like.

Collaborative filtering learns from patterns across users.

Hybrid systems combine multiple signals to produce better recommendations.

In the next section, we’ll zoom out from individual algorithms and look at how these pieces come together in an end-to-end recommendation system.

Subscribe now

How Does a Recommendation System Work?

A recommendation system doesn’t usually compare every user with every possible item and simply pick the best matches. For a platform with millions of users and products, that would quickly become too slow and computationally expensive.

Instead, most real-world recommendation systems break the process into stages. Each stage narrows the possibilities until we’re left with a small set of relevant recommendations for the user.

Step 1: Learn From User Interactions

It starts with user behavior. Every time someone views a product, watches a video, makes a purchase, rates an item, or skips a song, they leave behind a signal about their preferences.

Not all signals mean the same thing. Purchasing a product usually indicates stronger interest than viewing it once, while repeatedly skipping certain content may suggest the opposite. Over time, these interactions help the system build a better understanding of each user’s interests.

Step 2: Generate Candidates

Imagine an online retailer with millions of products. Evaluating every product for every customer would be inefficient, so the system first creates a much smaller set of potentially relevant items. This is known as candidate generation.

Candidates can come from several sources. The system might retrieve products similar to previous purchases, items popular among users with similar behavior, or products that are currently trending.

At this stage, the goal isn’t to identify the perfect recommendation. It’s simply to reduce millions of possibilities to a manageable set of promising candidates.

Step 3: Rank the Candidates

Once the candidate set is smaller, the system can evaluate each item more carefully. A ranking model estimates how relevant each candidate is likely to be for that particular user.

The model might consider the user’s past behavior, item characteristics, popularity, recency, price, and other contextual information. Items receiving stronger scores are then placed higher in the recommendation list.

This is why ranking matters so much. A highly relevant product appearing near the top of the page is much more likely to be discovered than the same product buried much further down.

Step 4: Apply Business Rules

The highest-ranked items aren’t always the ones that ultimately reach the user. Real-world systems also need to account for practical constraints and business objectives.

For example, a retailer may remove products that are out of stock. A streaming platform may avoid showing several nearly identical recommendations, while another system may intentionally introduce newer items to encourage discovery.

These rules help connect the machine learning model with the actual experience the business wants to create.

Step 5: Learn From What Happens Next

Once recommendations are shown, users respond to them. They might click, purchase, watch, save, skip, or simply ignore what they see.

Those responses become new information that the system can learn from. Over time, this creates a continuous feedback loop: the system makes recommendations, observes what users do, and uses those interactions to improve future recommendations.

This is an important distinction. A machine learning model may produce a relevance score, but the recommendation system determines what the user ultimately sees.

How Do We Know if Recommendations Are Good?

Building a recommendation system is only half the problem. Once we have one, we still need to answer a more important question: Are the recommendations actually useful?

At first, this might seem easy to measure. If users click on more recommendations, the system must be getting better. In reality, recommendation quality is more nuanced than any single metric can capture.

Offline Evaluation

Before exposing a new recommendation model to users, data scientists typically evaluate it using historical data. One common approach is to hide some of the items a user actually interacted with and see whether the model can successfully recommend them.

This is where metrics such as Precision@K, Recall@K, and NDCG come in. Each looks at recommendation quality from a slightly different perspective.

Precision@K measures how many of the top K recommendations are relevant. If we recommend 10 products and 4 are relevant to the user, Precision@10 helps capture that quality.

Recall@K looks at the problem from the other direction. Instead of asking how many recommendations were relevant, it asks how many of all the relevant items we successfully retrieved.

NDCG also considers ranking. A relevant item appearing near the top of the list is more valuable than the same item appearing much further down.

These metrics are useful for comparing models quickly before deployment. But they only tell us how well the model performs against historical behavior, which may not always reflect how users will respond to recommendations in the real world.

Online and Business Metrics

Once a recommendation system reaches real users, evaluation becomes much more practical. We want to know whether the recommendations are actually changing user behavior in a useful way.

The metrics we care about depend on the product and business objective. A retailer might track click-through rate, conversion, or revenue, while a streaming platform may care more about watch time, engagement, or retention.

This is why A/B testing plays such an important role in recommendation systems. Instead of assuming that a model with better offline performance will create a better experience, we can test it with real users and measure the actual impact.

When Better Offline Doesn’t Mean Better Online

Imagine that a new recommendation model improves NDCG by 5%. Based on offline evaluation, it looks like a clear improvement.

Then you run an A/B test and discover that click-through rate has dropped.

There could be several reasons. Perhaps the model became very good at recommending items users already knew about. Maybe the recommendations became too similar to one another, reducing discovery. Or the model may have learned historical patterns that don’t translate well to current user behavior.

This is an important lesson in recommendation systems: a better model metric doesn’t automatically mean a better recommendation experience.

A strong evaluation strategy therefore looks at the system from multiple angles. Offline metrics help us evaluate ranking quality, online experiments tell us how users respond, and business metrics tell us whether those changes ultimately create value.

🎤 Interview Q: How would you evaluate a recommendation system?

A: I would start with offline ranking metrics such as Precision@K, Recall@K, and NDCG to compare candidate models. Once a promising model is identified, I would validate it through an A/B test using relevant product and business metrics such as click-through rate, conversion, engagement, or retention. I wouldn’t rely on offline metrics alone because better offline performance doesn’t always translate into better user outcomes.

Subscribe now

Real-World Challenges in Recommendation Systems

Recommendation systems can look straightforward on paper: understand what a user likes, find relevant items, and rank them.

Real-world systems are much messier. Users change their preferences, new products constantly enter the catalog, interaction data is often sparse, and the recommendations themselves can influence what users do next.

Let’s look at some of the most common challenges.

The Cold-Start Problem

Imagine a new user joining a streaming platform. They haven’t watched, rated, or searched for anything yet, so the system has very little information about their preferences.

The same problem occurs with new items. A newly added movie or product has no interaction history, making it difficult for collaborative filtering approaches to know who might like it.

Platforms often address this by using information available at the start, such as item characteristics, popular content, onboarding preferences, or contextual signals. As more interactions are collected, recommendations can gradually become more personalized.

Data Sparsity

Most users interact with only a tiny fraction of the items available on a platform.

Someone may have watched 100 movies, for example, while the catalog contains tens of thousands. In an e-commerce setting with millions of products, the gap can be even larger.

This creates a very sparse user-item interaction matrix. Finding meaningful patterns becomes harder when most user-item combinations contain no interaction at all.

Popularity Bias

Recommendation systems learn from historical behavior, and popular items naturally generate more interactions. That gives the system more evidence about those items, which can lead to them being recommended even more frequently.

Over time, this can create a feedback loop where already-popular items receive most of the exposure while less popular or newer items struggle to be discovered.

A good recommendation experience therefore needs to balance relevance with discovery rather than simply reinforcing what is already popular.

Filter Bubbles and Lack of Diversity

If someone watches several crime documentaries, recommending more crime documentaries might initially seem like a great strategy.

But if every recommendation becomes increasingly similar, the experience can quickly feel repetitive. The system may become very good at predicting existing preferences while becoming worse at helping users discover something new.

This is why recommendation systems often consider diversity, novelty, and serendipity alongside relevance. Sometimes a slightly unexpected recommendation can create more value than another perfectly predictable one.

Scalability

Finally, recommendation systems need to work quickly.

A platform may have millions of users and millions of items, creating an enormous number of possible user-item combinations. Calculating detailed scores for every combination whenever someone opens an app isn’t practical.

This is one reason real-world systems often separate candidate generation from ranking. Candidate generation quickly narrows the catalog, allowing more sophisticated models to focus on a much smaller set of promising items.

These challenges highlight an important point: a recommendation system isn’t successful simply because its model predicts preferences accurately.

It also needs to handle new users and items, surface enough variety, operate at scale, and ultimately create an experience that people find useful.

🎤 Interview Q: What are some of the biggest challenges when building recommendation systems?

A: A strong answer might discuss cold start, data sparsity, popularity bias, diversity, and scalability. Rather than simply listing them, explain why each challenge matters and how it could affect the recommendations users actually see.

Key Takeaways

Recommendation systems are ultimately about helping users navigate an overwhelming number of choices. The algorithms may vary, but the goal remains the same: surface items that are relevant to each user.

Here are the main ideas to remember:

• Recommendation systems personalize what users see by learning from users, items, and the interactions between them.

• Content-based filtering recommends items similar to those a user has liked before, while collaborative filtering learns from patterns across many users. Real-world systems often combine multiple approaches.

• At scale, recommendations are commonly generated in stages. Candidate generation narrows millions of possible items to a smaller set, and ranking determines which of those candidates should appear first.

• Offline metrics are only part of the story. Precision@K, Recall@K, and NDCG help compare models, but online experiments are needed to understand how recommendations affect real user behavior and business outcomes.

• Real-world recommenders face challenges beyond model accuracy, including cold start, sparse data, popularity bias, lack of diversity, and scalability.

Perhaps the most important idea is that a recommendation system shouldn’t be judged only by how accurately it predicts what a user already likes.

A useful system also helps users discover what they might like next.

What’s Next?

We’ve covered the foundation of recommendation systems: what they are, how they work, the major approaches, how they’re evaluated, and some of the challenges that appear in the real world.

But we’ve only scratched the surface.

In the next article, we’ll take a closer look at Content-Based Filtering, one of the most intuitive approaches to building recommendations.

We’ll explore how a system can use information about items and a user’s past preferences to answer a simple question:

“If you liked this, what else might you like?”

We’ll also look at how similarity is calculated, walk through a practical example, and discuss where content-based recommendations work well—and where they begin to fall short.

From there, we’ll continue building our understanding of recommendation systems one concept at a time.

Because before building sophisticated recommenders, it helps to understand the ideas that make personalization possible in the first place.

Introduction

Imagine you’re a data scientist at an online retailer. You have data on millions of customers, including:

• Their purchase history

• Average order value

• Shopping frequency

• Product categories they buy

• Time since their last purchase

Your marketing team asks a simple question:

“How can personalize our marketing campaigns? We want to send different marketing to different groups of customers.”

There’s just one problem.

No customer has been labelled as a “high-value customer,” “bargain shopper,” or “occasional buyer.” The data simply contains customer behaviour—nothing more.

So how do we discover groups of similar customers?

This is where clustering comes in.

Unlike supervised learning, where models learn from labelled examples to make predictions, clustering looks for hidden patterns within the data itself. Instead of predicting an outcome, it groups together observations that are similar to one another.

One of the most popular algorithms for this task is K-Means Clustering.

K-Means has applications far beyond customer segmentation. It can be used to group stores with similar sales patterns, identify products with similar purchasing behaviour, organize documents by topic, detect unusual observations, and even compress images.

Despite its simplicity, K-Means remains one of the most widely used unsupervised learning algorithms because it is intuitive, scalable, and often provides valuable insights before any predictive modeling begins.

In this article, we’ll explore how K-Means works, how to choose the right number of clusters, where it performs well, where it struggles, and how it’s used to solve real-world business problems.

By the end of this issue, you’ll understand why one of the most powerful ways to learn from data isn’t by making predictions—it’s by discovering patterns you didn’t know were there.

What Is Clustering?

Most machine learning algorithms I’ve written about so far have likely been supervised learning algorithms. Models such as Linear Regression, Logistic Regression, Decision Trees, and Random Forests all have one thing in common—they learn from labelled data.

For example, if you are building a model that identifies which customer will make a purchase, your predictions would be based on historical data that actually tells which customer made a purchase and which ones didn’t.

But, what do you do when those labels don’t exist.

Imagine you’re analysing customer purchasing behaviour for the first time. You know how often customers shop, how much they spend, and the types of products they buy, but you don’t know which customers belong to similar groups. There are no labels such as “loyal customers,” “budget shoppers,” or “seasonal buyers.”

This is where clustering becomes useful.

Clustering is an unsupervised learning technique that automatically groups similar observations based on their characteristics. Instead of predicting a known outcome, it searches for natural patterns within the data.

Think of it as organising a box of mixed LEGO bricks. Without knowing anything about them beforehand, you naturally begin sorting them by colour, size, or shape because similar pieces belong together. Clustering algorithms do something similar—they group observations that are more alike than they are different.

The key difference between supervised and unsupervised learning is the goal:

• Supervised learning answers, “Can I predict what will happen?”

• Unsupervised learning answers, “What patterns already exist in my data?”

These patterns often reveal insights that aren’t immediately obvious. A retailer might discover distinct customer segments, a bank might identify groups of customers with different spending behaviours, or a manufacturer might find stores with similar inventory patterns. These insights can then guide marketing strategies, operational decisions, or even serve as inputs to future predictive models.

One of the simplest and most widely used algorithms for discovering these hidden groups is K-Means Clustering.

What Is K-Means Clustering?

Now that we understand what clustering is, let’s look at one of the most popular clustering algorithms: K-Means.

At its core, K-Means tries to answer a simple question:

How can we divide our data into groups so that observations within the same group are as similar as possible?

The algorithm does this by creating K distinct clusters, where K is a number chosen by the user.

For example, if you’re segmenting customers, you might decide to create:

• 3 customer segments

• 5 customer segments

• 8 customer segments

Each cluster is represented by a point called a centroid. You can think of a centroid as the “centre” of a cluster. Customers that are most similar to one another are grouped around the same centroid.

The goal of K-Means is straightforward:

• Make observations within the same cluster as similar as possible.

• Make different clusters as distinct as possible.

To achieve this, the algorithm follows a simple, iterative process. It begins with an initial set of centroids, assigns every observation to its nearest centroid, recalculates the centroid based on the observations assigned to it, and then repeats the process until the clusters stop changing significantly.

Although this may sound computationally intensive, modern implementations are highly efficient and can cluster millions of observations in a relatively short amount of time.

Let’s return to our customer segmentation example.

Suppose you have three hundred thousand customers and choose K = 4. After running K-Means, you might discover four distinct groups:

• Customers who shop frequently and spend a lot.

• Customers who make occasional large purchases.

• Price-sensitive customers who mainly buy during promotions.

• Infrequent customers with low overall spending.

Notice that these labels weren’t provided to the algorithm. K-Means simply grouped customers based on similarities in their behaviour. It’s up to the data scientist and business stakeholders to interpret each cluster and decide how to use those insights.

This highlights an important point: K-Means discovers patterns, but humans give those patterns meaning.

In the next section, we’ll walk through the algorithm step by step to see exactly how these clusters are formed.

How K-Means Works

Although K-Means may sound sophisticated, the algorithm follows a surprisingly simple process. It performs two tasks iteratively: assigning observations to clusters and updating the centre of each cluster.

Let’s walk through the process using our customer segmentation example.

Step 1: Choose the Number of Clusters (K)

The first step is deciding how many clusters you want the algorithm to create.

Suppose you believe your customers can be grouped into four distinct segments. You would set K = 4.

At this point, the algorithm doesn’t know what those four groups look like—it only knows that it needs to create four of them. The first K can be chosen at random based on intuition.

Step 2: Initialize the Centroids

Next, K-Means randomly selects K observations to serve as the initial centroids.

Think of these centroids as temporary “cluster centres.” Since they are chosen randomly, they usually aren’t the final centres, but they provide a starting point for the algorithm.

Step 3: Assign Every Observation to the Nearest Centroid

The algorithm now looks at every observation in the dataset and calculates which centroid it is closest to.

Each observation is assigned to the nearest centroid, forming the first version of the clusters.

At this stage, the clusters are almost always imperfect because the centroids were selected randomly.

Step 4: Update the Centroids

Once all observations have been assigned, the algorithm recalculates the centre of each cluster. The new centroid is simply the average location of all the observations assigned to that cluster.

Because the centre has changed, some observations may now be closer to a different centroid than before.

Step 5: Repeat Until the Clusters Stabilize

The algorithm repeats the previous two steps:

• Assign observations to the nearest centroid.

• Recalculate each centroid.

With every iteration, the clusters become more stable and the centroids move less and less. Eventually, the centroids stop changing significantly, indicating that the algorithm has converged.

At this point, K-Means has found a set of clusters where observations within the same cluster are as similar as possible while remaining as different as possible from observations in other clusters.

Although the algorithm doesn’t guarantee the perfect clustering solution, it often produces highly useful groupings that can reveal valuable business insights.

🎤 Interview Q: Why does K-Means repeat the assignment and update steps?

A: Assigning observations changes the composition of each cluster, which changes the centroid. Once the centroids move, some observations may become closer to a different centroid. The algorithm repeats these steps until the clusters stop changing significantly.

Subscribe now

Choosing the Right Value of K

One of the first questions you’ll encounter when using K-Means is surprisingly simple:

How many clusters should I create?

Unfortunately, there isn’t a single correct answer.

If you choose too few clusters, you may group together observations that are actually quite different. If you choose too many, you may split meaningful groups into several smaller clusters, making the results difficult to interpret.

So, how do data scientists decide?

1. Start with the Business Problem

Before looking at any metrics, ask yourself why you’re clustering the data in the first place.

For example:

• A retailer may want three customer segments for a simple marketing strategy.

• A bank may need five risk groups to support different lending policies.

• A logistics company may cluster stores into four operational profiles for inventory planning.

The “best” value of K should support the business decision you’re trying to make.

2. Use Evaluation Metrics as a Guide

Once you have a reasonable range of values for K, evaluation metrics can help narrow down your options.

The most common approaches include:

Elbow Method

The Elbow Method measures how much variation exists within each cluster. As you increase K, the clusters become tighter and this variation decreases.

At some point, however, adding another cluster provides only a small improvement. This point, often called the elbow, is a good candidate for the optimal value of K.

Silhouette Score

The Silhouette Score measures two things at the same time:

• How similar an observation is to others within its own cluster.

• How different it is from observations in neighbouring clusters.

Higher scores generally indicate well-separated, well-defined clusters.

Gap Statistic

The Gap Statistic compares your clustering results to what would be expected if the data had no natural grouping at all.

If your clusters are significantly better than random, the chosen value of K is likely meaningful.

3. Validate the Results

Even if a metric suggests an “optimal” value of K, the final decision shouldn’t stop there. Ask yourself:

• Do these clusters make business sense?

• Can you explain what makes each cluster unique?

• Would different actions be taken for each cluster?

If the answer is no, then the clustering may not be useful—even if the metrics look excellent.

The ultimate goal of clustering isn’t to maximise a mathematical score. It’s to discover groups that lead to better business decisions.

🎤 Interview Q: How do you choose the value of K?

A: I start with the business objective to determine a reasonable range of values for K. Then I use techniques such as the Elbow Method, Silhouette Score, or Gap Statistic to compare different options. Finally, I validate whether the resulting clusters are meaningful and actionable from a business perspective.

Applications, Advantages, and Limitations

K-Means is one of the most widely used clustering algorithms because it strikes a balance between simplicity, speed, and effectiveness. While it isn’t the right solution for every problem, it often serves as an excellent starting point for exploring patterns in data.

Where Is K-Means Used?

One of the most common applications of K-Means is customer segmentation. Instead of treating every customer the same, businesses can group customers based on purchasing behaviour, spending habits, or engagement levels. Marketing teams can then tailor promotions and campaigns for each segment rather than adopting a one-size-fits-all approach.

Retailers also use K-Means to cluster stores with similar sales patterns, inventory needs, or customer demographics. This allows operational strategies to be customised for different groups of stores instead of applying the same approach across the entire network.

Beyond retail, K-Means has applications in fraud detection, where unusual transactions may form their own cluster, document clustering, where similar articles or documents are grouped together, and image compression, where similar colours are combined to reduce file size.

Why Is K-Means So Popular?

K-Means remains popular because it is:

• Simple to understand and implement.

• Fast and scalable, making it suitable for large datasets.

• Easy to interpret, with each cluster representing a meaningful group of similar observations.

• Often a strong baseline algorithm before exploring more advanced clustering techniques.

For many business problems, these advantages make K-Means the first algorithm data scientists try.

When Should You Avoid K-Means?

Like every machine learning algorithm, K-Means makes certain assumptions about the data.

It works best when clusters are relatively compact and well separated. If the clusters have irregular shapes or overlap significantly, K-Means may struggle to identify meaningful groups.

The algorithm is also sensitive to outliers. A single extreme observation can pull a centroid away from the true centre of a cluster, affecting the final results.

Another important consideration is feature scaling. Since K-Means relies on distance to measure similarity, variables measured on larger scales can dominate those measured on smaller scales. Standardising or normalising the data before clustering is therefore an important preprocessing step.

Finally, K-Means requires you to choose the number of clusters in advance. While methods such as the Elbow Method and Silhouette Score can provide useful guidance, they don’t replace business understanding and domain expertise.

🎤 Interview Q: What are the main limitations of K-Means?

A: K-Means requires the number of clusters to be specified beforehand, is sensitive to outliers and feature scaling, and performs best when clusters are compact and well separated. It may not perform well on datasets with irregularly shaped or overlapping clusters.

Key Takeaways

Let’s recap the key concepts from this article:

• K-Means is an unsupervised learning algorithm that groups similar observations into clusters without requiring labelled data.

• The objective of K-Means is to maximise similarity within a cluster while keeping different clusters as distinct as possible.

• The algorithm works iteratively by assigning observations to the nearest centroid and updating the centroids until the clusters stabilise.

• Choosing the right value of K requires both data and domain knowledge. Techniques such as the Elbow Method, Silhouette Score, and Gap Statistic provide useful guidance, but the final decision should align with the business problem.

• K-Means is simple, fast, and scalable, making it one of the most widely used clustering algorithms in industry.

• Like any algorithm, K-Means has limitations. It performs best on compact, well-separated clusters and is sensitive to feature scaling, outliers, and the choice of K.

The most important lesson to remember is this:

K-Means doesn’t predict outcomes—it discovers patterns. Those patterns become valuable only when they’re interpreted in the context of the business problem you’re trying to solve.

What’s Next?

K-Means is an excellent introduction to clustering and remains one of the most widely used unsupervised learning algorithms. However, it isn’t always the best choice.

What happens if your data contains outliers? Or if the clusters aren’t neat, circular groups? What if you don’t know how many clusters exist in the first place?

These are situations where K-Means can struggle.

In the next issue of The Practical Data Scientist, we’ll explore DBSCAN, a density-based clustering algorithm that takes a very different approach. Unlike K-Means, DBSCAN doesn’t require you to specify the number of clusters in advance, can identify clusters of arbitrary shapes, and naturally detects outliers.

By the end of the next article, you’ll understand not only how DBSCAN works, but also when to choose it over K-Means in real-world data science projects.

Keep building, keep learning—wishing you the best in your data journey.

Imagine two data scientists are given the same dataset and asked to predict customer churn.

The first data scientist spends hours tuning hyperparameters and experimenting with increasingly sophisticated machine learning models.

The second spends their time understanding the data.

Instead of using raw variables, they create features such as:

• Days since the customer’s last purchase

• Average monthly spending

• Number of purchases in the last 30 days

• Change in spending over time

• Customer tenure

Surprisingly, the second data scientist achieves better results using a much simpler model.

How is that possible?

Because great machine learning models aren’t built with algorithms alone. They’re built with meaningful representations of the underlying problem.

This is the essence of feature engineering.

Feature engineering is often described as the process of transforming raw data into features that machine learning models can learn from. While that’s true, it doesn’t fully capture its importance.

At its core, feature engineering is about helping models understand what matters.

Consider the difference between these two inputs:

Purchase Date: January 15, 2026

and

Days Since Last Purchase: 7

The second feature immediately provides meaningful information about customer behavior. Similarly, a customer’s date of birth may be less informative than their age, and a list of transaction timestamps may be less useful than their average monthly spending or purchase frequency.

The goal isn’t simply to create more features—it’s to create better ones.

In many real-world machine learning projects, improvements in feature engineering can produce larger performance gains than switching from one algorithm to another. In fact, it’s not uncommon for a well-engineered Logistic Regression model to outperform a poorly designed gradient boosting model.

This is why feature engineering remains one of the most valuable skills for data scientists. It sits at the intersection of domain knowledge, business understanding, and machine learning.

In this article, we’ll explore how data scientists transform raw data into meaningful signals, discuss common feature engineering techniques, examine real-world examples, and learn how better features often lead to better predictions.

By the end of this issue, you’ll understand why great models are frequently built long before the training process even begins.

What Is Feature Engineering?

Before we explore specific techniques, let’s answer an important question:

What exactly is feature engineering?

Feature engineering is the process of transforming raw data into meaningful inputs that help machine learning models identify patterns and make better predictions.

Simply put, it’s the art of representing a problem in a way that a model can understand.

Consider the following examples:

Notice that none of the engineered features introduced any new information. Instead, they transformed the existing data into representations that are often more meaningful and informative.

This is what makes feature engineering so powerful.

Machine learning models don’t understand business context. They don’t know that customers who haven’t made a purchase in six months may be more likely to churn or that spending patterns often matter more than individual transactions.

Feature engineering helps us incorporate that understanding into the data.

Why Raw Data Isn’t Always Enough

Imagine we’re building a model to predict customer churn.

Suppose our dataset contains the following variables:

• Customer ID

• Date of Birth

• Purchase Dates

• Transaction Amounts

• Account Creation Date

These variables may be useful, but they don’t necessarily capture customer behavior.

By engineering additional features, we can provide the model with much more meaningful information:

• Customer Age

• Customer Tenure

• Average Monthly Spending

• Days Since Last Purchase

• Purchase Frequency

• Change in Spending Over Time

These features tell a richer story about the customer and often make it easier for the model to learn meaningful patterns.

Feature Engineering Is More Than Data Transformation

A common misconception is that feature engineering is simply applying techniques such as one-hot encoding or scaling numerical variables.

While those techniques are important, feature engineering begins much earlier.

It starts by asking questions such as:

• What behavior are we trying to capture?

• Which variables are likely to influence the outcome?

• How can we represent this business problem more effectively?

For example, when predicting whether a customer will churn, a customer’s age may matter less than how frequently they use the product. Similarly, when predicting fraudulent transactions, the transaction amount may be less informative than whether it is unusually large for that particular customer.

The best features are often those that reflect an understanding of both the business problem and the data itself.

Important Insight

Feature engineering isn’t about creating more variables.

It’s about creating variables that better represent the patterns we want our models to learn.

In many cases, improving the representation of the problem can have a greater impact on model performance than changing the algorithm itself.

Interview Pro Tip

🎤 Interview Q: What is feature engineering, and why is it important?

A: Feature engineering is the process of transforming raw data into meaningful features that help machine learning models learn more effectively. It is important because better feature representations often improve model performance, interpretability, and generalization more than simply choosing a more sophisticated algorithm.

Why Feature Engineering Matters

It’s easy to assume that better predictions come from better algorithms.

When a model performs poorly, our first instinct is often to try something more sophisticated.

“Let’s use XGBoost instead of Logistic Regression.”

“Maybe we should tune the hyperparameters.”

“Perhaps we need a larger neural network.”

Sometimes, those changes help. But surprisingly often, the biggest improvements come from better features rather than better models.

This is because machine learning models can only learn from the information we provide. If the features fail to capture meaningful patterns in the data, even the most sophisticated algorithm will struggle to produce accurate predictions.

Consider the following example.

Suppose we’re building a model to predict customer churn.

One approach might be to provide the model with raw transactional data such as:

• Account creation date

• Purchase timestamps

• Individual transaction amounts

• Website visit logs

A better approach might be to engineer features that summarize customer behavior, such as:

• Customer tenure

• Days since the last purchase

• Average monthly spending

• Number of website visits in the last 30 days

• Change in spending over time

Although both models are built using the same underlying data, the second set of features provides much richer information about customer behavior.

The algorithm hasn’t changed. The representation of the problem has.

Feature Engineering Helps Models Learn Patterns

Machine learning models are excellent at identifying patterns, but they don’t understand business context.

For example, consider the following feature:

Transaction Timestamp: 2026-07-15 14:35:26

On its own, this value isn’t particularly meaningful.

However, we can transform it into features such as:

• Weekend vs. Weekday

• Month of the Year

• Days Since Last Purchase

• Number of Transactions This Week

• Time Since Last Login

These engineered features provide signals that are often much more informative than the original variable.

Similarly, suppose we’re predicting whether a loan application should be approved.

Instead of using only:

• Annual Income

• Monthly Expenses

we might create features such as:

• Debt-to-Income Ratio

• Savings Rate

• Percentage of Income Spent on Housing

These engineered features may better capture a customer’s financial behavior than the raw variables themselves.

One of the most valuable lessons in machine learning is that sophisticated models cannot compensate for poorly designed features. It is not uncommon for a well-engineered Logistic Regression model to outperform a poorly designed gradient boosting model.

This is why experienced data scientists spend a significant amount of their time understanding the data and designing meaningful features before training a model.

Feature engineering isn’t an afterthought—it’s often one of the most important parts of the entire machine learning pipeline.

Machine learning models learn from the features we give them. If those features fail to represent the underlying problem effectively, no amount of hyperparameter tuning or model complexity can fully compensate for that limitation.

Great models aren’t built by asking:

“Which algorithm should I use?”

They’re built by first asking:

“Am I teaching the model what actually matters?”

Better features don’t simply improve predictions—they help models learn the right patterns in the data.

Interview Pro Tip

🎤 Interview Q: Can feature engineering improve model performance more than changing the algorithm?

A: Yes. Feature engineering often has a significant impact on model performance because it determines how well the underlying problem is represented. In many cases, meaningful features can provide larger improvements than switching to a more sophisticated algorithm or tuning additional hyperparameters.

Subscribe now

Common Feature Engineering Techniques

Feature engineering isn’t a single technique—it’s a collection of approaches for representing data more effectively.

The best data scientists don’t simply ask, “What variables do I have?” They ask:

“What information am I trying to help the model learn?”

Sometimes we’re interested in understanding how often something happens. Other times, we’re trying to capture changes over time, identify relationships between variables, or provide additional business context.

Let’s look at some of the most commonly used feature engineering techniques.

Numerical Features

Numerical variables often benefit from transformations that make patterns easier for models to learn.

For example, we might transform:

• Annual salary into salary brackets

• Transaction amounts using logarithmic transformations

• Customer age into age groups

• Numerical variables through standardization or normalization

These transformations can make the data more interpretable and improve model performance.

Categorical Features

Machine learning models generally cannot work directly with categorical values such as:

• Product category

• Payment method

• Country

• Customer segment

These variables are typically transformed using techniques such as:

• One-hot encoding

• Label encoding

• Frequency encoding

• Target encoding

Choosing the appropriate encoding strategy often depends on both the model being used and the characteristics of the data.

Date and Time Features

Date and time variables frequently contain valuable information that isn’t immediately apparent.

For example, a purchase timestamp can be transformed into:

• Day of the week

• Month of the year

• Weekend vs. weekday

• Days since the last purchase

• Customer tenure

• Time since the last login

These engineered features often capture behavioral patterns more effectively than the original timestamp itself.

Aggregated Features

Some of the most valuable features summarize behavior across multiple observations. For example, rather than using individual transactions, we might create features such as:

• Average monthly spending

• Number of purchases in the last 30 days

• Average basket size

• Customer lifetime value

• Number of support tickets submitted

Aggregation allows us to transform large amounts of raw data into meaningful summaries that models can learn from more effectively.

Interaction Features

Sometimes, individual variables aren’t particularly informative on their own, but become highly predictive when combined.

For example:

• Debt-to-income ratio combines income and expenses.

• Revenue per customer combines total revenue and customer count.

• Conversion rate combines purchases and website visits.

These interaction features often capture relationships that would otherwise remain hidden in the raw data.

Feature engineering is rarely about applying a single technique. Instead, it involves combining domain knowledge, business understanding, and statistical intuition to create features that better represent the underlying problem.

When experienced data scientists look at a dataset, they don’t simply see columns and rows—they think about the behaviors, relationships, and patterns those variables might represent.

Interview Pro Tip

🎤 Interview Q: What are some common feature engineering techniques?

A: Common feature engineering techniques include scaling and transforming numerical variables, encoding categorical variables, extracting information from dates and timestamps, creating aggregated features, and engineering interaction features that capture relationships between variables. The appropriate technique depends on both the problem being solved and the characteristics of the data.

Feature Engineering in the Real World

Feature engineering rarely begins with the question:

“What transformations should I apply?”

Instead, it begins with a much more important question:

“What behavior am I trying to capture?”

Experienced data scientists don’t simply look at datasets—they look for signals that represent the underlying business problem.

Let’s consider a few examples.

Predicting Customer Churn

Suppose we’re building a model to predict whether customers are likely to stop using a product. Our dataset contains the following variables:

• Customer ID

• Purchase history

• Transaction amounts

• Website activity

• Account creation date

A beginner might use these variables exactly as they appear in the dataset.

An experienced data scientist might instead ask:

“What behaviors are typically associated with churn?”

Possible answers include:

• Customers are purchasing less frequently.

• Customers haven’t logged in recently.

• Customers are spending less than they used to.

• Customers are becoming less engaged over time.

That way of thinking naturally leads to features such as:

• Days since the last purchase

• Customer tenure

• Average monthly spending

• Change in spending over time

• Number of website visits in the last 30 days

Notice that none of these features existed in the original dataset. They were created by combining domain knowledge with an understanding of customer behavior.

Detecting Fraudulent Transactions

Feature engineering is equally important in fraud detection.

Suppose we have the following information:

• Transaction amount

• Transaction timestamp

• Customer location

• Payment method

These variables may be useful, but they don’t necessarily tell us whether a transaction is suspicious.

Instead, we might create features such as:

• Is the transaction amount unusually large for this customer?

• How many transactions occurred in the last hour?

• Is this transaction occurring in a new location?

• How much time has passed since the previous transaction?

These features provide much stronger signals about potentially fraudulent behavior than the raw variables themselves.

Building Recommendation Systems

Consider a product recommendation model.

Rather than using individual purchases alone, we might engineer features such as:

• Purchase frequency

• Average order value

• Categories purchased most frequently

• Products viewed recently

• Time since the customer’s last interaction

These features help us better understand customer preferences and purchasing patterns.

Thinking Like a Data Scientist

Notice that feature engineering looks very different across these examples.

That’s because there is no universal recipe for creating good features.

The best features are almost always driven by the business problem we’re trying to solve.

Experienced data scientists don’t begin by asking:

“Which algorithm should I use?”

They begin by asking:

“What information would help me make this prediction?”

That question often leads to better features—and ultimately, better models.

🎤 Interview Q: How do you approach feature engineering for a new machine learning problem?

A: I begin by understanding the business problem and identifying the behaviors or patterns that are likely to influence the outcome I’m trying to predict. I then use domain knowledge to engineer features that better represent those patterns before selecting and training a model.

Common Mistakes to Avoid

Feature engineering can significantly improve model performance, but it can also introduce problems when applied without careful thought.

Ironically, some of the most common mistakes aren’t technical—they stem from focusing on the data instead of the problem we’re trying to solve.

Creating Features Without Understanding the Business Problem

One of the biggest mistakes data scientists make is engineering features simply because they seem useful.

Feature engineering should always begin with the question:

“What information would help explain or predict the outcome?”

Without understanding the business problem, it’s easy to create features that add complexity without adding meaningful information.

For example, when predicting customer churn, customer engagement may be far more informative than demographic information. Similarly, in fraud detection, unusual behavior may matter more than transaction amounts alone.

The most useful features are rarely the most complicated—they’re often the most meaningful.

Assuming More Features Are Always Better

More features do not necessarily lead to better models.

Adding hundreds of variables can introduce noise, increase computational complexity, and sometimes even reduce model performance.

A smaller set of meaningful features will often outperform a large collection of poorly designed ones.

The goal of feature engineering isn’t to maximize the number of features—it’s to maximize the amount of useful information they provide.

Ignoring Data Leakage

Data leakage occurs when information that would not be available at prediction time unintentionally makes its way into the training data.

For example, suppose we’re predicting whether a customer will churn next month.

Features such as:

• Purchases made after the prediction date

• Future account activity

• Information collected after the customer has already churned

would all introduce information that the model would never have access to in production.

This often leads to unrealistically high model performance during development and disappointing results once the model is deployed.

Forgetting That Features Must Generalize

A feature may work exceptionally well on one dataset but fail when applied to new data.

Good features should capture meaningful and consistent patterns rather than accidental relationships that exist only in the training data.

This is one reason why domain knowledge remains so valuable in machine learning. Understanding why a feature should be predictive is often just as important as observing that it is predictive.

Feature engineering isn’t simply about improving performance—it’s about creating features that continue to be useful when the model encounters new and unseen data.

Feature engineering is often described as both an art and a science, and for good reason. The best features are rarely created by applying a checklist of transformations. They emerge from understanding the business problem, asking the right questions, and carefully considering what information is truly useful for making predictions.

Interview Pro Tip

🎤 Interview Q: What are some common mistakes in feature engineering?”

A: Common mistakes include creating features without understanding the business problem, assuming that more features always improve performance, introducing data leakage, and engineering features that do not generalize well to new data. Effective feature engineering requires both domain knowledge and careful validation.

Subscribe now

Final Takeaway

Let’s return to the question that started this article.

What makes a great machine learning model?

It’s tempting to believe that the answer lies in more sophisticated algorithms, larger datasets, or extensive hyperparameter tuning.

While all of those things matter, they aren’t where great models usually begin.

Great models begin with a deep understanding of the problem we’re trying to solve.

Feature engineering is much more than transforming variables or applying preprocessing techniques. It’s the process of helping machine learning models understand what matters by representing the underlying problem more effectively.

Throughout this article, we’ve seen that:

• Better features often matter more than better algorithms.

• Meaningful features are driven by business understanding and domain knowledge.

• Feature engineering is about representing behaviors and patterns—not simply creating more variables.

• The best features are often those that generalize well to new and unseen data.

Perhaps the most important lesson is that experienced data scientists don’t approach feature engineering by asking:

“What transformations should I apply?”

They begin by asking:

“What information would help me make this prediction?”

That shift in thinking is what separates feature engineering from data preprocessing.

Machine learning models don’t understand customers, products, or businesses. They understand patterns in data. Feature engineering is what allows us to translate real-world problems into representations that models can learn from effectively.

Great models aren’t simply trained.

They’re engineered.

And that process often begins long before the first model is ever trained.

What’s Next?

So far, we’ve focused primarily on supervised machine learning—from building models and tuning hyperparameters to regularization and feature engineering.

But not every data science problem comes with labeled data or a clearly defined outcome variable.

Sometimes, the goal isn’t to predict what will happen next. Instead, we’re trying to answer questions such as:

• Are there natural groups hidden within our data?

• Which customers behave similarly?

• How can we reduce complexity without losing important information?

• What patterns exist that we haven’t discovered yet?

This is where unsupervised learning comes in.

In the next issue of The Practical Data Scientist, we’ll begin a new series exploring some of the most widely used unsupervised learning techniques and the business problems they help solve.

We’ll start with one of the most popular clustering algorithms in machine learning—K-Means Clustering—and learn how data scientists use it to uncover meaningful patterns hidden within their data.

Because sometimes, the most valuable insights aren’t the ones we predict—they’re the ones we discover.

Keep building, keep learning—wishing you the best in your data journey.

Imagine you’re a data scientist at a global retail company.

To improve profitability, leadership decides to increase prices by 5% in Canada while keeping prices unchanged in the United States.

A month later, the results are in.

Sales in Canada have declined.

Leadership immediately asks:

“Did the price increase reduce sales?”

At first glance, answering this question seems straightforward.

Simply compare sales before and after the price increase.

If sales decreased, the pricing strategy must have been responsible.

Or was it?

What if consumer spending declined in both countries because of broader economic conditions?

What if inflation reduced discretionary spending during the same period?

Or what if demand naturally softened after the holiday season?

These factors can influence sales regardless of whether prices changed.

Simply comparing sales in Canada before and after the intervention doesn’t tell us whether the pricing strategy caused the decline.

To answer that question, we need to separate the impact of the price increase from all the other factors affecting sales over time.

This is where Difference-in-Differences (DiD) comes in.

Difference-in-Differences is one of the most widely used causal inference techniques for estimating the impact of a policy, business decision, or intervention when randomized experiments aren’t possible. Rather than looking only at how outcomes change over time, it compares those changes with a similar group that was not exposed to the intervention.

In the previous article, we explored how Propensity Score Matching creates fair comparisons between similar individuals. In this issue, we’ll tackle a different kind of problem—one where we have observations before and after an intervention.

Using the pricing strategy example throughout this article, we’ll see how Difference-in-Differences helps answer one of the most common questions in business analytics:

Did the price increase actually reduce sales, or would sales have declined even without it?

By the end of this issue, you’ll understand the intuition behind Difference-in-Differences, how it works, the assumptions it relies on, and when it should be used.

Why Before-and-After Comparisons Can Be Misleading

Let’s return to our pricing example.

Suppose the company increased prices in Canada on January 1.

At the end of January, the analytics team reports that monthly sales have declined by 8% compared to December.

Leadership immediately concludes:

“The price increase reduced sales by 8%.”

But can we really make that claim?

Not necessarily.

The problem is that many factors can influence sales over time.

For example:

• Changes in the economy

• Seasonal shopping patterns

• Competitor promotions

• Consumer confidence

• Inflation

Any one of these factors—or several acting together—could have affected sales during the same period. As a result, simply comparing sales before and after the price increase mixes together two effects:

• The impact of the pricing strategy

• All the other factors that changed over time

Without separating these effects, we cannot confidently determine how much of the decline was actually caused by the price increase.

Why This Matters

Imagine that consumer spending declined across North America because of worsening economic conditions.

Even if the company had not increased prices in Canada, sales might still have fallen.

In that case, attributing the entire 8% decline to the pricing strategy would overestimate its impact.

A before-and-after comparison tells us that something changed. It does not tell us why it changed.

To estimate the true impact of an intervention, we need a way to account for changes that would have occurred even if the intervention had never happened.

This naturally raises the next question:

How can we separate the effect of the price increase from all the other factors influencing sales?

Difference-in-Differences provides one answer.

🎤 Interview Q: Why isn’t a before-and-after comparison sufficient for measuring causal effects?

A: Because many external factors can change over time alongside the intervention. A before-and-after comparison cannot distinguish the effect of the intervention from other events that occurred during the same period, making it difficult to estimate the true causal effect.

The Core Idea Behind Difference-in-Differences

We’ve seen that a simple before-and-after comparison isn’t enough to estimate the impact of the price increase.

So how can we do better?

The key idea behind Difference-in-Differences (DiD) is surprisingly simple.

Instead of looking only at how sales changed in Canada, we compare that change with what happened in a similar country where prices did not change.

In our example:

• Canada is the treatment group because prices were increased.

• The United States is the control group because prices remained unchanged.

Now, instead of asking:

“Did sales change after the price increase?”

we ask:

“Did sales change more in Canada than they did in the United States?”

This distinction is important.

If sales declined by a similar amount in both countries, the decline was likely driven by broader economic conditions rather than the pricing strategy.

However, if sales fell significantly more in Canada than in the United States, the additional decline provides evidence that the price increase contributed to the change.

In other words, the United States helps us estimate what might have happened in Canada if prices had never changed.

Difference-in-Differences doesn’t compare sales levels. It compares changes in sales.

By comparing how outcomes evolve over time in both the treatment and control groups, it removes many of the external factors that affect both groups equally.

This gives us a much clearer picture of the intervention’s true impact.

You can think of Difference-in-Differences as answering the question:

“How much more (or less) did the treatment group change compared with what we would have expected based on the control group?”

That simple idea is what makes Difference-in-Differences one of the most widely used causal inference methods in business, economics, and public policy.

🎤 Interview Q: What is the intuition behind Difference-in-Differences?

A: Difference-in-Differences estimates the causal effect of an intervention by comparing how outcomes change over time in a treatment group relative to a control group. By focusing on the difference in changes rather than the difference in outcomes, it helps account for external factors that affect both groups.

Subscribe now

How Difference-in-Differences Works

Now that we understand the intuition behind Difference-in-Differences, let’s calculate the treatment effect using our pricing example.

The figure below summarizes the sales before and after the price increase in both countries.

The first step is to calculate how much sales changed in each country.

• Canada: Sales decreased by $0.8 million.

• United States: Sales decreased by $0.4 million.

Notice that sales declined in both countries.

This suggests that part of the decline was likely caused by broader economic conditions rather than the pricing strategy itself.

To estimate the impact of the price increase, we subtract the change observed in the control group from the change observed in the treatment group:

Difference-in-Differences Estimate = Treatment Change − Control Change

For our example:

• Treatment change = −$0.8M

• Control change = −$0.4M

Estimated treatment effect:

−$0.8M − (−$0.4M) = −$0.4M

In other words, after accounting for the broader decline observed in the United States, the price increase is estimated to have reduced monthly sales in Canada by an additional $0.4 million.

Difference-in-Differences doesn’t compare sales levels. It compares how much the treatment group changed relative to the control group.

By removing changes that affected both groups, it provides a more credible estimate of the intervention’s impact.

🎤 Interview Q: Why do we subtract the change in the control group?

A: The control group captures changes that would likely have occurred even without the intervention. Subtracting those changes helps isolate the portion of the outcome that is more likely attributable to the treatment itself.

The Parallel Trends Assumption

Difference-in-Differences is a powerful technique, but it relies on one important assumption.

This assumption is known as the Parallel Trends Assumption.

Simply put, it states that if the price increase had never happened, sales in Canada and the United States would have followed similar trends over time.

Notice that this assumption does not require the two countries to have identical sales.

Canada may consistently generate lower sales than the United States because of differences in population, market size, or consumer behavior.

That’s perfectly acceptable.

What matters is that the pattern of change over time would have been similar in both countries if the intervention had not occurred.

Why This Matters

Imagine that sales in Canada had already been declining much faster than sales in the United States before the price increase.

If we later observe a larger decline in Canada, we cannot confidently attribute it to the pricing strategy.

The decline may simply reflect an existing trend that began long before the intervention.

In that case, the Difference-in-Differences estimate would be biased.

How Do We Check This?

Although we can never prove the Parallel Trends Assumption with certainty, we can look for evidence that supports it.

One common approach is to examine historical data before the intervention.

If sales in the treatment and control groups move in a similar pattern over time before the price change, the assumption becomes more plausible.

This is why Difference-in-Differences studies often include a graph showing the trends for both groups before the intervention.

Difference-in-Differences doesn’t assume that the treatment and control groups are identical. It assumes that without the intervention, they would have changed in similar ways over time.

That distinction is what makes the Parallel Trends Assumption the foundation of the entire method.

🎤 Interview Q: What is the Parallel Trends Assumption in Difference-in-Differences?

A: It assumes that, in the absence of the treatment, the treatment and control groups would have followed similar trends over time. This allows the control group to serve as a credible estimate of what would have happened to the treatment group had the intervention never occurred.

Limitations of Difference-in-Differences

Difference-in-Differences is a powerful technique, but like every causal inference method, it has limitations.

Understanding these limitations is just as important as knowing how to apply the method.

The Control Group Matters

Difference-in-Differences relies on having a control group that provides a reasonable estimate of what would have happened to the treatment group in the absence of the intervention.

If the control group differs substantially from the treatment group, the estimated treatment effect may be biased.

External Events Can Affect the Results

The method assumes that no major event affects only one of the groups during the study period.

For example, suppose a new competitor entered the Canadian market shortly after the price increase while the U.S. market remained unchanged.

Any decline in Canadian sales could then be caused by the competitor, the price increase, or both.

Difference-in-Differences alone cannot distinguish between these effects.

Timing Is Important

Choosing the right time window is critical.

If the “after” period is too short, the full impact of the intervention may not yet be visible.

If it is too long, additional events may occur that make it harder to isolate the treatment effect.

Randomized Experiments Are Still Preferred

Whenever randomized experiments are practical and ethical, they remain the gold standard for estimating causal effects.

Difference-in-Differences is most valuable when randomization is impossible or impractical and suitable observational data are available.

Important Insight

Difference-in-Differences doesn’t eliminate every source of bias.

Instead, it helps account for changes that affect both the treatment and control groups over time.

The quality of the estimate depends on selecting an appropriate control group, validating the Parallel Trends Assumption, and carefully considering other events that may have influenced the results.

🎤 Interview Q: What are the limitations of Difference-in-Differences?

A: Difference-in-Differences relies on the Parallel Trends Assumption and requires an appropriate control group. It can produce biased estimates if external events affect only one group, if the groups follow different trends before the intervention, or if the timing of the analysis is not chosen carefully.

Final Takeaway

Let’s return to the question that started this article.

Did the price increase reduce sales in Canada?

At first glance, the answer seemed simple.

Sales declined after the prices increased.

But as we’ve seen, that observation alone doesn’t establish causality.

Sales can change for many reasons—economic conditions, seasonality, competitor actions, and other external factors. A simple before-and-after comparison cannot separate the effect of the pricing strategy from everything else happening at the same time.

That’s where Difference-in-Differences comes in.

Rather than asking whether sales changed, it asks a more meaningful question:

Did sales change more in the treatment group than they did in a comparable control group?

By comparing changes over time instead of raw outcomes, Difference-in-Differences helps account for common trends affecting both groups and provides a more credible estimate of the intervention’s impact.

Like any causal inference technique, it has assumptions and limitations. But when those assumptions are reasonable, Difference-in-Differences becomes one of the most practical and widely used tools for measuring the real-world impact of business decisions.

Key Takeaways

• Before-and-after comparisons alone cannot establish causality.

• Difference-in-Differences compares changes over time, not just outcomes.

• A control group helps account for external factors that affect both groups.

• The Parallel Trends Assumption is essential for producing credible estimates.

Difference-in-Differences doesn’t ask:

“Did the outcome change?”

It asks:

“How much more (or less) did the outcome change because of the intervention?”

That shift in thinking is what makes Difference-in-Differences such a valuable tool for business analytics, economics, public policy, and causal inference.

🎤 Interview Q: When would you use Difference-in-Differences?

A: I would use Difference-in-Differences when I have observations before and after an intervention for both a treatment group and a comparable control group. By comparing how outcomes change over time between the two groups, it helps estimate the causal effect of the intervention while accounting for common trends.

What’s Next?

So far in this causal inference series, we’ve explored two powerful approaches for estimating causal effects using observational data.

In the previous article, we learned how Propensity Score Matching creates fair comparisons between similar individuals. In this issue, we saw how Difference-in-Differences compares changes over time to isolate the impact of an intervention.

But what happens when we can’t find comparable groups, or when the assumptions behind these methods don’t hold?

In the next issue of The Practical Data Scientist, we’ll explore another approach to answering difficult causal questions—one that helps estimate causal effects even when traditional comparisons become challenging.

As always, we’ll work through a real-world business case study, focusing not just on the mathematics behind the method, but on the intuition that helps you decide when and why to use it.

By the end of this series, you’ll have a practical toolkit for choosing the right causal inference technique based on the business problem you’re trying to solve—not just the data you have.

Keep building, keep learning—wishing you the best in your data journey.

Introduction

Imagine you’re a data scientist at an e-commerce company.

The marketing team launches an email campaign promoting a new loyalty program. Over 100,000 customers receive the campaign, while the remaining customers do not.

A month later, the results look promising.

Customers who received the email spent 18% more, on average, than those who didn’t.

Leadership is excited and asks a simple question:

“Great! So how much did the campaign increase sales?”

At first glance, the answer seems obvious.

The campaign group spent more money, so the campaign must have worked.

Or did it?

What if the marketing team intentionally targeted customers who were already more likely to make a purchase?

Perhaps they selected customers who had:

• Purchased recently

• Spent more in the past

• Opened previous marketing emails

• Shown higher engagement with the brand

If that’s the case, these customers may have spent more even if they had never received the campaign.

Suddenly, the question becomes much harder.

The challenge isn’t measuring what happened.

It’s estimating what would have happened if those same customers had not received the campaign.

That is the central challenge of causal inference.

In the previous issue, we introduced the concepts of correlation and causation and discussed why observational data alone is often insufficient to establish cause-and-effect relationships.

In this article, we’ll build on that foundation by exploring one of the most widely used causal inference techniques: Propensity Score Matching (PSM).

Using the marketing campaign example throughout the article, we’ll see how PSM helps us create fair comparisons between treated and untreated customers, allowing us to estimate the true impact of an intervention when randomized experiments aren’t possible.

By the end of this issue, you’ll understand not only how Propensity Score Matching works, but also why it has become one of the most important tools in the causal inference toolbox for data scientists.

The Fundamental Problem of Causal Inference

Let’s return to our marketing campaign.

Leadership wants to know:

“How much did the campaign increase sales?”

To answer that question, we would ideally compare two scenarios for the same customer:

Scenario 1: The customer receives the marketing campaign.

Scenario 2: The same customer does not receive the marketing campaign.

The difference in spending between these two scenarios would tell us the true effect of the campaign. Unfortunately, there is one problem.

We can only observe one of these outcomes. For every customer, only one reality exists.

If a customer received the campaign, we observe how much they spent after receiving it. We can never observe how much that same customer would have spent during the same period had they not received the campaign.

This unobserved outcome is called the counterfactual.

The counterfactual represents the outcome that would have occurred under the alternative scenario. For example, suppose a customer who received the campaign spent $150.

The question we really want to answer is:

How much would this same customer have spent if they had never received the campaign?

If the answer were $130, we could conclude that the campaign increased spending by $20.

But because we can never observe both realities for the same customer, the true treatment effect is fundamentally unknowable at the individual level.

This challenge is known as the Fundamental Problem of Causal Inference.

Why This Matters

If we can’t compare a customer with themselves, the next best option is to compare them with someone who is as similar as possible.

That simple idea forms the foundation of many causal inference techniques, including Propensity Score Matching.

Rather than asking:

“What happened to this customer?”

we ask:

“What would likely have happened to a very similar customer who did not receive the treatment?”

Causal inference is fundamentally about estimating the missing outcome—the counterfactual.

Since we can never observe both realities for the same individual, we rely on carefully designed methods to approximate what would have happened in the absence of the treatment.

🎤 Interview Q: What is the fundamental problem of causal inference?

A: For any individual, we can observe either the treated outcome or the untreated outcome, but never both. Because the counterfactual is unobservable, causal inference methods aim to estimate it using comparable individuals or carefully designed experiments.

Why Comparing Everyone Doesn’t Work

At this point, you might be wondering:

Why can’t we simply compare customers who received the campaign with those who didn’t?

Suppose we calculate the average spending for the two groups.

• Average Spending of People who Received Campaign: $126

• Average Spending of People who Did Not Receive Campaign: $102

At first glance, the conclusion seems straightforward. The campaign group spent $24 more.

Did the campaign increase spending by $24?

Not necessarily.

The problem is that the two groups may not have been comparable to begin with.

Customers who received the campaign may have already been more likely to make a purchase than those who did not. If that’s true, part—or even all—of the difference in spending could simply reflect the characteristics of the customers rather than the impact of the campaign itself.

In other words, the treatment and control groups started from different baselines.

As a result, simply comparing the two groups mixes together two effects:

• The effect of the campaign itself

• The pre-existing differences between the customers

This is known as selection bias.

The treatment group was selected, rather than randomly assigned, making it difficult to isolate the true effect of the intervention.

Why This Matters

Imagine comparing the performance of professional athletes with that of casual weekend players after giving only the professionals access to a new training program.

If the professionals perform better, was it because of the training program?

Or because they were already better athletes?

The comparison isn’t fair because the two groups were fundamentally different from the start.

The same principle applies to marketing campaigns, loyalty programs, pricing experiments, and countless other business decisions.

To estimate a causal effect, we need treatment and control groups that are as similar as possible before the treatment occurs. Only then can differences in outcomes be more confidently attributed to the intervention itself.

This naturally leads to the question:

How can we create fair comparisons when treatments weren’t assigned randomly?

One of the most widely used answers is Propensity Score Matching.

🎤 Interview Q: Why can’t we directly compare treatment and control groups in observational data?

A: Because the treatment and control groups may differ in important ways before the treatment occurs. These pre-existing differences introduce selection bias, making it difficult to determine whether the observed outcome was caused by the treatment or by the characteristics of the individuals themselves.

Enter Propensity Score Matching

We’ve established two important ideas:

• We cannot compare a customer with themselves because the counterfactual is unobservable.

• We cannot simply compare all treated customers with all untreated customers because the groups may be fundamentally different.

So what can we do instead?

The answer is surprisingly intuitive.

Rather than comparing every treated customer with every untreated customer, we compare each treated customer with an untreated customer who looked similar before the treatment occurred.

This is the central idea behind Propensity Score Matching (PSM).

Instead of asking:

“How did customers who received the campaign perform compared with everyone else?”

PSM asks:

“How did customers who received the campaign perform compared with similar customers who did not receive it?”

Returning to our marketing campaign example, imagine we identify a customer who:

• Is 35 years old

• Has been a customer for three years

• Spends about $120 per month

• Shops once or twice each month

If this customer received the campaign, we would look for another customer with nearly identical characteristics who did not receive the campaign.

By comparing these two customers, we create a much fairer comparison than comparing the entire treatment group with the entire control group.

Of course, finding a perfect match for every customer is rarely possible.

Instead, Propensity Score Matching uses a statistical approach to identify customers who have a similar likelihood of receiving the treatment based on their observed characteristics.

The goal is not to make the customers identical.

The goal is to make the treatment and control groups as comparable as possible before the intervention, so that differences observed afterward are more likely to reflect the effect of the treatment rather than pre-existing differences.

Why This Matters

Propensity Score Matching doesn’t eliminate bias completely.

However, it substantially reduces selection bias by ensuring that treated customers are compared with untreated customers who had similar characteristics before the treatment.

This produces a much more credible estimate of the treatment effect than a simple comparison of group averages.

You can think of Propensity Score Matching as creating apples-to-apples comparisons.

Instead of comparing everyone with everyone else, it compares people who were similar before the treatment, allowing us to estimate what might have happened had the treated individuals never received the intervention.

🎤 Interview Q: What is the intuition behind Propensity Score Matching?

A: Propensity Score Matching reduces selection bias by matching treated individuals with untreated individuals who had similar characteristics before the treatment. This creates fairer comparisons and helps estimate the causal effect of the intervention using observational data.

How Propensity Score Matching Works

We’ve seen that comparing all treated customers with all untreated customers can lead to biased conclusions.

Instead, Propensity Score Matching compares customers who were similar before the treatment.

But how do we determine who is “similar”?

Matching customers on every characteristic individually quickly becomes impractical. A customer may differ in age, purchase history, income, geography, browsing behavior, and dozens of other attributes.

Rather than comparing customers across every variable separately, Propensity Score Matching summarizes these characteristics into a single value called the propensity score.

The propensity score represents the probability that a customer would receive the treatment based on their observed characteristics.

For our marketing campaign, we might estimate each customer’s propensity score using variables such as:

• Age

• Purchase history

• Average monthly spending

• Website activity

• Loyalty status

Customers with similar propensity scores had a similar likelihood of receiving the campaign, even if one actually received it and the other did not.

Step 1: Estimate the Propensity Score

The first step is to build a model that predicts whether a customer would receive the treatment.

In practice, this is often done using Logistic Regression, although other models can also be used.

The output is a probability between 0 and 1 for every customer.

For example:

Step 2: Match Similar Customers

Next, each treated customer is matched with one or more untreated customers who have a similar propensity score.

For example:

• Customer A (0.82) → Customer B (0.80)

• Customer D (0.25) → Customer C (0.27)

Because these customers had similar probabilities of receiving the campaign, they form much fairer comparison pairs than randomly selected customers.

Step 3: Compare Outcomes

Once the matches are created, we compare the outcomes within each matched pair.

If the treated customer consistently spends more than their matched counterpart, we have stronger evidence that the campaign itself contributed to the increase in spending.

Repeating this process across all matched pairs allows us to estimate the Average Treatment Effect for the campaign.

The propensity score itself is not the quantity we ultimately care about. Its purpose is to help us create treatment and control groups that are comparable before the intervention.

The causal effect is estimated after the matching is complete by comparing the outcomes of these matched groups.

🎤 Interview Q: Why do we estimate a propensity score instead of matching directly on every variable?

A: Matching on many variables simultaneously becomes difficult as the number of features grows. The propensity score summarizes all observed characteristics into a single probability, making it much easier to create comparable treatment and control groups.

Limitations of Propensity Score Matching

Propensity Score Matching is a powerful technique, but it is not a magic solution.

Like every causal inference method, it relies on assumptions and has important limitations that data scientists should understand before applying it.

It Only Accounts for Observed Variables

Propensity Score Matching can only balance characteristics that are included in the model.

For our marketing campaign, we might match customers based on variables such as purchase history, average spending, website activity, and loyalty status.

But what if customer motivation influenced both the likelihood of receiving the campaign and their future spending?

If motivation wasn’t measured, it can’t be included in the propensity score.

As a result, unobserved confounding may still bias the estimated treatment effect.

Good Matches Are Essential

Propensity Score Matching works best when treated and untreated customers have similar characteristics.

Suppose a campaign targeted only the company’s highest-value customers.

If there are no comparable customers in the control group, meaningful matches cannot be created. Without comparable matches, estimating the treatment effect becomes unreliable.

This concept is often referred to as common support or overlap.

Matching Reduces the Sample Size

Not every customer will find a suitable match. Customers without a good match are typically excluded from the analysis.

While this improves the quality of the comparisons, it also reduces the amount of data available for estimating the treatment effect.

Randomized Experiments Are Still Preferred

When randomized experiments are feasible, they remain the preferred approach for estimating causal effects.

Random assignment naturally balances both observed and unobserved characteristics between the treatment and control groups.

Propensity Score Matching attempts to mimic this process using observational data, but it cannot fully replace a well-designed randomized experiment.

Important Insight

Propensity Score Matching does not eliminate bias. Instead, it reduces bias by creating more comparable treatment and control groups based on the information that is available.

The quality of the results depends not only on the matching algorithm but also on the quality and completeness of the data.

🎤 Interview Q: What are the limitations of Propensity Score Matching?

A: Propensity Score Matching only controls for observed variables. It cannot account for unmeasured confounders, requires sufficient overlap between treatment and control groups, may reduce the sample size after matching, and does not replace randomized experiments when those are feasible.

Final Takeaway

Let’s return to the question that started this article.

Did the marketing campaign actually increase sales?

At first glance, the answer seemed simple.

Customers who received the campaign spent more than those who did not.

But as we’ve seen, that difference alone doesn’t prove the campaign caused the increase.

The treatment and control groups may have been different long before the campaign was launched.

To estimate the true impact of the campaign, we need a fair comparison.

That’s exactly what Propensity Score Matching helps us achieve.

By matching treated customers with untreated customers who had similar characteristics before the intervention, PSM reduces selection bias and provides a more credible estimate of the treatment effect.

While it cannot account for unobserved confounders or replace a randomized experiment, it remains one of the most widely used techniques for estimating causal effects from observational data.

Key Takeaways

• We can never observe the counterfactual for the same individual.

• Simply comparing treatment and control groups can lead to biased conclusions.

• Propensity Score Matching creates fairer comparisons by matching similar individuals.

• The quality of the estimate depends on the quality of the matches and the available data.

Machine learning helps us answer:

What is likely to happen?

Propensity Score Matching helps us answer:

What difference did the intervention actually make?

That distinction is what makes causal inference such a valuable skill for modern data scientists.

🎤 Interview Q: When would you use Propensity Score Matching?

A: I would use Propensity Score Matching when I need to estimate the causal effect of a treatment using observational data and a randomized experiment isn’t possible. By matching treated and untreated individuals with similar characteristics, PSM helps reduce selection bias and create more credible comparisons.

What’s Next?

In this article, we explored how Propensity Score Matching helps us estimate causal effects by creating fair comparisons between treated and untreated groups.

But what if we have data collected before and after an intervention?

In many real-world situations, businesses, governments, and researchers evaluate policies, product launches, pricing changes, or marketing campaigns over time. Rather than matching similar individuals, they compare how outcomes change before and after an intervention.

In the next issue of The Practical Data Scientist, we’ll explore this approach and learn how it helps answer one of the most common questions in analytics:

Did the intervention truly change the outcome, or would it have changed anyway?

We’ll continue building our intuition for causal inference—one business problem at a time.

Keep building, keep learning—wishing you the best in your data journey.

Introduction

Over the past several issues of The Practical Data Scientist, we’ve focused on predictive machine learning. We’ve explored topics ranging from Logistic Regression and Decision Trees to Random Forests, XGBoost, Cross Validation, and Regularization. Together, these techniques help us answer an important question:

What is likely to happen?

Prediction is one of the most powerful applications of data science, but many business decisions require us to answer a different question:

What will happen if we change something?

Suppose a company launches a new marketing campaign and sales increase by 15%.

Did the campaign actually cause the increase? Or would sales have increased anyway because of seasonal demand?

Similarly, imagine a retailer introduces free shipping and observes a higher conversion rate.

Did free shipping drive more purchases, or were customers already more likely to buy because of a holiday shopping season?

These are not prediction problems. They are causal inference problems.

Causal inference is the field of data science that helps us estimate the effect of an intervention—whether it’s a marketing campaign, a pricing change, a product feature, or a public policy. Instead of asking what happened, it asks what would have happened if we had acted differently?

This article marks the beginning of a new series on causal inference. Over the next few issues, we’ll build an intuitive understanding of how data scientists move beyond correlation to estimate cause and effect. We’ll cover topics such as propensity score matching, difference-in-differences, instrumental variables, and other techniques commonly used when randomized experiments aren’t feasible.

In this first article, we’ll establish the foundation by exploring one of the most misunderstood concepts in data science:

Correlation does not imply causation.

By the end of this issue, you’ll understand why identifying relationships in data is only the first step—and why answering causal questions requires a completely different way of thinking.

Correlation vs. Causation

One of the most important lessons in data science is that correlation does not imply causation.

Although the phrase is widely quoted, it is also one of the most misunderstood concepts in analytics. Let’s start with the definitions.

Correlation means that two variables are associated with one another. As one variable changes, the other tends to change as well.

Causation, on the other hand, means that a change in one variable directly produces a change in another.

The difference may seem subtle, but it fundamentally changes the kinds of conclusions we can draw from data. Consider the following example.

Every summer:

• Ice cream sales increase.

• Drowning incidents also increase.

These two variables are positively correlated. Does this mean eating more ice cream causes drowning?

Clearly not.

The fact that two variables move together does not necessarily mean that one causes the other.

Business examples are just as common. Suppose a retailer launches a new recommendation system, and customer engagement increases.

Did the recommendation system improve engagement?

Or suppose a company offers discount coupons, and customers who receive them spend more money.

Did the coupons increase spending?

From the observed data alone, we cannot confidently answer these questions.

We know that two events occurred together, but we do not yet know whether one caused the other.

Why This Matters

Many machine learning models are designed to identify relationships and make predictions.

For example, a model might accurately predict which customers are likely to make a purchase or which products are likely to sell well.

But businesses often need to answer a different type of question:

• Should we launch this marketing campaign?

• Should we introduce free shipping?

• Should we redesign our website?

• Should we recommend this product?

These questions require us to estimate the effect of an intervention—not simply identify patterns in historical data.

Correlation tells us what tends to happen together. Causation tells us what happens because of an action.

Understanding this distinction is the first step toward making better data-driven decisions.

In the next section, we’ll explore why determining causation is much more difficult than identifying correlation, and introduce the challenges that make causal inference such an important field in data science.

Interview Pro Tip

🎤 Interview Q: What is the difference between correlation and causation?

A: Correlation means two variables are associated with each other, while causation means that a change in one variable directly produces a change in another. Although causal relationships often exhibit correlation, correlated variables do not necessarily have a cause-and-effect relationship.

Why Is Causation So Hard?

If correlation doesn’t prove causation, you might wonder:

Why can’t we simply compare the outcomes before and after an intervention?

The answer is that many other factors may influence the outcome at the same time.

These factors can make it difficult—or even impossible—to determine whether the intervention itself caused the observed change.

Let’s look at some of the most common challenges.

Confounding Variables

A confounding variable is a factor that influences both the treatment and the outcome.

As a result, it creates the illusion of a causal relationship when one may not actually exist. Consider the ice cream example from the previous section.

During the summer:

• Ice cream sales increase.

• Drowning incidents increase.

It would be incorrect to conclude that eating ice cream causes drowning.

The real driver is temperature.

Hot weather encourages more people to buy ice cream and also leads more people to swim, increasing the likelihood of drowning incidents.

Because temperature affects both variables, it is called a confounder.

Business problems are full of confounders.

Suppose a retailer offers discount coupons to its most loyal customers. If those customers spend more money, can we conclude that the coupons caused the increase?

Not necessarily.

Loyal customers may have spent more even without receiving the coupon. In this case, customer loyalty is the confounding variable.

Selection Bias

Selection bias occurs when the individuals receiving a treatment are systematically different from those who do not. For example, imagine evaluating a premium loyalty program.

Customers who voluntarily join the program may already be your most engaged customers. If they spend more than non-members, we cannot immediately attribute the difference to the loyalty program.

Part of the difference may simply be due to who chose to enroll.

Reverse Causality

Sometimes, we correctly identify a relationship between two variables but misunderstand the direction of the relationship.

For example, suppose stores with higher sales also employ more staff.

Did hiring more employees increase sales?

Or did stores hire more employees because sales were already higher?

Without additional evidence, both explanations are plausible.

Important Insight

Observational data can reveal relationships, but it rarely tells us why those relationships exist.

Before concluding that one variable causes another, we must account for confounders, selection bias, and other factors that can influence the results. This is precisely why causal inference methods are needed.

🎤 Interview Q: What is a confounding variable?”

A: A confounding variable is a factor that influences both the treatment and the outcome, making it difficult to determine whether the observed relationship is truly causal.

How Do Data Scientists Estimate Causal Effects?

Once we’ve identified the challenges of observational data, the next question becomes:

How can we estimate causal effects despite these challenges?

There isn’t a single technique that works in every situation. Instead, data scientists choose from a toolbox of causal inference methods depending on the problem, the available data, and whether randomized experiments are feasible.

Some of the most commonly used approaches include:

• Randomized Controlled Trials (A/B Testing) — the gold standard when randomization is possible.

• Propensity Score Matching (PSM) — creates comparable treatment and control groups using observational data.

• Difference-in-Differences (DiD) — measures the impact of an intervention by comparing changes over time.

• Instrumental Variables (IV) — helps estimate causal effects when unobserved confounding exists.

• Regression Discontinuity Design (RDD) — leverages treatment assignment based on a threshold or cutoff.

If you’d like a deeper dive into these techniques, I’ve previously written about several of them, and you can find those articles below.

This new series will revisit each method individually, building a more intuitive understanding of when to use it, how it works, and the assumptions behind it.

Further Reading

The Math Behind A/B Testing: What You Need to Know to Interpret Results

What to Do When You Can’t Run an A/B Test: Practical Techniques with Examples

Final Takeaway

Throughout this article, we’ve explored a simple but powerful idea:

Correlation tells us what happens together. Causation tells us what happens because of an action.

This distinction lies at the heart of many business decisions.

Organizations rarely want to know whether two variables are related. They want to know whether an intervention—such as a marketing campaign, a pricing change, a new product feature, or a recommendation algorithm—actually caused an improvement in outcomes.

Answering these questions is far more challenging than building a predictive model. It requires careful reasoning, an understanding of bias, and the right analytical methods.

In this article, we’ve laid the foundation by understanding:

• The difference between correlation and causation

• Why establishing causality is difficult

• The challenges posed by confounders, selection bias, and reverse causality

• The different approaches data scientists use to estimate causal effects

In the coming issues, we’ll dive deeper into each of these techniques, exploring when to use them, how they work, and the assumptions they rely on.

Important Insight

Prediction helps us anticipate the future. Causal inference helps us understand how to change it.

For data scientists, that difference often determines whether an analysis simply describes the world—or helps improve it.

🎤 Interview Q: Why is causal inference important in data science?

A: Many business decisions require understanding the effect of an intervention rather than simply predicting an outcome. Causal inference provides the framework for estimating whether an action actually caused a change, enabling organizations to make more informed decisions.

What’s Next?

This article marks the beginning of our new series on causal inference.

In the next issue of The Practical Data Scientist, we’ll take a deeper dive into one of the most widely used approaches for estimating causal effects using observational data.

Rather than asking whether two variables are related, we’ll explore how data scientists determine what would have happened if an intervention had never occurred—a question that lies at the heart of business decision-making.

As the series progresses, we’ll continue building on these ideas, developing the intuition behind the methods that organizations use to move from correlation to causation.

If you’ve ever wondered how companies measure the true impact of a marketing campaign, a product launch, or a pricing change, this series is for you.

Keep building, keep learning—wishing you the best in your data journey.

We like to think collaboration between Product Managers and Analysts is natural.

It isn’t.

Even in high-performing teams, this relationship quietly breaks down—not because either side lacks skill, but because of how they work together day to day.

At its best, PM–Analyst collaboration is investigative. You’re both trying to understand the same system from different angles. But when collaboration becomes transactional, everyone loses.

Here are ten common breakdowns—and how both sides can work better together.

5 Collaboration Mistakes PMs Make with Analysts

Most PMs don’t intentionally create friction with analysts.

In fact, many of these behaviors come from good intentions—trying to move quickly, provide clarity, or keep the team aligned. But without realizing it, PMs can sometimes turn analysts into dashboard builders and query executors rather than thought partners.

The strongest PM–Analyst relationships aren’t built on requests and deliverables. They’re built on shared curiosity and a common goal: making better decisions.

Here are five ways PMs unintentionally make collaboration harder—and what better collaboration looks like.

1. Defining the Goal Without the Analyst

You walk into a meeting with a fully formed question: “Can we analyze churn for users in segment X over the last 90 days?”

It sounds clear. It’s actually limiting.

When PMs define the goal alone, they often lock into a specific lens before the analyst even gets context. The analyst becomes a query executor, not a thought partner.

What to do instead:

Bring the analyst in earlier when the problem is still messy. Frame it as:
“We’re seeing a drop in retention in segment X. I’d love your help figuring out what’s really happening.”

You’ll often get a better question than the one you started with.

2. Focusing on Tasks Instead of the Problem

PMs often default to task-based requests:
“Can you pull this dashboard?”
“Can you run this cohort analysis?”

But tasks without context force analysts to reverse-engineer intent, which slows everything down and reduces insight quality.

What to do instead:

Anchor every request in the underlying problem: “We’re trying to understand why activation dropped after onboarding changes.”

Now the analyst can challenge assumptions, suggest better approaches, and even tell you if you’re solving the wrong problem.

3. Treating Data as a Validation Tool, Not a Discovery Tool

A subtle one.

Sometimes PMs already have a hypothesis and they look to data to confirm it.
“Can you check if feature X improved engagement?”

This narrows the analysis to validation instead of exploration.

What to do instead:

Ask open-ended questions alongside your hypothesis: “My hypothesis is that feature X improved engagement, but I’d also like to understand what actually changed in user behavior.”

This creates space for unexpected insights the kind that actually move product decisions.

4. Not Closing the Loop After Analysis

The analysis gets delivered. You say thanks. Then… silence.

From the analyst’s perspective, this is frustrating. They don’t know:

• Did the work influence a decision?

• Was it useful?

• Did anything ship because of it?

What to do instead:

Close the loop deliberately:
“Your analysis helped us deprioritize feature Y and focus on onboarding fixes. We’re rolling out changes next sprint.”

This builds trust and helps analysts refine future work based on real impact.

5. Skipping Retrospectives on Data Work

PMs are used to sprint retros but rarely apply the same thinking to analytics collaboration. So, the same issues repeat:

• Misaligned expectations

• Last-minute requests

• Over-scoped analyses

What to do instead:

Run lightweight retros specifically for collaboration:

• What worked in how we framed the problem?

• Where did we lose time?

• What should we do differently next time?

None of these mistakes come from a lack of product sense or leadership. They’re simply habits that emerge when teams become busy and communication becomes transactional.

But collaboration is a two-way street.

Just as PMs can unintentionally limit the impact of analytics, analysts can unintentionally make it harder for PMs to move quickly, make decisions, and communicate with stakeholders.

Let’s flip the perspective.

5 Collaboration Mistakes Analysts Make with PMs

Analysts often talk about wanting a seat at the table.

But with that seat comes responsibility.

The best analysts don’t just answer questions—they help shape decisions. And that requires understanding the pressures PMs operate under: ambiguity, competing priorities, stakeholder expectations, and the constant need to balance speed with rigor.

Many collaboration breakdowns aren’t caused by technical gaps. They’re caused by misaligned expectations and communication styles.

Here are five common mistakes analysts make—and how to become a better partner to your PM.

1. Jumping Into Analysis Before Understanding the Decision

Analysts are trained to answer questions. So, when a request comes in, it’s tempting to immediately start pulling data.

But sometimes the most important question isn’t: “What analysis should I run?”

It’s: “What decision are we trying to make?”

Without understanding the decision, analysts can spend days producing work that doesn’t actually influence anything.

What to do instead:

Start with questions like:

• “What decision will this analysis inform?”

• “What would you do differently depending on the result?”

The goal isn’t just to answer questions—it’s to enable decisions.

2. Optimizing for Technical Correctness Instead of Business Relevance

Analysts love rigor. PMs love progress.

Sometimes analysts spend weeks building the perfect analysis when a directional answer would have been enough.

Meanwhile, the decision window closes.

What to do instead:

Match the depth of analysis to the importance and urgency of the decision.

Ask: “Do we need a precise answer, or do we need a useful answer?”

Perfect analyses delivered too late are rarely impactful.

3. Speaking in Metrics Instead of Narratives

Analysts spend so much time with data that it’s easy to assume the numbers speak for themselves.

They don’t.

PMs aren’t looking for charts—they’re looking for understanding.

What to do instead:

Move beyond: “Activation decreased by 3.2%.”

To: “Users are dropping off earlier in onboarding, which suggests the new flow may be creating friction.”

Help connect:

• What happened?

• Why did it happen?

• What should we do next?

Insights are far more valuable than metrics.

4. Treating Requests as Tickets Instead of Partnerships

Sometimes analysts unintentionally position themselves as service providers:

“Tell me what you need, and I’ll deliver it.”

But the strongest analyst-PM relationships are collaborative, not transactional.

What to do instead: Challenge assumptions. Suggest alternatives. Push back when necessary.

Instead of saying:

“Sure, I’ll build that dashboard.”

Try: “Before we build this, can we talk about the problem we’re trying to solve?”

Great analysts don’t just provide answers—they help frame better questions.

5. Disappearing Into Analysis and Returning Weeks Later

Nothing creates anxiety for PMs faster than silence. Without visibility, PMs don’t know:

• Is the analysis blocked?

• Are assumptions changing?

• Is the scope growing?

What to do instead: Share progress early and often.

Even quick updates like:

• “I’m seeing something unexpected.”

• “Initial results don’t support our hypothesis.”

• “I may need another day to validate this.”

These conversations prevent surprises and build trust.

Ultimately, great products are rarely built by PMs or analysts in isolation.

The best decisions happen when product intuition and analytical thinking reinforce each other rather than compete with each other.

PMs don’t need analysts who simply answer questions.

Analysts don’t need PMs who simply assign tasks.

They both need partners who are willing to challenge assumptions, communicate openly, and stay focused on the outcome rather than the process.

Because when that happens, data stops being a reporting function and product stops being a guessing game. And that’s when teams do their best work.

Introduction

Over the past few issues of The Practical Data Scientist, we’ve explored increasingly powerful machine learning models and learned how to evaluate and tune them properly.

But regardless of the algorithm we use, one challenge remains:

How do we prevent models from memorizing the training data?

Suppose we train two models.

One achieves 99% training accuracy, while the other achieves 95%.

Does that automatically make the first model better?

Not necessarily.

A model that fits the training data too closely may end up learning noise rather than underlying patterns. As a result, it performs exceptionally well on the data it has already seen but struggles when presented with new observations.

This phenomenon, known as overfitting, is one of the most common problems in machine learning.

Ideally, we want models that capture meaningful relationships while ignoring random fluctuations in the data.

This is where regularization comes in.

Regularization introduces a penalty for model complexity, encouraging models to learn simpler and more generalizable patterns. Instead of focusing solely on fitting the training data, regularization helps strike a balance between accuracy and complexity.

In this issue, we’ll explore the intuition behind regularization, understand how Ridge (L2), Lasso (L1), and Elastic Net work, and discuss when to use each approach.

By the end of this issue, you’ll understand one of the most important ideas in machine learning:

A model that fits the training data perfectly is not always the model that performs best.

The Idea Behind Regularization

In the previous section, we saw that highly complex models can overfit the training data and struggle to generalize to unseen observations.

So how do we prevent this?

The answer is regularization.

At a high level, regularization discourages the model from becoming overly complex by penalizing large coefficients.

The intuition is simple:

Simpler models tend to generalize better.

Consider a linear model:

where:

• (x_i) represents the input features

• (w_i) represents the coefficients (weights)

• (b) is the intercept

The coefficients determine how strongly each feature influences the prediction.

Without any constraints, the model is free to assign very large values to these coefficients if doing so helps reduce training error. While this may improve performance on the training data, it can also make the model more sensitive to noise and increase the risk of overfitting.

To understand how regularization works, we first need to introduce the concept of a loss function.

A loss function measures how far the model’s predictions are from the true values. During training, the model tries to minimize this quantity.

Regularization modifies the loss function by adding a penalty term that discourages complexity.

Instead of minimizing only the prediction error, the model now tries to balance two objectives:

• Fit the data well

• Keep the model reasonably simple

Mathematically, the regularized loss becomes:

where:

• _prediction measures prediction error

• lambda controls the strength of regularization

• The penalty term discourages large coefficients

By introducing this penalty, the model may sacrifice a small amount of training accuracy, but it often achieves much better performance on unseen data.

Why Does Regularization Work?

Large coefficients can make a model highly sensitive to small changes in the input data.

By shrinking the coefficients, regularization:

• Reduces model complexity

• Lowers variance

• Improves generalization

• Helps prevent overfitting

The tradeoff is that regularization may slightly increase bias. However, this increase in bias is often outweighed by a larger reduction in variance, leading to better overall performance.

Regularization does not try to fit the training data perfectly. Instead, it intentionally sacrifices a little training accuracy in exchange for better generalization.

This idea lies at the heart of the bias-variance tradeoff and is one of the reasons regularization is so effective.

🎤 Interview Q: What is the intuition behind regularization?

A: Regularization penalizes model complexity, encouraging simpler models that generalize better to unseen data.

Ridge Regression (L2)

One of the most common forms of regularization is Ridge Regression, also known as L2 regularization.

The central idea behind Ridge Regression is simple:

Instead of eliminating features, Ridge shrinks their coefficients toward zero.

Recall that in the previous section, we introduced the idea of adding a penalty term to the loss function. In Ridge Regression, this penalty is proportional to the sum of the squared coefficients:

where:

• _prediction is the original loss function

• (w_i) are the model coefficients

• lambda controls the strength of regularization

What Does Ridge Regression Do?

Ridge discourages large coefficients by making them more expensive. As the value of lambda increases:

• Coefficients become smaller

• Model complexity decreases

• Variance decreases

• Generalization improves

However, unlike Lasso Regression, Ridge does not force coefficients exactly to zero. Instead, it keeps all features in the model while reducing their influence.

Why Does Squaring Matter?

Because when the coefficients are squared:

• Large coefficients receive a much larger penalty

• Small coefficients are penalized less severely

This allows Ridge Regression to prevent any single feature from dominating the model.

When Should You Use Ridge Regression?

Ridge is particularly useful when:

• Most features are believed to contain useful information

• Features are highly correlated

• The goal is prediction rather than feature selection

• You want a more stable model

For example, if multiple features contain overlapping information, Ridge tends to distribute the weights across them rather than relying heavily on a single feature.

Ridge Regression reduces model complexity without removing features. You can think of Ridge as:

“Keep all the features, but don’t let any one of them become too important.”

🎤 Interview Q: What is the effect of L2 regularization?

A: L2 regularization shrinks the coefficients toward zero, reducing model complexity and improving generalization, but it does not perform feature selection because the coefficients rarely become exactly zero.

Subscribe now

Lasso Regression (L1)

While Ridge Regression shrinks coefficients toward zero, Lasso Regression, also known as L1 regularization, can go one step further:

It can eliminate them entirely.

In Lasso Regression, the penalty term is proportional to the sum of the absolute values of the coefficients.

The regularized loss function becomes:

where:

• _prediction is the original loss function

• (w_i) are the model coefficients

• lambda controls the strength of regularization

What Does Lasso Regression Do?

Lasso encourages sparsity. As the value of (\lambda) increases:

• Some coefficients shrink to exactly zero

• Unimportant features are removed

• The model becomes simpler

• Variance decreases and generalization improves

Unlike Ridge Regression, which retains all features, Lasso automatically performs feature selection.

Why Is Feature Selection Useful?

In many real-world problems, only a subset of features may truly be important. By driving some coefficients to zero, Lasso:

• Removes irrelevant features

• Produces simpler models

• Improves interpretability

• Reduces storage and computational requirements

This makes Lasso especially attractive when working with high-dimensional datasets.

When Should You Use Lasso Regression?

Lasso is particularly useful when:

• You suspect that only a few features are truly important

• Interpretability is important

• You want the model to automatically perform feature selection

• The dataset contains many features

The key difference between Ridge and Lasso is: Ridge shrinks coefficients. Lasso can eliminate them.

This ability to set coefficients exactly to zero is what makes Lasso a powerful tool for feature selection.

🎤 Interview Q: Why does Lasso perform feature selection but Ridge does not?

A: L1 regularization can drive some coefficients exactly to zero, effectively removing those features from the model. In contrast, L2 regularization shrinks coefficients toward zero but rarely makes them exactly zero.”

Elastic Net

So far, we’ve seen that Ridge Regression shrinks coefficients while Lasso Regression can eliminate them entirely. But what if we want the benefits of both?

This is where Elastic Net comes in.

Elastic Net combines the penalties used in Ridge and Lasso, allowing the model to simultaneously shrink coefficients and perform feature selection.

The regularized loss function becomes:

where:

• _prediction is the original loss function

• The first penalty term corresponds to L1 regularization

• The second penalty term corresponds to L2 regularization

• lambda_1 and lambda_2 control the strength of the two penalties

What Does Elastic Net Do?

Elastic Net inherits properties from both Ridge and Lasso. As a result, it:

• Shrinks coefficients toward zero

• Sets some coefficients exactly to zero

• Performs feature selection

• Reduces overfitting

• Produces more stable models

Why Not Always Use Lasso?

One limitation of Lasso is that when features are highly correlated, it often selects one feature and discards the others. For example, suppose we are predicting house prices using:

• House size in square feet

• Number of bedrooms

• Number of bathrooms

These features are strongly related. Lasso may arbitrarily keep one feature and remove the others.

Elastic Net behaves differently.

Instead of choosing a single feature, it tends to keep groups of correlated features together, making the resulting model more stable and often more accurate.

When Should You Use Elastic Net?

Elastic Net is particularly useful when:

• The dataset contains many features

• Features are highly correlated

• You want both shrinkage and feature selection

• Lasso appears unstable

In practice, Elastic Net is often preferred over pure Lasso when working with real-world datasets containing correlated variables.

You can think of the three methods as follows:

• Ridge: Keep all features, but shrink them.

• Lasso: Remove unnecessary features.

• Elastic Net: Combine the strengths of both.

🎤 Interview Q: When would you choose Elastic Net over Lasso?”

A: Elastic Net is preferred when features are highly correlated because it combines the shrinkage effect of Ridge with the feature selection ability of Lasso, resulting in more stable models.

Subscribe now

Ridge vs Lasso vs Elastic Net

We’ve now explored the three most common regularization techniques individually. While they all aim to reduce overfitting and improve generalization, they do so in different ways.

The image below summarizes their key differences.

• Ridge Regression (L2) shrinks coefficients toward zero while keeping all features in the model. It is often a good choice when most features contain useful information and prediction accuracy is the primary goal.

• Lasso Regression (L1) goes one step further by driving some coefficients exactly to zero. This makes it particularly useful for feature selection and building more interpretable models.

• Elastic Net combines the strengths of both Ridge and Lasso. By performing shrinkage and feature selection simultaneously, it tends to work well when features are highly correlated.

There is no universally best regularization technique.

The right choice depends on the problem, the nature of the data, and whether prediction accuracy or model interpretability is more important.

Rapid Fire Interview Questions and Final Takeaway

1. What is regularization?

Regularization is a technique that penalizes model complexity to reduce overfitting and improve generalization.

2. Why do we need regularization?

Highly complex models can memorize noise in the training data. Regularization encourages simpler models that perform better on unseen data.

3. What is the difference between L1 and L2 regularization?

L2 regularization (Ridge) shrinks coefficients toward zero but keeps all features.

L1 regularization (Lasso) can shrink some coefficients exactly to zero, effectively performing feature selection.

4. Why does Lasso perform feature selection?

Because the L1 penalty can drive some coefficients to exactly zero, removing those features from the model.

5. When would you use Elastic Net?

Elastic Net is often preferred when features are highly correlated because it combines the shrinkage effect of Ridge with the feature selection ability of Lasso.

6. Does regularization always improve training accuracy?

No. Regularization may slightly reduce training accuracy, but it often improves performance on unseen data by reducing overfitting.

Final Takeaway

The goal of machine learning is not to fit the training data perfectly. The goal is to build models that generalize well.

Regularization helps achieve this by controlling model complexity and striking a balance between bias and variance.

If there’s one idea to remember, it’s this:

A slightly simpler model that generalizes well is often more valuable than a complex model that memorizes the training data.

What’s Next?

Earlier in The Practical Data Scientist, we explored A/B testing and briefly introduced several techniques that help us move beyond simple experimentation.

In the next issue, we’ll begin taking a deeper dive into these methods with Causal Inference for Data Science Interviews.

We’ll explore one of the most important questions in data science:

Did this action actually cause the outcome?

We’ll start by understanding the difference between correlation and causation, discuss common sources of bias, and introduce the tools data scientists use to estimate causal effects when randomized experiments aren’t possible.

Over the coming issues, we’ll go deeper into techniques such as propensity score matching, difference-in-differences, instrumental variables, and other approaches that help answer causal questions in the real world.

Because finding patterns is useful—but understanding causes is powerful.

Keep building, keep learning—wishing you the best in your data journey.

Over the past few issues of The Practical Data Scientist, we’ve explored several machine learning models, from Logistic Regression and Decision Trees to Random Forests and XGBoost.

But regardless of how powerful a model may be, one question remains: how do we know if it will perform well on unseen data?

Suppose we train two models and obtain the following training accuracies:

• Model A: 99%

• Model B: 95%

Which model is better?

At first glance, Model A might seem like the obvious choice. However, training performance alone tells us very little about how the model will behave in the real world.

A model that performs exceptionally well on the training data may simply be memorizing it rather than learning patterns that generalize to new observations. This phenomenon, known as overfitting, is one of the most common challenges in machine learning.

To build models that generalize well, we need reliable ways to estimate their performance on unseen data and systematic approaches for choosing their settings.

This is where cross validation and hyperparameter tuning come into play.

In this issue, we’ll explore how to properly evaluate machine learning models, understand why a single train-test split is often not enough, and learn how to tune models to achieve better performance without falling into common pitfalls.

By the end of this issue, you’ll understand one of the most important ideas in machine learning:

“A model is only as good as the way it is evaluated”

Train, Validation, and Test Sets

To understand cross validation and hyperparameter tuning, we first need to understand how data is typically divided during model development.

A machine learning model should not be evaluated on the same data it was trained on. Doing so can give an overly optimistic estimate of performance and make it difficult to detect overfitting.

For this reason, datasets are usually split into three parts:

Training Set

The training set is used to fit the model. During this phase, the algorithm learns patterns and relationships from the data.

Examples:

• Estimating coefficients in Logistic Regression

• Building trees in Random Forests

• Learning residuals in XGBoost

The model has direct access to this data.

Validation Set

The validation set is used during model development. Its primary purposes are:

• Hyperparameter tuning

• Model selection

• Comparing different algorithms

For example, we might use the validation set to answer questions like:

• Should the Random Forest have 100 trees or 500 trees?

• What learning rate should we use for XGBoost?

• Is Logistic Regression performing better than Random Forest?

Importantly, the model does not learn directly from the validation set.

Instead, the validation set helps us make decisions about the model.

Test Set

The test set is used only once, after all model development is complete. Its purpose is to provide an unbiased estimate of how the final model will perform on unseen data.

Think of the test set as a final exam.

Just as students should not see the exam before taking it, the model should not use information from the test set while it is being developed.

The typical workflow looks like this:

Dataset
   ↓
Training Set
   ↓
Validation Set
   ↓
Test Set

• Train on the training set

• Tune on the validation set

• Evaluate once on the test set

Many beginners focus heavily on training accuracy. However, what ultimately matters is how well the model performs on data it has never seen before.

🎤 Interview Q: Why can’t we tune hyperparameters using the test set?

A: Because the test set should remain untouched until the very end. Using it during model development can lead to overly optimistic performance estimates and poor generalization.

Cross Validation

In the previous section, we saw how datasets are typically divided into training, validation, and test sets. But there is a problem with this approach.

Suppose we randomly split our data and obtain a validation accuracy of 92%.

How much confidence should we have in that number? What if we had chosen a different validation set?

A different split could produce:

• 90% accuracy

• 94% accuracy

• Or something entirely different

In other words, a single train-validation split may give us a misleading estimate of model performance. This is where cross validation comes in.

The Idea Behind Cross Validation

Instead of relying on a single validation set, cross validation repeatedly trains and evaluates the model on different subsets of the data.

By averaging performance across multiple splits, we obtain a more reliable estimate of how well the model will generalize to unseen data.

K-Fold Cross Validation

The most common approach is K-Fold Cross Validation.

The process works as follows:

1. Divide the dataset into K equal folds

2. Use K−1 folds for training

3. Use the remaining fold for validation

4. Repeat the process K times so that every fold serves as the validation set exactly once

5. Average the performance across all K runs

For example, in 5-fold cross validation:

• Train on folds 1–4 and validate on fold 5

• Train on folds 1–3 and 5, validate on fold 4

• Continue until each fold has been used as the validation set

The final score is the average across all five runs.

Why Use Cross Validation?

Cross validation offers several advantages:

• Makes better use of the available data

• Reduces dependence on a single random split

• Produces a more stable estimate of performance

• Helps compare different models more fairly

For these reasons, cross validation is widely used in machine learning competitions and real-world projects.

The most commonly used values are:

• 5-Fold Cross Validation

• 10-Fold Cross Validation

In practice, 5-fold cross validation provides a good balance between computational cost and reliable performance estimates.

Cross validation does not improve the model itself. Instead, it improves our confidence in the model’s estimated performance.

🎤 Interview Q: Why do we use cross validation instead of a single train-test split?”

A: Cross validation reduces the dependence on a particular split and provides a more reliable estimate of how the model will perform on unseen data.”

Hyperparameter Tuning

So far, we’ve focused on evaluating models. But once we have a reliable evaluation strategy, another question naturally arises:

How do we choose the best version of a model?

This is where hyperparameter tuning comes in.

Hyperparameters are settings that control how a machine learning algorithm learns. Unlike model parameters, they are not learned from the data and must be specified before training begins.

For example: Random Forest

• Number of trees (n_estimators)

• Maximum depth (max_depth)

• Minimum samples per split (min_samples_split)

XGBoost

• Learning rate

• Maximum depth

• Number of trees

• Subsampling rate

Different hyperparameter values can lead to very different models and, consequently, very different performance.

Parameters vs Hyperparameters

It is important to distinguish between parameters and hyperparameters.

Parameters

• Learned automatically from the data

• Examples:

  • Coefficients in Logistic Regression

  • Split points in Decision Trees

  • Leaf values in XGBoost

Hyperparameters

• Chosen before training

• Control how the model learns

• Examples:

  • Learning rate

  • Maximum depth

  • Number of trees

Grid Search

One approach to hyperparameter tuning is Grid Search. The idea is simple:

1. Define a set of possible values for each hyperparameter.

2. Train models using every possible combination.

3. Select the combination that produces the best validation performance.

For example:

max_depth: [3, 5, 7]
n_estimators: [100, 300, 500]

Grid Search evaluates all nine combinations.

Advantages

• Simple to understand

• Exhaustive search

Limitations

• Computationally expensive

• Doesn’t scale well with many hyperparameters

Random Search

Instead of trying every combination, Random Search samples combinations randomly. For example, instead of evaluating all 100 possible combinations, we might randomly test 20.

Surprisingly, Random Search often performs just as well while requiring significantly less computation.

Advantages

• More efficient

• Scales better to large search spaces

Limitations

• Not guaranteed to find the absolute best combination

The typical workflow looks like this:

Choose Hyperparameters
          ↓
Train Model
          ↓
Cross Validation
          ↓
Evaluate Performance
          ↓
Select Best Combination

Hyperparameter tuning is not about finding the perfect model. It is about finding a model that generalizes well to unseen data.

🎤 Interview Q: What is the difference between Grid Search and Random Search?

A: Grid Search evaluates every possible combination of hyperparameters, while Random Search samples combinations randomly. Random Search is often more computationally efficient and performs surprisingly well in practice.

Subscribe now

Common Pitfalls

Cross validation and hyperparameter tuning are powerful tools, but they can also give misleading results if used incorrectly. Understanding these pitfalls is just as important as understanding the techniques themselves.

Data Leakage

Data leakage occurs when information from outside the training set unintentionally influences the model. Examples include:

• Using future information to predict the past

• Scaling the entire dataset before splitting

• Performing feature selection on the full dataset

Data leakage can produce unrealistically high performance and often leads to poor results in production.

Tuning on the Test Set

The purpose of the test set is to estimate how the final model will perform on unseen data. If we repeatedly evaluate models on the test set and use those results to make decisions, the test set effectively becomes part of the training process.

This leads to overly optimistic performance estimates. The test set should only be used once, after all model development is complete.

Overfitting to the Validation Set

Trying too many models and hyperparameter combinations can cause us to unknowingly tailor our model to the validation data.

As a result, the validation score may no longer reflect true generalization performance. Cross validation helps reduce this problem, but it cannot eliminate it entirely.

Stratified K-Fold Cross Validation

For imbalanced datasets, a regular K-Fold split can produce folds with very different class distributions. This may lead to unstable evaluation metrics.

Stratified K-Fold preserves the class proportions in each fold, making performance estimates more reliable. For classification problems with class imbalance, Stratified K-Fold is often preferred over standard K-Fold.

Time Series Cross Validation

Traditional cross validation randomly shuffles observations. This approach breaks the temporal order of time series data and can introduce leakage from the future.

Instead, time series problems use expanding or rolling windows. The model is always trained on past observations and validated on future observations.

Important Insight

The biggest mistakes in machine learning often come not from the algorithms themselves, but from how models are evaluated. A sophisticated model evaluated incorrectly can be far worse than a simple model evaluated properly.

Common interview questions include:

• What is data leakage?

• Why shouldn’t we tune on the test set?

• When would you use Stratified K-Fold?

• Why is regular K-Fold inappropriate for time series data?

Being able to answer these questions demonstrates a strong understanding of practical machine learning, not just theory.

Rapid Fire Interview Questions and Final Takeaway

1. Why do we need a validation set?

The validation set is used for model selection and hyperparameter tuning, while the test set is reserved for the final evaluation.

2. Why can’t we tune hyperparameters using the test set?

Using the test set during model development can lead to overly optimistic performance estimates and poor generalization.

3. Why do we use cross validation?

Cross validation provides a more reliable estimate of model performance by reducing dependence on a single train-validation split.

4. What is the difference between parameters and hyperparameters?

Parameters are learned from the data during training, while hyperparameters are specified before training and control how the model learns.

5. Grid Search vs Random Search?

Grid Search evaluates every possible hyperparameter combination, while Random Search evaluates a random subset of combinations.

Random Search is often more computationally efficient.

6. When should you use Stratified K-Fold?

For classification problems with imbalanced classes, Stratified K-Fold ensures that each fold preserves the class distribution of the original dataset.

7. Why can’t we use regular K-Fold for time series data?

Randomly shuffling time series observations can introduce information from the future into the training set, leading to data leakage.

Time series data requires specialized validation strategies that preserve temporal order.

Final Takeaway

No matter how sophisticated a machine learning algorithm may be, its success ultimately depends on how it is evaluated.

Cross validation and hyperparameter tuning help us build models that generalize to unseen data rather than simply memorizing the training set.

If there’s one idea to remember, it’s this: A model is only as good as the way it is evaluated.

What’s Next?

In the next issue of The Practical Data Scientist, we’ll explore Regularization for Data Science Interviews.

We’ll dive into overfitting, the bias-variance tradeoff, and how techniques like Ridge (L2), Lasso (L1), and Elastic Net help machine learning models generalize better to unseen data.

Keep building, keep learning—wishing you the best in your data journey.

Introduction

In the last issue of The Practical Data Scientist, we explored Random Forests and saw how combining many decision trees can produce models that are more stable and less prone to overfitting.

Random Forests improve performance by averaging the predictions of many independent trees. This helps reduce variance and makes the overall model more reliable.

But what if we took a completely different approach?

Instead of building many trees independently and averaging their predictions, what if we built trees sequentially and allowed each new tree to focus on correcting the mistakes made by the previous ones?

This simple idea gave rise to Gradient Boosting, one of the most powerful concepts in machine learning.

Over the years, several boosting algorithms have been developed, but none have had a greater impact than XGBoost. From Kaggle competitions to production systems, XGBoost became the algorithm of choice for many structured data problems and helped popularize boosting across the machine learning community.

In this issue, we’ll explore the intuition behind Gradient Boosting, understand how XGBoost improves upon it, and discuss why these models have become staples of both real-world machine learning and data science interviews.

By the end of this issue, you’ll understand not just how boosting works, but why learning from mistakes can sometimes be more powerful than averaging predictions.

The Key Idea Behind Gradient Boosting

In the previous issue, we saw that Random Forests improve predictions by averaging many independent trees.

Gradient Boosting takes a very different approach.

Instead of building trees independently, it builds them sequentially.

The key idea is simple: Each new tree tries to correct the mistakes made by the previous trees.

Imagine we’re trying to predict house prices.

Suppose the true price of a house is:

Actual Value = 100

Our first tree predicts:

Prediction = 80

The model has made an error of:

Residual = Actual - Prediction = 20

Instead of starting from scratch, Gradient Boosting trains the next tree to learn this residual—the part that the first tree missed.

The second tree might predict:

Residual Prediction = 15

Now our updated prediction becomes:

80 + 15 = 95

We’re closer, but we’re still not perfect. The remaining error is:

100 - 95 = 5

A third tree is trained to learn this remaining error.

This process continues, with each new tree focusing on what previous trees failed to capture. As a result:

• Errors become progressively smaller

• Predictions become increasingly accurate

• The ensemble improves step by step

Random Forests vs Gradient Boosting

Although both methods combine many trees, their philosophies are very different.

Random Forests

• Build trees independently

• Reduce variance through averaging

• Rely on the wisdom of crowds

Gradient Boosting

• Build trees sequentially

• Reduce errors through correction

• Learn from previous mistakes

Random Forests improve predictions through averaging. Gradient Boosting improves predictions through learning.

This seemingly small difference is what makes boosting one of the most powerful ideas in machine learning.

🎤 Interview Q: What is the intuition behind Gradient Boosting?

A: Gradient Boosting builds trees sequentially, where each new tree learns from the errors made by previous trees.

How Gradient Boosting Works

Now that we understand the intuition behind boosting, let’s see how the algorithm works in practice.

The process begins by training a simple decision tree on the original dataset. This first tree produces an initial prediction.

Since the prediction is not perfect, the model computes the errors, also known as residuals, which represent the part of the target that the current model failed to explain.

A new tree is then trained to predict these residuals. Instead of predicting the target directly, this tree learns what the previous model got wrong.

This process repeats multiple times:

1. Train a tree

2. Calculate the residuals

3. Train the next tree on those residuals

4. Add the new predictions to the existing ones

5. Repeat until the model reaches the desired performance

As more trees are added, the ensemble gradually improves its predictions.

Additive Learning

Unlike Random Forests, where predictions are averaged, Gradient Boosting builds an additive model. Each new tree contributes a small correction to the existing prediction. The final prediction can be thought of as:

Final Prediction =
Tree 1
+ Tree 2
+ Tree 3
+ ...
+ Tree N

Every tree is responsible for explaining what the previous trees missed.

The Role of Learning Rate

Not all trees contribute equally.

Gradient Boosting introduces a parameter called the learning rate, which controls how much each new tree influences the final prediction.

A smaller learning rate:

• Learns more slowly

• Requires more trees

• Often generalizes better

A larger learning rate:

• Learns faster

• Requires fewer trees

• Can increase the risk of overfitting

The learning rate acts like a step size, ensuring that the model improves gradually rather than making large corrections all at once.

Gradient Boosting does not try to build one perfect tree. Instead, it builds many simple trees, each making small improvements to the overall prediction. This gradual correction process is one of the reasons boosting algorithms are so powerful.

🎤 Interview Q: What does the learning rate do in Gradient Boosting?

A: The learning rate controls how much each new tree contributes to the ensemble. Smaller learning rates lead to slower learning but often better generalization.”

Subscribe now

What Makes XGBoost Special?

Gradient Boosting provided the core idea, but XGBoost transformed that idea into one of the most successful machine learning algorithms ever created.

For many years, XGBoost dominated Kaggle competitions and became the go-to algorithm for structured and tabular data problems.

So what made it so special?

Regularization

Unlike traditional Gradient Boosting, XGBoost includes regularization to control model complexity and reduce overfitting.

This allows the model to achieve high predictive performance while maintaining better generalization.

Built-in Handling of Missing Values

Many machine learning algorithms require missing values to be imputed before training.

XGBoost can automatically learn how to handle missing values, making preprocessing simpler and more robust.

Feature Subsampling

Similar to Random Forests, XGBoost can randomly sample features during training.

This introduces diversity and helps reduce overfitting.

Efficient and Scalable Training

XGBoost was designed with speed and scalability in mind. It supports:

• Parallel computation

• Efficient memory usage

• Distributed training

These improvements make it suitable for large datasets and production environments.

Why Did XGBoost Dominate Kaggle Competitions?

XGBoost offered several advantages:

• Strong predictive performance

• Excellent handling of tabular data

• Robustness to noisy features

• Flexibility through extensive hyperparameters

• Effective regularization

As a result, it became the default starting point for many machine learning competitions and real-world applications.

XGBoost is not a completely different algorithm. It is an optimized implementation of Gradient Boosting that adds several practical improvements to improve speed, robustness, and generalization.

🎤 Interview Q: “What is the difference between Gradient Boosting and XGBoost?”

A: “XGBoost is an optimized implementation of Gradient Boosting that adds regularization, feature subsampling, efficient training, and automatic handling of missing values.”

Advantages and Limitations of XGBoost

XGBoost became one of the most popular machine learning algorithms because it combines strong predictive performance with several practical improvements over traditional Gradient Boosting. It is often one of the first algorithms data scientists try.

Advantages

XGBoost offers several important benefits:

• Excellent predictive performance

• Handles complex non-linear relationships

• Built-in regularization helps reduce overfitting

• Supports missing values natively

• Provides feature importance scores

• Scales efficiently to large datasets

• Frequently achieves state-of-the-art results on structured data

These advantages made XGBoost the dominant algorithm in many machine learning competitions and production systems.

Limitations

Despite its strengths, XGBoost is not always the best choice. Some common limitations include:

• Larger number of hyperparameters

• Longer training times compared to simpler models

• Greater risk of overfitting if not properly tuned

• Less interpretable than simpler models

• Requires more experimentation to achieve optimal performance

When Should You Use XGBoost?

XGBoost is often a strong choice when:

• Predictive performance is more important than interpretability

• Relationships between features are complex and non-linear

• You are willing to spend time tuning hyperparameters

In practice, XGBoost is commonly used for:

• Fraud detection

• Customer churn prediction

• Credit risk modeling

• Recommendation systems

• Demand forecasting

• Marketing response modeling

XGBoost is often considered the “power tool” of machine learning. It can achieve exceptional performance, but that performance usually comes at the cost of increased complexity and tuning effort.

🎤 Interview Q: When would you choose XGBoost over simpler models?

A: When predictive performance is the primary objective and I’m working with structured data, XGBoost is often an excellent choice because of its ability to capture complex patterns while maintaining strong generalization.”

Random Forest vs XGBoost

Random Forests and XGBoost are both ensemble methods built on decision trees, but they improve performance in very different ways.

Random Forests reduce variance by averaging the predictions of many independent trees. XGBoost, on the other hand, builds trees sequentially, with each new tree correcting the mistakes made by the previous ones.

Which One Should You Use?

If you want:

• A robust model with minimal tuning

• Faster training

• Lower risk of overfitting

then Random Forest is often an excellent choice.

If you want:

• The highest possible predictive performance

• Greater flexibility

• Better results on many tabular datasets

then XGBoost is frequently the preferred option.

Neither algorithm is universally better. Random Forests are often easier to train and interpret, while XGBoost typically offers superior performance when properly tuned.

A good rule of thumb is: Start with Random Forests. Optimize with XGBoost.

🎤 Interview Q: Random Forest or XGBoost—which would you choose?

A: Random Forest is a great baseline because it is simple and robust. If maximizing predictive performance is critical and I have time for tuning, I would consider XGBoost.

Subscribe now

Rapid Fire Interview Questions

Q: What is the difference between bagging and boosting?

A: Bagging builds trees independently and combines their predictions through averaging. Boosting builds trees sequentially, with each new tree correcting the mistakes of the previous ones.

Q: What are residuals in Gradient Boosting?

A: Residuals are the errors made by the current model. Each new tree is trained to predict these residuals, gradually improving the overall prediction.

Q: What does the learning rate do?

A: The learning rate controls how much each new tree contributes to the ensemble. Smaller learning rates lead to slower learning but often better generalization.

Q: Why is XGBoost so powerful?

A: XGBoost combines the idea of Gradient Boosting with several practical improvements, including regularization, feature subsampling, efficient training, and automatic handling of missing values.

Q: Why can boosting overfit?

A: Because each new tree focuses on correcting previous errors, adding too many trees or using an aggressive learning rate can cause the model to learn noise in the training data.

Q: Which is better: Random Forest or XGBoost?

A: Neither model is universally better. Random Forests are simpler and more robust, while XGBoost often achieves higher predictive performance when properly tuned.

Final Takeaway

Random Forests improve predictions through averaging.Gradient Boosting improves predictions through correction.

While Random Forests rely on the wisdom of crowds, Gradient Boosting and XGBoost learn from their mistakes.

This simple idea—building models sequentially and correcting errors step by step—helped make XGBoost one of the most successful machine learning algorithms ever created.

What’s Next?

In the next issue of The Practical Data Scientist, we’ll shift our focus from building models to evaluating and selecting them.

We’ll explore Cross Validation and Hyperparameter Tuning, covering concepts like train-validation-test splits, K-fold cross validation, and common tuning strategies that help machine learning models generalize to unseen data.

Keep building, keep learning—wishing you the best in your data journey.

Introduction

In the last issue of The Practical Data Scientist, we explored Decision Trees and saw why they became one of the most popular machine learning algorithms.

They are intuitive, easy to visualize, and capable of capturing complex non-linear relationships. They can automatically learn feature interactions and require very little preprocessing.

But decision trees have an important weakness: A single decision tree can be highly unstable.

Imagine training a decision tree on a customer churn dataset. If you slightly change the training data—perhaps by removing a few observations or collecting a new batch of customers—the resulting tree may look completely different.

Different data can lead to:

• Different tree structures

• Different decision rules

• Different predictions

This behavior is known as high variance.

Decision trees are also prone to overfitting. As the tree grows deeper, it may begin learning random noise and dataset-specific patterns instead of meaningful relationships. While a deep tree can capture complex patterns, it may not generalize well to unseen data.

As a result, a single decision tree often suffers from:

• High variance

• Overfitting

• Unstable predictions

This naturally raises an important question: How do we keep the flexibility of decision trees while reducing their instability?

The answer is surprisingly simple. Instead of relying on one tree, build many trees and combine their predictions.

This idea forms the foundation of Random Forests.

Random Forests are one of the most successful machine learning algorithms ever created. They are widely used for fraud detection, churn prediction, recommendation systems, risk modeling, and countless other real-world applications.

In this issue, we’ll explore the key ideas behind Random Forests, including bagging, bootstrap sampling, feature randomness, and ensemble learning. More importantly, we’ll focus on the intuition behind why combining many trees often produces a model that is more stable, more accurate, and less prone to overfitting than any individual tree.

📌 Pro Tip: “What problem do Random Forests solve?”

“Random Forests primarily reduce the variance of decision trees, making predictions more stable and improving generalization.”

Subscribe now

The Wisdom of Crowds

Suppose you walk into a room and show 1,000 people a jar filled with jellybeans.

You ask each person: “How many jellybeans are in the jar?”

Most individual guesses will be wrong.

Some people will underestimate.
Some will overestimate.

But something interesting happens when you average all the guesses together. The average estimate is often surprisingly close to the true answer.

This phenomenon is known as the wisdom of crowds.

The idea is simple:

A collection of imperfect opinions can often be more accurate than any single opinion.

Random Forests apply this exact principle to machine learning.

Instead of relying on a single decision tree, Random Forests build many decision trees and combine their predictions. Each individual tree:

• Sees a slightly different version of the data

• Learns slightly different patterns

• Makes slightly different mistakes

While some trees may make poor predictions on certain observations, those mistakes tend to cancel out when we aggregate predictions across hundreds of trees. As a result, the combined model is often:

• More stable

• More accurate

• Less prone to overfitting

This process of building many models and combining their predictions is called ensemble learning.

Random Forests are one of the most successful examples of ensemble learning, and they rely on a specific technique called bagging, which we’ll explore next.

📌 Pro Tip: A strong interview explanation is -

“Random Forests work because averaging many diverse decision trees reduces variance and produces more stable predictions.”

That’s often more valuable than diving straight into implementation details.

What Is Bagging?

Bagging stands for Bootstrap Aggregating.

It is an ensemble learning technique that improves model performance by combining the predictions of multiple models instead of relying on a single model.

The idea is simple:

Rather than training one decision tree, we train many decision trees and aggregate their predictions.

The process works as follows:

1. Create multiple versions of the training dataset

2. Train a separate decision tree on each dataset

3. Generate predictions from all trees

4. Combine the predictions into a final result

For classification problems:

• Each tree votes for a class

• The majority vote becomes the final prediction

For regression problems:

• Each tree produces a numerical prediction

• The average becomes the final prediction

The Key Requirement for Bagging

For bagging to be effective, the models in the ensemble cannot all be identical.

Imagine training 100 decision trees on exactly the same dataset using the same settings. You would end up with 100 identical trees making the same predictions. In that case, combining their predictions provides no benefit.

For bagging to work, each tree must learn something slightly different. This creates diversity within the ensemble.

The challenge then becomes: How do we create many different trees while starting from the same dataset?

The solution is bootstrap sampling.

Instead of training every tree on the entire dataset, we train each tree on a different randomly generated sample of the data.

As a result:

• Each tree sees slightly different observations

• Each tree learns slightly different patterns

• Each tree produces slightly different predictions

This diversity is one of the key reasons bagging works so well in practice.

We’ll see exactly how bootstrap sampling creates these different datasets in the next section.

📌 Pro Tip: A common interview question is:

“Why doesn’t training multiple trees on the same dataset improve performance?”

A strong answer is:

“Because the trees would be nearly identical. Bagging works because each model is trained on a different bootstrap sample, creating diversity within the ensemble.”

Bootstrap Sampling

In the previous section, we saw that bagging requires multiple decision trees that are slightly different from one another.

This raises an important question:

How do we create different training datasets when we only have one dataset?

The answer is bootstrap sampling.

Bootstrap sampling is a technique where we repeatedly sample observations from the original dataset with replacement.

The phrase with replacement is important.

It means that after selecting an observation, we put it back into the dataset before drawing the next observation. As a result, the same observation can be selected multiple times. For example, suppose our original dataset contains five observations:

A B C D E

One bootstrap sample might look like:

A B B D E

Another bootstrap sample might be:

A C D D E

Notice that:

• Some observations appear multiple times

• Some observations are missing entirely

Even though each bootstrap sample has the same size as the original dataset, the composition is slightly different. This is exactly what we want.

By training each decision tree on a different bootstrap sample:

• Every tree sees a slightly different dataset

• Every tree learns slightly different patterns

• Every tree makes slightly different predictions

These differences create the diversity needed for bagging to be effective.

Why Sample With Replacement?

If we sampled without replacement, every bootstrap sample would contain exactly the same observations, just in a different order.

The resulting trees would be very similar.

Sampling with replacement introduces variability, which helps create diverse trees and ultimately improves the performance of the ensemble.

Although each bootstrap sample contains the same number of rows as the original dataset, it does not contain all of the original observations.

On average:

• About 63% of the original observations appear in a bootstrap sample

• About 37% are left out

These left-out observations are called Out-of-Bag (OOB) samples, which can later be used to estimate model performance without needing a separate validation dataset.

We’ll revisit OOB samples when discussing Random Forest evaluation.

Subscribe now

How Random Forests Work

At this point, we have all the building blocks needed to understand Random Forests.

A Random Forest combines:

• Bootstrap sampling

• Decision trees

• Random feature selection

• Aggregated predictions

The training process is surprisingly straightforward.

Step 1: Create Bootstrap Samples

Multiple bootstrap samples are generated from the original dataset.

Each sample contains:

• The same number of observations as the original dataset

• A slightly different combination of observations

Every decision tree will be trained on a different bootstrap sample.

Step 2: Train a Decision Tree on Each Sample

A separate decision tree is trained on each bootstrap sample.

Because every tree sees a slightly different dataset:

• The trees learn different patterns

• The trees make different mistakes

• The trees produce different predictions

This diversity is critical to the success of the ensemble.

Step 3: Randomly Select Features at Each Split

This is the step that makes Random Forests different from ordinary bagged trees.

Instead of considering all features when creating a split, each tree only evaluates a random subset of features.

For example, if a dataset contains 20 features:

• A tree may only consider 4 or 5 randomly selected features at a split

• Another tree may consider a different subset

This introduces additional diversity into the forest.

We’ll explore why this matters in the next section.

Step 4: Aggregate Predictions

Once all trees have been trained, their predictions are combined.

For classification:

• Each tree votes for a class

• The majority vote becomes the final prediction

For regression:

• Each tree produces a numerical prediction

• The average prediction becomes the final output

Putting It All Together, random Forests do not rely on a single “best” tree.

Instead, they rely on the collective intelligence of many trees trained on slightly different data and slightly different features.

This combination produces a model that is often:

• More accurate

• More stable

• Less prone to overfitting

than any individual decision tree.

📌 Pro Tip: Random Forests combine bootstrap sampling, decision trees, random feature selection, and prediction aggregation to create a low-variance ensemble model.

Why Random Feature Selection Matters

So far, we’ve seen that Random Forests use bootstrap sampling to train each tree on a different version of the data. But Random Forests add one more layer of randomness.

When a decision tree creates a split, it normally evaluates all available features and chooses the best one. For example, if a dataset contains 20 features, a traditional decision tree considers all 20 features at every split.

Random Forests work differently. At each split, the algorithm randomly selects a subset of features and only evaluates those features when deciding how to split the data.

For example:

• Dataset contains 20 features

• Random Forest randomly selects 4 features

• The tree chooses the best split from those 4 features only

The next split may use a completely different subset of features.

Why Is This Necessary?

Imagine a dataset where one feature is extremely predictive.

If every tree always has access to all features, most trees will repeatedly choose the same feature near the top of the tree. As a result:

• Trees become very similar

• Trees make similar predictions

• Trees make similar mistakes

In other words, the forest loses diversity.

Random feature selection helps prevent this problem. By forcing trees to consider different subsets of features:

• Trees become more diverse

• Correlation between trees decreases

• The ensemble becomes more effective

Bagged Trees vs Random Forests

This distinction is important:

Bagged Trees

• Use bootstrap sampling

• Train multiple trees

• Consider all features at each split

Random Forests

• Use bootstrap sampling

• Train multiple trees

• Randomly select features at each split

This additional randomness is what makes Random Forests significantly more powerful than ordinary bagged trees.

A Random Forest does not try to build the best individual tree. Instead, it tries to build many strong but diverse trees. The combination of bootstrap sampling and random feature selection creates the diversity needed for the ensemble to outperform any single tree.

📌 Pro Tip: A very common interview question is:

“What is the difference between bagging and Random Forests?”

A strong answer is:

“Random Forests use bagging, but they also introduce random feature selection at each split. This reduces correlation between trees and improves ensemble performance.”

Advantages and Limitations of Random Forests

Random Forests became one of the most popular machine learning algorithms because they combine strong predictive performance with relatively simple training procedures.

They often perform well out of the box and require less tuning than many other machine learning models.

Advantages

Random Forests offer several important benefits:

• Reduce overfitting compared to a single decision tree

• Lower variance through averaging

• Capture complex non-linear relationships

• Handle feature interactions automatically

• Require little preprocessing

• Do not require feature scaling

• Work well on both classification and regression problems

Because Random Forests combine many trees, they are typically more stable and reliable than individual decision trees.

This is one reason they are often used as a strong baseline model in machine learning projects.

Limitations

Despite their strengths, Random Forests are not perfect.

Some common limitations include:

• Less interpretable than a single decision tree

• Larger memory footprint

• Slower training and inference than simpler models

• Can become difficult to explain to stakeholders

• May struggle when extrapolation beyond the training data is required

For example, while a single decision tree can be visualized and explained relatively easily, understanding the behavior of hundreds of trees simultaneously is much more challenging.

When Should You Use Random Forests?

Random Forests are often a strong choice when:

• You are working with structured/tabular data

• Relationships are non-linear

• Predictive performance is more important than model simplicity

• You want a strong baseline before exploring more advanced models

In practice, Random Forests are commonly used for:

• Fraud detection

• Customer churn prediction

• Credit risk modeling

• Marketing response modeling

• Demand forecasting

Random Forests are often described as a “safe default” machine learning model.

They may not always be the best-performing algorithm, but they frequently provide strong results with relatively little effort.

Subscribe now

Decision Tree vs Random Forest

Decision Trees and Random Forests are closely related, but they solve very different problems.

A Decision Tree focuses on building a single model that learns patterns from the data. Random Forests take a different approach by combining many decision trees to produce a more stable and accurate prediction.

The table below summarizes the key differences:

Which One Should You Use?

If interpretability is your primary goal, a Decision Tree may be the better choice. Since the entire model can be visualized, it is often easier to explain predictions to stakeholders.

If predictive performance is the priority, Random Forests are usually the stronger option. By averaging many diverse trees, they reduce overfitting and often generalize better to unseen data.

Random Forests do not replace Decision Trees. Instead, they build upon them.

You can think of a Random Forest as: Many Decision Trees working together to produce a more reliable prediction.

This is one of the most important ideas in ensemble learning.

Rapid Fire Interview Questions

1. What problem do Random Forests solve?

Random Forests primarily reduce the variance of decision trees.

By combining predictions from many diverse trees, they produce more stable predictions and improve generalization on unseen data.

2. What is bagging?

Bagging (Bootstrap Aggregating) is an ensemble learning technique that trains multiple models on different bootstrap samples of the data and combines their predictions.

Its primary goal is to reduce variance.

3. What is bootstrap sampling?

Bootstrap sampling is the process of randomly sampling observations with replacement from the original dataset.

Because sampling is performed with replacement:

• Some observations appear multiple times

• Some observations are omitted

This creates different training datasets for different trees.

4. Why does Random Forest use random feature selection?

If every tree always considered all features, many trees would look very similar.

Random feature selection:

• Increases diversity

• Reduces correlation between trees

• Improves ensemble performance

5. What is the difference between bagging and Random Forests?

Bagging:

• Uses bootstrap sampling

• Trains multiple trees

• Uses all features at each split

Random Forest:

• Uses bootstrap sampling

• Trains multiple trees

• Uses a random subset of features at each split

You can think of Random Forests as:

Random Forest = Bagging + Random Feature Selection

6. Does Random Forest reduce bias or variance?

Random Forests primarily reduce variance.

Individual decision trees tend to have:

• Low bias

• High variance

Averaging many trees reduces variance while maintaining the flexibility of the underlying trees.

7. Do Random Forests require feature scaling?

No.

Random Forests split data based on feature thresholds rather than distances, so feature scaling typically has little impact on performance.

8. Why are Random Forests less interpretable than Decision Trees?

A Decision Tree consists of a single set of decision rules that can be visualized and explained.

A Random Forest may contain hundreds of trees, making it much harder to understand the exact reasoning behind an individual prediction.

9. What are Out-of-Bag (OOB) samples?

Out-of-Bag samples are observations that were not selected in a tree’s bootstrap sample.

These observations can be used to estimate model performance without requiring a separate validation dataset.

10. Why does a Random Forest usually outperform a Decision Tree?

A Random Forest reduces variance by averaging predictions from many diverse decision trees, which makes the model more stable and less prone to overfitting.

Final Takeaway

Random Forests demonstrate one of the most powerful ideas in machine learning:

Combining many models often works better than relying on a single model.

By using bootstrap sampling, random feature selection, and prediction aggregation, Random Forests transform unstable decision trees into models that are more accurate, more stable, and less prone to overfitting.

If there’s one idea to remember, it’s this:

Decision Trees are powerful. Random Forests make them reliable.

What’s Next?

I’m currently working on something a little different for the next issue of The Practical Data Scientist—potentially a collaboration with another creator in the data science space.

Nothing is finalized yet, but if it comes together, I think you’ll enjoy it.

Until then, keep building, keep learning—wishing you the best in your data journey.

In the previous issues of The Practical Data Scientist, we explored:

• Logistic Regression

• Evaluation Metrics

• Model performance trade-offs

Those topics form the foundation of machine learning interviews.

But many of the most powerful modern machine learning models are not linear models.

They are tree-based models.

Models like Random Forests, XGBoost, LightGBM and CatBoost have become some of the most widely used algorithms in practical machine learning because they:

• Handle non-linear relationships well

• Capture feature interactions automatically

• Require less feature engineering

• Perform extremely well on structured/tabular data

At the core of all these models is one simple idea: Decision Trees

Understanding decision trees deeply makes it much easier to understand modern ensemble methods later.

In this issue, we’ll build that foundation by covering:

• How decision trees work

• Entropy and information gain

• Gini impurity

• Tree splitting logic

• Overfitting and pruning

• Feature importance

• Advantages and limitations of tree-based models

• Common interview questions and pro tips

As always, the focus will be on intuition, practical understanding, and interview-oriented thinking rather than memorizing formulas.

📌 Pro Tip: Interviewers often use decision trees to test whether candidates truly understand machine learning intuition — not just equations and definitions.

Why Tree-Based Models Became Popular

Tree-based models became extremely popular because they solve many practical problems that traditional linear models struggle with.

Unlike models such as logistic regression, tree-based models can naturally capture:

• Non-linear relationships

• Complex feature interactions

• Different behaviors across segments of data

For example:

• The relationship between income and loan default may behave differently for different age groups

• Customer churn patterns may depend on combinations of features rather than single variables

Tree-based models handle these patterns automatically without requiring heavy feature engineering.

Key Advantages

Handles Non-Linear Relationships: Decision trees do not assume a linear relationship between features and the target. This makes them flexible for many real-world problems.

Captures Feature Interactions Automatically: Linear models often require manually creating interaction features. Tree-based models naturally capture interactions through splits. For example:

• A tree may first split on age

• Then create different income rules within each age group

Requires Less Preprocessing: Tree-based models usually -

• Do not require feature scaling

• Are less sensitive to outliers

• Can handle numerical and categorical features

This makes them very practical in real-world workflows.

Strong Performance on Tabular Data: Tree-based ensemble models like Random Forests, XGBoost, LightGBM often dominate structured/tabular machine learning problems. This is one reason they appear frequently in:

• Kaggle competitions

• Production ML systems

• Data science interviews

Compared to logistic regression:

• Logistic regression creates linear decision boundaries

• Tree-based models create flexible non-linear decision boundaries

This allows trees to model more complex patterns with less manual feature engineering.

📌 Pro Tip: A strong interview answer is “Tree-based models became popular because they combine strong predictive performance with relatively low preprocessing requirements on structured data.”

Subscribe now

What Is a Decision Tree?

A decision tree is a machine learning model that makes predictions by repeatedly splitting the data into smaller groups based on feature values.

The structure resembles a flowchart:

• Each split represents a decision rule

• Each branch represents an outcome of that rule

• Each final node (leaf node) represents a prediction

The goal of a decision tree is simple: Split the data in a way that creates increasingly “pure” groups.

For example, imagine building a model to predict customer churn. The tree might learn rules like:

• Customers with low engagement are more likely to churn

• Among those users, customers with many support tickets churn even more frequently

The tree keeps splitting the data step-by-step until the groups become sufficiently homogeneous.

Components of a Decision Tree

Root Node: The first split of the tree. This usually represents the feature that best separates the data initially.

Internal Nodes: Intermediate decision points where the data is split further. Example:

“Is account age > 12 months?”

Branches: The possible outcomes of a split. For example:

• Yes branch

• No branch

Leaf Nodes: The final prediction outputs. For classification problems:

• Class 0

• Class 1

For regression problems:

• Numerical prediction values

Recursive Partitioning

Decision trees use a process called recursive partitioning:

1. Find the best split

2. Split the data

3. Repeat the process on each subgroup

This continues until:

• The data becomes sufficiently pure

• Or stopping criteria are reached

Why Trees Feel Intuitive

One reason decision trees are popular is because they mimic human decision-making. For example:

• “If income is low and debt is high → higher default risk”

• “If engagement is low and complaints are high → higher churn risk”

This makes trees highly interpretable compared to many other machine learning models.

📌 Pro Tip: A strong interview explanation is:

“Decision trees recursively split the data to create increasingly pure groups.”

That demonstrates deeper understanding than simply saying:

“Trees make decisions using branches.”

How Decision Trees Learn

Now that we understand what a decision tree is, the next question becomes:

How does the tree decide where to split?

At every step, the tree evaluates different possible splits and chooses the one that creates the “purest” groups. For example, in a churn prediction problem, the model may test rules such as:

• Is engagement score < 20?

• Is account age > 12 months?

• Is number of support tickets > 5?

The goal is to reduce uncertainty and create groups where most observations belong to a single class. Decision trees learn using a recursive process:

• Find the best split at the current node

• Split the data into smaller groups

• Repeat the process on each subgroup

This continues until the groups become sufficiently pure or the model reaches stopping criteria. A “good” split is one that:

• Separates the classes clearly

• Reduces impurity within each group

For example, a node that originally contains a 50/50 mix of churners and non-churners may become much cleaner after splitting:

• One subgroup may contain mostly churners

• Another subgroup may contain mostly non-churners

Decision trees use a greedy learning approach. At each step, the model chooses the best split available at that moment instead of searching for the globally optimal tree. This makes trees computationally efficient, but it can sometimes lead to suboptimal solutions.

To measure split quality, decision trees use impurity metrics such as entropy, information gain, and Gini impurity. These concepts help quantify how “pure” or “mixed” a node is, which we’ll explore next.

📌 Pro Tip: A strong interview answer is

“Decision trees recursively choose splits that maximize purity improvement at each step.”

Entropy and Information Gain

To decide the best place to split the data, decision trees need a way to measure how “pure” or “mixed” a node is.

One common measure is called entropy.

Entropy measures the amount of uncertainty or randomness in a node.

Where:

• p_i = proportion (probability) of class iii in the node

For example:

• If 70% of observations belong to Class 1, then p_1=0.7

Intuition Behind Entropy

• High entropy → highly mixed classes

• Low entropy → mostly one class

• Zero entropy → completely pure node

For example:

• A node with 50% churners and 50% non-churners has high entropy

• A node with 95% churners has much lower entropy

The goal of a decision tree is to reduce entropy after every split.

Information Gain

Decision trees use a metric called Information Gain to evaluate how useful a split is.

Where:

• w_i = proportion of samples that go to child node i

• H(Child_i) = entropy of child node i

For example:

• If 30% of observations go left and 70% go right:

  • w1=0.3

  • w2=0.7

Information Gain measures how much uncertainty is reduced after splitting the data.

A split with high information gain creates cleaner and more homogeneous groups. For example:

• Before splitting, a node may contain a mixed set of churners and non-churners

• After splitting, the groups may become much more separated

The tree chooses the split that produces the highest information gain.

At every step, the decision tree is essentially asking: “Which split gives me the largest reduction in uncertainty?”

This process repeats recursively until the tree stops growing.

📌 Pro Tip: A strong interview explanation is:

“Entropy measures impurity, and information gain measures how much a split reduces that impurity.”

That demonstrates much stronger intuition than simply memorizing formulas.

Subscribe now

Gini Impurity

Another common way decision trees measure impurity is through Gini Impurity.

where:

• p_i = proportion of class (i) in the node

Like entropy, Gini Impurity measures how mixed the classes are within a node.

Intuition Behind Gini Impurity

• Low Gini → purer node

• High Gini → more mixed classes

• Gini = 0 → completely pure node

For example:

• A node containing only churners has Gini = 0

• A node with a 50/50 class split has high impurity

The decision tree tries to find splits that reduce Gini Impurity as much as possible.

Gini vs Entropy

Both entropy and Gini are impurity measures used to evaluate splits. In practice:

• They often produce very similar trees

• Both aim to create purer groups after splitting

Main differences:

• Gini is computationally simpler and slightly faster

• Entropy comes from information theory and is sometimes considered more interpretable mathematically

Most modern decision tree implementations are based on CART (Classification and Regression Trees), which typically use Gini Impurity by default.

The exact impurity metric is usually less important than the core intuition: Decision trees learn by repeatedly creating cleaner and more homogeneous groups.

📌 Pro Tip: If an interviewer asks:

“What’s the difference between entropy and Gini?”

you usually do not need a deeply mathematical answer. A strong practical answer is:

“Both measure impurity. Gini is computationally simpler and often used in practice, while entropy is based on information theory.”

Overfitting in Decision Trees

One of the biggest challenges with decision trees is overfitting.

Because trees can continue splitting the data repeatedly, they may eventually start memorizing the training data instead of learning generalizable patterns.

An overfit tree often:

• Achieves extremely high training accuracy

• Performs poorly on unseen data

• Learns noise instead of meaningful relationships

This usually happens when the tree becomes very deep and creates overly specific decision rules.

For example, instead of learning broad churn behavior, the tree may begin memorizing very small customer segments that only exist in the training dataset.

Decision trees are especially prone to overfitting because they are highly flexible models. They can continue splitting until nodes become almost completely pure, even if those splits are not useful for generalization.

This leads to a high-variance problem:

• Small changes in training data can produce very different trees

• Different splits may lead to entirely different model structures

• Predictions can become unstable

From a bias-variance perspective:

• Deep trees → low bias, high variance

• Shallow trees → higher bias, lower variance

The challenge is finding the right balance between:

• Learning meaningful patterns

• Avoiding memorization of noise

A very important machine learning insight is:

A perfectly accurate training model is not necessarily a good model.

What matters most is how well the model performs on unseen data.

📌 Pro Tip: A strong interview answer is:

“Decision trees overfit because they can continue splitting until they memorize the training data, which increases variance and hurts generalization.”

Controlling Tree Complexity

Since decision trees can easily overfit, we need ways to control how complex the tree becomes.

Instead of allowing the tree to grow indefinitely, we can introduce constraints that help the model generalize better to unseen data. Some common techniques include:

• Limiting the maximum depth of the tree

• Requiring a minimum number of samples before splitting

• Requiring a minimum number of samples in leaf nodes

• Pruning unnecessary branches

Max Depth: Max depth limits how deep the tree can grow. A shallow tree:

• Simpler model

• Lower variance

• Higher bias

A very deep tree:

• More flexible

• Higher variance

• Greater risk of overfitting

Choosing the right depth is part of balancing the bias-variance tradeoff.

Minimum Samples for Split: This parameter requires a minimum number of observations before creating another split. Without this restriction, the tree may create highly specific branches for very small groups of data.

Minimum Samples per Leaf: This controls the minimum number of observations allowed in a final leaf node. Larger leaf sizes usually:

• Produce smoother decision boundaries

• Reduce overfitting

• Improve stability

Pruning: Pruning removes branches that add little predictive value. The idea is simple:

Not every split improves generalization.

By removing weak or unnecessary branches, pruning helps create simpler and more robust trees.

A more complex tree is not always a better tree.

The goal is not to perfectly memorize the training data, but to learn patterns that generalize well to new data.

📌 Pro Tip: A strong interview answer is:

“We control tree complexity to reduce overfitting and improve generalization on unseen data.”

Advantages and Limitations of Tree-Based Models

Tree-based models became popular because they are powerful, flexible, and relatively easy to use on structured data.

One of their biggest strengths is their ability to model complex patterns without requiring heavy preprocessing or feature engineering.

Advantages

Tree-based models:

• Handle non-linear relationships naturally

• Capture feature interactions automatically

• Require little preprocessing

• Do not require feature scaling

• Work well with both numerical and categorical features

• Are relatively easy to interpret

Unlike linear models, decision trees can create flexible decision boundaries and adapt to different segments of the data automatically. For example:

• Customer churn behavior may differ across age groups

• Fraud patterns may depend on combinations of features rather than individual variables

Tree-based models can learn these relationships directly from the data.

Another major advantage is interpretability. Since the model makes decisions through sequential splits, it is often easier to explain predictions compared to more complex machine learning models.

Limitations

Despite their strengths, tree-based models also have important limitations.

Decision trees:

• Can overfit easily

• Are sensitive to small changes in data

• Often have high variance

• Use greedy splitting strategies

• May not generalize well individually

For example, a small change in the training data can sometimes produce a very different tree structure.

This instability is one reason individual decision trees are often replaced by ensemble methods such as:

• Random Forests

• Gradient Boosting

• XGBoost

These models combine many trees together to improve stability and predictive performance.

Important Insight: Individual decision trees are usually not the final goal. They are the foundation for many of the most powerful machine learning models used in production today.

📌 Pro Tip: A strong interview answer is:

“Decision trees are powerful because they handle non-linear relationships and interactions naturally, but individual trees can be unstable and prone to overfitting.”

Feature Importance and Interpretability

One reason tree-based models are popular is that they are often easier to interpret than many other machine learning models.

As the tree makes splits, it learns which features are most useful for separating the data. Features that:

• Reduce impurity significantly

• Appear near the top of the tree

• Are used frequently across splits

are often considered more important.

How Feature Importance Works

In decision trees, feature importance is typically calculated using impurity reduction.

The intuition is simple: Features that create better splits contribute more to the model.

For example:

• If “engagement score” consistently creates strong splits in a churn model

• The model may assign high importance to that feature

Why Interpretability Matters

Interpretability is especially valuable in domains where understanding model decisions is important. Examples include:

• Healthcare

• Finance

• Risk modeling

• Marketing

Unlike many black-box models, decision trees allow us to trace:

• Which features influenced a prediction

• What decision path the model followed

This makes tree-based models easier to explain to non-technical stakeholders.

Important Limitation

Feature importance should still be interpreted carefully.

Some importance methods can become biased toward:

• Features with many unique values

• High-cardinality variables

• Correlated features

This means: A feature with high importance is not always truly “causal” or inherently more valuable.

Important Insight: Interpretability is one of the major reasons tree-based models remain widely used, even when more complex models exist.

In many real-world applications:

• Explainability matters

• Stakeholder trust matters

• Transparency matters

📌 Pro Tip: “Feature importance in trees is typically based on impurity reduction, but importance scores should still be interpreted carefully because they can be biased.”

Decision Trees vs Logistic Regression

Important Insight: Neither model is universally better. In practice:

• Logistic regression is often a strong interpretable baseline

• Tree-based models are often better for capturing complex non-linear patterns

Many real-world machine learning workflows start with logistic regression and later move toward tree-based ensemble methods if additional predictive power is needed.

📌 Pro Tip: Logistic regression works well when relationships are relatively linear and interpretability is critical, while tree-based models are better at capturing complex non-linear interactions automatically.

Rapid Fire Interview Questions

1. Why are decision trees considered non-linear models?

Decision trees create decision boundaries through sequential splits instead of fitting a single linear equation. This allows them to model:

• Complex patterns

• Feature interactions

• Segment-specific behavior

A strong answer is:

“Trees create piecewise decision boundaries through recursive splits, which makes them non-linear.”

2. Why don’t decision trees require feature scaling?

Decision trees split based on feature thresholds rather than distances.

For example: “Is income > 50K?”

Since splits depend only on ordering, scaling usually does not affect the tree structure.

Distance-based models like KNN or K-Means require scaling much more than trees do.

3. Why are decision trees unstable?

Small changes in training data can lead to very different splits and tree structures. This happens because trees:

• Use greedy splitting

• Depend heavily on early split decisions

This instability contributes to high variance.

This is one of the major motivations behind Random Forests.

4. Entropy vs Gini — what’s the difference?

Both measure node impurity.

• Entropy is based on information theory

• Gini is computationally simpler and commonly used in CART trees

In practice:

• They often produce very similar trees

You usually do not need a deeply mathematical explanation in interviews. Focus on intuition.

5. Why do decision trees overfit?

Decision trees can continue splitting until they memorize the training data. Deep trees:

• Low bias

• High variance

This hurts generalization on unseen data.

“Trees overfit because they can create highly specific rules for small groups of observations.”

6. What is pruning?

Pruning removes branches that contribute little predictive value. The goal is to:

• Reduce complexity

• Improve generalization

• Prevent overfitting

Pruning is essentially a way of simplifying the model to improve performance on unseen data.

Final Takeaway

Decision trees are one of the most important foundations in machine learning.

Individually, they are simple and intuitive models that help explain core concepts such as:

• Non-linear decision boundaries

• Recursive splitting

• Entropy and impurity

• Overfitting and variance

• Feature importance

But their importance goes far beyond individual trees.

Modern machine learning systems often rely on powerful ensemble methods built on top of decision trees, including:

• Random Forests

• Gradient Boosting

• XGBoost

• LightGBM

Understanding decision trees deeply makes these advanced models much easier to learn later.

One of the biggest interview advantages of studying decision trees is that they force you to think about:

• Model behavior

• Trade-offs

• Generalization

• Interpretability

instead of simply memorizing algorithms.

What’s Next?

In the next issue of The Practical Data Scientist, we’ll move from individual decision trees to ensemble methods by exploring Random Forests and Bagging.

We’ll cover how combining multiple trees improves performance, reduces overfitting, and forms the foundation of many modern machine learning systems.

Keep building, keep learning—wishing you the best in your data journey.

In the previous issue of The Practical Data Scientist, we explored logistic regression — one of the most commonly discussed algorithms in data science interviews. But building a model is only half the problem.

The next challenge is understanding whether the model is actually good.

And surprisingly, this is where many candidates struggle in interviews.

A model with 99% accuracy can still completely fail in production. A model with lower accuracy might actually deliver far more business value. Understanding why this happens is what separates candidates who memorize machine learning from candidates who truly understand it.

In this issue, we’ll break down the most important evaluation metrics used in machine learning interviews:

• Accuracy

• Precision

• Recall

• F1 Score

• ROC-AUC

• PR Curves

• Threshold tuning

More importantly, we’ll focus on the intuition behind these metrics, the trade-offs they represent, and how to connect them to real business problems during interviews.

Subscribe now

The Confusion Matrix

Before discussing metrics like Precision, Recall, or ROC-AUC, we first need to understand the foundation behind all of them: the confusion matrix.

Almost every classification metric in machine learning is derived from four possible prediction outcomes:

Here’s what each term means:

• True Positive (TP):
  The model correctly predicts the positive class.

• True Negative (TN):
  The model correctly predicts the negative class.

• False Positive (FP):
  The model predicts positive when the actual class is negative.

• False Negative (FN):
  The model predicts negative when the actual class is positive.

Most evaluation metrics are simply different ways of measuring the trade-offs between these four quantities.

For example:

• Precision focuses on False Positives

• Recall focuses on False Negatives

• Accuracy combines all four values

Understanding the confusion matrix deeply makes evaluation metrics much easier to understand intuitively.

📌 Pro Tip: Many candidates memorize metric formulas without understanding what the errors actually represent.

Strong interview answers usually focus on:

• What kind of mistakes the model is making

• Which mistakes are more costly for the business

• How the metric captures those trade-offs

Accuracy

Accuracy is one of the simplest and most commonly used evaluation metrics.

It measures the proportion of predictions the model got correct:

Where:

• TP = True Positives

• TN = True Negatives

• FP = False Positives

• FN = False Negatives

Accuracy answers the question: “Out of all predictions, how many did the model get right?”

For balanced datasets, accuracy can be a useful metric. For example:

• Predicting pass/fail outcomes in a balanced exam dataset

• Predicting whether a customer clicks or not when both classes are similarly distributed

However, accuracy becomes misleading when dealing with imbalanced datasets.

Why Accuracy Can Fail

Imagine a fraud detection dataset where:

• 99% of transactions are not fraud

• 1% are fraud

A model that predicts: “Not fraud” for every transaction would still achieve:

Accuracy = 99%

Even though the model completely fails to detect fraud. This is why relying only on accuracy can lead to poor business decisions.

Accuracy is most useful when:

• Classes are balanced

• False positives and false negatives have similar costs

📌 Pro Tip: If an interviewer says:

“The model has 99% accuracy”

your first instinct should be:

“What does the class distribution look like?”

That usually signals an imbalanced dataset question.

Precision

Precision measures how often the model is correct when it predicts the positive class.

The formula for precision is:

Precision answers the question: “When the model predicts positive, how often is it actually correct?”

A high precision model makes fewer false positive mistakes. For example:

• In spam detection, low precision means important emails may be incorrectly marked as spam

• In marketing campaigns, low precision means promotions may be sent to users unlikely to respond

Precision Focuses on False Positives

Precision becomes important when false positives are costly. Examples:

• Spam filtering

• Ad targeting

• Recommendation systems

• Loan approval systems

Precision vs Conservativeness

A model can increase precision by becoming more conservative about predicting the positive class. For example:

• Instead of predicting 100 positives, it may only predict 20 highly confident positives

• Precision may improve, but recall may decrease

This is why precision should usually not be analyzed in isolation.

📌 Pro Tip: A strong interview answer is:

“Precision matters when acting on a prediction has a cost, so we want predicted positives to be highly reliable.”

Subscribe now

Recall

Recall measures how many actual positive cases the model successfully identifies.

The formula for recall is:

Recall answers the question: “Out of all actual positive cases, how many did the model correctly capture?”

A high recall model makes fewer false negative mistakes.

Recall Focuses on False Negatives

Recall becomes important when missing a positive case is costly or dangerous. Examples:

• Fraud detection

• Disease diagnosis

• Security systems

• Predicting equipment failures

For example:

• In fraud detection, a false negative means fraudulent activity goes undetected

• In medical diagnosis, a false negative could mean failing to detect a disease

Recall vs Aggressiveness

A model can improve recall by becoming more aggressive about predicting the positive class. For example:

• Lowering the prediction threshold may capture more true positives

• But this can also increase false positives

This is why improving recall often reduces precision.

📌 Pro Tip

A strong interview answer is:

“Recall matters when missing a positive case is more costly than raising false alarms.”

Precision vs Recall Tradeoff

Precision and recall are often in tension with each other. Improving one can reduce the other. For example:

• Lowering the prediction threshold makes the model more likely to predict the positive class

• This usually increases recall because more actual positives are captured

• But it can also increase false positives, reducing precision

Similarly:

• Raising the threshold makes the model more conservative

• Precision may improve

• But recall may decrease because more positives are missed

Intuition

• High Precision:
  “When the model says positive, it is usually correct.”

• High Recall:
  “The model captures most of the actual positive cases.”

In practice, the right balance depends on the business problem.

Real-World Examples

Fraud Detection

Recall is usually prioritized because missing fraud can be very costly.

Spam Detection

Precision is often prioritized because users do not want important emails incorrectly marked as spam.

Medical Diagnosis

High recall is critical because missing a disease can be dangerous.

Threshold Tuning

The prediction threshold directly controls the precision-recall tradeoff. For example:

• Lower threshold → higher recall, lower precision

• Higher threshold → higher precision, lower recall

This is why threshold selection is often a business decision rather than a purely technical one.

📌 Pro Tip: One of the biggest mistakes candidates make is saying:

“Precision is more important” or “Recall is more important.”

Strong candidates instead say:

“The right balance depends on the cost of false positives versus false negatives.”

F1 Score

The F1 Score combines precision and recall into a single metric. It is defined as the harmonic mean of precision and recall:

F1 Score answers the question:

“How well does the model balance precision and recall?”

Unlike accuracy, the F1 Score becomes especially useful when dealing with imbalanced datasets. A high F1 Score means:

• The model captures positive cases effectively

• The model also keeps false positives relatively low

Why Harmonic Mean?

The harmonic mean penalizes situations where one metric is much lower than the other. For example:

• Precision = 1.0

• Recall = 0.1

The F1 Score will still be low because the model is performing poorly on recall. This prevents models from appearing strong when only one metric is high.

When F1 Score Is Useful

F1 Score is commonly used when:

• Classes are imbalanced

• Both false positives and false negatives matter

• A balance between precision and recall is important

Examples:

• Fraud detection

• Churn prediction

• Medical diagnosis

Limitations of F1 Score

F1 Score does not consider true negatives directly. This means:

• Two models with very different true negative performance can still have similar F1 Scores

Because of this, F1 Score should not always be used alone.

📌 Pro Tip: A very common interview question is:

“Why do we use the harmonic mean instead of the regular average?”

The answer:

“Because the harmonic mean penalizes extreme imbalance between precision and recall.”

ROC Curve and ROC-AUC

The ROC Curve (Receiver Operating Characteristic Curve) measures how well a model separates the positive and negative classes across different classification thresholds.

The curve plots:

• True Positive Rate (TPR) on the Y-axis

• False Positive Rate (FPR) on the X-axis

Where:

Intuition

ROC-AUC measures the model’s ability to rank positive examples higher than negative examples. A higher ROC-AUC means:

• Better class separation

• Better ranking ability

• Stronger discrimination between classes

General interpretation:

• ROC-AUC = 0.5 → Random guessing

• ROC-AUC = 1.0 → Perfect separation

Why ROC Curves Matter

Different classification thresholds produce different:

• Precision

• Recall

• False positive rates

ROC curves allow us to evaluate model performance across all thresholds instead of relying on a single threshold like 0.5.

ROC-AUC evaluates ranking quality, not classification accuracy at a specific threshold.

This is why:

• A model can have a strong ROC-AUC

• But still perform poorly at a badly chosen threshold

One important thing to notice in the ROC curve is how different classification thresholds change the model’s behavior.

As the threshold decreases:

• The model becomes more aggressive about predicting the positive class

• Recall (TPR) increases

• False positives also increase

As the threshold increases:

• The model becomes more conservative

• Precision usually improves

• Recall decreases because fewer positives are predicted

Each point on the ROC curve represents a different threshold value.

This is why ROC curves are useful:
they show model performance across all possible thresholds instead of evaluating performance at just one fixed threshold like 0.5.

Limitations of ROC-AUC

ROC-AUC can sometimes appear overly optimistic on highly imbalanced datasets.

This is because:

• False Positive Rate may remain small even when the number of false positives is large in absolute terms

This is one reason Precision-Recall curves are often preferred for rare-event prediction problems.

📌 Pro Tip: A strong interview explanation is:

“ROC-AUC measures how well the model separates the classes independent of a specific threshold.”

That shows deeper understanding than simply saying:

“Higher AUC is better.”

Subscribe now

PR Curve vs ROC Curve

Both ROC Curves and Precision-Recall (PR) Curves are used to evaluate classification models across different thresholds.

However, they focus on different aspects of model performance.

ROC Curves plot:

• True Positive Rate (Recall)

• False Positive Rate

ROC-AUC measures how well the model separates the classes overall.

PR Curves plot:

• Precision

• Recall

Instead of focusing on class separation broadly, PR Curves focus specifically on performance for the positive class.

Why PR Curves Matter for Imbalanced Data

On highly imbalanced datasets, ROC-AUC can sometimes appear overly optimistic.

This happens because:

• The False Positive Rate may remain small even when the model generates many false positives in absolute terms

PR Curves expose this issue more clearly because precision directly accounts for false positives.

This makes PR Curves especially useful for:

• Fraud detection

• Disease prediction

• Rare-event modeling

• Anomaly detection

ROC Curves answer:

“How well can the model separate the classes overall?”

PR Curves answer:

“When the model predicts positive, how reliable are those predictions while still capturing positives?”

General rule:

• Balanced datasets → ROC-AUC is often sufficient

• Imbalanced datasets → PR Curves are usually more informative

📌 Pro Tip: Mentioning PR Curves for imbalanced datasets is a strong signal of practical machine learning experience.

A strong interview answer is:

“ROC-AUC can look strong even when precision is poor on rare-event problems, which is why PR curves are often preferred for imbalanced datasets.”

Threshold Tuning

Most classification models output probabilities, not final class predictions.

To convert probabilities into class labels, we choose a classification threshold.

For example:

• Probability > 0.5 → Positive class

• Probability ≤ 0.5 → Negative class

However, the default threshold of 0.5 is not always optimal.

Why Thresholds Matter

Changing the threshold directly changes:

• Precision

• Recall

• False Positive Rate

• False Negative Rate

This means the threshold controls the type of mistakes the model makes.

A lower threshold makes the model more aggressive about predicting the positive class. Effects:

• Higher recall

• More true positives

• More false positives

• Lower precision

A higher threshold makes the model more conservative. Effects:

• Higher precision

• Fewer false positives

• Lower recall

• More false negatives

Real-World Examples

Fraud Detection

Lower thresholds are often preferred because missing fraud can be very costly.

Spam Detection

Higher thresholds may be preferred to avoid incorrectly classifying important emails as spam.

Medical Diagnosis

Thresholds are often chosen carefully to balance patient safety and unnecessary interventions.

Threshold selection is usually a business decision, not just a modeling decision.

The “best” threshold depends on:

• Cost of false positives

• Cost of false negatives

• Operational constraints

• User experience

📌 Pro Tip: Many candidates assume the threshold is fixed at 0.5.

Mentioning threshold tuning immediately makes your answer more practical and business-oriented.

A strong interview answer is:

“The optimal threshold depends on the business trade-off between false positives and false negatives.”

Choosing the Right Metric

One important distinction to make is that the metrics discussed in this issue are model performance metrics. These metrics evaluate how well the machine learning model performs statistically.

However, business success is often measured using entirely different metrics, such as:

• Revenue

• Retention

• Conversion rate

• Customer lifetime value

• Cost savings

A model with strong ROC-AUC or F1 Score does not automatically guarantee strong business impact. That is why choosing the right business metric is equally important.

I covered this topic in a previous issue of The Practical Data Scientist on selecting the right business metrics for machine learning problems.

There is no universally “best” evaluation metric. The right metric depends entirely on:

• The business problem

• The type of mistakes that matter

• The operational costs of those mistakes

Different Problems Need Different Metrics

Business Context Matters

Two models can have:

• Similar accuracy

• Similar ROC-AUC

• Very different business impact

For example:

• A fraud model with high recall may detect more fraud

• But excessive false positives may overwhelm investigators

This is why evaluation metrics should never be discussed without business context.

Many candidates in an interview would answer:

“I would optimize for accuracy.”

without first understanding:

• Class imbalance

• Cost of mistakes

• Operational impact

Strong candidates instead ask:

• Which errors are more expensive?

• What happens when the model is wrong?

• What are the business constraints?

Metrics are not just mathematical tools.

They are ways of translating business priorities into model behavior.

📌 Pro Tip: This is one of the highest-signal interview sections.

The strongest answers usually:

1. Identify the business objective

2. Discuss trade-offs

3. Explain which mistakes matter most

4. Justify the metric choice accordingly

Common Interview Questions

Here are some of the most commonly asked evaluation metric interview questions:

1. Why is accuracy misleading on imbalanced datasets?

Accuracy can appear very high even when the model completely fails on the minority class.

For example:

• If 99% of transactions are non-fraud

• A model predicting “not fraud” for every case still achieves 99% accuracy

This is why metrics like Precision, Recall, F1 Score, and PR-AUC are often more useful for imbalanced problems.

📌 Pro Tip: Whenever you hear:

“The model has very high accuracy”

immediately think about:

• Class imbalance

• Minority class performance

• Cost of mistakes

2. Precision vs Recall — which is more important?

Neither metric is universally better.

The right choice depends on the business problem and the cost of errors.

Examples:

• Fraud detection → prioritize Recall

• Spam detection → prioritize Precision

📌 Pro Tip: Avoid absolute answers like:

“Recall is better.”

Strong candidates discuss trade-offs instead.

3. Why use F1 Score?

F1 Score balances Precision and Recall into a single metric.

It becomes useful when:

• Classes are imbalanced

• Both false positives and false negatives matter

📌 Pro Tip: Mention that F1 Score ignores true negatives directly.
That shows deeper understanding.

4. ROC-AUC vs PR-AUC

• ROC-AUC measures overall class separation ability

• PR-AUC focuses more directly on positive-class performance

PR-AUC is often preferred for highly imbalanced datasets because it captures precision-recall trade-offs more clearly.

📌 Pro Tip: Mentioning PR-AUC for rare-event prediction problems is a strong practical ML signal.

5. How do thresholds impact model performance?

Changing the threshold changes:

• Precision

• Recall

• False positive rate

• False negative rate

Lower thresholds:

• Increase recall

• Reduce precision

Higher thresholds:

• Increase precision

• Reduce recall

📌 Pro Tip: Mention that threshold tuning is usually a business decision, not just a modeling decision.

6. Which metric would you optimize for?

The correct answer depends on:

• Business objectives

• Operational constraints

• Cost of false positives

• Cost of false negatives

There is rarely one universally correct metric.

📌 Pro Tip: The strongest candidates:

1. Clarify the business goal

2. Discuss trade-offs

3. Justify the metric choice using business impact

Final Takeaway

Evaluation metrics are not just mathematical formulas.

They are tools for understanding:

• How a model behaves

• What kinds of mistakes it makes

• Whether those mistakes align with business goals

One of the biggest differences between beginner and experienced data scientists is the ability to think beyond accuracy and reason about trade-offs.

In interviews, strong candidates usually:

• Explain metrics intuitively

• Connect them to real-world business problems

• Discuss false positives vs false negatives

• Justify why a particular metric matters

The goal is not to memorize formulas.

The goal is to understand what the model is optimizing for — and whether that aligns with the actual problem being solved.

What’s Next?

In the next issue of The Practical Data Scientist, we’ll move beyond linear models and step into the world of tree-based models with Decision Trees

Decision trees are one of the most important foundations for understanding modern machine learning models like Random Forests and Gradient Boosting.

As always, the focus will be on intuition, practical understanding, and interview-oriented thinking rather than memorizing formulas.

Keep building, keep learning—wishing you the best in your data journey.

What Is Logistic Regression?

Logistic regression is a supervised machine learning algorithm used for classification problems. It predicts the probability that an observation belongs to a particular class.

For example, it can be used to predict:

• Whether a customer will churn

• Whether a transaction is fraudulent

• Whether a user will click on an ad

Unlike linear regression, which predicts continuous values, logistic regression predicts probabilities between 0 and 1.

It does this using the sigmoid function:

The predicted probability is then converted into a class label using a threshold (commonly 0.5). For example:

• Probability > 0.5 → Class 1

• Probability ≤ 0.5 → Class 0

One of the biggest advantages of logistic regression is interpretability, which is why it remains one of the most commonly asked algorithms in data science interviews.

Subscribe now

The Core Intuition

Logistic regression first creates a weighted combination of input features:

This value is then passed through the sigmoid function:

The sigmoid function transforms the output into a smooth S-shaped curve, allowing the model to map predictions into probabilities.

Key intuition:

• Higher values of z push predictions closer to 1

• Lower values of z push predictions closer to 0

This makes logistic regression effective for binary classification problems while keeping the model simple and interpretable.

Key Interview Concept: Log-Odds

One of the most important concepts in logistic regression interviews is log-odds.

Instead of modeling probabilities directly, logistic regression models the logarithm of the odds:

Here:

• p represents the probability of the positive class

• p/(1−p)​ represents the odds

• Taking the logarithm converts the relationship into a linear equation

But why do we use log-odds instead of probabilities directly?

Probabilities are bounded between 0 and 1, making them difficult to model with a linear equation. A linear model could otherwise produce invalid predictions like 1.5 or -0.2.

By converting probabilities into odds and then taking the logarithm, the output range becomes:

• Negative infinity to positive infinity

This makes it suitable for linear modeling.

Intuition

• Probability = 0.5 → log-odds = 0

• Probability > 0.5 → positive log-odds

• Probability < 0.5 → negative log-odds

Another reason this formulation is important is interpretability:

• A positive coefficient increases the odds of the positive class

• A negative coefficient decreases the odds

A common interview question is:

“How do you interpret logistic regression coefficients?”

In logistic regression:

• A one-unit increase in a feature changes the log-odds by the value of its coefficient

• Exponentiating the coefficient, eβe^{\beta}eβ, gives the odds ratio

For example:

• If e^β = 2, the odds are doubled

• If e^β = 0.5, the odds are reduced by half

This impacts the probability p indirectly through the sigmoid function:

• Positive coefficients increase the probability of the positive class

• Negative coefficients decrease it

However, the change in probability is not constant. It depends on the current value of p.

For example:

• Increasing log-odds from 0 to 1 changes probability from 0.50 to 0.73

• Increasing log-odds from 4 to 5 changes probability only from 0.98 to 0.99

This happens because the sigmoid curve becomes flatter near 0 and 1.

How Logistic Regression Is Trained

Logistic regression is trained by finding the coefficients that best separate the classes.

Instead of using Mean Squared Error (MSE), logistic regression uses a loss function called Log Loss (or Binary Cross-Entropy):

Where:

• y is the actual class label

• p is the predicted probability

The loss function penalizes incorrect predictions:

• Predicting a high probability for the wrong class results in a large penalty

• Correct and confident predictions result in lower loss

The model then uses optimization algorithms such as Gradient Descent to minimize this loss and learn the best coefficients.

Why Not Use MSE?

Mean Squared Error is commonly used in linear regression, but it is not ideal for logistic regression because:

• The sigmoid function makes the optimization problem non-linear

• MSE can lead to non-convex loss surfaces

• Log Loss provides better probabilistic interpretation and optimization behavior

  Subscribe now

Important Interview Topics

Assumptions

Logistic regression makes a few important assumptions:

• Linear relationship with log-odds:
  The features should have a linear relationship with the log-odds of the target, not necessarily with the target itself.

• Independent observations:
  Each observation should be independent. Repeated or highly related observations can bias the model.

• Low multicollinearity:
  Features should not be highly correlated with each other, as this can make coefficients unstable and difficult to interpret.

• Limited influence of outliers:
  Extreme values can heavily influence the decision boundary and coefficient estimates.

📌 Pro Tip: A common mistake is saying logistic regression assumes a linear relationship with the target variable.
The correct answer is:

“Logistic regression assumes a linear relationship between the features and the log-odds of the target.”

Evaluation Metrics

Accuracy alone is often misleading, especially with imbalanced datasets.

Common evaluation metrics include:

• Precision

• Recall

• F1 Score

• ROC-AUC

Interview tip:

• Fraud detection problems usually prioritize recall

• Marketing targeting problems often prioritize precision

I’ll cover evaluation metrics in much more detail in a separate article since they are extremely important for interviews.

📌 Pro Tip: When discussing metrics, always tie them back to business impact instead of giving textbook definitions only.

Imbalanced Data

In many real-world datasets, one class appears much less frequently than the other.

Examples:

• Fraud transactions

• Rare diseases

• Customer churn

A model predicting only the majority class may still achieve high accuracy, making accuracy unreliable.

Common solutions:

• Oversampling the minority class

• Undersampling the majority class

• Using class weights

• Adjusting the classification threshold

📌 Pro Tip: If an interviewer mentions “99% accuracy,” immediately think about class imbalance and ask about the class distribution.

Regularization

Regularization helps prevent overfitting by penalizing large coefficients.

Two common types:

• L1 Regularization (Lasso):
  Can shrink some coefficients to zero, effectively performing feature selection.

• L2 Regularization (Ridge):
  Reduces coefficient magnitudes smoothly without eliminating features entirely.

Regularization is especially useful when:

• There are many features

• Features are correlated

• The model is overfitting training data

📌 Pro Tip: A strong interview answer is:

“L1 helps with feature selection, while L2 helps stabilize the model by shrinking coefficients.”

Decision Threshold

Logistic regression outputs probabilities, which are converted into class labels using a threshold.

Default threshold:

• Probability > 0.5 → Positive class

• Probability ≤ 0.5 → Negative class

However, the threshold can be adjusted depending on business objectives.

Examples:

• Lower threshold → higher recall

• Higher threshold → higher precision

📌 Pro Tip: Many candidates forget that the threshold is configurable. Mentioning threshold tuning shows practical understanding beyond theory.

Multicollinearity

Multicollinearity occurs when features are highly correlated with each other.

Problems caused by multicollinearity:

• Unstable coefficient estimates

• Difficulty interpreting feature importance

• Increased variance in predictions

Common ways to detect it:

• Correlation matrix

• Variance Inflation Factor (VIF)

Possible solutions:

• Remove correlated features

• Combine related variables

• Apply regularization techniques

📌 Pro Tip: If interpretability matters, always mention multicollinearity because unstable coefficients can lead to misleading business conclusions.

When to Use Logistic Regression

Logistic regression is often one of the best starting models for classification problems because it is simple, fast, and highly interpretable.

It works especially well when:

• The relationship between features and the target is relatively linear

• Interpretability is important

• The dataset is not extremely large or complex

• Probability estimates are needed

Common real-world applications include:

• Customer churn prediction

• Fraud detection

• Credit risk modeling

• Ad click prediction

• Medical diagnosis

One major advantage of logistic regression is explainability. Unlike many complex machine learning models, its coefficients can be interpreted directly, making it popular in industries where transparency matters.

However, logistic regression may struggle when:

• Relationships are highly non-linear

• There are complex feature interactions

• The decision boundary is not approximately linear

In such cases, models like decision trees, random forests, or gradient boosting may perform better.

📌 Pro Tip: A strong interview answer is “Logistic regression is usually my first baseline model because it is interpretable, fast to train, and provides strong performance on many structured datasets.”

Common Interview Questions

Here are some of the most commonly asked logistic regression interview questions:

1. Why not use linear regression for classification?

Linear regression can produce predictions outside the range of 0 and 1, making it unsuitable for probability estimation. Logistic regression solves this using the sigmoid function.

2. Why does logistic regression use log-odds?

Log-odds transform probabilities from a bounded range (0,1)(0,1)(0,1) to an unbounded range (−∞,+∞)(-\infty, +\infty)(−∞,+∞), allowing a linear relationship with the features.

3. How do you interpret logistic regression coefficients?

• Coefficients represent changes in log-odds

• e^β represents the odds ratio

• Positive coefficients increase the probability of the positive class

4. What happens when features are highly correlated?

Highly correlated features can lead to unstable coefficient estimates and poor interpretability. This issue is known as multicollinearity.

5. How do you handle imbalanced datasets?

Common approaches include:

• Class weights

• Oversampling / undersampling

• Threshold tuning

• Using better evaluation metrics such as Precision, Recall, and F1-score

6. What is regularization and why is it important?

Regularization prevents overfitting by penalizing large coefficients.

• L1 regularization can perform feature selection

• L2 regularization shrinks coefficients smoothly

7. What is the role of the decision threshold?

The threshold converts predicted probabilities into class labels. Adjusting the threshold changes the balance between precision and recall.

📌 Pro Tip: In interviews, avoid giving one-line textbook answers. Explain:

1. The intuition

2. The business impact

3. The trade-offs