AI training data is the single biggest reason a model works or quietly falls apart, and this tutorial explains where datasets come from, what a label really is, how bias sneaks in, and why the train/validation/test split is the habit that keeps you honest. If you remember one thing from the whole AI Foundations chapter, make it this: the data you feed a model matters more than the model itself.

“Real stupidity beats artificial intelligence every time.”

Terry Pratchett, Hogfather

Last Updated: July 2026 | Tested on: Python 3.14.6, scikit-learn 1.9.0 | Difficulty: Beginner | Reading Time: 17 minutes

Here is the everyday version. Imagine teaching a child what a dog is. You do not hand them a rulebook about fur length and ear shape. You point at dogs on hundreds of walks and say “dog”, and one day it just clicks. Now imagine half the time you pointed at a cat and said “dog” by mistake. The child would end up confused, calling cats dogs with total confidence. A machine learning model learns the exact same way, from examples, and it is just as helpless if the examples are wrong. That pile of examples is the training data, and this post is about where it comes from and how it goes wrong.

Prerequisites

Garbage In, Garbage Out

A model is a function with learnable numbers, and those numbers are shaped entirely by the examples it sees during training. There is no hidden source of wisdom. If the examples are clean, balanced, and correctly labeled, the model has a fair shot. If the examples are wrong, skewed, or missing whole groups of people, the model faithfully copies those flaws and hands them back to you at scale. This is the first law of machine learning, older than any framework: garbage in, garbage out.

Think about a chef learning only from photos of overcooked food. They will confidently plate overcooked food, because that is all “correct” ever looked like to them. The chef is not broken. The training set was. Most real-world ML failures are data failures wearing a model costume, which is why senior engineers spend far more time on the data than on the algorithm. A fancier model on bad data just gives you a more confident wrong answer.

Where Training Data Actually Comes From

So where do these piles of examples come from? In practice, training data flows in from five main places, and most real projects mix several of them:

• Logged behavior: the clicks, purchases, and searches your product already records. A streaming service knows what you watched to the end. This is the richest and most common source, and it grows for free as people use the product.
• Public datasets: ready-made, downloadable collections. Kaggle Datasets, Hugging Face Datasets, and the UCI Machine Learning Repository are the go-to homes; government open-data portals are another. Great for learning and benchmarking, though rarely a perfect fit for your exact problem.
• Web scraping: pulling data from public web pages. Powerful but the ethical and legal minefield of the bunch: respect robots.txt, site terms, and rate limits. We cover the how and the etiquette in the web scraping tutorial.
• Human labeling: paying people to tag examples (“this review is positive”, “this scan shows a fracture”). Slow and expensive, but often the only way to get high-quality labels for a new problem.
• Synthetic data: examples generated by a program or another model. It is rising fast because it is cheap and sidesteps privacy issues, though it can bake in the quirks of whatever created it. Treat it as a useful supplement, not a free lunch.

Whatever the source, the data usually ends up as a plain table: one row per example, columns for the features, and one column for the answer. Let us load a real public dataset that ships with scikit-learn and look at its shape, so “dataset” stops being an abstract word.

📄 peek_dataset.py: a real dataset is just a table with an answer column

from sklearn.datasets import load_breast_cancer

data = load_breast_cancer()
X, y = data.data, data.target

print("A dataset is just a table of numbers with an answer column")
print(f"Rows (patients):     {X.shape[0]}")
print(f"Columns (features):  {X.shape[1]}")
print(f"First 3 feature names: {[str(n) for n in data.feature_names[:3]]}")
print(f"Answer column values:  {sorted(set(y.tolist()))}  "
      f"({data.target_names[0]}=0, {data.target_names[1]}=1)")
print(f"First patient, first 3 features: {X[0][:3].round(2).tolist()}  -> label {y[0]}")

▶ Output

A dataset is just a table of numbers with an answer column
Rows (patients):     569
Columns (features):  30
First 3 feature names: ['mean radius', 'mean texture', 'mean perimeter']
Answer column values:  [0, 1]  (malignant=0, benign=1)
First patient, first 3 features: [17.99, 10.38, 122.8]  -> label 0

What happened here: This famous public dataset is 569 rows and 30 feature columns, each row a tumor described by measurements like mean radius and mean texture, plus one answer column that says malignant (0) or benign (1). That is the whole shape of supervised training data: a grid of features and a column of known answers. When someone says “we trained on a million examples”, they mean a table like this with a million rows. The model never sees the word “tumor” or understands medicine; it only ever sees numbers and the answer beside them.

Labels: What Supervision Really Means

That answer column has a name: the label. Supervised learning just means every training example comes with its correct answer attached, so the model can compare its guess to the truth and adjust. The “supervision” is those labels. No labels, no supervision, and you are in unsupervised territory instead (grouping similar things with no answer key), which we cover in the machine learning introduction.

Here is the part beginners underestimate: labels are expensive. Features are often free (you already logged them), but someone has to decide the correct answer for every single row. A radiologist tagging scans, a fluent speaker marking translations, a moderator judging whether a comment is toxic. This is slow, costs real money, and humans disagree with each other more than you would expect. Say a labeler named Aditi marks a borderline review as negative while her colleague Anvay marks the same one neutral. Multiply that across a hundred thousand rows and you get label noise: labels that are simply wrong. And wrong labels do real damage, as the next demo shows.

When Labels Lie: A Label-Noise Demo

Let us make label noise concrete. We start with a clean, learnable dataset and hold out a clean test set as our honest judge. Then we deliberately flip a growing share of the training labels (turning some 0s into 1s and back) and watch what it does to accuracy. The test set is never corrupted, so any drop is caused purely by teaching the model wrong answers.

📄 label_noise.py: flip training labels, measure the damage

import numpy as np
from sklearn.datasets import make_classification
from sklearn.neighbors import KNeighborsClassifier
from sklearn.model_selection import train_test_split

rng = np.random.default_rng(0)

# A learnable dataset with a bit of overlap between the two classes
X, y = make_classification(n_samples=800, n_features=6, n_informative=4,
                           n_redundant=0, class_sep=1.0, random_state=0)

# Hold out a CLEAN test set we never corrupt: this is the honest judge
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=0)

print(f"{'Label noise':<14}{'Flipped labels':<16}{'Test accuracy'}")
print("-" * 44)
for noise in (0.0, 0.10, 0.20, 0.30, 0.40):
    y_dirty = y_train.copy()
    n_flip = int(noise * len(y_dirty))
    flip_idx = rng.choice(len(y_dirty), size=n_flip, replace=False)
    y_dirty[flip_idx] = 1 - y_dirty[flip_idx]        # flip 0<->1

    model = KNeighborsClassifier(n_neighbors=5)
    model.fit(X_train, y_dirty)
    acc = model.score(X_test, y_test)                # judged on CLEAN labels
    print(f"{noise:<14.0%}{n_flip:<16}{acc:.3f}")

▶ Output

Label noise   Flipped labels  Test accuracy
--------------------------------------------
0%            0               0.912
10%           56              0.883
20%           112             0.804
30%           168             0.717
40%           224             0.654

What happened here: With clean labels the model hits 91.2 percent. Flip just 10 percent of the training labels and it slips to 88.3 percent. Push the noise to 30 percent and accuracy craters to 71.7 percent, and at 40 percent it is barely better than a coin-aware guess. Nothing about the model changed between runs. The features were identical every time. The only thing we damaged was the answer key, and the model got proportionally worse. This is why teams obsess over label quality and why "just collect more data" is bad advice if the new data is sloppily labeled. More wrong answers do not average out into a right one.

Train, Validation, and Test: The Exam-Paper Rule

You may have noticed both demos held out a chunk of data they never trained on. That is not optional; it is the most important discipline in all of machine learning. Think of a student and an exam. The training set is the textbook they study from. The test set is the sealed exam paper. If the student sees the exact exam questions while studying, a perfect score proves nothing, because they memorized instead of learned. In ML, letting the model see the test data before the final check is called leakage, and it makes weak models look brilliant right up until they meet real users.

So we split the data three ways. The training set (around 60 percent) is where the model learns its weights. The validation set (around 20 percent) is a practice exam you can look at as often as you like, to tune settings and pick the best version of the model. The test set (around 20 percent) stays locked in a drawer and is touched exactly once, at the very end, for a single honest estimate of real-world performance. The diagram below shows the flow and the one rule that holds it all together.

Tip: click the diagram to open it full screen, then use the zoom and pan controls for a closer look.

Here is the split in code. We carve the data into train, validation, and test, use the validation set to choose a tree depth (a knob that controls how much the model can memorize), then open the test set only once for the final number.

📄 three_way_split.py: learn, tune, then judge once

from sklearn.datasets import make_classification
from sklearn.tree import DecisionTreeClassifier
from sklearn.model_selection import train_test_split

X, y = make_classification(n_samples=1000, n_features=10, n_informative=6,
                           random_state=42)

# Split ONCE, three ways: 60% train, 20% validation, 20% test
X_tmp, X_test, y_tmp, y_test = train_test_split(
    X, y, test_size=0.20, random_state=42)
X_train, X_val, y_train, y_val = train_test_split(
    X_tmp, y_tmp, test_size=0.25, random_state=42)   # 0.25 * 0.8 = 0.20

print(f"Train rows:      {len(y_train)}  (model learns weights)")
print(f"Validation rows: {len(y_val)}  (pick the best setting)")
print(f"Test rows:       {len(y_test)}  (touched once, at the very end)")
print()
print(f"{'tree depth':<12}{'train acc':<12}{'val acc'}")
print("-" * 32)

best_depth, best_val = None, -1.0
for depth in (2, 4, 6, 10, None):
    m = DecisionTreeClassifier(max_depth=depth, random_state=42)
    m.fit(X_train, y_train)
    train_acc = m.score(X_train, y_train)
    val_acc = m.score(X_val, y_val)
    label = str(depth) if depth is not None else "none"
    print(f"{label:<12}{train_acc:<12.3f}{val_acc:.3f}")
    if val_acc > best_val:
        best_depth, best_val = depth, val_acc

# Only NOW do we unlock the test set, exactly once
final = DecisionTreeClassifier(max_depth=best_depth, random_state=42)
final.fit(X_train, y_train)
print(f"\nBest depth on validation: {best_depth}")
print(f"Honest test accuracy: {final.score(X_test, y_test):.3f}")

▶ Output

Train rows:      600  (model learns weights)
Validation rows: 200  (pick the best setting)
Test rows:       200  (touched once, at the very end)

tree depth  train acc   val acc
--------------------------------
2           0.827       0.835
4           0.897       0.885
6           0.945       0.865
10          1.000       0.885
none        1.000       0.885

Best depth on validation: 4
Honest test accuracy: 0.805

What happened here: Watch the two accuracy columns pull apart. At depth 10 (and unlimited depth), the tree scores a perfect 1.000 on the training data. It has literally memorized every row. But its validation accuracy is only 0.885, which tells the truth: memorizing is not learning. The validation set is what lets us catch that and pick a sensible depth of 4 instead of the show-off deep tree. Then, and only then, we open the test set once and get 0.805, an honest estimate of how this model behaves on data it has never seen. If we had tuned on the test set instead, we would have fooled ourselves. The exam paper stays sealed until the end.

Bias In, Bias Out

Label noise is random damage. Bias is worse, because it is patterned damage: the data leans a certain way, so the model leans that way too, consistently and confidently. If your training data mostly describes one kind of person, place, or situation, the model works well for that slice and poorly for everyone else, and the accuracy score can still look fine on average. Two well-documented public cases make this vivid.

• The resume screener that learned to prefer men. A large tech company built an internal tool to rank job applicants and reportedly scrapped it around 2018 after finding it downgraded resumes that mentioned women's activities. It was trained on a decade of past hires in a male-dominated field, so it simply learned the historical pattern and repeated it. The model was not malicious; the data was skewed, and the model was loyal to the skew.
• Face analysis that failed on darker skin. The widely cited Gender Shades study (2018) showed several commercial face-classification systems had error rates under 1 percent for lighter-skinned men but well over 20 percent for darker-skinned women. The training images had skewed toward lighter faces, so the systems were far less accurate on the groups that were underrepresented.

The fix starts with knowing what is in your data. That is why the field now writes documentation for datasets and models: a datasheet describes how a dataset was collected, who is in it, and its known gaps, while a model card records what a model was trained on, how it was evaluated, and where it should not be used. We come back to model cards when you ship your own project. For now the takeaway is simple: check who and what your data represents before you trust the average score, because the average hides the group it is failing.

Licensing and PII: What You May Train On

Being able to get data is not the same as being allowed to use it. Two guardrails matter from day one. First, licensing: a public dataset still carries a license that says what you may do with it. Some allow any use, some are research-only, some forbid commercial use, and scraped web content may be copyrighted even though it is visible to everyone. Read the license before you build a product on it.

Second, personal data: names, emails, health records, and location trails are personally identifiable information (PII), and privacy laws such as GDPR-style rules give people rights over it. At the time of writing, the safe defaults are to collect only what you truly need, remove or mask identifiers you do not, and never quietly repurpose data people gave you for something else.

None of this requires a law degree. The common-sense version: would the person who created this data be surprised or upset to learn you trained a model on it? If yes, stop and get the licensing and consent right first. Doing this early is far cheaper than unwinding a model that learned from data you were never allowed to use.

Common Mistakes

• Chasing more data before checking label quality. As the noise demo showed, doubling a sloppily labeled dataset can make a model worse, not better. Fix the labels first, then scale.
• Tuning on the test set. Peeking at test results to pick settings quietly leaks the answer key. Keep a separate validation set for all tuning and touch the test set once.
• Trusting the average accuracy. A high overall score can hide total failure on an underrepresented group. Always look at performance sliced by the groups that matter.
• Assuming public means free to use. Visible on the web is not the same as licensed for your product. Check the license and the presence of personal data before training.

Best Practices

• Split before you touch anything. Make the train/validation/test split the very first step, so no test information can leak into learning or tuning.
• Audit the data before the model. Count the classes, check for missing values, and look at who is represented. An hour here saves days of confused debugging later.
• Measure label quality. Have two people label the same sample and compare; low agreement means your labels are noisy and your ceiling is low.
• Document your dataset. Write down where it came from, its license, and its known gaps. Your future self and your teammates will thank you.

Conclusion

You now hold the mental model that the rest of the AI journey leans on. AI training data is the examples a model learns from, and its quality sets the ceiling on everything the model can ever do. You saw where data comes from (logged behavior, public datasets, scraping, human labeling, and synthetic data), what a label is and why good ones cost real money, how label noise steadily wrecks accuracy, why bias in the data becomes bias in the predictions, and how the train/validation/test split keeps you honest. Above all, remember the first law: garbage in, garbage out. Better data usually beats a fancier model.

Next up in the AI Foundations chapter, you will look at the hardware that actually runs all this training, and why matrix math loves parallel chips. If you want the full path from Python basics through deep learning and GenAI, follow it in order from the Python + AI/ML tutorial series home.

Frequently Asked Questions

Interview Questions on AI Training Data

The same ideas as they show up in real interviews, framed as scenarios you can practice out loud.

Q: What does "garbage in, garbage out" mean for machine learning?

It means a model can only be as good as the data it learned from. The model has no independent knowledge; it just captures patterns in the training examples. If those examples are wrong, biased, or unrepresentative, the model reproduces those flaws confidently, so investing in clean, well-labeled, representative data usually pays off more than choosing a more complex algorithm.

Q: Why do we split data into training, validation, and test sets instead of just two?

The training set fits the model and the test set gives the final honest score, but you also need somewhere to make tuning decisions like choosing hyperparameters. If you tune against the test set, its results leak into your choices and stop being an unbiased estimate. The validation set is that middle ground: you can look at it repeatedly to pick the best model, while the test set stays sealed for a single final measurement.

Q: What is data leakage and how do you prevent it?

Data leakage is when information the model should not have at prediction time sneaks into training, making offline scores look great but real performance poor. Classic causes are tuning on the test set, fitting preprocessing on the full dataset before splitting, and features that secretly encode the answer. Prevent it by splitting first, learning every transformation from the training set alone, and touching the test set only once at the very end.

Q: How does bias enter a model, and how would you catch it?

Bias enters through the data: if the training set overrepresents one group or reflects past unfair decisions, the model learns and repeats that pattern while overall accuracy can still look fine. You catch it by measuring performance separately for each important group rather than trusting the average, and by documenting how the dataset was collected and who is in it. If a group is missing or thin in the data, expect the model to fail on it.

Q: A teammate wants to boost accuracy by adding a large batch of cheaply labeled data. What is your concern?

My concern is label quality. Cheaply labeled data often carries high label noise, and as the noise demo shows, adding wrong answers can lower accuracy rather than raise it. I would first measure the new data's label agreement on a sample, check that it represents the same population as the target use case, and confirm its license and privacy status before adding a single row.

Series: Python + AI/ML Cookbook, Part 5: AI Foundations

Further reading: for the full reference, see the official Python documentation.

Related Posts

Previous: What is an AI Model? Weights, Training, and Inference

Next: Central Processing Unit (CPU) vs Graphics Processing Unit (GPU) vs Tensor Processing Unit (TPU) vs Neural Processing Unit (NPU): AI Hardware Explained Simply

Series Home: Python + AI/ML Tutorial Series