This Kaggle project walks you through a real getting-started competition from the first look at the data to a submission file you upload for a live leaderboard score. You will explore the data, split it the right way, build an honest baseline, add features, cross-validate, train one boosted model, and dodge the two data leaks that quietly wreck most first attempts. Everything runs on a laptop in seconds, no Graphics Processing Unit (GPU) and no paid account.

“The public leaderboard is a lie until your own cross-validation agrees with it. Trust your local score, not the ranking.”

Kaggle competition folklore

Last Updated: July 2026 | Tested on: Python 3.14.6, scikit-learn 1.9.0 | Difficulty: Intermediate | Reading Time: 24 minutes

A Kaggle competition is like a public cooking contest where every cook gets the same box of vegetables and the judges score the same dish against a hidden rubric. Everyone starts equal, the scoring is honest, and your result sits on a public board next to thousands of others. A first Kaggle project is a fantastic way to learn, because the feedback is brutal and immediate. It is also a fantastic way to fool yourself, because it is very easy to build a model that looks brilliant on your screen and then collapses the moment the hidden judges taste it.

The competition this Kaggle project is built around is the classic binary survival problem: given a passenger’s class, sex, age, fare, and family size, predict whether they survived. At the time of writing this is the Titanic getting-started competition on Kaggle, and if that one ever retires, the Spaceship Titanic competition or any current “Getting Started” contest works with the exact same code and the exact same workflow. The dataset does not matter. The habit does.

What We Are Building

By the end you have a folder that runs top to bottom and produces a valid submission.csv, plus a clean story you can tell in an interview: here is my baseline, here is how much each improvement bought me, here is how I proved I was not fooling myself. The whole thing follows one repeatable shape you can reuse on any competition or any classification task at work. The diagram below is that shape in one picture.

One honest note up front about how this Kaggle project runs. In the real competition you download train.csv and test.csv from the competition Data tab. So that every code block here always runs and can never break because a download link died, we generate a stand-in dataset in code with the exact same columns as the real Titanic file. The moment you swap in the real CSV, every line below works unchanged.

Prerequisites

The Kaggle Competition Loop

Read the order carefully, because the order is the whole point. You explore, then you split before you touch anything, then you get an honest baseline, then you validate with cross-validation, and only then do you start improving. The loop between Improve and Validate is where most of your time goes: try an idea, re-check it on cross-validation, keep it only if the honest number moves. The last diamond is the reality check that catches leakage: if your leaderboard score is far worse than your local score, you leaked information somewhere and the model was never as good as it looked.

Step 1: Get the Data and Explore

In a real entry you click Join Competition, download the files, and load them with pd.read_csv("train.csv"). Here we generate titanic_train.csv with the same columns. Run this once to create the file everyone below reads.

📄 make_data.py: write a Titanic-style training file (stand-in for the real download)

import numpy as np
import pandas as pd

rng = np.random.default_rng(42)
N = 891  # same size as the real Titanic training set

pclass = rng.choice([1, 2, 3], size=N, p=[0.24, 0.21, 0.55])
sex = rng.choice(["male", "female"], size=N, p=[0.65, 0.35])
age = np.clip(rng.normal(29, 13, size=N), 0.5, 80).round(1)
sibsp = rng.choice([0, 1, 2, 3, 4], size=N, p=[0.68, 0.23, 0.05, 0.02, 0.02])
parch = rng.choice([0, 1, 2, 3], size=N, p=[0.76, 0.13, 0.09, 0.02])
fare = np.round(np.where(pclass == 1, rng.normal(84, 40, N),
                np.where(pclass == 2, rng.normal(21, 10, N),
                         rng.normal(13, 8, N))).clip(4, 512), 2)
embarked = rng.choice(["S", "C", "Q"], size=N, p=[0.72, 0.19, 0.09])

# A survival signal with a Sex-by-Class interaction: third-class women fared
# far worse than "female + third class" added up would suggest. A straight-line
# model cannot capture that product, but a tree can.
family = sibsp + parch + 1
is_child = age < 14
sweet_spot = (family >= 2) & (family <= 4)
sexclass = {("female", 1): 3.0, ("female", 2): 2.4, ("female", 3): -1.4,
            ("male", 1): 0.3, ("male", 2): -1.1, ("male", 3): -2.5}
base = np.array([sexclass[(s, int(p))] for s, p in zip(sex, pclass)])
child_third = is_child & (pclass == 3)

logit = 1.3 * (base - 0.010 * age + 0.003 * fare
               + 1.2 * is_child + 2.2 * child_third
               + 0.6 * sweet_spot - 0.6 * (family > 4))
prob = 1 / (1 + np.exp(-logit))
survived = (rng.random(N) < prob).astype(int)

# Real files have holes. Punch missing values into Age and Embarked.
age_obj = age.astype(object)
age_obj[rng.random(N) < 0.20] = np.nan
emb_obj = embarked.astype(object)
emb_obj[rng.random(N) < 0.005] = np.nan

df = pd.DataFrame({"PassengerId": np.arange(1, N + 1), "Survived": survived,
                   "Pclass": pclass, "Sex": sex, "Age": age_obj, "SibSp": sibsp,
                   "Parch": parch, "Fare": fare, "Embarked": emb_obj})
df.to_csv("titanic_train.csv", index=False)
print(f"Wrote titanic_train.csv: {df.shape[0]} rows, {df.shape[1]} columns")
print("Overall survival rate:", round(df["Survived"].mean(), 3))

▶ Output

Wrote titanic_train.csv: 891 rows, 9 columns
Overall survival rate: 0.404

Every Kaggle project starts the same way: explore before modelling anything. Exploring is not busywork, it is how you decide which columns even matter and where the missing values hide. Three cheap moves answer most of it: count missing values, and check the survival rate broken down by the two columns everyone suspects, sex and class.

📄 explore.py: look before you model

import pandas as pd

df = pd.read_csv("titanic_train.csv")
print("Shape:", df.shape)
print("Missing per column:\n", df.isna().sum()[lambda s: s > 0])

print("\nSurvival rate by sex:")
print(df.groupby("Sex")["Survived"].mean().round(3))

print("\nSurvival rate by passenger class:")
print(df.groupby("Pclass")["Survived"].mean().round(3))

▶ Output

Shape: (891, 9)
Missing per column:
 Age         180
Embarked      8
dtype: int64

Survival rate by sex:
Sex
female    0.550
male      0.322
Name: Survived, dtype: float64

Survival rate by passenger class:
Pclass
1    0.826
2    0.514
3    0.173
Name: Survived, dtype: float64

What happened here: Two things jump out. First, Age is missing for 180 of 891 rows and Embarked for 8, so any model needs a plan for filling those before it can train. Second, sex and class both clearly move survival: women survive more than men, and first class survives at 83% while third class survives at just 17%. That 17% is the interesting one, because it is much lower than sex alone would predict, which is a hint that sex and class interact rather than simply add up. Hold that thought, because it is exactly the pattern that decides which model wins later.

Step 2: Split First, Then a Leak-Proof Baseline

Here is the single most important rule in any Kaggle project: split the data before you fit anything, and fit every transform on the training part only. Think of the validation set as a sealed exam you are not allowed to peek at. The moment your imputer or scaler gets to see the validation rows while it is learning, that exam is no longer sealed, and your score becomes a fiction. Wrapping every step inside a scikit-learn Pipeline is what keeps the wall solid, because the pipeline refits itself on only the training rows of each split.

Before any clever model, get two baselines. The first is the dumbest possible model: always guess the majority class, which here is “did not survive.” If your fancy model cannot beat that, it has learned nothing. The second is a plain logistic regression, which is the honest yardstick every heavier model has to beat.

📄 baseline.py: split first, then a majority-class and a logistic baseline

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression
from sklearn.dummy import DummyClassifier
from sklearn.metrics import accuracy_score

df = pd.read_csv("titanic_train.csv")
y = df["Survived"]
X = df.drop(columns=["Survived", "PassengerId"])

# SPLIT FIRST. Everything after this fits on X_train only.
X_train, X_valid, y_train, y_valid = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)

num_cols = ["Age", "SibSp", "Parch", "Fare"]
cat_cols = ["Pclass", "Sex", "Embarked"]
pre = ColumnTransformer([
    ("num", Pipeline([("impute", SimpleImputer(strategy="median")),
                      ("scale", StandardScaler())]), num_cols),
    ("cat", Pipeline([("impute", SimpleImputer(strategy="most_frequent")),
                      ("onehot", OneHotEncoder(handle_unknown="ignore"))]), cat_cols),
])

# Baseline 0: always guess the majority class.
dummy = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)
print("Majority-class accuracy:", round(accuracy_score(y_valid, dummy.predict(X_valid)), 3))

# Baseline 1: logistic regression inside a leak-proof Pipeline.
logreg = Pipeline([("prep", pre),
                   ("clf", LogisticRegression(max_iter=1000, random_state=42))])
logreg.fit(X_train, y_train)
print("Logistic regression accuracy:", round(accuracy_score(y_valid, logreg.predict(X_valid)), 3))

▶ Output

Majority-class accuracy: 0.598
Logistic regression accuracy: 0.816

What happened here: The majority-class guess scores 0.598, so roughly 60% of passengers did not survive and a model that always says “died” is right that often. That is your floor. Logistic regression lands at 0.816, a genuine 22 points above the floor, which tells you the columns really do carry signal. Notice the ColumnTransformer did all the messy prep in one place: it filled missing ages with the median, filled missing ports with the most common one, scaled the numbers, and one-hot encoded the categories, all fitted on the training rows only. That single number, 0.816, is now the bar to beat.

Step 3: Feature Engineering and Cross-Validation

A single train-validation split gives one number, and that number wobbles depending on which rows happened to land in the validation set. Cross-validation fixes that. Instead of one judge, you use five: split the data into five parts, train on four and score on the fifth, rotate five times, and average. Picture five different tasting judges at that cooking contest scoring your dish and reporting the average rather than trusting whichever judge you happened to draw.

While we are here, we add a few hand-built features. FamilySize adds up siblings, spouses, parents, and children plus the passenger. IsAlone flags solo travellers, and FarePerPerson splits a group ticket across the group. These are the kinds of features a human notices and a raw column does not spell out.

📄 features_cv.py: engineer features, then judge with 5-fold cross-validation

import numpy as np
import pandas as pd
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression

df = pd.read_csv("titanic_train.csv")
y = df["Survived"]

def add_features(frame):
    out = frame.copy()
    out["FamilySize"] = out["SibSp"] + out["Parch"] + 1
    out["IsAlone"] = (out["FamilySize"] == 1).astype(int)
    out["FarePerPerson"] = out["Fare"] / out["FamilySize"]
    return out

base_X = df.drop(columns=["Survived", "PassengerId"])
feat_X = add_features(base_X)
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

def make_pipe(num_cols, cat_cols):
    pre = ColumnTransformer([
        ("num", Pipeline([("impute", SimpleImputer(strategy="median")),
                          ("scale", StandardScaler())]), num_cols),
        ("cat", Pipeline([("impute", SimpleImputer(strategy="most_frequent")),
                          ("onehot", OneHotEncoder(handle_unknown="ignore"))]), cat_cols),
    ])
    return Pipeline([("prep", pre),
                     ("clf", LogisticRegression(max_iter=1000, random_state=42))])

base_pipe = make_pipe(["Age", "SibSp", "Parch", "Fare"], ["Pclass", "Sex", "Embarked"])
feat_pipe = make_pipe(["Age", "SibSp", "Parch", "Fare", "FamilySize", "IsAlone", "FarePerPerson"],
                      ["Pclass", "Sex", "Embarked"])

base_scores = cross_val_score(base_pipe, base_X, y, cv=cv, scoring="accuracy")
feat_scores = cross_val_score(feat_pipe, feat_X, y, cv=cv, scoring="accuracy")
print(f"Baseline features CV:   {base_scores.mean():.3f} +/- {base_scores.std():.3f}")
print(f"Engineered features CV: {feat_scores.mean():.3f} +/- {feat_scores.std():.3f}")
print("Per-fold (engineered):", np.round(feat_scores, 3).tolist())

▶ Output

Baseline features CV:   0.806 +/- 0.015
Engineered features CV: 0.805 +/- 0.022
Per-fold (engineered): [0.816, 0.809, 0.781, 0.837, 0.781]

What happened here: Be honest about the result, because this is the lesson most tutorials skip. Adding FamilySize, IsAlone, and FarePerPerson did not move the logistic model at all: 0.805 versus 0.806 is a tie, and the per-fold spread of 0.781 to 0.837 is wider than any gain. That is not a failure, it is information. Those features are mostly re-expressions of columns the model already had, so a straight-line model has nothing new to chew on. The lift, when it comes, will not come from more features. It will come from a model that can actually use the interaction hiding in them, which is the next step.

Step 4: Add One Boosted Model

Remember the clue from exploring: third-class women survived far less than sex and class added together would predict. Logistic regression cannot see that, because it treats each column as a separate additive push and never multiplies them. A tree-based model does exactly that multiplying, splitting on sex and then splitting again on class inside that branch. Gradient boosting builds many small trees where each one fixes the last one’s mistakes, and it is the model that wins the majority of tabular Kaggle competitions.

We use scikit-learn’s own HistGradientBoostingClassifier, which needs no extra install and is fast. In serious competitions most people reach for XGBoost (3.3.0 at the time of writing) or LightGBM instead, and they usually squeeze out another point or two, but the sklearn-native version is the safest default and the fallback that always works. Trees do not need scaling, so the prep here just imputes and encodes.

📄 boosted.py: a gradient-boosted model on the same folds

import numpy as np
import pandas as pd
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OrdinalEncoder
from sklearn.ensemble import HistGradientBoostingClassifier

df = pd.read_csv("titanic_train.csv")
y = df["Survived"]
X = df.drop(columns=["Survived", "PassengerId"])
X["FamilySize"] = X["SibSp"] + X["Parch"] + 1
X["IsAlone"] = (X["FamilySize"] == 1).astype(int)
X["FarePerPerson"] = X["Fare"] / X["FamilySize"]

num_cols = ["Age", "SibSp", "Parch", "Fare", "FamilySize", "IsAlone", "FarePerPerson"]
cat_cols = ["Pclass", "Sex", "Embarked"]

# Trees do not need scaling; they just need categories as numbers.
pre = ColumnTransformer([
    ("num", SimpleImputer(strategy="median"), num_cols),
    ("cat", Pipeline([("impute", SimpleImputer(strategy="most_frequent")),
                      ("ord", OrdinalEncoder(handle_unknown="use_encoded_value",
                                             unknown_value=-1))]), cat_cols),
])
gb = Pipeline([("prep", pre),
               ("clf", HistGradientBoostingClassifier(
                   learning_rate=0.05, max_iter=300, max_leaf_nodes=15,
                   min_samples_leaf=25, l2_regularization=1.0, random_state=42))])

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(gb, X, y, cv=cv, scoring="accuracy")
print(f"HistGradientBoosting CV: {scores.mean():.3f} +/- {scores.std():.3f}")
print("Per-fold:", np.round(scores, 3).tolist())

▶ Output

HistGradientBoosting CV: 0.845 +/- 0.017
Per-fold: [0.866, 0.837, 0.831, 0.865, 0.826]

What happened here: The boosted model reached 0.845 on the same five folds, a real gain of about four points over the logistic 0.806, and every single fold beat the logistic average. This is the payoff the feature engineering set up: the trees exploit the sex-and-class interaction that the linear model was blind to. Four points may sound small, but on a Kaggle leaderboard four points is often hundreds of ranks. Notice we tuned a few settings (a slow learning_rate with more trees, small leaves, some regularization) to keep the model from memorizing noise on just 891 rows.

Step 5: The Leakage Trap

Data leakage is when your model gets to peek at information it would never have at prediction time, so it looks brilliant in training and falls apart in the wild. It is the single most common reason a beginner’s Kaggle project scores far worse on the leaderboard than it did in the notebook. The script below has two leaks planted in it on purpose. Before you read the explanation, try to spot both. Say a newcomer named Aviraj wrote this and asked you to review it.

📄 leakage.py: two planted leaks. Can you find them before the output?

import numpy as np
import pandas as pd
from sklearn.model_selection import cross_val_score, StratifiedKFold
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

df = pd.read_csv("titanic_train.csv")
y = df["Survived"]

# A lifeboat number "came with" the export. It is only filled for survivors.
rng = np.random.default_rng(0)
df["Boat"] = np.where(y == 1, rng.integers(1, 17, len(df)), np.nan)

num = ["Age", "SibSp", "Parch", "Fare", "Boat"]
X = df[num].fillna(-1)
cv = StratifiedKFold(5, shuffle=True, random_state=42)

# Leak #2 hides here: the scaler is fit on ALL rows before cross-validation.
X_scaled_all = StandardScaler().fit_transform(X)
leaky = cross_val_score(LogisticRegression(max_iter=1000), X_scaled_all, y, cv=cv)
print(f"Leaky CV accuracy:  {leaky.mean():.3f}   <- too good to be true")

# Fix #1 only: drop the Boat leak, keep the (still wrong) global scaler.
X_nb = X.drop(columns=["Boat"])
X_nb_all = StandardScaler().fit_transform(X_nb)
fix1 = cross_val_score(LogisticRegression(max_iter=1000), X_nb_all, y, cv=cv)
print(f"After dropping Boat: {fix1.mean():.3f}   worth +{leaky.mean()-fix1.mean():.3f}")

# Fix #2: move scaling inside the pipeline so it fits on train folds only.
honest = cross_val_score(
    Pipeline([("scale", StandardScaler()), ("clf", LogisticRegression(max_iter=1000))]),
    X_nb, y, cv=cv)
print(f"Scaler inside pipe: {honest.mean():.3f}   <- the honest number")

▶ Output

Leaky CV accuracy:  0.987   <- too good to be true
After dropping Boat: 0.737   worth +0.249
Scaler inside pipe: 0.737   <- the honest number

What happened here: The two planted leaks are these:

1. Target leakage (the big one): the Boat column is only filled for people who survived, so it basically is the answer. Including it rockets CV accuracy to 0.987. In the real Titanic data a lifeboat column exists for exactly this reason, and it is the classic trap. The tell is that 0.987 is absurdly high, and the rule is that any feature you would only know after the outcome cannot be a feature. Dropping it alone costs 0.249, which is how much fake score it was worth.
2. Preprocessing leakage (the subtle one): StandardScaler().fit_transform(X) runs on every row before cross-validation, so each fold’s scaler already saw the rows it is about to be tested on. Here the honest number is identical to four decimals, because scaling is a mild transform on this data. That is worth being honest about: this particular leak barely moved the needle. But the same mistake with a target encoder, a feature selector, or on time-ordered data quietly inflates your score by a lot, and reviewers will call it out on sight. The fix is free: put the transform inside the Pipeline, as the last line does, so it refits on train folds only.

Step 6: Make a Submission

Once your model choice is locked in, retrain it on every labelled row you have (no need to hold out a validation set anymore, cross-validation already told you the honest score) and predict the competition’s test.csv. The competition wants a CSV in an exact shape: one column of ids and one column of predictions. Get that format wrong and the upload is rejected, so this step is worth doing carefully.

On the real competition you would download test.csv. To keep our follow-along promise, we generate a stand-in the same way we generated the training file: same columns and the same recipe, a different random seed so the passengers are genuinely unseen, and no Survived column because that is exactly what the competition withholds. Run this once before submit.py.

📄 make_test_data.py: write a Titanic-style test file (stand-in for the real test.csv)

# Stand-in for the competition's test.csv: same columns, no Survived label.
import numpy as np
import pandas as pd

rng = np.random.default_rng(7)   # different seed: unseen passengers
N = 418  # same size as the real Titanic test set

pclass = rng.choice([1, 2, 3], size=N, p=[0.24, 0.21, 0.55])
sex = rng.choice(["male", "female"], size=N, p=[0.65, 0.35])
age = np.clip(rng.normal(29, 13, size=N), 0.5, 80).round(1)
sibsp = rng.choice([0, 1, 2, 3, 4], size=N, p=[0.68, 0.23, 0.05, 0.02, 0.02])
parch = rng.choice([0, 1, 2, 3], size=N, p=[0.76, 0.13, 0.09, 0.02])
fare = np.round(np.where(pclass == 1, rng.normal(84, 40, N),
                np.where(pclass == 2, rng.normal(21, 10, N),
                         rng.normal(13, 8, N))).clip(4, 512), 2)
embarked = rng.choice(["S", "C", "Q"], size=N, p=[0.72, 0.19, 0.09])

# Same holes as the training file: some ages and ports are missing.
age_obj = age.astype(object)
age_obj[rng.random(N) < 0.20] = np.nan
emb_obj = embarked.astype(object)
emb_obj[rng.random(N) < 0.005] = np.nan

test = pd.DataFrame({"PassengerId": np.arange(892, 892 + N), "Pclass": pclass,
                     "Sex": sex, "Age": age_obj, "SibSp": sibsp,
                     "Parch": parch, "Fare": fare, "Embarked": emb_obj})
test.to_csv("titanic_test.csv", index=False)
print(f"Wrote titanic_test.csv: {test.shape[0]} rows, {test.shape[1]} columns (no Survived)")

▶ Output

Wrote titanic_test.csv: 418 rows, 8 columns (no Survived)

📄 submit.py: train on all data, predict the test set, write submission.csv

import pandas as pd
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OrdinalEncoder
from sklearn.ensemble import HistGradientBoostingClassifier

def add_features(frame):
    out = frame.copy()
    out["FamilySize"] = out["SibSp"] + out["Parch"] + 1
    out["IsAlone"] = (out["FamilySize"] == 1).astype(int)
    out["FarePerPerson"] = out["Fare"] / out["FamilySize"]
    return out

train = pd.read_csv("titanic_train.csv")
test = pd.read_csv("titanic_test.csv")   # the real test.csv has no Survived column

y = train["Survived"]
X = add_features(train.drop(columns=["Survived", "PassengerId"]))
X_test = add_features(test.drop(columns=["PassengerId"]))

num = ["Age", "SibSp", "Parch", "Fare", "FamilySize", "IsAlone", "FarePerPerson"]
cat = ["Pclass", "Sex", "Embarked"]
pre = ColumnTransformer([
    ("num", SimpleImputer(strategy="median"), num),
    ("cat", Pipeline([("impute", SimpleImputer(strategy="most_frequent")),
                      ("ord", OrdinalEncoder(handle_unknown="use_encoded_value",
                                             unknown_value=-1))]), cat),
])
model = Pipeline([("prep", pre),
                  ("clf", HistGradientBoostingClassifier(
                      learning_rate=0.05, max_iter=300, max_leaf_nodes=15,
                      min_samples_leaf=25, l2_regularization=1.0, random_state=42))])

model.fit(X, y)                       # train on ALL labelled data now
predictions = model.predict(X_test)

submission = pd.DataFrame({"PassengerId": test["PassengerId"],
                           "Survived": predictions.astype(int)})
submission.to_csv("submission.csv", index=False)
print("Wrote submission.csv:", submission.shape)
print(submission.head().to_string(index=False))
print("Predicted survival rate:", round(submission["Survived"].mean(), 3))

▶ Output

Wrote submission.csv: (418, 2)
 PassengerId  Survived
         892         0
         893         0
         894         0
         895         1
         896         0
Predicted survival rate: 0.361

What happened here: The file has exactly the two columns Kaggle asks for and one row per test passenger, and the predicted survival rate of 0.361 sits close to the 0.404 we saw in training, which is a reassuring sanity check. On the real competition you now go to the competition page, click Submit Predictions, drag in submission.csv, and within seconds you get a public leaderboard score. That score is what the code below represents, an honest stand-in since we cannot upload to a live board from here.

▶ Example output (real leaderboard, after uploading)

Your submission scored 0.79186, which is an improvement of your
previous score of 0.76555. Great job!

The number that matters is not the leaderboard rank, it is the gap between your local cross-validation score and the leaderboard score. If your CV said 0.845 and the board says 0.79, that small gap is normal and healthy. If your CV said 0.98 and the board says 0.62, you leaked, exactly like the Boat trap above, and you go back to the Improve step in the loop.

Ship It: Notebook, GitHub, and Hiring Managers

A Kaggle project is only a portfolio piece once someone else can read it. Two homes work well together, and you want both. First, a public Kaggle notebook: it hosts the code and the dataset in one place and runs in the browser, so a reviewer clicks once and sees your work execute. Second, a GitHub repository with the scripts and a short README. Say a hiring manager named Aditi opens your repo with 60 seconds to spare. Here is what she actually reads, in order:

• The README’s first paragraph: what the problem is, and your final honest CV score. Not your best leaderboard fluke, your validated number.
• The baseline-to-final story: majority class 0.60, logistic 0.81, boosted 0.85. A clear progression shows you understand why each step helped.
• Whether you split before preprocessing and used a Pipeline. This is the fastest way she judges if you understand leakage.
• One honest sentence about what did not work, like the feature engineering that did not help the linear model. Honesty about negative results reads as senior.
• That the whole thing runs top to bottom on a fresh machine with one command.

Notice what is not on that list: your exact leaderboard rank. A junior candidate leads with “top 15%.” A candidate who gets hired leads with “here is my validation strategy and here is how I know I did not leak.” The rank fades, the method is the thing that transfers to a real job.

Common Mistakes

Mistake 1: Fitting transforms before the split

Scaling or imputing on the full dataset and then splitting lets every fold peek at the rows it will be tested on. The score you report is quietly optimistic. Put every transform inside a Pipeline and let cross-validation refit it on each training fold. This one habit prevents most leakage.

Mistake 2: Trusting the public leaderboard over your own CV

The public leaderboard is scored on a slice of the test set, and chasing it leads you to overfit that slice. When the private leaderboard is revealed at the end, those chasers drop hundreds of ranks. Trust your local cross-validation, and only submit changes that improve it, not changes that improve the public score alone.

Mistake 3: Skipping the dumb baseline

Jumping straight to a boosted model with no baseline means you have no idea whether 0.82 is good or terrible. The majority-class number is your floor, and the logistic number is your yardstick. Without them, a score is just a number floating in space with nothing to compare it to.

Best Practices

• DO split before you touch the data, and wrap every transform in a Pipeline so it fits on training rows only
• DO get a majority-class and a logistic baseline before any fancy model, so every later gain is measured against something
• DO judge every change with k-fold cross-validation, and report the mean and the spread, not a single lucky split
• DO seed everything (random_state=42) so your results are reproducible and your teammates can match them
• DON’T include any feature you would only know after the outcome, like a lifeboat number or a refund flag
• DON’T reach for a heavy model or a stack of them until the boosted single model has clearly hit its ceiling

The scikit-learn core used here (Pipeline, ColumnTransformer, cross-validation) is stable and has looked the same for years, so this workflow will outlive any single competition. When you outgrow the sklearn-native booster, XGBoost and LightGBM are the two standard next steps, and the rest of the pipeline around them does not change.

Conclusion

You ran a full Kaggle project end to end. You explored the data, split before touching anything, built a majority-class floor and a logistic yardstick, cross-validated honestly, and added a boosted model that genuinely beat the baseline by exploiting an interaction the linear model could not see. Then you hunted down two planted data leaks and wrote a valid submission file. Most importantly, you learned to trust your own validation over the leaderboard, which is the one habit that separates people who climb the rankings from people who quietly overfit and crash.

Enter a live getting-started competition this week and run the same Kaggle project loop on it, from exploration to submission. For the full path from Python basics through machine learning and deployment, browse the Python + AI/ML tutorial series home.

Frequently Asked Questions

Interview Questions on Kaggle and ML Projects

Interviewers rarely ask for definitions. They ask what happens in situations like these.

Q: Why must you split the data before fitting any preprocessing step?

If you fit a scaler, imputer, or encoder on the full dataset, it learns statistics from the rows you are about to test on, which leaks information and inflates your score. The clean approach is to split first and wrap every transform in a Pipeline, so cross-validation refits the transform on each training fold and the validation rows stay a fair proxy for unseen data. That single discipline prevents the most common form of leakage.

Q: A model scores 0.99 in cross-validation but 0.62 on the leaderboard. What happened?

That enormous gap is the signature of target leakage. Some feature encodes the answer, like a lifeboat number that only exists for survivors, or a transform was fit on the full data before splitting. I would audit every feature for whether it would actually be known at prediction time, remove any that would not, and confirm all preprocessing sits inside the pipeline. A trustworthy model shows only a small gap between local CV and the leaderboard.

Q: Why start with a majority-class baseline instead of going straight to a strong model?

The majority-class baseline is your floor: it tells you the accuracy of guessing the most common label every time. On an imbalanced problem that floor can be surprisingly high, so without it you cannot tell whether a 0.82 model is impressive or barely better than guessing. A logistic regression then gives an honest yardstick. Every heavier model has to beat both, and measuring against them is how you prove a gain is real.

Q: Why did gradient boosting beat logistic regression on this data when extra features did not?

The survival signal contained an interaction: third-class women survived far less than sex and class considered separately would predict. Logistic regression is additive and cannot represent that product without an explicit interaction term. Gradient boosting splits on one column and then splits again inside that branch, so it captures interactions automatically. The engineered features did not help the linear model because they duplicated existing signal, while the boosted model gained by modelling the interaction the linear model was structurally blind to.

Q: What is the difference between the public and private leaderboard, and why does it matter?

The public leaderboard scores your submission on a small visible slice of the test set during the competition, while the private leaderboard, revealed at the end, scores it on the rest. If you tune to the public slice you overfit it and drop when the private score appears. The defense is to trust your own cross-validation and treat the public board as a rough sanity check, not the objective you optimize.

Q: When you finish tuning, should you retrain on all the data before predicting the test set?

Yes. Cross-validation was only there to estimate the honest score and pick the model. Once that decision is made, refit the chosen pipeline on every labelled row so it learns from as much data as possible, then predict the competition test set. Holding data out at that final stage would waste signal you already know how to use safely, since the pipeline handles preprocessing on the full training set correctly.

Related Posts

Previous: ML: Sentiment Analysis, Bag-of-Words to ML

Next: Python ML Model Serving: From Notebook to Production Application Programming Interface (API)

Series Home: Python + AI/ML Tutorial Series