Perform hypothesis testing in Python: null and alternative hypotheses, p-values, t-tests (one-sample, independent, paired), chi-square, and a decision tree for choosing the right test (including where ANOVA fits) in this hypothesis testing python guide.

“To call in the statistician after the experiment is done may be no more than asking him to perform a post-mortem examination.”

Ronald Fisher, Statistical Methods

Last Updated: July 2026 | Tested on: Python 3.14.6, SciPy 1.18.0 | Difficulty: Advanced | Reading Time: 16 minutes

Your A/B test shows that the new landing page has a 3% higher conversion rate. Is that a real win, or did you just get a lucky run of visitors? Hypothesis testing is the math that answers that one question. Instead of arguing about gut feelings, you put a number on it: what is the chance you would see a difference this big if the new page were actually no better? When that chance is tiny, you stop second-guessing and call the improvement real.

Here is a quick way to picture it. Think of a courtroom. The null hypothesis is “the defendant is innocent”, that is the boring default everyone starts with. The data is the evidence. You only convict (reject the null) when the evidence would be very unlikely for an innocent person. The p-value is exactly that: how surprising your data would be if the boring default were true. A small p-value means “this evidence is hard to explain by chance alone”, so you reject the default.

The steps never change, no matter which test you run: state what you assume (the null hypothesis), state what you want to prove (the alternative hypothesis), collect data, compute a test statistic, and read off the p-value. If p < 0.05, you reject the null hypothesis. That 0.05 cutoff is the standard across science and industry, but there is nothing magic about it. It is a convention, like driving on the right side of the road. Some fields use 0.01 or 0.001 when the cost of a wrong call is high.

Rahul, a 27-year-old data analyst, ran an A/B test on a checkout flow and saw a 5% lift. His manager wanted to ship it that same afternoon. But the hypothesis test came back with p = 0.23, nowhere near significant, which meant the 5% could easily be noise. So they kept the test running. Two weeks later, with four times the sample size, p dropped to 0.003. The improvement was real all along. They just did not have enough data to prove it yet. Patience, not a bigger gut feeling, is what closed the case.

Tip: click the diagram to open it full screen, then use the zoom and pan controls for a closer look.

This decision tree answers the question every beginner gets stuck on: which test do I even use? Start at the top with “what am I comparing?” and follow the branches. One sample against a known number sends you to a one-sample t-test. Two separate groups send you to an independent t-test (or Mann-Whitney U if the data is not normal). The same group measured before and after sends you to a paired t-test.

Three or more groups send you to ANOVA (Analysis of Variance), which this post covers at the decision level only; the worked examples below stick to t-tests and chi-square. Counts in categories send you to chi-square. The workflow itself never changes, only the function name at the end of the branch does.

Prerequisites

Work through the probability distributions tutorial first. You should be comfortable with the normal distribution and the basic idea of a p-value. On the code side you need SciPy and NumPy installed (pip install scipy numpy); if NumPy arrays are still new to you, the NumPy introduction covers everything this post uses. No heavy math background is required. If you can read a fraction and follow an “if this, then that” decision, you can follow every test in this post.

The Hypothesis Testing Framework

Every test in this post follows the same five steps. Learn them once and you can run any test SciPy offers. Here they are end to end on a tiny real example. A coffee shop claims each cup has 200mg of caffeine, and Niranjan, a 24-year-old curious about his afternoon jitters, measures 25 cups to check. We seed the random number generator so you get the exact same numbers shown here.

📄 hypothesis_framework.py: the five steps applied

from scipy import stats
import numpy as np

# Scenario: A coffee shop claims their average cup has 200mg caffeine
# Niranjan measures 25 cups and gets these results:
rng = np.random.default_rng(42)
measurements = rng.normal(195, 10, 25)  # Actually less than 200

print("Step 1: State hypotheses")
print("  H0 (null): mu = 200mg (shop's claim is true)")
print("  H1 (alternative): mu != 200mg (claim is false)")

print("\nStep 2: Choose significance level")
alpha = 0.05
print(f"  alpha = {alpha} (5% chance of false positive)")

print("\nStep 3: Compute test statistic")
t_stat, p_value = stats.ttest_1samp(measurements, popmean=200)
print(f"  t-statistic: {t_stat:.4f}")
print(f"  p-value: {p_value:.4f}")

print(f"\nStep 4: Compare p-value to alpha")
print(f"  p-value ({p_value:.4f}) {'<' if p_value < alpha else '>='} alpha ({alpha})")

print(f"\nStep 5: Conclusion")
if p_value < alpha:
    print(f"  REJECT H0. The average caffeine is significantly different from 200mg.")
    print(f"  Sample mean: {measurements.mean():.1f}mg")
else:
    print(f"  FAIL TO REJECT H0. Not enough evidence to dispute the claim.")
    print(f"  Sample mean: {measurements.mean():.1f}mg")

▶ Output

Step 1: State hypotheses
  H0 (null): mu = 200mg (shop's claim is true)
  H1 (alternative): mu != 200mg (claim is false)

Step 2: Choose significance level
  alpha = 0.05 (5% chance of false positive)

Step 3: Compute test statistic
  t-statistic: -3.2210
  p-value: 0.0037

Step 4: Compare p-value to alpha
  p-value (0.0037) < alpha (0.05)

Step 5: Conclusion
  REJECT H0. The average caffeine is significantly different from 200mg.
  Sample mean: 194.6mg

What happened here: Niranjan's 25 cups averaged 194.6mg, not the promised 200mg. The one-sample t-test asked the real question: could a true average of 200mg produce a sample this low just by random luck? The p-value of 0.0037 says that would happen less than 4 times in 1,000. That is rare enough to reject the shop's claim. Notice that the code never decided anything on its own. It handed you a t-statistic and a p-value, and the simple rule "p < 0.05 means reject" did the rest. Run any of the other tests below and you are still just reading that same p-value.

The Three t-tests

The t-test is the workhorse of hypothesis testing, and it comes in three flavours. The trick is matching the flavour to your situation. An independent t-test compares two separate groups, like two different teams of people. A paired t-test compares the same group twice, like the same people before and after a training course. There is also the one-sample t-test you already saw above, which compares one group against a fixed number. Think of it like weighing fruit: independent is two different baskets on the scale, paired is the same basket weighed yesterday and today.

📄 t_tests.py: one-sample, independent, and paired

from scipy import stats
import numpy as np

rng = np.random.default_rng(42)

# 1. Independent two-sample t-test
# Are engineering salaries different from marketing salaries?
eng_salaries = rng.normal(82000, 8000, 50)
mkt_salaries = rng.normal(75000, 9000, 45)

t_stat, p_val = stats.ttest_ind(eng_salaries, mkt_salaries)
print(f"Independent t-test (Eng vs Mkt salaries):")
print(f"  Eng mean: {eng_salaries.mean():,.0f}, Mkt mean: {mkt_salaries.mean():,.0f}")
print(f"  t = {t_stat:.3f}, p = {p_val:.4f}")
print(f"  Significant difference: {p_val < 0.05}\n")

# 2. Paired t-test
# Did a training program improve scores? (same people, before/after)
before = rng.normal(70, 10, 30)
after = before + rng.normal(5, 8, 30)  # Improvement + noise

t_stat, p_val = stats.ttest_rel(before, after)
print(f"Paired t-test (Before vs After training):")
print(f"  Before mean: {before.mean():.1f}, After mean: {after.mean():.1f}")
print(f"  t = {t_stat:.3f}, p = {p_val:.4f}")
print(f"  Significant improvement: {p_val < 0.05}")

▶ Output

Independent t-test (Eng vs Mkt salaries):
  Eng mean: 82,730, Mkt mean: 73,937
  t = 6.685, p = 0.0000
  Significant difference: True

Paired t-test (Before vs After training):
  Before mean: 68.4, After mean: 71.9
  t = -2.223, p = 0.0342
  Significant improvement: True

What happened here: The engineering and marketing salary gap (about 82.7k versus 73.9k) came back with a p-value so small it rounded to 0.0000 (the real value is about 0.0000000017). A gap that large almost never shows up by chance, so it is clearly a real difference. The training program is the more interesting case. The before-and-after scores moved from 68.4 to 71.9, and p = 0.0342, which squeaks under 0.05. That is a real but modest improvement, the kind that could easily have looked like noise with a smaller group. One detail to notice: the t-statistic is negative simply because ttest_rel(before, after) subtracts in that order. The sign tells you the direction, the p-value tells you whether to care.

Chi-Square Test for Categorical Data

t-tests compare numbers like salaries or scores, the kind of values you can summarize with a mean and standard deviation. But what if your data is just labels, like which department someone is in and whether they prefer working from home? You cannot average a department. For that you use the chi-square test of independence. Think of it like asking whether umbrella sales and rainy days go together: if umbrellas fly off the shelf exactly on the wet days, the two are linked, but if sales look the same rain or shine, they are independent.

The chi-square test asks one plain question: are these two categories related, or do they just vary on their own? It works by comparing the counts you actually saw against the counts you would expect if the two categories had nothing to do with each other. If the real counts are far from the expected counts, the categories are linked.

📄 chi_square.py: testing independence of categorical variables

from scipy import stats
import numpy as np

# Is there a relationship between department and work-from-home preference?
#                   WFH    Office   Hybrid
# Engineering       45      15       40
# Marketing         20      30       25
# Data Science      50      10       35

observed = np.array([
    [45, 15, 40],
    [20, 30, 25],
    [50, 10, 35]
])

chi2, p_val, dof, expected = stats.chi2_contingency(observed)
print(f"Chi-square test of independence:")
print(f"  Chi-square statistic: {chi2:.2f}")
print(f"  p-value: {p_val:.4f}")
print(f"  Degrees of freedom: {dof}")
print(f"  Significant relationship: {p_val < 0.05}")
print(f"\nExpected frequencies (if independent):\n{expected.round(1)}")

▶ Output

Chi-square test of independence:
  Chi-square statistic: 27.49
  p-value: 0.0000
  Degrees of freedom: 4
  Significant relationship: True

Expected frequencies (if independent):
[[42.6 20.4 37. ]
 [31.9 15.3 27.8]
 [40.5 19.4 35.2]]

What happened here: The expected table is what each cell would hold if department and work preference were unrelated. Compare it to the real counts: Data Science actually had 50 work-from-home folks, but the "no relationship" expectation was only 40.5. Engineering leaned the same way. Those gaps add up into a chi-square statistic of 27.49, and the p-value rounds to 0.0000 (the real value is about 0.0000158). That is well under 0.05, so department and work-from-home preference are linked, not independent.

The chi-square test cannot tell you which department drives the link, only that a link exists. For that you eyeball the gaps between the real and expected tables, exactly as we just did. And when both of your variables are numeric rather than categorical, you measure the strength of that kind of relationship with correlation and regression instead.

Which Test Should I Use?

Bookmark this table. Picking the wrong test is the single most common beginner mistake, and it usually comes from skipping the one question that matters: what shape is my data, and how many groups am I comparing? Match your situation to a row and copy the function name. The decision tree at the top of this post is the same logic in picture form.

Common Mistakes

Mistake 1: Reading the p-value backwards

This is the one almost everyone gets wrong at first. A p-value is NOT the chance that your hypothesis is true. It is the chance of seeing data this extreme if the boring default (the null) were true. The difference sounds small but it flips the whole meaning.

📄 The one sentence to memorise

# WRONG: "p = 0.03 means there is a 3% chance the null hypothesis is true"
# RIGHT: "p = 0.03 means there is a 3% chance of seeing this data
#         (or more extreme) IF the null hypothesis were true"

# The p-value is about the data, not about the hypothesis.

Mistake 2: Testing over and over until something turns up

Here is the trap that ruins real experiments. If you keep re-running a test on fresh data until you finally see p < 0.05, you will eventually get there even when there is nothing to find. Picture rolling a die over and over: roll it enough times and a six is guaranteed, even though no single roll is special. Run twenty honest tests on two groups that are truly identical and, on average, one of them will cross 0.05 by pure luck. Let us prove it.

📄 p_hacking_demo.py: noise can fake a result if you test enough

from scipy import stats
import numpy as np

# Two groups that are TRULY identical (same distribution, no real difference).
# If we test them 20 times on fresh samples, how often does p < 0.05
# fool us into "finding" a difference that does not exist?
rng = np.random.default_rng(7)
false_positives = 0
trials = 20

for i in range(trials):
    group_a = rng.normal(100, 15, 30)
    group_b = rng.normal(100, 15, 30)  # identical setup
    _, p = stats.ttest_ind(group_a, group_b)
    if p < 0.05:
        false_positives += 1
        print(f"  Trial {i+1:2d}: p = {p:.4f}  <-- FALSE positive!")

print(f"\nRan {trials} tests on groups with NO real difference.")
print(f"Got {false_positives} 'significant' result(s) purely by chance.")
print("Lesson: test something enough times and noise will look real.")

▶ Output

  Trial  4: p = 0.0187  <-- FALSE positive!

Ran 20 tests on groups with NO real difference.
Got 1 'significant' result(s) purely by chance.
Lesson: test something enough times and noise will look real.

What happened here: Both groups were drawn from the exact same distribution, so there is genuinely nothing to detect. Yet trial 4 came back with p = 0.0187 and shouted "significant". That is a false positive, and it is not a bug. An alpha of 0.05 literally means you accept a 5% false-positive rate, so roughly 1 in 20 honest tests will trip the wire by chance. The fix is to decide your single test up front, and if you must run many comparisons, correct for it (the Bonferroni correction, for example, divides 0.05 by the number of tests). Never go fishing for a small p-value.

Practice Exercises

1. Exercise 1: A factory claims its bolts are 50mm long. Generate 40 measurements with rng.normal(49.6, 1.5, 40) and run a one-sample t-test against 50. Is the claim safe?
2. Exercise 2: Write a small function choose_two_group_test(group_a, group_b) that first checks normality with stats.shapiro, then runs an independent t-test if both groups look normal, or Mann-Whitney U if they do not. Return the test name and the p-value.
3. Exercise 3: Build a mini A/B test report. Take conversion counts for two landing pages, run a chi-square test, and print a one-line verdict ("ship it" or "keep testing") based on whether p < 0.05.

Conclusion

You now have the full hypothesis testing toolkit: the five-step framework that never changes, the three t-tests for comparing numbers, the chi-square test for categorical labels, and a decision tree that tells you which one to reach for. The big idea to carry away is that every test hands you the same thing, a p-value, and the same rule reads it: a small p-value means the boring default is hard to defend, so you reject it. You also saw the two traps that catch beginners, reading the p-value backwards and testing until noise finally looks real.

Next up is correlation and regression, where instead of asking "are these groups different?" you ask "do these two numbers move together, and can I predict one from the other?". For the full roadmap from beginner basics all the way to machine learning, head back to the Python + AI/ML tutorial series home.

Frequently Asked Questions

Interview Questions on Hypothesis Testing

The same ideas as they show up in real interviews, framed as scenarios you can practice out loud.

Q: What is the difference between a one-sample, an independent, and a paired t-test?

A one-sample t-test compares one group against a fixed known value, like checking measured cups of coffee against a claimed 200mg. An independent t-test compares two separate groups, like engineering versus marketing salaries. A paired t-test compares the same subjects measured twice, like test scores before and after a training course. The key is whether the two sets of numbers are linked subject by subject: if they are, you use the paired test because it cancels out per-person variation.

Q: Why do we say "fail to reject the null" instead of "accept the null"?

A hypothesis test can only measure evidence against the null, never evidence that proves it true. A large p-value means you did not find enough evidence to reject the default, but that could be because the effect is genuinely zero or because your sample was too small to detect a real effect. Both look identical from the outside. Saying "fail to reject" keeps you honest about that uncertainty, the same way a court returns "not guilty" rather than "proven innocent".

Q: Your A/B test comes back with p = 0.06 and your manager wants to ship the change anyway. What do you tell them and what do you check?

p = 0.06 is above the usual 0.05 cutoff, so by the pre-agreed rule the result is not significant and you cannot claim the change worked. Before deciding, I would check the sample size and effect size: a promising lift that just missed the line often becomes significant with more data, so the honest move is to keep the test running rather than ship on a borderline number. I would also confirm the 0.05 threshold was fixed up front and not moved after seeing the data, since bending the cutoff to fit the result is exactly how false positives sneak in.

Q: You have two groups to compare, but Shapiro-Wilk reports the data is not normally distributed. What do you do?

The independent t-test assumes roughly normal data, so when that assumption fails I switch to the non-parametric equivalent, the Mann-Whitney U test (stats.mannwhitneyu). It compares ranks instead of raw means, so it does not care about the shape of the distribution. For paired non-normal data the equivalent is the Wilcoxon signed-rank test, and for three or more groups it is Kruskal-Wallis. With large samples the t-test is fairly robust to mild non-normality, so I weigh sample size before switching automatically.

Q: What does a chi-square test of independence tell you, and what does it not tell you?

It tells you whether two categorical variables are related, by comparing the counts you observed against the counts you would expect if the variables were independent. A significant result means the two categories are linked, not just varying on their own. What it does not tell you is which specific cell drives the link or how strong the relationship is: for that you inspect the gaps between the observed and expected tables, and for numeric variables you would use correlation instead.

Q: What is the multiple comparisons problem, and how does the Bonferroni correction help?

Every test at alpha = 0.05 carries a 5% chance of a false positive, so if you run many tests, the odds that at least one trips by pure luck climb fast. Run twenty tests on groups with no real difference and on average one will look "significant". The Bonferroni correction fixes this by dividing your alpha by the number of tests, so twenty comparisons would each be judged against 0.05 / 20 = 0.0025. It is a conservative fix that keeps the overall false-positive rate near 5% at the cost of some power to detect real effects.

Want more? the official Python documentation documents everything this post could not fit.

Related Posts

Previous: Statistics: Probability Distributions, Normal and Binomial

Next: Statistics: Correlation & Regression Analysis

Series Home: Python + AI/ML Tutorial Series