Statistical Hypothesis Testing Interactive Visualization: A/B Testing with Chi Squared Test

2020 · Posted by Sebastien Lemieux-Codere

Count Histograms

2002503003500.000.020.040.060.080.100.120.140.160.18Group AGroup BPositive Outcome CountsFraction of Simulations
Fraction Positive Histograms

0.40.50.60.70.000.020.040.060.080.100.120.140.160.18Group AGroup BFraction of Positive OutcomesFraction of Simulations
Test Statistic Histogram

05101520253035400.000.010.020.030.040.050.060.070.080.090.100.110.120.130.14Computed StatisticFraction of SimulationsSignificance Threshold (> 3.84)Distribution Under Null Hypothesis
Computed P-Value Histogram

0.00.10.20.30.40.50.60.70.80.91.00.00.10.20.30.40.50.60.70.80.9Computed P-ValueFraction of SimulationsSignificance Threshold (< 0.05)Distribution Under Null Hypothesis
Simulation Parameters
Group SizeProbability of Positive Outcome for Group
Group A
Group B
The Null Hypothesis is not True because Group A and Group B have different probabilities of a positive outcome.

Suggested Simulation Parameters to Try:










 Metrics for Significance Level: α =



Type 1 Errors = 0.00% of simulations

Type 2 Errors = 11.35% of simulations


Correctly rejected null hypothesis in ~ 88.65% of simulations (Approx. Statistical Power)

Failed to reject null hypothesis in ~ 11.35% of simulations

See results of specific simulation: / 2000

Counts

Group A Negative CountGroup A Positive CountPositive + Negative
Group A239261500
Group B214286500
Group A + B4535471000

Means (Percentages)

Group A Negative FractionGroup A Positive Fraction
Group A0.478 (47.80%)0.522 (52.20%)
Group B0.428 (42.80%)0.572 (57.20%)
0.453 (45.3%)0.547 (54.7%)


Chi Square Test Statistic = 2.52229
P-Value = 0.11225


The Author

Sebastien Lemieux-Codere

Sebastien is a Data Scientist and Software Developer.