CorrectedPairedTTest
Corrected Paired t-test for comparing two models evaluated with cross-validation.
Based on Nadeau & Bengio (2003), this test corrects the standard paired t-test to account for the fact that CV folds are not independent, sharing training data. The standard paired t-test underestimates variance in this setting, leading to inflated Type I error rates. The correction factor is:
var_corrected = (1/k + n_test/n_train) * var_differences
where k is the number of folds, n_test is the test set size per fold, and n_train is the training set size per fold.
References
Nadeau, C., & Bengio, Y. (2003). Inference for the Generalization Error. Machine Learning, 52(3), 239-281.
Methods
get_metadata(cls) -> dict
CorrectedPairedTTestMetadata for Corrected Paired T-Test.
run(self, scores: dict[str, list[float]], alpha: float = 0.05, alternative: str = 'two-sided', correction_method: str | None = None, **kwargs) -> DashAI.back.statistical_tests.statistical_test_result.StatisticalTestResult
CorrectedPairedTTestRun the corrected paired t-test for cross-validated score differences.
Parameters
- scores : dict[str, list[float]]
- Mapping from model/run names to paired score vectors collected over the same cross-validation folds.
- alpha : float, optional
- Significance level used to decide whether the null hypothesis is rejected, by default 0.05.
- alternative : str, optional
- Direction of the alternative hypothesis:
two-sided,greater, orless. - correction_method : str or None, optional
- Method used to adjust p-values when more than two models are being compared.
Returns
- StatisticalTestResult
- A result object containing the corrected test statistic, p-value, significance decision, and additional details about the variance correction applied to the fold-wise differences.