Exploring the Advantages and Limitations of Bayesian Methods for Causal Inference in Small-Sample Randomized Experiments in Education: A Monte Carlo Study
Guardado en:
| Publicado en: | ProQuest Dissertations and Theses (2025) |
|---|---|
| Autor principal: | |
| Publicado: |
ProQuest Dissertations & Theses
|
| Materias: | |
| Acceso en línea: | Citation/Abstract Full Text - PDF |
| Etiquetas: |
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| Resumen: | Background: Randomized controlled trials (RCTs) are foundational in educational research for establishing causal effects, yet small-sample pilot studies—a common reality in the field—pose persistent methodological challenges. Classical frequentist methods, which rely heavily on p-values and null-hypothesis significance testing, often suffer from low statistical power, wide and imprecise intervals, and limited interpretability in these settings (Schoot & Miočević, 2020). These limitations can lead to exaggerated effect sizes or misleading conclusions, undermining the credibility of intervention evaluations and the evidence base for educational policy and practice.Purpose: The primary goal of this proposed dissertation is to explore the advantages and limitations of Bayesian methods for causal inference in small-sample randomized experiments in education. Specifically, the research aims to demonstrate through a simulation study how Bayesian methods compare with frequentist methods in terms of precision, statistical conclusion, interpretability, and flexibility. The Bayesian models utilize prior knowledge, posterior distributions, and credible intervals, whereas the frequentist models rely on asymptotics (large-sample assumptions) and p-values (McNeish, 2016). The research specifically examines scenarios with negligible, small, and medium/large true effects, and explores the impact of prior-data alignment and misalignment on inference quality.Research Design: A Monte Carlo simulation study with at least 5,000 replications per condition was used to compare the performance of Bayesian and frequentist methods. The Monte Carlo simulation is designed to replicate the small-sample conditions typically observed in underpowered educational experiments intended to improve students’ achievement test scores. The experimental designs considered include the basic singlelevel experiment and the two-level cluster-randomized experiment. Key factors such as sample size, effect sizes, prior-data conflict are systematically varied to evaluate their impact on the performance of both methods. In particular, the study examines misalignments between the prior distribution specified in the analysis and the true data-generating process, focusing on discrepancies in both location (central tendency) and dispersion (variability).Data Analysis: Simulated datasets were analyzed using both Bayesian and frequentist linear regression models under single-level and multilevel designs. Key performance metrics included bias, root mean squared error (RMSE), interval coverage, standard error, posterior standard deviation, and robustness to prior specification. Statistical conclusion validity was assessed using calibration, Type M (magnitude) and Type S (sign) errors, and interpretability was compared through the lens of posterior probabilities and credible intervals (Bayesian) versus p-values and confidence intervals (frequentist).Findings: Bayesian methods demonstrated advantages in small-sample settings, producing more precise estimates, with narrower and better-calibrated intervals than frequentist methods—especially when well-aligned or weakly informative priors were used. Bayesian models also showed superior control over Type M and Type S errors, reducing the risk of effect size exaggeration and incorrect directionality. However, these benefits depended critically on thoughtful prior specification. When priors were overly optimistic or misaligned with the data-generating process, Bayesian inference became overconfident, leading to poor coverage and inflated error rates. Computational demands and the need for transparent, interpretable communication of posterior probabilities were additional practical considerations. In contrast, frequentist methods, while familiar and computationally efficient, tended to yield wider intervals.Implications: These findings demonstrate that Bayesian techniques provide a flexible, precise, and interpretable alternative to traditional frequentist methods in small-sample educational experiments. The capacity to make direct probability statements about parameters—such as the likelihood that an effect exceeds a meaningful threshold—offers a more intuitive way to assess practical significance. Combined with improved calibration and control of inferential errors, Bayesian inference emerges as a valuable tool for both researchers and decision-makers. The growing availability of Bayesian software has made these methods increasingly accessible to applied educational researchers. This dissertation encourages researchers to leverage the strengths of the Bayesian framework—particularly its suitability for threshold-based decision-making and rich, probabilistic interpretation— while also recognizing the importance of thoughtful prior specification, the computational demands of some models, and the complementary role of frequentist validation. By adopting a transparent and well-justified approach to statistical modeling, educational researchers can improve the accuracy, clarity, and trustworthiness of their causal inferences, ultimately strengthening the foundation for evidence-based policy and practice. |
|---|---|
| ISBN: | 9798290963709 |
| Fuente: | ProQuest Dissertations & Theses Global |