Designing optimal, data-driven policies from multisite randomized trials. (English) Zbl 1532.62074
Summary: Optimal treatment regimes (OTRs) have been widely employed in computer science and personalized medicine to provide data-driven, optimal recommendations to individuals. However, previous research on OTRs has primarily focused on settings that are independent and identically distributed, with little attention given to the unique characteristics of educational settings, where students are nested within schools and there are hierarchical dependencies. The goal of this study is to propose a framework for designing OTRs from multisite randomized trials, a commonly used experimental design in education and psychology to evaluate educational programs. We investigate modifications to popular OTR methods, specifically Q-learning and weighting methods, in order to improve their performance in multisite randomized trials. A total of 12 modifications, 6 for Q-learning and 6 for weighting, are proposed by utilizing different multilevel models, moderators, and augmentations. Simulation studies reveal that all Q-learning modifications improve performance in multisite randomized trials and the modifications that incorporate random treatment effects show the most promise in handling cluster-level moderators. Among weighting methods, the modification that incorporates cluster dummies into moderator variables and augmentation terms performs best across simulation conditions. The proposed modifications are demonstrated through an application to estimate an OTR of conditional cash transfer programs using a multisite randomized trial in Colombia to maximize educational attainment.
MSC:
| 62P15 | Applications of statistics to psychology |
Keywords:
optimal treatment regimes; optimal treatment rules; personalized learning; Q-learning; weighting; heterogeneous treatment effects; multilevel data; clustered data; conditional cash transfer programReferences:
| [1] | Agniel, D.; Almirall, D.; Burkhart, Q.; Grant, S.; Hunter, SB; Pedersen, ER; Ramchand, R.; Griffin, BA, Identifying optimal level-of-care placement decisions for adolescent substance use treatment, Drug and Alcohol Dependence, 212 (2020) · doi:10.1016/j.drugalcdep.2020.107991 |
| [2] | Arboleda, FLT; Valverde, M., The travels of a set of numbers: The multiple networks enabled by the Colombian ‘Estrato’ system, Social & Legal Studies, 30, 5, 685-703 (2021) · doi:10.1177/0964663920960536 |
| [3] | Athey, S.; Tibshirani, J.; Wager, S., Generalized random forests, The Annals of Statistics, 47, 2, 1148-1178 (2019) · Zbl 1418.62102 · doi:10.1214/18-AOS1709 |
| [4] | Barrera-Osorio, F.; Bertrand, M.; Linden, LL; Perez-Calle, F., Replication data for: Improving the design of conditional transfer programs: Evidence from a randomized education experiment in Colombia (tech. rep.), Inter-University Consortium for Political and Social Research (2019) · doi:10.3886/E113783V1 |
| [5] | Barrera-Osorio, F.; Bertrand, M.; Linden, LL; Perez-Calle, F., Improving the design of conditional transfer programs: Evidence from a randomized education experiment in Colombia, American Economic Journal: Applied Economics, 3, 2, 167-195 (2011) · doi:10.1257/app.3.2.167 |
| [6] | Barrera-Osorio, F., Bertrand, M., Linden, L. L., & Perez-Calle, F. (2011b). Replication data for: Improving the design of conditional transfer programs: Evidence from a randomized education experiment in Colombia [Ann Arbor, MI: Inter-University Consortium for Political and Social Research [distributor], 2019-10-12]. doi:10.3886/E113783V1 |
| [7] | Barrera-Osorio, F.; Linden, LL; Saavedra, JE, Medium-and long-term educational consequences of alternative conditional cash transfer designs: Experimental evidence from Colombia, American Economic Journal: Applied Economics, 11, 3, 54-91 (2019) · doi:10.1257/app.20170008 |
| [8] | Bates, D.; Mächler, M.; Bolker, B.; Walker, S., Fitting linear mixed-effects models using lme4, Journal of Statistical Software, 67, 1, 1-48 (2015) · doi:10.18637/jss.v067.i01 |
| [9] | Chakraborty, B.; Laber, EB; Zhao, Y., Inference for optimal dynamic treatment regimes using an adaptive m-out-of-n bootstrap scheme, Biometrics, 69, 3, 714-723 (2013) · Zbl 1418.62182 · doi:10.1111/biom.12052 |
| [10] | Chakraborty, B., & Moodie, E. (2013). Statistical methods for dynamic treatment regimes. Springer. doi:10.1007/978-1-4614-7428-9 · Zbl 1278.62169 |
| [11] | Chen, S.; Tian, L.; Cai, T.; Yu, M., A general statistical framework for subgroup identification and comparative treatment scoring, Biometrics, 73, 1199-1209 (2017) · Zbl 1405.62164 · doi:10.1111/biom.12676 |
| [12] | Chernozhukov, V.; Chetverikov, D.; Demirer, M.; Duflo, E.; Hansen, C.; Newey, W.; Robins, J., Double/debiased machine learning for treatment and structural parameters, The Econometrics Journal, 21, 1, C1-C68 (2018) · Zbl 07565928 · doi:10.1111/ectj.12097 |
| [13] | Feller, A., & Gelman, A. (2015). Hierarchical models for causal effects.doi:10.1002/9781118900772.etrds0160 |
| [14] | Firebaugh, G., Warner, C., & Massoglia, M. (2013). Fixed effects, random effects, and hybrid models for causal analysis. In S. L. Morgan (Ed.), Handbook of causal analysis for social research (pp. 113-132). Springer. doi:10.1007/978-94-007-6094-3_7 |
| [15] | Hill, JL, Bayesian nonparametric modeling for causal inference, Journal of Computational and Graphical Statistics, 20, 1, 217-240 (2011) · doi:10.1198/jcgs.2010.08162 |
| [16] | Huling, JD; Yu, M., Subgroup identification using the personalized package, Journal of Statistical Software, 98, 5, 1-60 (2021) · doi:10.18637/jss.v098.i05 |
| [17] | Imbens, G. W., & Rubin, D. B. (2015). Causal inference in statistics, social, and biomedical sciences. Cambridge University Press. doi:10.1017/cbo9781139025751 · Zbl 1355.62002 |
| [18] | Kim, K., & Zubizarreta, J. R. (2023). Fair and robust estimation of heterogeneous treatment effects for policy learning. In Proceedings of the 40-th international conference on machine learning. doi:10.48550/arXiv.2306.03625 |
| [19] | Kosorok, M. R., & Moodie, E. E. M. (2015). Adaptive treatment strategies in practice: Planning trials and analyzing data for personalized medicine. Society for Industrial and Applied Mathematics, Philadelphia, PA. doi:10.1137/1.9781611974188 · Zbl 1334.92005 |
| [20] | Lee, Y.; Nguyen, TQ; Stuart, EA, Partially pooled propensity score models for average treatment effect estimation with multilevel data, Journal of the Royal Statistical Society: Series A (Statistics in Society), 184, 4, 1578-1598 (2021) · doi:10.1111/rssa.12741 |
| [21] | Liang, K-Y; Zeger, SL, Longitudinal data analysis using generalized linear models, Biometrika, 73, 1, 13-22 (1986) · Zbl 0595.62110 · doi:10.1093/biomet/73.1.13 |
| [22] | Logan, BR; Sparapani, R.; McCulloch, RE; Laud, PW, Decision making and uncertainty quantification for individualized treatments using Bayesian additive regression trees, Statistical Methods in Medical Research, 28, 4, 1079-1093 (2019) · doi:10.1177/0962280217746191 |
| [23] | Mitchell, S.; Potash, E.; Barocas, S.; D’Amour, A.; Lum, K., Algorithmic fairness: Choices, assumptions, and definitions, Annual Review of Statistics and its Application, 8, 141-163 (2021) · doi:10.1146/annurev-statistics-042720-125902 |
| [24] | Murphy, SA, Optimal dynamic treatment regimes, Journal of the Royal Statistical Society: Series B (Statistical Methodology), 65, 2, 331-355 (2003) · Zbl 1065.62006 · doi:10.1111/1467-9868.00389 |
| [25] | Murphy, SA, An experimental design for the development of adaptive treatment strategies, Statistics in Medicine, 24, 10, 1455-1481 (2005) · doi:10.1002/sim.2022 |
| [26] | Murphy, SA; Lynch, KG; Oslin, D.; McKay, JR; TenHave, T., Developing adaptive treatment strategies in substance abuse research, Drug and Alcohol Dependence, 88, S24-S30 (2007) · doi:10.1016/j.drugalcdep.2006.09.008 |
| [27] | Murphy, SA; Oslin, DW; Rush, AJ; Zhu, J., Methodological challenges in constructing effective treatment sequences for chronic psychiatric disorders, Neuropsychopharmacology, 32, 2, 257-262 (2007) · doi:10.1038/sj.npp.1301241 |
| [28] | Murray, TA; Yuan, Y.; Thall, PF, A Bayesian machine learning approach for optimizing dynamic treatment regimes, Journal of the American Statistical Association, 113, 523, 1255-1267 (2018) · Zbl 1402.62281 · doi:10.1080/01621459.2017.1340887 |
| [29] | Nabi, R., Malinsky, D., & Shpitser, I. (2019). Learning optimal fair policies. In Proceedings of the 36th international conference on machine learning (vol. 32, no. 1, pp. 4674-4682). doi:10.1609/aaai.v32i1.11553 |
| [30] | Neyman, JS, On the application of probability theory to agricultural experiments: Essay on principles. Section 9 (with discussion), Statistical Science, 4, 465-480 (1923) |
| [31] | Park, C.; Kang, H., Efficient semiparametric estimation of network treatment effects under partial interference [asac009], Biometrika, 109, 4, 1015-1031 (2022) · Zbl 1515.62050 · doi:10.1093/biomet/asac009 |
| [32] | Qian, M.; Murphy, SA, Performance guarantees for individualized treatment rules, The Annals of Statistics, 39, 2, 1180-1210 (2011) · Zbl 1216.62178 · doi:10.1214/10-AOS864 |
| [33] | Raudenbush, SW, Adaptive centering with random effects: An alternative to the fixed effects model for studying time-varying treatments in school settings, Education Finance and Policy, 4, 4, 468-491 (2009) · doi:10.1162/edfp.2009.4.4.468 |
| [34] | Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (Vol. 1). Sage. · Zbl 1001.62004 |
| [35] | Raudenbush, SW; Schwartz, D., Randomized experiments in education, with implications for multilevel causal inference, Annual Review of Statistics and Its Application, 7, 1, 177-208 (2020) · doi:10.1146/annurev-statistics-031219-041205 |
| [36] | Robins, J. M. (2004). Optimal structural nested models for optimal sequential decisions. In Proceedings of the second Seattle symposium in biostatistics (pp. 189-326). doi:10.1007/978-1-4419-9076-1_11 · Zbl 1279.62024 |
| [37] | Rubin, DB, Estimating causal effects of treatments in randomized and nonrandomized studies, Journal of Educational Psychology, 66, 5, 688-701 (1974) · doi:10.1037/h0037350 |
| [38] | Rubin, DB, Comment: Which ifs have causal answers, Journal of the American Statistical Association, 81, 396, 961-962 (1986) · doi:10.2307/2289065 |
| [39] | Stefanski, LA; Boos, DD, The calculus of M-estimation, The American Statistician, 56, 1, 29-38 (2002) · doi:10.1198/000313002753631330 |
| [40] | Suk, Y., A within-group approach to ensemble machine learning methods for causal inference in multilevel studies, Journal of Educational and Behavioral Statistics (2023) · doi:10.3102/10769986231162096 |
| [41] | Suk, Y.; Han, KT, A psychometric framework for evaluating fairness in algorithmic decision making: Differential algorithmic functioning, Journal of Educational and Behavioral Statistics (2023) · doi:10.3102/10769986231171711 |
| [42] | Suk, Y.; Kang, H., Robust machine learning for treatment effects in multilevel observational studies under cluster-level unmeasured confounding, Psychometrika, 87, 1, 310-343 (2022) · Zbl 1486.62311 · doi:10.1007/s11336-021-09805-x |
| [43] | Suk, Y.; Kang, H., Tuning random forests for causal inference under cluster-level unmeasured confounding, Multivariate Behavioral Research (2022) · doi:10.1080/00273171.2021.1994364 |
| [44] | Suk, Y.; Kang, H.; Kim, J-S, Random forests approach for causal inference with clustered observational data, Multivariate Behavioral Research, 56, 6, 829-852 (2021) · doi:10.1080/00273171.2020.1808437 |
| [45] | Tibshirani, R., Regression shrinkage and selection via the lasso, Journal of the Royal Statistical Society. Series B (Methodological), 58, 1, 267-288 (1996) · Zbl 0850.62538 · doi:10.1111/j.2517-6161.1996.tb02080.x |
| [46] | Tsiatis, A. A., Davidian, M., Holloway, S. T., & Laber, E. B. (2019). Dynamic treatment regimes: Statistical methods for precision medicine. Hall/CRC. doi:10.1201/9780429192692 · Zbl 1473.62011 |
| [47] | van der Vaart, A. W. (2000). Asymptotic statistics (Vol. 3). Cambridge University Press. doi:10.1017/cbo978051180225 · Zbl 0910.62001 |
| [48] | Wager, S.; Athey, S., Estimation and inference of heterogeneous treatment effects using random forests, Journal of the American Statistical Association, 113, 523, 1228-1242 (2018) · Zbl 1402.62056 · doi:10.1080/01621459.2017.1319839 |
| [49] | Watkins, CJ; Dayan, P., Q-learning, Machine learning, 8, 3, 279-292 (1992) · Zbl 0773.68062 · doi:10.1023/a:1022676722315 |
| [50] | Wooldridge, J. M. (2010). Econometric analysis of cross section and panel data. The MIT press. · Zbl 1327.62009 |
| [51] | Zhang, B.; Tsiatis, AA; Davidian, M.; Zhang, M.; Laber, E., Estimating optimal treatment regimes from a classification perspective, Stat, 1, 1, 103-114 (2012) · Zbl 07847463 · doi:10.1002/sta.411 |
| [52] | Zhao, Y.; Zeng, D.; Rush, AJ; Kosorok, MR, Estimating individualized treatment rules using outcome weighted learning, Journal of the American Statistical Association, 107, 499, 1106-1118 (2012) · Zbl 1443.62396 · doi:10.1080/01621459.2012.695674 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
