×

Learning networks from Gaussian graphical models and Gaussian free fields. (English) Zbl 07832561

Summary: We investigate the problem of estimating the structure of a weighted network from repeated measurements of a Gaussian graphical model (GGM) on the network. In this vein, we consider GGMs whose covariance structures align with the geometry of the weighted network on which they are based. Such GGMs have been of longstanding interest in statistical physics, and are referred to as the Gaussian free field (GFF). In recent years, they have attracted considerable interest in the machine learning and theoretical computer science. In this work, we propose a novel estimator for the weighted network (equivalently, its Laplacian) from repeated measurements of a GFF on the network, based on the Fourier analytic properties of the Gaussian distribution. In this pursuit, our approach exploits complex-valued statistics constructed from observed data, that are of interest in their own right. We demonstrate the effectiveness of our estimator with concrete recovery guarantees and bounds on the required sample complexity. In particular, we show that the proposed statistic achieves the parametric rate of estimation for fixed network size. In the setting of networks growing with sample size, our results show that for Erdos-Renyi random graphs \(G(d, p)\) above the connectivity threshold, network recovery takes place with high probability as soon as the sample size \(n\) satisfies \(n \gg d^4 \log d \cdot p^{-2}\).

MSC:

62F12 Asymptotic properties of parametric estimators
62F10 Point estimation
62F35 Robustness and adaptive procedures (parametric inference)

Software:

glasso; CHOMPACK

References:

[1] Anandkumar, A.; Tan, V.; Willsky, A., High-dimensional Gaussian graphical model selection: walk summability and local separation criterion, J. Mach. Learn. Res., 13, 07, 2011
[2] Anandkumar, A.; Tan, VY; Huang, F.; Willsky, AS, High-dimensional structure estimation in Ising models: local separation criterion, Ann. Stat., 40, 1346-1375, 2012 · Zbl 1297.62124 · doi:10.1214/12-AOS1009
[3] Banerjee, O.; Ghaoui, L., Model selection through sparse max likelihood estimation model selection through sparse maximum likelihood estimation for multivariate gaussian or binary data, J. Mach. Learn. Res., 9, 08, 2007
[4] Banerjee, S.; Ghosal, S., Posterior convergence rates for estimating large precision matrices using graphical models, Electron. J. Stat., 8, 2, 2111-2137, 2014 · Zbl 1302.62124 · doi:10.1214/14-EJS945
[5] Banerjee, S.; Ghosal, S., Bayesian structure learning in graphical models, J. Multivar. Anal., 136, 147-162, 2015 · Zbl 1308.62119 · doi:10.1016/j.jmva.2015.01.015
[6] Basso, K., Margolin, A.A., Stolovitzky, G., Klein, U., Dalla-Favera, R., Califano, A.: Reverse engineering of regulatory networks in human B cells. Nat. Genet. 37(4), 382-390 (2005). doi:10.1038/ng1532
[7] Belomestny, D., Trabs, M., Tsybakov, A.B.: Sparse covariance matrix estimation in high-dimensional deconvolution. Bernoulli 25(3), 8 (2019). doi:10.3150/18-BEJ1040A · Zbl 1466.62332
[8] Berestycki, N.: Introduction to the Gaussian free field and Liouville quantum gravity. Lecture notes, 2018-2019 (2015)
[9] Berthet, Q.; Rigollet, P.; Srivastava, P., Exact recovery in the ising blockmodel, Ann. Stat., 47, 4, 1805-1834, 2019 · Zbl 1420.62268 · doi:10.1214/17-AOS1620
[10] Bhattacharya, B. B., Mukherjee, S.: Inference in ising models. (2018) · Zbl 1408.62033
[11] Bickel, PJ; Levina, E., Regularized estimation of large covariance matrices, Ann. Stat., 36, 1, 199-227, 2008 · Zbl 1132.62040 · doi:10.1214/009053607000000758
[12] Bickel, PJ; Levina, E., Covariance regularization by thresholding, Ann. Stat., 36, 6, 2577-2604, 2008 · Zbl 1196.62062 · doi:10.1214/08-AOS600
[13] Bresler, G.: Efficiently learning ising models on arbitrary graphs. In: Proceedings of the forty-seventh annual ACM symposium on Theory of computing, pp. 771-782 (2015) · Zbl 1321.68397
[14] Cai, TT; Zhang, C-H; Zhou, HH, Optimal rates of convergence for covariance matrix estimation, Ann. Stat., 38, 4, 2118-2144, 2010 · Zbl 1202.62073 · doi:10.1214/09-AOS752
[15] Cai, TT; Li, H.; Liu, W.; Xie, J., Covariate-adjusted precision matrix estimation with an application in genetical genomics, Biometrika, 100, 1, 139-156, 2012 · Zbl 1284.62648 · doi:10.1093/biomet/ass058
[16] Cai, T.; Liu, W.; Zhou, HH, Estimating sparse precision matrix: optimal rates of convergence and adaptive estimation, Ann. Stat., 44, 2, 455-488, 2016 · Zbl 1341.62115 · doi:10.1214/13-AOS1171
[17] Cai, TT; Ren, Z.; Zhou, HH, Estimating structured high-dimensional covariance and precision matrices: optimal rates and adaptive estimation, Electron. J. Stat., 10, 1, 1-59, 2016 · Zbl 1331.62272
[18] Cai, T.; Liu, W.; Luo, X., A constrained \(\ell_1\) minimization approach to sparse precision matrix estimation, J. Am. Stat. Assoc., 106, 494, 594-607, 2011 · Zbl 1232.62087 · doi:10.1198/jasa.2011.tm10155
[19] Dahl, J.; Vandenberghe, L.; Roychowdhury, V., Covariance selection for non-chordal graphs via chordal embedding, Optim. Methods Softw., 23, 4, 501-520, 2008 · Zbl 1151.90514 · doi:10.1080/10556780802102693
[20] d’Aspremont, A.; Banerjee, O.; El Ghaoui, L., First-order methods for sparse covariance selection, SIAM J. Matrix Anal. Appl., 30, 1, 56-66, 2008 · Zbl 1156.90423 · doi:10.1137/060670985
[21] Dempster, AP, Covariance selection, Biometrics, 28, 1, 157-175, 1972 · doi:10.2307/2528966
[22] El Karoui, N., Operator norm consistent estimation of large-dimensional sparse covariance matrices, Ann. Stat., 36, 6, 2717-2756, 2008 · Zbl 1196.62064
[23] Fan, J.; Feng, Y.; Yichao, W., Network exploration via the adaptive LASSO and SCAD penalties, Ann. Appl. Stat., 3, 2, 521-541, 2009 · Zbl 1166.62040 · doi:10.1214/08-AOAS215
[24] Friedman, J.; Hastie, T.; Tibshirani, R., Sparse inverse covariance estimation with the graphical lasso, Biostatistics, 9, 3, 432-441, 2007 · Zbl 1143.62076 · doi:10.1093/biostatistics/kxm045
[25] Ghosh, S., Mukherjee, S. S.: Learning with latent group sparsity via heat flow dynamics on networks. arXiv:2201.08326 (2022)
[26] Huang, JZ; Liu, N.; Pourahmadi, M.; Liu, L., Covariance matrix selection and estimation via penalised normal likelihood, Biometrika, 93, 1, 85-98, 2006 · Zbl 1152.62346 · doi:10.1093/biomet/93.1.85
[27] Huang, S.; Li, J.; Sun, L.; Ye, J.; Fleisher, A.; Teresa, W.; Chen, K.; Reiman, E., Learning brain connectivity of Alzheimer’s disease by sparse inverse covariance estimation, NeuroImage, 50, 3, 935-949, 2010 · doi:10.1016/j.neuroimage.2009.12.120
[28] Kelner, J.; Koehler, F.; Meka, R.; Moitra, A.; Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, MF; Lin, H., Learning some popular gaussian graphical models without condition number bounds, Advances in Neural Information Processing Systems, 10986-10998, 2020, New York: Curran Associates Inc, New York
[29] Kelner, J.; Koehler, F.; Meka, R.; Moitra, A., Learning some popular gaussian graphical models without condition number bounds, Adv. Neural. Inf. Process. Syst., 33, 10986-10998, 2020
[30] Lam, C.; Fan, J., Sparsistency and rates of convergence in large covariance matrix estimation, Ann. Stat., 37, 6, 4254-4278, 2009 · Zbl 1191.62101 · doi:10.1214/09-AOS720
[31] Lei, J.; Rinaldo, A., Consistency of spectral clustering in stochastic block models, Ann. Stat., 43, 1, 215-237, 2015 · Zbl 1308.62041 · doi:10.1214/14-AOS1274
[32] Liu, H.; Lafferty, J.; Wasserman, L., The nonparanormal: semiparametric estimation of high dimensional undirected graphs, J. Mach. Learn. Res., 10, 80, 2295-2328, 2009 · Zbl 1235.62035
[33] Loh, P-L; Bühlmann, P., High-dimensional learning of linear causal networks via inverse covariance estimation, J. Mach. Learn. Res., 15, 1, 3065-3105, 2014 · Zbl 1318.68148
[34] Ma, Y., Garnett, R., Schneider, J.: \( \sigma \)-optimality for active learning on gaussian random fields. Advances in Neural Information Processing Systems, 26 (2013)
[35] Malioutov, DM; Johnson, JK; Willsky, AS, Walk-sums and belief propagation in gaussian graphical models, J. Mach. Learn. Res., 7, 73, 2031-2064, 2006 · Zbl 1222.68254
[36] Meinshausen, N.; Bühlmann, P., High-dimensional graphs and variable selection with the Lasso, Ann. Stat., 34, 3, 1436-1462, 2006 · Zbl 1113.62082 · doi:10.1214/009053606000000281
[37] Menéndez, P.; Kourmpetis, YA; ter Braak, CJ; van Eeuwijk, FA, Gene regulatory networks from multifactorial perturbations using graphical lasso: application to the DREAM4 challenge, PLoS ONE, 5, 12, e14147, 2010 · doi:10.1371/journal.pone.0014147
[38] Misra, S., Vuffray, M., Lokhov, A.Y.: Information theoretic optimal learning of gaussian graphical models. In: Jacob Abernethy and Shivani Agarwal, editors, Proceedings of Thirty Third Conference on Learning Theory, volume 125 of Proceedings of Machine Learning Research, pp. 2888-2909. PMLR, 09-12 (2020)
[39] Müller, A.; Scarsini, M., Archimedean copulae and positive dependence, J. Multivar. Anal., 93, 2, 434-445, 2005 · Zbl 1065.60018 · doi:10.1016/j.jmva.2004.04.003
[40] Ravikumar, P., Wainwright, M.J., Lafferty, J.D.: High-dimensional ising model selection using \(\ell_1\)-regularized logistic regression (2010) · Zbl 1189.62115
[41] Ravikumar, P.; Wainwright, MJ; Raskutti, G.; Bin, Yu, High-dimensional covariance estimation by minimizing \(\ell_1\)-penalized log-determinant divergence, Electron. J. Stat., 5, none, 935-980, 2011 · Zbl 1274.62190 · doi:10.1214/11-EJS631
[42] Rish, I.; Thyreau, B.; Thirion, B.; Plaze, M.; Paillere-martinot, M.; Martelli, C.; Martinot, J.; Poline, J.; Cecchi, G.; Bengio, Y.; Schuurmans, D.; Lafferty, J.; Williams, C.; Culotta, A., Discriminative network models of schizophrenia, Advances in Neural Information Processing Systems, 2009, New York: Curran Associates Inc, New York
[43] Rothman, AJ; Bickel, PJ; Levina, E.; Zhu, J., Sparse permutation invariant covariance estimation, Electron. J. Stat., 2, 494-515, 2008 · Zbl 1320.62135 · doi:10.1214/08-EJS176
[44] Schafer, J., Strimmer, K.: Learning large-scale graphical Gaussian models from genomic data. In: AIP Conference Proceedings, pp. 263-276. AIP (2005). doi:10.1063/1.1985393
[45] Sheffield, S., Gaussian free fields for mathematicians, Probab. Theory Relat. Fields, 139, 3-4, 521-541, 2007 · Zbl 1132.60072 · doi:10.1007/s00440-006-0050-1
[46] Shi, W.; Ghosal, S.; Martin, R., Bayesian estimation of sparse precision matrices in the presence of Gaussian measurement error, Electron. J. Stat., 15, 2, 4545-4579, 2021 · Zbl 1478.62124 · doi:10.1214/21-EJS1904
[47] Tropp, JA, Just relax: convex programming methods for identifying sparse signals in noise, IEEE Trans. Inf. Theory, 52, 3, 1030-1051, 2006 · Zbl 1288.94025 · doi:10.1109/TIT.2005.864420
[48] Varoquaux, G., Baronnet, F., Kleinschmidt, A., Fillard, P., Thirion, B.: Detection of brain functional-connectivity difference in post-stroke patients using group-level covariance modeling. In: T. Jiang, N. Navab, J.P.W. Pluim, M.A. Viergever, (eds) Medical Image Computing and Computer-Assisted Intervention - MICCAI 2010, pp. 200-208, Springer, Berlin (2010)
[49] Varoquaux, G.; Gramfort, A.; Poline, J.; Thirion, B.; Lafferty, J.; Williams, C.; Shawe-Taylor, J.; Zemel, R.; Culotta, A., Brain covariance selection: better individual functional connectivity models using population prior, Advances in Neural Information Processing Systems, 2010, New York: Curran Associates Inc, New York
[50] Vershynin, R., High-Dimensional Probability: An Introduction with Applications in Data Science, 2018, Cambridge: Cambridge University Press, Cambridge · Zbl 1430.60005
[51] Wainwright, MJ, Sharp thresholds for high-dimensional and noisy sparsity recovery using \(\ell_1\)-constrained quadratic programming (lasso), IEEE Trans. Inf. Theory, 55, 5, 2183-2202, 2009 · Zbl 1367.62220 · doi:10.1109/TIT.2009.2016018
[52] Wille, A.; Zimmermann, P.; Vranová, E.; Fürholz, A.; Laule, O.; Bleuler, S.; Hennig, L.; Prelić, A.; von Rohr, P.; Thiele, L.; Zitzler, E.; Gruissem, W.; Bühlmann, P., Sparse graphical Gaussian modeling of the isoprenoid gene network in Arabidopsis thaliana, Genome Biol., 5, 11, R92, 2004 · doi:10.1186/gb-2004-5-11-r92
[53] Woodbury, MA, Inverting Modified Matrices, 1950, Princeton: Princeton University, Department of Statistics, Princeton
[54] Wu, WB; Pourahmadi, M., Non-parametric estimation of large covariance matrices of longitudinal data, Biometrika, 90, 4, 831-844, 2003 · Zbl 1436.62347 · doi:10.1093/biomet/90.4.831
[55] Yuan, M., High dimensional inverse covariance matrix estimation via linear programming, J. Mach. Learn. Res., 11, 79, 2261-2286, 2010 · Zbl 1242.62043
[56] Yuan, M.; Lin, Y., Model selection and estimation in the Gaussian graphical model, Biometrika, 94, 19-35, 2007 · Zbl 1142.62408 · doi:10.1093/biomet/asm018
[57] Zhao, P.; Bin, Yu, On model selection consistency of lasso, J. Mach. Learn. Res., 7, 90, 2541-2563, 2006 · Zbl 1222.62008
[58] Zhu, X., Ghahramani, Z., Lafferty, J.D.: Semi-supervised learning using gaussian fields and harmonic functions. In: Proceedings of the 20th International conference on Machine learning (ICML-03), pp. 912-919 (2003)
[59] Zhu, X., Lafferty, J., Ghahramani, Z.: Combining active learning and semi-supervised learning using Gaussian fields and harmonic functions. In: ICML 2003 workshop on the continuum from labeled to unlabeled data in machine learning and data mining, vol. 3 (2003)
[60] Zwiernik, P., Semialgebraic Statistics and Latent Tree Models, 2016, Boca Raton: Chapman & Hall/CRC, Boca Raton · Zbl 1378.62004
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.