Large covariance estimation through elliptical factor models. (English) Zbl 1402.62124
Summary: We propose a general Principal Orthogonal complEment Thresholding (POET) framework for large-scale covariance matrix estimation based on the approximate factor model. A set of high-level sufficient conditions for the procedure to achieve optimal rates of convergence under different matrix norms is established to better understand how POET works. Such a framework allows us to recover existing results for sub-Gaussian data in a more transparent way that only depends on the concentration properties of the sample covariance matrix. As a new theoretical contribution, for the first time, such a framework allows us to exploit conditional sparsity covariance structure for the heavy-tailed data. In particular, for the elliptical distribution, we propose a robust estimator based on the marginal and spatial Kendall’s tau to satisfy these conditions. In addition, we study conditional graphical model under the same framework. The technical tools developed in this paper are of general interest to high-dimensional principal component analysis. Thorough numerical results are also provided to back up the developed theory.
MSC:
| 62H25 | Factor analysis and principal components; correspondence analysis |
| 62H12 | Estimation in multivariate analysis |
Keywords:
principal component analysis; approximate factor model; sub-Gaussian family; elliptical distribution; conditional graphical model; marginal and spatial Kendall’s tauReferences:
| [1] | Agarwal, A., Negahban, S. and Wainwright, M. J. (2012). Noisy matrix decomposition via convex relaxation: Optimal rates in high dimensions. Ann. Statist.40 1171–1197. · Zbl 1274.62219 · doi:10.1214/12-AOS1000 |
| [2] | Amini, A. A. and Wainwright, M. J. (2009). High-dimensional analysis of semidefinite relaxations for sparse principal components. Ann. Statist.37 2877–2921. · Zbl 1173.62049 · doi:10.1214/08-AOS664 |
| [3] | Antoniadis, A. and Fan, J. (2001). Regularization of wavelet approximations. J. Amer. Statist. Assoc.96 939–967. · Zbl 1072.62561 · doi:10.1198/016214501753208942 |
| [4] | Bai, J. and Li, K. (2012). Statistical analysis of factor models of high dimension. Ann. Statist.40 436–465. · Zbl 1246.62144 · doi:10.1214/11-AOS966 |
| [5] | Bai, J. and Ng, S. (2002). Determining the number of factors in approximate factor models. Econometrica70 191–221. · Zbl 1103.91399 · doi:10.1111/1468-0262.00273 |
| [6] | Bai, J. and Ng, S. (2013). Principal components estimation and identification of static factors. J. Econometrics176 18–29. · Zbl 1284.62350 · doi:10.1016/j.jeconom.2013.03.007 |
| [7] | Belloni, A. and Chernozhukov, V. (2011). \(ℓ_{1}\)-penalized quantile regression in high-dimensional sparse models. Ann. Statist.39 82–130. · Zbl 1209.62064 · doi:10.1214/10-AOS827 |
| [8] | Beran, R. (1978). An efficient and robust adaptive estimator of location. Ann. Statist.6 292–313. · Zbl 0378.62051 · doi:10.1214/aos/1176344125 |
| [9] | Berthet, Q. and Rigollet, P. (2013a). Optimal detection of sparse principal components in high dimension. Ann. Statist.41 1780–1815. · Zbl 1277.62155 · doi:10.1214/13-AOS1127 |
| [10] | Berthet, Q. and Rigollet, P. (2013b). Complexity theoretic lower bounds for sparse principal component detection. In Proceedings of the 26th Annual Conference on Learning Theory 1046–1066. PMLR, Princeton, NJ. · Zbl 1277.62155 · doi:10.1214/13-AOS1127 |
| [11] | Bickel, P. J. (1982). On adaptive estimation. Ann. Statist.10 647–671. · Zbl 0489.62033 · doi:10.1214/aos/1176345863 |
| [12] | Bickel, P. J. and Levina, E. (2008a). Covariance regularization by thresholding. Ann. Statist.36 2577–2604. · Zbl 1196.62062 · doi:10.1214/08-AOS600 |
| [13] | Bickel, P. J. and Levina, E. (2008b). Regularized estimation of large covariance matrices. Ann. Statist.36 199–227. · Zbl 1132.62040 · doi:10.1214/009053607000000758 |
| [14] | Birnbaum, A., Johnstone, I. M., Nadler, B. and Paul, D. (2013). Minimax bounds for sparse PCA with noisy high-dimensional data. Ann. Statist.41 1055–1084. · Zbl 1292.62071 · doi:10.1214/12-AOS1014 |
| [15] | Cai, T. and Liu, W. (2011). Adaptive thresholding for sparse covariance matrix estimation. J. Amer. Statist. Assoc.106 672–684. · Zbl 1232.62086 · doi:10.1198/jasa.2011.tm10560 |
| [16] | Cai, T., Liu, W. and Luo, X. (2011). A constrained \(ℓ_{1}\) minimization approach to sparse precision matrix estimation. J. Amer. Statist. Assoc.106 594–607. · Zbl 1232.62087 |
| [17] | Cai, T. T., Ma, Z. and Wu, Y. (2013). Sparse PCA: Optimal rates and adaptive estimation. Ann. Statist.41 3074–3110. · Zbl 1288.62099 · doi:10.1214/13-AOS1178 |
| [18] | Cai, T., Ma, Z. and Wu, Y. (2015). Optimal estimation and rank detection for sparse spiked covariance matrices. Probab. Theory Related Fields161 781–815. · Zbl 1314.62130 · doi:10.1007/s00440-014-0562-z |
| [19] | Cai, T. T., Ren, Z. and Zhou, H. H. (2013). Optimal rates of convergence for estimating Toeplitz covariance matrices. Probab. Theory Related Fields156 101–143. · Zbl 06176807 · doi:10.1007/s00440-012-0422-7 |
| [20] | Cai, T. T., Zhang, C.-H. and Zhou, H. H. (2010). Optimal rates of convergence for covariance matrix estimation. Ann. Statist.38 2118–2144. · Zbl 1202.62073 · doi:10.1214/09-AOS752 |
| [21] | Cai, T. T. and Zhou, H. H. (2012). Optimal rates of convergence for sparse covariance matrix estimation. Ann. Statist.40 2389–2420. · Zbl 1373.62247 · doi:10.1214/12-AOS998 |
| [22] | Cai, T. T., Li, H., Liu, W. and Xie, J. (2013). Covariate-adjusted precision matrix estimation with an application in genetical genomics. Biometrika100 139–156. · Zbl 1284.62648 · doi:10.1093/biomet/ass058 |
| [23] | Catoni, O. (2012). Challenging the empirical mean and empirical variance: A deviation study. Ann. Inst. Henri Poincaré Probab. Stat.48 1148–1185. · Zbl 1282.62070 · doi:10.1214/11-AIHP454 |
| [24] | Chamberlain, G. and Rothschild, M. (1983). Arbitrage, factor structure, and mean-variance analysis on large asset markets. Econometrica51 1281–1304. · Zbl 0523.90017 · doi:10.2307/1912275 |
| [25] | Chandrasekaran, V., Parrilo, P. A. and Willsky, A. S. (2012). Latent variable graphical model selection via convex optimization. Ann. Statist.40 1935–1967. · Zbl 1257.62061 · doi:10.1214/11-AOS949 |
| [26] | Chandrasekaran, V., Sanghavi, S., Parrilo, P. A. and Willsky, A. S. (2011). Rank-sparsity incoherence for matrix decomposition. SIAM J. Optim.21 572–596. · Zbl 1226.90067 · doi:10.1137/090761793 |
| [27] | Choi, K. and Marden, J. (1998). A multivariate version of Kendall’s \(τ\). J. Nonparametr. Stat.9 261–293. · Zbl 0945.62062 |
| [28] | Christensen, D. (2005). Fast algorithms for the calculation of Kendall’s \(τ\). Comput. Statist.20 51–62. · Zbl 1089.62067 |
| [29] | Čížek, P., Härdle, W. and Weron, R., eds. (2005). Statistical Tools for Finance and Insurance. Springer, Berlin. · Zbl 1078.62112 |
| [30] | Croux, C., Ollila, E. and Oja, H. (2002). Sign and rank covariance matrices: Statistical properties and application to principal components analysis. In Statistical Data Analysis Based on the \(L_{1}\)-Norm and Related Methods (Neuchâtel, 2002) 257–269. Birkhäuser, Basel. · Zbl 1145.62343 |
| [31] | Davis, C. and Kahan, W. M. (1970). The rotation of eigenvectors by a perturbation. III. SIAM J. Numer. Anal.7 1–46. · Zbl 0198.47201 · doi:10.1137/0707001 |
| [32] | Dupuis Lozeron, E. and Victoria-Feser, M. P. (2010). Robust estimation of constrained covariance matrices for confirmatory factor analysis. Comput. Statist. Data Anal.54 3020–3032. · Zbl 1284.62196 · doi:10.1016/j.csda.2009.08.014 |
| [33] | Dürre, A., Vogel, D. and Tyler, D. E. (2014). The spatial sign covariance matrix with unknown location. J. Multivariate Anal.130 107–117. · Zbl 1292.62079 · doi:10.1016/j.jmva.2014.05.004 |
| [34] | El Karoui, N. (2008). Operator norm consistent estimation of large-dimensional sparse covariance matrices. Ann. Statist.36 2717–2756. · Zbl 1196.62064 · doi:10.1214/07-AOS559 |
| [35] | Fan, J., Fan, Y. and Barut, E. (2014). Adaptive robust variable selection. Ann. Statist.42 324–351. · Zbl 1296.62144 · doi:10.1214/13-AOS1191 |
| [36] | Fan, J., Fan, Y. and Lv, J. (2008). High dimensional covariance matrix estimation using a factor model. J. Econometrics147 186–197. · Zbl 1429.62185 · doi:10.1016/j.jeconom.2008.09.017 |
| [37] | Fan, J., Feng, Y. and Wu, Y. (2009). Network exploration via the adaptive lasso and SCAD penalties. Ann. Appl. Stat.3 521–541. · Zbl 1166.62040 · doi:10.1214/08-AOAS215 |
| [38] | Fan, J., Li, Q. and Wang, Y. (2017). Estimation of high dimensional mean regression in the absence of symmetry and light tail assumptions. J. R. Stat. Soc. Ser. B. Stat. Methodol.79 247–265. · Zbl 1414.62178 |
| [39] | Fan, J., Liao, Y. and Mincheva, M. (2011). High-dimensional covariance matrix estimation in approximate factor models. Ann. Statist.39 3320–3356. · Zbl 1246.62151 · doi:10.1214/11-AOS944 |
| [40] | Fan, J., Liao, Y. and Mincheva, M. (2013). Large covariance estimation by thresholding principal orthogonal complements. J. R. Stat. Soc. Ser. B. Stat. Methodol.75 603–680. With 33 discussions by 57 authors and a reply by Fan, Liao and Mincheva. · Zbl 1411.62138 |
| [41] | Fan, J., Liao, Y. and Wang, W. (2016). Projected principal component analysis in factor models. Ann. Statist.44 219–254. · Zbl 1331.62295 · doi:10.1214/15-AOS1364 |
| [42] | Fan, J., Liu, H. and Wang, W. (2018). Supplement to “Large covariance estimation through elliptical factor models”. DOI:10.1214/17-AOS1588SUPP. |
| [43] | Fan, J., Wang, W. and Zhong, Y. (2016). Robust covariance estimation for approximate factor models. Available at arXiv:1602.00719. |
| [44] | Fan, J., Wang, W. and Zhong, Y. (2017). An \(ℓ_{∞}\) eigenvector perturbation bound and its application to robust covariance estimation. Available at arXiv:1603.03516. |
| [45] | Fan, J., Ke, Z. T., Liu, H. and Xia, L. (2015). QUADRO: A supervised dimension reduction method via Rayleigh quotient optimization. Ann. Statist.43 1498–1534. · Zbl 1317.62054 · doi:10.1214/14-AOS1307 |
| [46] | Fan, J., Liu, H., Wang, W. and Zhu, Z. (2016). Heterogeneity adjustment with applications to graphical model inference. Available at arXiv:1602.05455. |
| [47] | Fang, K. T., Kotz, S. and Ng, K. W. (1990). Symmetric Multivariate and Related Distributions. Monographs on Statistics and Applied Probability36. Chapman & Hall, London. · Zbl 0699.62048 |
| [48] | Friedman, J., Hastie, T. and Tibshirani, R. (2008). Sparse inverse covariance estimation with the graphical lasso. Biostatistics9 432–441. · Zbl 1143.62076 · doi:10.1093/biostatistics/kxm045 |
| [49] | Hallin, M. and Paindaveine, D. (2006). Semiparametrically efficient rank-based inference for shape. I. Optimal rank-based tests for sphericity. Ann. Statist.34 2707–2756. · Zbl 1114.62066 · doi:10.1214/009053606000000731 |
| [50] | Hampel, F. R. (1974). The influence curve and its role in robust estimation. J. Amer. Statist. Assoc.69 383–393. · Zbl 0305.62031 · doi:10.1080/01621459.1974.10482962 |
| [51] | Han, F. and Liu, H. (2014). Scale-invariant sparse PCA on high-dimensional meta-elliptical data. J. Amer. Statist. Assoc.109 275–287. · Zbl 1367.62185 · doi:10.1080/01621459.2013.844699 |
| [52] | Han, F. and Liu, H. (2018). ECA: High dimensional elliptical component analysis in non-Gaussian distributions. J. Amer. Statist. Assoc.113 252–268. · Zbl 1398.62153 |
| [53] | Han, F. and Liu, H. (2017). Statistical analysis of latent generalized correlation matrix estimation in transelliptical distribution. Bernoulli23 23–57. · Zbl 1359.62186 · doi:10.3150/15-BEJ702 |
| [54] | Hsu, D., Kakade, S. M. and Zhang, T. (2011). Robust matrix decomposition with sparse corruptions. IEEE Trans. Inform. Theory57 7221–7234. · Zbl 1365.15018 · doi:10.1109/TIT.2011.2158250 |
| [55] | Hsu, D. and Sabato, S. (2014). Heavy-tailed regression with a generalized median-of-means. In Proceedings of the 31st International Conference on Machine Learning (ICML-14) 37–45. PMLR, Bejing, China. |
| [56] | Huber, P. J. (1964). Robust estimation of a location parameter. Ann. Math. Stat.35 73–101. · Zbl 0136.39805 · doi:10.1214/aoms/1177703732 |
| [57] | Hult, H. and Lindskog, F. (2002). Multivariate extremes, aggregation and dependence in elliptical distributions. Adv. in Appl. Probab.34 587–608. · Zbl 1023.60021 · doi:10.1239/aap/1033662167 |
| [58] | Johnstone, I. M. and Lu, A. Y. (2009). On consistency and sparsity for principal components analysis in high dimensions. J. Amer. Statist. Assoc.104 682–693. · Zbl 1388.62174 · doi:10.1198/jasa.2009.0121 |
| [59] | Kendall, M. G. (1948). Rank Correlation Methods. C. Griffin, London. · Zbl 0032.17602 |
| [60] | Knight, W. R. (1966). A computer method for calculating Kendall’s tau with ungrouped data. J. Amer. Statist. Assoc.61 436–439. · Zbl 0138.13202 · doi:10.1080/01621459.1966.10480879 |
| [61] | Koenker, R. (2005). Quantile Regression. Econometric Society Monographs38. Cambridge Univ. Press Cambridge. · Zbl 1111.62037 |
| [62] | Lam, C. and Fan, J. (2009). Sparsistency and rates of convergence in large covariance matrix estimation. Ann. Statist.37 4254–4278. · Zbl 1191.62101 · doi:10.1214/09-AOS720 |
| [63] | Levina, E. and Vershynin, R. (2012). Partial estimation of covariance matrices. Probab. Theory Related Fields153 405–419. · Zbl 1318.62179 · doi:10.1007/s00440-011-0349-4 |
| [64] | Liu, H., Han, F. and Zhang, C. (2012). Transelliptical graphical models. In Advances in Neural Information Processing Systems 800–808. Curran Associates, Inc., Lake Tahoe, NV. |
| [65] | Liu, L., Hawkins, D. M., Ghosh, S. and Young, S. S. (2003). Robust singular value decomposition analysis of microarray data. Proc. Natl. Acad. Sci. USA100 13167–13172. · Zbl 1108.62117 · doi:10.1073/pnas.1733249100 |
| [66] | Ma, Z. (2013). Sparse principal component analysis and iterative thresholding. Ann. Statist.41 772–801. · Zbl 1267.62074 · doi:10.1214/13-AOS1097 |
| [67] | Marden, J. I. (1999). Some robust estimates of principal components. Statist. Probab. Lett.43 349–359. · Zbl 0939.62055 · doi:10.1016/S0167-7152(98)00272-7 |
| [68] | Meinshausen, N. and Bühlmann, P. (2006). High-dimensional graphs and variable selection with the lasso. Ann. Statist.34 1436–1462. · Zbl 1113.62082 · doi:10.1214/009053606000000281 |
| [69] | Mitra, R. and Zhang, C.-H. (2014). Multivariate analysis of nonparametric estimates of large correlation matrices. Available at arXiv:1403.6195. |
| [70] | Pang, H., Liu, H. and Vanderbei, R. (2014). The fastclime package for linear programming and large-scale precision matrix estimation in R. J. Mach. Learn. Res.15 489–493. · Zbl 1318.90002 |
| [71] | Paul, D. and Johnstone, I. M. (2012). Augmented sparse principal component analysis for high dimensional data. Available at arXiv:1202.1242. |
| [72] | Pison, G., Rousseeuw, P. J., Filzmoser, P. and Croux, C. (2003). Robust factor analysis. J. Multivariate Anal.84 145–172. · Zbl 1038.62055 · doi:10.1016/S0047-259X(02)00007-6 |
| [73] | Posekany, A., Felsenstein, K. and Sykacek, P. (2011). Biological assessment of robust noise models in microarray data analysis. Bioinformatics27 807–814. |
| [74] | Rachev, S. T. (2003). Handbook of Heavy Tailed Distributions in Finance. Handbooks in Finance1. Elsevier, Amsterdam. |
| [75] | Ravikumar, P., Wainwright, M. J., Raskutti, G. and Yu, B. (2011). High-dimensional covariance estimation by minimizing \(ℓ_{1}\)-penalized log-determinant divergence. Electron. J. Stat.5 935–980. · Zbl 1274.62190 · doi:10.1214/11-EJS631 |
| [76] | Rothman, A. J., Levina, E. and Zhu, J. (2009). Generalized thresholding of large covariance matrices. J. Amer. Statist. Assoc.104 177–186. · Zbl 1388.62170 · doi:10.1198/jasa.2009.0101 |
| [77] | Rothman, A. J., Levina, E. and Zhu, J. (2010). Sparse multivariate regression with covariance estimation. J. Comput. Graph. Statist.19 947–962. Supplementary materials available online. |
| [78] | Rousseeuw, P. J. and Croux, C. (1993). Alternatives to the median absolute deviation. J. Amer. Statist. Assoc.88 1273–1283. · Zbl 0792.62025 · doi:10.1080/01621459.1993.10476408 |
| [79] | Ruttimann, U. E., Unser, M., Rawlings, R. R., Rio, D., Ramsey, N. F., Mattay, V. S., Hommer, D. W., Frank, J. A. and Weinberger, D. R. (1998). Statistical analysis of functional MRI data in the wavelet domain. IEEE Trans. Med. Imag.17 142–154. |
| [80] | Shen, D., Shen, H. and Marron, J. S. (2013). Consistency of sparse PCA in high dimension, low sample size contexts. J. Multivariate Anal.115 317–333. · Zbl 1258.62072 · doi:10.1016/j.jmva.2012.10.007 |
| [81] | Tsybakov, A. B. (2009). Introduction to Nonparametric Estimation. Springer, New York. Revised and extended from the 2004 French original. Translated by Vladimir Zaiats. · Zbl 1029.62034 |
| [82] | Tyler, D. E. (1982). Radial estimates and the test for sphericity. Biometrika69 429–436. · Zbl 0501.62041 · doi:10.1093/biomet/69.2.429 |
| [83] | Vanderbei, R. J. (2008). Linear Programming: Foundations and Extensions, 3rd ed. International Series in Operations Research & Management Science114. Springer, New York. · Zbl 1139.90002 |
| [84] | Vershynin, R. (2012). Introduction to the non-asymptotic analysis of random matrices. In Compressed Sensing 210–268. Cambridge Univ. Press Cambridge. |
| [85] | Visuri, S., Koivunen, V. and Oja, H. (2000). Sign and rank covariance matrices. J. Statist. Plann. Inference91 557–575. · Zbl 0965.62049 · doi:10.1016/S0378-3758(00)00199-3 |
| [86] | Vogel, D. and Fried, R. (2011). Elliptical graphical modelling. Biometrika98 935–951. · Zbl 1228.62069 · doi:10.1093/biomet/asr037 |
| [87] | Vu, V. Q. and Lei, J. (2013). Minimax sparse principal subspace estimation in high dimensions. Ann. Statist.41 2905–2947. · Zbl 1288.62103 · doi:10.1214/13-AOS1151 |
| [88] | Wang, W. and Fan, J. (2017). Asymptotics of empirical eigenstructure for high dimensional spiked covariance. Ann. Statist.45 1342–1374. · Zbl 1373.62299 · doi:10.1214/16-AOS1487 |
| [89] | Wegkamp, M. and Zhao, Y. (2016). Adaptive estimation of the copula correlation matrix for semiparametric elliptical copulas. Bernoulli22 1184–1226. · Zbl 1388.62162 · doi:10.3150/14-BEJ690 |
| [90] | Wu, Y. and Liu, Y. (2009). Variable selection in quantile regression. Statist. Sinica19 801–817. · Zbl 1166.62012 |
| [91] | Yuan, M. (2010). High dimensional inverse covariance matrix estimation via linear programming. J. Mach. Learn. Res.11 2261–2286. · Zbl 1242.62043 |
| [92] | Yuan, M. and Lin, Y. (2007). Model selection and estimation in the Gaussian graphical model. Biometrika94 19–35. · Zbl 1142.62408 · doi:10.1093/biomet/asm018 |
| [93] | Zhao, T., Roeder, K. and Liu, H. (2014). Positive semidefinite rank-based correlation matrix estimation with application to semiparametric graph estimation. J. Comput. Graph. Statist.23 895–922. |
| [94] | Zou, H., Hastie, T. and Tibshirani, R. (2006). Sparse principal component analysis. J. Comput. Graph. Statist.15 265–286. |
| [95] | Zou, H. and Yuan, M. (2008). Composite quantile regression and the oracle model selection theory. Ann. Statist.36 1108–1126. · Zbl 1360.62394 · doi:10.1214/07-AOS507 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
