Fisher lecture: Dimension reduction in regression. (English) Zbl 1246.62148
Summary: Beginning with a discussion of R.A. Fisher’s early written remarks that relate to dimension reduction, this article revisits principal components as a reductive method in regression, develops several model-based extensions and ends with descriptions of general approaches to model-based and model-free dimension reduction in regression. It is argued that the role for principal components and related methodology may be broader than previously seen and that the common practice of conditioning on observed values of the predictors may unnecessarily limit the choice of regression methodology.
MSC:
| 62H25 | Factor analysis and principal components; correspondence analysis |
| 62J99 | Linear inference, regression |
Keywords:
central subspace; Grassmann manifolds; inverse regression; minimum average variance estimation; principal components; principal fitted components; sliced inverse regression; sufficient dimension reductionReferences:
| [1] | Adcock, R. J. (1878). A problem in least squares. The Analyst 5 53–54. |
| [2] | Aldrich, J. (2005). Fisher and regression. Statist. Sci. 20 401–417. · Zbl 1130.62300 · doi:10.1214/088342305000000331 |
| [3] | Alter, O., Brown, P. and Botstein, D. (2000). Singular value decomposition for genome-wide expression data processing and modeling. Proc. Natl. Acad. Sci. U.S.A. 97 10,101–10,106. |
| [4] | Anderson, T. W. (1984). Estimating linear statistical relationships. Ann. Statist. 12 1–45. · Zbl 0542.62039 · doi:10.1214/aos/1176346390 |
| [5] | Anscombe, F. J. (1961). Examination of residuals. Proc. 4th Berkeley Symp. Math. Statist. Probab. 1 1–36. Univ. California Press, Berkeley. · Zbl 0101.35703 |
| [6] | Anscombe, F. J. and Tukey, J. W. (1963). The examination and analysis of residuals. Technometrics 5 141–160. JSTOR: · Zbl 0118.13903 · doi:10.2307/1266059 |
| [7] | Box, G. E. P. (1979). Robustness in the strategy of scientific model building. In Robustness in Statistics (R. Launer and G. Wilkinson, eds.) 201–235. Academic Press, New York. · Zbl 0441.62033 |
| [8] | Box, G. E. P. (1980). Sampling and Bayes’ inference in scientific modeling and robustness (with discussion). J. Roy. Statist. Soc. Ser. A 143 383–430. JSTOR: · Zbl 0471.62036 · doi:10.2307/2982063 |
| [9] | Breiman, L. (2001). Statistical modeling: The two cultures (with discussion). Statist. Sci. 16 199–231. · Zbl 1059.62505 · doi:10.1214/ss/1009213726 |
| [10] | Bura, E. and Cook, R. D. (2001). Extending sliced inverse regression: The weighted chi-squared test. J. Amer. Statist. Assoc. 96 996–1003. JSTOR: · Zbl 1047.62035 · doi:10.1198/016214501753208979 |
| [11] | Bura, E. and Pfeiffer, R. M. (2003). Graphical methods for class prediction using dimension reduction techniques on DNA microarray data. Bioinformatics 19 1252–1258. |
| [12] | Chiaromonte, F. and Martinelli, J. (2002). Dimension reduction strategies for analyzing global gene expression data with a response. Math. Biosci. 176 123–144. · Zbl 0999.62090 · doi:10.1016/S0025-5564(01)00106-7 |
| [13] | Chikuse, Y. (2003). Statistics on Special Manifolds . Lecture Notes in Statist. 174 . Springer, New York. · Zbl 1026.62051 |
| [14] | Christensen, R. (2001). Advanced Linear Modeling , 2nd ed. Springer, New York. · Zbl 0983.62057 |
| [15] | Cook, R. D. (1986). Assessment of local influence (with discussion). J. Roy. Statist. Soc. Ser. B 48 133–169. JSTOR: · Zbl 0608.62041 |
| [16] | Cook, R. D. (1994). Using dimension-reduction subspaces to identify important inputs in models of physical systems. In Proc. Section on Physical and Engineering Sciences 18–25. Amer. Statist. Assoc., Alexandria, VA. |
| [17] | Cook, R. D. (1998). Regression Graphics : Ideas for Studying Regressions Through Graphics. Wiley, New York. · Zbl 0903.62001 |
| [18] | Cook, R. D. and Ni, L. (2005). Sufficient dimension reduction via inverse regression: A minimum discrepancy approach. J. Amer. Statist. Assoc. 100 410–428. · Zbl 1117.62312 · doi:10.1198/016214504000001501 |
| [19] | Cook, R. D. and Weisberg, S. (1982). Residuals and Influence in Regression . Chapman and Hall, London. · Zbl 0564.62054 |
| [20] | Cook, R. D. and Weisberg, S. (1994). An Introduction to Regression Graphics. Wiley, New York. · Zbl 0925.62287 |
| [21] | Cox, D. R. (1968). Notes on some aspects of regression analysis. J. Roy. Statist. Soc. Ser. A 131 265–279. |
| [22] | Cox, D. R. (1990). Role of models in statistical analysis. Statist. Sci. 5 169–174. · Zbl 0955.62518 · doi:10.1214/ss/1177012165 |
| [23] | de Leeuw, L. (2006). Principal component analysis of binary data by iterated singular value decomposition. Comput. Statist. Data Anal. 50 21–39. · Zbl 1429.62218 |
| [24] | Edelman, A., Arias, T. A. and Smith, S. T. (1998). The geometry of algorithms with orthogonality constraints. SIAM J. Matrix Anal. Appl. 20 303–353. · Zbl 0928.65050 · doi:10.1137/S0895479895290954 |
| [25] | Edgeworth, F. Y. (1884). On the reduction of observations. Philosophical Magazine 135–141. |
| [26] | Eubank, R. L. (1988). Spline Smoothing and Nonparametric Regression. Dekker, New York. · Zbl 0702.62036 |
| [27] | Fearn, T. (1983). A misuse of ridge regression in the calibration of a near infrared reflectance instrument. Appl. Statist. 32 73–79. |
| [28] | Fisher, R. A. (1922). On the mathematical foundations of theoretical statistics. Philos. Trans. Roy. Soc. London Ser. A 222 309–368. · JFM 48.1280.02 |
| [29] | Fisher, R. A. (1924). The influence of rainfall on the yield of wheat at Rothamsted. Philos. Trans. Roy. Soc. London Ser. B 213 89–142. |
| [30] | Fisher, R. A. (1936). Uncertain inference. Proc. American Academy of Arts and Sciences 71 245–258. · Zbl 0015.36301 |
| [31] | Fisher, R. A. (1941). Statistical Methods for Research Workers , 8th ed. Oliver and Boyd, London. · Zbl 0063.01385 |
| [32] | George, E. I. and Oman, S. D. (1996). Multiple-shrinkage principal component regression. The Statistician 45 111–124. |
| [33] | Gould, S. J. (1981). The Mismeasure of Man . Norton, New York. |
| [34] | Hadi, A. S. and Ling, R. F. (1998). Some cautionary notes on the use of principal components regression. Amer. Statist. 52 15–19. |
| [35] | Hawkins, D. M. and Fatti, L. P. (1984). Exploring multivariate data using the minor principal components. The Statistician 33 325–338. |
| [36] | Helland, I. S. (1992). Maximum likelihood regression on relevant components. J. Roy. Statist. Soc. Ser. B 54 637–647. JSTOR: · Zbl 0774.62053 |
| [37] | Helland, I. S. and Almøy, T. (1994). Comparison of prediction methods when only a few components are relevant. J. Amer. Statist. Assoc. 89 583–591. JSTOR: · Zbl 0799.62080 · doi:10.2307/2290861 |
| [38] | Hocking, R. R. (1976). The analysis and selection of variables in linear regression. Biometrics 32 1–49. JSTOR: · Zbl 0328.62042 · doi:10.2307/2529336 |
| [39] | Hotelling, H. (1933). Analysis of a complex statistical variable into principal components. J. Educational Psychology 24 417–441. · JFM 59.1182.04 |
| [40] | Hotelling, H. (1957). The relationship of the newer multivariate statistical methods to factor analysis. British J. Statist. Psychology 10 69–79. |
| [41] | Hwang, J. T. G. and Nettleton, D. (2003). Principal components regression with data-chosen components and related methods. Technometrics 45 70–79. · doi:10.1198/004017002188618716 |
| [42] | Jolliffe, I. T. (1982). A note on the use of principal components in regression. Appl. Statist. 31 300–303. |
| [43] | Jolliffe, I. T. (2002). Principal Component Analysis , 2nd ed. Springer, New York. · Zbl 1011.62064 · doi:10.1007/b98835 |
| [44] | Jong, J. and Kotz, S. (1999). On a relation between principal components and regression analysis. Amer. Statist. 53 349–351. JSTOR: · doi:10.2307/2686055 |
| [45] | Kendall, M. G. (1957). A Course in Multivariate Analysis . Griffin, London. · Zbl 0077.24208 |
| [46] | Lehmann, E. L. (1990). Model specification: The views of Fisher, Neyman, and later developments. Statist. Sci. 5 160–168. · Zbl 0955.62516 · doi:10.1214/ss/1177012164 |
| [47] | Li, K.-C. (1991). Sliced inverse regression for dimension reduction (with discussion). J. Amer. Statist. Assoc. 86 316–342. JSTOR: · Zbl 0742.62044 · doi:10.2307/2290563 |
| [48] | Li, L. and Li, H. (2004). Dimension reduction methods for microarrays with application to censored survival data. Bioinformatics 20 3406–3412. · Zbl 1022.68519 |
| [49] | Maronna, R. (2005). Principal components and orthogonal regression based on robust scales. Technometrics 47 264–273. · doi:10.1198/004017005000000166 |
| [50] | Marx, B. D. and Smith, E. P. (1990). Principal component estimation for generalized linear regression. Biometrika 77 23–31. JSTOR: · Zbl 0692.62051 · doi:10.1093/biomet/77.1.23 |
| [51] | McCullagh, P. (2002). What is a statistical model? (with discussion). Ann. Statist. 30 1225–1310. · Zbl 1039.62003 · doi:10.1214/aos/1035844977 |
| [52] | Mosteller, F. and Tukey, J. W. (1977). Data Analysis and Regression : A Second Course in Statistics . Addison–Wesley, Reading, MA. |
| [53] | Muirhead, R. J. (1982). Aspects of Multivariate Statistical Theory. Wiley, New York. · Zbl 0556.62028 |
| [54] | Oman, S. D. (1991). Random calibration with many measurements: An application of Stein estimation. Technometrics 33 187–195. |
| [55] | Pearson, K. (1901). On lines and planes of closest fit to systems of points in space. Philosophical Magazine (6) 2 559–572. · JFM 32.0710.04 |
| [56] | Rao, C. R. (1962). Use of discriminant and allied functions in multivariate analysis. Sankhyā Ser. A 24 149–154. · Zbl 0105.12401 |
| [57] | Savage, L. J. (1976). On rereading R. A. Fisher (with discussion). Ann. Statist. 4 441–500. · Zbl 0325.62008 · doi:10.1214/aos/1176343456 |
| [58] | Scott, D. (1992). Multivariate Density Estimation . Wiley, New York. · Zbl 0850.62006 |
| [59] | Seber, G. A. F. (1984). Multivariate Observations. Wiley, New York. · Zbl 0627.62052 |
| [60] | Spearman, C. (1904). “General intelligence,” objectively determined and measured. Amer. J. Psychology 15 201–292. |
| [61] | Stigler, S. M. (1973). Studies in the history of probability and statistics. XXXII. Laplace, Fisher and the discovery of the concept of sufficiency. Biometrika 60 439–445. JSTOR: · Zbl 0286.01010 |
| [62] | Stigler, S. M. (1976). Discussion of “On rereading R. A. Fisher,” by L. J. Savage. Ann. Statist. 4 498–500. · Zbl 0325.62008 · doi:10.1214/aos/1176343456 |
| [63] | Stigler, S. M. (2005). Fisher in 1921. Statist. Sci. 20 32–49. · Zbl 1100.01511 · doi:10.1214/088342305000000025 |
| [64] | Tipping, M. E. and Bishop, C. M. (1999). Probabilistic principal component analysis. J. R. Stat. Soc. Ser. B Stat. Methodol. 61 611–622. JSTOR: · Zbl 0924.62068 · doi:10.1111/1467-9868.00196 |
| [65] | Xia, Y., Tong, H., Li, W. K. and Zhu, L.-X. (2002). An adaptive estimation of dimension reduction space (with discussion). J. R. Stat. Soc. Ser. B Stat. Methodol. 64 363–410. JSTOR: · Zbl 1091.62028 · doi:10.1111/1467-9868.03411 |
| [66] | Zhao, L. P. and Prentice, R. L. (1990). Correlated binary regression using a quadratic exponential model. Biometrika 77 642–648. JSTOR: · doi:10.1093/biomet/77.3.642 |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
