Functional-bandwidth kernel for support vector machine with functional data: an alternating optimization algorithm. (English) Zbl 1430.90450
Summary: Functional data analysis (FDA) is devoted to the study of data which are functions. Support vector machine (SVM) is a benchmark tool for classification, in particular, of functional data. SVM is frequently used with a kernel (e.g., Gaussian) which involves a scalar bandwidth parameter. In this paper, we propose to use kernels with functional bandwidths. In this way, accuracy may be improved, and the time intervals critical for classification are identified. Tuning the functional parameters of the new kernel is a challenging task expressed as a continuous optimization problem, solved by means of a heuristic. Our experiments with benchmark data sets show the advantages of using functional parameters and the effectiveness of our approach.
MSC:
| 90C20 | Quadratic programming |
| 68T10 | Pattern recognition, speech recognition |
| 62H30 | Classification and discrimination; cluster analysis (statistical aspects) |
References:
| [1] | Akaike, H., A new look at the statistical model identification, IEEE Transactions on Automatic Control, 19, 6, 716-723 (1974) · Zbl 0314.62039 |
| [2] | Baíllo, A.; Cuevas, A.; Cuesta-Albertos, J. A., Supervised classification for a family of gaussian functional models, Scandinavian Journal of Statistics, 38, 3, 480-498 (2011) · Zbl 1246.62155 |
| [3] | Berrendero, J. R.; Cuevas, A.; Torrecilla, J. L., Variable selection in functional data classification: a maxima-hunting proposal, Statistica Sinica, 26, 619-638 (2016) · Zbl 1356.62079 |
| [4] | Biau, G.; Bunea, F.; Wegkamp, M. H., Functional classification in Hilbert spaces, IEEE Transactions on Information Theory, 51, 6, 2163-2172 (2005) · Zbl 1285.94015 |
| [5] | Biernacki, C.; Celeux, G.; Govaert, G., Assessing a mixture model for clustering with the integrated completed likelihood, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22, 7, 719-725 (2000) |
| [6] | Blanquero, R.; Carrizosa, E.; Chis, O.; Esteban, N.; Jiménez-Cordero, A.; Rodríguez, J. F.; Sillero-Denamiel, M. R., On extreme concentrations in chemical reaction networks with incomplete measurements, Industrial & Engineering Chemistry Research, 55, 11417-11430 (2016) |
| [7] | Blanquero, R.; Carrizosa, E.; Jiménez-Cordero, A.; Martín-Barragán, B., Variable Selection in Classification for Multivariate Functional Data, Technical Report (2017), University of Edinburgh - Universidad de Sevilla |
| [8] | Blanquero, R.; Carrizosa, E.; Jiménez-Cordero, A.; Rodríguez, J. F., A global optimization method for model selection in chemical reactions networks, Computers & Chemical Engineering, 93, 52-62 (2016) |
| [9] | Brooks, J. P., Support vector machines with the ramp loss and the hard margin loss, Operations Research, 59, 2, 467-479 (2011) · Zbl 1228.90057 |
| [10] | Bugeau, A., & Pérez, P. (2007). Bandwidth selection for kernel estimation in mixed multi-dimensional spaces.; Bugeau, A., & Pérez, P. (2007). Bandwidth selection for kernel estimation in mixed multi-dimensional spaces. |
| [11] | Cai, Z.; Fan, J.; Yao, Q., Functional-coefficient regression models for nonlinear time series, Journal of the American Statistical Association, 95, 451, 941-956 (2000) · Zbl 0996.62078 |
| [12] | Carrizosa, E.; Martín-Barragán, B.; Romero Morales, D., A nested heuristic for parameter tuning in support vector machines, Computers & Operations Research, 43, 328-334 (2014) · Zbl 1349.62260 |
| [13] | Carrizosa, E.; Romero Morales, D., Supervised classification and mathematical optimization, Computers & Operations Research, 40, 1, 150-165 (2013) · Zbl 1349.68135 |
| [14] | Cauwenberghs, G.; Poggio, T., Incremental and decremental support vector machine learning, Advances in neural information processing systems, 409-415 (2001) |
| [15] | Chen, Q.; Wynne, R.; Goulding, P.; Sandoz, D., The application of principal component analysis and kernel density estimation to enhance process monitoring, Control Engineering Practice, 8, 5, 531-543 (2000) |
| [16] | Chen, Y., Keogh, E., Hu, B., Begum, N., Bagnall, A., Mueen, A., & Batista, G. (2015). The UCR time series classification archive. www.cs.ucr.edu/ eamonn/time_series_data/; Chen, Y., Keogh, E., Hu, B., Begum, N., Bagnall, A., Mueen, A., & Batista, G. (2015). The UCR time series classification archive. www.cs.ucr.edu/ eamonn/time_series_data/ |
| [17] | Claeskens, G.; Croux, C.; Kerckhoven, J. V., An information criterion for variable selection in support vector machines, Journal of Machine Learning Research, 9, Mar, 541-558 (2008) · Zbl 1225.68166 |
| [18] | Colson, B.; Marcotte, P.; Savard, G., An overview of bilevel optimization, Annals of Operations Research, 153, 1, 235-256 (2007) · Zbl 1159.90483 |
| [19] | Cortes, C.; Vapnik, V., Support-vector networks, Machine Learning, 20, 3, 273-297 (1995) · Zbl 0831.68098 |
| [20] | Cristianini, N.; Shawe-Taylor, J., An introduction to support vector machines and other kernel-based learning methods (2000), Cambridge University Press |
| [21] | Cruz-Cano, R.; Chew, D. S.; Choi, K.-P.; Leung, M.-Y., Least-squares support vector machine approach to viral replication origin prediction, INFORMS Journal on Computing, 22, 3, 457-470 (2010) · Zbl 1243.68236 |
| [22] | Cuevas, A.; Febrero, M.; Fraiman, R., On the use of the bootstrap for estimating functions with functional data, Computational Statistics & Data Analysis, 51, 2, 1063-1074 (2006) · Zbl 1157.62390 |
| [23] | Cuevas, A.; Febrero, M.; Fraiman, R., Robust estimation and classification for functional data via projection-based depth notions, Computational Statistics, 22, 3, 481-496 (2007) · Zbl 1195.62032 |
| [24] | De Boor, C., A practical guide to splines. A practical guide to splines, Applied Mathematical Sciences, 27 (1978), Springer-Verlag New York · Zbl 0406.41003 |
| [25] | Delaigle, A.; Hall, P., Achieving near perfect classification for functional data, Journal of the Royal Statistical Society: Series B (Statistical Methodology), 74, 2, 267-286 (2012) · Zbl 1411.62164 |
| [26] | Duong, T.; Cowling, A.; Koch, I.; Wand, M., Feature significance for multivariate kernel density estimation, Computational Statistics & Data Analysis, 52, 9, 4225-4242 (2008) · Zbl 1452.62265 |
| [27] | Febrero-Bande, M.; Oviedo de la Fuente, M., Statistical computing in functional data analysis: the r package fda.usc, Journal of Statistical Software, 51, 4, 1-28 (2012) |
| [28] | Ferraty, F.; Vieu, P., Nonparametric functional data analysis: theory and practice (2006), Springer Science & Business Media · Zbl 1119.62046 |
| [29] | Ferris, M. C.; Munson, T. S., Semismooth support vector machines, Mathematical Programming, 101, 1, 185-204 (2004) · Zbl 1076.90041 |
| [30] | Friedman, J., Hastie, T., & Tibshirani, R. (2001a). Datasets for The Elements of Statistical Learninghttps://web.stanford.edu/ hastie/ElemStatLearn/data.html; Friedman, J., Hastie, T., & Tibshirani, R. (2001a). Datasets for The Elements of Statistical Learninghttps://web.stanford.edu/ hastie/ElemStatLearn/data.html · Zbl 0973.62007 |
| [31] | Friedman, J.; Hastie, T.; Tibshirani, R., The elements of statistical learning. The elements of statistical learning, Springer Series in Statistics, 1 (2001), Springer, Berlin · Zbl 0973.62007 |
| [32] | Hammer, B.; Villmann, T., Generalized relevance learning vector quantization, Neural Networks, 15, 8, 1059-1068 (2002) |
| [33] | Hofmann, T.; Schölkopf, B.; Smola, A. J., Kernel methods in machine learning, The Annals of Statistics, 36, 3, 1171-1220 (2008) · Zbl 1151.30007 |
| [34] | Jiménez-Cordero, A.; Maldonado, S., Automatic feature scaling and selection for support vector machine classification with functional data, Technical Report (2018), Universidad de los Andes - Universidad de Sevilla |
| [35] | Kadri, H.; Duflos, E.; Preux, P.; Canu, S.; Davy, M., Nonlinear functional regression: a functional RKHS approach, Proceedings of the International conference on artificial intelligence and statistics, 374-380 (2010) |
| [36] | Kästner, M.; Hammer, B.; Biehl, M.; Villmann, T., Functional relevance learning in generalized learning vector quantization, Neurocomputing, 90, 85-95 (2012) |
| [37] | Keerthi, S. S.; Lin, C.-J., Asymptotic behaviors of support vector machines with gaussian kernel, Neural computation, 15, 7, 1667-1689 (2003) · Zbl 1086.68569 |
| [38] | Laukaitis, A.; Račkauskas, A., Functional data analysis for clients segmentation tasks, European Journal of Operational Research, 163, 1, 210-216 (2005) · Zbl 1067.90118 |
| [39] | Lessmann, S.; Voß, S., A reference model for customer-centric data mining with support vector machines, European Journal of Operational Research, 199, 2, 520-530 (2009) · Zbl 1176.90340 |
| [40] | López-Pintado, S.; Romo, J., On the concept of depth for functional data, Journal of the American Statistical Association, 104, 486, 718-734 (2009) · Zbl 1388.62139 |
| [41] | Maldonado, S.; Pérez, J.; Bravo, C., Cost-based feature selection for support vector machines: An application in credit scoring, European Journal of Operational Research, 261, 2, 656-665 (2017) · Zbl 1403.90526 |
| [42] | Maldonado, S.; Weber, R.; Basak, J., Simultaneous feature selection and classification using kernel-penalized support vector machines, Information Sciences, 181, 1, 115-128 (2011) |
| [43] | Martín-Barragán, B.; Lillo, R.; Romo, J., Interpretable support vector machines for functional data, European Journal of Operational Research, 232, 1, 146-155 (2014) |
| [44] | Muñoz, A.; González, J., Representing functional data using support vector machines, Pattern Recognition Letters, 31, 6, 511-516 (2010) |
| [45] | Preda, C.; Saporta, G.; Lévéder, C., PLS classification of functional data, Computational Statistics, 22, 2, 223-235 (2007) · Zbl 1196.62086 |
| [46] | Ramsay, J. O.; Silverman, B. W., Applied functional data analysis: methods and case studies. Applied functional data analysis: methods and case studies, Springer Series in Statistics, 77 (2002), Springer-Verlag · Zbl 1011.62002 |
| [47] | Ramsay, J. O.; Silverman, B. W., Functional data analysis. Functional data analysis, Springer Series in Statistics (2005), Springer-Verlag · Zbl 1079.62006 |
| [48] | Richtárik, P.; Takáč, M., Parallel coordinate descent methods for big data optimization, Mathematical Programming, 156, 1-2, 433-484 (2016) · Zbl 1342.90102 |
| [49] | Rossi, F.; Villa, N., Support vector machine for functional data classification, Neurocomputing, 69, 7, 730-742 (2006) |
| [50] | Rossi, F.; Villa, N., Recent advances in the use of SVM for functional data classification, Functional and operatorial statistics, 273-280 (2008), Physica-Verlag HD: Physica-Verlag HD Heidelberg |
| [51] | Ruiz-Meana, M.; Garcia-Dorado, D.; Pina, P.; Inserte, J.; Agulló, L.; Soler-Soler, J., Cariporide preserves mitochondrial proton gradient and delays atp depletion in cardiomyocytes during ischemic conditions, American Journal of Physiology-Heart and Circulatory Physiology, 285, 3, H999-H1006 (2003) |
| [52] | Sain, S. R., Multivariate locally adaptive density estimation, Computational Statistics & Data Analysis, 39, 2, 165-186 (2002) · Zbl 1132.62329 |
| [53] | Sato, A.; Yamada, K., Generalized learning vector quantization, Advances in neural information processing systems, 423-429 (1996) |
| [54] | Schwarz, G., Estimating the dimension of a model, The Annals of Statistics, 6, 2, 461-464 (1978) · Zbl 0379.62005 |
| [55] | Sood, A.; James, G. M.; Tellis, G. J., Functional regression: A new model for predicting market penetration of new products, Marketing Science, 28, 1, 36-51 (2009) |
| [56] | Suykens, J.; Vandewalle, J., Least squares support vector machine classifiers, Neural Processing Letters, 9, 3, 293-300 (1999) |
| [57] | Székely, G. J.; Rizzo, M. L.; Bakirov, N. K., Measuring and testing dependence by correlation of distances, The Annals of Statistics, 35, 6, 2769-2794 (2007) · Zbl 1129.62059 |
| [58] | Torrecilla Noguerales, J. L., On the theory and practice of variable selection for functional data (2015), Universidad Autónoma de Madrid, (Ph.D. thesis) |
| [59] | Tuddenham, R. D.; Snyder, M. M., Physical growth of california boys and girls from birth to eighteen years., 1, 183-364 (1954), Publications in Child Development. University of California, Berkeley |
| [60] | Wang, H.; Yao, M., Fault detection of batch processes based on multivariate functional kernel principal component analysis, Chemometrics and Intelligent Laboratory Systems, 149, 78-89 (2015) |
| [61] | Wang, H.; Zheng, B.; Yoon, S. W.; Ko, H. S., A support vector machine-based ensemble algorithm for breast cancer diagnosis, European Journal of Operational Research, 267, 2, 687-699 (2018) · Zbl 1403.92109 |
| [62] | Wang, J.-L.; Chiou, J.-M.; Müller, H.-G., Functional Data Analysis, Annual Review of Statistics and Its Application, 3, 1, 257-295 (2016) |
| [63] | Wei, L. (2006). http://alumni.cs.ucr.edu/ wli/selfTraining/; Wei, L. (2006). http://alumni.cs.ucr.edu/ wli/selfTraining/ |
| [64] | Wei, L.; Keogh, E., Semi-supervised time series classification, Proceedings of the twelfth ACM SIGKDD international conference on knowledge discovery and data mining. Proceedings of the twelfth ACM SIGKDD international conference on knowledge discovery and data mining, KDD ’06, 748-753 (2006), ACM |
| [65] | Wu, C. O.; Chiang, C.-T.; Hoover, D. R., Asymptotic confidence regions for kernel smoothing of a varying-coefficient model with longitudinal data, Journal of the American Statistical Association, 93, 444, 1388-1402 (1998) · Zbl 1064.62523 |
| [66] | Xing, Z.; Pei, J.; Philip, S. Y., Early prediction on time series: A nearest neighbor approach, IJCAI, 1297-1302 (2009) |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
