×

Strong convergence of trajectories via inertial dynamics combining Hessian-driven damping and Tikhonov regularization for general convex minimizations. (English) Zbl 1530.37112

Summary: Let \(\mathscr{H}\) be a real Hilbert space, and \(f:\mathscr{H} \to \mathbb{R}\) be a convex twice differentiable function whose solution set \(\operatorname{argmin}f\) is nonempty. We investigate the long time behavior of the trajectories of the vanishing damped dynamical system with Tikhonov regularizing term and Hessian-driven damping \[ \ddot{x}(t) + \alpha \dot{x}(t) + \delta\nabla^2 f (x(t))\dot{x}(t) + \beta(t)\nabla f (x(t)) + cx(t) = 0 \] where \(\alpha\), \(c\), \(\delta\) are three positive constants, and the time scale parameter \(\beta\) is a positive nondecreasing function such that \(\lim_{t\to +\infty} \beta(t) = +\infty\). Under some assumptions on the parameter \(\beta\), we will show rapid convergence of values, strong convergence toward the minimum norm element of \(\operatorname{argmin}f\), and rapid convergence of the gradients toward zero. Note that the Hessian-driven damping significantly reduces the oscillatory aspects, and the time scale parameter \(\beta\) improves the rate of convergences mentioned above. As particular cases of \(\beta\), we set \(\beta(t) = t^p \ln^q(t)\), for \((p, q) \in (\mathbb{R}_+)^2 \backslash \{(0, 0)\}\), and \(\beta(t) = e^{\gamma t^p}\), for \(p \in ]0, 1[\) and \(\gamma > 0\). The manuscript concludes with two numerical examples and comments on their performance.

MSC:

37N40 Dynamical systems in optimization and economics
46N10 Applications of functional analysis in optimization, convex analysis, mathematical programming, economics
65K05 Numerical mathematical programming methods
65K10 Numerical optimization and variational techniques
90B50 Management decision making, including multiple objectives
90C25 Convex programming
Full Text: DOI

References:

[1] Cabot, A., Proximal point algorithm controlled by a slowly vanishing term: applications to hierarchical minimization, SIAM J. Optim., 15, 555-572 (2005) · Zbl 1079.90098 · doi:10.1137/S105262340343467X
[2] Attouch, H.; Czarnecki, M.-O., Asymptotic behavior of coupled dynamical systems with multiscale aspects, J. Differ. Equ, 248, 1315-1344 (2010) · Zbl 1190.37090 · doi:10.1016/j.jde.2009.06.014
[3] Attouch, H.; Czarnecki, M.-O., Asymptotic behavior of gradient-like dynamical systems involving inertia and multiscale aspects, J. Differ. Equ, 262, 3, 2745-2770 (2017) · Zbl 1375.37188 · doi:10.1016/j.jde.2016.11.009
[4] Chadli, O.; Chbani, Z.; Riahi, H., Equilibrium problems with generalized monotone bifunctions and applications to variational inequalities, J. Optim. Theory Appl, 105, 299-323 (2000) · Zbl 0966.91049 · doi:10.1023/A:1004657817758
[5] Attouch, H., Applicable Mathematics Series, Variational Convergence for Functions and Operators (1984), Boston: Pitman Advanced Publishing Program, Boston · Zbl 0561.49012
[6] Rockafellar, R. T., Extension of Fenchel’s duality theorem for convex functions, Duke Math. J., 33, 81-89 (1966) · Zbl 0138.09301 · doi:10.1215/S0012-7094-66-03312-6
[7] Attouch, H., Viscosity solutions of minimization problems, SIAM J. Optim., 6, 3, 769-806 (1996) · Zbl 0859.65065 · doi:10.1137/S1052623493259616
[8] Attouch, H.; Cominetti, R., A dynamical approach to convex minimization coupling approximation with the steepest descent method, J. Differ. Equ, 128, 2, 519-540 (1996) · Zbl 0886.49024 · doi:10.1006/jdeq.1996.0104
[9] Polyak, B., Some methods of speeding up the convergence of iteration methods, USSR Comput. Math. Math. Phys, 4, 1-17 (1964) · Zbl 0147.35301 · doi:10.1016/0041-5553(64)90137-5
[10] Polyak, B., Introduction to Optimization (1987), New York: Optimization Software-Inc, New York · Zbl 0625.62093
[11] Attouch, H.; Chbani, Z.; Fadili, J.; Riahi, H., First-order algorithms via inertial systems with Hessian driven damping, Math. Program., 193, 113-155 (2022) · Zbl 1497.37121 · doi:10.1007/s10107-020-01591-1
[12] Attouch, H.; Balhag, A.; Chbani, Z.; Riahi, H., Damped inertial dynamics with vanishing Tikhonov regularization: strong asymptotic convergence towards the minimum norm solution, J. Differ. Equ, 311, 29-58 (2022) · Zbl 1489.37113 · doi:10.1016/j.jde.2021.12.005
[13] Boţ, R. I.; Csetnek, E. R.; László, S. C., Tikhonov regularization of a second order dynamical system with Hessian driven damping, Math. Program, 189, 151-186 (2021) · Zbl 1489.34088 · doi:10.1007/s10107-020-01528-8
[14] Cabot, A., Inertial gradient-like dynamical system controlled by a stabilizing term, J. Optim. Theory Appl, 120, 275-303 (2004) · Zbl 1070.90107 · doi:10.1023/B:JOTA.0000015685.21638.8d
[15] Cabot, A.; Engler, H.; Gadat, S., On the long time behavior of second order differential equations with asymptotically small dissipation, Trans. Amer. Math. Soc., 361, 5983-6017 (2009) · Zbl 1191.34078 · doi:10.1090/S0002-9947-09-04785-0
[16] Jendoubi, M. A.; May, R., On an asymptotically autonomous system with Tikhonov type regularizing term, Arch. Math. (Basel, 95, 4, 389-399 (2010) · Zbl 1217.34085 · doi:10.1007/s00013-010-0181-6
[17] Su, W.; Boyd, S.; Candès, E. J., A differential equation for modeling Nesterov’s accelerated gradient method: Theory and insights, J. Mach. Learn. Res, 17, 5312-5354 (2016) · Zbl 1391.90667 · doi:10.5555/2946645.3053435
[18] Attouch, H.; Chbani, Z.; Riahi, H., Combining fast inertial dynamics for convex optimization with Tikhonov regularization, J. Math. Anal. Appl, 457, 1065-1094 (2018) · Zbl 1375.65080 · doi:10.1016/j.jmaa.2016.12.017
[19] Attouch, H.; Cabot, A.; Redont, P., The dynamics of elastic shocks via epigraphical regularization of a differential inclusion, Adv. Math. Sci. Appl, 12, 273-306 (2002) · Zbl 1038.49029
[20] Nesterov, Y., A method of solving a convex programming problem with convergence rate, Soviet Math. Dokl., 27, 372-376 (1983) · Zbl 0535.90071
[21] Nesterov, Y., Introductory Lectures on Convex Optimization: A Basic Course, Volume 87 of Applied Optimization (2004), Boston, MA: Kluwer Academic Publishers, Boston, MA · Zbl 1086.90045
[22] Nesterov, Y., How to make the gradients small, Optima, MPS Newsletter, 88, 10-11 (2012)
[23] Nesterov, Y., Gradient methods for minimizing composite functions, Math. Program., 140, 1, 125-161 (2013) · Zbl 1287.90067 · doi:10.1007/s10107-012-0629-5
[24] Bo̧t, R. I.; Csetnek, E. R.; László, S. C., Tikhonov regularization of a second order dynamical system with Hessian driven damping, 189, 151-186 (2021) · Zbl 1489.34088 · doi:10.1007/s10107-020-01528-8
[25] Csetnek, R. E., Karapetyants, M. A. (2022). A fast continuous time approach for non-smooth convex optimization with time scaling and Tikhonov regularization, preprint.
[26] Attouch, H., László, S. C. (2021). Convex optimization via inertial algorithms with vanishing Tikhonov regularization: fast convergence to the minimum norm solution. arXiv.2104.11987. DOI: .
[27] László, S. C. (2022). On the strong convergence of the trajectories of a Tikhonov regularized second order dynamical system with asymptotically vanishing damping. arXiv:2202.08980.
[28] Attouch, H., Balhag, A., Chbani, Z., Riahi, H. (2022). Accelerated gradient methods combining Tikhonov regularization with geometric damping driven by the Hessian. arXiv.2203.05457. DOI: . · Zbl 1522.37096
[29] Attouch, H., Boţ, R. I., Nguyen, D.-K. (2022). Fast convex optimization via time scale and averaging of the steepest descent. arXiv.2208.08260. DOI: .
[30] Attouch, H., Chbani, Z., Riahi, H. (2022). Accelerated gradient methods with strong convergence to the minimum norm minimizer: a dynamic approach combining time scaling, averaging, and Tikhonov regularization. arXiv.2211.10140. DOI: .
[31] Attouch, H.; Chbani, Z.; Riahi, H., Fast proximal methods via time scaling of damped inertial dynamics, SIAM J. Optim., 29, 3, 2227-2256 (2019) · Zbl 07105235 · doi:10.1137/18M1230207
[32] Attouch, H.; Chbani, Z.; Riahi, H., Fast convex optimization via time scaling of damped inertial gradient dynamics, Pure Appl. Funct. Anal., 6, 6, 1081-1117 (2021) · Zbl 1489.37095
[33] Attouch, H.; Balhag, A.; Chbani, Z.; Riahi, H., Fast convex optimization via inertial dynamics combining viscous and Hessian-driven damping with time rescaling, Evol. Equ. Control Theory, 11, 2, 487-514 (2022) · Zbl 1495.37066 · doi:10.3934/eect.2021010
[34] Bagy, A. C.; Chbani, Z.; Riahi, H., The Heavy ball method regularized by Tikhonov term. Simultaneous convergence of values and trajectories, Evol. Equ. Control Theory, 12, 2, 687-702 (2023) · Zbl 1521.37098 · doi:10.3934/eect.2022046
[35] Brézis, H., Analyse Fonctionnelle, Théorie et Applications (1983), Paris: Masson, Paris · Zbl 0511.46001
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.