Nonconvex Optimization for Low-rank Matrix Related Problems
Author: Zhenzhen Li
Publisher:
Published: 2020
Total Pages: 170
ISBN-13:
DOWNLOAD EBOOKPosts
Optimization on Low Rank Nonconvex Structures
Author: Hiroshi Konno
Publisher: Springer Science & Business Media
Published: 2013-12-01
Total Pages: 462
ISBN-13: 1461540984
DOWNLOAD EBOOKGlobal optimization is one of the fastest developing fields in mathematical optimization. In fact, an increasing number of remarkably efficient deterministic algorithms have been proposed in the last ten years for solving several classes of large scale specially structured problems encountered in such areas as chemical engineering, financial engineering, location and network optimization, production and inventory control, engineering design, computational geometry, and multi-objective and multi-level optimization. These new developments motivated the authors to write a new book devoted to global optimization problems with special structures. Most of these problems, though highly nonconvex, can be characterized by the property that they reduce to convex minimization problems when some of the variables are fixed. A number of recently developed algorithms have been proved surprisingly efficient for handling typical classes of problems exhibiting such structures, namely low rank nonconvex structures. Audience: The book will serve as a fundamental reference book for all those who are interested in mathematical optimization.
Optimality Guarantees for Non-convex Low Rank Matrix Recovery Problems
Author: Christopher Dale White
Publisher:
Published: 2015
Total Pages: 196
ISBN-13:
DOWNLOAD EBOOKLow rank matrices lie at the heart of many techniques in scientific computing and machine learning. In this thesis, we examine various scenarios in which we seek to recover an underlying low rank matrix from compressed or noisy measurements. Specifically, we consider the recovery of a rank r positive semidefinite matrix XX[superscript T] [element] R[superscript n x n] from m scalar measurements of the form [mathematic equation] via minimization of the natural l2 loss function [mathematic equation]; we also analyze the quadratic nonnegative matrix factorization (QNMF) approach to clustering where the matrix to be factorized is the transition matrix for a reversible Markov chain. In all of these instances, the optimization problem we wish to solve has many local optima and is highly non-convex. Instead of analyzing convex relaxations, which tend to be complicated and computationally expensive, we operate directly on the natural non-convex problems and prove both local and global optimality guarantees for a family of algorithms.
Non-convex Optimization for Machine Learning
Author: Prateek Jain
Publisher: Foundations and Trends in Machine Learning
Published: 2017-12-04
Total Pages: 218
ISBN-13: 9781680833683
DOWNLOAD EBOOKNon-convex Optimization for Machine Learning takes an in-depth look at the basics of non-convex optimization with applications to machine learning. It introduces the rich literature in this area, as well as equips the reader with the tools and techniques needed to apply and analyze simple but powerful procedures for non-convex problems. Non-convex Optimization for Machine Learning is as self-contained as possible while not losing focus of the main topic of non-convex optimization techniques. The monograph initiates the discussion with entire chapters devoted to presenting a tutorial-like treatment of basic concepts in convex analysis and optimization, as well as their non-convex counterparts. The monograph concludes with a look at four interesting applications in the areas of machine learning and signal processing, and exploring how the non-convex optimization techniques introduced earlier can be used to solve these problems. The monograph also contains, for each of the topics discussed, exercises and figures designed to engage the reader, as well as extensive bibliographic notes pointing towards classical works and recent advances. Non-convex Optimization for Machine Learning can be used for a semester-length course on the basics of non-convex optimization with applications to machine learning. On the other hand, it is also possible to cherry pick individual portions, such the chapter on sparse recovery, or the EM algorithm, for inclusion in a broader course. Several courses such as those in machine learning, optimization, and signal processing may benefit from the inclusion of such topics.
Optimization Algorithms on Matrix Manifolds
Author: P.-A. Absil
Publisher: Princeton University Press
Published: 2009-04-11
Total Pages: 240
ISBN-13: 1400830249
DOWNLOAD EBOOKMany problems in the sciences and engineering can be rephrased as optimization problems on matrix search spaces endowed with a so-called manifold structure. This book shows how to exploit the special structure of such problems to develop efficient numerical algorithms. It places careful emphasis on both the numerical formulation of the algorithm and its differential geometric abstraction--illustrating how good algorithms draw equally from the insights of differential geometry, optimization, and numerical analysis. Two more theoretical chapters provide readers with the background in differential geometry necessary to algorithmic development. In the other chapters, several well-known optimization methods such as steepest descent and conjugate gradients are generalized to abstract manifolds. The book provides a generic development of each of these methods, building upon the material of the geometric chapters. It then guides readers through the calculations that turn these geometrically formulated methods into concrete numerical algorithms. The state-of-the-art algorithms given as examples are competitive with the best existing algorithms for a selection of eigenspace problems in numerical linear algebra. Optimization Algorithms on Matrix Manifolds offers techniques with broad applications in linear algebra, signal processing, data mining, computer vision, and statistical analysis. It can serve as a graduate-level textbook and will be of interest to applied mathematicians, engineers, and computer scientists.
Non-convex Optimization Methods for Sparse and Low-rank Reconstruction
Author: Penghang Yin
Publisher:
Published: 2016
Total Pages: 93
ISBN-13: 9781339830124
DOWNLOAD EBOOKAn algorithmic framework, based on the difference of convex functions algorithm, is proposed for minimizing difference of ℓ1 and ℓ 2 norms (ℓ1-2 minimization) as well as a wide class of concave sparse metrics for compressed sensing problems. The resulting algorithm iterates a sequence of ℓ1 minimization problems. An exact sparse recovery theory is established to show that the proposed framework always improves on the basis pursuit (ℓ1 minimization) and inherits robustness from it. Numerical examples on success rates of sparse solution recovery illustrate further that, unlike most existing non-convex compressed sensing solvers in the literature, our method always out-performs basis pursuit, no matter how ill-conditioned the measurement matrix is.As the counterpart of ℓ1-2 minimization for low-rank matrix recovery, we present a phase retrieval method via minimization of the difference of trace and Frobenius norms which we call PhaseLiftOff. The associated least squares minimization with this penalty as regularization is equivalent to the original rank-one least squares problem under a mild condition on the measurement noise. Numerical results show that PhaseLiftOff outperforms the convex PhaseLift and its non-convex variant (log-determinant regularization), and successfully recovers signals near the theoretical lower limit on the number of measurements without the noise.
Handbook of Robust Low-Rank and Sparse Matrix Decomposition
Author: Thierry Bouwmans
Publisher: CRC Press
Published: 2016-05-27
Total Pages: 553
ISBN-13: 1498724639
DOWNLOAD EBOOKHandbook of Robust Low-Rank and Sparse Matrix Decomposition: Applications in Image and Video Processing shows you how robust subspace learning and tracking by decomposition into low-rank and sparse matrices provide a suitable framework for computer vision applications. Incorporating both existing and new ideas, the book conveniently gives you one-stop access to a number of different decompositions, algorithms, implementations, and benchmarking techniques. Divided into five parts, the book begins with an overall introduction to robust principal component analysis (PCA) via decomposition into low-rank and sparse matrices. The second part addresses robust matrix factorization/completion problems while the third part focuses on robust online subspace estimation, learning, and tracking. Covering applications in image and video processing, the fourth part discusses image analysis, image denoising, motion saliency detection, video coding, key frame extraction, and hyperspectral video processing. The final part presents resources and applications in background/foreground separation for video surveillance. With contributions from leading teams around the world, this handbook provides a complete overview of the concepts, theories, algorithms, and applications related to robust low-rank and sparse matrix decompositions. It is designed for researchers, developers, and graduate students in computer vision, image and video processing, real-time architecture, machine learning, and data mining.
Handbook of Variational Methods for Nonlinear Geometric Data
Author: Philipp Grohs
Publisher: Springer Nature
Published: 2020-04-03
Total Pages: 701
ISBN-13: 3030313514
DOWNLOAD EBOOKThis book covers different, current research directions in the context of variational methods for non-linear geometric data. Each chapter is authored by leading experts in the respective discipline and provides an introduction, an overview and a description of the current state of the art. Non-linear geometric data arises in various applications in science and engineering. Examples of nonlinear data spaces are diverse and include, for instance, nonlinear spaces of matrices, spaces of curves, shapes as well as manifolds of probability measures. Applications can be found in biology, medicine, product engineering, geography and computer vision for instance. Variational methods on the other hand have evolved to being amongst the most powerful tools for applied mathematics. They involve techniques from various branches of mathematics such as statistics, modeling, optimization, numerical mathematics and analysis. The vast majority of research on variational methods, however, is focused on data in linear spaces. Variational methods for non-linear data is currently an emerging research topic. As a result, and since such methods involve various branches of mathematics, there is a plethora of different, recent approaches dealing with different aspects of variational methods for nonlinear geometric data. Research results are rather scattered and appear in journals of different mathematical communities. The main purpose of the book is to account for that by providing, for the first time, a comprehensive collection of different research directions and existing approaches in this context. It is organized in a way that leading researchers from the different fields provide an introductory overview of recent research directions in their respective discipline. As such, the book is a unique reference work for both newcomers in the field of variational methods for non-linear geometric data, as well as for established experts that aim at to exploit new research directions or collaborations. Chapter 9 of this book is available open access under a CC BY 4.0 license at link.springer.com.
Statistical Inference and Optimization for Low-rank Matrix and Tensor Learning
Author: Yuetian Luo (Ph.D.)
Publisher:
Published: 2022
Total Pages: 0
ISBN-13:
DOWNLOAD EBOOKHigh dimensional statistical problems with matrix or tensor type data are ubiquitous in modern data analysis. In many applications, the dimension of the matrix or tensor is high and much bigger than the sample size and some structural assumptions are often imposed to ensure the problem is well-posed. One of the most popular structures in matrix and tensor data analysis is the low-rankness. In this thesis, we make contributions to the statistical inference and optimization in low-rank matrix and tensor data analysis from the following three aspects. First, first-order algorithms have been the workhorse in modern data analysis, including matrix and tensor problems, for their simplicity and efficiency. Second-order algorithms suffer from high computational costs and instability. The first part of the thesis explores the following question: can we develop provable efficient second-order algorithms for high-dimensional matrix and tensor problems with low-rank structures? We provide a positive answer to this question, where the key idea is to explore smooth Riemannian structures of the sets of low-rank matrices and tensors and the connection to the second-order Riemannian optimization methods. In particular, we demonstrate that for a large class of tensor-on-tensor regression problems, the Riemannian Gauss-Newton algorithm is computationally fast and achieves provable second-order convergence. We also discuss the case when the intrinsic rank of the parameter matrix/tensor is unknown and a natural rank overspecification is implemented. In the second part of the thesis, we explore an interesting question: is there any connection between different non-convex optimization approaches for solving the general low-rank matrix optimization? We find from a geometric point of view, the common non-convex factorization formulation has a close connection with the Riemannian formulation and there exists an equivalence between them. Moreover, we discover that two notable Riemannian formulations, i.e., formulations under Riemannian embedded and quotient geometries, are also closely related from a geometric point of view. In the final part of the thesis, we are dedicated to studying one intriguing phenomenon in high dimensional statistical problems, statistical and computational trade-offs, which refers to the commonly appearing gaps between the different signal-to-noise ratio thresholds that make the problem information-theoretically solvable or polynomial-time solvable. Here we focus on the statistical-computational trade-offs induced by tensor structures. We would provide rigorous evidence for the computational barriers to two important classes of problems: tensor clustering and tensor regression. We show these computational limits by the average-case reduction and restricted class of low-degree polynomials arguments.
Provable Non-convex Optimization for Learning Parametric Models
Author: Kai Zhong (Ph. D.)
Publisher:
Published: 2018
Total Pages: 866
ISBN-13:
DOWNLOAD EBOOKNon-convex optimization plays an important role in recent advances of machine learning. A large number of machine learning tasks are performed by solving a non-convex optimization problem, which is generally NP-hard. Heuristics, such as stochastic gradient descent, are employed to solve non-convex problems and work decently well in practice despite the lack of general theoretical guarantees. In this thesis, we study a series of non-convex optimization strategies and prove that they lead to the global optimal solution for several machine learning problems, including mixed linear regression, one-hidden-layer (convolutional) neural networks, non-linear inductive matrix completion, and low-rank matrix sensing. At a high level, we show that the non-convex objectives formulated in the above problems have a large basin of attraction around the global optima when the data has benign statistical properties. Therefore, local search heuristics, such as gradient descent or alternating minimization, are guaranteed to converge to the global optima if initialized properly. Furthermore, we show that spectral methods can efficiently initialize the parameters such that they fall into the basin of attraction. Experiments on synthetic datasets and real applications are carried out to justify our theoretical analyses and illustrate the superiority of our proposed methods.
