Convex optimization

Convex optimization is a subfield of mathematical optimization that studies the problem of minimizing convex functions over convex sets (or, equivalently, maximizing concave functions over convex sets). Many classes of convex optimization problems admit polynomial-time algorithms, whereas mathematical optimization is in general NP-hard.

Definition

Abstract form

A convex optimization problem is defined by two ingredients:

The objective function, which is a real-valued convex function of n variables, <math>f :\mathcal D \subseteq \mathbb{R}^n \to \mathbb{R}</math>;
The feasible set, which is a convex subset <math>C\subseteq \mathbb{R}^n</math>.

The goal of the problem is to find some <math>\mathbf{x^\ast} \in C</math> attaining

:<math>\inf \{ f(\mathbf{x}) : \mathbf{x} \in C \}</math>.

In general, there are three options regarding the existence of a solution:

If such a point x* exists, it is referred to as an optimal point or solution; the set of all optimal points is called the optimal set; and the problem is called solvable.
If <math>f</math> is unbounded below over <math>C</math>, or the infimum is not attained, then the optimization problem is said to be unbounded.
Otherwise, if <math>C</math> is the empty set, then the problem is said to be infeasible.

Standard form

A convex optimization problem is in standard form if it is written as

:<math>\begin{align}

&\underset{\mathbf{x{\operatorname{minimize& & f(\mathbf{x}) \\

&\operatorname{subject\ to}

& &g_i(\mathbf{x}) \leq 0, \quad i = 1, \dots, m \\

&&&h_i(\mathbf{x}) = 0, \quad i = 1, \dots, p,

\end{align}</math>

where:

Epigraph form (standard form with linear objective)

In the standard form it is possible to assume, without loss of generality, that the objective function f is a linear function. This is because any program with a general objective can be transformed into a program with a linear objective by adding a single variable t and a single constraint, as follows:

:<math>\begin{align}

&\underset{\mathbf{x},t}{\operatorname{minimize& & t \\

&\operatorname{subject\ to}

& &f(\mathbf{x}) - t \leq 0 \\

&& &g_i(\mathbf{x}) \leq 0, \quad i = 1, \dots, m \\

&&&h_i(\mathbf{x}) = 0, \quad i = 1, \dots, p,

\end{align}</math>

Conic form

Every convex program can be presented in a conic form, which means minimizing a linear objective over the intersection of an affine plane and a convex cone:

thumb|A hierarchy of convex optimization problems. (LP: [[linear programming, QP: quadratic programming, SOCP second-order cone program, SDP: semidefinite programming, CP: conic optimization.)]]

Linear programming problems are the simplest convex programs. In LP, the objective and constraint functions are all linear.
Quadratic programming are the next-simplest. In QP, the constraints are all linear, but the objective may be a convex quadratic function.
Second order cone programming are more general.
Semidefinite programming are more general.
Conic optimization are even more general - see figure to the right,

Other special cases include;

Least squares
Quadratic minimization with convex quadratic constraints
Geometric programming
Entropy maximization with appropriate constraints.

Properties

The following are useful properties of convex optimization problems:

Bundle methods (Wolfe, Lemaréchal, Kiwiel), and
Subgradient projection methods (Polyak),
Interior-point methods, and self-regular barrier functions.
Cutting-plane methods
Ellipsoid method
Subgradient method
Dual subgradients and the drift-plus-penalty method

Subgradient methods can be implemented simply and so are widely used. Dual subgradient methods are subgradient methods applied to a dual problem. The drift-plus-penalty method is similar to the dual subgradient method, but takes a time average of the primal variables.

Lagrange multipliers

Consider a convex minimization problem given in standard form by a cost function <math>f(x)</math> and inequality constraints <math>g_i(x)\leq 0</math> for <math> 1 \leq i \leq m</math>. Then the domain <math>\mathcal{X}</math> is:

:<math>\mathcal{X} = \left\{x\in X \vert g_1(x), \ldots, g_m(x)\leq 0\right\}.</math>

The Lagrangian function for the problem is

:<math>L(x,\lambda_{0},\lambda_1, \ldots ,\lambda_{m})=\lambda_{0} f(x) + \lambda_{1} g_{1} (x)+\cdots + \lambda_{m} g_{m} (x).</math>

For each point <math>x</math> in <math>X</math> that minimizes <math>f</math> over <math>X</math>, there exist real numbers <math>\lambda_{0},\lambda_1, \ldots, \lambda_{m},</math> called Lagrange multipliers, that satisfy these conditions simultaneously:

<math>x</math> minimizes <math>L(y,\lambda_{0},\lambda_{1},\ldots ,\lambda_{m})</math> over all <math>y \in X,</math>
<math>\lambda_{0},\lambda_{1},\ldots ,\lambda_{m} \geq 0,</math> with at least one <math>\lambda_{k} > 0,</math>
<math>\lambda_{1}g_{1}(x)=\cdots = \lambda_{m}g_{m}(x) = 0</math> (complementary slackness).

If there exists a "strictly feasible point", that is, a point <math>z</math> satisfying

:<math>g_{1}(z), \ldots, g_{m}(z)<0,</math>

then the statement above can be strengthened to require that <math>\lambda_{0}=1</math>.

Conversely, if some <math>x</math> in <math>X</math> satisfies (1)–(3) for scalars <math>\lambda_{0},\ldots,\lambda_{m} </math> with <math>\lambda_{0}=1</math> then <math>x</math> is certain to minimize <math>f</math> over <math>X</math>.

Software

There is a large software ecosystem for convex optimization. This ecosystem has two main categories: solvers on the one hand and modeling tools (or interfaces) on the other hand.

Solvers implement the algorithms themselves and are usually written in C. They require users to specify optimization problems in very specific formats which may not be natural from a modeling perspective. Modeling tools are separate pieces of software that let the user specify an optimization in higher-level syntax. They manage all transformations to and from the user's high-level model and the solver's input/output format.

Below are two tables. The first shows modelling tools (such as CVXPY and JuMP.jl) and the second shows solvers (such as SCS and MOSEK). They are by no means exhaustive.

{| class="wikitable sortable"

!Program

!Language

!Description

!FOSS?

|CVX

|MATLAB

|Interfaces with SeDuMi and SDPT3 solvers; designed to only express convex optimization problems.

|CVXPY

|Python

|Convex.jl

|Julia

|Disciplined convex programming, supports many solvers.

|CVXR

|GAMS

|Modeling system for linear, nonlinear, mixed integer linear/nonlinear, and second-order cone programming problems.

|ROME

|Modeling system for robust optimization. Supports distributionally robust optimization and uncertainty sets.

| and structural optimization, where the approximation concept has proven to be efficient. Convex optimization can be used to model problems in the following fields:

Portfolio optimization.
Worst-case risk analysis.).
Electricity generation optimization.
Localization using wireless signals

Extensions

Extensions of convex optimization include the optimization of biconvex, pseudo-convex, and quasiconvex functions. Extensions of the theory of convex analysis and iterative methods for approximately solving non-convex minimization problems occur in the field of generalized convexity, also known as abstract convex analysis.