In mathematics, a norm in general is a function from a vector space to non-negative numbers. When the vector space comprises matrices, such norms are referred to as matrix norms. Matrix norms behave in certain ways like the distance from the zero matrix. They are distinguished from the norms in general, because they also interact with matrix multiplication in certain senses.
Many specific matrix norms can be defined. Most of them arise from the following three perspectives, though different perspectives may sometimes yield the same norm.
- Consider the matrix as a linear operator; then a matrix norm may describe how much the operator can stretch vectors. Such matrix norms induced by vector norms are called operator norms.
- Consider the matrix as a rectangular array of numbers; then a matrix norm may be defined as a certain sum of the entries. Such matrix norms are sometimes called "entry-wise" norms.
- The singular value decomposition is useful in analyzing matrices. A vector norm of the singular values of a matrix may be taken as a matrix norm. Such norms are called Schatten norms.
Matrix norms are often denoted by double vertical bars with optional subscripts (e.g., <math>\|A\|</math> or <math>\|A\|_2</math>). However, the meaning of the subscript may vary, since matrix norms in different perspectives relate to <math>\ell^p</math>-norms in different ways.
{| class="wikitable" style="margin: auto; border: none;"
|+ Matrix norms in different perspectives
|-
! Operator norms !! "Entry-wise" norms !! Schatten norms !! Also known as
|-
| induced from <math>\ell^1</math>-norm || maximum column sum || ||
|-
| || || <math>\ell^1</math>-norm (sum of singular values) || nuclear norm
|-
| || square root of the sum of squares || <math>\ell^2</math>-norm of singular values || Frobenius norm
|-
| induced from <math>\ell^2</math>-norm || || <math>\ell^\infty</math>-norm (the largest singular value) || spectral norm
|-
| induced from <math>\ell^\infty</math>-norm || maximum row sum || ||
|}
Preliminaries
Given a field <math>\ K\ </math> of either real or complex numbers (or any complete subset thereof), let <math>\ K^{m \times n}\ </math> be the -vector space of matrices with <math>m</math> rows and <math>n</math> columns and entries in the field <math>\ K ~.</math> A matrix norm is a norm on <math>\ K^{m \times n}~.</math>
The matrix norm is a function <math>\ \|\cdot\| : K^{m \times n} \to \R^{0+}\ </math> that must satisfy the following properties:
For all scalars <math>\ \alpha \in K\ </math> and matrices <math>\ A, B \in K^{m \times n}\ ,</math>
- <math> \|A\| \ge 0\ </math> (positive-valued)
- <math> \|A\| = 0 \iff A=0_{m,n}</math> (definite)
- <math> \left\| \alpha\ A \right\| = \left| \alpha \right|\ \left\|A\right\|\ </math> (absolutely homogeneous)
- <math> \| A + B \| \le \| A \| + \| B \|\ </math> (sub-additive or satisfying the triangle inequality)
The only feature distinguishing matrices from rearranged vectors is multiplication. Matrix norms are particularly useful if they are also sub-multiplicative:
- <math>\ \left\| AB \right\| \le \left\| A \right\| \left\| B \right\|\ </math>
Every norm on <math>\ K^{n\times n}\ </math> can be rescaled to be sub-multiplicative; in some books, the terminology matrix norm is reserved for sub-multiplicative norms.
Possible properties
Unitary invariance
A matrix norm is called unitarily invariant if for all unitary matrices <math>U,V </math> and matrix <math>A </math>, <math>\lVert UAV \rVert = \lVert A \rVert </math>.
A symmetric gauge function is an absolute vector norm <math>\phi: \C^p \to \R^+ </math> such that <math>\phi(Px) = \phi(x) </math> for any permutation matrix <math>P </math>. That is:
- Non-negativity: <math>\phi(x)\geq0</math>, and <math>\phi(x)=0</math> if and only if <math>x=0</math>.
- Positive homogeneity: <math>\phi(\alpha x)=|\alpha|\phi(x)</math> for any real number <math> \alpha </math>.
- Triangle inequality: <math>\phi(x+y)\leq \phi(x)+\phi(y)</math>.
- Symmetry: <math> \phi(Px) = \phi(x) </math> for any permutation matrix <math>P</math>.
A norm is a unitarily invariant matrix norm if and only if it is a symmetric gauge function on the vector of singular values.
Matrix norms induced by vector norms
Suppose a vector norm <math>\|\cdot\|_{\alpha}</math> on <math>K^n</math> and a vector norm <math>\|\cdot\|_{\beta}</math> on <math>K^m</math> are given. Any <math>m \times n</math> matrix induces a linear operator from <math>K^n</math> to <math>K^m</math> with respect to the standard basis, and one defines the corresponding induced norm or operator norm or subordinate norm on the space <math>K^{m \times n}</math> of all <math>m \times n</math> matrices as follows:
<math display="block">
\|A\|_{\alpha, \beta} = \sup\{ \|Ax\|_\beta : x \in K^n \text{ such that } \|x\|_\alpha \leq 1 \}
</math>
where <math> \sup </math> denotes the supremum. This norm measures how much the mapping induced by <math>A</math> can stretch vectors.
Depending on the vector norms <math>\|\cdot\|_{\alpha}</math>, <math>\|\cdot\|_{\beta}</math> used, notation other than <math>\|\cdot\|_{\alpha,\beta}</math> can be used for the operator norm.
Matrix norms induced by vector p-norms
If the p-norm for vectors (<math>1 \leq p \leq \infty</math>) is used for both spaces <math>K^n</math> and <math>K^m,</math> then the corresponding operator norm is:<math display="block"> \|A\|_2 = \sqrt{\lambda_{\max}\left(A^* A\right)} = \sigma_{\max}(A).</math>where <math>\sigma_{\max}(A)</math> represents the largest singular value of matrix <math>A.</math>
There are further properties:
- <math display="inline">\|A \|_2 = \sup\{x^* A y : x \in K^m, y \in K^n \text{ with }\|x\|_2 = \|y\|_2 = 1\}.</math> Proved by the Cauchy–Schwarz inequality.
- <math display="inline"> \| A^* A\|_2 = \| A A^* \|_2 = \|A\|_2^2</math>. Proven by singular value decomposition (SVD) on <math>A</math>.
- <math display="inline"> \|A\| _2 = \sigma_{\mathrm{max(A) \leq \|A\|_{\rm F} = \sqrt{\sum_i \sigma_{i}(A)^2}</math>, where <math>\|A\|_\textrm{F}</math> is the Frobenius norm. Equality holds if and only if the matrix <math>A</math> is a rank-one matrix or a zero matrix.
- Conversely, <math>\|A\|_\textrm{F} \leq \min(m,n)^{1/2}\|A\|_2</math>.
- <math> \|A\|_2 = \sqrt{\rho(A^{*}A)}\leq\sqrt{\|A^{*}A\|_\infty}\leq\sqrt{\|A\|_1\|A\|_\infty} </math>.
Matrix norms induced by vector α- and β-norms
We can generalize the above definition. Suppose we have vector norms <math>\|\cdot\|_{\alpha}</math> and <math>\|\cdot\|_{\beta}</math> for spaces <math>K^n</math> and <math>K^m</math> respectively; the corresponding operator norm is
<math display="block">
\|A\|_{\alpha, \beta} = \sup\{ \|Ax\|_\beta : x \in K^n \text{ such that } \|x\|_\alpha \leq 1 \}
</math>
In particular, the <math>\|A\|_{p}</math> defined previously is the special case of <math>\|A\|_{p, p}</math>.
In the special cases of <math>\alpha = 2</math> and <math>\beta=\infty</math>, the induced matrix norms can be computed by<math display="block"> \|A\|_{2,\infty}= \max_{1\le i\le m}\|A_{i:}\|_2, </math> where <math>A_{i:}</math> is the i-th row of matrix <math> A </math>.
In the special cases of <math>\alpha = 1</math> and <math>\beta=2</math>, the induced matrix norms can be computed by<math display="block"> \|A\|_{1, 2} = \max_{1\le j\le n}\|A_{:j}\|_2, </math> where <math>A_{:j}</math> is the j-th column of matrix <math> A </math>.
Hence, <math> \|A\|_{2,\infty} </math> and <math> \|A\|_{1, 2} </math> are the maximum row and column 2-norm of the matrix, respectively.
Properties
Any operator norm is consistent with the vector norms that induce it, giving
<math display="block">\|Ax\|_\beta \leq \|A\|_{\alpha,\beta}\|x\|_\alpha.</math>
Suppose <math>\|\cdot\|_{\alpha,\beta}</math>; <math>\|\cdot\|_{\beta,\gamma}</math>; and <math>\|\cdot\|_{\alpha,\gamma}</math> are operator norms induced by the respective pairs of vector norms <math>(\|\cdot\|_\alpha, \|\cdot\|_\beta)</math>; <math>(\|\cdot\|_\beta, \|\cdot\|_{\gamma})</math>; and <math>(\|\cdot\|_\alpha, \|\cdot\|_\gamma)</math>. Then,
:<math>\|AB\|_{\alpha,\gamma} \leq \|A\|_{\beta, \gamma} \|B\|_{\alpha, \beta} ;</math>
this follows from
<math display="block">\|ABx\|_\gamma \leq \|A\|_{\beta, \gamma} \|Bx\|_\beta \leq \|A\|_{\beta, \gamma} \|B\|_{\alpha, \beta} \|x\|_\alpha </math>
and
<math display="block">\sup_{\|x\|_\alpha = 1} \|ABx \|_\gamma = \|AB\|_{\alpha, \gamma} .</math>
Square matrices
Suppose <math>\|\cdot\|_{\alpha, \alpha}</math> is an operator norm on the space of square matrices <math>K^{n \times n}</math>
induced by vector norms <math>\|\cdot\|_{\alpha}</math> and <math>\|\cdot\|_\alpha</math>.
Then, the operator norm is a sub-multiplicative matrix norm:
<math display="block">\|AB\|_{\alpha, \alpha} \leq \|A\|_{\alpha, \alpha} \|B\|_{\alpha, \alpha}.</math>
Moreover, any such norm satisfies the inequality
for all positive integers r, where is the spectral radius of . For symmetric or hermitian , we have equality in () for the 2-norm, since in this case the 2-norm is precisely the spectral radius of . For an arbitrary matrix, we may not have equality for any norm; a counterexample would be
<math display="block">A = \begin{bmatrix} 0 & 1 \\ 0 & 0 \end{bmatrix},</math>
which has vanishing spectral radius. In any case, for any matrix norm, we have the spectral radius formula:
<math display="block">\lim_{r\to\infty}\|A^r\|^{1/r}=\rho(A). </math>
Energy norms
If the vector norms <math>\|\cdot\|_{\alpha}</math> and <math>\|\cdot\|_{\beta}</math> are given in terms of energy norms based on symmetric positive definite matrices <math>P</math> and <math>Q</math> respectively, the resulting operator norm is given as
<math display="block">
\|A\|_{P, Q} = \sup \{ \|Ax\|_Q : \|x\|_P \leq 1 \}.
</math>
Using the symmetric matrix square roots of <math>P</math> and <math>Q</math> respectively, the operator norm can be expressed as the spectral norm of a modified matrix:
<math display="block">
\|A\|_{P, Q} = \|Q^{1/2} A P^{-1/2}\|_{2}.
</math>
"Entry-wise" matrix norms
These norms treat an <math> m \times n </math> matrix as a vector of size <math> m \cdot n </math>, and use one of the familiar vector norms. For example, using the p-norm for vectors, , we get:
:<math>\| A \|_{p,p} = \| \mathrm{vec}(A) \|_p = \left( \sum_{i=1}^m \sum_{j=1}^n |a_{ij}|^p \right)^{1/p}</math>
This is a different norm from the induced p-norm (see above) and the Schatten p-norm (see below), but the notation is the same.
The special case p = 2 is the Frobenius norm, and p = ∞ yields the maximum norm.
and norms
Let <math>(a_1, \ldots, a_n) </math> be the dimension columns of matrix <math>A</math>. From the original definition, the matrix <math> A </math> presents data points in an -dimensional space. The <math>L_{2,1}</math> norm is the sum of the Euclidean norms of the columns of the matrix:
:<math>\| A \|_{2,1} = \sum_{j=1}^n \| a_{j} \|_2 = \sum_{j=1}^n \left( \sum_{i=1}^m |a_{ij}|^2 \right)^{1/2}</math>
The <math>L_{2,1}</math> norm as an error function is more robust, since the error for each data point (a column) is not squared. It is used in robust data analysis and sparse coding.
For , the <math>L_{2,1}</math> norm can be generalized to the <math>L_{p,q}</math> norm as follows:
:<math>\| A \|_{p,q} = \left(\sum_{j=1}^n \left( \sum_{i=1}^m |a_{ij}|^p \right)^{\frac{q}{p\right)^{\frac{1}{q.</math>
Frobenius norm
When for the <math>L_{p,q}</math> norm, it is called the Frobenius norm or the Hilbert–Schmidt norm, though the latter term is used more frequently in the context of operators on (possibly infinite-dimensional) Hilbert space. This norm can be defined in various ways:
:<math>\|A\|_\text{F} = \sqrt{\sum_{i}^m\sum_{j}^n |a_{ij}|^2} = \sqrt{\operatorname{trace}\left(A^* A\right)} = \sqrt{\sum_{i=1}^{\min\{m, n\ \sigma_i^2(A)},</math>
where the trace is the sum of diagonal entries, and <math>\sigma_i(A)</math> are the singular values of <math>A</math>. The second equality is proven by explicit computation of <math>\mathrm{trace}(A^*A)</math>. The third equality is proven by singular value decomposition of <math>A</math>, and the fact that the trace is invariant under circular shifts.
The Frobenius norm is an extension of the Euclidean norm to <math>K^{n \times n}</math> and comes from the Frobenius inner product on the space of all matrices.
The Frobenius norm is sub-multiplicative and is very useful for numerical linear algebra. The sub-multiplicativity of Frobenius norm can be proved using the Cauchy–Schwarz inequality. In fact, it is more than sub-multiplicative, as <math display="block">\|AB\|_F \leq\|A\|_{op}\|B\|_F</math>where the operator norm <math>\|\cdot\|_{op} \leq \|\cdot\|_{F}</math>.
Frobenius norm is often easier to compute than induced norms, and has the useful property of being invariant under rotations (and unitary operations in general). That is, <math>\|A\|_\text{F} = \|AU\|_\text{F} = \|UA\|_\text{F}</math> for any unitary matrix <math>U</math>. This property follows from the cyclic nature of the trace (<math>\operatorname{trace}(XYZ) =\operatorname{trace}(YZX) = \operatorname{trace}(ZXY)</math>):
:<math>\|AU\|_\text{F}^2 = \operatorname{trace}\left( (AU)^{*}A U \right)
= \operatorname{trace}\left( U^{*} A^{*}A U \right)
= \operatorname{trace}\left( UU^{*} A^{*}A \right)
= \operatorname{trace}\left( A^{*} A \right)
= \|A\|_\text{F}^2,</math>
and analogously:
:<math>\|UA\|_\text{F}^2 = \operatorname{trace}\left( (UA)^{*}UA \right)
= \operatorname{trace}\left( A^{*} U^{*} UA \right)
= \operatorname{trace}\left( A^{*}A \right)
= \|A\|_\text{F}^2,</math>
where we have used the unitary nature of <math>U</math> (that is, <math>U^* U = U U^* = \mathbf{I}</math>).
It also satisfies
:<math>\|A^* A\|_\text{F} = \|AA^*\|_\text{F} \leq \|A\|_\text{F}^2</math>
and
:<math>\|A + B\|_\text{F}^2 = \|A\|_\text{F}^2 + \|B\|_\text{F}^2 + 2 \operatorname{Re} \left( \langle A, B \rangle_\text{F} \right),</math>
where <math>\langle A, B \rangle_\text{F}</math> is the Frobenius inner product, and Re is the real part of a complex number (irrelevant for real matrices)
Max norm
The max norm is the elementwise norm in the limit as goes to infinity:
:<math> \|A\|_{\max} = \max_{i, j} |a_{ij}|. </math>
This norm is not sub-multiplicative; but modifying the right-hand side to <math>\sqrt{m n} \max_{i, j} \vert a_{i j} \vert</math> makes it so.
Note that in some literature (such as Communication complexity), an alternative definition of max-norm, also called the <math>\gamma_2</math>-norm, refers to the factorization norm:
:<math> \gamma_2(A) = \min_{U,V: A = UV^T} \| U \|_{2,\infty} \| V \|_{2,\infty} = \min_{U,V: A = UV^T} \max_{i,j} \| U_{i,:} \|_2 \| V_{j,:} \|_2 </math>
Schatten norms
The Schatten p-norms arise when applying the p-norm to the vector of singular values of a matrix.), defined as:
: <math>\|A\|_{*} = \operatorname{trace} \left(\sqrt{A^*A}\right) = \sum_{i=1}^{\min\{m,n\ \sigma_i(A),</math>
where <math>\sqrt{A^*A}</math> denotes a positive semidefinite matrix <math>B</math> such that <math>BB=A^*A</math>. More precisely, since <math>A^*A</math> is a positive semidefinite matrix, its square root is well defined. The nuclear norm <math>\|A\|_{*}</math> is a convex envelope of the rank function <math>\text{rank}(A)</math>, so it is often used in mathematical optimization to search for low-rank matrices.
Combining von Neumann's trace inequality with Hölder's inequality for Euclidean space yields a version of Hölder's inequality for Schatten norms for <math> 1/p + 1/q = 1 </math>:
: <math>\left|\operatorname{trace}(A^*B)\right| \le \|A\|_p \|B\|_q,</math>
In particular, this implies the Schatten norm inequality
: <math> \|A\|_F^2 \le \|A\|_p \|A\|_q. </math>
Cut norms
Another source of inspiration for matrix norms arises from considering a matrix as the adjacency matrix of a weighted, directed graph. The so-called "cut norm" measures how close the associated graph is to being bipartite:
<math display="block">\|A\|_{\Box}=\max_{S\subseteq[n], T\subseteq[m]}{\left|\sum_{s\in S,t\in T}{A_{t,s\right|}</math>
where . Equivalent definitions (up to a constant factor) impose the conditions ; ; or .
- <math>\|A\|_2\le\|A\|_F\le\sqrt{r}\|A\|_2</math>
- <math>\|A\|_F \le \|A\|_{*} \le \sqrt{r} \|A\|_F</math>
- <math>\|A\|_{\max} \le \|A\|_2 \le \sqrt{mn}\|A\|_{\max}</math>
- <math>\frac{1}{\sqrt{n\|A\|_\infty\le\|A\|_2\le\sqrt{m}\|A\|_\infty</math>
- <math>\frac{1}{\sqrt{m\|A\|_1\le\|A\|_2\le\sqrt{n}\|A\|_1.</math>
See also
- Dual norm
- Logarithmic norm
Notes
References
Bibliography
- James W. Demmel, Applied Numerical Linear Algebra, section 1.7, published by SIAM, 1997.
- Carl D. Meyer, Matrix Analysis and Applied Linear Algebra, published by SIAM, 2000. [http://www.matrixanalysis.com]
- John Watrous, Theory of Quantum Information, 2.3 Norms of operators, lecture notes, University of Waterloo, 2011.
- Kendall Atkinson, An Introduction to Numerical Analysis, published by John Wiley & Sons, Inc 1989