Inner products and Norms

• Inner products
  • Definition
    Defiition 1 (Inner product). A function $\langle \cdot , \cdot \rangle$ : $\mathbb{R}^n \times \mathbb{R}^n → \mathbb{R}$ is an inner product if
    1. $\langle x , x \rangle \geq 0$, $\langle x , x \rangle = 0 \Leftrightarrow x = 0$ (positivity)
    2. $\langle x , y \rangle = \langle y , x \rangle$ (symmetry)
    3. $\langle x + y , z \rangle = \langle x , z \rangle + \langle y , z \rangle$ (additivity)
    4. $\langle rx , y \rangle = r \langle x , y \rangle$ for all $r \in \mathbb{R}$ (homogeneity)
  • Examples
    1. The standard inner product is $$ \langle x , y \rangle = x^T y = \sum x_i y_i, ~~~ x, y \in \mathbb{R}^n $$
    2. The standard inner product between matrices is $$ \langle X , Y \rangle = Tr(X^T Y) = \sum_i \sum_j X_{ij} Y_{ij}, ~~~ X, Y \in \mathbb{R}^{m \times n} $$
  • Properties of inner products
    1. Definition 2 (Orthogonality). We say that $x$ and $y$ are orthogonal if $$ \langle x , y \rangle = 0 $$
    2. Theorem 1 (Cauchy Schwarz). For $x, y ∈ \mathbb{R}^n$, $$ | \langle x , y \rangle | \leq  \| x \| ~ \| y \|, $$ where $ \| x \| := \sqrt{\langle x , x \rangle}$.

 

• Norms
  • Definition
    Definition 3 (Norm). A function $f : \mathbb{R}^n → \mathbb{R}$ is a norm if
    1. $f(x) \geq 0$, $f(x) = 0 \Leftrightarrow x = 0$ (positivity)
    2. $f(\alpha x) = |\alpha| f(x)$ for all $\alpha \in \mathbb{R}$ (homogeneity)
    3. $f(x + y) \leq f(x) + f(y)$ (triangle inequality)
  • Examples
    1. The 2-norm : $ \| x \| = \sqrt{\sum_i x_i^2} $
    2. The 1-norm : $ \| x \|_1 = \sum_i |x_i| $
    3. The inf-norm : $ \| x \|_\infty = max_i |x_i| $
    4. The p-norm : $ \| x \|_p = (\sum_i |x_i|^p )^{1 / p}, p \geq 1 $
  • Properties of norms
    1. Lemma 1. Take any inner product $\langle \cdot , \cdot \rangle$ and define $f(x) = \sqrt{\langle x, x \rangle}$. Then $f$ is a norm.

 

• Norms on Spaces of random variables
  • $L^p$ norm : $ \| X \|_{L^p} = (E |X|^p)^{1/p}, ~~~ p \in (0, \infty) $
    $L^p$ space : $\{ X : \| X \|_{L^p} < \infty \}$
  • $L^\infty$ norm : $ \| X \|_{L^\infty} = \text{ess sup }|X| $