Stat/Junk

Inner products and Norms

repaired_stat 2023. 2. 17. 13:33
  • Inner products
    • Definition
      Defiition 1 (Inner product). A function $\langle \cdot , \cdot \rangle$ : $\mathbb{R}^n \times \mathbb{R}^n → \mathbb{R}$ is an inner product if
      1. $\langle x , x \rangle \geq 0$, $\langle x , x \rangle = 0 \Leftrightarrow x = 0$ (positivity)
      2. $\langle x , y \rangle = \langle y , x \rangle$ (symmetry)
      3. $\langle x + y , z \rangle = \langle x , z \rangle + \langle y , z \rangle$ (additivity)
      4. $\langle rx , y \rangle = r \langle x , y \rangle$ for all $r \in \mathbb{R}$ (homogeneity)
    • Examples
      1. The standard inner product is $$ \langle x , y \rangle = x^T y = \sum x_i y_i, ~~~ x, y \in \mathbb{R}^n $$
      2. The standard inner product between matrices is $$ \langle X , Y \rangle = Tr(X^T Y) = \sum_i \sum_j X_{ij} Y_{ij}, ~~~ X, Y \in \mathbb{R}^{m \times n} $$
    • Properties of inner products
      1. Definition 2 (Orthogonality). We say that $x$ and $y$ are orthogonal if $$ \langle x , y \rangle = 0 $$
      2. Theorem 1 (Cauchy Schwarz). For $x, y ∈ \mathbb{R}^n$, $$ | \langle x , y \rangle | \leq  \| x \| ~ \| y \|, $$ where $ \| x \| := \sqrt{\langle x , x \rangle}$.

 

  • Norms
    • Definition
      Definition 3 (Norm). A function $f : \mathbb{R}^n → \mathbb{R}$ is a norm if
      1. $f(x) \geq 0$, $f(x) = 0 \Leftrightarrow x = 0$ (positivity)
      2. $f(\alpha x) = |\alpha| f(x)$ for all $\alpha \in \mathbb{R}$ (homogeneity)
      3. $f(x + y) \leq f(x) + f(y)$ (triangle inequality)
    • Examples
      1. The 2-norm : $ \| x \| = \sqrt{\sum_i x_i^2} $
      2. The 1-norm : $ \| x \|_1 = \sum_i |x_i| $
      3. The inf-norm : $ \| x \|_\infty = max_i |x_i| $
      4. The p-norm : $ \| x \|_p = (\sum_i |x_i|^p )^{1 / p}, p \geq 1 $
    • Properties of norms
      1. Lemma 1. Take any inner product $\langle \cdot , \cdot \rangle$ and define $f(x) = \sqrt{\langle x, x \rangle}$. Then $f$ is a norm.

 

  • Norms on Spaces of random variables
    • $L^p$ norm : $ \| X \|_{L^p} = (E |X|^p)^{1/p}, ~~~ p \in (0, \infty) $
      $L^p$ space : $\{ X : \| X \|_{L^p} < \infty \}$
    • $L^\infty$ norm : $ \| X \|_{L^\infty} = \text{ess sup }|X| $