Reproducing kernel Hilbert space

• Intro
  Reproducing kernel Hilbert spaces are particularly important because of the celebrated representer theorem which states that every function in an RKHS that minimizes an empirical risk functional can be written as a linear combination of the kernel function evaluated at training samples.

• Spaces
  1. Vector space : A set of vectors with operations of addition and scalar multiplication satisfying certain axioms.
  2. Normed space : A vector space with a function that assigns lengths to vectors.
  3. Banach space : A complete normed space where every Cauchy sequence converges within the space.
  4. Inner product space : A vector space with a function that defines an angle and length between vectors.
  5. Hilbert space : A complete inner product space where every Cauchy sequence converges within the space.
  
  
• Reproducing property
  $X$ is a non-empty set.
  $k : X \times X \to R$ is a kernel s.t. $k(x, y) = \langle \phi(x), \phi(y) \rangle_H$
  $\phi : X \to H$ is a feature map
  $$ f(x) = \langle f(\cdot), k(x, \cdot) \rangle_H $$
  
  
• Definition 
  Let $k : X \times X \to R$ be a positive definite kernel. The RKHS, $H_k$, generated by the kernel $k$, is a linear span of ${k(x, \cdot) : x \in X}$ equipped with the inner product $$ \langle f, g \rangle_H = \sum^r_{i=1} \sum^s_{j=1} \alpha_i \beta_j k(x_i, x_j), $$ where $f(\cdot) = \sum^r_{i = 1} \alpha_i k(x_i, \cdot)$ and $g(\cdot) = \sum^s_{j=1} \beta_j k(x_j, \cdot).$