Radial basis function kernel

The RBF kernel on two samples x , x ′ ∈ R k {\displaystyle \mathbf {x} ,\mathbf {x'} \in \mathbb {R} ^{k}} , represented as feature vectors in some input space, is defined as K ( x , x ′ ) = exp ⁡ ( − ‖ x − x ′ ‖ 2 2 σ 2 ) {\displaystyle K(\mathbf {x} ,\mathbf {x'} )=\exp \left(-{\frac {\|\mathbf {x} -\mathbf {x'} \|^{2}}{2\sigma ^{2}}}\right)} ‖ x − x ′ ‖ 2 {\displaystyle \textstyle \|\mathbf {x} -\mathbf {x'} \|^{2}} may be recognized as the squared Euclidean distance between the two feature vectors. σ {\displaystyle \sigma } is a free parameter. An equivalent definition involves a parameter γ = 1 2 σ 2 {\displaystyle \textstyle \gamma ={\tfrac {1}{2\sigma ^{2}}}} : K ( x , x ′ ) = exp ⁡ ( − γ ‖ x − x ′ ‖ 2 ) {\displaystyle K(\mathbf {x} ,\mathbf {x'} )=\exp(-\gamma \|\mathbf {x} -\mathbf {x'} \|^{2})} Since the value of the RBF kernel decreases with distance and ranges between zero (in the infinite-distance limit) and one (when x = x'), it has a ready interpretation as a similarity measure. The feature space of the kernel has an infinite number of dimensions; for σ = 1 {\displaystyle \sigma =1} , its expansion using the multinomial theorem is: exp ⁡ ( − 1 2 ‖ x − x ′ ‖ 2 ) = exp ⁡ ( 2 2 x ⊤ x ′ − 1 2 ‖ x ‖ 2 − 1 2 ‖ x ′ ‖ 2 ) = exp ⁡ ( x ⊤ x ′ ) exp ⁡ ( − 1 2 ‖ x ‖ 2 ) exp ⁡ ( − 1 2 ‖ x ′ ‖ 2 ) = ∑ j = 0 ∞ ( x ⊤ x ′ ) j j ! exp ⁡ ( − 1 2 ‖ x ‖ 2 ) exp ⁡ ( − 1 2 ‖ x ′ ‖ 2 ) = ∑ j = 0 ∞ ∑ n 1 + n 2 + ⋯ + n k = j exp ⁡ ( − 1 2 ‖ x ‖ 2 ) x 1 n 1 ⋯ x k n k n 1 ! ⋯ n k ! exp ⁡ ( − 1 2 ‖ x ′ ‖ 2 ) x ′ 1 n 1 ⋯ x ′ k n k n 1 ! ⋯ n k ! = ⟨ φ ( x ) , φ ( x ′ ) ⟩ {\displaystyle {\begin{alignedat}{2}\exp \left(-{\frac {1}{2}}\|\mathbf {x} -\mathbf {x'} \|^{2}\right)&=\exp \left({\frac {2}{2}}\mathbf {x} ^{\top }\mathbf {x'} -{\frac {1}{2}}\|\mathbf {x} \|^{2}-{\frac {1}{2}}\|\mathbf {x'} \|^{2}\right)\\[5pt]&=\exp \left(\mathbf {x} ^{\top }\mathbf {x'} \right)\exp \left(-{\frac {1}{2}}\|\mathbf {x} \|^{2}\right)\exp \left(-{\frac {1}{2}}\|\mathbf {x'} \|^{2}\right)\\[5pt]&=\sum _{j=0}^{\infty }{\frac {(\mathbf {x} ^{\top }\mathbf {x'} )^{j}}{j!}}\exp \left(-{\frac {1}{2}}\|\mathbf {x} \|^{2}\right)\exp \left(-{\frac {1}{2}}\|\mathbf {x'} \|^{2}\right)\\[5pt]&=\sum _{j=0}^{\infty }\quad \sum _{n_{1}+n_{2}+\dots +n_{k}=j}\exp \left(-{\frac {1}{2}}\|\mathbf {x} \|^{2}\right){\frac {x_{1}^{n_{1}}\cdots x_{k}^{n_{k}}}{\sqrt {n_{1}!\cdots n_{k}!}}}\exp \left(-{\frac {1}{2}}\|\mathbf {x'} \|^{2}\right){\frac {{x'}_{1}^{n_{1}}\cdots {x'}_{k}^{n_{k}}}{\sqrt {n_{1}!\cdots n_{k}!}}}\\[5pt]&=\langle \varphi (\mathbf {x} ),\varphi (\mathbf {x'} )\rangle \end{alignedat}}} φ ( x ) = exp ⁡ ( − 1 2 ‖ x ‖ 2 ) ( a ℓ 0 ( 0 ) , a 1 ( 1 ) , … , a ℓ 1 ( 1 ) , … , a 1 ( j ) , … , a ℓ j ( j ) , … ) {\displaystyle \varphi (\mathbf {x} )=\exp \left(-{\frac {1}{2}}\|\mathbf {x} \|^{2}\right)\left(a_{\ell _{0}}^{(0)},a_{1}^{(1)},\dots ,a_{\ell _{1}}^{(1)},\dots ,a_{1}^{(j)},\dots ,a_{\ell _{j}}^{(j)},\dots \right)} where ℓ j = ( k + j − 1 j ) {\displaystyle \ell _{j}={\tbinom {k+j-1}{j}}} , a ℓ ( j ) = x 1 n 1 ⋯ x k n k n 1 ! ⋯ n k ! | n 1 + n 2 + ⋯ + n k = j ∧ 1 ≤ ℓ ≤ ℓ j {\displaystyle a_{\ell }^{(j)}={\frac {x_{1}^{n_{1}}\cdots x_{k}^{n_{k}}}{\sqrt {n_{1}!\cdots n_{k}!}}}\quad |\quad n_{1}+n_{2}+\dots +n_{k}=j\wedge 1\leq \ell \leq \ell _{j}} Because support vector machines and other models employing the kernel trick do not scale well to large numbers of training samples or large numbers of features in the input space, several approximations to the RBF kernel (and similar kernels) have been introduced. Typically, these take the form of a function z that maps a single vector to a vector of higher dimensionality, approximating the kernel: ⟨ z ( x ) , z ( x ′ ) ⟩ ≈ ⟨ φ ( x ) , φ ( x ′ ) ⟩ = K ( x , x ′ ) {\displaystyle \langle z(\mathbf {x} ),z(\mathbf {x'} )\rangle \approx \langle \varphi (\mathbf {x} ),\varphi (\mathbf {x'} )\rangle =K(\mathbf {x} ,\mathbf {x'} )}