Least Squared Error Fitting

A method of linear regression that aims to reduce the sum of the square of the errors.

The sum of the square of errors for $n$ data-points is notated as:

$$E^2=\sum _{i=1}^n{ϵ}_i^2=\sum _{i=1}^n(y_i-{\hat{y}}_i{)}^2$$

Bad notation

$E^2$ is not $(E{)}^2$. This is just an example of shitty notation, because we should all know that:

$$(a+b+c+\dots {)}^2\ne a^2+b^2+c^2$$

Similiarly:

$$(E{)}^2\ne E^2$$

Using Linear Algebra §

Recall that $\hat{y}=a+bx$

$E^2$ can be written as:

$$E^2=\Vert \vec{y}-\mathbf{A}\vec{u}{\Vert }^2$$

where:

$$\vec{y}=\left[\begin{array}{c}{y}_1\\ y_2\\ ⋮\\ y_n\end{array}\right]\mathbf{A}=\left[\begin{array}{cc}1& x_1\\ 1& x_2\\ ⋮& ⋮\\ 1& x_n\end{array}\right]\vec{u}=\left[\begin{array}{c}a\\ b\end{array}\right]\mathbf{A}\vec{u}=\left[\begin{array}{c}a+b{x}_1\\ a+b{x}_2\\ ⋮\\ a+b{x}_n\end{array}\right]$$

Then, $E^2$ is reduced whenever $\mathbf{A}\vec{u}$ is close to $\vec{y}$. Since we can’t actually change the values of $\mathbf{A}$, we will have to adjust $\vec{u}$ (and through it, $a$ and $b$).

We do so by taking the column space of $\mathbf{A}$. Let $W$ be the column space of $\mathbf{A}$. We assume that all values are real, so $W$ is a subspace of ${\mathbb{R}}^2$

$$W=\{\mathbf{A}\vec{v}|\vec{v}\in {\mathbb{R}}^2\}$$

We also defined $W$ to be an inner product space, with it’s inner product being the dot product.

Then, our goal is to find a value of $\vec{u}$ such that the distance between $\mathbf{A}\vec{u}$ and $\vec{y}$ is minimised.

We can use the properties of orthogonal projection:

So the vector closest to $\vec{y}$ in $W$ is ${\text{proj}}_W(\vec{y})$. Let $\mathbf{A}\vec{u}={\text{proj}}_W(\vec{y})$. Another property of orthogonal projection can be used:

Let $\vec{w}\in W$ (i.e. $\vec{w}=\mathbf{A}\vec{v},\vec{v}\in {\mathbb{R}}^2$). Then

$$⟨\vec{w},\vec{y}-{\text{proj}}_W(\vec{y})⟩=0$$

$$⟨\mathbf{A}\vec{v},\vec{y}-\mathbf{A}\vec{u}⟩=0$$

Since the inner product of $W$ is the dot product:

$$(\mathbf{A}\vec{v})\cdot (\vec{y}-\mathbf{A}\vec{u})=0$$

Now we use a property unique to the dot product:

$$(\mathbf{A}\vec{v}{)}^T(\vec{y}-\mathbf{A}\vec{u})=0$$

Then using properties of matrix transposition:

$${\vec{v}}^T{\mathbf{A}}^T(\vec{y}-\mathbf{A}\vec{u})=0$$

Remember that $\vec{v}$ is any vector in ${\mathbb{R}}^2$. That means $\vec{v}$ can also be ${\mathbf{A}}^T(\vec{y}-\mathbf{A}\vec{u})$. In that case, we can take the positivity of the inner product:

$$⟨\vec{v},{\mathbf{A}}^T(\vec{y}-\mathbf{A}\vec{u})⟩=0⟺{\mathbf{A}}^T(\vec{y}-\mathbf{A}\vec{u})=\vec{0}$$

(Since we just look at the case where $\vec{v}={\mathbf{A}}^T(\vec{y}-\mathbf{A}\vec{u})$)

$${\mathbf{A}}^T(\vec{y}-\mathbf{A}\vec{u})=\vec{0}$$

$${\mathbf{A}}^T\vec{y}-{\mathbf{A}}^T\mathbf{A}\vec{u}=\vec{0}$$

$${\mathbf{A}}^T\mathbf{A}\vec{u}={\mathbf{A}}^T\vec{y}$$

And, if ${\mathbf{A}}^T\mathbf{A}$ is an invertible matrix, then:

$$\vec{u}=({\mathbf{A}}^T\mathbf{A}{)}^{-1}{\mathbf{A}}^T\vec{y}$$