Linear Algebra – Orthogonality and least squares – Applications

The least-squares solution can for instance be used to find a curve (such as a straight line) that best fits a set of data points.

Example: Consider the data points: \((−3, −2)\), \((−2, 0)\), \((1, 3)\) and \((2, 3)\). Now we are looking for a straight line \(y=ax+b\) that best fits these data points. By substituting the corresponding values for \(x\) and \(y\) we see that \(a\) and \(b\) should satisfy:

\[\left\{\begin{array}{rcr}-3a+b&=&-2\\-2a+b&=&0\\a+b&=&3\\2a+b&=&3\end{array}\right.\quad\Longrightarrow\quad A\mathbf{x}=\mathbf{b} \quad\text{with}\quad A=\begin{pmatrix}-3&1\\-2&1\\1&1\\2&1\end{pmatrix}\quad\text{and}\quad\mathbf{b}=\begin{pmatrix}-2\\0\\3\\3\end{pmatrix}.\]

It is clear that \(A\mathbf{x}=\mathbf{b}\) is inconsistent. So the four points do not lie on a straight line. Now we determine a least-squares solution of \(A\mathbf{x}=\mathbf{b}\):

\[A^TA=\begin{pmatrix}-3&-2&1&2\\1&1&1&1\end{pmatrix}\begin{pmatrix}-3&1\\-2&1\\1&1\\2&1\end{pmatrix}=\begin{pmatrix}18&-2\\-2&4\end{pmatrix} \quad\text{and}\quad A^T\mathbf{b}=\begin{pmatrix}-3&-2&1&2\\1&1&1&1\end{pmatrix}\begin{pmatrix}-2\\0\\3\\3\end{pmatrix}=\begin{pmatrix}15\\4\end{pmatrix}.\]

Hence:

\[A^TA\hat{\mathbf{x}}=A^T\mathbf{b}:\quad\left(\left.\begin{matrix}18&-2\\-2&4\end{matrix}\,\right|\,\begin{matrix}15\\4\end{matrix}\right) \sim\left(\left.\begin{matrix}17&0\\-1&2\end{matrix}\,\right|\,\begin{matrix}17\\2\end{matrix}\right) \sim\left(\left.\begin{matrix}1&0\\0&2\end{matrix}\,\right|\,\begin{matrix}1\\3\end{matrix}\right)\quad\Longrightarrow\quad \hat{\mathbf{x}}=\frac{1}{2}\begin{pmatrix}2\\3\end{pmatrix}.\]

Now we choose \(a=1\) and \(b=\dfrac{3}{2}\), then we obtain the so-calles least-squares line \(y=x+\dfrac{3}{2}\).

In this way also other kinds of least-squares curves can be obtained that best fit a set of data points. The substitution of the corresponding values of \(x\) and \(y\) lead to a system of equations with the coefficients as unknowns. Then we choose the entries of the least-squares solution as the values for these coefficients.

Examples:

1) Consider the data points: \((-3,8)\), \((-2,3)\), \((0,2)\), \((1,4)\) and \((2,8)\). We look for a curve of the form \(y=a+bx^2\) that best fits these data points. By substituting the corresponding values for \(x\) and \(y\) we see that \(a\) and \(b\) should satisfy:

\[\left\{\begin{array}{rrcr}a&+9b&=&8\\a&+4b&=&3\\a&&=&2\\a&+b&=&4\\a&+4b&=&8\end{array}\right.\quad\Longrightarrow\quad A\mathbf{x}=\mathbf{b}\quad\text{with}\quad A=\begin{pmatrix}1&9\\1&4\\1&0\\1&1\\1&4\end{pmatrix}\quad\text{and}\quad \mathbf{b}=\begin{pmatrix}8\\3\\2\\4\\8\end{pmatrix}.\]

This sytem \(A\mathbf{x}=\mathbf{b}\) is inconsistent too. So we determine a least-squares solution of \(A\mathbf{x}=\mathbf{b}\):

\[A^TA=\begin{pmatrix}1&1&1&1&1\\9&4&0&1&4\end{pmatrix}\begin{pmatrix}1&9\\1&4\\1&0\\1&1\\1&4\end{pmatrix}=\begin{pmatrix}5&18\\18&114\end{pmatrix} \quad\text{and}\quad A^T\mathbf{b}=\begin{pmatrix}1&1&1&1&1\\9&4&0&1&4\end{pmatrix}\begin{pmatrix}8\\3\\2\\4\\8\end{pmatrix}=\begin{pmatrix}24\\111\end{pmatrix}.\]

Hence:

\[A^TA\hat{\mathbf{x}}=A^T\mathbf{b}:\quad\left(\left.\begin{matrix}5&18\\18&114\end{matrix}\,\right|\,\begin{matrix}24\\111\end{matrix}\right) \sim\left(\left.\begin{matrix}5&18\\-2&42\end{matrix}\,\right|\,\begin{matrix}24\\15\end{matrix}\right) \sim\left(\left.\begin{matrix}1&102\\-2&42\end{matrix}\,\right|\,\begin{matrix}54\\15\end{matrix}\right) \sim\left(\left.\begin{matrix}1&102\\0&246\end{matrix}\,\right|\,\begin{matrix}54\\123\end{matrix}\right) \sim\left(\left.\begin{matrix}1&0\\0&2\end{matrix}\,\right|\,\begin{matrix}3\\1\end{matrix}\right)\quad\Longrightarrow\quad \hat{\mathbf{x}}=\frac{1}{2}\begin{pmatrix}6\\1\end{pmatrix}.\]

Now we choose \(a=3\) and \(b=\dfrac{1}{2}\), then we obtain the so-called least-squares curve \(y=3+\dfrac{1}{2}x^2\).

2) Consider the data points: \((-2,5)\), \((-1,6)\), \((0,5)\), \((1,2)\) and \((2,-3)\). We look for a parabola of the form \(y=a+bx+cx^2\) that best fits these data points. By substituting the corresponding values for \(x\) and \(y\) we see that \(a\), \(b\) and \(c\) should satisfy:

\[\left\{\begin{array}{rrrcr}a&-2b&+4c&=&5\\a&-b&+c&=&6\\a&&&=&5\\a&+b&+c&=&2\\a&+2b&+4c&=&-3\end{array}\right.\quad\Longrightarrow\quad A\mathbf{x}=\mathbf{b}\quad\text{with}\quad A=\begin{pmatrix}1&-2&4\\1&-1&1\\1&0&0\\1&1&1\\1&2&4\end{pmatrix}\quad\text{and}\quad \mathbf{b}=\begin{pmatrix}5\\6\\5\\2\\-3\end{pmatrix}.\]

This system \(A\mathbf{x}=\mathbf{b}\) is inconsistent too. So we determine a least-squares solution of \(A\mathbf{x}=\mathbf{b}\):

\[A^TA=\begin{pmatrix}1&1&1&1&1\\-2&-1&0&1&2\\4&1&0&1&4\end{pmatrix}\begin{pmatrix}1&-2&4\\1&-1&1\\1&0&0\\1&1&1\\1&2&4\end{pmatrix} =\begin{pmatrix}5&0&10\\0&10&0\\10&0&34\end{pmatrix}\quad\text{and}\quad A^T\mathbf{b}=\begin{pmatrix}1&1&1&1&1\\-2&-1&0&1&2\\4&1&0&1&4\end{pmatrix} \begin{pmatrix}5\\6\\5\\2\\-3\end{pmatrix}=\begin{pmatrix}15\\-20\\16\end{pmatrix}.\]

Hence:

\[A^TA\hat{\mathbf{x}}=A^T\mathbf{b}:\quad\left(\left.\begin{matrix}5&0&10\\0&10&0\\10&0&34\end{matrix}\,\right|\,\begin{matrix}15\\-20\\16\end{matrix}\right) \sim\left(\left.\begin{matrix}1&0&2\\0&1&0\\5&0&17\end{matrix}\,\right|\,\begin{matrix}3\\-2\\8\end{matrix}\right) \sim\left(\left.\begin{matrix}1&0&2\\0&1&0\\0&0&7\end{matrix}\,\right|\,\begin{matrix}3\\-2\\-7\end{matrix}\right) \sim\left(\left.\begin{matrix}1&0&0\\0&1&0\\0&0&1\end{matrix}\,\right|\,\begin{matrix}5\\-2\\-1\end{matrix}\right) \quad\Longrightarrow\quad\hat{\mathbf{x}}=\begin{pmatrix}5\\-2\\-1\end{pmatrix}.\]

Now we choose \(a=5\), \(b=-2\) and \(c=-1\), then we obtain the least-squares curve \(y=5-2x-x^2\).

3) Consider the data points: \((0,5)\), \((\frac{1}{2}\pi,-3)\), \((\pi,-2)\), \((\frac{3}{2}\pi,5)\) and \((2\pi,3)\). We look for a curve of the form \(y=a\cos(x)+b\sin(x)+c\) that best fits these data points. By substituting the corresponding values for \(x\) and \(y\) we see that \(a\), \(b\) and \(c\) should satisfy:

\[\left\{\begin{array}{rrcr}a&&+c&=&5\\&b&+c&=&-3\\-a&&+c&=&-2\\&-b&+c&=&5\\a&&+c&=&3\end{array}\right.\quad\Longrightarrow\quad A\mathbf{x}=\mathbf{b} \quad\text{with}\quad A=\begin{pmatrix}1&0&1\\0&1&1\\-1&0&1\\0&-1&1\\1&0&1\end{pmatrix}\quad\text{and}\quad\mathbf{b}=\begin{pmatrix}5\\-3\\-2\\5\\3\end{pmatrix}.\]

It is clear that this sytem is inconsistent, so we look for a least-squares solution:

\[A^TA=\begin{pmatrix}1&0&-1&0&1\\0&1&0&-1&0\\1&1&1&1&1\end{pmatrix}\begin{pmatrix}1&0&1\\0&1&1\\-1&0&1\\0&-1&1\\1&0&1\end{pmatrix} =\begin{pmatrix}3&0&1\\0&2&0\\1&0&5\end{pmatrix}\quad\text{and}\quad A^T\mathbf{b}=\begin{pmatrix}1&0&-1&0&1\\0&1&0&-1&0\\1&1&1&1&1\end{pmatrix} \begin{pmatrix}5\\-3\\-2\\5\\3\end{pmatrix}=\begin{pmatrix}10\\-8\\8\end{pmatrix}.\]

Hence:

\[A^TA\hat{\mathbf{x}}=A^T\mathbf{b}:\quad\left(\left.\begin{matrix}3&0&1\\0&2&0\\1&0&5\end{matrix}\,\right|\,\begin{matrix}10\\-8\\8\end{matrix}\right) \sim\left(\left.\begin{matrix}1&0&5\\0&1&0\\3&0&1\end{matrix}\,\right|\,\begin{matrix}8\\-4\\10\end{matrix}\right) \sim\left(\left.\begin{matrix}1&0&5\\0&1&0\\0&0&-14\end{matrix}\,\right|\,\begin{matrix}8\\-4\\-14\end{matrix}\right) \sim\left(\left.\begin{matrix}1&0&0\\0&1&0\\0&0&1\end{matrix}\,\right|\,\begin{matrix}3\\-4\\1\end{matrix}\right) \quad\Longrightarrow\quad\hat{\mathbf{x}}=\begin{pmatrix}3\\-4\\1\end{pmatrix}.\]

Now we choose \(a=3\), \(b=-4\) and \(c=1\), then we obtain the least-squares curve \(y=3\cos(x)-4\sin(x)+1\).

---

Last modified on May 2, 2021