Least Square Solutions

LEAST SQUARE SOLUTIONS

A least square solution for a matrix equation Ax = b is a vector x that minimize the squared error or the squared Euclidean norm of the residual vector ( ||b−Ax||² ). This is typically used for over−determined systems (where there are more equations than unknowns) where no exact solutions exist. To find this solution we can solve the normal equations A^TAx = A^Tb.

1. How the least square solutions work

1. To find the closest vector

A least square solutions finds the vector Ax in the column space of A that is closest to b. This is equivalent to projecting b onto the column space of A.

2. The normal equations

The vector b−Ax must be orthogonal to the column space of A. This condition is expressed by the normal equations A^T(b−Ax) = 0, which simplifies to A^TAx=A^Tb.

3. To solve the matrix for x

The least square solution x can be found by solving the normal equations. It A^TA is invertible, the solution is unique and given by x = (A^TA)^‒1 A^T b.

4. The rule of the pseudo inverse

The term (A^TA)^‒1A^T is known as the pseudo inverse of A and can be used to find the least square solution for any matrix A.

2. Uses of least square solutions

1. Over−determined systems

Least square is mostly used for systems with more equations than unknowns. (Fitting a line to data points)

2. In consistent systems

It provides the "best approximate" solution for systems that have no exact solutions, minimizing the overall error.

3. Regression analysis

It is a standard method in regression analysis to find the best fitting model by minimizing the sum of the squares of the differences between observed and predicted values.

3. Steps to compute least squares solutions

Let A be an m×n matrix and let b be a vector in Rⁿ. The method for computing a least squares solution of Ax=b is

1. Compute the matrix A^TA and the vector A^Tb.

2. Form the augmented matrix for the matrix equation A^TAx = A^Tb and row reduce.

3. This equation is always, consistent, and any solution x is the least squares solution.

Note:

Let A be an m×n matrix and let "b" be a vector in R^m. Then the following statements are equivalent.

1. Ax=b has a unique least−squares solution.

2. The columns of A are linearly independent.

3. A^TA is invertible.

In this case, the least squares solution is

 x = (A^TA)⁻¹A^Tb.