Harvard University,FAS
Fall 2003

Mathematics Math21b
Fall 2003

Linear Algebra and Differential Equations

Course Head: Oliver knill
Office: SciCtr 434
Email: knill@math.harvard.edu

Data Fitting

(least square solutions)

Least Square Solution

You have seen the method to fit data (x1,y1),..., (xm,ym) by functions of the form f=a1 f1(x)+ ... + an fn(x). The idea was to write a system A a = b of linear equations which tell that all the data can be fitted by the functions f(xi)=yi, then find the least square solution a* = (AT A)-1 AT b to this system.

(The least square solution formula was obtained from AT (b-A a) = 0 which paraphrases that the "error" (b-A a) is perpendicular to the image of A. Geometrically this means that A a is the projection of b onto the image of A. )


Regression

Examples. Linear fitting with a linear function f(x) = a1 x + a2, where n=2: 
 
a1 x1 + a2 = y2
a1 x1 + a2 = y2
    .....      
a1 xm + a2 = ym


where 



      | x1   1  |
      | x2   2  |
A  =  |  ...    | 
      |  ...    |   
      | xm   m  |




and  the formula  a = (AT A)-1 AT b can be written
as a1 = Cov[X,Y]/Var[X], a2 = E[Y]-a1 E[X],
where E[X]=x1 + ...+xn is the expectation of the x data,
E[Y]=y1 + ...+yn is the expectation of the y data, 
Var[X]=E[X2]-E[X] is the variance and Cov[X,Y] = E[X Y]-E[X] E[Y]
is the covariance of X and Y. The solution line 
y = a1 x + a2 is called the regression line. 



(The same formula can also be obtained using Lagrange extrema by minimizing
Var[a1 X + a2 - Y] under the constraint
E[a1 X + a2 -Y]=0.)






 
 Higher dimensions 
 
 


 



We can also fit data (x1,y1,z1),
..., (xm,ym,zm)in three dimensions by functions 
z=f(x,y) = a1 f1(x,y) + ... + an fn(x,y)
or implicitly by surfaces g(x,y,z) = 0 like for example 
g(x,y,z) = a1x2 + a2 y2 + a3 z2 + a4,
in which case one would want to find the best ellipsoid centered at the origin which fits the three
dimensional data. 










 Back to the main page