What exactly is the physical interpretation of variance (in terms of a data set) and the difference between variance and co-variance matrices?
$\endgroup$1 Answer
$\begingroup$The physical interpretation of variance is "how spread the data is". The higher the variance - the data is more spread. There is no such things as a variance matrix.
The Covariance is a measure of correlation between two variables. for $X,Y$ random variables (scalars) - $cov(X,Y) = E[(X-E[X])(Y-E[Y])]$, and is a scalar too. note that $var(X)=cov(X,X)=E[(X-E[X])^2]$.
lastly - a Covariance matrix is used for random vectors. for a single random vector $\bar{X}=[X_1,...,X_n]^T$, a covariance matrix (which can be seen as a variance matrix, but this is not the correct term) is defined as $cov(X)=E[(X-E[X])(X-E[X])^T]$. Note that the diagonal elements in this matrix are the variances of the random variables $X_1,...,X_n$.
For two random vectors $X,Y$, we define the Covariance between them as $cov(X,Y)=E[(X-E[X])(Y-E[Y])^T]$.
$\endgroup$ 2
