Introduction to Factor Analysis

Factor analysis is used to describe the covariance relationships among many variables in terms of a few underlying, but unobservable random quantities called factors. If variables can be grouped (not the observations) by their correlations, then all variables within a particular group are highly correlated among themselves. This means, each group of variables represents a single underlying hidden factor that is responsible for the observed correlations.

For example, correlations from a group of test scores in Mathematics, Statistics, Chemistry and Physics might correspond to a factor named “intelligence”. A second group of variables representing physical fitness scores might correspond to another factor named “fitness”.

We can use factor analysis to identify the hidden factors behind observed variables which cannot be measured directly. The variables that cannot be measured directly are called latent variables or common factors. However, the latent variables can be examined indirectly by measuring observed variables which are called indicators or manifest variables.

After extracting factors using factor analysis, meaningful names should be given to these factors. A well-defined factor should have at least three high-loading variables.

When considering the theory behind the factor analysis, we can see a model with some underlying assumptions.
For example, suppose we have p number of variables, \(X_{1}\), \(X_{2}\), …. , \(X_{p}\), and assume that m latent (common) factors, \(F_{1}\), \(F_{2}\), …, \(F_{m}\), are hidden behind these p variables. Then, the first variable \({X}_{1}\) can be explained by these factors as follows:

\[X_{1}-\mu{_{1}}={l}_{11}F_{1}+{l}_{12}F_{2}+...+{l}_{1m}F_{m}+\epsilon{_{1}}\]
Now the second variable \({X}_{2}\) will write as

\[X_{2}-\mu{_{2}}={l}_{21}F_{1}+{l}_{22}F_{2}+...+{l}_{2m}F_{m}+\epsilon{_{2}} \] Then, the other variables also can be written in the same way. Finally the \(p^{th}\) variable will write as

\[X_{p}-\mu{_{p}}={l}_{p1}F_{1}+{l}_{p2}F_{2}+...+{l}_{pm}F_{m}+\epsilon{_{p}}\]
Here \(\mu{_{i}}\) is the mean of the variable \(X_{i}\), and \({l}_{ij}, i=1,2,...,p, j=1,2,...,m\) indicates the correlation of each variable with the underlying factor which is usually called as Factor loading. For example, \({l}_{ij}\) is the correlation between the variable\({X}_{i}\) with factor \({F}_{j}\).
The symbol \(\epsilon{_{i}}\) represents the error variance, and it is also known as the \(i^{th}\) specific factor which cannot be explained by the common factors. The above equations can be written in the matrix notation as follows:
\[\mathbf{X}-\mathbf{\mu}=\mathbf{L}\mathbf{F}+\mathbf{\epsilon}\]
where \(\mathbf{\mu}\)=\(\left( \mu_{1}, \mu _{2},...,\mu _{p}\right)^{'}\), \(\mathbf{F}=\left (F_{1},F_{2},...,F_{m} \right)\) and \(\mathbf{\epsilon}=\left (\epsilon_{1},\epsilon_{2},...,\epsilon_{m} \right)\) and

\(\mathbf{L}=\begin{pmatrix} l_{11} & l_{12} &. & l_{1m} \\ l_{21} & l_{22} &. & l_{2m}\\ .& .& .& .\\ l_{p1}&l_{p2} &. &l_{pm} \end{pmatrix}\).

Assumptions on \(\mathbf{F}\) and \(\mathbf{\epsilon}\)
(a) Common factors have zero mean and they are uncorrelated.
i.e. \(E\left(\mathbf{F}\right )=\mathbf{0}\) and \(Cov\left(\mathbf{F}\right )=E\left(FF^{'}\right )=\mathbf{I}\)
(b) \(E\left(\mathbf{\epsilon }\right )=\mathbf{0}\) and
\[Cov\left(\mathbf{\epsilon}\right )=E\left(\epsilon\epsilon^{'}\right )=\mathbf{\Psi}=\begin{pmatrix} \psi_{1} &0 &0 &0 \\ 0& \psi_{2}& 0& 0\\ .& .& .& .\\ 0 & 0 & 0 &\psi_{p} \end{pmatrix}\]
(c) Specific factors \(\mathbf{\epsilon }\) and common factors \(\mathbf{F}\) are uncorrelated. i.e. \(Cov\left(\mathbf{\epsilon, F}\right )=E\left(\epsilon F^{'}\right )=\mathbf{I}\).

Model \(\mathbf{X}-\mathbf{\mu}=\mathbf{L}\mathbf{F}+\mathbf{\epsilon}\) with the above assumptions are called Orthogonal factor model. According to the properties of this model, it can be shown that the covariance matrix \(\mathbf{\Sigma}\) of this model is \(\mathbf{\Sigma}=\mathbf{L}\mathbf{L}'+\mathbf{\psi}\), and
\[Var(X_{i})=\sigma_{ii}=l_{i1}^{2}+l_{i2}^{2}+...+l_{im}^{2}+\psi_{i}=h_{i}^{2}+\psi_{i}\] for each \(i=1,2,...,p\). Here \(h_{i}^{2}\) is called the \(i^{th}\) communality and \(\psi_{i}\) is called the uniqueness or \(i^{th}\) specific variance. According to this result we can see that a portion of the variance of the \(i^th\) variable is explained by the m common factors.
Since the latent factors, \(F_{1}\), \(F_{2}\), …, \(F_{m}\), and the variance of the specific factors \(\psi_{1}\), \(\psi_{2}\), …, \(\psi_{p}\),cannot be measured directly, we have to estimate them.
Several methods of estimation are availble in literature that are used in Factor analysis. Two such methods are principal component method and maximum likelihood method. Principal component method is a non-parametric method, and use principal component analysis to obtain factors. To apply maximum likelihood method the data should have a multivariate normal distribution; and hence it is a parametric method. First we consider the principal component method.