Factor Analysis

Description

Factor analysis looks for a small number of hidden factors that explain why a larger set of measured variables, such as a model's input features, tend to move together. Each variable is modelled as a weighted mix of the shared factors plus a part that is unique to it, including measurement error. The weights are called loadings, and they show which variables each factor is built from; for example, several credit-history features might all load on one 'repayment behaviour' factor. Unlike principal component analysis, which summarises as much of the total variation as it can, factor analysis models only the variation that variables share. Analysts usually rotate the factors afterwards, using varimax (factors stay uncorrelated) or oblimin (factors may correlate), so that each variable loads mainly on one factor and the factors are easier to name.

Example Use Cases

Explainability

A bank's model validation team finds that forty macroeconomic and credit-bureau inputs to a default-risk model are heavily correlated. Exploratory factor analysis groups them into four factors the team can name, such as growth, inflation, household debt and credit conditions, and the model is rebuilt on the factor scores so its drivers can be explained to the risk committee in four terms instead of forty.

A flood-forecasting team with sixty correlated gauge, soil-moisture and satellite inputs runs factor analysis on the training data to find a few underlying drivers, such as rainfall intensity, catchment wetness and river level. Forecast explanations are then reported per driver, which operators can relate to what they see on the ground.

Limitations

The standard model is linear, with each variable a weighted sum of factors plus noise. Maximum-likelihood extraction also assumes roughly normal data, so skewed, binary or strongly non-linear variables can distort the loadings. Check distributions first, or choose an extraction method that does not require normality.
Factors are statistical summaries of correlation, not discovered causes. A factor labelled 'economic confidence' is the analyst's reading of which variables load together, and a different rotation or sample may support a different label.
Sensitive to the choice of number of factors and rotation method, which have no single correct answer and can change both which variables group together and how readable the factors are. Report the choices made and check that the structure is stable.
Requires a sample that is large relative to the number of variables. With small samples the loadings are unstable and change when a few cases are added or removed.
Describes the structure of the data fed into it, not how a trained model uses that data. Factors of a model's inputs say nothing about which factors the model relies on, unless the model is also built on or analysed with the factor scores.

Resources

Research Papers

"General Intelligence," Objectively Determined and Measured

C. Spearman•Apr 1, 1904

Software Packages

factor_analyzer

Dec 5, 2017

A Python module to perform exploratory & confirmatory factor analyses.

Tutorials

Factor Analysis (with rotation) to visualize patterns — scikit-learn

Investigating the Iris dataset, we see that sepal length, petal length and petal width are highly correlated. Sepal width is less redundant. Matrix decomposition techniques can uncover these latent...

A CFA example – lavaan tutorial

Documentations

Factor Analysis — scikit-learn user guide (section 2.5.5)

Principal component analysis (PCA): Exact PCA and probabilistic interpretation: PCA is used to decompose a multivariate dataset in a set of successive orthogonal components that explain a maximum a...

FactorAnalyzer documentation

Related Techniques

Name	Description	Assurance Goals
Principal Component Analysis	Principal component analysis (PCA) summarises a dataset that has many numeric features by finding a few new axes along which the data varies most. Each axis, called a principal component, is a weighted mix of the original features. The first axis follows the direction of greatest spread in the data, the second follows the greatest remaining spread at right angles to the first, and so on. Plotting the data on the first two or three axes gives a view of its structure, and the weights show which features move together.	Explainability
t-SNE	t-SNE (t-distributed Stochastic Neighbour Embedding) turns high-dimensional data into a two- or three-dimensional map that can be plotted and inspected. It is a non-linear dimensionality reduction method, so it can unfold curved structure that linear methods such as principal component analysis, which find only straight-line directions of greatest variation, would flatten. For each pair of points it works out how likely they are to be neighbours in the original space. It then moves the points around on the map until the same neighbour probabilities hold there as closely as possible. Because it concentrates on keeping near neighbours together, clusters of similar items usually appear as visible groups, whether the points are raw data or a model's internal representations such as embeddings.	Explainability
UMAP	UMAP (Uniform Manifold Approximation and Projection) is a non-linear dimensionality reduction method that turns high-dimensional data into a two- or three-dimensional map for plotting. It assumes the data lie on a lower-dimensional curved surface, called a manifold, and links each point to its nearest neighbours, giving each link a strength that says how likely the two points are to be neighbours. It then arranges the points on the map so that the link strengths there match those in the original space as closely as possible. Like t-SNE, it concentrates on keeping near neighbours together, so groups of similar items appear as clusters, whether the points are raw data or a model's embeddings. Its authors argue that it keeps more of the large-scale layout than t-SNE and runs faster, but gaps between clusters and cluster sizes should not be read as exact.	Explainability