Principal Component Analysis

Description

Principal component analysis (PCA) summarises a dataset that has many numeric features by finding a few new axes along which the data varies most. Each axis, called a principal component, is a weighted mix of the original features. The first axis follows the direction of greatest spread in the data, the second follows the greatest remaining spread at right angles to the first, and so on. Plotting the data on the first two or three axes gives a view of its structure, and the weights show which features move together.

Example Use Cases

Explainability

A retail analyst runs PCA on three dozen customer behaviour measures and finds that two components account for most of the variation. The weights show that one axis mixes spend and basket size and the other mixes visit frequency and recency, which gives the team a short list of dimensions to examine before building a segmentation model.

A team building a face verification system runs PCA on its training images and displays the leading components as images. If those components mainly capture lighting or background rather than face shape, the team knows the dataset's main variation reflects how the photos were taken, and can change what it collects.

A researcher auditing a word embedding takes the difference vectors for ten gendered word pairs, such as he and she, and runs PCA. The first component explains far more variance than the rest and far more than for random vectors, which suggests the embedding encodes gender mostly along one direction (Bolukbasi et al., 2016).

Limitations

Each component mixes many original features, so it often has no clear real-world meaning, and the weights are hard to read when there are many features.
PCA captures only linear structure, so it can miss curved or clustered patterns in the data and interactions between features.
Results depend on feature scaling. PCA centres the data but does not rescale it, so a feature measured in large units can dominate the leading components. Standardise features first, or the components may reflect units and not structure.
PCA ranks directions by how much the data varies, not by how much they matter for an outcome. A high-variance component may be noise or irrelevant to a decision, and a low-variance one may carry the signal that separates classes, so leading components are not a guide to what a model needs.
Dropping components discards the variance they carry. Deciding how many to keep is a judgement call, for example a target share of variance explained, and it trades a simpler summary against lost detail.

Resources

Research Papers

On lines and planes of closest fit to systems of points in space

Karl Pearson•Nov 1, 1901

Analysis of a complex of statistical variables into principal components.

H. Hotelling•Sep 1, 1933

Software Packages

pca

Jan 11, 2020

pca: A Python Package for Principal Component Analysis.

scikit-learn

scikit-learn: machine learning in Python. Contribute to scikit-learn/scikit-learn development by creating an account on GitHub.

Tutorials

Principal Component Analysis (PCA) on Iris Dataset — scikit-learn

This example shows a well known decomposition technique known as Principal Component Analysis (PCA) on the Iris dataset. This dataset is made of 4 features: sepal length, sepal width, petal length,...

Documentations

Principal component analysis (PCA) — scikit-learn user guide (section 2.5.1)

Principal component analysis (PCA): Exact PCA and probabilistic interpretation: PCA is used to decompose a multivariate dataset in a set of successive orthogonal components that explain a maximum a...

Related Techniques

Name	Description	Assurance Goals
Factor Analysis	Factor analysis looks for a small number of hidden factors that explain why a larger set of measured variables, such as a model's input features, tend to move together. Each variable is modelled as a weighted mix of the shared factors plus a part that is unique to it, including measurement error. The weights are called loadings, and they show which variables each factor is built from; for example, several credit-history features might all load on one 'repayment behaviour' factor. Unlike principal component analysis, which summarises as much of the total variation as it can, factor analysis models only the variation that variables share. Analysts usually rotate the factors afterwards, using varimax (factors stay uncorrelated) or oblimin (factors may correlate), so that each variable loads mainly on one factor and the factors are easier to name.	Explainability
t-SNE	t-SNE (t-distributed Stochastic Neighbour Embedding) turns high-dimensional data into a two- or three-dimensional map that can be plotted and inspected. It is a non-linear dimensionality reduction method, so it can unfold curved structure that linear methods such as principal component analysis, which find only straight-line directions of greatest variation, would flatten. For each pair of points it works out how likely they are to be neighbours in the original space. It then moves the points around on the map until the same neighbour probabilities hold there as closely as possible. Because it concentrates on keeping near neighbours together, clusters of similar items usually appear as visible groups, whether the points are raw data or a model's internal representations such as embeddings.	Explainability
UMAP	UMAP (Uniform Manifold Approximation and Projection) is a non-linear dimensionality reduction method that turns high-dimensional data into a two- or three-dimensional map for plotting. It assumes the data lie on a lower-dimensional curved surface, called a manifold, and links each point to its nearest neighbours, giving each link a strength that says how likely the two points are to be neighbours. It then arranges the points on the map so that the link strengths there match those in the original space as closely as possible. Like t-SNE, it concentrates on keeping near neighbours together, so groups of similar items appear as clusters, whether the points are raw data or a model's embeddings. Its authors argue that it keeps more of the large-scale layout than t-SNE and runs faster, but gaps between clusters and cluster sizes should not be read as exact.	Explainability