Principal Component Analysis

Description

Principal Component Analysis transforms high-dimensional data into a lower-dimensional representation by finding the directions (principal components) that capture the maximum variance in the data. Each component is a linear combination of original features, with the first component explaining the most variance, the second component the most remaining variance orthogonal to the first, and so on. This technique reveals underlying patterns in data structure, enables visualization of complex datasets, and helps identify which combinations of features drive the most variation in the data.

Example Use Cases

Explainability

Analysing customer behavior data with dozens of variables (purchase frequency, spending patterns, demographics) to identify the 2-3 main dimensions that explain customer segmentation, revealing whether customers cluster by spending level, product preferences, or shopping frequency.

Reducing dimensionality of image data for facial recognition systems by finding the principal components that capture the most variation in face shapes and expressions, helping understand which facial features contribute most to distinguishing between individuals.

Limitations

Principal components are abstract linear combinations of original features that often lack clear real-world interpretation or meaning.
Only captures linear relationships between features, missing non-linear patterns and complex interactions in the data.
Results are highly sensitive to feature scaling - features with larger numerical ranges can dominate the principal components.
Information loss is inherent when reducing dimensions, and choosing the optimal number of components requires balancing simplicity with retained variance.

Resources

erdogant/pca

Software Package

How to Calculate Principal Component Analysis (PCA) from Scratch ...

Tutorial

A One-Stop Shop for Principal Component Analysis | Towards Data ...

Tutorial

Principal Component Analysis (PCA) with Scikit-Learn - KDnuggets

Tutorial

willtownes/glmpca-py

Software Package

Related Techniques

Name	Description	Assurance Goals
ANCHOR	ANCHOR generates high-precision if-then rules that explain individual predictions by identifying the minimal set of feature conditions that guarantee a specific prediction with high confidence. It searches for 'anchor' conditions (e.g., 'age > 30 AND income < £50k') that ensure the model gives the same prediction at least 95% of the time when those conditions are met. This creates human-readable rules that users can trust as sufficient conditions for understanding why a particular decision was made.	Explainability Transparency
Contextual Decomposition	Contextual Decomposition explains LSTM and RNN predictions by decomposing the final hidden state into contributions from individual inputs and their interactions. Unlike simpler attribution methods, it separates the direct contribution of specific words or phrases from the contextual effects of surrounding words. This is particularly useful for understanding how sequential models process language, as it can identify whether a word's influence comes from its individual meaning or from its interaction with nearby words in the sequence.	Explainability Transparency
Permutation Tests	Permutation tests assess the statistical significance of observed results (such as model accuracy, feature importance, or group differences) by comparing them to what would occur purely by chance. The technique randomly shuffles labels or data thousands of times, recalculating the metric of interest each time to build an empirical null distribution. If the actual observed result falls in the extreme tail of this distribution (typically beyond the 95th or 99th percentile), it provides strong evidence that the relationship is genuine rather than due to random chance, without requiring parametric assumptions about data distributions.	Explainability Reliability
Contrastive Explanation Method	The Contrastive Explanation Method (CEM) explains model decisions by generating contrastive examples that reveal what makes a prediction distinctive. It identifies 'pertinent negatives' (minimal features that could be removed to change the prediction) and 'pertinent positives' (minimal features that must be present to maintain the prediction). This approach helps users understand not just what led to a decision, but what would need to change to achieve a different outcome, providing actionable insights for decision-making.	Explainability Transparency
Factor Analysis	Factor analysis is a statistical technique that identifies latent variables (hidden factors) underlying observed correlations in data. It works by analysing how variables relate to each other, finding a smaller number of unobserved factors that explain patterns among multiple observed variables. Unlike PCA which maximises total variance, factor analysis focuses on shared variance (communalities - the variance variables have in common) whilst separating out unique variance and measurement error. After extracting factors, rotation methods like varimax (which creates uncorrelated factors) or oblimin (allowing correlated factors) help make factors more interpretable by aligning them with distinct groups of variables.	Explainability Transparency
Taylor Decomposition	Taylor Decomposition is a mathematical technique that explains neural network predictions by computing first-order and higher-order derivatives of the network's output with respect to input features. It decomposes the prediction into relevance scores that indicate how much each input feature and feature interaction contributes to the final decision. The method uses Layer-wise Relevance Propagation (LRP) principles to trace prediction contributions backwards through the network layers, providing precise mathematical attributions for each input element.	Explainability

Principal Component Analysis

Description

Example Use Cases

Explainability

Limitations

Resources

erdogant/pca

How to Calculate Principal Component Analysis (PCA) from Scratch ...

A One-Stop Shop for Principal Component Analysis | Towards Data ...

Principal Component Analysis (PCA) with Scikit-Learn - KDnuggets

willtownes/glmpca-py

Related Techniques

Tags