Dimensionality Reduction and Feature Extraction

PCA, factor analysis, feature selection, feature extraction, and more

Feature transformation techniques reduce the dimensionality in the data by transforming data into new features. Feature selection techniques are preferable when transformation of variables is not possible, e.g., when there are categorical variables in the data. For a feature selection technique that is specifically suitable for least-squares fitting, see Stepwise Regression.

Live Editor Tasks

Reduce Dimensionality

Reduce dimensionality using Principal Component Analysis (PCA) in Live Editor (Since R2022b)

Functions

expand all

Feature Selection

`fscchi2`	Univariate feature ranking for classification using chi-square tests (Since R2020a)
`fscmrmr`	Rank features for classification using minimum redundancy maximum relevance (MRMR) algorithm
`fscnca`	Feature selection using neighborhood component analysis for classification
`fsrftest`	Univariate feature ranking for regression using F-tests (Since R2020a)
`fsrmrmr`	Rank features for regression using minimum redundancy maximum relevance (MRMR) algorithm (Since R2022a)
`fsrnca`	Feature selection using neighborhood component analysis for regression
`fsulaplacian`	Rank features for unsupervised learning using Laplacian scores
`partialDependence`	Compute partial dependence (Since R2020b)
`plotPartialDependence`	Create partial dependence plot (PDP) and individual conditional expectation (ICE) plots
`oobPermutedPredictorImportance`	Out-of-bag predictor importance estimates for random forest of classification trees by permutation
`oobPermutedPredictorImportance`	Out-of-bag predictor importance estimates for random forest of regression trees by permutation
`predictorImportance`	Estimates of predictor importance for classification tree
`predictorImportance`	Estimates of predictor importance for classification ensemble of decision trees
`predictorImportance`	Estimates of predictor importance for regression tree
`predictorImportance`	Estimates of predictor importance for regression ensemble of decision trees
`relieff`	Rank importance of predictors using ReliefF or RReliefF algorithm
`sequentialfs`	Sequential feature selection using custom criterion
`stepwiselm`	Perform stepwise regression
`stepwiseglm`	Create generalized linear regression model by stepwise regression

Feature Extraction

`rica`	Feature extraction by using reconstruction ICA
`sparsefilt`	Feature extraction by using sparse filtering
`transform`	Transform predictors into extracted features

t-SNE Multidimensional Visualization

tsne t-Distributed Stochastic Neighbor Embedding

PCA and Canonical Correlation

`barttest`	Bartlett’s test
`canoncorr`	Canonical correlation
`pca`	Principal component analysis of raw data
`pcacov`	Principal component analysis on covariance matrix
`pcares`	Residuals from principal component analysis
`ppca`	Probabilistic principal component analysis

Incremental PCA

`incrementalPCA`	Incremental principal component analysis (Since R2024a)
`fit`	Fit principal component analysis model to streaming data (Since R2024a)
`transform`	Transform data into principal component scores (Since R2024a)
`reset`	Reset incremental principal component analysis model (Since R2024a)

Factor Analysis

`factoran`	Factor analysis
`rotatefactors`	Rotate factor loadings

Nonnegative Matrix Factorization

nnmf Nonnegative matrix factorization

Multidimensional Scaling

`cmdscale`	Classical multidimensional scaling
`mahal`	Mahalanobis distance to reference samples
`mdscale`	Nonclassical multidimensional scaling
`pdist`	Pairwise distance between pairs of observations
`squareform`	Format distance matrix

Procrustes Analysis

procrustes Procrustes analysis

Objects

expand all

Feature Selection

`FeatureSelectionNCAClassification`	Feature selection for classification using neighborhood component analysis (NCA)
`FeatureSelectionNCARegression`	Feature selection for regression using neighborhood component analysis (NCA)

Feature Extraction

`ReconstructionICA`	Feature extraction by reconstruction ICA
`SparseFiltering`	Feature extraction by sparse filtering

Topics

Feature Selection

Introduction to Feature Selection
Learn about feature selection algorithms and explore the functions available for feature selection.
Sequential Feature Selection
This topic introduces sequential feature selection and provides an example that selects features sequentially using a custom criterion and the sequentialfs function.
Neighborhood Component Analysis (NCA) Feature Selection

Neighborhood component analysis (NCA) is a non-parametric method for selecting features with the goal of maximizing prediction accuracy of regression and classification algorithms.
- Robust Feature Selection Using NCA for Regression
- Tune Regularization Parameter to Detect Features Using NCA for Classification
Regularize Discriminant Analysis Classifier
Make a more robust and simpler model by removing predictors without compromising the predictive power of the model.
Select Predictors for Random Forests
Select split-predictors for random forests using interaction test algorithm.

Feature Extraction

Feature Extraction
Feature extraction is a set of methods to extract high-level features from data.
Feature Extraction Workflow
This example shows a complete workflow for feature extraction from image data.
Extract Mixed Signals
This example shows how to use rica to disentangle mixed audio signals.

t-SNE Multidimensional Visualization

t-SNE
t-SNE is a method for visualizing high-dimensional data by nonlinear reduction to two or three dimensions, while preserving some features of the original data.
Visualize High-Dimensional Data Using t-SNE
This example shows how t-SNE creates a useful low-dimensional embedding of high-dimensional data.
tsne Settings
This example shows the effects of various tsne settings.
t-SNE Output Function
Output function description and example for t-SNE.

PCA and Canonical Correlation

Principal Component Analysis (PCA)
Principal Component Analysis reduces the dimensionality of data by replacing several correlated variables with a new set of variables that are linear combinations of the original variables.
Analyze Quality of Life in U.S. Cities Using PCA
Perform a weighted principal components analysis and interpret the results.

Factor Analysis

Factor Analysis
Factor analysis is a way to fit a model to multivariate data to estimate interdependence of measured variables on a smaller number of unobserved (latent) factors.
Analyze Stock Prices Using Factor Analysis
Use factor analysis to investigate whether companies within the same sector experience similar week-to-week changes in stock prices.
Perform Factor Analysis on Exam Grades
This example shows how to perform factor analysis using Statistics and Machine Learning Toolbox™.

Nonnegative Matrix Factorization

Nonnegative Matrix Factorization
Nonnegative matrix factorization (NMF) is a dimension-reduction technique based on a low-rank approximation of the feature space.
Perform Nonnegative Matrix Factorization
Perform nonnegative matrix factorization using the multiplicative and alternating least-squares algorithms.

Multidimensional Scaling

Multidimensional Scaling
Multidimensional scaling allows you to visualize how near points are to each other for many kinds of distance or dissimilarity metrics and can produce a representation of data in a small number of dimensions.
Classical Multidimensional Scaling
Use cmdscale to perform classical (metric) multidimensional scaling, also known as principal coordinates analysis.
Classical Multidimensional Scaling Applied to Nonspatial Distances
This example shows how to perform classical multidimensional scaling using the cmdscale function in Statistics and Machine Learning Toolbox™.
Nonclassical Multidimensional Scaling
This example shows how to visualize dissimilarity data using nonclassical forms of multidimensional scaling (MDS).
Nonclassical and Nonmetric Multidimensional Scaling
Perform nonclassical multidimensional scaling using mdscale.

Procrustes Analysis

Compare Handwritten Shapes Using Procrustes Analysis
Use Procrustes analysis to compare two handwritten numerals.

Featured Examples

Select Features for Classifying High-Dimensional Data

Select features for classifying high-dimensional data. More specifically, it shows how to perform sequential feature selection, which is one of the most popular feature selection algorithms. It also shows how to use holdout and cross-validation to evaluate the performance of the selected features.

Open Live Script

Partial Least Squares Regression and Principal Components Regression

Apply partial least squares regression (PLSR) and principal components regression (PCR), and explore the effectiveness of the two methods.

Open Live Script

Fitting an Orthogonal Regression Using Principal Components Analysis

Use Principal Components Analysis (PCA) to fit a linear regression. PCA minimizes the perpendicular distances from the data to the fitted model. This is the linear case of what is known variously as Orthogonal Regression, Total Least Squares, Deming Regression, or Errors-in-variables, and is appropriate when there is no natural distinction between predictor and response variables, or when all variables are measured with error. This is in contrast to the usual regression assumption that predictor variables are measured exactly, and only the response variable has an error component.

Open Live Script