20 Questions to Test Your Skills On Dimensionality Reduction (PCA)

PCA stands for Principal Component analysis. It is a dimensionality reduction technique that summarizes a large set of correlated variables (basically high dimensional data) into a smaller number of representative variables, called the Principal Components, that explains most of the variability of the original set i.e, not losing that much of the information.

PCA is a deterministic algorithm in which we have not any parameters to initialize and it doesn’t have a problem of local minima, like most of the machine learning algorithms has.

Image Source: Google Images

The major steps which are to be followed while using the PCA algorithm are as follows:

Step-1: Get the dataset.

Step-2: Compute the mean vector (µ).

Step-3: Subtract the means from the given data.

Step-4: Compute the covariance matrix.

Step-5: Determine the eigenvectors and eigenvalues of the covariance matrix.

Step-6: Choosing Principal Components and forming a feature vector.

Step-7: Deriving the new data set by taking the projection on the weight vector.

Usually, the aim of standardization is to assign equal weights to all the variables. PCA finds new axes based on the covariance matrix of original variables. As the covariance matrix is sensitive to the standardization of variables therefore if we use features of different scales, we often get misleading directions.

Moreover, if all the variables are on the same scale, then there is no need to standardize the variables.

Yes, the idea behind rotation i.e, orthogonal Components is so that we are able to capture the maximum variance of the training set.

If we don’t rotate the components, the effect of PCA will diminish and we’ll have to select more Principal Components to explain the maximum variance of the training dataset.

The assumptions needed for PCA are as follows:

1. PCA is based on Pearson correlation coefficients. As a result, there needs to be a linear relationship between the variables for applying the PCA algorithm.

2. For getting reliable results by using the PCA algorithm, we require a large enough sample size i.e, we should have sampling adequacy.

3. Your data should be suitable for data reduction i.e., we need to have adequate correlations between the variables to be reduced to a smaller number of components.

4. No significant noisy data or outliers are present in the dataset.

While applying the PCA algorithm, If we get all eigenvectors the same, then the algorithm won’t be able to select the Principal Components because in such cases, all the Principal Components are equal.

The properties of principal components in PCA are as follows:

1. These Principal Components are linear combinations of original variables that result in an axis or a set of axes that explain/s most of the variability in the dataset.

2. All Principal Components are orthogonal to each other.

3. The first Principal Component accounts for most of the possible variability of the original data i.e, maximum possible variance.

4. The number of Principal Components for n-dimensional data should be at utmost equal to n(=dimension). For Example, There can be only two Principal Components for a two-dimensional data set.

The Principal Component represents a line or an axis along which the data varies the most and it also is the line that is closest to all of the n observations in the dataset.

In mathematical terms, we can say that the first Principal Component is the eigenvector of the covariance matrix corresponding to the maximum eigenvalue.

Accordingly,

• Sum of squared distances = Eigenvalue for PC-1
• Sqrt of Eigenvalue = Singular value for PC-1

If we project all the points on the Principal Component, they tell us that the independent variable 2 is N times as important as of independent variable 1.

Yes, we can use Principal Components for regression problem statements.

PCA would perform well in cases when the first few Principal Components are sufficient to capture most of the variation in the independent variables as well as the relationship with the dependent variable.

The only problem with this approach is that the new reduced set of features would be modeled by ignoring the dependent variable Y when applying a PCA and while these features may do a good overall job of explaining the variation in X, the model will perform poorly if these variables don’t explain the variation in Y.

Feature selection refers to choosing a subset of the features from the complete set of features.

No, PCA is not used as a feature selection technique because we know that any Principal Component axis is a linear combination of all the original set of feature variables which defines a new set of axes that explain most of the variations in the data.

Therefore while it performs well in many practical settings, it does not result in the development of a model that relies upon a small set of the original features.

PCA does not take the nature of the data i.e, linear or non-linear into considerations during its algorithm run but PCA focuses on reducing the dimensionality of most datasets significantly. PCA can at least get rid of useless dimensions.

However, reducing dimensionality with PCA will lose too much information if there are no useless dimensions.

A dimensionality reduction algorithm is said to work well if it eliminates a significant number of dimensions from the dataset without losing too much information. Moreover, the use of dimensionality reduction in preprocessing before training the model allows measuring the performance of the second algorithm.

We can therefore infer if an algorithm performed well if the dimensionality reduction does not lose too much information after applying the algorithm.

Comprehension Type Question: (16 – 18)

Consider a set of 2D points {(-3,-3), (-1,-1),(1,1),(3,3)}. We want to reduce the dimensionality of these points by 1 using PCA algorithms. Assume sqrt(2)=1.414.

Now, Answer the Following Questions:

Here the original data resides in R² i.e, two-dimensional space, and our objective is to reduce the dimensionality of the data to 1 i.e, 1-dimensional data ⇒ K=1

We try to solve these set of problem step by step so that you have a clear understanding of the steps involved in the PCA algorithm:

Step-1: Get the Dataset

Here data matrix X is given by [ [ -3, -1, 1 ,3 ], [ -3, -1, 1, 3 ] ]

Step-2:  Compute the mean vector (µ)

Mean Vector: [ {-3+(-1)+1+3}/4, {-3+(-1)+1+3}/4 ] = [ 0, 0 ]

Step-3: Subtract the means from the given data

Since here the mean vector is 0, 0 so while subtracting all the points from the mean we get the same data points.

Step-4: Compute the covariance matrix

Therefore, the covariance matrix becomes XX^T since the mean is at the origin.

Therefore, XX^T becomes [ [ -3, -1, 1 ,3 ], [ -3, -1, 1, 3 ] ] ( [ [ -3, -1, 1 ,3 ], [ -3, -1, 1, 3 ] ] )^T

= [ [ 20, 20 ], [ 20, 20 ] ]

Step-5: Determine the eigenvectors and eigenvalues of the covariance matrix

det(C-λI)=0 gives the eigenvalues as 0 and 40.

Now, choose the maximum eigenvalue from the calculated and find the eigenvector corresponding to λ = 40 by using the equations CX = λX :

Accordingly, we get the eigenvector as (1/√ 2 ) [ 1, 1 ]

Therefore, the eigenvalues of matrix XX^T are 0 and 40.

Step-6: Choosing Principal Components and forming a weight vector

Here, U = R^2×1 and equal to the eigenvector of XX^T corresponding to the largest eigenvalue.

Now, the eigenvalue decomposition of C=XX^T

And W (weight matrix) is the transpose of the U matrix and given as a row vector.

Therefore, the weight matrix is given by  [1 1]/1.414

Step-7: Deriving the new data set by taking the projection on the weight vector

Now, reduced dimensionality data is obtained as x_i = U^T X_i = WX_i

x₁ = WX₁= (1/√ 2 ) [ 1, 1 ] [ -3, -3 ]^T = – 3√ 2

x₂ = WX₂= (1/√ 2)  [ 1, 1 ] [ -1, -1 ]^T = – √ 2

x₃ = WX₃= (1/√ 2)  [ 1, 1 ] [ 1, 1]^T = – √ 2

x₄ = WX₄= (1/√ 2 ) [ 1, 1 ] [ 3, 3 ]^T = – 3√ 2

Therefore, the reduced dimensionality will be equal to {-3*1.414, -1.414,1.414, 3*1.414}.

This completes our example!

 

Some of the advantages of Dimensionality reduction are as follows:

1. Less misleading data means model accuracy improves.

2. Fewer dimensions mean less computing. Less data means that algorithms train faster.

3. Less data means less storage space required.

4. Removes redundant features and noise.

5. Dimensionality Reduction helps us to visualize the data that is present in higher dimensions in 2D or 3D.

Some of the disadvantages of Dimensionality reduction are as follows:

1. While doing dimensionality reduction, we lost some of the information, which can possibly affect the performance of subsequent training algorithms.

2. It can be computationally intensive.

3. Transformed features are often hard to interpret.

4. It makes the independent variables less interpretable.