Question

[Machine learning] Principal Component Analysis (PCA) and Linear Discriminant Analysis (LDA)

In this problem two linear dimenionality reduction methods will be discussed. They are principal component analysis (PCA) and linear discriminant analysis (LDA).

In a supervised binary classification task we have a total of 15 features, among which only 4 are useful for predicting the target variable, the other features are pure random noise with very high variance. What complicates matters even worse is that the 4 features when considered individually show no predictive power, and only work when considered together.

Consider each of the following dimension reduction (or classification techniques) and indicate whether it may be able to successfully identify the relevant dimensions (Yes or No). Briefly explain why.

(I) Principle Component Analysis
(II) Linear Discriminant Analysis
(III) AdaBoost with decision stumps as the weak learner
(IV) AdaBoost with a linear support vector machine as the weak learner

Answer 1

i) Yes, PCA creates a new set of features that is a linear combination of the input features and thus will be able to identify those 4 features linear combination.

ii) No because LDA tries to minimize the within-class covariance and maximize in between class covariance but since the other noisy features have high covariance it will have a great effect on LDA and will lead to incorrect results.

iii) No because a decision stump makes a prediction based on the value of just a single input feature

iv) No, SVM does not perform well when the data set has more noise.