20 Questions to Test Your Skills on Feature Engineering and EDA

Different types of data require different types of cleaning, but the most important steps of data Cleaning are:

• Data Quality
• Removing Duplicate Data (also irrelevant data)
• Structural errors
• Outliers or noisy data
• Treatment for Missing Data

Data Cleaning is an important step that involves about 80% of the time before analyzing data, it helps to increase the accuracy of the model. This helps organizations to make informed decisions.

In the data cleaning process, we can see there are lots of missing values that are not filled or collected during the survey for data collection. So, we have to handle such missing values using the following methods:

1. Delete the missing records: Ignoring such rows or dropping such records.

2. Central Tendency: Fill values with mean, mode, and median.

3. Different mean for different classes: Fill values using mean but for different classes, different means can be used.

4. Model Building: You can also fill the most probable value using regression, Bayesian formula, or decision tree, KNN, and Prebuilt imputing libraries.

5. Use constant Value: Fill with a constant value.

6. To fill the values manually.

                                                                   Image Source: Google Images

The two useful methods of Pandas Library are –

• isnull( )
• dropna( )

These help to find the columns of data with missing or corrupted data and drop those values. In cases where we want to fill the invalid values with a placeholder value such as 0, we can use the fillna() method.

Both classification and regression belong to the category of supervised machine learning algorithms. However, when the target is categorical, classification is used whereas when your target variable is continuous, regression is used.

The classification produces discrete values and datasets to strict the categories, while regression gives continuous results that allow to better distinguish differences between individual points.

In order to represent the belonging of the data points to certain categories, we use classification over regression.

For example, If you wanted to know whether a name was male or female rather than just how correlated they were with male and female names.

The Curse of Dimensionality refers to the situation when your data has too many features, but for better performance of our model, we need only some optimal no of features.

                                                                           Image Source: Google Images

The phrase is used to describe the difficulty of using the brute force approach or grid search to optimize a function having too many inputs. Due to the increase in dimensionality, the following issues have come up:

• Overfitting: If we have more features than observations, we have a risk of overfitting the model.
• Sparsity: When we have too many features, observations become harder to cluster. Due to too many dimensions in our data, every data point in the dataset appears equidistant from all others and as a result, no meaningful clusters can be formed.

As a result, dimensionality reduction techniques like PCA have come into the picture to solve the problem of increased dimensionality.

A feature splitting is an approach to generate some new features from the existing one to improve the performance of our model.

For Example, splitting names of objects into two parts- first and last names.

For Neural networks: Batch size with a Numpy array will work.

Steps:

• Load the whole data into a Numpy array since the Numpy array creates a mapping of the complete data set. So, there is no need to load the dataset completely in the memory.
• To get the required data, you can pass an index to a Numpy array.
• Use this data and pass it to the Neural network as an input.
• Have a small batch size.

For Support Vector Machines (SVMs): Partial fit will work

Steps:

• Divide one big data set into small-size data sets.
• By using a partial fit method of SVM, and then gives the subset of the complete data set as input.
• Repeat step 2 for all the remaining subsets.

To use a dataset without having an output column, we first give the input dataset into a clustering algorithm, which generates an optimal number of clusters, and then labels the cluster numbers as the new target variable.

Now, the dataset has both independent and dependent variables i.e, target column present.

So, this completes our objective to apply the supervised learning algorithms to the unlabeled data.

Yes, we can test for the probability of improving the accuracy of the model without using cross-validation techniques.

For doing this, We have to run our ML model for say K number of iterations, and then recording the accuracy. After that try to plot all the accuracies and remove the 5% of low probability values. Then, measure the left [low]and right [high] threshold, decided by the problem statement itself.

Now, with the remaining 95% confidence, we can say that the model can go as low or as high i.e, determine the range.

Both One-hot encoding and Label Encoding are used to encode the categorical variables to numerical variables so that we can feed that encoded data to our ML algorithms and be able to find insights about the data.

One-hot encoding is the representation of categorical variables as binary vectors while Label Encoding converts labels or words into numeric form.

One-hot encoding increases the dimensionality of the data set while Label encoding doesn’t affect the dimensionality of the data set since One-hot encoding creates a new variable for each level in the variable whereas, in Label encoding, the levels of a variable get encoded as 1 and 0 in replacement of the same column.

              Fig. One Hot Encoding

     Fig. Label Encoding

                                                                   Image Source: Google Images

 

There are various methods to select important features from a data set that include the following:

1. Multicollinearity: Identify and discard correlated variables before finalizing important variables.

2. Using the Linear Regression model: More explanatory variables could be selected based on the p-values obtained from the Linear Regression.

3. Wrapper Methods: Forward, Backward, and Stepwise selection

4. Regularization: Lasso Regression

5. Ensemble Technique: Apply Random Forest and then plot the variable chart

6. Information Gain: Top features can be selected based on information gain for the available set of features.

 

There is no fixed rule of thumb to choose an algorithm for a classification problem, but the following points can be kept in mind while selecting an algorithm:

• If accuracy is our major focus: Test different algorithms and cross-validate them.
• WhatIf we have a small training dataset: Use models that have low variance and high bias.
• If we have a large training dataset: Use models that have high variance and little bias.

The popular methods which are used to reduce the dimensionality of a dataset are as follows:

1. Use Feature Engineering and try to Combine the features and make a new set of features.

2. Use some form of algorithmic dimensionality reduction like Independent Component Analysis(ICA), Principal component analysis(PCA), or Linear Discriminant Analysis(LDA), etc.

Ideally, we need to do a scaling post-train and test split. Scaling post or pre-split should not make much difference when our data is closely packed.

A categorical independent variable can be treated as a continuous one when the nature of data points the categorical data represents is ordinal in nature.

So, if the independent variable is having ordinal data then it can be treated as continuous and its inclusion in the model increases the performance of the model.

From a date column, we can get lots of important features such as day of the week, day of the month, day of the quarter, and day of the year.

Moreover, we can extract the date, month, and year from that column also. All these features can impact your prediction and make our model robust.

For Example, Sales of the business can be impacted by month or day of the week.

                                                                     Image Source: Google Images

An imbalanced dataset is when you have, For Example, a classification problem statement and let’s 90% of the data is in one class. As a result, an accuracy of 90% can be skewed if you have no predictive power on the other category of data!

Here are a few suggestions to get rid of these problems:

• To collect more dataset that helps to even the imbalances in the dataset.
• Resample the dataset to correct for imbalances.
• Try a different algorithm altogether on your dataset.

So, it’s important that you have a keen sense of what damage an unbalanced dataset can cause, and how to balance that.

In cases where the number of variables is greater than the number of observations, it represents a high-dimensional dataset.

So, it is not possible to calculate a unique least-square coefficient estimate, and therefore we used the penalized regression methods like Least Angle regression(LARS), LASSO, or Ridge which seems to work well under these circumstances as they tend to shrink the coefficients in order to reduce the variance.

Moreover, in situations of higher variance of least square estimates Ridge Regression works better.