In machine learning and data analytics data visualization is one of the most important steps. Cleaning and arranging data is done by different algorithms. Sometimes in data sets, we get NaN (not a number) values which are not possible to use for data visualization. To solve this problem, one possible method is to replace nan values with an average of columns. Given below are a few methods to solve this problem.
Method #1: Using np.colmean and np.take
Python3
# Python code to demonstrate# to replace nan values# with an average of columnsimport numpy as np# Initialising numpy arrayini_array = np.array([[1.3, 2.5, 3.6, np.nan], [2.6, 3.3, np.nan, 5.5], [2.1, 3.2, 5.4, 6.5]])# printing initial arrayprint ("initial array", ini_array)# column meancol_mean = np.nanmean(ini_array, axis = 0)# printing column meanprint ("columns mean", str(col_mean))# find indices where nan value is presentinds = np.where(np.isnan(ini_array))# replace inds with avg of columnini_array[inds] = np.take(col_mean, inds[1])# printing final arrayprint ("final array", ini_array) |
Output:
initial array [[ 1.3 2.5 3.6 nan] [ 2.6 3.3 nan 5.5] [ 2.1 3.2 5.4 6.5]] columns mean [ 2. 3. 4.5 6. ] final array [[ 1.3 2.5 3.6 6. ] [ 2.6 3.3 4.5 5.5] [ 2.1 3.2 5.4 6.5]]
Method #2: Using np.ma and np.where
Python3
# Python code to demonstrate# to replace nan values# with average of columnsimport numpy as np# Initialising numpy arrayini_array = np.array([[1.3, 2.5, 3.6, np.nan], [2.6, 3.3, np.nan, 5.5], [2.1, 3.2, 5.4, 6.5]])# printing initial arrayprint ("initial array", ini_array)# replace nan with col meansres = np.where(np.isnan(ini_array), np.ma.array(ini_array, mask = np.isnan(ini_array)).mean(axis = 0), ini_array) # printing final arrayprint ("final array", res) |
Output:
initial array [[ 1.3 2.5 3.6 nan] [ 2.6 3.3 nan 5.5] [ 2.1 3.2 5.4 6.5]] final array [[ 1.3 2.5 3.6 6. ] [ 2.6 3.3 4.5 5.5] [ 2.1 3.2 5.4 6.5]]
Method #3: Using Naive and zip
Python3
# Python code to demonstrate# to replace nan values# with average of columnsimport numpy as np# Initialising numpy arrayini_array = np.array([[1.3, 2.5, 3.6, np.nan], [2.6, 3.3, np.nan, 5.5], [2.1, 3.2, 5.4, 6.5]])# printing initial arrayprint ("initial array", ini_array)# indices where values is nan in arrayindices = np.where(np.isnan(ini_array))# Iterating over numpy array to replace nan with valuesfor row, col in zip(*indices): ini_array[row, col] = np.mean(ini_array[ ~np.isnan(ini_array[:, col]), col])# printing final arrayprint ("final array", ini_array) |
Output:
initial array [[ 1.3 2.5 3.6 nan] [ 2.6 3.3 nan 5.5] [ 2.1 3.2 5.4 6.5]] final array [[ 1.3 2.5 3.6 6. ] [ 2.6 3.3 4.5 5.5] [ 2.1 3.2 5.4 6.5]]
Approach#4: Using list comprehension and built-in functions
This approach uses list comprehension and built-in functions to compute the column means and replace the NaN values in the array with the corresponding column means. It first computes the column means using a list comprehension with the help of the filter and zip functions. Then, it replaces the NaN values in the array with the corresponding column means using another list comprehension with the help of the enumerate function. Finally, it returns the modified list.
Algorithm
1. Compute the column means.
2. Replace the NaN values in the array with the corresponding column means using list comprehension and built-in functions.
3. Return the modified list.
Python3
def replace_nan_with_mean(arr): col_means = [sum(filter(lambda x: x is not None, col))/len(list(filter(lambda x: x is not None, col))) for col in zip(*arr)] for i in range(len(arr)): arr[i] = [col_means[j] if x is None else x for j, x in enumerate(arr[i])] return arrarr=[[1.3, 2.5, 3.6, None], [2.6, 3.3, None, 5.5], [2.1, 3.2, 5.4, 6.5]]print(replace_nan_with_mean(arr)) |
[[1.3, 2.5, 3.6, 6.0], [2.6, 3.3, 4.5, 5.5], [2.1, 3.2, 5.4, 6.5]]
Time complexity: O(n*m), where n is the number of rows and m is the number of columns.
Auxiliary Space: O(n*m).
Approach#4: Using zip()+lambda()
Compute the column means excluding None values using a loop over the transposed array zip(*arr). Replace None values with column means using map() and lambda functions.
Algorithm
1. Initialize an empty list means to store the column means.
2. Loop over the transposed array zip(*arr) to iterate over columns.
3. For each column, filter out None values and compute the mean of the remaining values. If there are no remaining values, set the mean to 0.
4. Append the mean to the means list.
5. Use map() and lambda functions to replace None values with the corresponding column mean in each row of the array arr.
6. Return the modified array arr.
Python3
# initial arrayarr = [[1.3, 2.5, 3.6, None], [2.6, 3.3, None, 5.5], [2.1, 3.2, 5.4, 6.5]]# compute column meansmeans = []for col in zip(*arr): values = [x for x in col if x is not None] means.append(sum(values)/len(values) if values else 0)# replace NaN values with column meansarr = list(map(lambda row: [means[j] if x is None else x for j,x in enumerate(row)], arr))# print final arrayprint(arr) |
[[1.3, 2.5, 3.6, 6.0], [2.6, 3.3, 4.5, 5.5], [2.1, 3.2, 5.4, 6.5]]
Time Complexity: O(n^2), where n is the number of values in the array.
Auxiliary Space: O(n), where n is the number of values in the array.
