Python – Remove duplicate values across Dictionary Values

Sometimes, while working with Python dictionaries, we can have a problem in which we need to remove all the duplicate values across all the dictionary value lists. This problem can have applications in data domains and web development domains. Let’s discuss certain ways in which this task can be performed.

Input: test_dict = {‘Manjeet’: [1], ‘Akash’: [1, 8, 9]} 
Output: {‘Manjeet’: [], ‘Akash’: [8, 9]} 

Input: test_dict = {‘Manjeet’: [1, 1, 1], ‘Akash’: [1, 1, 1]} 
Output: {‘Manjeet’: [], ‘Akash’: []}

Method #1: Using Counter() + list comprehension 

The combination of the above functions can be used to solve this problem. In this, we use Counter() to extract all frequencies and list comprehension to assign the value with a single occurrence in the value list. 

The original dictionary : {'Manjeet': [1, 4, 5, 6], 'Akash': [1, 8, 9], 'Nikhil': [10, 22, 4], 'Akshat': [5, 11, 22]}
Uncommon elements records : {'Manjeet': [6], 'Akash': [8, 9], 'Nikhil': [10], 'Akshat': [11]}

Time complexity: O(n*m), where n is number of keys in dictionary and m is length of the longest list in dictionary.
Auxiliary space: O(n*m), where n is number of keys in dictionary and m is length of the longest list in dictionary.

Method #2 : Using extend(),count(),keys() methods and for loops

The original dictionary : {'Manjeet': [1, 4, 5, 6], 'Akash': [1, 8, 9], 'Nikhil': [10, 22, 4], 'Akshat': [5, 11, 22]}
Uncommon elements records : {'Manjeet': [6], 'Akash': [8, 9], 'Nikhil': [10], 'Akshat': [11]}

Time Complexity: O(n*n)
Auxiliary Space: O(n)

Method #3: Using extend(),operator.countOf(),keys() methods and for loops

The original dictionary : {'Manjeet': [1, 4, 5, 6], 'Akash': [1, 8, 9], 'Nikhil': [10, 22, 4], 'Akshat': [5, 11, 22]}
Uncommon elements records : {'Manjeet': [6], 'Akash': [8, 9], 'Nikhil': [10], 'Akshat': [11]}

Time Complexity: O(n*n)
Auxiliary Space: O(n)

Method #4: Using defaultdict() and set()

Step-by-step approach:

• Import defaultdict from collections
• Initialize a defaultdict with set as its default_factory
• Loop through the values of the test_dict
  • Loop through the items of each value
    • Add each item to the corresponding set in the defaultdict
• Loop through the items of the test_dict
  • Initialize a new list for each item
  • Loop through the items of the value and check if the length of the corresponding set in the defaultdict is 1
    • If the length is 1, append the item to the new list
    • Add the new list to the result dictionary using the item as the key
• Print the result dictionary

Below is the implementation of the above approach:

The original dictionary : {'Manjeet': [1, 4, 5, 6], 'Akash': [1, 8, 9], 'Nikhil': [10, 22, 4], 'Akshat': [5, 11, 22]}
Uncommon elements records : {'Manjeet': [6], 'Akash': [8, 9], 'Nikhil': [10], 'Akshat': [11]}

Time complexity: O(n*m), where n is the number of keys in the dictionary and m is the length of the longest list in the dictionary.
Auxiliary space: O(n*m), for the defaultdict and the result dictionary.

Method #5: Using list comprehension and dictionary comprehension

Step-by-step approach:

• Create a flattened list of all values in the dictionary using list comprehension and the values() method of the dictionary.
• Create a new list unique_values containing only the values that occur exactly once in the flattened list using list comprehension and the count() method.
• Create a new dictionary res using dictionary comprehension where each key is a key from the original dictionary and each value is a list of values from the original dictionary that occur in unique_values.

Below is the implementation of the above approach:

The original dictionary : {'Manjeet': [1, 4, 5, 6], 'Akash': [1, 8, 9], 'Nikhil': [10, 22, 4], 'Akshat': [5, 11, 22]}
Uncommon elements records : {'Manjeet': [6], 'Akash': [8, 9], 'Nikhil': [10], 'Akshat': [11]}

Time complexity: O(n + m + k), where n is the total number of values in the original dictionary, m is the number of unique values in the original dictionary, and k is the number of keys in the original dictionary.
Auxiliary space: O(n + m + k), where n is the total number of values in the original dictionary, m is the number of unique values in the original dictionary, and k is the number of keys in the original dictionary. 

Method #6:  Using numpy:

Algorithm:

1. Create an empty dictionary ‘d’ with default values as a set.
2. Iterate through each value list in the input dictionary and add the id of the list to the set corresponding to each element in the list in the dictionary ‘d’.
3. Create a new empty dictionary ‘res’.
4. Iterate through each key-value pair in the input dictionary and for each list in the value, create a new list ‘new_lst’.
5. For each item in the list, check if its corresponding set in ‘d’ has length 1, indicating it is unique.
6. If it is unique, append it to the ‘new_lst’.
7. Add the ‘new_lst’ as the value for the current key in the ‘res’ dictionary.
8. Print the ‘res’ dictionary as the output.

Output:

The original dictionary : {‘Manjeet’: [1, 4, 5, 6], ‘Akash’: [1, 8, 9], ‘Nikhil’: [10, 22, 4], ‘Akshat’: [5, 11, 22]}
Uncommon elements records : {‘Manjeet’: [6], ‘Akash’: [8, 9], ‘Nikhil’: [10], ‘Akshat’: [11]}

Time complexity:
The time complexity of this code is O(n * m), where ‘n’ is the number of keys in the input dictionary and ‘m’ is the maximum number of elements in any list in the dictionary. This is because we iterate through each key-value pair in the input dictionary once and for each element in the lists, we check the corresponding set in the dictionary ‘d’, which has a maximum size of ‘m’.

Auxiliary Space:
The space complexity of this code is O(n * m), where ‘n’ is the number of keys in the input dictionary and ‘m’ is the maximum number of elements in any list in the dictionary. This is because we create a dictionary ‘d’ with ‘n’ keys and each key has a set with maximum size ‘m’. We also create a new dictionary ‘res’ with ‘n’ keys and the maximum length of any list in ‘res’ is also ‘m’.