How to Remove Duplicates from a Python List?

A Python list can contain multiple data values, and to access an individual value, we can use the corresponding index number. In this tutorial, we will learn how to remove duplicates from a Python list using different methods. A list can have multiple elements, and those elements can be similar.

Moreover, in a Python interview , the interviewer might ask about removing duplicate elements from a list. Python list is a very flexible and widely used Python data structure .

Python does not support static arrays like other high-level programming languages. Instead, it has the list, which is an alternative for the Python array.

Prerequisites

• Python list
• The Python enumerate() function
• Python set
• Python dictionary
• List comprehension

How to Remove Duplicates from a Python List?

In this Python article, we will discuss four different techniques to remove duplicate elements from a Python list. These are:

1. The Naïve method
2. The set() method
3. List comprehension and the enumerate() function
4. The dictionary .formkeys() method

1. How to Remove Duplicate Elements from a Python List Using the Naïve Method?

In the naïve method, we will create a procedural algorithm to remove duplicate elements from a Python list. In this approach, first, we will create a new empty list and traverse through every element from our existing list. Inside the loop, we will check if each element is present in the new list. If yes, then we will move further. If not, then we will insert that element in our new list. When the loop ends, we will have a Python list with no duplicate values.

Code Example

exis_list = [ 1,2,2,3,4,5,6,7,5,4,2,3,7,8]
new_list = []
for i in exis_list:
    if not i in new_list:
        new_list.append(i)

print("The old list is:", exis_list)
print("The new list is:", new_list)

Output

The old list is: [1, 2, 2, 3, 4, 5, 6, 7, 5, 4, 2, 3, 7, 8]
The new list is: [1, 2, 3, 4, 5, 6, 7, 8]

2. Removing Duplicate Elements from a Python List Using the set() Method

A Python set is an unordered and immutable data structure. Also, a set in Python does not store duplicate values, and with the help of the set() method, we can convert a list object to a set object. If we convert the list into a set, all the duplicate elements from the list are omitted.

Code Example

exis_list = [ 1,2,2,3,4,5,6,7,5,4,2,3,7,8]
new_set = set(exis_list)
new_list = list(new_set)

print("The old list is:", exis_list)
print("The new list is:", new_list)

Output

The old list is: [1, 2, 2, 3, 4, 5, 6, 7, 5, 4, 2, 3, 7, 8]
The new list is: [1, 2, 3, 4, 5, 6, 7, 8]

3. How to Remove Duplicates from a Python List Using List Comprehension and Enumerate()

The enumerate() function accepts a list object and returns an enumerate object, which contains a tuple pair with index numbers and corresponding list values. Using list comprehension, we can create a new list that looks for the elements that are occurring the first time and add them.

Code Example

exis_list = [ 1,2,2,3,4,5,6,7,5,4,2,3,7,8]
new_list = [ value for index,value in enumerate(exis_list) if value not in exis_list[:index]]

print("The old list is:", exis_list)
print("The new list is:", new_list)

Output

The old list is: [1, 2, 2, 3, 4, 5, 6, 7, 5, 4, 2, 3, 7, 8]
The new list is: [1, 2, 3, 4, 5, 6, 7, 8] 

4. Removing Duplicate Elements from a Python List Using the .fromkeys() Method

The .formkeys() is a dictionary method that is used to convert list values to dictionary keys. A dictionary can only have immutable and unique keys, and the .formkeys() method accepts a 1D list and creates a dictionary with list values as keys, and sets their values to None. After creating the dictionary using the dict.fromkeys() method, we can change back dictionary to list using the list() method.

Code Example

exis_list = [ 1,2,2,3,4,5,6,7,5,4,2,3,7,8]
new_dict = dict.fromkeys(exis_list)
new_list = list(new_dict)

print("The old list is:", exis_list)
print("The new list is:", new_list)

Output

The old list is: [1, 2, 2, 3, 4, 5, 6, 7, 5, 4, 2, 3, 7, 8]
The new list is: [1, 2, 3, 4, 5, 6, 7, 8]

Conclusion

With this, we have reached the end of our tutorial on how to remove duplicates from a Python list. In most cases, you will be using Naïve and .formkeys() methods. However, using the set() method to create a unique list is a very clean and fast method. Have any suggestions regarding the article? Let us know via comments.

People are also reading:

• Extract Image Metadata in Python
• Crack a ZIP File Password in Python
• How to Translate Languages in Python?
• MySQL Database in Python
• Get Domain Name Information in Python
• How to Get Geolocation in Python?
• SIFT Feature Extraction in Python
• SYN Flooding Attack in Python
• Extract YouTube Comments in Python
• Edge Detection in Python