Python Remove Duplicates From List With Examples

Duplicates generally mean exactly like something else, especially through having been copied. So if you want to Python Remove Duplicates from List, there are basically 5 methods by which you can achieve the duplication or copy-free Python list.

Contents

Ways to remove duplicates from a list in Python

So talking about the ways to remove the duplicates from Python list there are generally following 5 techniques or methods:

Using set() function
Using a temporary list
By using Enumerate
Using itertools groupby
By Maintaining the order using OrderedDict

Using set() function to remove duplicates from Python list

The easiest way is to convert it to a set, because sets, by definition, don’t have duplicates:

As we know, that set is the unordered collection of unique elements. Python internally uses the hash technique to process sets. It is a quite simple and fast method to remove duplicate items from Python list.

The original order of the sequence (list) may not be preserved using this approach. However, this method will also not work if you have dict or similar object in the list because dict the object is not hashable in Python.

The set is formed by placing all the items inside the curly braces {} and the items are separated by comma or by using the built-in function set().

The set can have any number of items, and they may be of different types (integer, float, tuple, string, etc.). But the set cannot have a mutable element, like a list, or dictionary, as its element.

numbers = [1, 2, 1, 2, 3, 5, 3]
numbers = list(set(numbers))
print('The list after removing duplicate elements',numbers)

Output:
The list after removing duplicate elements [1, 2, 3, 5]

By using a temporary list to remove duplicates from Python list

Here we use the brute-force approach to remove duplicate elements from a list. We will create a temporary list and append elements to it only if it’s not present.

ints_list = [1, 2, 1, 2, 3, 5, 3]
temp = []

for x in ints_list:
    if x not in temp:
        temp.append(x)

ints_list = temp

print(f'The list after removing duplicate elements = {temp}')

Create a basic list with the elements separated by commas enclosed in square brackets.
After that let’s create an empty or a temporary list.
Using the python for loop lets check each element in the initialized list that is the basic list.
After that let’s append elements to the temporary list if those elements are not present.
Now let’s equate the basic list with the temporary list.

Output:
The list after removing duplicate elements [1, 2, 3, 5]

By using Enumerate to remove duplicates from Python list

A lot of times when dealing with iterators, we also get a need to keep a count of iterations. Python eases the programmers’ task by providing a built-in function enumerate() for this task.

So in this example, we will use enumerate to remove python remove duplicates from the list.
Enumerate() method adds a counter to an iterable and returns it in a form of enumerating object. This enumerate object can then be used directly in for loops or be converted into a list of tuples using list() method.

x = [1, 2, 1, 2, 3, 5, 3]

print([v for i, v in enumerate(x) if i == 0 or v != x[i-1]])

Output:
The list after removing duplicate elements [1, 2, 3, 5]

Using itertools groupby to Remove Duplicates from List

The 4th way to Remove Duplicates from a Python list is using the itertools library.
In addition, the itertools module implements a number of iterator building blocks. Each has been recast in a form suitable for Python.

However, this module standardizes a core set of fast, memory-efficient. In addition, they form an “iterator algebra” making it possible to construct specialized tools succinctly and efficiently in pure Python.

For instance, SML provides a tabulation tool: tabulate(f) which produces a sequence f(0), f(1), .... Similarly, the same effect can be achieved in Python by combining map.

These tools work well with the high-speed functions in the operator module. For example, the multiplication operator.

In addition, we can say itertools groupby is one of the best methods to remove Duplicates from the Python list. As it uses the builtin module, so by default it’s speed will be higher than the above-mentioned methods to remove duplication in Python list.

from itertools import groupby

x = [1, 2, 1, 2, 3, 5, 3]
print([i[0] for i in groupby(x)])

Output:
The list after removing duplicate elements [1, 2, 3, 5]

Must Read

By Maintaining the order using OrderedDict

In Python 3.5 and above OrderedDict of collections library has C implementation.

An OrderedDict is a dictionary subclass that remembers the order that keys were first inserted. However, the only difference between dict() and OrderedDict() is that:

OrderedDict preserves the order in which the keys are inserted. As the regular dict doesn’t track the insertion order. However, the order the items are inserted is remembered by OrderedDict.

This means it is a quite fast and best technique to remove duplication elements from the list in Python. You should also give a try too:

from collections import OrderedDict 
# initializing list 
test_list = [1, 2, 1, 2, 3, 5, 3]
res = list(OrderedDict.fromkeys(test_list)) 
# printing list after removal 
print ("The list after removing duplicates : " + str(res))

Output:
The list after removing duplicate elements [1, 2, 3, 5]

Conclusion

In conclusion, we hope you enjoyed out this article.

We have tried to cover all ways for Python Remove Duplicates From List. In addition let us know if you have a better way of doing this, through the comments.