Home » Python » Find unique rows in numpy.array

# Find unique rows in numpy.array

Questions:

I need to find unique rows in a `numpy.array`.

For example:

``````>>> a # I have
array([[1, 1, 1, 0, 0, 0],
[0, 1, 1, 1, 0, 0],
[0, 1, 1, 1, 0, 0],
[1, 1, 1, 0, 0, 0],
[1, 1, 1, 1, 1, 0]])
>>> new_a # I want to get to
array([[1, 1, 1, 0, 0, 0],
[0, 1, 1, 1, 0, 0],
[1, 1, 1, 1, 1, 0]])
``````

I know that i can create a set and loop over the array, but I am looking for an efficient pure `numpy` solution. I believe that there is a way to set data type to void and then I could just use `numpy.unique`, but I couldn’t figure out how to make it work.

As of NumPy 1.13, one can simply choose the axis for selection of unique values in any N-dim array. To get unique rows, one can do:

`unique_rows = np.unique(original_array, axis=0)`

Questions:

Yet another possible solution

``````np.vstack({tuple(row) for row in a})
``````

Questions:

Another option to the use of structured arrays is using a view of a `void` type that joins the whole row into a single item:

``````a = np.array([[1, 1, 1, 0, 0, 0],
[0, 1, 1, 1, 0, 0],
[0, 1, 1, 1, 0, 0],
[1, 1, 1, 0, 0, 0],
[1, 1, 1, 1, 1, 0]])

b = np.ascontiguousarray(a).view(np.dtype((np.void, a.dtype.itemsize * a.shape[1])))
_, idx = np.unique(b, return_index=True)

unique_a = a[idx]

>>> unique_a
array([[0, 1, 1, 1, 0, 0],
[1, 1, 1, 0, 0, 0],
[1, 1, 1, 1, 1, 0]])
``````

EDIT
Added `np.ascontiguousarray` following @seberg’s recommendation. This will slow the method down if the array is not already contiguous.

EDIT
The above can be slightly sped up, perhaps at the cost of clarity, by doing:

``````unique_a = np.unique(b).view(a.dtype).reshape(-1, a.shape[1])
``````

Also, at least on my system, performance wise it is on par, or even better, than the lexsort method:

``````a = np.random.randint(2, size=(10000, 6))

%timeit np.unique(a.view(np.dtype((np.void, a.dtype.itemsize*a.shape[1])))).view(a.dtype).reshape(-1, a.shape[1])
100 loops, best of 3: 3.17 ms per loop

%timeit ind = np.lexsort(a.T); a[np.concatenate(([True],np.any(a[ind[1:]]!=a[ind[:-1]],axis=1)))]
100 loops, best of 3: 5.93 ms per loop

a = np.random.randint(2, size=(10000, 100))

%timeit np.unique(a.view(np.dtype((np.void, a.dtype.itemsize*a.shape[1])))).view(a.dtype).reshape(-1, a.shape[1])
10 loops, best of 3: 29.9 ms per loop

%timeit ind = np.lexsort(a.T); a[np.concatenate(([True],np.any(a[ind[1:]]!=a[ind[:-1]],axis=1)))]
10 loops, best of 3: 116 ms per loop
``````

Questions:

If you want to avoid the memory expense of converting to a series of tuples or another similar data structure, you can exploit numpy’s structured arrays.

The trick is to view your original array as a structured array where each item corresponds to a row of the original array. This doesn’t make a copy, and is quite efficient.

As a quick example:

``````import numpy as np

data = np.array([[1, 1, 1, 0, 0, 0],
[0, 1, 1, 1, 0, 0],
[0, 1, 1, 1, 0, 0],
[1, 1, 1, 0, 0, 0],
[1, 1, 1, 1, 1, 0]])

ncols = data.shape[1]
dtype = data.dtype.descr * ncols
struct = data.view(dtype)

uniq = np.unique(struct)
uniq = uniq.view(data.dtype).reshape(-1, ncols)
print uniq
``````

To understand what’s going on, have a look at the intermediary results.

Once we view things as a structured array, each element in the array is a row in your original array. (Basically, it’s a similar data structure to a list of tuples.)

``````In [71]: struct
Out[71]:
array([[(1, 1, 1, 0, 0, 0)],
[(0, 1, 1, 1, 0, 0)],
[(0, 1, 1, 1, 0, 0)],
[(1, 1, 1, 0, 0, 0)],
[(1, 1, 1, 1, 1, 0)]],
dtype=[('f0', '<i8'), ('f1', '<i8'), ('f2', '<i8'), ('f3', '<i8'), ('f4', '<i8'), ('f5', '<i8')])

In [72]: struct[0]
Out[72]:
array([(1, 1, 1, 0, 0, 0)],
dtype=[('f0', '<i8'), ('f1', '<i8'), ('f2', '<i8'), ('f3', '<i8'), ('f4', '<i8'), ('f5', '<i8')])
``````

Once we run `numpy.unique`, we’ll get a structured array back:

``````In [73]: np.unique(struct)
Out[73]:
array([(0, 1, 1, 1, 0, 0), (1, 1, 1, 0, 0, 0), (1, 1, 1, 1, 1, 0)],
dtype=[('f0', '<i8'), ('f1', '<i8'), ('f2', '<i8'), ('f3', '<i8'), ('f4', '<i8'), ('f5', '<i8')])
``````

That we then need to view as a “normal” array (`_` stores the result of the last calculation in `ipython`, which is why you’re seeing `_.view...`):

``````In [74]: _.view(data.dtype)
Out[74]: array([0, 1, 1, 1, 0, 0, 1, 1, 1, 0, 0, 0, 1, 1, 1, 1, 1, 0])
``````

And then reshape back into a 2D array (`-1` is a placeholder that tells numpy to calculate the correct number of rows, give the number of columns):

``````In [75]: _.reshape(-1, ncols)
Out[75]:
array([[0, 1, 1, 1, 0, 0],
[1, 1, 1, 0, 0, 0],
[1, 1, 1, 1, 1, 0]])
``````

Obviously, if you wanted to be more concise, you could write it as:

``````import numpy as np

def unique_rows(data):
uniq = np.unique(data.view(data.dtype.descr * data.shape[1]))
return uniq.view(data.dtype).reshape(-1, data.shape[1])

data = np.array([[1, 1, 1, 0, 0, 0],
[0, 1, 1, 1, 0, 0],
[0, 1, 1, 1, 0, 0],
[1, 1, 1, 0, 0, 0],
[1, 1, 1, 1, 1, 0]])
print unique_rows(data)
``````

Which results in:

``````[[0 1 1 1 0 0]
[1 1 1 0 0 0]
[1 1 1 1 1 0]]
``````

Questions:

`np.unique` when I run it on `np.random.random(100).reshape(10,10)` returns all the unique individual elements, but you want the unique rows, so first you need to put them into tuples:

``````array = #your numpy array of lists
new_array = [tuple(row) for row in array]
uniques = np.unique(new_array)
``````

That is the only way I see you changing the types to do what you want, and I am not sure if the list iteration to change to tuples is okay with your “not looping through”

Questions:

np.unique works by sorting a flattened array, then looking at whether each item is equal to the previous. This can be done manually without flattening:

``````ind = np.lexsort(a.T)
a[ind[np.concatenate(([True],np.any(a[ind[1:]]!=a[ind[:-1]],axis=1)))]]
``````

This method does not use tuples, and should be much faster and simpler than other methods given here.

NOTE: A previous version of this did not have the ind right after a[, which mean that the wrong indices were used. Also, Joe Kington makes a good point that this does make a variety of intermediate copies. The following method makes fewer, by making a sorted copy and then using views of it:

``````b = a[np.lexsort(a.T)]
b[np.concatenate(([True], np.any(b[1:] != b[:-1],axis=1)))]
``````

This is faster and uses less memory.

Also, if you want to find unique rows in an ndarray regardless of how many dimensions are in the array, the following will work:

``````b = a[lexsort(a.reshape((a.shape[0],-1)).T)];
b[np.concatenate(([True], np.any(b[1:]!=b[:-1],axis=tuple(range(1,a.ndim)))))]
``````

An interesting remaining issue would be if you wanted to sort/unique along an arbitrary axis of an arbitrary-dimension array, something that would be more difficult.

Edit:

To demonstrate the speed differences, I ran a few tests in ipython of the three different methods described in the answers. With your exact a, there isn’t too much of a difference, though this version is a bit faster:

``````In [87]: %timeit unique(a.view(dtype)).view('<i8')
10000 loops, best of 3: 48.4 us per loop

In [88]: %timeit ind = np.lexsort(a.T); a[np.concatenate(([True], np.any(a[ind[1:]]!= a[ind[:-1]], axis=1)))]
10000 loops, best of 3: 37.6 us per loop

In [89]: %timeit b = [tuple(row) for row in a]; np.unique(b)
10000 loops, best of 3: 41.6 us per loop
``````

With a larger a, however, this version ends up being much, much faster:

``````In [96]: a = np.random.randint(0,2,size=(10000,6))

In [97]: %timeit unique(a.view(dtype)).view('<i8')
10 loops, best of 3: 24.4 ms per loop

In [98]: %timeit b = [tuple(row) for row in a]; np.unique(b)
10 loops, best of 3: 28.2 ms per loop

In [99]: %timeit ind = np.lexsort(a.T); a[np.concatenate(([True],np.any(a[ind[1:]]!= a[ind[:-1]],axis=1)))]
100 loops, best of 3: 3.25 ms per loop
``````

Questions:

Here is another variation for @Greg pythonic answer

``````np.vstack(set(map(tuple, a)))
``````

Questions:

I didn’t like any of these answers because none handle floating-point arrays in a linear algebra or vector space sense, where two rows being “equal” means “within some ?”. The one answer that has a tolerance threshold, https://stackoverflow.com/a/26867764/500207, took the threshold to be both element-wise and decimal precision, which works for some cases but isn’t as mathematically general as a true vector distance.

Here’s my version:

``````from scipy.spatial.distance import squareform, pdist

def uniqueRows(arr, thresh=0.0, metric='euclidean'):
"Returns subset of rows that are unique, in terms of Euclidean distance"
distances = squareform(pdist(arr, metric=metric))
idxset = {tuple(np.nonzero(v)[0]) for v in distances <= thresh}
return arr[[x[0] for x in idxset]]

# With this, unique columns are super-easy:
def uniqueColumns(arr, *args, **kwargs):
return uniqueRows(arr.T, *args, **kwargs)
``````

The public-domain function above uses `scipy.spatial.distance.pdist` to find the Euclidean (customizable) distance between each pair of rows. Then it compares each each distance to a `thresh`old to find the rows that are within `thresh` of each other, and returns just one row from each `thresh`-cluster.

As hinted, the distance `metric` needn’t be Euclidean—`pdist` can compute sundry distances including `cityblock` (Manhattan-norm) and `cosine` (the angle between vectors).

If `thresh=0` (the default), then rows have to be bit-exact to be considered “unique”. Other good values for `thresh` use scaled machine-precision, i.e., `thresh=np.spacing(1)*1e3`.

Questions:

The numpy_indexed package (disclaimer: I am its author) wraps the solution posted by Jaime in a nice and tested interface, plus many more features:

``````import numpy_indexed as npi
new_a = npi.unique(a)  # unique elements over axis=0 (rows) by default
``````

Questions:

Why not use `drop_duplicates` from pandas:

``````>>> timeit pd.DataFrame(image.reshape(-1,3)).drop_duplicates().values
1 loops, best of 3: 3.08 s per loop

>>> timeit np.vstack({tuple(r) for r in image.reshape(-1,3)})
1 loops, best of 3: 51 s per loop
``````

Questions:

I’ve compared the suggested alternative for speed and found that, surprisingly, the void view `unique` solution is even a bit faster than numpy’s native `unique` with the `axis` argument. The difference is notable only for small arrays though, and given the increased clarity,

``````numpy.unique(a, axis=0)
``````

is probably the best solution if you have NumPy 1.13 or later.

Code to reproduce the plot:

``````import numpy
import perfplot

def unique_void_view(a):
return numpy.unique(
a.view(numpy.dtype((numpy.void, a.dtype.itemsize*a.shape[1])))
).view(a.dtype).reshape(-1, a.shape[1])

def lexsort(a):
ind = numpy.lexsort(a.T)
return a[ind[
numpy.concatenate((
[True], numpy.any(a[ind[1:]] != a[ind[:-1]], axis=1)
))
]]

def vstack(a):
return numpy.vstack({tuple(row) for row in a})

def unique_axis(a):
return numpy.unique(a, axis=0)

perfplot.show(
setup=lambda n: numpy.random.randint(2, size=(n, 20)),
kernels=[unique_void_view, lexsort, vstack, unique_axis],
n_range=[2**k for k in range(15)],
logx=True,
logy=True,
xlabel='len(a)',
equality_check=None
)
``````

Questions:

np.unique works given a list of tuples:

``````>>> np.unique([(1, 1), (2, 2), (3, 3), (4, 4), (2, 2)])
Out[9]:
array([[1, 1],
[2, 2],
[3, 3],
[4, 4]])
``````

With a list of lists it raises a `TypeError: unhashable type: 'list'`

Questions:

Based on the answer in this page I have written a function that replicates the capability of MATLAB’s `unique(input,'rows')` function, with the additional feature to accept tolerance for checking the uniqueness. It also returns the indices such that `c = data[ia,:]` and `data = c[ic,:]`. Please report if you see any discrepancies or errors.

``````def unique_rows(data, prec=5):
import numpy as np
d_r = np.fix(data * 10 ** prec) / 10 ** prec + 0.0
b = np.ascontiguousarray(d_r).view(np.dtype((np.void, d_r.dtype.itemsize * d_r.shape[1])))
_, ia = np.unique(b, return_index=True)
_, ic = np.unique(b, return_inverse=True)
return np.unique(b).view(d_r.dtype).reshape(-1, d_r.shape[1]), ia, ic
``````

Questions:

Beyond @Jaime excellent answer, another way to collapse a row is to uses `a.strides[0]` (assuming `a` is C-contiguous) which is equal to `a.dtype.itemsize*a.shape[0]`. Furthermore `void(n)` is a shortcut for `dtype((void,n))`. we arrive finally to this shortest version :

``````a[unique(a.view(void(a.strides[0])),1)[1]]
``````

For

``````[[0 1 1 1 0 0]
[1 1 1 0 0 0]
[1 1 1 1 1 0]]
``````

Questions:

For general purpose like 3D or higher multidimensional nested arrays, try this:

``````import numpy as np

def unique_nested_arrays(ar):
origin_shape = ar.shape
origin_dtype = ar.dtype
ar = ar.reshape(origin_shape[0], np.prod(origin_shape[1:]))
ar = np.ascontiguousarray(ar)
unique_ar = np.unique(ar.view([('', origin_dtype)]*np.prod(origin_shape[1:])))
return unique_ar.view(origin_dtype).reshape((unique_ar.shape[0], ) + origin_shape[1:])
``````

``````a = np.array([[1, 1, 1, 0, 0, 0],
[0, 1, 1, 1, 0, 0],
[0, 1, 1, 1, 0, 0],
[1, 1, 1, 0, 0, 0],
[1, 1, 1, 1, 1, 0]])
unique_nested_arrays(a)
``````

gives:

``````array([[0, 1, 1, 1, 0, 0],
[1, 1, 1, 0, 0, 0],
[1, 1, 1, 1, 1, 0]])
``````

But also 3D arrays like:

``````b = np.array([[[1, 1, 1], [0, 1, 1]],
[[0, 1, 1], [1, 1, 1]],
[[1, 1, 1], [0, 1, 1]],
[[1, 1, 1], [1, 1, 1]]])
unique_nested_arrays(b)
``````

gives:

``````array([[[0, 1, 1], [1, 1, 1]],
[[1, 1, 1], [0, 1, 1]],
[[1, 1, 1], [1, 1, 1]]])
``````

Questions:

None of these answers worked for me. I’m assuming as my unique rows contained strings and not numbers. However this answer from another thread did work:

You can use .count() and .index() list’s methods

``````coor = np.array([[10, 10], [12, 9], [10, 5], [12, 9]])
coor_tuple = [tuple(x) for x in coor]
unique_coor = sorted(set(coor_tuple), key=lambda x: coor_tuple.index(x))
unique_count = [coor_tuple.count(x) for x in unique_coor]
unique_index = [coor_tuple.index(x) for x in unique_coor]
``````

Questions:

We can actually turn m x n numeric numpy array into m x 1 numpy string array, please try using the following function, it provides count, inverse_idx and etc, just like numpy.unique:

``````import numpy as np

def uniqueRow(a):
#This function turn m x n numpy array into m x 1 numpy array storing
#string, and so the np.unique can be used

#Input: an m x n numpy array (a)
#Output unique m' x n numpy array (unique), inverse_indx, and counts

s = np.chararray((a.shape[0],1))
s[:] = '-'

b = (a).astype(np.str)

s2 = np.expand_dims(b[:,0],axis=1) + s + np.expand_dims(b[:,1],axis=1)

n = a.shape[1] - 2

for i in range(0,n):
s2 = s2 + s + np.expand_dims(b[:,i+2],axis=1)

s3, idx, inv_, c = np.unique(s2,return_index = True,  return_inverse = True, return_counts = True)

return a[idx], inv_, c
``````

Example:

``````A = np.array([[ 3.17   9.502  3.291],
[ 9.984  2.773  6.852],
[ 1.172  8.885  4.258],
[ 9.73   7.518  3.227],
[ 8.113  9.563  9.117],
[ 9.984  2.773  6.852],
[ 9.73   7.518  3.227]])

B, inv_, c = uniqueRow(A)

Results:

B:
[[ 1.172  8.885  4.258]
[ 3.17   9.502  3.291]
[ 8.113  9.563  9.117]
[ 9.73   7.518  3.227]
[ 9.984  2.773  6.852]]

inv_:
[3 4 1 0 2 4 0]

c:
[2 1 1 1 2]
``````

Questions:

Lets get the entire numpy matrix as a list, then drop duplicates from this list, and finally return our unique list back into a numpy matrix:

``````matrix_as_list=data.tolist()
matrix_as_list:
[[1, 1, 1, 0, 0, 0], [0, 1, 1, 1, 0, 0], [0, 1, 1, 1, 0, 0], [1, 1, 1, 0, 0, 0], [1, 1, 1, 1, 1, 0]]

uniq_list=list()
uniq_list.append(matrix_as_list[0])

[uniq_list.append(item) for item in matrix_as_list if item not in uniq_list]

unique_matrix=np.array(uniq_list)
unique_matrix:
array([[1, 1, 1, 0, 0, 0],
[0, 1, 1, 1, 0, 0],
[1, 1, 1, 1, 1, 0]])
``````

Questions:

The most straightforward solution is to make the rows a single item by making them strings. Each row then can be compared as a whole for its uniqueness using numpy. This solution is generalize-able you just need to reshape and transpose your array for other combinations. Here is the solution for the problem provided.

``````import numpy as np

original = np.array([[1, 1, 1, 0, 0, 0],
[0, 1, 1, 1, 0, 0],
[0, 1, 1, 1, 0, 0],
[1, 1, 1, 0, 0, 0],
[1, 1, 1, 1, 1, 0]])

uniques, index = np.unique([str(i) for i in original], return_index=True)
cleaned = original[index]
print(cleaned)
``````

Will Give:

`````` array([[0, 1, 1, 1, 0, 0],
[1, 1, 1, 0, 0, 0],
[1, 1, 1, 1, 1, 0]])
``````

Send my nobel prize in the mail

Questions:
``````import numpy as np