"pythonic" way to fill bag of words

Question

I've got a list of words, about 273000 of them in the list Word_array There are about 17000 unique words, and they're stored in Word_arrayU

I want a count for each one

#make bag of worsds   
Word_arrayU = np.unique(Word_array)
wordBag = [['0','0'] for _ in range(len(Word_array))] #prealocate necessary space
i=0
while i< len(Word_arrayU): #for each unique word
    wordBag[i][0] = Word_arrayU[i]
    #I think this is the part that takes a long time.  summing up a list comprehension with a conditional.  Just seems sloppy
    wordBag[i][1]=sum([1 if x == Word_arrayU[i] else 0 for x in Word_array])
    i=i+1

summing up a list comprehension with a conditional. Just seems sloppy; is there a better way to do it?

have you looked at using Counter from collections? It's sole purpose is to give you a dictionary where the keys are the items from an iterable and the value is the count of those items from the iterable... — bravosierra99, Oct 13 '16 at 20:09
If you're calling `np.unique` then wouldn't you remove all the duplicates anyway, meaning every count is 1? — roganjosh, Oct 13 '16 at 20:15
_sum( x == Word_arrayU[i] for x in Word_array)_ will do the trick - Pytnon True and False may be used whenever _int_ type is used. That will probably save some execution time - but Counter approach still looks better — volcano, Oct 13 '16 at 20:18
BTW, the main reason it's so slow is that your cycle is O(17,000*273,000) - for each unique word you scan the original list. — volcano, Oct 13 '16 at 20:36

score 2 · Answer 1 · edited Oct 13 '16 at 20:35

2

from collections import Counter
counter = Counter(Word_array)
the_count_of_some_word = counter["some_word"]

#printing the counts
for word, count in counter.items():
   print("{} appears {} times.".format(word, count)

edited Oct 13 '16 at 20:35

Padraic Cunningham

160,756
20
201
286

answered Oct 13 '16 at 20:15

bravosierra99

1,221
9
21

Padraic Cunningham · Answer 2 · 2016-10-13T20:33:36.923

Since you are already using numpy.unique, just set return_counts=True in the unique call:

import numpy as np

unique,  count = np.unique(Word_array, return_counts=True)

That will give you two arrays, the unique elements and their counts:

n [10]: arr = [1,3,2,11,3,4,5,2,3,4]

In [11]: unique,  count = np.unique(arr, return_counts=True)

In [12]: unique
Out[12]: array([ 1,  2,  3,  4,  5, 11])

In [13]: count
Out[13]: array([1, 2, 3, 2, 1, 1])

score 0 · Answer 3 · answered Oct 13 '16 at 20:16

Building on the suggestion from @jonrsharpe...

from collections import Counter

words = Counter()

words['foo'] += 1
words['foo'] += 1
words['bar'] += 1

Output

Counter({'bar': 1, 'foo': 2})

It's really convenient because you don't have to initialize words.

You can also initialize directly from a list of words:

Counter(['foo', 'foo', 'bar'])

Output

Counter({'bar': 1, 'foo': 2})

work.bin · Answer 4 · 2016-10-13T20:51:05.370

0

I don't know about most 'Pythonic' but definitely the easiest way of doing this would be to use collections.Counter.

from collections import Counter

Word_array = ["word1", "word2", "word3", "word1", "word2", "word1"]

wordBag = Counter(Word_array).items()

edited Oct 13 '16 at 20:51

answered Oct 13 '16 at 20:16

work.bin

1,048
5
25

Terrible suggestion - count twice? If you - for some obscure reason - want tuples, _Counter(Word\_array).items()_ will do the trick – volcano Oct 13 '16 at 20:32
1

If you call a function over t the same data twice - that is bad practice. Since OP is talking about list of 237,000 words - yep, it is terrible. – volcano Oct 13 '16 at 21:16

score 0 · Answer 5 · answered Oct 13 '16 at 20:26

In python 3 there is a built-in list.count function. For example:

>>> h = ["a", "b", "a", "a", "c"]
>>> h.count("a")
3
>>>

So, you could make it more efficient by doing something like:

Word_arrayU = np.unique(Word_array)
wordBag = []
for uniqueWord in Word_arrayU:
    wordBag.append([uniqueWord, Word_array.count(uniqueWord)])

score -1 · Answer 6 · answered Oct 13 '16 at 20:17

-1

If you want a less efficient (than Counter), but more transparent solution, you can use collections.defaultdict

from collections import defaultdict
my_counter = defaultdict(int)
for word in word_array:
    my_counter[word] += 1

answered Oct 13 '16 at 20:17

Patrick Haugh

49,982
11
66
73

3

why would he use defaultdict over Counter? – bravosierra99 Oct 13 '16 at 20:18

"pythonic" way to fill bag of words

6 Answers6