missing values writing a csv file with python

Question

I am trying to write a csv file with python from a dictionnary which looks like follows:

1: [(3456,12),(4521,78)]
2: [(5478,45),(5609,43)]
3: [(8756,34),(6721,90)]

etc ...

I have several dictionaries like this. They can have until 30 keys. To write all the dictionnaries in a csv file, I wrote the following code which will be applied to each dictionnary:

for i,j in dictionnary.iteritems():
   for k in j:
     cr=csv.reader(open('/home/file.csv','rb'))
     datalist=list(cr)
     c=writerows(datalist+[[self.user_id, i,k[0],k[1]])

When I open the csv file, only a part of the dictionary is written. When I run multiple times the code, not the same entries of the dictionary are missing. I don't understand what happens. Where could the problem come from?

Try this: http://stackoverflow.com/questions/14693646/writing-to-csv-file-python — Sharif Mamun, Mar 08 '14 at 23:19
I thought I understood your problem, but I don't. Obviously, we're missing an essential part of your code. I don't understand why you **read** the same csv file on each iteration and the what the `writerows` function does. It cannot be the `csv.writer.writerows` as it's not attached to any object. See also the [example in the official documentation](http://docs.python.org/3.3/library/csv.html?highlight=csv#csv.writer) on how to use the writer. — Cilyan, Mar 08 '14 at 23:28
I think they're trying to append data to the end of the CSV. Which would explain why OP would need to refresh the `datalist` every loop — Tejas Pendse, Mar 08 '14 at 23:33
I read the csv file on each iteration because if I create the csv.writer before the iteration, i am said that the file is not open. The writerows function adds the content of my dictionnary at the end of what has already been written in the csv file. What is strange is that the code works, but only few items of my dictionnary are missing. And they are not the same at each run — user2879969, Mar 08 '14 at 23:40

Tejas Pendse · Answer 1 · 2014-03-09T00:14:15.323

0

You're reading the file everytime you write to it. This means, you read the data twice. As others have suggested take the csv.reader( open(...) ) part out of the loop. Also, I think you're trying to append data to the end of CSV. If that's what you're after, try using the with command and opening the file in ab (append binary) mode:

with open('/home/file.csv','ab') as csvfile:
    csvwriter = csv.writer( csvfile, delimiter=' ', quoting=csv.QUOTE_MINIMAL )
    for i,j in dictionary.iteritems():
        for k in j:
            csvwriter.writerow( [ self.user_id, i, k[0], k[1] ] )

edited Mar 09 '14 at 00:14

answered Mar 08 '14 at 23:31

Tejas Pendse

558
6
16

I obtain the following message:'expecting for a character buffer object' – user2879969 Mar 08 '14 at 23:49
I fixed a few things. I'm using a `csvwriter` and opening the file in `ab` which is append in binary. Check the edits. – Tejas Pendse Mar 09 '14 at 00:15
Did it work? If it did, would you mind accepting an answer? Thanks! – Tejas Pendse Mar 09 '14 at 00:33

score 0 · Answer 2 · answered Mar 09 '14 at 03:32

Maybe this code will give you ideas about how to manage to complete a CSV file:

import csv
d1 = {1: [(3456,12),(4521,78)],
      2: [(5478,45),(5609,43)],
      3: [(8756,34),(6721,90)] }

d2 = {1: [(34,1),(4,7)],
      2: [(58,5),(56,4)],
      3: [(87,3),(67,9)] }

d3 = {1: [(111,22),(333,44)],
      3: [(1000,300),(2000,400)] }

d4 = {'AAA': [('arrze','iifuiyu')],
      'BBB': [(575757,'434343434')] }

def yi(input):
    for user_id,d in input:
        for i in d:
            for tu in d[i]:
                    yield [user_id, i, tu[0], tu[1]]

def csvcompl(fn,ds):           
    with open(fn,'ab') as cc:
        cr = csv.writer(cc)
        cr.writerows(yi(ds))

ds = (('Lavillenie',d1),('Abba',d2),(987654321,d3))
csvcompl('fifiroro.csv',ds)
csvcompl('fifiroro.csv',(('OPO',d4),))

with open('fifiroro.csv','r') as f:
    print f.read()

result

Lavillenie,1,3456,12
Lavillenie,1,4521,78
Lavillenie,2,5478,45
Lavillenie,2,5609,43
Lavillenie,3,8756,34
Lavillenie,3,6721,90
Abba,1,34,1
Abba,1,4,7
Abba,2,58,5
Abba,2,56,4
Abba,3,87,3
Abba,3,67,9
987654321,1,111,22
987654321,1,333,44
987654321,3,1000,300
987654321,3,2000,400
OPO,AAA,arrze,iifuiyu
OPO,BBB,575757,434343434

missing values writing a csv file with python

2 Answers2