How can i save a Panda dataframe to a django model?

Question

I am trying to read a csv file using panda and parse it and then upload the results in my django database. Well, for now i am converting each dataframe to a list and then iterating over the list to save it in the DB. But my solution is inefficient when the list is really big for each column. How can i make it better ?

 fileinfo = pd.read_csv(csv_file, sep=',',
                        names=['Series_reference', 'Period', 'Data_value', 'STATUS',
                               'UNITS', 'Subject', 'Group', 'Series_title_1', 'Series_title_2',
                               'Series_title_3','Series_tile_4','Series_tile_5'],
                        skiprows = 1)

# serie = fileinfo[fileinfo['Series_reference']]
s = fileinfo['Series_reference'].values.tolist()
p = fileinfo['Period'].values.tolist()
d = fileinfo['Data_value'].values.tolist()
st = fileinfo['STATUS'].values.tolist()
u = fileinfo['UNITS'].values.tolist()
sub = fileinfo['Subject'].values.tolist()
gr = fileinfo['Group'].values.tolist()
stt= fileinfo['Series_title_1'].values.tolist()

while count < len(s):    
    b = Testdata(
        Series_reference = s[count],
        Period = p[count],
        Data_value = d[count],
        STATUS = st[count],
        UNITS = u[count],
        Subject = sub[count],
        Group = gr[count],
        Series_title_1 = stt[count]
    )
    b.save()
    count = count + 1

My first instinct would use pandas.DataFrame.to_dict Then iterate through the dict to create a pile of instinated Testdata instances. I found this SO link helpful. https://stackoverflow.com/questions/1639174/creating-class-instance-properties-from-a-dictionary — Timothy Lombard, Jun 15 '18 at 14:06

score 0 · Answer 1 · answered Jun 15 '18 at 14:04

You can use pandas apply function. You can pass axis=1 to apply a given function to every row:

df.apply(
    creational_function, # Method that creates your structure
    axis=1, # Apply to every row
    args=(arg1, arg2) # Additional args to creational_function
)

in creational_function the first argument received is the row, where you can access specific columns likewise the original dataframe

def creational_function(row, arg1, arg2):
    s = row['Series_reference']
    # For brevity I skip the others arguments...
    # Create TestData
    # Save

Note that arg1 and arg2 are the same for every row.

If you want to do something more with your created TestData objects, you can change creational_function to return a value, then df.apply will return a list containing all elements returned by the passed function.

How can i save a Panda dataframe to a django model?

1 Answers1