Writing dictionary of dataframes to file

Question

I have a dictionary, and for each key in my dictionary, I have one pandas dataframe. The dataframes from key to key are of unequal length.

It takes some time to get to the dataframes that are connected to each key, and therefore I wish to save my dictionary of dataframes to a file, so I can just read the file into Python instead of running my script every time I open Python.

My question is: How would you suggest to write the dictionary with dataframes to a file - and to read it in again? I have tried the following, where dictex is the dictionary:

w = csv.writer(open("output.csv", "w"))
for key, val in dictex.items():
    w.writerow([key, val])

But I am not really sure if I get what I want, as I struggle to read the file into Python again.

Thank you for your time.

Do these dataframes have the same set of columns? If so, you can just add an additional column to each dataframe and store a key there, then merge all dataframes and save the result into a file. — Michael O., Jun 10 '18 at 17:36
But it would be fine if you added a minimal working example here. — Michael O., Jun 10 '18 at 17:38
Nope the dataframes are of unequal dimensions (both unequal rows and/or unequal columns) — C. Refsgaard, Jun 10 '18 at 19:01

score 2 · Accepted Answer · edited Aug 21 '19 at 15:25

2

Regarding the rule of saving data frames independently and not using SQL solution (or another database format) it could be the following code:

import csv
import pandas as pd 

def saver(dictex):
    for key, val in dictex.items():
        val.to_csv("data_{}.csv".format(str(key)))

    with open("keys.txt", "w") as f: #saving keys to file
        f.write(str(list(dictex.keys())))

def loader():
    """Reading data from keys"""
    with open("keys.txt", "r") as f:
        keys = eval(f.read())

    dictex = {}    
    for key in keys:
        dictex[key] = pd.read_csv("data_{}.csv".format(str(key)))

    return dictex

(...)

dictex = loader()

edited Aug 21 '19 at 15:25

Nilesh Kesar

147
12

answered Jun 10 '18 at 18:51

artona

978
7
11

Hi @artona, thank you for your answer. It doesn't work with the dictionary I need to write to file, because its values are both dataframes and integers. I've posted my question here: https://stackoverflow.com/questions/65098346/writing-dictionary-of-mix-of-integers-and-dataframes-to-file, in case you can help :) – Elsa Dec 01 '20 at 20:55

Writing dictionary of dataframes to file

1 Answers1