Retrieve all pickled data in a folder

Question

I'm new to Python.

I have matrices saved in a specific folder as .pkl. I would like to retrieve pickled data with a for loop over this folder. I also would like the new variable to have the same name as the file from which I'm reading the data.

In the end, what I want is to generalize the operation below for all data in the folder Output

with open('../Output/MatrixA.pkl', 'rb') as fin:
     MatrixA = pickle.load(fin)

score 1 · Accepted Answer · answered Sep 24 '20 at 14:54

Use of sys.modules and setattr will work like charm:

import pickle
from sys import modules
from os import listdir

main_script = modules['__main__']

for file in [path for path in listdir() if path.endswith('pk1')]:
    with open(file, "rb") as fp:
        setattr(main_script, fp.name.split('.')[0], pickle.load(fp))


print(a, b, c, d, e, sep='\n')

code used to generate sample files - a~e.pk1:

import pickle

for i, n in enumerate('abcde'):
    with open(f"{n}.pk1", "wb") as fp:
        pickle.dump([*iter(range(i))], fp)

output:

[]
[0]
[0, 1]
[0, 1, 2]
[0, 1, 2, 3]

But would be nice if you convert those pickles to json as pickled file are dependent to python versions, and are potential threat.

Use this to convert all .pk1 in directory:

import json
import pickle
from os import listdir


for file in [path for path in listdir() if path.endswith('pk1')]:
    with open(file, "rb") as fp:
        with open(file.split('.')[0] + ".json", "wt") as fp_json:
            json.dump(pickle.load(fp), fp_json)

Retrieve all pickled data in a folder

1 Answers1