compress the text text file full of integer[python]

Asked Nov 29 '20 at 17:29

Active Nov 29 '20 at 18:47

Viewed 39 times

My python code generates a list of list of integers like this [[1, 311 ,8], [5, 3, 23]]. however it has thousands of hundreds of integer and i have to run the code a hundred times. I have to store these results.

I keep them in a file where each list of integers is space separated and takes one line with following code:

with open(courpFile, 'a+', newline='') as myfile:
     wr = csv.writer(myfile, delimiter=' ')
     for simluate in corpus:
          for walk in simluate:
              wr.writerow(walk)

I have problem to keep these files as they take Gigabytes of space and I do not have. How can I compress them and extract them as needed any other recommendation can be good.

edited Nov 29 '20 at 18:47

asked Nov 29 '20 at 17:29

Mohsen Fazaeli

1

If your data size is large I would suggest using `pyspark` and store data in `parquet` format – Kuldeep Singh Sidhu Nov 29 '20 at 17:31
1

Separate the files into batches is one approach. For every `10,000` newlines for example. – etch_45 Nov 29 '20 at 17:33
@etch_45 there is nothing wrong with scale the problems is keeping file in asci format and uncompressed. wastes lots of disk. – Mohsen Fazaeli Nov 29 '20 at 18:49

compress the text text file full of integer[python]

0 Answers0