Convert SAS file (sas7bdat) to a flat file using R/Python without memory constraints

Question

I need to convert a SAS file into a flat file. These files can be pretty big that can go up to 60 GB in size. I wrote a script in R (below) but it reads the entire data and then exports to a CSV file. Is there a way I could convert such big files without any memory constraints. I am open to using either R or Python. I working on a machine that has 16 GB RAM.

args = commandArgs(trailingOnly=TRUE)

library(sas7bdat)

MyData <-  read.sas7bdat(file = args[1])
write.csv(MyData, file = args[2], row.names = FALSE)

I don't have SAS software. All i got is a bunch of SAS files :/ — Ramaranjan Ruj, Jul 03 '18 at 14:52
What are you going to do with that csv file? If you're working with R or Python, you'll have to read it into memory anyway. — Hong Ooi, Jul 03 '18 at 15:00
`heaven::read_sas` can read by column, so you could read column by column, and write those as different csvs, then you could work with only the columns you're interested in, or probably better, put all this data in a SQL database. — Moody_Mudskipper, Jul 03 '18 at 15:08

s3nh · Accepted Answer · 2019-07-09T09:48:44.863

1

In my opinion, you can aquire solution using pandas.read_sas and chunksize arg:

Pandas read sas docs

For example, iterate through 10k observations:

import pandas as pd

chunk_size =  10**4
for chunk in pd.read_sas(filename, chunksize=chunksize):
    process(chunk)

where process() are instructions which you want to provide (append, etc.).

edited Jul 09 '19 at 09:48

answered Jul 09 '19 at 06:53

s3nh

482
1
8

Convert SAS file (sas7bdat) to a flat file using R/Python without memory constraints

1 Answers1