Decode Stream to CSV in Python by Byte (Translate from C# code)

Question

I am trying to consume a streamed response in Python from a soap API, and output a CSV file. The response outputs a string coded in base 64, which I do not know what to do with. Also the api documentation says that the response must be read to a destination buffer-by-buffer.

Here is the C# code was provided by the api's documentation:

byte[] buffer = new byte[4000];
bool endOfStream = false;
int bytesRead = 0;
using (FileStream localFileStream = new FileStream(destinationPath, FileMode.Create, FileAccess.Write))
{
   using (Stream remoteStream = client.DownloadFile(jobId))
   {
     while (!endOfStream)
     {
         bytesRead = remoteStream.Read(buffer, 0, buffer.Length);
         if (bytesRead > 0)
         {
              localFileStream.Write(buffer, 0, bytesRead);
              totalBytes += bytesRead;
         }
         else
         {
              endOfStream = true;
         }
      }
   }
}

I have tried many different things to get this stream to a readable csv file, but non have worked.

with open('test.csv', 'w') as f: f.write(FileString)

Returns a csv with the base64 string spread over multiple lines

Here is my latest attempt:

with open('csvfile13.csv', 'wb') as csvfile:
          FileString = client.service.DownloadFile(yyy.JobId, False)
          stream = io.BytesIO(str(FileString))
          with open(stream,"rt",4000) as readstream:
             csvfile.write(readstream)

This produces the error:

TypeError: coercing to Unicode: need string or buffer, _io.BytesIO

Any help would be greatly appreciated, even if it is just to point me in the right direction. I will be ensure to award the points to whoever is the most helpful, even if I do not completely solve the issue!

I have asked several questions similar to this one, but I have yet to find an answer that works completely: What is the Python equivalent to FileStream in C#?

Write Streamed Response(file-like object) to CSV file Byte by Byte in Python

How to replicate C# 'byte' and 'Write' in Python

Let me know if you need further clarification!

Update: I have tried print(base64.b64decode(str(FileString)))

This gives me a page full of webdings like

]�P�O�J��Y��KW �

I have also tried

for data in client.service.DownloadFile(yyy.JobId, False):
    print data

But this just loops through the output character by characater like any other string.

I have also managed to get a long string of bytes like \xbc\x97_D\xfb(not actual bytes, just similar format) by decoding the entire string, but I do not know how to make this readable.

Edit: Corrected the output of the sample python, added more example code, formatting

Answer 1

It sounds like you need to use the base64 module to decode the downloaded data.

It might be as simple as:

with open(destinationPath, 'w') as localFile:
    remoteFile = client.service.DownloadFile(yyy.JobId, False)
    remoteData = str(remoteFile).decode('base64')
    localFile.write(remoteData)

I suggest you break the problem down and determine what data you have at each stage. For example what exactly are you getting back from client.service.DownloadFile?

Decoding your sample downloaded data (given in the comments):

'UEsYAItH7brgsgPutAG\AoAYYAYa='.decode('base64')

gives

'PK\x18\x00\x8bG\xed\xba\xe0\xb2\x03\xee\xb4\x01\x80\xa0\x06\x18\x01\x86'

This looks suspiciously like a ZIP file header. I suggest you rename the file .zip and open it as such to investigate.

If remoteData is a ZIP something like the following should extract and write your CSV.

import io
import zipfile

remoteFile = client.service.DownloadFile(yyy.JobId, False)
remoteData = str(remoteFile).decode('base64')

zipStream = io.BytesIO(remoteData)
z = zipfile.ZipFile(zipStream, 'r')
csvData = z.read(z.infolist()[0])

with open(destinationPath, 'w') as localFile:
    localFile.write(csvData)

Note: BASE64 can have some variations regarding padding and alternate character mapping but once you can see the data it should be reasonably clear what you need. Of course carefully read the documentation on your SOAP interface.

Answer 2

Are you sure FileString is a Base64 string? Based on the source code here, suds.sax.text.Text is a subclass of Unicode. You can write this to a file as you would a normal string but whatever you use to read the data from the file may corrupt it unless it's UTF-8-encoded.

You can try writing your Text object to a UTF-8-encoded file using io.open:

import io
with io.open('/path/to/my/file.txt', 'w', encoding='utf_8') as f:
    f.write(FileString)

Bear in mind, your console or text editor may have trouble displaying non-ASCII characters but that doesn't mean they're not encoded properly. Another way to inspect them is to open the file back up in the Python interactive shell:

import io
with io.open('/path/to/my/file.txt', 'r', encoding='utf_8') as f:
    next(f) # displays the representation of the first line of the file as a Unicode object

In Python 3, you can even use the built-in csv to parse the file, however in Python 2, you'll need to pip install backports.csv because the built-in module doesn't work with Unicode objects:

from backports import csv
import io
with io.open('/path/to/my/file.txt', 'r', encoding='utf_8') as f:
    r = csv.reader(f)
    next(r) # displays the representation of the first line of the file as a list of Unicode objects (each value separated)