Getting OutOfMemory Exception while converting tif file into base64 encoding

Question

We have created console application in C#, where will read multi-page tif/tiff files, splitting pagewise and then convert into base64 encoding to upload the file in other target application (as this accepts only base64 encoding to upload the documents) and we are getting this Out of Memory Exception whenever file size exceeded more than 500 MB Exception of type 'System.OutOfMemoryException' was thrown

Exception at System.Convert.ToBase64String(Byte[] inArray, Int32 offset, Int32 length, Base64FormattingOptions options)
   at System.Convert.ToBase64String(Byte[] inArray)

Code Snippet:

Byte[] bytes = File.ReadAllBytes(filepath);
String base64stringofdocument = Convert.ToBase64String(bytes);

Above filepath refers > absolute path of the file

You will probably have to do some [memory usage analysis with Visual Studio](https://docs.microsoft.com/en-us/visualstudio/profiling/memory-usage?view=vs-2019) to find the root cause of this. — Matthew Watson, Aug 07 '20 at 14:02
A single string object can't exceed 2^31-1 chars in length in .NET. Base64 encoding takes up at least 4/3 space relative to the input size. That gives you an upper theoretical file size boundary of ~805MB, although practically it's probably lower. You probably want to compress the input before converting to base64 — Mathias R. Jessen, Aug 07 '20 at 14:06
Have you considered streaming the data instead of reading the whole file into memory? — Charleh, Aug 07 '20 at 14:07
Project > Properties > Build tab, untick the "Prefer 32-bit" checkbox. You don't prefer it. — Hans Passant, Aug 07 '20 at 14:57
@Charleh Just refer above code snippet, converting into byte array from the file before passing into base64 encoding. How to improve this ? Pls help us — Sheik, Aug 07 '20 at 15:37
Is the app x86 or x64? Cosider pure x64 build to expand a memory limit. But definetly `Stream`ing is solution - [this answer](https://stackoverflow.com/a/57820500/12888024) may help. — aepot, Aug 07 '20 at 19:31

score 1 · Answer 1 · answered Aug 07 '20 at 14:33

1

There is overhead involved in working with strings. For very large amounts of data, you are better off working with arrays or streams. In this case, you can start by rewriting your code to use Convert.ToBase64CharArray. So your code would change to something like this:

Byte[] bytes = File.ReadAllBytes(filePath);
// Compute the number of Base64 converted characters required (must be divisible by 4)
long charsRequired = (long)((4.0d/3.0d) * bytes.Length);
if (charsRequired % 4 != 0) {
    charsRequired += 4 - charsRequired % 4;
}
// Allocate buffer for characters, and write converted data into the array
Char[] chars = new Char[charsRequired];
Convert.ToBase64CharArray(bytes, 0, bytes.Length, chars, 0);

Then you can upload the chars array to your target application.

answered Aug 07 '20 at 14:33

Jack A.

3,145
1
16
24

Hi Jack, Tried with the above suggestion but if convert char array again into string object, the same issue might occur as the target application accepts string object only for uploading a document. Is there any better way to handle large files. Pls share code snippet. – Sheik Aug 07 '20 at 15:02
@Sheik It seems unlikely that your target application can only accept .NET strings. If you edit your post to include the code used to upload to your target application, it may be possible to comment further. – Jack A. Aug 07 '20 at 15:11
After the above code lines, We will just pass encoded base64 string into Webservice call of the target application. Nothing much after this step. – Sheik Aug 07 '20 at 15:23

Getting OutOfMemory Exception while converting tif file into base64 encoding

1 Answers1