Downloading large file in ColdFusion using CFHTTP

Question

I'm trying to download a large (600MB) binary file to the server using CFHTTP on Coldfusion 8:

<cfhttp 
  method="get" 
  url="#fileURL#" 
  path="#filePath#" 
  file="#fileName#" 
  timeout="600" 
  getasbinary="auto"
>

It's working fine for smaller files (100something MB) but for the large one's I'm getting the Server 500 error: "java.lang.OutOfMemoryError: Java heap space".

The file is being downloaded from a Dropbox folder - so only available option is to use HTTP GET.

Does anyone have idea how to download it, so it wouldn't kill the server or timeout?

If it's a file in Dropbox, why not just install the Dropbox client on the server, and watch the directory for new/updated files? — Ben Doom, Jan 19 '11 at 22:14
Because we have couple separate clients on the same server - can't install multiple Dropbox clients. But thanks for the idea Ben. — Lucas, Jan 20 '11 at 09:06

score 14 · Accepted Answer · answered Jan 19 '11 at 19:26

You can do this by calling Java from CF code. The buffered input and output stream classees are designed to hold onto chunks of data, rather than the whole thing, avoiding OutOfMemory errors.

getByteArray() is a helper method because there's not way to declare something like byte buf[]=new byte[1024]; in CF directly.

In the example change the source and destination variables.

Example

<cfset source = 'protocol://domain/path/to/file.ext'>
<cfset destination = getDirectoryFromPath(getCurrentTemplatePath()) & listlast(source, "/")>
<cffunction name="getByteArray" access="private" returnType="binary" output="no">
    <cfargument name="size" type="numeric" required="true"/>
    <cfset var emptyByteArray =
        createObject("java", "java.io.ByteArrayOutputStream").init().toByteArray()/>
    <cfset var byteClass = emptyByteArray.getClass().getComponentType()/>
    <cfset var byteArray =
        createObject("java","java.lang.reflect.Array").newInstance(byteClass, arguments.size)/>
    <cfreturn byteArray/>
</cffunction>

<cfscript>
uri = createObject("java", "java.net.URL").init(source);
uis = uri.openStream();
bis = createObject("java", "java.io.BufferedInputStream").init(uis);
fos = createObject("java", "java.io.FileOutputStream").init(destination);
bos = createObject("java", "java.io.BufferedOutputStream").init(fos);
buffer = getByteArray(1024);
len = bis.read(buffer);
while(len > 0) {
    bos.write(buffer,0,len);
    len = bis.read(buffer);
}
bos.close();
bis.close();
fos.close();
uis.close();
</cfscript>

Nice howto, especially for people without Java background (like me). — Sergey Galashyn, Jan 19 '11 at 20:27
Java is great to know with CF because it allows you to work around some of the problems with built-in functionality. — orangepips, Jan 19 '11 at 20:34
What would I need to add to use this to connect through a proxy server? My cf server is Coldfusion9. Thanks. — garyv, Oct 12 '12 at 19:55
@garv: I'm not working with CF these days. I'd suggest asking a new question. — orangepips, Oct 13 '12 at 00:16

score 2 · Answer 2 · answered Jan 19 '11 at 18:13

The problem is is that it's too large. ColdFusion reads the entire thing into memory before writing it to disk.

You'll be better off using some other method to get the file. wget can do http requests from a command-line. That with judicious use of CFEXECUTE is probably a good way to go.

Downloading large file in ColdFusion using CFHTTP

2 Answers2