38

I'm going crazy, I created a file object, so it can be read with ObjectInputStream, and I placed the assets folder. The method works with a file smaller than 1M, and give error with larger files. I read that is a limit of Android platform, but I also know that can be "easily" avoided. Those who have downloaded the game Reging Thunder, for example, can easily see that in their assets folder is a file 18.9M large. This is my code for read 1 object from a ObjecInputStream

File f = File.createTempFile("mytempfile", "dat");
FileOutputStream fos = new FileOutputStream(f);

InputStream is = mc.getAssets().open(path,3);

ObjectInputStream ois=new ObjectInputStream(is);
byte[] data = (byte[]) ois.readObject();
fos.write(data);

fos.flush();
fos.close();
ois.close();
is.close();

now I have an uncompressed file and I can use it without worrying about the error "This file can not be opened as a file descriptor; it is probably compressed"

This function works well with files smaller than 1M, with bigger files return an java.io.IOException on line "ObjectInputStream ois=new ObjectInputStream(is);"

why??

Syco
  • 755
  • 2
  • 13
  • 19

11 Answers11

49

Faced the same issue. I've cut up my 4MB file into 1 MB chunks, and on the first run I join the chunks into a data folder on the phone. As an added bonus, the APK is properly compressed. The chunk files are called 1.db, 2.db, etc. The code goes like this:

File Path = Ctxt.getDir("Data", 0);
File DBFile = new File(Path, "database.db");

if(!DBFile.exists() || DatabaseNeedsUpgrade)  //Need to copy...
    CopyDatabase(Ctxt, DBFile);


static private void CopyDatabase(Context Ctxt, File DBFile) throws IOException
{
    AssetManager assets = Ctxt.getAssets();
    OutputStream outstream = new FileOutputStream(DBFile);
    DBFile.createNewFile();
    byte []b = new byte[1024];
    int i, r;
    String []assetfiles = assets.list("");
    Arrays.sort(assetfiles);
    for(i=1;i<10;i++) //I have definitely less than 10 files; you might have more
    {
        String partname = String.format("%d.db", i);
        if(Arrays.binarySearch(assetfiles, partname) < 0) //No such file in assets - time to quit the loop
            break;
        InputStream instream = assets.open(partname);
        while((r = instream.read(b)) != -1)
            outstream.write(b, 0, r);
        instream.close();
    }
    outstream.close();
}
Seva Alekseyev
  • 55,897
  • 22
  • 151
  • 252
  • 1
    thanks for you answer. But which kind of tool you use to cut your large db to each smaller chunks? – anticafe Nov 23 '10 at 03:19
  • Tried few ready-made freeware ones, then wrote another. :) It's a 20-line piece in straight C. – Seva Alekseyev Nov 23 '10 at 14:56
  • 1
    Thanks Seva.. Loop did not work for me but I did it manually. I had the database of 5.3MB. It saved my time. Thanks again. For Splitting the Database, I used HJSplit tool. – Shraddha May 22 '12 at 13:00
  • Still getting IOException in android 2.2, after splitting database. Can anyone help. I have 6MB database and could not find way to work with that. Thanks – djk Apr 02 '13 at 06:50
  • 2djk: ask a separate question. This is not a forum, the OP's question has been long answered. – Seva Alekseyev Apr 02 '13 at 13:47
  • Is this still relevant for newer versions of Android? I've read that this is only needed for Android versions < 2.3. – smg Jun 08 '17 at 21:57
  • Not really :))) – Seva Alekseyev Jun 09 '17 at 20:25
33

The limitation is on compressed assets. If the asset is uncompressed, the system can memory-map the file data and use the Linux virtual memory paging system to pull in or discard 4K chunks as appropriate. (The "zipalign" tool ensures that uncompressed assets are word-aligned in the file, which means they'll also be aligned in memory when directly mapped.)

If the asset is compressed, the system has to uncompress the entire thing to memory. If you have a 20MB asset, that means 20MB of physical memory is tied up by your application.

Ideally the system would employ some sort of windowed compression, so that only parts need to be present, but that requires some fanciness in the asset API and a compression scheme that works well with random access. Right now APK == Zip with "deflate" compression, so that's not practical.

You can keep your assets uncompressed by giving them a suffix of a file type that doesn't get compressed (e.g. ".png" or ".mp3"). You may also be able to add them manually during the build process with "zip -0" instead of having them bundled up by aapt. This will likely increase the size of your APK.

fadden
  • 48,613
  • 5
  • 104
  • 152
  • I just faced this problem today and could get it working by changing the file name suffix to .mp3 as suggested. .png didn't work though. AssetManager says it can't find the file :( Thank you very much for the suffix tip! – Francisco Junior Mar 22 '11 at 14:40
  • 2
    this [post](http://stackoverflow.com/questions/3177026/problem-while-opening-asset-file-with-the-help-of-content-provider) has a list of such extensions. – Samuel Apr 11 '11 at 00:32
9

Like Seva suggested you can split up your file in chunks. I used this to split up my 4MB file

public static void main(String[] args) throws Exception {  
    String base = "tracks";  
    String ext = ".dat";  
    int split = 1024 * 1024;  
    byte[] buf = new byte[1024];  
    int chunkNo = 1;  
    File inFile = new File(base + ext);  
    FileInputStream fis = new FileInputStream(inFile);  
    while (true) {  
      FileOutputStream fos = new FileOutputStream(new File(base + chunkNo + ext));  
      for (int i = 0; i < split / buf.length; i++) {  
        int read = fis.read(buf);  
        fos.write(buf, 0, read);  
        if (read < buf.length) {  
          fis.close();  
          fos.close();  
          return;  
        }  
      }  
      fos.close();  
      chunkNo++;  
    }  
  }  

If you do not need to combine the files into a single file on the device again, just use this InputStream, which combines them into one on the fly.

import java.io.IOException;  
import java.io.InputStream;  

import android.content.res.AssetManager;  

public class SplitFileInputStream extends InputStream {  

  private String baseName;  
  private String ext;  
  private AssetManager am;  
  private int numberOfChunks;  
  private int currentChunk = 1;  
  private InputStream currentIs = null;  

  public SplitFileInputStream(String baseName, String ext, int numberOfChunks, AssetManager am) throws IOException {  
    this.baseName = baseName;  
    this.am = am;  
    this.numberOfChunks = numberOfChunks;  
    this.ext = ext;  
    currentIs = am.open(baseName + currentChunk + ext, AssetManager.ACCESS_STREAMING);  
  }  

  @Override  
  public int read() throws IOException {  
    int read = currentIs.read();  
    if (read == -1 && currentChunk < numberOfChunks) {  
      currentIs.close();  
      currentIs = am.open(baseName + ++currentChunk + ext, AssetManager.ACCESS_STREAMING);  
      return read();  
    }  
    return read;  
  }  

  @Override  
  public int available() throws IOException {  
    return currentIs.available();  
  }  

  @Override  
  public void close() throws IOException {  
    currentIs.close();  
  }  

  @Override  
  public void mark(int readlimit) {  
    throw new UnsupportedOperationException();  
  }  

  @Override  
  public boolean markSupported() {  
    return false;  
  }  

  @Override  
  public int read(byte[] b, int offset, int length) throws IOException {  
    int read = currentIs.read(b, offset, length);  
    if (read < length && currentChunk < numberOfChunks) {  
      currentIs.close();  
      currentIs = am.open(baseName + ++currentChunk + ext, AssetManager.ACCESS_STREAMING);  
      read += read(b, offset + read, length - read);  
    }  
    return read;  
  }  

  @Override  
  public int read(byte[] b) throws IOException {  
    return read(b, 0, b.length);  
  }  

  @Override  
  public synchronized void reset() throws IOException {  
    if (currentChunk == 1) {  
      currentIs.reset();  
    } else {  
      currentIs.close();  
      currentIs = am.open(baseName + currentChunk + ext, AssetManager.ACCESS_STREAMING);  
      currentChunk = 1;  
    }  
  }  

  @Override  
  public long skip(long n) throws IOException {  
    long skipped = currentIs.skip(n);  
    if (skipped < n && currentChunk < numberOfChunks) {  
      currentIs.close();  
      currentIs = am.open(baseName + ++currentChunk + ext, AssetManager.ACCESS_STREAMING);  
      skipped += skip(n - skipped);  
    }  
    return skipped;  
  }  
}

Usage:
ObjectInputStream ois = new ObjectInputStream(new SplitFileInputStream("mytempfile", ".dat", 4, getAssets()));

Oliver
  • 131
  • 1
  • 5
  • Your method(s) seems to work but in practice I experienced two issues. At first method (main - java - splitter method) I faces issue when file was exactly 1024 * n (n>0) long. And after last bytes was read `int read` has value `-1` and `fos.write` failed. I changed writing mechanism like `if (read > -1) { fos.write(buf, 0, read); }'. Next issue belongs to `SplitFileInputStream` class. Method `public int read(byte[] b)` does not work well. So I changed it analogically like `public int read(byte[] b, int offset, int length)` method. – zmeda Jun 14 '11 at 13:09
  • It looks something like `public int read(byte[] b) throws IOException { int read = currentIs.read(b); if (read < b.length && currentChunk < numberOfChunks) { currentIs.close(); currentIs = am.open(baseName + ++currentChunk + ext, AssetManager.ACCESS_STREAMING); byte[] buffer = new byte[b.length - read]; read += read(buffer); } return read; }` – zmeda Jun 14 '11 at 13:12
7

Change the file extension to .mp3

xarlymg89
  • 2,205
  • 2
  • 24
  • 35
yassine
  • 89
  • 1
  • 1
2

I found another solution, maybe you're interested in it.

In the root of your sources, where you have the build.xml file, you can overwrite the -package-resources target in the custom_rules.xml file, which is used for adding/modifying targets in ant without breaking anything in the standard android app build system.

Just create a file with this content:

<?xml version="1.0" encoding="UTF-8"?>
<project name="yourAppHere" default="help">

    <target name="-package-resources" depends="-crunch">
        <!-- only package resources if *not* a library project -->
        <do-only-if-not-library elseText="Library project: do not package resources..." >
            <aapt executable="${aapt}"
                    command="package"
                    versioncode="${version.code}"
                    versionname="${version.name}"
                    debug="${build.is.packaging.debug}"
                    manifest="${out.manifest.abs.file}"
                    assets="${asset.absolute.dir}"
                    androidjar="${project.target.android.jar}"
                    apkfolder="${out.absolute.dir}"
                    nocrunch="${build.packaging.nocrunch}"
                    resourcefilename="${resource.package.file.name}"
                    resourcefilter="${aapt.resource.filter}"
                    libraryResFolderPathRefid="project.library.res.folder.path"
                    libraryPackagesRefid="project.library.packages"
                    libraryRFileRefid="project.library.bin.r.file.path"
                    previousBuildType="${build.last.target}"
                    buildType="${build.target}"
                    ignoreAssets="${aapt.ignore.assets}">
                <res path="${out.res.absolute.dir}" />
                <res path="${resource.absolute.dir}" />
                <nocompress /> <!-- forces no compression on any files in assets or res/raw -->
                <!-- <nocompress extension="xml" /> forces no compression on specific file extensions in assets and res/raw -->
            </aapt>
        </do-only-if-not-library>
    </target>
</project>
Redhart
  • 99
  • 1
  • 6
Ottavio Campana
  • 3,820
  • 5
  • 28
  • 55
2

I know this is an old question but I thought of a good solution. Why not store the file already pre-zipped in the asset folder. Then since it is already a zip file and therefore compressed it won't need to be compressed again. So if you wanted the file to be compressed to decrease the size of your apk but you don't want to deal with splitting up files I think this is easier.

When you need to read that file off the device just wrap the inputstream in a zipinputstream http://developer.android.com/reference/java/util/zip/ZipInputStream.html

Matt Wolfe
  • 8,585
  • 7
  • 56
  • 75
  • Here is code example how to open zip file instead of gz file https://github.com/jgilfelt/android-sqlite-asset-helper/blob/master/library/src/main/java/com/readystatesoftware/sqliteasset/SQLiteAssetHelper.java#L449 This seems to work with files bigger than 1M. – Greg Dan Dec 11 '17 at 18:49
0

add file extension is mp3.I use mydb.mp3in assets folder and copy .this run without error.show check it.

santosh Kundkar
  • 149
  • 1
  • 3
0

Using GZIP would be another method. you only need to wrap InputStream inside GZIPInputStream.

I used this for a database which size about 3.0 MB and output compress file was about 600KB.

  • For copying DB in firs run, I gzipped my source .db file using GZIP tool.
  • Then renamed it to .jpg in order to avoid more compression (these processes are done before compile APK FILE).
  • Then for reading compressed GZIP file from assetss

and copying it:

private void copydatabase() throws IOException {
        // Open your local db as the input stream
        InputStream myinput = mContext.getAssets().open(DB_NAME_ASSET);
        BufferedInputStream buffStream = new BufferedInputStream(myinput);
        GZIPInputStream zis = new GZIPInputStream(buffStream);

        // Path to the just created empty db
        String outfilename = DB_PATH + DB_NAME;

        // Open the empty db as the output stream
        OutputStream myoutput = new FileOutputStream(outfilename);


        // transfer byte to inputfile to outputfile
        byte[] buffer = new byte[1024];
        int length;
        while ((length = zis.read(buffer)) > 0) {
            myoutput.write(buffer, 0, length);
        }

        // Close the streams
        myoutput.flush();
        myoutput.close();
        zis.close();
        buffStream.close();
        myinput.close();
    }
VSB
  • 7,800
  • 11
  • 59
  • 121
0

Instead of assets folder, I have put my large files in the raw folder. It works for me.

Redhart
  • 99
  • 1
  • 6
the100rabh
  • 4,009
  • 4
  • 29
  • 39
  • I got always the same problem: "Data exceeds UNCOMPRESS_DATA_MAX (1314625 vs 1048576)" thank you anyway – Syco May 19 '10 at 13:29
  • I am using Eclipse and my application Navkar Matra http://www.androlib.com/android.application.com-app-navkarmantra-Cxin.aspx works wonderfully with a 2 MB file in it. – the100rabh May 21 '10 at 06:57
0

set database to multiple part with a program e.g "Win Hex", you can download from Link

and continue Load files bigger than 1M from assets folder

Community
  • 1
  • 1
Ehsan Jelodar
  • 1,394
  • 18
  • 24
-1

I use NetBeans to build the package and I not found how to change the settings of AAPT. I have not tried the png, but the mp3 are compressed. I can compile the package and then enter the assets folder with the parameter -0? what would be the correct command to use?

Syco
  • 1