I'd like to use IndexedDB to process a lot of data. Too much data to fit in memory. To do this, I would like to use Firefox's IndexedDB persistent storage, which allows me to store more than 2GB worth of data (Firefox apparently has a limit of 2GB imposed on non-persistent storage).
However, I've run into an issue. Firefox does not appear to be imposing a limit on the amount of data I can store in persistent storage. In fact, if I leave the following sample running, it will apparently run until the disk is full!
Sample (Online) (Must be run in Firefox!):
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8" />
<title>Firefox IndexedDB Limit Test</title>
</head>
<body>
<script>
(function() {
'use strict';
var IDBReq = indexedDB.open('testdb', {
version: 1,
storage: 'persistent'
});
IDBReq.onupgradeneeded = function() {
this.result.createObjectStore('data');
};
var logmsg;
IDBReq.onsuccess = function() {
var DB = this.result;
var size = 0;
var next = function(i) {
var data = new Uint8Array(0xFFFF);
crypto.getRandomValues(data);
size += data.length;
logmsg = 'size: ' + size + 'b ' + (size / (1024 * 1024 * 1024)) + 'gb';
var store = DB.transaction(['data'], 'readwrite').objectStore('data');
var storeReq = store.add(data, 'data-' + i);
storeReq.onsuccess = function() {
next(i + 1);
};
storeReq.onerror = function() {
console.log('storeReq error');
console.log(this.error);
};
};
next(1);
};
setInterval(function() {
if (logmsg) {
console.log(logmsg);
logmsg = null;
}
}, 1000);
})();
</script>
</body>
</html>
For obvious reasons, filling up a user's drive is not ideal. If the user does not have enough free disk space, it would be better not to try to run it at all, or at-least stop when more-than X% full.
Strangely, according to MDN, it seems like the browser should be imposing a limit on the amount of data stored:
The maximum browser storage space is dynamic — it is based on your hard drive size. The global limit is calculated as 50% of free disk space. In Firefox, an internal browser tool called the Quota Manager keeps track of how much disk space each origin is using up, and deletes data if necessary.
So if your hard drive is 500GB, then the total storage for a browser is 250GB. If this is exceeded, a process called origin eviction comes into play, deleting entire origin's worth of data until the storage amount goes under the limit again. There is no trimming effect put in place, to delete parts of origins — deleting one database of an origin could cause problems with inconsistency.
There's also another limit called group limit — this is defined as 20% of the global limit. Each origin is part of a group (group of origins). There's one group for each eTLD+1 domain.
However, my testing never threw any errors while it filled up the drive.
Now my question is, is there any way to ensure putting a lot of data into Firefox's IndexedDB persistent storage will not fill up a user's drive, and make them very, very angry?
