8

How would you emulate a memory mapped file in FirefoxOS, Tizen or any other mobile pure-JS solution?

The use case is for a mobile browser and you need lots of data which does not fit in the RAM or you don't want to waste RAM for it yet and prefer to lazy load it.

The only thing I found is IndexedDB or what can I do about it? Any better tricks or APIs?

Hmmh it looks like Web SQL Database could be also a solution on Android, Tizen or iOS. But Firefox does not support it (?)

Update: I'm asking because of some experiments

Karussell
  • 16,303
  • 14
  • 88
  • 188

1 Answers1

3

First thing first, Web SQL won't be ever standardised as explained in the specification, so it should be considered only for WebKit/Blink-based browsers.

There is an awesome overview of offline storage options in this queston, even though the map tiles are considered in that question, I think it is still relevant to your use case.

I believe you are on the right track with IndexedDB for the graph data. On a high level it is a key-value asynchronous object store (see the Basic Concepts document). For your use case, you could index graph nodes in an object store. There is, for example, LevelGraph library which stores graph data in IndexedDB, though it is built for Semantic Web triples. HeliosJS is also worth mentioning, though it is an in-memory graph database.

Edit: Current API for IndexedDB is asynchronous. There is synchronous API drafted in the spec, which could be used only in web workers. Unfortunately no engine currently implements this feature. There is a pending patch for Gecko, but I did not find any plans for Blink or WebKit, so it is not a meaningful option now.

It is possible to access raw files through Web APIs. You could use XHR2 to load a (local) file as a binary Blob. Unfortunately XHR2 is mostly designed for streaming files and not for a random access, though you could split-up the data into multiple files and request them on demand, but that may be slow. The direct access to files is currently quite limited, FileList and createObjectURL are primarily used for direct file user input (through Drag and Drop or file input field), FileSystem API was recently killed, and the DeviceStorage is non-standard and privileged (Firefox OS-specific). You can also store files in IndexedDB, which is described for FileHandle API. However, once you manage to get access to the raw File object, you can use the Blob.slice method to load chunks of the file – there is a great example of reading file chunks via upload form. You may also want to look at jDataView library & friends, which eases handling of binary data through the more efficient ArrayBuffer.

Edit: As for the synchronous API, localStorage (aka DOM Storage) could be considered too. It is also a key-value storage, but much much simpler and more limited than IndexedDB:

  • Size of the storage is limited, usually to 5 MB
  • Only one localStorage per domain/application (you can have multiple named object stores in IndexedDB).
  • Only strings can be stored.

In general, localStorage is useful cookies replacement, but it is not really useful for storing large offline data.

So to sum it up:

  • IndexedDB is the easiest and widely available option, though it may be slow, inefficient or hit memory limits with very large data; also, only asynchronous API is currenlty possible.
  • Raw file access is hard to obtain without user interaction and the APIs are unstable and non-standard.

In the end, you can combine both approaches, two options come in mind:

  • Use XHR2 to parse the large file in chunks and store the parsed nodes into IndexedDB
  • Store the large file into IndexedDB (via XHR), use FileHandle.getFile to load the File object and Blob.slice to read its content.

In all cases, you can (should) use Web Workers to handle data manipulation and calculations in the background.

Anyway, GraphHopper looks great, we really lack such non-trivial offline applications for Firefox OS, so good luck!

Community
  • 1
  • 1
jnv
  • 657
  • 9
  • 15
  • Wow, thanks for this in-depth answer. I'll have a look at all your suggestions in the next days! – Karussell May 22 '14 at 09:12
  • @jnv, the worst thing with such APIs is their asynchronousity, while Java uses synchronous API, and so does GraphHopper. And we have no chance to overwrite either GraphHopper or TeaVM to use continuation-passing style or something else. For GraphHopper it will give overcomplicated Java code, for TeaVM it will give a great performance loss, due to most of the methods would be rewritten in CPS. So we are looking for the way to run storage APIs synchronously in a web worker. But it seems that such thing is impossible for now. – Alexey Andreev May 22 '14 at 10:09
  • @jnv I think your answer is a possible way to go (still I'll wait a bit before accepting it maybe other people have more ideas :)) – Karussell May 22 '14 at 14:46
  • @AlexeyAndreev yes that is the key limitation when going from Java to JS as in Java you currently don't use async APIs. But TeaVM could emulate sync access via async+polling. I know this sounds creepy but combined with a WeakMap (see docs: loosing keys if GC sees it is unused) and the Blob.slice (or via jDataView) technique from @ jnv this could work. The critical variable will be the size of one such slice (or 'segment') because if it is too large then too much mem is allocated (+initial waiting is too long) and if it is too small then too many such polling operations can occur – Karussell May 22 '14 at 14:51
  • 1
    I understand, the async nature of JS is a real problem here. @AlexeyAndreev: there is a sync API for IDB planned, but not implemented anywhere: https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API#Synchronous_API – so the Blob.slice seems like the way to go, since you need only a few wrappers for async code. Also I recommend to take a look how existing transpilers handle sync-to-async flow: https://github.com/jashkenas/coffeescript/wiki/List-of-languages-that-compile-to-JS#synchronous-to-asynchronous-javascript-compilers-cps – jnv May 22 '14 at 15:27
  • I think there is no way to go further without sync API. It is just impossible to wrap async API with "few wrappers" and make it sync. How soon the sync API will be implemented? Is there a roadmap? – Alexey Andreev May 22 '14 at 15:43
  • @AlexeyAndreev I have updated the answer. Unfortunately sync API is not a viable alternative now. Anyway, even though I understand your issues with transpilation, the requirements for synchronous API were never raised in the question in the first place. – jnv May 24 '14 at 13:50