I have been trying to read data from a large (~270 MB) text file in Java and store it in an ArrayList of Mats and 3D point arrays in order to do Flann-Based Matching. Although these methods work, it takes too much time (roughly 20-30 minutes) to process and store the data. Is there a way to speed up adding the values to the ArrayList?
My text file in the following format: -1.78894 -0.260657 -5.01526 54 15 2899.69 817.542 14 0 0 1 ... (repeats for 124 more ints)
This is my code using BufferedReader, where mDescriptors is an ArrayList of Mat objects and mPoints3d is a 2D ArrayList of 3D points.
public void loadData() {
// initialize mDescriptors and mPoints3d by iterating through it;
for(int i = 0; i < 256; i++)
{
mDescriptors.add(new Mat());
mPoints3d.add(new ArrayList<Point3>());
}
try {
// open the database data file and read from it
Log.i(TAG, "Opening table.txt");
File fTable = new File(Environment.getExternalStorageDirectory().getPath() + "/GlassProject/table.txt");
BufferedReader br = new BufferedReader(new FileReader(fTable));
String str;
// keep reading data until there is no more data to read
while((str = br.readLine()) != null)
{
String[] sa = str.split(" ");
Point3 xyz = new Point3(Float.valueOf(sa[0]), Float.valueOf(sa[1]), Float.valueOf(sa[2]));
int imgidx = Integer.valueOf(sa[3]);
//int featidx = Integer.valueOf(sa[4]);
// Iterate through the rest of the line to get the descriptor data (128 values)
for(int i = 7; i < sa.length; i++)
{
mDes.put(0,i - 7,Integer.valueOf(sa[i]));
}
mDescriptors.get(imgidx - 1).push_back(mDes); // Index issues happen here
mPoints3d.get(imgidx - 1).add(xyz);
}
Log.v(TAG, "Loaded the Database data!");
}
catch(IOException ex) {
Log.e(TAG, "Error: could not open table");
ex.printStackTrace();
}
}