1

Here is my code that throws java.lang.OutOfMemoryError: Java heap space:

StringBuilder sb = new StringBuilder();
    for (int i = 0; i< ranks.size(); i++) {
        Map<String, String> map = ranks.get(i);
        sb.append("SELECT ");
        int iter = 0;
        for (String key : map.keySet()) {
            if (key.equals("ware_md5")) {
                sb.append("'").append(map.get(key)).append(" ' AS ").append(key);
            }
            else {
                sb.append(map.get(key)).append(" AS ").append(key);
            }
            if (iter < map.keySet().size() - 1) {
                sb.append(",");
            }
            iter++;
        }
        if (i < ranks.size() - 1) {
            sb.append(" UNION ALL ");
        }

    }
    return sb.toString();

'ranks' ArrayList contains about 30 000 values

Is there any way to rewrite this peace of code to avoid this error ?

Bogdan Timofeev
  • 908
  • 3
  • 10
  • 30
  • What are you going to do with a string this big? Maybe you should write directly to a file? What is the size of the `map.keySet()`? – 9000 Jun 06 '17 at 20:46
  • Here I am trying ti generate some type of SQL request, size of map.keySet() is about 220 records – Bogdan Timofeev Jun 06 '17 at 20:50
  • 1
    There are number of issues that you may run into even if you resolved memory issue. 1 big query that the db might not be able to handle. Large number of unions which performance will be a big hit on db server as well as the number of rows returned. I would suggest do-less-but-do-more approach – Minh Kieu Jun 06 '17 at 20:53
  • 1
    I'm afraid an SQL _request_ size is limited in most databases by some reasonably small amount, like 1MB in MySQL (1GB in Postgres, though). Even if you can send a query this big, chances are executing it will face certain difficulties. Are you sure you want to query 30k tables in exactly one query? Chances are that running several dozens or hundreds of smaller queries in parallel, using a `ThreadPoolExecutor`, may result in better overall performance. You can easily implement the effect of `union all` on the receiving side. – 9000 Jun 06 '17 at 20:59

2 Answers2

4

Launch the JVM with a larger heap. java -Xmx256M as an example, for an example with a 256MB heap.

user3745362
  • 229
  • 1
  • 5
1
  • More memory is the easiest solution, already given
  • new StringBuilder(100_000) - at least faster
  • AS key is not needed but for the first union.
  • Note that SortedMap is what you need: the keys always in the same order.
  • consume ranks.get(i)
Joop Eggen
  • 96,344
  • 7
  • 73
  • 121