0

I am unable to store the csv files in a hashmap of hashmap due to OutOfMemoryError.

CSV file format: word,count

Outer Hashmap: HashMap> - For storing language code and hashmap

Inner Hashmap: HashMap - For storing word and count

I am trying to loop through a list of .csv files(51) of total memory size 145mb in Tomcat server, Eclipse. But I'm getting OutOfMemoryError. Is there any API to convert csv to hashmap in java instead of iterating the csv file line by line?

private static HashMap<String, HashMap<String, Long>> wordCountAllLang = new HashMap<String, HashMap<String, Long>>();
    public static void fetchWordCount(){
        HashMap<String, Long> wordCountMap;
        String language, line;
        String[] lineArr;
        BufferedReader in = null;
        try{
            final File folder = new File(ProofingConstants.DICTIONARY.WORD_COUNT_DATA_FOLDER_PATH);
            for (File fileEntry : folder.listFiles()) {
                language = fileEntry.getName().replace(".csv", ""); //No i18n
                LOGGER.log(Level.WARNING, "Language:###"+language);
                if (!isDictionaryLanguage(language)){
                    continue;
                }
                //Reading the file contents and putting it in the hash map
                wordCountMap = new HashMap<String, Long>();
                try{
                    in = new BufferedReader(new FileReader(fileEntry));
                    while ((line = in.readLine()) != null) {
                        lineArr = line.split(",");
                        wordCountMap.put(lineArr[0], Long.parseLong(lineArr[1]));
                    }

                }
                catch(Exception e) {
                    LOGGER.log(Level.WARNING, "SEVERE_ERROR: Exception while reading csv file data into hashmap", e);
                }
                finally{
                    in.close();
                }

                //Storing the hashmap in another hash map using language as its key
                wordCountAllLang.put(language, wordCountMap);
                LOGGER.log(Level.WARNING, "Language completed");
            }
        }
        catch(Exception e) {
            LOGGER.log(Level.WARNING, "SEVERE_ERROR: Exception while getting word count from csv file", e);
        }
    }

"exception: OutOfMemoryError"

prasad
  • 1
  • Hi, prasad; try to increase JVM memory using -Xsmax – stuck Nov 04 '19 at 05:56
  • Possible duplicate of [Read large CSV in java](https://stackoverflow.com/questions/20043181/read-large-csv-in-java) – Sudhir Ojha Nov 04 '19 at 05:56
  • @stuck it's `-Xmx`. You can also tune the `HashMap` constructor parameters (loadfactor, initial capacity) to make it use less memory. – Kayaman Nov 04 '19 at 05:59

0 Answers0