PHP creating large JSON string

Question

I'm working on an application where the user should see a list with charts. The data for the charts should be fetched from a database (which currently counts about 785 rows) and then get sorted to form a valid JSON string. Right not I try to do it like this

while($row = $res->fetch_assoc()) {
    if(count($appData) == 0 ){
        $appData[] = array(
            "name" => $row["name"],
            "date" => array($row["date"]),
            "android" => array($row["android_count"]),
            "ios" => array($row["apple_count"])
        );
    }else {
        for($i = 0; $i < count($appData); $i++) {
            if($appData[$i]["name"] == $row["name"]){
                $appData[$i]["date"][] = $row["date"];
                $appData[$i]["android"][] = $row["android_count"];
                $appData[$i]["ios"][] = $row["apple_count"];
            }else {
                $appData[] = array(
                    "name" => $row["name"],
                    "date" => array($row["date"]),
                    "android" => array($row["android_count"]),
                    "ios" => array($row["apple_count"])
                );
            }
        }
    }
}
echo json_encode($appData);

When I try to run the code it will give an "Fatal error: Allowed memory size of 536870912 bytes exhausted (tried to allocate 71 bytes)" error. I've tried to increase the maximum allowed memory, just to see what would happen, but I got the same result.

Is there any way to avoid having to do so many loops? Or should I approach this from a completely different way, and if so which one?

The end result should look something like this

[{"name":"Some name", "date":["2016-05-09", "2016-05-10", "2016-05-11"], "android":["3", "1", "8"], "ios":["4", "7", "5"]},...]

All help would be appreciated!

Possible duplicate of [Allowed memory size of 33554432 bytes exhausted (tried to allocate 43148176 bytes) in php](http://stackoverflow.com/questions/415801/allowed-memory-size-of-33554432-bytes-exhausted-tried-to-allocate-43148176-byte) — Hamza Zafeer, May 17 '16 at 15:00

JesusTheHun · Answer 1 · 2016-05-17T14:57:04.627

Your problem is not the number of loops, but the size of your $appData array and the memory_limit value of your php configuration.

If you cannot shrink the size of the data you pass on, you must increase the memory_limit value. But be careful while increasing this value, since it's the value for each running php script your server will execute. I would recommand to paginate or send to the output buffer on each loop.

If you need exemple of code, just ask for it.

Pagination means your javascript page will call X times the PHP script to retrieve N rows each times, until the PHP scripts does not allow it anymore. Therefore you have to return an array such as :

return array(
    'nextPage' => 2, // More data available on this page
    'data' => $json
);

// Or

return array(
    'nextPage' => null, // No more data available
    'data' => $json
);

Or send to the output buffer on each loop and release memory :

$first = true;
echo '[';

while($row = $res->fetch_assoc()) {

    if(!$first) {
        echo ',';
    } else {
        $first = false;
    }

    // some logic
    $row_data = array(...);

    echo json_encode($row_data);
}

echo ']';

This way you do not stack all the data in php's variables.

If you have an example laying around, that would be great! – Bas Pauw May 17 '16 at 14:52 — Bas Pauw, May 17 '16 at 14:52

Dan Wyman · Accepted Answer · 2016-05-17T17:29:52.623

The memory problem is in the "for" loop. It can be adding a bunch of items to $appData per loop, instead of "just one if there is no matching name." For example, if $appData already has 100 items in it, and $row['name'] matches the last item in $appData, then 99 items will be added to $appData before the last item in $appData is updated. I bet the current code is generating an $appData with a lot more than 785 items.

To fix the memory issue, change the "for" loop to something like this:

    $matchFound = false;
    for($i = 0; $i < count($appData); $i++) {
        if($appData[$i]["name"] == $row["name"]){
            $appData[$i]["date"][] = $row["date"];
            $appData[$i]["android"][] = $row["android_count"];
            $appData[$i]["ios"][] = $row["apple_count"];
            $matchFound = true;
            break;
        }
    }
    if (!$matchFound) {
        $appData[] = array(
            "name" => $row["name"],
            "date" => array($row["date"]),
            "android" => array($row["android_count"]),
            "ios" => array($row["apple_count"])
        );
    }

On an efficiency note, using an associative array as suggested by maximkou will be a big speedup.

score 0 · Answer 3 · answered May 17 '16 at 15:05

Index your array by $row['name']. This strong simplify your code. Arrays in php allocate many memory, so encode nested data by row. Or try to use SplFixedArray if you know result array size.

Try this:

while($row = $res->fetch_assoc()) {
    $appData[ $row["name"] ] = json_encode(array(
        "name" => $row["name"],
        "date" => array($row["date"]),
        "android" => array($row["android_count"]),
        "ios" => array($row["apple_count"])
    ));
}
echo "[".implode(',', $appData)."]";

score 0 · Answer 4 · answered May 17 '16 at 15:25

This should create the exact same results (tho haven't tested), and uses a map array and array_key_exists() to avoid extra loops. This does it in a single loop.

$nameMap = array(); // hold name and keys
while($row = $res->fetch_assoc()){
    $key = array_key_exists($row['name'], $nameMap) ? $nameMap[$row['name']] : count($appData);
    $nameMap[$row['name']] = $key;
    if(empty($appData[$key])) 
        $appData[$key] = array("name"=>$row['name'], "date"=>array(), "android"=>array(), "ios"=>array());
    $appData[$key]['date'][] = $row['date'];
    $appData[$key]['android'][] = $row['android'];
    $appData[$key]['ios'][] = $row['ios'];
}

PHP creating large JSON string

4 Answers4