The issue:
I'm working with PHP, cURL and a public API to fetch json strings. These json strings are formatted like this (simplified, average size is around 50-60 kB):
{
"data": {},
"previous": "url",
"next": "url"
}
What am trying to do is fetch all the json strings starting from the first one by checking for the "next" attribute. So I have a while loop and as long as there's a "next" attribute, I fetch the next URL.
The problem is sometimes, randomly the loop stops before the end and I cannot figure out why after many tests.
I say randomly because sometimes the loop goes through to the end and no problem occurs. Sometimes it crashes after N loops.
And so far I couldn't extract any information to help me debug it.
I'm using PHP 7.3.0 and launching my code from the CLI.
What I tried so far:
Check the headers:
No headers are returned. Nothing at all.
Use curl_errno() and curl_error():
I tried the following code right after executing the request (curl_exec($ch)) but it never triggers.
if(curl_errno($ch)) {
echo 'curl error ' . curl_error($ch) . PHP_EOL;
echo 'response received from curl error :' . PHP_EOL;
var_dump($response); // the json string I should get from the server.
}
Check if the response came back null:
if(is_null($response))
or if my json string has an error:
if(!json_last_error() == JSON_ERROR_NONE)
Though I think it's useless because it will never be valid if the cURL response is null or empty. When this code triggers, the json error code is 3 (JSON_ERROR_CTRL_CHAR)
The problematic code:
function apiCall($url) {
...
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch);
}
$inc = 0;
$url = 'https://api.example.com/' . $id;
$jsonString = apiCall($url);
if(!is_null($jsonString)) {
file_put_contents('pathToDirectory/' . $id + $inc, $jsonString);
$nextUrl = getNextUrl($jsonString);
while ($nextUrl) {
$jsonString = apiCall($url . '?page=' . $nextUrl);
if(!is_null($jsonString)) {
$inc++;
file_put_contents('pathToDirectory/' . $id + $inc, $jsonString);
$nextUrl = getNextUrl($jsonString);
}
}
}
What I'm expecting my code to do:
Not stop randomly, or at least give me a clear error code.