How could I allocate a variable length command output?

Question

I have a bash script which prints a single huge line of variable output. All the examples I've seen uses a fixed buffer of 1024 bytes or so, in order to read line by line.

#include <stdio.h>
#include <stdlib.h>

int main( int argc, char *argv[] ) {

  FILE *fp;
  char path[1024];

  /* Open the command for reading. */
  fp = popen("bash /home/ouhma/myscript.sh", "r");
  if (fp == NULL) {
    printf("Failed to run command\n" );
    exit(1);
  }

  /* Read the output a line at a time - output it. */
  while (fgets(path, sizeof(path)-1, fp) != NULL) {
    printf("%s", path);
  }

  /* close */
  pclose(fp);

  return 0;
}

Link reference: C: Run a System Command and Get Output?

But what if I don't know if the output line has a length even bigger of 1024 bytes? How could I handle it through reading with popen() command?

You can use POSIX [`getline()`](http://pubs.opengroup.org/onlinepubs/9699919799/functions/getline.html) to read as long a line as possible subject only to memory constraints, or you can repeatedly call `fgets()` when you haven't yet received a newline, adding a buffer that you dynamically allocate as needed. Neither is very complex, though it is most certainly easier to use `getline()` than to have to do it yourself. — Jonathan Leffler, Oct 28 '16 at 07:06
Note that if a line is longer than 1024, your current code will continue to read the rest of the line in the next fgets() call(s). Thus your current code which simply prints out the text will work without any issues. (however if you want to store an entire line inside your program and manipulate that line as a while - you'll need to look at the answer from alk) — nos, Oct 28 '16 at 08:52

alk · Accepted Answer · 2016-10-28T09:19:55.323

But what if I don't know if the output line has a length even bigger of 1024 bytes

You then need to handle the storage for incoming data dynamically.

To do so you would additionally to what you show add a dynamically allocated "string" which grows if fully used by reallocating it to provide more room.

Code doing so might look like this:

#include <stdlib.h>
#include <stdio.h>


#define BUFFER_SIZE (1024)
#define START_SIZE (1) /* Has to be > 0 */


int main(void) 
{
  size_t s = START_SIZE;
  char * path = malloc(s);
  if (NULL == path)
  {
    perror("malloc() failed");
    return EXIT_FAILURE);
  }

  path[0] = '\0';

  {
    /* Open the command for reading. */
    FILE * fp = popen("bash /home/ouhma/myscript.sh", "r");
    if (NULL == fp) 
    {
      perror("popen() failed");
      return EXIT_FAILURE); /* By returning here the code leaks the memory
                               already allocated to path as well as fp. */
    }

    {
      char buffer[BUFFER_SIZE];

      /* Read the output a line at a time - output it. */
      while (NULL != fgets(buffer, sizeof buffer, fp)) 
      {
        fprintf(stderr, "partly read: '%s'\n", buffer);

        while ((s - 1) < strlen(buffer))
        {
          void * p = realloc(path, s *= 2); /* Grow s exponentially. */
          if (NULL == p)
          {
            perror("realloc() failed");
            return EXIT_FAILURE; /* By returning here the code leaks the memory
                                    already allocated to path as well as fp. */
          }

          path = p;
        }

        /* Concatenate what just had been read to final "string". */
        strcat(path, buffer);
      }
    }

    if (!feof(fp))
    {
      perror("fgets() failed");
    }

    /* Close file. */
    if (-1 == pclose(fp))
    {
      perror("pclose() failed");
    }
  }

  /* Print result. */
  printf("read: '%s'\n", path);

  /* Clean up. */
  free(path);

  return EXIT_SUCCESS;
}

Clean up of open file descriptors and dynamically allocated memory in case of any failure is left as an exercise to the reader ... ;-)

How could I allocate a variable length command output?

1 Answers1