Python - process a chunk of lines in a file

Question

I have a file containing x number of values each on their own line. I need to be able to take n number of value from this file, put them into an array, pass that array into a new process, clear the array and then take another n number of values from the file to give to the next process.

The problem I'm having is when x is a value like 12 and I'm trying to give, let's say, 10 chunks of values of each process.

The first process will get it's first 10 values no problem, but I'm having trouble giving the remaining 2 to the last process.

The problem would also arise if, let's say, you tell the program to give each process 10 values from the file, but the file only has 1, or even 9 values.

I need know when I'm at the last set of values that is less than n

I want to avoid taking every value in the file and storing it in an array all at once since I could run into memory problems if there was millions of values in that file.

Here's an example of what I've tried to do:

chunk = 10
value_list = []
with open ('file.txt', 'r') as f:
    for value in f:
        value_list.append(value)
        if (len(value_list) >= chunk):
            print 'Got %d' % len(value_list)
            value_list = [] # Clear the list
            # Put array into new process

This will catch every 10 in this example, but it wont work if there even happend to be less than 10 in the file to begin with.

just use the remaining data from `value_list` after your loop is done to call the process one last time (if `value_list` is not empty) — Jean-François Fabre, Oct 20 '16 at 10:30

Tom Barron · Accepted Answer · 2016-10-20T19:16:27.073

2

What I typically do in this situation is just handle the last (short) array after the for loop. For example,

chunk = 10
value_list = []
with open ('file.txt', 'r') as f:
    for value in f:
        if (len(value_list) >= chunk):
            print 'Got %d' % len(value_list)
            value_list = [] # Clear the list
            # Put array into new process
        value_list.append(value)
    # send left overs to new process
    if value_list:
        print 'Got %d' % len(value_list)
        # Put final array into new process

edited Oct 20 '16 at 19:16

answered Oct 20 '16 at 10:31

Tom Barron

1,354
17
20

I edited my post to include the `value_list.append(value)`. Not sure why that wasn't there. Just wanted to point out that if you have `value_list.append(value)` within that else as you have it, it ends up skipping the `n`th value if it hits that if statement at least once. – Shaun Aran Oct 20 '16 at 10:42
1

The above code has a logical error - it will skip one line each time it reaches `chunk`size. Solution: move the `append` out of the `else` branch. – bruno desthuilliers Oct 20 '16 at 11:46
@ShaunAran, bruno, good catch! Thank you both. I have corrected the example. – Tom Barron Oct 20 '16 at 19:17

Python - process a chunk of lines in a file

1 Answers1