3

I have a short script which reads 1,048,574 records out of a CSV with two columns, and puts them into two lists, one for each column. So each list should end up with 1,048,574 elements.

In PyCharm, this works fine. In the console, the program hangs around 1,048,345.

I can't figure it out. Both are using the same Python, 3.6.5.

Here is the script:

import csv

xtime = []
voltage = []

count = 0
with open ('audio2.csv', newline='') as csvfile:
    reader = csv.reader(csvfile)
    for row in reader:
        xtime.append(row[0])
        voltage.append(row[1])
        print(count)
        count += 1
    print('finished reading')

Each line of the CSV looks like -0.000951453,-1.56 and there are 1,048,574 lines. PyCharm prints finished reading at the end, but GitBash just hangs after printing 1048345.

What could be the problem?

temporary_user_name
  • 30,801
  • 41
  • 120
  • 186
  • Did you consider removing the `print(count)`? It seems excessive to print a million line to the console. – Samuel Dion-Girardeau May 25 '18 at 15:47
  • I wanted to know how far it was getting and didn't know a better way when running from console. Do you know a better way in the console? – temporary_user_name May 25 '18 at 15:47
  • You could maybe print the count once every n iterations (e.g. `if count % 1000`). If you want to have row-level precision, you can also try redirecting the output to a file, which will be faster that displaying it, e.g. `python myscript.py > log.txt` – Samuel Dion-Girardeau May 25 '18 at 15:50

2 Answers2

0

I am not sure, but if I were you I would either use python from windows command line or if launched from git bash I will try winpty python

It may be related to this issue

akasolace
  • 546
  • 5
  • 15
0

If you are very sure the Python path is the same, maybe you should try to use less data for testing with Python in console to checking whether it's the memory problem.

forAllBright
  • 61
  • 1
  • 8
  • this should be indeed the first test – akasolace May 25 '18 at 15:34
  • You are correct, removing 250 records causes it to run fine. But why would I have enough memory in PyCharm but not in GitBash? I'm very surprised to run into this issue with a list containing only a million items when I have 16GB of memory and nothing else running. – temporary_user_name May 25 '18 at 15:35
  • I am not sure but if it could be the git bash shell issue? A conserved suggestion is to run python in powershell or cmd. Just a try. – forAllBright May 25 '18 at 15:57