I have a mysql table which contains a Primary auto_increment key.
I have 500 csv files, each about 3gb worth of data and the bulk of the data in one column.
Currently I'm loading the files into mysql using:
#!/bin/bash
for file in /files/*.csv
do
mysql -e "load data local infile '$f' into table myTable FIELDS TERMINATED BY ','
ENCLOSED BY '\"' escaped by '\"' IGNORE 1 LINES" -u user -ppass
done
Are there any ways to improve performance? Maybe removing the primary key while inserting and then adding it afterwards? Or is there a way to insert in parallel instead of one file at a time?