How can I insert millions or rows from a .txt file into a SQLite3 database using Flask-SQLAlchemy? I tried just reading one line at the time from the .txt file and adding and committing them in a loop but noticed that tis takes an enormous amount of time to do. How can I do this effectively? I tried implementing this solution https://stackoverflow.com/a/7137270 to my code but couldn't get it to work.
The table schema looks like this:
class table(db.Model):
id = db.Column(db.Integer, primary_key=True)
col1 = db.Column(db.Integer)
col2 = db.Column(db.Integer)
col3 = db.Column(db.String(50))
col4 = db.Column(db.String(50))
col5 = db.Column(db.String(50))
col6 = db.Column(db.Integer)
col7 = db.Column(db.String(50))
col8 = db.Column(db.Integer)
col9 = db.Column(db.Integer)
col10 = db.Column(db.Integer)
col11 = db.Column(db.Integer)
col12 = db.Column(db.Integer)
The rows in the .txt
file look like this:
hjk;28770930;Y;T;C;;asd;;1;1233;1233;0.00081103
And there are about 85M rows to add to the db.