Hi I recently took up a project about bitcoin analysis and need to download financial data from Yahoo! Finance via Python. I tried fix_yahoo_finance and pandas datareader but there seems to be a bug on the website when downloading files. It always miss some days. So I decided to use beautiful soup, the code is as follow:
import requests
import time
import pandas as pd
from bs4 import BeautifulSoup
def time_convert(dt):
time.strptime(dt,'%Y-%m-%d %H:%M:%S')
s = time.mktime(time.strptime(dt,'%Y-%m-%d %H:%M:%S'))
return str(int(s))
s = requests.Session()
start = time_convert("2016-02-15 00:00:00")
end = time_convert("2018-02-15 00:00:00")
r = s.get("https://uk.finance.yahoo.com/quote/BTC-USD/history?period1="+start+"&period2="+end+"&interval=1d&filter=history&frequency=1d"
soup = BeautifulSoup(r.text, 'lxml')
tables = soup.select('table')
df_list = []
for table in tables:
df_list.append(pd.concat(pd.read_html(table.prettify())))
df = pd.concat(df_list)
df.to_excel("E:\PythonData\price_"+'.xlsx')
It works but the data is not complete because the website loads the data when your mouse scroll s down to the end of the page, but codes doesn't do that. How can I fix this?