0

I've been web scraping yahoo finance stock options for a couple of years and I use rvest to do it. I am currently using Rstudio server on ubuntu with no problem. I was working on adding certain features on my macbook pro when i ran into a problem: I would not get the options date data. It works on ubuntu but not on my macbook. I'm running the exact same code and i have the same package version on both machines. On my mac this returns an empty string and on ubuntu i see all the selectors dates (correct information). 

library(rvest)
library(dplyr)

url_date <- "https://finance.yahoo.com/quote/SPY/options?p=SPY"

webpage_date <- 
 read_html(url_date) %>%
 html_nodes(".drop-down-selector") %>%
 html_text

In Ubuntu these return data but on the macbook they return an empty character vector

webpage_date <- 
  read_html(url_date) %>%
  html_nodes("option") %>%
  html_text

webpage_date <- 
  read_html(url_date) %>%
  html_nodes("#Main") %>%
  html_text

furthermore, i get different results on both machines when I search for this ID

webpage_date <- 
  read_html(url_date) %>%
  html_nodes("#YDC-Col1")

I'm very confused about the different results.

vicm159
  • 1
  • 1
  • The difference may lie in having different user agents on Ubuntu vs macOS. You could check this via read_html("http://httpbin.org/user-agent"). If they differ, try setting the macOS one equal to the Ubuntu one, via httr::set_config(httr::user_agent("PASTE HERE")) . – Bas Dec 16 '19 at 11:29
  • thank you for the quick reply but i dont think thats the problem. I used this code to get the user agent `se – vicm159 Dec 18 '19 at 06:07
  • 1
    Not sure why this works but it does: `read_html(content(GET(url_date), "text"))`. If i add the content and GET function (they come from the httr pacakge) inside the read_html function with the html node as 'option' then i get the same result as in Ubuntu. Or the result i want. i'll ask this question on the rvest github repo. FYI @Bas – vicm159 Mar 14 '20 at 06:33

0 Answers0