Efficiant way to load Wold Bank Data via WDI in shiny

Question

I am working on a shiny dashboard project to visualize data from the World Bank. In total i need 14 different variables that I want to show in a data table. It works in R but when I try to do it in shiny I get a connection error. I think it takes too much time to download. Is there a way to load the data more efficient or in background or whatever? I can't find a solution to make it run and I don't want to store the data in a file on the server because then I don't have an automatic update... :/

Here's a minimal vital code snippet

library(shiny)
library(dplyr)
library(WDI)
library(wbstats)
library(lubridate)

ui <- fluidPage(

   titlePanel('WDI Example'),
   mainPanel(tableOutput('table'))

   )


server <- function(input, output) {

  startDate <- as.numeric(format(Sys.Date(), "%Y")) -20
  endDate <- as.numeric(format(Sys.Date(), "%Y"))

  fetch.data <- list() 

  list <- c("SP.POP.TOTL",
            "NY.ADJ.NNTY.PC.CD",
            "SH.XPD.CHEX.PC.CD",
            "SH.XPD.OOPC.CH.ZS",
            "SH.MED.BEDS.ZS",
            "SH.MED.PHYS.ZS",
            "NY.GDP.MKTP.CD",
            "NY.GDP.MKTP.KD",
            "NY.GDP.MKTP.KD.ZG",
            "SH.XPD.CHEX.GD.ZS",
            "SP.DYN.LE00.IN",
            "SP.POP.65UP.TO",
            "SP.POP.DPND",
            "SP.POP.DPND.OL") 

  for (i in 1:length(list)){
    fetch.data[[i]] <- WDI(indicator = as.character(list[i]),
                           start = startDate, 
                           end = endDate,
                           country = "all",
                           extra = TRUE)
  }

  data <- fetch.data


  for (i in 1:length(list)){

    data[[i]] <- subset(data[[i]], region != "Aggregates")
    data[[i]] <- data[[i]] %>% select(country, iso3c, region, year, noquote(list[i]))
    data[[i]]$date <- as.Date(as.character(date), format = "%Y")
    data[[i]] <- data[[i]][complete.cases(data[[i]][ ,5]),]
    data[[i]] <- data[[i]] %>% group_by(country) %>% arrange(desc(date)) %>% slice(1)
  }


  indicators <- fetch.data[[1]] %>% 
    select(country, iso3c, region) %>% 
    subset(region != "Aggregates") %>% 
    arrange(desc(country)) %>% 
    group_by(country) %>% 
    slice(1)


  for (i in 1:length(list)){
    indicators <- merge(indicators, data[[i]][ ,c(2,5)], by.x = "iso3c", by.y = "iso3c", all = TRUE)
  }


  indicators <- indicators %>% 
    arrange(country) %>% 
    mutate_if(is.numeric, round, 2)

  output$table <- renderTable(indicators)

}

shinyApp(ui = ui, server = server)

and here is an example that does not work… https://thera-trainer.shinyapps.io/example/

Thanks for advice

Can you put the error message in your question? – Wil Jan 16 '20 at 13:11 — Wil, Jan 16 '20 at 13:11

Wil · Accepted Answer · 2020-01-16T14:06:28.790

The Error

Your error is in this line of code:

data[[i]]$date <- as.Date(as.character(date), format = "%Y")

date is a function (closure), which is why you are getting an error. If you mean to refer to a column in your dataframe, you would need to use data[[i]]$date, as in

data[[i]]$date <- as.Date(as.character(data[[i]]$date), format = "%Y")

You are likely used to the tidyverse where you aren't required to prefix columns with their dataframe, but when using non-tidyverse operations, like <-, you must include the dataframe name as well. In addition, there is no column in data[[i]] called date, so even if you had used as.Date(as.character(data[[i]]$date), format = "%Y") you would have received an error because that column doesn't exist.

Further, it looks like you are trying to convert a date to a year, but there is already a column in data[[i]] called year, you can refer to it with data[[i]]$year.

Code Style

It is not good practice to name your objects with names that match those of a function. You have done this twice, once by assigning fetch.data to data (there is already a function utils::data(), and more importantly you have assigned your indicators to a variable called list. This is extremely dangerous, as you have overwritten the list() function in your global environment, and to use the list function again later on you will have to call it with base::list(). I would recommend a more emotive name for the indicators, such as:

indicators_list <- c("SP.POP.TOTL",
            "NY.ADJ.NNTY.PC.CD",
            "SH.XPD.CHEX.PC.CD",
            "SH.XPD.OOPC.CH.ZS",
            "SH.MED.BEDS.ZS",
            "SH.MED.PHYS.ZS",
            "NY.GDP.MKTP.CD",
            "NY.GDP.MKTP.KD",
            "NY.GDP.MKTP.KD.ZG",
            "SH.XPD.CHEX.GD.ZS",
            "SP.DYN.LE00.IN",
            "SP.POP.65UP.TO",
            "SP.POP.DPND",
            "SP.POP.DPND.OL")

withProgress

There is a Shiny progress indicator you can use to monitor loops that take a long time. You would include it like this;

  withProgress(
    for (i in 1:length(indicators_list)){
      incProgress(1/length(indicators_list), detail = paste("Doing part", i))
      fetch.data[[i]] <- WDI(indicator = indicators_list[i],
                             start = startDate, 
                             end = endDate,
                             country = "all",
                             extra = TRUE)
  })

Complete Example

Here is a complete example that I was able to get working. I have only included the server component as that is the only one where I made changes.

server <- function(input, output) {

  startDate <- as.numeric(format(Sys.Date(), "%Y")) -20
  endDate <- as.numeric(format(Sys.Date(), "%Y"))

  fetch.data <- list() 

  indicators_list <- c("SP.POP.TOTL",
                  "NY.ADJ.NNTY.PC.CD",
                  "SH.XPD.CHEX.PC.CD",
                  "SH.XPD.OOPC.CH.ZS",
                  "SH.MED.BEDS.ZS",
                  "SH.MED.PHYS.ZS",
                  "NY.GDP.MKTP.CD",
                  "NY.GDP.MKTP.KD",
                  "NY.GDP.MKTP.KD.ZG",
                  "SH.XPD.CHEX.GD.ZS",
                  "SP.DYN.LE00.IN",
                  "SP.POP.65UP.TO",
                  "SP.POP.DPND",
                  "SP.POP.DPND.OL") 

  withProgress(
    for (i in 1:length(indicators_list)){
      incProgress(1/length(indicators_list), detail = paste("Doing part", i))
      fetch.data[[i]] <- WDI(indicator = indicators_list[i],
                             start = startDate, 
                             end = endDate,
                             country = "all",
                             extra = TRUE)
  })

  data <- fetch.data


  for (i in 1:length(indicators_list)){

    data[[i]] <- subset(data[[i]], region != "Aggregates")
    data[[i]] <- data[[i]] %>% select(country, iso3c, region, year, noquote(indicators_list[i]))
    data[[i]]$date <- data[[i]]$year
    data[[i]] <- data[[i]][complete.cases(data[[i]][ ,5]),]
    data[[i]] <- data[[i]] %>% group_by(country) %>% arrange(desc(date)) %>% slice(1)
  }


  indicators <- fetch.data[[1]] %>% 
    select(country, iso3c, region) %>% 
    subset(region != "Aggregates") %>% 
    arrange(desc(country)) %>% 
    group_by(country) %>% 
    slice(1)


  for (i in 1:length(indicators_list)){
    indicators <- merge(indicators, data[[i]][ ,c(2,5)], by.x = "iso3c", by.y = "iso3c", all = TRUE)
  }


  indicators <- indicators %>% 
    arrange(country) %>% 
    mutate_if(is.numeric, round, 2)

  output$table <- renderTable(indicators)

}

You're welcome. If you found the answer useful, consider [accepting](https://stackoverflow.com/help/someone-answers) it so that others with the same question may find the answer. — Wil, Jan 16 '20 at 15:44

Efficiant way to load Wold Bank Data via WDI in shiny

1 Answers1