0

I have a df frame containing TV viewing data, I would like to run a QC check for overlapping viewing. Let's say for the same day, same household, for each individual, each minute should be credited to one station or channel only.

for example, I would like to flag line 8 , 9 , because it seem impossible an individual in a unique household watched two TV stations (62,67) at the same time (start_hour_minute) . I am wondering is there a way to flag this rows? A sort of min by min view by individual by day. 

df <- data.frame(stringsAsFactors=FALSE,
         date = c("2018-09-02", "2018-09-02", "2018-09-02", "2018-09-02",
                  "2018-09-02", "2018-09-02", "2018-09-02", "2018-09-02",
                  "2018-09-02"),
         householdID = c(18101276L, 18101276L, 18102843L, 18102843L, 18102843L,
                  18102843L, 18104148L, 18104148L, 18104148L),
   Station_id = c(74L, 74L, 62L, 74L, 74L, 74L, 62L, 62L, 67L),
        IndID = c("aa", "aa", "aa", "aa", "aa", "aa", "aa", "aa", "aa"),
        Start = c(111300L, 143400L, 030000L, 034900L, 064400L, 070500L, 060400L,
                  075100L, 075100L),
          End = c(111459L, 143759L, 033059L, 035359L, 064759L, 070559L, 060459L,
                  81559L, 81559L),
   start_hour_minute = c(1113L, 1434L, 0300L, 0349L, 0644L, 0705L, 0604L, 0751L, 0751L),
     end_hour_minute = c(1114L, 1437L, 0330L, 0353L, 0647L, 0705L, 0604L, 0815L, 0815L))
Ann
  • 178
  • 3
  • 13

2 Answers2

1

You can just group by the variables that you think should correspond to single rows (e.g. household-date-minute combinations) and then count the rows (or the unique values in Station_id) and add flag = 1 if that row should be flagged, else flag = 0

df %>% 
    group_by(date, householdID, start_hour_minute) %>% 
    mutate(flag = if_else(n() == 1, 0, 1))

Alternatively, if you want all other variables to match except Station_id, you can do 

df %>% 
    group_by_at(vars(-Station_id)) %>% 
    mutate(flag = if_else(n() == 1, 0, 1))
konvas
  • 13,106
  • 2
  • 34
  • 43
  • Thank you so much! I was struggling with `lubridate::int_overlaps()`. Thanks man – Ann Sep 12 '18 at 20:02
1

The lubridate package has an inteval class object and the %within% function that checks if a timestamp is within an interval. You can use this to get flags.

Using the dummy data you provided above...

data_out <- df %>% 
# Get the hour, minute, and second values as standalone numerics.
mutate(
    date = ymd(date),
    Start_Hour = floor(Start / 10000),
    Start_Minute = floor((Start - Start_Hour*10000) / 100),
    Start_Second = (Start - Start_Hour*10000) - Start_Minute*100,
    End_Hour = floor(End / 10000),
    End_Minute = floor((End - End_Hour*10000) / 100),
    End_Second = (End - End_Hour*10000) - End_Minute*100,
# Use the hour, minute, second values to create a start-end timestamp.
    Start_TS = ymd_hms(date + hours(Start_Hour) + minutes(Start_Minute) + seconds(Start_Second)),
    End_TS = ymd_hms(date + hours(Start_Hour) + minutes(Start_Minute) + seconds(Start_Second)),
# Create an interval object.
    Watch_Interval = interval(start = Start_TS, end = End_TS)
) %>% 
# Group by the IDs.
group_by(householdID, Station_id) %>% 
# Flag where the household's interval overlaps with another time.
mutate(
    overlap_flag = case_when(
        sum(Start_TS %within% as.list(Watch_Interval)) == 0 ~ 0,
        sum(Start_TS %within% as.list(Watch_Interval)) > 0 ~ 1,
        TRUE ~ NA_real_
    )
) %>% 
# dplyr doesn't play nice with interval objects, so we should remove Watch_Interval.
select(-Watch_Interval)

See the flagged values using data_out %>% filter(overlap_flag == 1).

Note: The dplyr and lubridate packages don't always play nice together, especially older versions. You may need to update the package versions for each.

Michael Herman
  • 21
  • 4