6  OPTIONAL: Time Series

Week 6 (study week): Cleaning and plotting time series data

This is an optional tutorial for anyone working with data collected over time: repeated surveys, data loggers, weather records or growth measurements.

Note

This tutorial is optional and not assessed, and it will not be in the exam. Work through it in your own time if time series are relevant to your project.

You will learn how to:

  1. Convert text into dates and times with lubridate, and extract useful parts (year, month, day of year).
  2. Plot long-term survey data and spot sudden changes.
  3. Explore seasonality, trends and autocorrelation.
  4. Highlight a study period on logger data.
  5. Fit a simple growth curve.

Most examples use datasets from the lterdatasampler package.

Code
# Week 6 workshop (optional) - Time series
# Written by Nicholas Wu, 1234578, 01/04/2027

# install.packages("lterdatasampler")  # run once in the console if needed

library(tidyverse)        # includes lubridate for dates and times
library(lterdatasampler)  # example ecological datasets
library(patchwork)        # combine plots

# setwd("C:/Users/YOUR-NAME/Documents/ECS200")

Dates and times with lubridate

When you import a CSV file, dates are often read as text (chr). R needs to know they are dates before it can sort, plot or subtract them. The lubridate functions are named after the order of the parts in your dates: ymd() for year-month-day, dmy() for day-month-year, mdy() for month-day-year, with _hm or _hms added when there is a time.

Code
lubridate::dmy("05-01-1987")              # day-month-year
[1] "1987-01-05"
Code
lubridate::mdy("01/05/1987")              # month-day-year
[1] "1987-01-05"
Code
lubridate::ymd_hms("1987-01-05 14:30:00") # date and time
[1] "1987-01-05 14:30:00 UTC"

Which lubridate function to use, depending on how the dates were written.

Once a column is a date, you can extract useful parts from it:

Code
stream_dates <- lterdatasampler::luq_streamchem %>%
  dplyr::select(sample_date, temp) %>%
  dplyr::mutate(year      = lubridate::year(sample_date),                 # year
                month     = lubridate::month(sample_date),                # month as a number
                month_lab = lubridate::month(sample_date, label = TRUE),  # month as a label (Jan, Feb...)
                doy       = lubridate::yday(sample_date))                 # day of the year (1-366)

head(stream_dates)
# A tibble: 6 × 6
  sample_date  temp  year month month_lab   doy
  <date>      <dbl> <dbl> <dbl> <ord>     <dbl>
1 1987-01-05     20  1987     1 Jan           5
2 1987-01-13     20  1987     1 Jan          13
3 1987-01-20     20  1987     1 Jan          20
4 1987-01-27     20  1987     1 Jan          27
5 1987-02-03     20  1987     2 Feb          34
6 1987-02-10     20  1987     2 Feb          41

Ecological time series are often sampled unevenly. The gap between consecutive samples, calculated with dplyr::lag(), shows missing sampling periods or changes in effort.

Code
stream_dates %>%
  dplyr::arrange(sample_date) %>%
  dplyr::mutate(days_since_last = as.numeric(sample_date - dplyr::lag(sample_date))) %>%
  dplyr::count(days_since_last, sort = TRUE) %>%  # how often each gap occurs
  head()
# A tibble: 6 × 2
  days_since_last     n
            <dbl> <int>
1               7   284
2               6    11
3               8    10
4               1     2
5               2     2
6               5     2

Examples

Bad time series data

Loggers can be damaged, exposed to direct sun, flooded or affected by software bugs. In this example something went wrong after 1 November:

Time series with bad data after 1 November.

These data need to be removed before analysis. Depending on the problem, you can:

  • filter out values beyond a biologically or physically possible threshold (e.g. humidity above 100%);
  • filter out outliers with the IQR rule (Week 1);
  • filter out the time period when you know the equipment was faulty (e.g. after a storm), using dplyr::filter(date_time < lubridate::ymd("2023-11-01")).

A worked example: cleaning a raw logger file

Other common problems with microclimate loggers are outliers, duplicate times/values and missing data points. logger_bad.csv is a (simulated) week of hourly air temperature from a logger placed in a tree hollow. It contains all of these problems, just like a real file straight off a logger.

logger_bad.csv

Code
logger_raw <- readr::read_csv("data/logger_bad.csv") %>%
  dplyr::mutate(date_time = lubridate::dmy_hm(date_time)) # text to date-time

summary(logger_raw)
   date_time                      air_temp     
 Min.   :2023-07-10 00:00:00   Min.   : 4.900  
 1st Qu.:2023-07-11 17:15:00   1st Qu.: 7.475  
 Median :2023-07-13 04:30:00   Median :11.650  
 Mean   :2023-07-13 09:56:23   Mean   :11.877  
 3rd Qu.:2023-07-15 05:45:00   3rd Qu.:15.825  
 Max.   :2023-07-16 23:00:00   Max.   :47.300  
                               NAs    :2       
Code
# Always plot the raw data first
ggplot2::ggplot(logger_raw, ggplot2::aes(x = date_time, y = air_temp)) +
  ggplot2::geom_line() +
  ggplot2::geom_point(size = 0.8) +
  ggplot2::labs(x = NULL, y = "Air temperature (°C)") +
  ggplot2::theme_bw()

The plot already shows a spike to over 45 °C and a gap where the line jumps across missing hours. The other problems are hidden, so we check for each one.

Compare raw and cleaned data

Finally, compare the raw and cleaned data. Filled values are shown in red, and the long gap is left empty:

Code
raw_plot <- ggplot2::ggplot(logger_raw, ggplot2::aes(x = date_time, y = air_temp)) +
  ggplot2::geom_line(colour = "grey50") +
  ggplot2::labs(x = NULL, y = "Air temp (°C)", title = "Raw") +
  ggplot2::theme_bw()

clean_plot <- ggplot2::ggplot(logger_clean, ggplot2::aes(x = date_time, y = air_temp_filled)) +
  ggplot2::geom_line() +                                          # breaks where values are NA
  ggplot2::geom_point(data = dplyr::filter(logger_clean, filled),
                      colour = "#e12744", size = 2) +             # interpolated values
  ggplot2::labs(x = NULL, y = "Air temp (°C)", title = "Cleaned") +
  ggplot2::theme_bw()

raw_plot / clean_plot

TipChecklist for raw logger data
  1. Convert the date-time column with lubridate and plot the raw data.
  2. Remove exact duplicate rows (dplyr::distinct(), Week 1).
  3. Find times with more than one reading (dplyr::count(date_time)) and keep one.
  4. Flag impossible values and sudden jumps (dplyr::lag()) and set them to NA.
  5. Add missing times with tidyr::complete().
  6. Interpolate only short gaps; leave long gaps as NA.
  7. Report every cleaning step in your methods, and keep the raw file unchanged.

Analysing time series

Time series analysis is beyond this unit, but here are some possibilities:

  • Repeated measures: if the same individuals or sites are measured over time, include them as a random effect (Week 12).
  • Trends and population dynamics: GLMs (Week 8), or generalised additive models (GAMs) for non-linear trends over time (Pedersen et al., 2019).
  • Forecasting: models such as autoregressive (AR), ARIMA and exponential smoothing handle autocorrelation and seasonality and can predict future values. See this live forecast of desert rodent populations.

Captures of Merriam’s kangaroo rat (Dipodomys merriami) over time.

Additional resources