---
title: "Exploring the openSenseMap Dataset"
author: "Norwin Roosen"
date: "`r Sys.Date()`"
output:
rmarkdown::html_vignette:
fig_margin: 0
fig_width: 6
fig_height: 4
vignette: >
%\VignetteIndexEntry{Exploring the openSenseMap Dataset}
%\VignetteEngine{knitr::rmarkdown}
%\VignetteEncoding{UTF-8}
---
```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```
This package provides data ingestion functions for almost any data stored on the
open data platform for environmental sensordata <https://opensensemap.org>.
Its main goals are to provide means for:
- big data analysis of the measurements stored on the platform
- sensor metadata analysis (sensor counts, spatial distribution, temporal trends)
### Exploring the dataset
Before we look at actual observations, lets get a grasp of the openSenseMap
datasets' structure.
```{r results = FALSE}
library(magrittr)
library(opensensmapr)
# all_sensors = osem_boxes(cache = '.')
all_sensors = readRDS('boxes_precomputed.rds') # read precomputed file to save resources
```
```{r}
summary(all_sensors)
```
This gives a good overview already: As of writing this, there are more than 700
sensor stations, of which ~50% are currently running. Most of them are placed
outdoors and have around 5 sensors each.
The oldest station is from May 2014, while the latest station was registered a
couple of minutes ago.
Another feature of interest is the spatial distribution of the boxes: `plot()`
can help us out here. This function requires a bunch of optional dependencies though.
```{r, message=FALSE, warning=FALSE}
plot(all_sensors)
```
It seems we have to reduce our area of interest to Germany.
But what do these sensor stations actually measure? Lets find out.
`osem_phenomena()` gives us a named list of of the counts of each observed
phenomenon for the given set of sensor stations:
```{r}
phenoms = osem_phenomena(all_sensors)
str(phenoms)
```
Thats quite some noise there, with many phenomena being measured by a single
sensor only, or many duplicated phenomena due to slightly different spellings.
We should clean that up, but for now let's just filter out the noise and find
those phenomena with high sensor numbers:
```{r}
phenoms[phenoms > 20]
```
Alright, temperature it is! Fine particulate matter (PM2.5) seems to be more
interesting to analyze though.
We should check how many sensor stations provide useful data: We want only those
boxes with a PM2.5 sensor, that are placed outdoors and are currently submitting
measurements:
```{r results = FALSE, eval=FALSE}
pm25_sensors = osem_boxes(
exposure = 'outdoor',
date = Sys.time(), # ±4 hours
phenomenon = 'PM2.5'
)
```
```{r}
pm25_sensors = readRDS('pm25_sensors.rds') # read precomputed file to save resources
summary(pm25_sensors)
plot(pm25_sensors)
```
Thats still more than 200 measuring stations, we can work with that.
### Analyzing sensor data
Having analyzed the available data sources, let's finally get some measurements.
We could call `osem_measurements(pm25_sensors)` now, however we are focusing on
a restricted area of interest, the city of Berlin.
Luckily we can get the measurements filtered by a bounding box:
```{r, results=FALSE, message=FALSE}
library(sf)
library(units)
library(lubridate)
library(dplyr)
```
Since the API takes quite long to response measurements, especially filtered on space and time, we do not run the following chunks for publication of the package on CRAN.
```{r bbox, results = FALSE, eval=FALSE}
# construct a bounding box: 12 kilometers around Berlin
berlin = st_point(c(13.4034, 52.5120)) %>%
st_sfc(crs = 4326) %>%
st_transform(3857) %>% # allow setting a buffer in meters
st_buffer(set_units(12, km)) %>%
st_transform(4326) %>% # the opensensemap expects WGS 84
st_bbox()
pm25 = osem_measurements(
berlin,
phenomenon = 'PM2.5',
from = now() - days(3), # defaults to 2 days
to = now()
)
```
```{r}
pm25 = readRDS('pm25_berlin.rds') # read precomputed file to save resources
plot(pm25)
```
Now we can get started with actual spatiotemporal data analysis.
First, lets mask the seemingly uncalibrated sensors:
```{r, warning=FALSE}
outliers = filter(pm25, value > 100)$sensorId
bad_sensors = outliers[, drop = TRUE] %>% levels()
pm25 = mutate(pm25, invalid = sensorId %in% bad_sensors)
```
Then plot the measuring locations, flagging the outliers:
```{r}
st_as_sf(pm25) %>% st_geometry() %>% plot(col = factor(pm25$invalid), axes = TRUE)
```
Removing these sensors yields a nicer time series plot:
```{r}
pm25 %>% filter(invalid == FALSE) %>% plot()
```
Further analysis: comparison with LANUV data `TODO`
