It may be useful to download data from openSenseMap only once. For reproducible results, the data could be saved to disk, and reloaded at a later point.

This avoids..

This vignette shows how to use this built in opensensmapr feature, and how to do it yourself, if you want to store to other data formats.

Using openSensMapr Caching Feature

All data retrieval functions of opensensmapr have a built in caching feature, which serializes an API response to disk. Subsequent identical requests will then return the serialized data instead of making another request. To do so, each request is given a unique ID based on its parameters.

To use this feature, just add a path to a directory to the cache parameter:

b = osem_boxes(cache = tempdir())
list.files(tempdir(), pattern = 'osemcache\\..*\\.rds')

## [1] "osemcache.c54710f66b662e29dd86b089962b0f598e47eddb.rds"

# the next identical request will hit the cache only!
b = osem_boxes(cache = tempdir())

# requests without the cache parameter will still be performed normally
b = osem_boxes()

You can maintain multiple caches simultaneously which allows to store only serialized data related to a script in its directory:

cacheDir = getwd() # current working directory
b = osem_boxes(cache = cacheDir)

# the next identical request will hit the cache only!
b = osem_boxes(cache = cacheDir)

To get fresh results again, just call osem_clear_cache() for the respective cache:

osem_clear_cache() # clears default cache

## [1] TRUE

osem_clear_cache(getwd()) # clears a custom cache

## [1] TRUE

Custom (De-) Serialization

If you want to roll your own serialization method to support custom data formats, here's how:

# this section requires:
library(opensensmapr)
library(jsonlite)
library(readr)

# first get our example data:
boxes = osem_boxes(grouptag = 'ifgi')
measurements = osem_measurements(boxes, phenomenon = 'PM10')

If you are paranoid and worry about .rds files not being decodable anymore in the (distant) future, you could serialize to a plain text format such as JSON. This of course comes at the cost of storage space and performance.

# serializing senseBoxes to JSON, and loading from file again:
write(jsonlite::serializeJSON(measurements), 'boxes.json')
boxes_from_file = jsonlite::unserializeJSON(readr::read_file('boxes.json'))

Both methods also persist the R object metadata (classes, attributes). If you were to use a serialization method that can't persist object metadata, you could re-apply it with the following functions:

# note the toJSON call
write(jsonlite::toJSON(measurements), 'boxes_bad.json')
boxes_without_attrs = jsonlite::fromJSON('boxes_bad.json')

boxes_with_attrs = osem_as_sensebox(boxes_without_attrs)
class(boxes_with_attrs)

## [1] "sensebox"   "data.frame"

The same goes for measurements via osem_as_measurements().

Workflow for reproducible code

For truly reproducible code you want it to work and return the same results – no matter if you run it the first time or a consecutive time, and without making changes to it.

Therefore we need a wrapper around the save-to-file & load-from-file logic. The following examples show a way to do just that, and where inspired by this reproducible analysis by Daniel Nuest.

# offline logic
osem_offline = function (func, file, format='rds', ...) {
  # deserialize if file exists, otherwise download and serialize
  if (file.exists(file)) {
    if (format == 'json')
      jsonlite::unserializeJSON(readr::read_file(file))
    else
      readRDS(file)
  } else {
    data = func(...)
    if (format == 'json')
      write(jsonlite::serializeJSON(data), file = file)
    else
      saveRDS(data, file)
    data
  }
}

# wrappers for each download function
osem_measurements_offline = function (file, ...) {
  osem_offline(opensensmapr::osem_measurements, file, ...)
}
osem_boxes_offline = function (file, ...) {
  osem_offline(opensensmapr::osem_boxes, file, ...)
}
osem_box_offline = function (file, ...) {
  osem_offline(opensensmapr::osem_box, file, ...)
}
osem_counts_offline = function (file, ...) {
  osem_offline(opensensmapr::osem_counts, file, ...)
}

Thats it! Now let's try it out:

# first run; will download and save to disk
b1 = osem_boxes_offline('mobileboxes.rds', exposure='mobile')

# consecutive runs; will read from disk
b2 = osem_boxes_offline('mobileboxes.rds', exposure='mobile')
class(b1) == class(b2)

## [1] TRUE TRUE

# we can even omit the arguments now (though thats not really the point here)
b3 = osem_boxes_offline('mobileboxes.rds')
nrow(b1) == nrow(b3)

## [1] TRUE

# verify that the custom sensebox methods are still working
summary(b2)

## boxes total: 55
## 
## boxes by exposure:
## mobile 
##     55 
## 
## boxes by model:
##                   custom             homeEthernet                 homeWifi 
##                        7                        2                        8 
##        homeWifiFeinstaub luftdaten_pms5003_bme280  luftdaten_sds011_bme280 
##                        6                        2                        9 
##   luftdaten_sds011_dht11   luftdaten_sds011_dht22 
##                        1                       20 
## 
## $last_measurement_within
##    1h    1d   30d  365d never 
##    16    18    24    43    12 
## 
## oldest box: 2017-05-24 08:16:36 (Feinstaub Hauptstrasse Steampunk-Design)
## newest box: 2018-05-24 07:08:32 (Josi Test)
## 
## sensors per box:
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##   1.000   4.000   4.000   4.618   5.000  22.000

plot(b3)

plot of chunk test

To re-download the data, just clear the files that were created in the process:

file.remove('mobileboxes.rds', 'boxes_bad.json', 'boxes.json', 'measurements.rds')

## Warning in file.remove("mobileboxes.rds", "boxes_bad.json", "boxes.json", :
## cannot remove file 'measurements.rds', reason 'No such file or directory'

A possible extension to this scheme comes to mind: Omit the specification of a filename, and assign a unique ID to the request instead. For example, one could calculate the SHA-1 hash of the parameters, and use it as filename.