Accessing Public Heatmap and Incident Information

There are three main public API calls exposed by the e-mission API server. All of them share the following characteristics: - they are HTTP POST calls - they do not currently require an API key for access. If we get popular enough to be the target for DDOS attacks, we will revisit this decision. - they take a time range as input. The time range can be specified either in UTC, as a range, or in local time, as a filter.

These APIs are all currently very slow, because the prototype system is running on a single AWS host. The performance will improve after Nov 1st, when I split the functionality among multiple hosts.

The three public API calls are: - /result/heatmap/pop.route/<time_type> - /result/heatmap/incidents/<time_type> - /result/metrics/<time_type>

The <time_type> is either timestamp or local_date. The actual query range is specified in POST fields.

Without any further ado, here are some examples of how to use these calls. I've chosen python as the language for demonstrating them since most data scientists are familiar with it, but any language that can send http post requests will work as well.

Travel point queries

Retrieving data from 2nd Oct to 3rd Oct across all modes and the entire world

$ ipython
Python 2.7.11 |Anaconda 2.0.1 (x86_64)| (default, Dec  6 2015, 18:57:58)
Type "copyright", "credits" or "license" for more information.

IPython 4.2.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import arrow

In [2]: start_ts = arrow.get("2017-10-02").timestamp

In [3]: end_ts = arrow.get("2017-10-03").timestamp

In [4]: post_fields = {
        "modes": None,
        "start_time": start_ts,
        "end_time": end_ts,
        "sel_region": None
}

In [5]: import requests

In [6]: result = requests.post("https://e-mission.eecs.berkeley.edu/result/heatmap/pop.route/timestamp", json=post_fields)

In [7]: result.status_code
Out[7]: 200

In [8]: result.json()['lnglat'][0:5]
Out[8]:
[[-117.07978918462588, 32.769079182443484],
 [-122.26395704827273, 37.87331020231461],
 [-110.77774474290837, 35.15781189961038],
 [-110.77774474290837, 35.15781189961038],
 [-110.77774474290837, 35.15781189961038]]

In [9]: len(result.json()['lnglat'])
Out[9]: 16607

Modifying the prior query to only retrieve walk and bike data

In [10]: post_fields["modes"] = ["WALKING", "ON_FOOT", "BICYCLING"]

In [11]: result = requests.post("https://e-mission.eecs.berkeley.edu/result/heatmap/pop.route/timestamp", json=post_fields)

In [12]: result.status_code
Out[12]: 200

In [13]: len(result.json()['lnglat'])
Out[13]: 3312

Modifying the prior query to only retrieve data around Hearst Street in Berkeley, CA

In [15]: result = requests.post("https://e-mission.eecs.berkeley.edu/result/heatmap/pop.route/timestamp", json=post_fields)

In [16]: len(result.json()['lnglat'])
Out[16]: 112

Incident queries

Incident queries are very similar to heatmap queries, only sent to a different URL. Some other differences are: - they don't support modes - since they are user-reported and not collected in the background, there are typically few of them.

Retrieving all incidents from June to Oct

$ ipython
Python 2.7.11 |Anaconda 2.0.1 (x86_64)| (default, Dec  6 2015, 18:57:58)
Type "copyright", "credits" or "license" for more information.

IPython 4.2.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import arrow

In [2]: start_ts = arrow.get("2017-06-01").timestamp

In [3]: end_ts = arrow.get("2017-10-01").timestamp

In [4]: post_fields = {
        "start_time": start_ts,
        "end_time": end_ts,
        "sel_region": None
}

In [5]: import requests

In [7]: result = requests.post("https://e-mission.eecs.berkeley.edu/result/heatmap/incidents/timestamp", json=post_fields)

In [8]: result.status_code
Out[8]: 200

In [9]: len(result.json()["incidents"])
Out[9]: 42

In [10]: result.json()["incidents"][0:5]
Out[10]:
[{u'fmt_time': u'2017-06-09T10:51:41-07:00',
  u'loc': {u'coordinates': [-122.06568002700806, 37.36993549376299],
   u'type': u'Point'},
  u'local_dt': {u'day': 9,
   u'hour': 10,
   u'minute': 51,
   u'month': 6,
   u'second': 41,
   u'timezone': u'America/Los_Angeles',
   u'weekday': 4,
   u'year': 2017},
  u'stress': 0,
  u'ts': 1497030701},
 {u'fmt_time': u'2017-06-09T10:04:41-07:00',
  u'loc': {u'coordinates': [-122.07375526777469, 37.38112434837758],
   u'type': u'Point'},
  u'local_dt': {u'day': 9,
   u'hour': 10,
   u'minute': 4,
   u'month': 6,
   u'second': 41,
   u'timezone': u'America/Los_Angeles',
   u'weekday': 4,
   u'year': 2017},
  u'stress': 0,
  u'ts': 1497027881},
....

This query can be also modified to return incidents for a particular area. Note that incidents cannot be filtered by mode, so any mode field in the query is ignored.