REST API

The REST API provides ways to create, update, query

Create

Create a new trend with a name, size and entities.

GET /1/create/name/entities
Required Authentication: signature

Path Components

name
The name of the trend to be created. Must be unique in this instance. Names should consist of characters allowed in a URL only.
entitites
The entities may only contain alphanumerical characters A-Z, a-z, 0-9 and the _ (underscore). Multiple entities may be separated by a : (colon).
Example: user_name:city

Parameters

size
optional, defaults to 10.000
The size of this trend. This gives the maximum length of your top items. It should be selected by taking a useful fraction of your maximum possible keys that may appear.
Example: size=100000
timescales
optional, defaults to "day,hour,minute"
Defines the time scales available in this trend. Possible values are comma separated lists of week,day, hour, minute and second. Timescales should be named from left to right where the leftmost is the longest time span. The leftmost timescale defines how long objects are kept in the trend.
Example: timescales=hour,minute

Example

Create a new trend of which family names are most present in which cities of a size of 1.000.000 with weekly, daily, hourly and very short 60s trends. In a very mobile society this would provide you with some information of family movements between cities.

GET /1/create/where-people-live/family_name:city?size=1000000&timescales=week,day,hour,minute

Update

Create a new trend with a name, size and entities.

GET /1/update/name/keys/value
Required Authentication: token or signature

Path Components

name
The name of the trend that is updated.
keys
The keys that should be set in this trend, separated by : (colon). The individual keys must be URL encoded if they contain non-URL characters or an extra-colon.
Example: Stephan%20Schmidt:Berlin
value
optional, defaults to 1.0
An optional double value that is to be used to calculate the score if it is different from 1.0. This value may be used to get average numbers for keys instead of just hits per timescale.

Parameters

ts
optional, defaults to current time on server
The time stamp of the event, encoded as milliseconds since midnight, January 1, 1970.
Example: ts=1357569425925 Set timestamp to 14:37:02 UTC, Jan 7, 2013.

Examples

A simple update:
GET /1/update/where-people-live/Stephan%20Schmidt:Berlin?ts=day
Updating to a specific value (4 new people arrived right now):
GET /1/update/average-people/Schmidt/4
Updating that 4 new people arrived, not right now, but on Jan 7, 2013, 14:37:02 UTC:
GET /1/update/average-people/Schmidt/4?ts=1357569425925

Query

Query the results from a trend.

GET /1/query/name
Required Authentication: signature

Path Components

name
The name of the trend that is queried.

Parameters

count
optional, defaults to the 20
The number of elements returned in this query.
Example: count=100
offset
optional, defaults to the 0
For paging, starts the results at the specified offset.
Example: offset=20
timescale
optional, defaults to the first time scale
The time scale you would like to update.
Example: timescale=hour
name of entity
Filter to entries which match the value for that entity.
Example: city=Berlin Only show results for Berlin.

Examples

Ask for the top 10:
GET /1/query/songs?count=10
Find the top 100 songs for the last week:
GET /1/query/songs?count=100&timescale=week
Find top 10 family names for Berlin:
GET /1/query/where-people-live?city=Berlin&count=10
Find top 10 cities for family name "Schmidt"
GET /1/query/where-people-live?family_name=Schmidt&count=10

These will return a structure containing the timestamp of the last update, and the actual trend list:

{
  "ts":1373882656991,
  "trend":[
    {"keys":["paula"],"score":4049.867495171355},
    {"keys":["justin"],"score":4016.8600688497877},
    {"keys":["frank"],"score":3998.862200478933},
    {"keys":["adele"],"score":3974.8609627587707},
    {"keys":["paul"],"score":3960.861581618778}
  ],
}
  
All timestamps (ts) are in Milliseconds since 1970 and based on the UTC timezone.

Scores

Query single entity scores.

POST /1/query/name/score
Required Authentication: signature

Path Components

name
The name of the trend that is queried.

Parameters

timescale
optional, defaults to the first time scale
The time scale you would like to update.
Example: timescale=hour
ts
optional, a time stamp as a long value
Example: ts=1234567890 Show results for this time stamp.
This REST method requires a message body containing the queried entitites as a list. Each entity should be a on a single line. It will return a JSON encoded array similar to the trend query with the entity keys and scores in a map.

Example

POST /1/query/myTrend/score

Schmidt
Braun
Jugel
    
This query will return
[
  {"keys":["Schmidt"],"score": 1.23},
  {"keys":["Braun"],"score": 3.2},
  {"keys":["Jugel"],"score": 0.2}
]
    

Clear

Clear a trend from all data, but do not delete it. This will free resources for new incoming data.

DELETE /1/clear/name
Required Authentication: signature
Important! This will irreversibly remove all data from the trend. Unless you have a snapshot of the instance you will not be able to recover it.

Path Components

name
The name of the trend that is to be cleared.

Example

In case you just tested the trending and want to start over with fresh data, clear it:
DELETE /1/clear/songs

Delete

Delete a trend from this instance. This will delete all data and free the resources.

DELETE /1/delete/name
Required Authentication: signature
Important! This will irreversibly remove the trend and all it's data from this instance. Unless you have a snapshot of the instance you will not be able to recover it.

Path Components

name
The name of the trend that is to be deleted.

Example

Wrongly configured or simply just not needed, delete it:
DELETE /1/delete/songs

Meta

Each trend can have meta-data attached to it. Currently this is used only for setting the link template used in the dashboard trend view.

GET /1/meta/name/property
Required Authentication: signature

Path Components

name
The name of the trend that you want to get or set a property on.
property
A property name to be set or queries. Currently only linkTemplate is supported. You can use it also for your own meta-data like a key/value store. Remember that it may require resources you want to use for your trends.

Parameters

value
optional
Only required for setting a property value, else the current value is returned.
Example: value=http://twitter.com/$1/status/$1

Examples

Query the current value of a property:
GET /1/meta/linkTemplate
Set a special link template for a twitter analysis:
This special template will replace $1 with the users screen_name and $2 with the tweet id.
GET /1/meta/linkTemplate?value=http://twitter.com/$1/status/$1

Authentication Basics

streamdrill basically does all authorization using a header based signature, but for some cases it also accepts an API token. The token is useful for javascript induced events where an anonymous web browser should ping the trend engine. For performance reasons it is advised to use the header signature together with streaming. If used internally on a secure network, the token based authentication is acceptable.

Header Signature

You can send your data using a simple HTTP connection. The authentication is similar to other REST services, like AWS Request Authentication, and just adds a signature to the header. streamdrill generates an apiKey and an apiSecret for you upon your first login. You need both to sign the request, which looks the similar to this:

GET /1/create/trend HTTP/1.1
Authorization: TPK 09c8baae-0eba-402f-9bde-4b843379be68:dLeE7QE0tZdNG6uE+GxwYGkr96k=
Date: Thu, 29 Nov 2012 11:55:27 +0000

The signature is created using the following code, concatenating the method (GET, POST, DELETE ...), the current date and the url-path without the query parameters delimited by newlines. Then the signature is generated by using HMAC with the apiSecret encoding the result Base64.

def sign(method: String, date: String, url: String, apiSecret: String) = {
  val secretKey = new SecretKeySpec(apiSecret.getBytes("UTF-8"), "HmacSHA1")
  val mac = Mac.getInstance("HmacSHA1")
  mac.init(secretKey)
  val message = method + "\n" + date + "\n" + url
  Base64.encodeBase64String(mac.doFinal(message.getBytes("UTF-8")))
}

The resulting signature is set in the Authorization together with the Date header:

Authorization: TPK apiKey:signature
Date: EEE, dd MMM yyyy HH:mm:ss zzz

Token Authentication

In cases where external code needs to send messages an API token for authorization is acceptable. Only the update API call accepts token authentication. All other API calls, like create,query or delete require a valid request signature. An API token request looks like this:

GET /1/update/trend/foo:bar HTTP/1.1
Authorization: APITOKEN 1c182c7f-40f0-45ca-8d55-7c5fad930173

Each time you create a new table/trend with the create method, a token will be returned that may be used for token authentication. All that is necessary to allow a client to make the request is adding the following header to the outgoing HTTP GET request:

Authorization: APITOKEN api-token