Event delivery is a fundamental piece of any monitoring solution. Continual's Rcvr API is where events typically enter the solution and are routed for processing.

We want the mechanics of injecting an event to be dead simple and the API reflects this: it's a simple HTTP POST of event data sent in a common format (e.g. JSON, web form, or plain text).

Concepts

The Rcvr API does not dictate a particular message broker technology, but it does use the concepts of topics and event streams.

Events related to the same basic subject (e.g. bank transactions, SNMP traps, weather updates) belong on the same topic.  This gives downstream event processors a single place to subscribe to events about that subject.

Event delivery order is often important, especially if events are carrying state updates. For example, when working with timestamped data, out-of-order arrival is harmless. The data is generally just pushed to a time-series database and indexed on timestamp, so the data is re-ordered later.

Stateful event streams, on the other hand, require order guarantees.  Suppose a Smart Home monitoring system reports that a door has opened, and a few seconds later that the door has closed. If downstream systems receive these events out of order, their understanding of the state of the "real world" will be out-of-synch.

There are various strategies for retaining message order for downstream delivery, but many of them struggle to scale out.  For the Rcvr API, we borrow Apache Kafka's topic partitioning concept and allow event senders to specify an event stream name.  The Rcvr implementation is expected to guarantee that all events with the same event stream name are delivered to downstream consumers in the order they were received. (If you're familiar with Kafka, the "event stream name" here is equivalent to Kafka's "partition key.")

Paths

When sending data to the Rcvr service, client systems HTTP POST their data to:

POST /events/{topic}/{streamName}

Events in the payload are pushed to the given topic using the given event stream name.

For scenarios where event ordering is not important, senders can use a simpler path:

POST /events/{topic}

Finally, events can be pushed into a default topic by leaving the topic name out of the path:

POST /events

Formats

Internally, the Rcvr API delivers events encoded as JSON objects. Inbound events are converted from their payload format into JSON objects as follows:

application/json

The payload may be a JSON array or a single object.  If it's an array, each member is considered an individual event. Any array member that's not an object will be converted to an object with a single member, "message" that has the original member value as its value.

text/plain

The entire payload is considered a single event. The event object has a single member "message" with the text payload as its value.

application/x-www-form-urlencoded
multipart/form-data

The form post is converted to an event object in which each <form> field is a member with its respective stringified value.  File upload data in a multipart/form-data post is currently ignored.

Other Considerations

  • Calls to the Rcvr API are authenticated, enabling multi-tenant use.
  • The Rcvr API does not require a specific message broker technology. Our codebase includes a Kafka-backed implementation but any topic-oriented broker with message ordering guarantees could work.