Cygnus configuration examples

Content:

Single source, single storage (basic configuration)

To be done

Top

Single source, multiple storages

To be done

Top

Single source, single storage, parallel sinking

To be done

Top

Single source, multiple storages, parallel sinking

To be done

Top

Multiple sources

To be done

Top

Using interceptors

Interceptors are components of the Flume agent architecture. Typically, such an agent is based on a source dealing with the input, a sink dealing with the output and a channel communicating them. The source processes the input, producing Flume events (an object based on a set of headers and a byte-based body) that are put in the channel; then the sink consumes the events by getting them from the channel. This basic architecture may be enriched by the addition of Interceptors, a chained sequence of Flume events preprocessors that intercept the events before they are put into the channel and performing one of these operations:

  • Drop the event.
  • Modify an existent header of the Flume event.
  • Add a new header to the Flume event.

Interceptors should never modify the body part. Once an event is preprocessed, it is put in the channel as usual.

As can be seen, this mechanism allows for very useful ways of enriching the basic Flume events a certain Flume source may generate. Let's see how Cygnus makes use of this concept in order to add certain information to the Flume events created from the Orion notifications.

Top