Flume extensions catalogue
This document details the catalogue of extensions developed for Cygnus on top of Apache Flume.
Intended audience
The Flume extensions catalogue is a basic piece of documentation for all those FIWARE users using Cygnus. It describes the available extra components added to the Flume technology in order to deal with Twitter-like data.
Software developers may also be interested in this catalogue since it may guide the creation of new components (specially, sinks) for Cygnus/Flume.
Structure of the document
This document describes the Twitter Source and Twitter HDFS sink. TwitterSource
is a source designed to collect data from Twitter. This document contains an explanation about TwitterSource
configuration and functionality. TwitterHDFSSink
sink is currently the only one supported by the cygnus-twitter agent. This document contains an explanation about TwitterHDFSSink
functionality (including how the information within a Flume event is mapped into the storage data structures), configuration, uses cases and implementation details are given.