- Installing Apache Flume
- Installing cygnus-common
- Installing dependencies
Maven (and thus Java SDK, since Maven is a Java tool) is needed in order to install cygnus-common.
In order to install Java SDK (not JRE), just type (CentOS machines):
$ yum install java-1.7.0-openjdk-devel
Remember to export the JAVA_HOME environment variable. In the case of using
yum install as shown above, it would be:
$ export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk.x86_64
In order to do it permanently, edit
root user) or
/etc/profile (other users).
Maven is installed by downloading it from maven.apache.org. Install it in a folder of your choice (represented by
$ wget http://www.eu.apache.org/dist/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.tar.gz $ tar xzvf apache-maven-3.2.5-bin.tar.gz $ mv apache-maven-3.2.5 APACHE_MAVEN_HOME
It is highly recommended to create a
cygnus Unix user, under which Cygnus will be installed and run. By the way, this is how the RPM proceeds.
Creating such a user is quite simple. As a sudoer user (root or any other allowed), type the following:
$ (sudo) useradd cygnus
You may add a password or not to the
$ (sudo) passwd cygnus
Once created, change to this new fresh user in order to proceed with the rest of the installation:
$ su - cygnus
Once the user is created is necessary to create the path
log4j purposes. Start by creating the path and then give permissions for
$ mkdir -p /var/log/cygnus $ chown -R cygnus:cygnus /var/log/cygnus
This step is important because if you don't have the log path created Cygnus will shutdown when running for the first time.
Apache Flume can be easily installed by downloading its latests version from flume.apache.org. Move the untared directory to a folder of your choice (represented by
$ wget http://www.eu.apache.org/dist/flume/1.4.0/apache-flume-1.4.0-bin.tar.gz $ tar xvzf apache-flume-1.4.0-bin.tar.gz $ mv apache-flume-1.4.0-bin APACHE_FLUME_HOME $ mv APACHE_FLUME_HOME/lib/httpclient-4.2.1.jar APACHE_FLUME_HOME/lib/httpclient-4.2.1.jar.old $ mv APACHE_FLUME_HOME/lib/httpcore-4.2.1.jar APACHE_FLUME_HOME/lib/httpcore-4.2.1.jar.old $ mv APACHE_FLUME_HOME/lib/libthrift-0.7.0.jar APACHE_FLUME_HOME/lib/libthrift-0.7.0.jar.old $ mkdir -p APACHE_FLUME_HOME/plugins.d/cygnus/ $ mkdir APACHE_FLUME_HOME/plugins.d/cygnus/lib $ mkdir APACHE_FLUME_HOME/plugins.d/cygnus/libext
- The creation of the
plugins.ddirectory is related to the installation of third-party software, like Cygnus.
- Please observe the version of
httpclient-4.3.1.jar and httpcore-4.3.1.jar) don't match match the version of such packages within the Flume bundle (
httpclient-4.2.1.jar and httpcore-4.2.1.jar). In order the most recent version of the libraries, the ones within the Flume bundle must be removed (or renamed).
Start by cloning the Github repository containing cygnus-common:
$ git clone https://github.com/telefonicaid/fiware-cygnus.git $ cd fiware-cygnus $ git checkout <branch>
<branch> should be typically a stable release branch, e.g.
release/0.13.0, but could also be
master (synchronized with the latest release) or
develop (contains the latest not stable changes).
The developed classes must be packaged in a Java jar file. This can be done as a fat Java jar containing all the third-party dependencies (recommended). You may need to edit the
$ cd cygnus-common $ APACHE_FLUME_HOME/bin/mvn clean compile exec:exec assembly:single $ cp target/cygnus-common-<x.y.z>-jar-with-dependencies.jar APACHE_FLUME_HOME/plugins.d/cygnus/libext
Or as a thin Java jar file containing only the
cygnus-common classes. You may need to edit the
pom.xml if necessary (*):
$ cd cygnus-common $ APACHE_MAVEN_HOME/bin/mvn exec:exec package $ cp target/cygnus-<x.y.z>.jar APACHE_FLUME_HOME/plugins.d/cygnus/libext
Please observe in this case, and depending on the Cygnus components you are going to use, you may need to install additional third-party .jar files under
APACHE_FLUME_HOME/plugins.d/cygnus/libext/. Typically, you can get these jar files from your Maven repository (under
.m2 in your user home directory) and use the
In both cases, the
cygnus-common dependency must be installed at Maven as well, in order to build any other agent depending on it. Just run the following command:
$ mvn install:install-file -Dfile=APACHE_MAVEN_HOME/plugins.d/cygnus/libext/cygnus-common-<x.y.z>-jar-with-dependencies.jar -DgroupId=com.telefonica.iot -DartifactId=cygnus-common -Dversion=<x.y.z> -Dpackaging=jar -DgeneratePom=false
(*) Please have into account from Cygnus 0.10.0 the version of
hadoop-core within the
pom.xml must match the Hadoop version you are going to use; on the contrary, the HDFS sink will not work. Of course, if you are not going to use the HDFS sink, simply use the default
hadoop-core version (1.2.1) within the downloaded
pom.xml for correct compilation purposes.
Finally, please find a
compile.sh script containing all the commands shown in this section. It must be parameterized with the version of the current branch and the Apache Flume base path.
The installation is completed by copying the
cygnus-flume-ng script into
$ cp target/classes/cygnus-flume-ng APACHE_FLUME_HOME/bin $ chmod a+x APACHE_FLUME_HOME/bin/cygnus-flume-ng
It may happen while compiling
cygnus-common the Maven JVM has not enough memory. This can be changed as detailed at the Maven official documentation:
$ export MAVEN_OPTS="-Xmx512m -XX:MaxPermSize=128m"
These are the packages you will need to install under
APACHE_FLUME_HOME/plugins.d/cygnus/libext/ if you did not included them in the Cygnus jar:
|Cygnus dependencies||Version||Required by / comments|
|httpclient||4.3.1||Overwrites the one bundled in Apache Flume|
|httpcore||4.3.1||Overwrites the one bundled in Apache Flume|
|libthrift||0.9.1||Overwrites the one bundled in Apache Flume|
|json-simple||1.1||Management Interface and Grouping Rules|