Kafka backend

KafkaBackend interface

This class enumerates the methods any Kafka backend implementation must expose. In this case, the following ones:

boolean topicExists(String topic) throws Exception;

Gets if the given topic exists.

void createTopic(String topic, int partitions, int replicationFactor);

Creates a topic given its name, number of partitions and replication factor.

void send(ProducerRecord<String, String> record);

Sends a record to Kafka. A record is composed by a topic name and the data to be send.

KafkaBackendImpl class

This is a convenience backend class for CKAN that implements the KafkaBackend interface described above.

KafkaBackendImpl really wraps the KafkaProducer and AdminUtils Java classes.

Nothing special is done with regards to the encoding. Since Cygnus generally works with UTF-8 character set, this is how the data is written into the topics. It will responsability of the Kafka consumer to convert the bytes read into UTF-8.