Grouping Rules

IMPORTANT NOTE: from release 1.6.0, this feature is deprecated in favour of Name Mappings. More details can be found here.

Grouping rules are an advanced global feature of Cygnus. It is global because it is available for all NGSI sinks. They allow changing the notified FIWARE service path and the concatenation of entity ID and entity type (this is called the destination).

Such rules are just a Json within a configuration file (that can be instantiated from grouping_rules.conf.template) located at conf/. The following code is just an example:

    "grouping_rules": [
            "id": 1,
            "fields": [
            "regex": "...",
            "destination": "...",
            "fiware_service_path": "..."

When a notification is sent to Cygnus, a special Flume interceptor called NGSIGroupingInterceptor intercepts the plain Flume Event generated by NGSIRestHandler, parses the Event’s body and adds certain new headers (these headers are lists, having one element per notified context element within the notification): the default destinations, the grouped service paths and the grouped destinations. Thus, a sink may choose among default o grouped version of the destination and the grouping. Once finished, the modified event is put again into the channel as a native Flume Event.

Explained fields of a Grouping Rule:

  • id: A unique unsigned integer-based identifier.
  • fields: These are the fields that will be concatenated for regular expression matching. The available dictionary of fields for concatenation is "entityId", "entityType" and "servicePath". The order of these fields is important since the concatenation is made from left to right.
  • regex: Java-like regular expression to be applied on the concatenated fields. Special characters like '\' must be escaped ('\' is escaped as "\\").
  • destination: Name of the HDFS file or CKAN resource where the data will be effectively persisted. In the case of MySQL, Mongo and STH Comet this sufixes the table/collection name.
  • fiware_service_path: New fiware-servicePath replacing the notified one. The sinks will translate this into the name of the HDFS folder or CKAN package where the above destination entity will be placed. In the case of MySQL, Mongo and STH Comet this prefixes the table/collection name. It must start with / or the whole rule will be discarded.

Important notes

Please observe the grouping rules definition is global to all the sinks, at NGSIRestHandler, but then the application is local to the sink, depending on the enable_grouping parameter. Thus, if any of your sinks is going to take advantage of the grouping rules, simply leave blank the grouping rules configuration file in the REST handler. That will avoid unnecessary interception with grouping actions.

Grouping rules vs Name mappings

As seen, the name mappings feature is quite similar to the already existent grouping rules. Both of them are Flume interceptors and both of them allow changing certain notified name elements. Thus, which are the differences? Mainly:

Grouping rules Name mappings
Allow changing the notified FIWARE service path and the concatenation of entity ID and entity type (this is called the destination). Allow changing the notified FIWARE service, FIWARE service path, entity IDs, entity types, attribute names and attribute types.
Plain Flume Event's are intercepted, and plain Event's are put into the channel. Thus, the sinks must parse the notification, despite the grouping interceptor already parsed it. Plain Flume Event's are intercepted, and NGSIEvent's are put into the channel. Because the interceptor needs to parse the original notification, a NGSIEvent already contains the original notification parsed, and the mapped version of the original notification, freeing the sinks to parse the notification.
Such a functionality is very hard to implement based on the current grouping interceptor code. It is expected a enable_content_mappings feature is implemented in the future. Such a content mapping will take advantage of the already mapped version on the original notification within NGSIEvent's.


Further reading

Please, check the specific documentation for this custom interceptor in the Flume extensions catalogue for cygnus-ngsi agent.