Name Mappings

Name Mappings is an advanced global feature of Cygnus. It is global because it is available for all NGSI sinks.

Name Mappings allow changing the notified FIWARE service, FIWARE service path, entity IDs, entity types, attribute names and attribute types, given a mapping. Such a mapping is just a Json within a configuration file detailing how original naming must be replaced by alternative naming.

{
   "serviceMappings": [
      {
         "originalService": "myservice1",
         "newService": "new_myservice1",
         "servicePathMappings": [
            {
               "originalServicePath": "/myservicepath1",
               "newServicePath": "/new_myservicepath1",
               "entityMappings": [
                  {
                     "originalEntityId": "myentityid1",
                     "originalEntityType": "myentitytype1",
                     "newEntityId": "new_myentityid1",
                     "newEntityType": "new_myentitytype1",
                     "attributeMappings": [
                        {
                           "originalAttributeName": "myattributename1",
                           "originalAttributeType": "myattributetype1",
                           "newAttributeName": "new_myattributename1",
                           "newAttributeType": "new_myattributetype1"
                        },
                        ...
                     ]
                  },
                  ...
               ]
            },
            ...
         ]
      },
      ...
   ]
}

When a notification is sent to Cygnus, a special Flume interceptor called NGSINameMappingsInterceptor intercepts the plain Event's generated by NGSIRestHandler, parses the Event's body and iterate on the configured mappings in order to create a mapped version of the original notification. Once finished, both versions of the notification, original and mapped ones, are put into the channel buy means of a NGSIEvent, a Java object able to handle:

  • The original headers sent by NGSIRestHandler, i.e. fiware-service, fiware-servicepath, fiware-correlator and transaction-id.
  • New headers regarding the mapped FIWARE service and FIWARE service path, i.e. mapped-fiware-service and mapped-fiware-service-path.
  • The original notification already parsed as a NotifyContextRequest object.
  • The mapped version of the original notification as a NotifyContextRequest object.

Please observe no raw bytes about the body are sent.

Whenever a sink takes one of these NGSIEvent's, it is only a matter of deciding if such a sink enables the mappings (enable_name_mappings parameter) or not. If mappings are enabled, then the already parsed NotifyContextRequest, mapped version, is used. If not, then the original version is used.

Creating your own Name Mappings

Please observe the mappings definition is global to all the sinks, at NGSIRestHandler, as a Flume interceptor. Nevertheless, the application is local to the sink, depending on the enable_name_mappings parameter. Thus, if none of your sinks is going to take advantage of the mappings, simply avoid configuring the NGSINameMappingsInterceptor in NGSIRestHandler. That will avoid unnecessary interception and iterations on the mappings and Cygnus will perform faster.

$ cat /path/to/conf/agent.conf
cygnus-ngsi.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnus-ngsi.sources.http-source.channels = hdfs-channel
cygnus-ngsi.sources.http-source ...
cygnus-ngsi.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
cygnus-ngsi.sources.http-source.handler ...
cygnus-ngsi.sources.http-source.interceptors = ts nmi
cygnus-ngsi.sources.http-source.interceptors.ts.type = timestamp
cygnus-ngsi.sources.http-source.interceptors.nmi.type = com.telefonica.iot.cygnus.interceptors.NGSINameMappingsInterceptor$Builder
cygnus-ngsi.sources.http-source.interceptors.nmi.name_mappings_conf_file = /path/to/conf/name_mappings.conf

Additionally, please observe if any of the original names is not present, then the mapping affects all the names of that type. For instance, if originalService is not present in the mapping, then the mapping affects all the FIWARE services:

$ cat /path/to/conf/name_mappings.conf
{
   "serviceMappings": [
      {
         "servicePathMappings": [
            {
               "originalServicePath": "/myservicepath1",
               "newServicePath": "/new_myservicepath1",
               "entityMappings": [
                  {
                     "originalEntityId": "myentityid1",
                     "originalEntityType": "myentitytype1",
                     "newEntityId": "new_myentityid1",
                     "newEntityType": "new_myentitytype1",
                     "attributeMappings": [
                        {
                           "originalAttributeName": "myattributename1",
                           "originalAttributeType": "myattributetype1",
                           "newAttributeName": "new_myattributename1",
                           "newAttributeType": "new_myattributetype1"
                        }
                     ]
                  }
               ]
            }
         ]
      }
   ]
}

Another relevant behaviour must be noticed: when any of the new names is not present, then the original name is used in the mapping. I.e. there is no mapping. In this example, the original service path is never changed, since newServicePath is missing:

$ cat /path/to/conf/name_mappings.conf
{
   "serviceMappings": [
      {
         "originalService": "myservice1",
         "newService": "new_myservice1",
         "servicePathMappings": [
            {
               "originalServicePath": "/myservicepath1",
               "entityMappings": [
                  {
                     "originalEntityId": "myentityid1",
                     "originalEntityType": "myentitytype1",
                     "newEntityId": "new_myentityid1",
                     "newEntityType": "new_myentitytype1",
                     "attributeMappings": [
                        {
                           "originalAttributeName": "myattributename1",
                           "originalAttributeType": "myattributetype1",
                           "newAttributeName": "new_myattributename1",
                           "newAttributeType": "new_myattributetype1"
                        }
                     ]
                  }
               ]
            }
         ]
      }
   ]
}

Last but not least, the original names support Java-based regular expressions. For instance, if you want all the service paths are re-named equals simply use /.* as value for originalServicePath:

$ cat /path/to/conf/name_mappings.conf
{
   "serviceMappings": [
      {
         "originalService": "myservice1",
         "newService": "new_myservice1",
         "servicePathMappings": [
            {
               "originalServicePath": "/.*",
               "newServicePath": "/new_all",
               "entityMappings": [
                  {
                     "originalEntityId": "myentityid1",
                     "originalEntityType": "myentitytype1",
                     "newEntityId": "new_myentityid1",
                     "newEntityType": "new_myentitytype1",
                     "attributeMappings": [
                        {
                           "originalAttributeName": "myattributename1",
                           "originalAttributeType": "myattributetype1",
                           "newAttributeName": "new_myattributename1",
                           "newAttributeType": "new_myattributetype1"
                        }
                     ]
                  }
               ]
            }
         ]
      }
   ]
}

In addition, above mentioned Java-based regular expressions can be also used in new entity IDs. For instance, if we want to rename certain entity IDs described as a string concatenated with a number, and we want to replace the string but maintaining the number:

{
    "serviceMappings": [{
        "originalService": "service",
        "newService": "new_service",
        "servicePathMappings": [{
            "originalServicePath": "/subservice",
            "newServicePath": "/new_subservice",
            "entityMappings": [{
                "originalEntityType": "myentitytype",
                "originalEntityId": "(myentityid)([0-9]*)",
                "newEntityId": "new_myentityid$2",
                "attributeMappings": []
            }]
        }]
    }]
}

Top

Name Mappings vs. grouping rules

As seen, the Name Mappings feature is quite similar to the already existent grouping rules. Both of them are Flume interceptors and both of them allow changing certain notified name elements. Thus, which are the differences? Mainly:

Name Mappings Grouping rules
Allow changing the notified FIWARE service, FIWARE service path, entity IDs, entity types, attribute names and attribute types. Allow changing the notified FIWARE service path and the concatenation of entity ID and entity type (this is called the destination).
Plain Flume Event's are intercepted, and NGSIEvent's are put into the channel. Because the interceptor needs to parse the original notification, a NGSIEvent already contains the original notification parsed, and the mapped version of the original notification, freeing the sinks to parse the notification. Plain Flume Event's are intercepted, and plain Event's are put into the channel. Thus, the sinks must parse the notification, despite the grouping interceptor already parsed it`.
It is expected a enable_content_mappings feature is implemented in the future. Such a content mapping will take advantage of the already mapped version on the original notification within NGSIEvent's. Such a functionality is very hard to implement based on the current grouping interceptor code.

IMPORTANT NOTE: from release 1.6.0, Grouping Rules are deprecated in favour of Name Mappings. More details can be found here.

Top

Further reading

Please, check the specific documentation for this custom interceptor in the Flume extensions catalogue for cygnus-ngsi agent.

Top