Name Mappings

Name Mappings is an advanced global feature of Cygnus. It is global because it is available for all NGSI sinks.

Name Mappings allow changing the notified FIWARE service, FIWARE service path, entity IDs, entity types, attribute names and attribute types, given a mapping. Such a mapping is just a Json within a configuration file detailing how original naming must be replaced by alternative naming.

{
   "serviceMappings": [
      {
         "originalService": "myservice1",
         "newService": "new_myservice1",
         "servicePathMappings": [
            {
               "originalServicePath": "/myservicepath1",
               "newServicePath": "/new_myservicepath1",
               "entityMappings": [
                  {
                     "originalEntityId": "myentityid1",
                     "originalEntityType": "myentitytype1",
                     "newEntityId": "new_myentityid1",
                     "newEntityType": "new_myentitytype1",
                     "attributeMappings": [
                        {
                           "originalAttributeName": "myattributename1",
                           "originalAttributeType": "myattributetype1",
                           "newAttributeName": "new_myattributename1",
                           "newAttributeType": "new_myattributetype1"
                        },
                        ...
                     ]
                  },
                  ...
               ]
            },
            ...
         ]
      },
      ...
   ]
}

When a notification is sent to Cygnus, a special Flume interceptor called NGSINameMappingsInterceptor intercepts the plain Event's generated by NGSIRestHandler, parses the Event's body and iterate on the configured mappings in order to create a mapped version of the original notification. Once finished, both versions of the notification, original and mapped ones, are put into the channel buy means of a NGSIEvent, a Java object able to handle:

  • The original headers sent by NGSIRestHandler, i.e. fiware-service, fiware-servicepath, fiware-correlator and transaction-id.
  • New headers regarding the mapped FIWARE service and FIWARE service path, i.e. mapped-fiware-service and mapped-fiware-service-path.
  • The original notification already parsed as a NotifyContextRequest object.
  • The mapped version of the original notification as a NotifyContextRequest object.

Please observe no raw bytes about the body are sent.

Whenever a sink takes one of these NGSIEvent's, it is only a matter of deciding if such a sink enables the mappings (enable_name_mappings parameter) or not. If mappings are enabled, then the already parsed NotifyContextRequest, mapped version, is used. If not, then the original version is used.

Creating your own Name Mappings

Please observe the mappings definition is global to all the sinks, at NGSIRestHandler, as a Flume interceptor. Nevertheless, the application is local to the sink, depending on the enable_name_mappings parameter. Thus, if none of your sinks is going to take advantage of the mappings, simply avoid configuring the NGSINameMappingsInterceptor in NGSIRestHandler. That will avoid unnecessary interception and iterations on the mappings and Cygnus will perform faster.

$ cat /path/to/conf/agent.conf
cygnus-ngsi.sources.http-source.type = org.apache.flume.source.http.HTTPSource
cygnus-ngsi.sources.http-source.channels = hdfs-channel
cygnus-ngsi.sources.http-source ...
cygnus-ngsi.sources.http-source.handler = com.telefonica.iot.cygnus.handlers.NGSIRestHandler
cygnus-ngsi.sources.http-source.handler ...
cygnus-ngsi.sources.http-source.interceptors = ts nmi
cygnus-ngsi.sources.http-source.interceptors.ts.type = timestamp
cygnus-ngsi.sources.http-source.interceptors.nmi.type = com.telefonica.iot.cygnus.interceptors.NGSINameMappingsInterceptor$Builder
cygnus-ngsi.sources.http-source.interceptors.nmi.name_mappings_conf_file = /path/to/conf/name_mappings.conf

Additionally, please observe if any of the original names is not present, then the mapping affects all the names of that type. For instance, if originalService is not present in the mapping, then the mapping affects all the FIWARE services:

$ cat /path/to/conf/name_mappings.conf
{
   "serviceMappings": [
      {
         "servicePathMappings": [
            {
               "originalServicePath": "/myservicepath1",
               "newServicePath": "/new_myservicepath1",
               "entityMappings": [
                  {
                     "originalEntityId": "myentityid1",
                     "originalEntityType": "myentitytype1",
                     "newEntityId": "new_myentityid1",
                     "newEntityType": "new_myentitytype1",
                     "attributeMappings": [
                        {
                           "originalAttributeName": "myattributename1",
                           "originalAttributeType": "myattributetype1",
                           "newAttributeName": "new_myattributename1",
                           "newAttributeType": "new_myattributetype1"
                        }
                     ]
                  }
               ]
            }
         ]
      }
   ]
}

Another relevant behaviour must be noticed: when any of the new names is not present, then the original name is used in the mapping. I.e. there is no mapping. In this example, the original service path is never changed, since newServicePath is missing:

$ cat /path/to/conf/name_mappings.conf
{
   "serviceMappings": [
      {
         "originalService": "myservice1",
         "newService": "new_myservice1",
         "servicePathMappings": [
            {
               "originalServicePath": "/myservicepath1",
               "entityMappings": [
                  {
                     "originalEntityId": "myentityid1",
                     "originalEntityType": "myentitytype1",
                     "newEntityId": "new_myentityid1",
                     "newEntityType": "new_myentitytype1",
                     "attributeMappings": [
                        {
                           "originalAttributeName": "myattributename1",
                           "originalAttributeType": "myattributetype1",
                           "newAttributeName": "new_myattributename1",
                           "newAttributeType": "new_myattributetype1"
                        }
                     ]
                  }
               ]
            }
         ]
      }
   ]
}

Last but not least, the original names support Java-based regular expressions. For instance, if you want all the service paths are re-named equals simply use /.* as value for originalServicePath:

$ cat /path/to/conf/name_mappings.conf
{
   "serviceMappings": [
      {
         "originalService": "myservice1",
         "newService": "new_myservice1",
         "servicePathMappings": [
            {
               "originalServicePath": "/.*",
               "newServicePath": "/new_all",
               "entityMappings": [
                  {
                     "originalEntityId": "myentityid1",
                     "originalEntityType": "myentitytype1",
                     "newEntityId": "new_myentityid1",
                     "newEntityType": "new_myentitytype1",
                     "attributeMappings": [
                        {
                           "originalAttributeName": "myattributename1",
                           "originalAttributeType": "myattributetype1",
                           "newAttributeName": "new_myattributename1",
                           "newAttributeType": "new_myattributetype1"
                        }
                     ]
                  }
               ]
            }
         ]
      }
   ]
}

In addition, above mentioned Java-based regular expressions can be also used in new entity IDs. For instance, if we want to rename certain entity IDs described as a string concatenated with a number, and we want to replace the string but maintaining the number:

{
    "serviceMappings": [{
        "originalService": "service",
        "newService": "new_service",
        "servicePathMappings": [{
            "originalServicePath": "/subservice",
            "newServicePath": "/new_subservice",
            "entityMappings": [{
                "originalEntityType": "myentitytype",
                "originalEntityId": "(myentityid)([0-9]*)",
                "newEntityId": "new_myentityid$2",
                "attributeMappings": []
            }]
        }]
    }]
}

Sumarizing these are some useful examples and their result in a sink like MySQL:

Case 1: groups matching exactly by service, subservice, entityid and entitytype

  • Service Mapping:
{
  "serviceMappings": [
    {
      "originalService": "city012",
      "newService": "database_name",
      "servicePathMappings": [
        {
          "originalServicePath": "/electricidad",
          "newServicePath": "/tableprefix",
          "entityMappings": [
            {
              "originalEntityType": "luminaria",
              "newEntityType": "tablesufix",
              "originalEntityId": "luminaria_1",
              "newEntityId": "tableid",
              "attributeMappings": []
            }
          ]
        }
      ]
    }
  ]
}
  • Headers:
  Fiware-Service: city012
  Fiware-ServicePath: /electricidad
  • Entity:
{
   "id":"luminaria1",
   "type":"luminaria",
   ...
}

This case creates in database_name db a table with name tableprefix_tableid_tablesufix

Case 2: groups match exactly by service, subservice and entityid and any entitytype

  • Service Mapping:
{
  "serviceMappings": [
    {
      "originalService": "city012",
      "newService": "database_name",
      "servicePathMappings": [
        {
          "originalServicePath": "/electricidad",
          "newServicePath": "/tableprefix",
          "entityMappings": [
            {
              "originalEntityId": "pro(.+)",
              "newEntityId": "tablepro",
              "originalEntityType": ".+",
              "newEntityType": "tablesufix",
              "attributeMappings": []
            }
          ]
        }
      ]
    }
  ]
}
  • Headers:
  Fiware-Service: city012
  Fiware-ServicePath: /electricidad
  • Entity:
{
   "id":"pro1",
   "type":"luminaria",
   ...
}

This case creates in database_name db a table with name tableprefix_tablepro_tablesufix

Case 3: groups maching service by subservice and entitytype all entityid

  • Service Mapping:
{
  "serviceMappings": [
    {
      "originalService": "city012",
      "newService": "database_name",
      "servicePathMappings": [
        {
          "originalServicePath": "/(.+)",
          "newServicePath": "/$1",
          "entityMappings": [
            {
              "originalEntityId": "(.+)",
              "newEntityId": "any",
              "originalEntityType": "(.+)",
              "attributeMappings": []
            }
          ]
        }
      ]
    }
  ]
}
  • Headers:
  Fiware-Service: city012
  Fiware-ServicePath: /electricidad
  • Entity:
{
   "id":"pro1",
   "type":"luminaria",
   ...
}

This case creates in database_name db a table with name elecricidad_any_luminaria

Case 4: groups maching service by entitytype all subservice and entityid

  • Service Mapping:
{
  "serviceMappings": [
    {
      "originalService": "city012",
      "newService": "database_name",
      "servicePathMappings": [
        {
          "originalServicePath": "/(.+)",
          "newServicePath": "/global",
          "entityMappings": [
            {
              "originalEntityId": "(.+)",
              "newEntityId": "any",
              "originalEntityType": "(.+)",
              "attributeMappings": []
            }
          ]
        }
      ]
    }
  ]
}
  • Headers:
  Fiware-Service: city012
  Fiware-ServicePath: /electricidad
  • Entity:
{
   "id":"pro1",
   "type":"luminaria",
   ...
}

This case creates db database_name db a table with name global_any_luminaria

Case 5: groups maching service, subservice, entitytype, entityid and attribute name and attribute type

  • Service Mapping:
{
  "serviceMappings": [
    {
      "originalService": "city012",
      "newService": "database_name",
      "servicePathMappings": [
        {
          "originalServicePath": "/electricidad",
          "newServicePath": "/tableprefix",
          "entityMappings": [
            {
              "originalEntityType": "luminaria",
              "newEntityType": "tablesufix",
              "originalEntityId": "luminaria_1",
              "newEntityId": "tableid",
              "attributeMappings": [
                {
                  "originalAttributeName": "temperatura",
                  "originalAttributeType": "string",
                  "newAttributeName": "new_myattributename1",
                  "newAttributeType": "new_myattributetype1"
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}

And agent is configured with option data_model=dm-by-attribute

  • Headers:
  Fiware-Service: city012
  Fiware-ServicePath: /electricidad
  • Entity:
{
   "id":"pro1",
   "type":"luminaria",
   "temperatura": {
       "type": "string",
       "value": "13"
   }
}

This case creates in database_name db a table with name tableprefix_tableid_tablesufix_new_myattributename1

Case 6: groups maching service, entitytype, entityid and attribute name and attribute type in the same tableprefix

  • Service Mapping:
{
  "serviceMappings": [
    {
      "originalService": "city012",
      "newService": "database_name",
      "servicePathMappings": [
        {
          "newServicePath": "/tableprefix",
          "entityMappings": [
            {
              "originalEntityType": "luminaria",
              "newEntityType": "tablesufix",
              "originalEntityId": "luminaria_1",
              "newEntityId": "tableid",
              "attributeMappings": [
                {
                  "originalAttributeName": "temperatura",
                  "originalAttributeType": "string",
                  "newAttributeName": "new_myattributename1",
                  "newAttributeType": "new_myattributetype1"
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}

And agent is configured with option data_model=dm-by-attribute

  • Headers:
  Fiware-Service: city012
  Fiware-ServicePath: /electricidad
  • Entity:
{
   "id":"pro1",
   "type":"luminaria",
   "temperatura": {
       "type": "string",
       "value": "13"
   }
}

This case creates in database_name db a table with name tableprefix_tableid_tablesufix_new_myattributename1

Case 7: groups maching service, entitytype, attribute name and attribute type in the same tableprefix depending on entityid

  • Service Mapping:
{
  "serviceMappings": [
    {
      "originalService": "city012",
      "newService": "database_name",
      "servicePathMappings": [
        {
          "originalServicePath": "/electricidad",
          "newServicePath": "/tableprefix",
          "entityMappings": [
             {
              "originalEntityType": "luminaria",
              "newEntityType": "tablesufix",
              "originalEntityId": "([cabeco,isa]+)(.+)",
              "newEntityId": "$1",
              "attributeMappings": [
                {
                  "originalAttributeName": "temperatura",
                  "originalAttributeType": "string",
                  "newAttributeName": "new_myattributename1",
                  "newAttributeType": "new_myattributetype1"
                }
              ]
            },
             {
              "originalEntityType": "(.+)",
              "originalEntityId": ".+",
              "newEntityId": "othertype",
              "attributeMappings": [
              ]
             }
          ]
        }
      ]
    }
  ]
}

And agent is configured with option data_model=dm-by-attribute

  • Headers:
  Fiware-Service: city012
  Fiware-ServicePath: /electricidad
  • Entity:
{
   "id":"cabeco_33",
   "type":"luminaria",
   "temperatura": {
       "type": "string",
       "value": "13"
   }
}

This case creates in database_name db a table with name tableprefix_cabeco_tablesufix_new_myattributename1

  • Headers:
  Fiware-Service: city012
  Fiware-ServicePath: /electricidad
  • Entity:
{
   "id":"otherthing_22",
   "type":"container",
   "temperatura": {
       "type": "string",
       "value": "13"
   }
}

This case creates in database_name db a table with name tableprefix_othertype_tablesufix

Top

Name Mappings API

The contents of /path/to/conf/name_mappings.conf provide the name mappings used by Cygnus at startup time. However, they can be managed using a REST-based API. Using such API, new name mappings can be defined, modified or deleted.

The name mappings API is described in this piece of documentation.

Top

Name Mappings & Docker

When Cygnus was deployed using a docker image Installation via docker and namemappings files are provided in a volume, then updating these namemappings should be done in a way that i-node of that namemapping file would not be modified. This way cygnus will read namemappings changes without need of be restarted. To a achieve that is recomended not use editors like vim or emacs which uses temporal files while editing, and use simple editors like nano.

Top

Name Mappings vs. grouping rules

As seen, the Name Mappings feature is quite similar to the already existent grouping rules. Both of them are Flume interceptors and both of them allow changing certain notified name elements. Thus, which are the differences? Mainly:

Name Mappings Grouping rules
Allow changing the notified FIWARE service, FIWARE service path, entity IDs, entity types, attribute names and attribute types. Allow changing the notified FIWARE service path and the concatenation of entity ID and entity type (this is called the destination).
Plain Flume Event's are intercepted, and NGSIEvent's are put into the channel. Because the interceptor needs to parse the original notification, a NGSIEvent already contains the original notification parsed, and the mapped version of the original notification, freeing the sinks to parse the notification. Plain Flume Event's are intercepted, and plain Event's are put into the channel. Thus, the sinks must parse the notification, despite the grouping interceptor already parsed it`.
It is expected a enable_content_mappings feature is implemented in the future. Such a content mapping will take advantage of the already mapped version on the original notification within NGSIEvent's. Such a functionality is very hard to implement based on the current grouping interceptor code.

IMPORTANT NOTE: from release 1.6.0, Grouping Rules are deprecated in favour of Name Mappings. More details can be found here.

Top

Further reading

Please, check the specific documentation for this custom interceptor in the Flume extensions catalogue for cygnus-ngsi agent.

Top