Skip to content

Lightweight connector that enables context data sharing from Orion Context Broker towards Apache Kafka

License

Notifications You must be signed in to change notification settings

Engineering-Research-and-Development/o2k-connector

Folders and files

NameName
Last commit message
Last commit date

Latest commit

e4a7541 · Oct 26, 2023

History

37 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

O2K-Connector: a FIWARE Orion to Kafka Connector

License badge
Quay badge Docker badge
Status

O2K-Connector is a lightweight connector, built in Python, that enables context data sharing from Orion Context Broker towards Apache Kafka. In more detail, O2K-Connector subscribes to Orion Context Broker and publishes the received context-data to a built-in kafka topic.

quay.io 🐳 Docker Hub

O2K-Connector example placement in a FIWARE/Apache Architecture

O2K-Connector-placement

Contents

Configuration

O2K-Connector can be fully customized and configured by the means of environment variables listed on o2k.env file.

Environment variables Description
LOG_LEVEL Level of O2K Debug. Possible values: INFO, DEBUG, ERROR
LOGS_PATH Path of O2K-Connector logs
O2K_HOST Listener for O2K-Connector
O2K_PORT Port of O2K-Connector
SCHEMA_URL Kafka Schema url for avro messages
BOOTSTRAP_SERVERS Kafka Boostrap Servers
KAFKA_TOPIC User-specified Kafka Topic, if left blank, it will be auto-generated on the fly taking into account FIWARE_SERVICE and FIWARE_SERVICEPATH environments for NGSI V2 and entity id for NGSI-LD
ORION_HOST Hostname of Orion Context Broker
ORION_PORT Port of Orion Context Broker
FIWARE_SERVICE Fiware Service for NGSI V2
FIWARE_SERVICEPATH Fiware ServicePath for NGSI V2s
MQTT_HOST Hostname of MQTT Server, if used
MQTT_PORT Port of MQTT Server, if used
MQTT_TOPIC Topic to which publish on MQTT Server, if used
SUBSCRIPTION_JSON_PATH Path of subscription configuration files
SUBSCRIPTION_JSON_FILENAME Filename of subscription configuration file for NGSI V2 subscription
SUBSCRIPTION_JSON_FILENAME_LD Filename of subscription configuration file for NGSI-LD subscription
SUBSCRIPTION_JSON_FILENAME_MQTT Filename of subscription configuration file for NGSI-V2 subscription and MQTT underlying subscription engine
SUBSCRIPTION_SCHEMA_FILE_PATH Filename of subscription schema file for NGSI V2 subscription
SUBSCRIPTION_SCHEMA_FILE_PATH_LD Filename of subscription schema file for NGSI-LD subscription
SUBSCRIPTION_SCHEMA_FILE_PATH_MQTT Filename of subscription schema file for NGSI-V2 subscription and MQTT underlying subscription engine
ORION_VERSION Orion Context Broker version. Possible values: V2, LD
ORION_SUBSCRIPTION Orion Context Broker subscription engine. Possible values: http, mqtt
MULTIPLE_SUBSCRIPTIONS For NGSI V2 only: set to "false" to handle a unique subscription with multiple attributes
KAFKA_ENABLE_SSL Set to "true" if want to use SSL connection to Kafka
KAFKA_SSL_CA Certificate Authority key (path file inside the container) to use for SSL connection to Kafka
KAFKA_SSL_KEY Client key (path file inside the container) to use for SSL connection to Kafka
KAFKA_SSL_CERTIFICATE Certificate key (path file inside the container) to use for SSL connection to Kafka

Getting Started - Install

To instantiate O2K-Connector you can use docker-compose which will take care of the creation of the container running python. Alternatively you can locally run the connector by running the python script.

N.B: The connector will create a Kafka Topic structured as follows: {FIWARE_SERVICE}_{FIWARE_SERVICEPATH*}_{NGSI_ENTITY_ID}_{NGSI_ENTITY_TYPE}

*FIWARE_SERVICEPATH is deprived of '/' character

Docker - Recommended

  1. Configure subscription payload configuration 'subscription.json' inside "conf" folder.
  2. Configure your environment variables in the 'docker-compose.yml' file
  3. Run docker-compose command into project root folder:
docker-compose up

API

HTTP Method Service Description
POST /notify V2 Context data ingestion route
POST /notifyld?subscriptionId=urn:ngsi-ld:Type:Entity LD Context data ingestion route

Testing

To test O2K-Connector, few services are required, in particular:

  • Orion Context Broker
  • A fully working Kafka Environment

At startup O2K-Connector subscribes to Orion Context Broker which is required to be running. Context-data will start being published to a given Kafka Topic (generate on-the-fly) as soon as possible.

To text context-data consumpion, run kafka-console-consumer command:

kafka-console-consumer --bootstrap-server broker-1:29092,broker-2:29093,broker-3:29094 --topic {FIWARE_SERVICE}_{FIWARE_SERVICEPATH*}_{NGSI_ENTITY_ID}_{NGSI_ENTITY_TYPE}

*FIWARE_SERVICEPATH is deprived of '/' character

License

O2K-Connector is licensed under

GNU Affero General Public License v3.0