Centralized Dashboard for the ntopng Infrastructure

[cardigliano]

It is requested to implement a centralized dashboard that consolidates and displays aggregated information from multiple ntopng instances monitoring a distributed network (e.g. different sites). In short the goal is to create a unified view that allows users to monitor multiple sites in a single place/dashboard, rather than jumping across ntopng instances.

Requirements:

• Aggregate data from various ntopng instances running at different sites, offering a unified view/dashboard
• Live monitoring: real-time data needs to be aggregated in the live dashboard, to monitor the whole infrastructure in real-time
• Historical reports: historical data, at least in reports, needs to be aggregated (e.g. to analyse bandwidth utilization, or top hosts, in a time interval)
• Scalability: design a solution that can scale with many sites, also when the traffic per site is high

Additional notes:

• Drill down (e.g. inspect flows or specific hosts) can be implemented by jumping on the specific ntopng instance

Design choices:

• Option 1: leverage on the pre-existing Infrastructure support and Token-based authentication, which is using the RESTful APIs for retrieving data. In this case it is requested on implement wrapper endpoints, one per dashboard/report component type, acting as proxies and aggregating information from multiple ntopng instances.
• Option 2: study the feasibility of adding an extra abstraction layer, where the centralised ntopng acts as an aggregation layer and query coordinator, similar to those used in database clusters, using the distributed ntopng instances as data nodes. This requires studying technologies implementing horizontal sharding and ETL pipelines, and evaluate the effort for implementing all this in ntopng