AI-based Coresets is a highly efficient set of algorithms that is highly relevant method for IoT network traffic anomaly detection.
As we discussed in our previous blog post it is imperative to discover the first signs of anomalous activity, resulting from various types of malfunctions and/or misconfigurations, before any damage to business operations and key services can take place.
It is obvious that in order to identify and mitigate such threats, a technology that can process huge amounts of data and detect these anomalies is required. The problem is that “traditional” anomaly detection algorithms are not applicable to the challenge at hand- processing huge amounts of data, finding the anomalies and doing it quickly at a low cost.
Most existing solutions have the one or more of the following shortcomings:
When it comes to anomaly detection, the situation is worse: Simple distributions that can be learned via classic solutions such as PCA are considered hard when the input may include outliers that are not part of the model and should be identified/excluded, turning this problem to be at the level of NP-hard. Effectively, it means that the available practice is to run exhaustive search over all the possible sets of outliers, and choose the one that minimizes our fitting function after exclusion. Such solutions take considerably long time to execute, rendering this to be infeasible.
AI-based Coresets is a powerful technique which enables the use of smaller sets of data instead of larger ones without compromising the quality of the output. This approach is based on highly efficient set of algorithms that reduce the data input on one hand, and execute anomaly detection routines on the reduced dataset (coreset), on the other hand. This technology can be applied to fixed big datasets, distributed data or to streaming data, which is highly relevant feature when we aim at network traffic anomaly detection.
Most of the cyber security threats to IoT networks can be effectively mitigated and even prevented using the proposed solution; AI-based anomaly detection service utilizing coresets.
Here are a few examples.
To summarize, the ability to accurately reduce the data in real-time (coresets), enables to use any proven off-the-shelf detection model (AI, unsupervised machine learning, PCA, clustering…), thus the same solution can be easily used to detect almost any type of cyber threat across different heterogeneous IoT networks. It is the only feasible solution for large-scale, cellular- based IoT deployments.