Early detection is critical when an IoT cyber-attack or major operational event threatens to take down key physical services, cause significant financial damage and even loss of life. Explore how Shield-IoT Coreset data compression can detect anomalies in minutes compared to days and weeks required by alternative methods.
The main challenge is to overcome the problem of scale. As IoT networks become larger (from dozens of thousands, to millions of devices), most of current off-the-shelf anomaly detection algorithms fail to deliver results within a reasonable timeframe. Performing anomaly detection on extremely large datasets can take hours and even days. These solutions are not viable when an IoT cyber-attack or significant operational events threatens to take down key physical services, cause significant financial damage and even loss of life.
Most anomaly detection algorithms perform heuristics without provable guarantees performance in either running time and especially quality of anomaly detection.
Many existing anomaly detection algorithms have provable guarantees that are totally impractical, usually in terms of running time.
AI-based coresets is a powerful technique which enables the use of smaller sets of data instead of larger ones without compromising the quality of the output. This approach is based on highly efficient set of algorithms that reduce the data input on one hand, and execute anomaly detection routines on the reduced dataset (coreset), on the other hand. This technology has been applied to fixed big datasets, distributed data or to streaming data, which is highly relevant feature when we aim at network traffic anomaly detection.
AI-based coresets is a powerful technique which enables the use of smaller sets of data instead of larger ones without compromising the quality of the output. This approach is based on highly efficient set of algorithms that reduce the data input on one hand, and execute anomaly detection routines on the reduced dataset (coreset), on the other hand. This technology has been applied to fixed big datasets, distributed data or to streaming data, which is highly relevant feature when we aim at network traffic anomaly detection.
The “magic” of ShieldIOT Coreset technology is that it delivers accurate analytics at scale in a fraction of the time required by alternative off-the-shelf products. In practice, Coreset does not directly solve the anomaly detection problem, instead it tries to solve the data problem by using much smaller datasets.
Coresets are mathematical constructs that compress the data in real-time streaming mode, from 1 million to 20 data points (n to log(n)), with almost zero loss of energy. By running Coresets on reduced datasets before applying existing AI/ML anomaly detection algorithms, Coreset-based solutions can deliver optimal results in minutes compared to hours or days. These results include significant lower false alarms levels (false positives) as well as improved detection rates (true positives).
Existing non-Coreset based solutions, need to perform various heuristics to enable analysis of mass scale data sets. This method results in very high levels of false alarms (in some cases exceeding 95% false positives).
Statistical research was undertaken to compare ShieldIOT AI error rate (false positive) versus RANSAC, a state of the art anomaly detection algorithm (data: security camera, 114 features per data sample).
Shield-IoT technology is based on over 15 years of academic research in MIT and Haifa University, and backed by over 70 academic papers and multiple patents, The philosophy behind the AI-based coresets is that running existing algorithms on the reduced dataset will probably give approximate result, as running them on the original big data. This field-proven technique allows us to reduce the data by unprecedented order of magnitude: from n to log(n)! This is done not by designing a new algorithm for solving the problem, but rather by running the existing algorithms on the reduced dataset.
Unlike other compression techniques like zip or mp4, our coreset is data reduction and not just compression of the input, in the sense that it is problem-dependent. our coreset, optimized for detecting network anomalies, removes most of the noise and therefore they are more effective and faster when executing the algorithms on them.
For more information about Coreset AI anomaly detection technology, download these publications.
Published by Dan Feldman and Yair Marom
Published by Dan Feldman and Leonard J. Schulman.
Shieild-IoT software solutions are deployed across a wide range of industry use cases and applications. Our purpose-built IoT security management solution provides continuous threat mitigation with no changes to your network.