Coreset Technology: Unravelling The “Magic”

Coreset Technology: Unravelling The “Magic”

Much of ShieldIOT’s “secret sauce” and key differentiators are based on its unique Coreset technology. In this blog we’ll try to provide additional insights to what are coresets and why can they act as a game changer in protecting mass scale IoT networks.

Current Challenge

Today there are many good and available (off-the-shelf) artificial intelligence / machine learning anomaly detection algorithms. The challenge is that as IoT networks become larger and larger (reaching dozens of thousands, hundreds of thousands (smart cities) and now even millions of devices (smart grid, smart transportation)), using these anomaly detection algorithms on large datasets can take hours and even days, which is not an option in case of an IoT cyber attack that can take down key physical services, cause significant financial damage and even loss of life.

The Coreset “Magic”

ShieldIOT’s Coreset “magic” (based on 12+ years of MIT and Haifa academic research, 40+ academic papers) is that it enables accurate analytics at scale.  We did not invent a magical algorithm to “rule them all”. In fact, we do not directly solve the anomaly detection problem (considered by many as impossible) – instead we try to solve the data problem.

We use Coresets. Coresets are mathematical constructs that compress the data in real-time streaming mode, from 1 million to 20 data points (n to log(n)), with almost zero loss of energy. By running Coresets before applying the existing best of breed AI/ML anomaly detection algorithms, we enable reaching optimal results in minutes (vs. hours/days) as we are running on much smaller data. These optimal results include significant lower false alarms levels (false positives) as well as improved detection rates (true positives).

Existing solutions, that are not using Coresets, are forced to perform various heuristics on the data (else they can’t analyze it as it’s too big) resulting in very high levels of false alarms (in some cases over 95% false). 

The Uniqueness

The 3 main ShieldIOT Coreset unique differentiators include:

  • Unmatched accuracy: up to 20x lower false alarms vs. competitive solutions 
  • Unlimited scalability: supporting networks of millions of devices today 
  • Any device, application: the solution is agnostic across any IoT device/application (solution has been successfully used on meters, lighting, gateways, industrial sensors, car sensors, air quality sensors, HVAC….)

The Future of Coreset Anomaly Detection

ShieldIOT is the only solution today in the world that uses Coresets for anomaly detection in IoT. In fact  there are only a handful of Coresets experts in the world today, with most of them focusing on pure academic research (and not on commercial applications).

Coresets is a new and complex field in mathematics/computer science, making it extremely difficult to leverage existing academic papers as “recipes” for commercial solutions. Having said that, we expect that as IoT networks continue to grow, coresets as an enabler of accurate analytics at scale, will become a main stream IoT technology applied across multiple IoT application and networks to ensure their protection.

NIST IoT Cybersecurity and Privacy Risk Report: Overview and Remarks


NIST IoT Cybersecurity and Privacy Risk Report: Overview and Remarks

The National Institute of Standards and Technology (NIST) has released the Considerations for Managing Internet of Things (IoT) Cybersecurity and Privacy Risks Report.

NIST IoT report (find the full document here) aims to help federal agencies and other organizations manage the cybersecurity and privacy risks associated with individual IoT devices.

This NIST IoT report identifies three high-level considerations that may affect the management of cybersecurity and privacy risks for IoT devices as compared to conventional IT devices:

Read moreNIST IoT Cybersecurity and Privacy Risk Report: Overview and Remarks

ShieldIoT Secures Urban Smart Lighting Deployments

ShieldIoT Secures Urban Smart Lighting Deployments

Smart Street Lighting growth is driven by the ability of connected (smart) street lights to address the shortcoming of traditional street lights. Smart (connected) lighting systems can substantially reduce energy consumption as it can be incorporated with various sensors such as natural light sensors occupancy sensors, and motion sensors.

This is why, today, we are witnessing a huge growth in worldwide urban smart lighting deployments, driven by the need to improve lighting energy efficiency, lower operational costs and improve citizens’ wellbeing. Globally, there are about 281.4 million streetlights in the world and is expected that it will reach 338.9 million by 2025.  However, there are some caveats to this positive trend. These mass scale lighting networks/devices are not designed with security in mind, posing a threat to critical city lighting availability and infrastructure.

ShieldIoT has identified the need for securing smart lighting deployments, and have already been deployed to secure such deployment.

Read moreShieldIoT Secures Urban Smart Lighting Deployments

Mitigating Security Risks: A Coresets-Based Approach

Mitigating Security Risks: A Coresets-Based Approach

AI-based Coresets is a highly efficient set of algorithms that is highly relevant method for IoT network traffic anomaly detection.

As we discussed in our previous blog post it is imperative to discover the first signs of anomalous activity, resulting from various types of malfunctions and/or misconfigurations, before any damage to business operations and key services can take place.

It is obvious that in order to identify and mitigate such threats, a technology that can process huge amounts of data and detect these anomalies is required. The problem is that “traditional” anomaly detection algorithms are not applicable to the challenge at hand- processing huge amounts of data, finding the anomalies and doing it quickly at a low cost.

Most existing solutions have the one or more of the following shortcomings:

  • They perform heuristics without provable guarantees for their performance in either running time but usually quality of anomaly detection.
  • They have provable guarantees but are totally impractical, usually in terms of running time
  • They do not support modern big data computation models such as streaming of unbounded data with very limited memory, distributed computation on the cloud, or parallel computations that uses e.g. GPUs devices.

When it comes to anomaly detection, the situation is worse: Simple distributions that can be learned via classic solutions such as PCA are considered hard when the input may include outliers that are not part of the model and should be identified/excluded, turning this problem to be at the level of NP-hard. Effectively, it means that the available practice is to run exhaustive search over all the possible sets of outliers, and choose the one that minimizes our fitting function after exclusion. Such solutions take considerably long time to execute, rendering this to be infeasible.

How AI-based Coresets Technology Works

AI-based Coresets is a powerful technique which enables the use of smaller sets of data instead of larger ones without compromising the quality of the output. This approach is based on highly efficient set of algorithms that reduce the data input on one hand, and execute anomaly detection routines on the reduced dataset (coreset), on the other hand. This technology can be applied to fixed big datasets, distributed data or to streaming data, which is highly relevant feature when we aim at network traffic anomaly detection.

Mitigating Security Risk With Corset Based AI

Most of the cyber security threats to IoT networks can be effectively mitigated and even prevented using the proposed solution; AI-based anomaly detection service utilizing coresets.

Here are a few examples.

  • Denial-of-service attack mitigation: any change to existing network nodes behavior, such as an increase in the amount of data transmitted or in transmission frequency, would automatically be detected by the solution as an anomaly. The solution would then initiate the required action (such as “disconnect device from network”) to be automatically carried out by the operator’s device management systems, thus mitigating any potential damages by neutralizing (disconnecting) the captured (or replicated) nodes. The uniqueness of the coresets-based approach is that it ensures the solution can monitor such changes in real time for massive millions-of-devices networks.
  • Communication hijacking / man-in the middle attack mitigation: the solution would be able to compare existing incoming/outgoing messaging vs. past ‘normality’ detecting the first signs of minor changes resulting from the link being hijacked by a malicious attacker. Being able to accurately detect these changes early on, would enable the operator to take the required actions before real damages can take place. The uniqueness of the coresets-based approach includes two main aspects: computation-wise enabling to perform these comparisons across countless potential ‘hijacking junctions’, as well as using the relevant AI detection model (unsupervised AI) to perform such comparisons and be able to detect this type of anomalous activity.
  • Note that these examples do not necessarily rely on being able to analyze the actual transmitted data (as in many cases it may be encrypted also at the operator cloud), but on analyzing the communication metadata. In cases where the actual transmitted data is also available, the solution is able to detect attacks that are specifically focused on sending falsified information (for example a captured node smart meter that sends the wrong readings as part of a fraudulent malicious activity).


To summarize, the ability to accurately reduce the data in real-time (coresets), enables to use any proven off-the-shelf detection model (AI, unsupervised machine learning, PCA, clustering…), thus the same solution can be easily used to detect almost any type of cyber threat across different heterogeneous IoT networks. It is the only feasible solution for large-scale, cellular- based IoT deployments.

5G and Cellular-Based IoT Networks: Market Trends and Security

IoT Networks

5G and Cellular-Based IoT Networks: Market Trends and Security

Market Overview

At first, there was the internet, then mobile devices came along, and then devices became “connected”, launching the IoT networks revolution. Gartner predicts that in 2019 we’ll witness 14.2 billion connected things in use, and that this number will nearly double in the next couple of years, totaling in 25 billion by 2021. This rapid growth is made possible by several factors, all maturing at roughly the same time:
  • Reduced cost of devices and connectivity modules
  • Reduced cost of communication, storage and analytics
  • Transition in architecture from gateway-based topologies (devices connect to a gateway, the gateway connects to the cloud) to device-to-cloud direct communication topologies.

IoT Networks

These technologies include U-LPWA unlicensed low power wide area networks (such as Sigfox and LoRa) as well as cellular licensed LPWAN technologies such as LTE-M, NB-IoT and 5G once available. By the end of 2020 it is expected that the number of cellular-based IoT devices will surpass U-LPWA to become the dominant wide area technology, and by 2022 70% of the wide area IoT devices would be cellular-based (1.5 billion devices in 2022 vs. 400 million in 2016). By 2022, the NB-IoT market size is expected to reach $8.2B with CAGR of 91.3%.

Vertical Use Cases

NB-IoT based applications are spread across multiple domains and use cases:
  • Smart Cities: Smart parking management, waste management, street lights…
  • Logistics and Asset Tracking: Containers, vehicles, devices.
  • Transportation: Connected cars.
  • Energy: Smart metering including water, gas, and electricity.
  • Smart Buildings: Alarm systems, access control, HVAC
  • Smart Home: Multiple sensors
  • Wearables: people tracking, animal tracking, health monitoring
  • Agriculture: Environmental monitoring, pollution monitoring

Complexity of the Cellular IoT Value Chain

The IoT value chain includes 4 key players:
  • Device manufacturers and solution providers
  • System integrators
  • IoT network operators and service providers
  • Actual end customers (network users)
Understanding the role of each player is crucial for succeeding in this new, rapidly changing world.
  • Device Manufacturers and Solution Providers: Represent the lowest level of the value chain, the commodity. This end of market encompasses almost any piece of electronic equipment that can be connected- from refrigerators to light bulbs. Advancements in technology and lower costs of connectivity modules will ensure that prices this will continue to be an extremely competitive market, dominated by huge electronics manufacturers. One caveat is, that in the journey to slash costs (and maintain the slimmest of margins) security and assurance will be swept aside in favor of “time to market” considerations. So, instead of more robust devices we’re expected to see more rudimentary devices flooding the global markets, facilitating cyber breaches and subsequent attacks.
  • System Integrators: Moving from the commodity to the corporate and government world, systems integrators are an integral part of the value chain. Similar to the role large SI played in “digital transformation” projects of the past 20 years, modern SIs will facilitate the adoption of large-scale IoT deployments, such as safe and smart cities.
  •  IoT Network Operators and Service Providers: IoT service providers play a crucial role in this eco-system. They are the actual backbone of the IoT, much like cellular providers were the infrastructure that enabled the mobile revolution 2 decades ago, or the cloud providers 10 years ago. IoT network operators and service providers are a combination of these two- both enabling the communication, the massive storage required and the analytics layer to monetize it. The actual constellation changes- sometimes these are the telco operators that own and operate the networks (such as 3G/4G, LoRa, NB-IoT and 5G networks), other times they are the providers of dedicated IoT cloud services, and sometimes a combination of them both.
  • End- Users: As is the case with IoT these are also multi-faceted. End-users could be actual consumers, electronic companies providing IoT services, municipalities consuming (and delivering) IoT services, corporates and governments. What is certain that just like the 2 great movements that preceded it (mobile, cloud) we will all be soon be dependent on the IoT eco-system functioning well and without interferences to manage our daily lives.


As such, it is imperative that proper security measures and controls will be put in place- but more on this in our next blog post.

5G and Cellular Based Direct Communication IoT challenges


As we’ve explored in our previous blog post,  security is essential for the rapid adoption of IoT. However, since IoT as whole, and its security in specific, are relatively new, they lack a clear and accepted definition. Some include the security of every connected device, be it a car or a piece of machinery to fall under the category of “IoT Security”. Others refer to the more traditional definition of “pure”, Cellular based IoT deployments, that consists of large quantities of relatively inexpensive devices deployed “in the field” and communicate with the cloud directly (or to a lesser degree, through dedicated gateways).

We will refer to these types of IoT devices and networks (i.e. cellular-based IoT )  in this blog post.

Challenges and Risks of Cellular Based Direct Communication

Security IoT devices and networks is a novel challenge. It is radically different to IT security in the sense that IoT is very diverse (the technologies these devices use to communicate with the cloud include U-LPWA unlicensed low power wide area networks, such as Sigfox and LoRa) as well as cellular licensed LPWAN technologies such as LTE-M, NB-IoT and 5G).

Read more5G and Cellular Based Direct Communication IoT challenges

Product Demo

Sign -up now

Let us show you how Shield-IoT can expand your IoT security and operational monitoring capabilities.

Complete the form to contact Sales

By supplying my contact information, I authorize Shield-IoT to contact me with personalized communications about Shield-IoT products and services.

Skip to content