Enhancing Latency in IoT Real-Time Data Processing in Massive Scale

Why Conduct This Research?

The spread of connected devices in the Internet of Things (IoT) has led to the emergence of Massive IoT (MIoT). Managing and processing the enormous volume of data generated by these devices in real time has become a significant challenge. While this interconnectedness offers unparalleled opportunities for innovation, it also introduces challenges in the realms of data processing, security and scalability. In response to this, our research proposes a comprehensive framework combining a publish/subscribe data processing model with blockchain technology to ensure both efficiency and security.

Methodology: A Brief Description

Our research addresses these challenges by proposing a comprehensive architectural framework integrating real-time data processing with blockchain-based security measures. As shown in Figure 1, at the heart of our system is the integration of Apache Kafka and Apache Druid. This allows real-time processing of vast amounts of data with minimal latency. In other words, Apache Kafka uses a publish/subscribe mechanism, particularly well suited for MIoT environments. This mechanism allows data generated by IoT devices to be relayed to a dedicated broker. The data is then distributed to subscribed applications in real time. The publish/subscribe model is beneficial because it decouples producers of data (publishers) from the consumers (subscribers), allowing for greater flexibility, scalability, and responsiveness in data handling.

Apache Kafka is used for managing the data flow between IoT sensors and brokers. This ensures that data is transmitted efficiently and reliably, even as the number of connected devices scales into the hundreds of thousands. Apache Druid complements this by providing a robust analytical database environment that excels in real-time query and analysis, making it an ideal choice for scenarios where rapid data insight is critical.

Enhancements in Data Transmission, Security and Latency

A key challenge in MIoT is ensuring the availability of data transmission, particularly in environments where connectivity may be unstable. To address this, our framework incorporates a fail-safe mechanism using two Software-Defined Radios (SDRs) based on LTE technology, as can be seen in Figure 2. These SDRs act as a “network-in-a-box,” providing redundancy and ensuring that data transmission continues uninterrupted, even if one network fails. This redundancy is crucial for maintaining data integrity and availability in scenarios where consistent connectivity is non-negotiable.

Security is another major concern in MIoT, given the sensitive nature of the data being transmitted and stored. To fortify our system’s data storage capabilities, we integrated Hyperledger Fabric, a blockchain technology known for its robust security features, including data immutability and integrity. By leveraging blockchain, we ensure that the data stored within our system is not only secure but also tamper-proof, providing a high level of trust and reliability. Our experimental results demonstrate that this blockchain-based approach can handle over 800 transactions per second in a dataset containing 14,000 transactions, making it both secure and efficient.

Latency is a critical metric in any real-time data processing system, particularly in MIoT where delays can have significant consequences. Figure 3 shows that our framework achieves a latency of less than 25 milliseconds for a network of 100,000 devices, a performance that qualifies our system as highly responsive and suitable for real-time applications. This low latency is essential for applications such as autonomous vehicles, industrial automation and smart cities, where rapid data processing is critical for safety and efficiency.

In the broader context of IoT research, our work builds on previous studies that have explored various methods for improving data processing, security and scalability in IoT environments. However, our approach is distinguished by integrating these elements into a unified framework that addresses the unique challenges of MIoT. While traditional IoT systems focus on smaller-scale applications with limited data streams, MIoT requires solutions that can handle the exponential growth in device numbers and data volumes, while maintaining security and low latency.

Advancing the Future of Massive IoT With Integrated Solutions

In conclusion, our research presents a novel approach to managing the complexities of Massive IoT by combining real-time data processing with blockchain-based security. Through the use of a publish/subscribe model, Apache Kafka, Apache Druid and Hyperledger Fabric, we offer a scalable, secure and efficient framework well suited to the demands of the MIoT ecosystem. As the number of connected devices continues to grow, the need for such robust solutions will only become more pressing, and our work provides a solid foundation for future developments in this rapidly evolving field.

Additional Information

For more information on this research, please read the following paper: Ataei, M., Eghmazi, A., Shakerian, A., Landry, R., & Chevrette, G. (2023). Publish/Subscribe Method for Real-Time Data Processing in Massive IoT Leveraging Blockchain for Secured Storage. Sensors, 23(24), 9692. https://doi.org/10.3390/s23249...