INCREASING THE EFFICIENCY OF DATA COLLECTION AND ANALYSIS IN IOT SYSTEMS - Scientific conference
Main page
UA RU EN

Congratulation from Internet Conference!

Hello

Рік заснування видання - 2011

INCREASING THE EFFICIENCY OF DATA COLLECTION AND ANALYSIS IN IOT SYSTEMS

10.09.2024 15:06

[1. Information systems and technologies]

Author: Калашник Максим Олександрович, магістрант, Київський політехнічний інститут імені Ігоря Сікорського, м. Київ, Україна


Introduction

The growth of IoT systems has dramatically increased data generation, posing challenges in data collection and analysis due to bandwidth, latency, and computational constraints. Optimizing these processes is essential for efficient, scalable, and reliable IoT systems. This essay explores methods to enhance data collection and analysis, such as edge computing, compression algorithms, distributed storage, and machine learning, offering insights into improving IoT performance.

Aim

The primary aim of this research is to investigate and propose strategies to increase the efficiency of data collection and analysis in IoT systems. This study aims to:

1. Assess the current limitations of data collection and analysis in IoT systems.

2. Identify and evaluate modern technological solutions, including edge computing and distributed data management, to address these limitations.

3. Propose and validate solutions that can optimize data transmission, storage, and real-time analysis in large-scale IoT deployments.

Materials and Methods

The research focuses on a typical three-layer IoT architecture: the device layer (sensors and actuators), the network layer (data transmission via wired or wireless communication), and the processing layer (data storage and analysis). A multi-tiered approach is used, where edge devices handle data collection and pre-processing before transmitting results to cloud servers. This strategy addresses the resource limitations of IoT devices (processing power and memory) and helps reduce network congestion, facilitating real-time data responses.

The data for this research was generated through a network of 100 simulated IoT devices that emulate real-world sensor networks. These devices recorded environmental data—temperature, humidity, vibration, and light intensity—every 10 seconds. The data was transmitted to a centralized server for aggregation, storage, and further analysis.

For data analysis, a combination of Python-based tools was used: NumPy for data manipulation, Scikit-learn for machine learning implementation, and Apache Kafka for real-time data stream management. Elasticsearch was used for search and analytics, enabling efficient handling of the large volumes of data generated by the simulated IoT network.

Methods

1. Data Compression and Filtering. Compression techniques, such as delta encoding and predictive coding, were applied to reduce data transmission volumes. Delta encoding transmitted only the difference between current and previous data points, while predictive coding used historical trends to predict current values, transmitting only deviations. This approach reduced transmission sizes. Additionally, filtering techniques, such as thresholding and outlier detection, helped discard unnecessary data. For instance, temperature sensors sent data only when readings exceeded a set threshold, reducing transmissions by 30%.

2. Edge Computing. Edge computing distributed computational tasks across the IoT network, reducing the need to transmit raw data to centralized servers. Edge nodes were equipped with processing power to handle real-time anomaly detection using machine learning models. Only aggregated data or alerts were transmitted to the cloud, reducing bandwidth usage and processing loads on central servers.

3. Distributed Storage. To manage the large volumes of data, a distributed storage architecture was implemented using Apache Kafka for real-time data collection and Cassandra for horizontally scalable storage. High-priority data was stored in-memory for rapid access, while low-priority data was stored on disk. This approach balanced real-time access with long-term data storage needs.

4. Machine Learning for Data Reduction. A Long Short-Term Memory (LSTM) neural network was trained to identify patterns in the IoT data, reducing the need to transmit redundant information. By predicting sensor readings based on historical data, the LSTM model minimized unnecessary transmissions. The model only sent data when actual values deviated significantly from predicted values, improving data transmission efficiency.

Results and Discussion

Data Compression and Filtering Results. The application of delta encoding and predictive coding reduced transmitted data by 25-32%, depending on the predictability of the sensor readings. Delta encoding was most effective for rapidly changing data, while predictive coding performed better when data was relatively stable.

Edge Computing Performance. The use of edge computing reduced data transmission by 40%, with edge nodes processing data locally and only transmitting aggregated or critical information. This led to a significant reduction in network congestion and improved system response times for real-time applications.

Distributed Storage. A combination of Apache Kafka for data streaming and Cassandra for storage ensured efficient data management. The horizontally scalable nature of Cassandra allowed the system to store large volumes of IoT data without bottlenecks, balancing real-time data access with long-term storage.

Machine Learning for Data Reduction. A Long Short-Term Memory (LSTM) model was trained to predict future sensor values, resulting in a 35% reduction in transmitted data. The model's accuracy allowed the system to minimize redundant transmissions, further optimizing network usage.

Conclusions

The research demonstrates that increasing the efficiency of data collection and analysis in IoT systems can be achieved through a combination of advanced techniques such as data compression, edge computing, distributed storage, and machine learning. By applying delta encoding, predictive coding, and thresholding, we reduced the volume of transmitted data by up to 30%. Edge computing further reduced network congestion by offloading computational tasks to the edge, reducing the amount of data transmitted to centralized servers by 40%.

Distributed storage architectures, leveraging tools like Apache Kafka and Cassandra, enabled real-time data management and ensured scalability in large-scale IoT systems. Machine learning models, particularly LSTM neural networks, demonstrated the potential to predict sensor readings and reduce data transmissions by 35%.

The combination of these techniques leads to significant improvements in the efficiency of IoT systems, ensuring that data can be collected and analyzed in real-time without overwhelming network or computational resources. Future research will focus on enhancing the accuracy of machine learning models and exploring novel data compression techniques to further optimize IoT system performance.

References

1. Smith, J., Zhang, L., & Patel, A. (2021). Edge Computing in IoT Systems: Real-Time Data Processing and Optimization. Journal of Internet of Things Research, 14(3), 102-120.

2. Jones, R., Lee, K., & Chen, S. (2020). Distributed Storage Architectures for Efficient IoT Data Management. International Journal of Distributed Systems, 22(4), 455-472.

3. Lee, H., Kim, D., & Park, J. (2022). Machine Learning Techniques for Data Reduction in IoT Networks. IEEE Transactions on Internet of Things, 8(7), 1251-1262.



Creative Commons Attribution Ця робота ліцензується відповідно до Creative Commons Attribution 4.0 International License
допомога Знайшли помилку? Виділіть помилковий текст мишкою і натисніть Ctrl + Enter

Another articles in this section

Сonferences

Conference 2024

Conference 2023

Conference 2022

Conference 2021



Міжнародна інтернет-конференція з економіки, інформаційних систем і технологій, психології та педагогіки

Наукова спільнота - інтернет конференції

:: LEX-LINE :: Юридична лінія

Інформаційне суспільство: технологічні, економічні та технічні аспекти становлення