Bridging the Gap: Integrating Batch and Streaming Data Paradigms for Holistic Analytics in the Age of Real-Time, Predictive, and Historical Insights
Αποθηκεύτηκε σε:
| Εκδόθηκε σε: | International Journal of Communication Networks and Information Security vol. 17, no. 2 (2025), p. 327-359 |
|---|---|
| Κύριος συγγραφέας: | |
| Έκδοση: |
Kohat University of Science and Technology (KUST)
|
| Θέματα: | |
| Διαθέσιμο Online: | Citation/Abstract Full Text Full Text - PDF |
| Ετικέτες: |
Δεν υπάρχουν, Καταχωρήστε ετικέτα πρώτοι!
|
| Περίληψη: | Background: As industries increasingly rely on data-driven decision-making, the growing volume and complexity of data have led to the widespread adoption of analytics across sectors like finance, healthcare, IoT, and smart cities. Traditional systems, however, often rely on either batch processing for historical data or streaming data for real-time analysis, leading to fragmented insights. The integration of these two paradigms- historical and real-time data-has become essential to achieve a comprehensive view that reflects both past trends and immediate conditions, ultimately enabling organizations to make more informed and proactive decisions. Problem Statement: A critical gap exists between batch processing for historical data and streaming data for real-time analytics. Existing solutions typically treat these two data types separately, resulting in limited capacity for delivering holistic insights. Batch processing excels in historical analysis but suffers from delays and inefficiencies when real-time decisions are needed. Conversely, streaming data systems provide immediate insights but lack the depth and context provided by historical data. This separation hampers the ability to create a unified analytics environment capable of supporting predictive and real-time decision-making across dynamic environments. Objective: This paper proposes a novel, dynamic hybrid framework that integrates both batch and streaming data into a single seamless analytics system. The proposed framework enables organizations to handle both historical and realtime data, supporting real-time analytics, historical analysis, and predictive modeling in an efficient, scalable manner. By combining the strengths of both data paradigms, the system provides a unified view that enhances decision-making capabilities, enabling proactive strategies across various industries. Methodology: The methodology involves the development of an adaptive hybrid architecture that dynamically switches between batch and streaming data processing based on the real-time needs and system load. The proposed system incorporates machine learning models to predict optimal processing modes, dynamically adjusting data flows for efficiency. Edge-AI integration is utilized to preprocess data at the source, reducing bandwidth usage and improving real-time responsiveness. A dynamic workflow management system ensures that data processing methods adapt to changing conditions, ensuring both high performance and resource efficiency. Additionally, advanced synchronization techniques like timestamp-based fusion and event-based models are employed to maintain data consistency across both processing modes. |
|---|---|
| ISSN: | 2073-607X 2076-0930 |
| Πηγή: | Science Database |