Terramycelium: a reference architecture for adaptive big data systems
Gorde:
| Argitaratua izan da: | Journal of Big Data vol. 12, no. 1 (Nov 2025), p. 260 |
|---|---|
| Egile nagusia: | |
| Beste egile batzuk: | |
| Argitaratua: |
Springer Nature B.V.
|
| Gaiak: | |
| Sarrera elektronikoa: | Citation/Abstract Full Text Full Text - PDF |
| Etiketak: |
Etiketarik gabe, Izan zaitez lehena erregistro honi etiketa jartzen!
|
| Laburpena: | Modern organizations generate and process unprecedented volumes of structured, semi-structured, and unstructured data from diverse sources, creating significant architectural and engineering challenges for traditional data processing systems. Industry analyses consistently report failure rates of 60-85% for Big Data projects, with architectural limitations identified as a primary contributing factor. Current reference architectures suffer from monolithic designs, inadequate cross-cutting concerns (security, privacy, metadata), and limited adaptability to evolving data ecosystems. This paper presents Terramycelium, a novel reference architecture for Big Data systems that addresses these limitations through a domain-driven, event-oriented approach. The architecture integrates principles from complex adaptive systems, domain-driven design, distributed systems, and event-driven architectures to enable autonomous domain-specific data ownership while maintaining system-wide coherence through asynchronous event communication. We developed Terramycelium following empirically grounded reference architecture guidelines and evaluated it through two complementary methods: a case-mechanism experiment and expert opinion assessment. The case-mechanism experiments demonstrated the architecture’s capability to process 1.693GB of data with 50-100 second latency, handle 771,305 streaming messages with 0.0000148 second ingestion latency, and maintain stable performance with 24% CPU utilization under high-volume scenarios. Expert evaluation (n=3, 10-32 years experience) validated the architecture’s innovative integration of domain-driven design with data engineering, while identifying implementation complexity and organizational readiness as adoption challenges. Terramycelium contributes a validated approach for building scalable, maintainable Big Data systems that addresses the limitations of existing monolithic architectures while aligning with modern software engineering practices. |
|---|---|
| ISSN: | 2196-1115 |
| DOI: | 10.1186/s40537-025-01311-6 |
| Baliabidea: | ABI/INFORM Global |