Terramycelium: a reference architecture for adaptive big data systems
Gespeichert in:
| Veröffentlicht in: | Journal of Big Data vol. 12, no. 1 (Nov 2025), p. 260 |
|---|---|
| 1. Verfasser: | |
| Weitere Verfasser: | |
| Veröffentlicht: |
Springer Nature B.V.
|
| Schlagworte: | |
| Online-Zugang: | Citation/Abstract Full Text Full Text - PDF |
| Tags: |
Keine Tags, Fügen Sie das erste Tag hinzu!
|
MARC
| LEADER | 00000nab a2200000uu 4500 | ||
|---|---|---|---|
| 001 | 3275425862 | ||
| 003 | UK-CbPIL | ||
| 022 | |a 2196-1115 | ||
| 024 | 7 | |a 10.1186/s40537-025-01311-6 |2 doi | |
| 035 | |a 3275425862 | ||
| 045 | 2 | |b d20251101 |b d20251130 | |
| 100 | 1 | |a Ataei, Pouya |u Scholar Spark, Mount Albert, New Zealand | |
| 245 | 1 | |a Terramycelium: a reference architecture for adaptive big data systems | |
| 260 | |b Springer Nature B.V. |c Nov 2025 | ||
| 513 | |a Journal Article | ||
| 520 | 3 | |a Modern organizations generate and process unprecedented volumes of structured, semi-structured, and unstructured data from diverse sources, creating significant architectural and engineering challenges for traditional data processing systems. Industry analyses consistently report failure rates of 60-85% for Big Data projects, with architectural limitations identified as a primary contributing factor. Current reference architectures suffer from monolithic designs, inadequate cross-cutting concerns (security, privacy, metadata), and limited adaptability to evolving data ecosystems. This paper presents Terramycelium, a novel reference architecture for Big Data systems that addresses these limitations through a domain-driven, event-oriented approach. The architecture integrates principles from complex adaptive systems, domain-driven design, distributed systems, and event-driven architectures to enable autonomous domain-specific data ownership while maintaining system-wide coherence through asynchronous event communication. We developed Terramycelium following empirically grounded reference architecture guidelines and evaluated it through two complementary methods: a case-mechanism experiment and expert opinion assessment. The case-mechanism experiments demonstrated the architecture’s capability to process 1.693GB of data with 50-100 second latency, handle 771,305 streaming messages with 0.0000148 second ingestion latency, and maintain stable performance with 24% CPU utilization under high-volume scenarios. Expert evaluation (n=3, 10-32 years experience) validated the architecture’s innovative integration of domain-driven design with data engineering, while identifying implementation complexity and organizational readiness as adoption challenges. Terramycelium contributes a validated approach for building scalable, maintainable Big Data systems that addresses the limitations of existing monolithic architectures while aligning with modern software engineering practices. | |
| 653 | |a Data processing | ||
| 653 | |a Software | ||
| 653 | |a Experiments | ||
| 653 | |a Metadata | ||
| 653 | |a Big Data | ||
| 653 | |a Data systems | ||
| 653 | |a Latency | ||
| 653 | |a Engineering | ||
| 653 | |a Architecture | ||
| 653 | |a Privacy | ||
| 653 | |a Limitations | ||
| 653 | |a Cross cutting | ||
| 653 | |a Coherence | ||
| 653 | |a Machine learning | ||
| 653 | |a Adaptive systems | ||
| 653 | |a Ingestion | ||
| 653 | |a Ownership | ||
| 653 | |a Network latency | ||
| 653 | |a Decentralization | ||
| 653 | |a Unstructured data | ||
| 653 | |a Complexity | ||
| 653 | |a Building engineers | ||
| 700 | 1 | |a Atemkeng, Marcellin |u Department of Mathematics, Rhodes University, Grahamstown, South Africa (GRID:grid.91354.3a) (ISNI:0000 0001 2364 1300); National Institute for Theoretical and Computational Sciences (NITheCS), Stellenbosch, South Africa (GRID:grid.91354.3a) | |
| 773 | 0 | |t Journal of Big Data |g vol. 12, no. 1 (Nov 2025), p. 260 | |
| 786 | 0 | |d ProQuest |t ABI/INFORM Global | |
| 856 | 4 | 1 | |3 Citation/Abstract |u https://www.proquest.com/docview/3275425862/abstract/embedded/L8HZQI7Z43R0LA5T?source=fedsrch |
| 856 | 4 | 0 | |3 Full Text |u https://www.proquest.com/docview/3275425862/fulltext/embedded/L8HZQI7Z43R0LA5T?source=fedsrch |
| 856 | 4 | 0 | |3 Full Text - PDF |u https://www.proquest.com/docview/3275425862/fulltextPDF/embedded/L8HZQI7Z43R0LA5T?source=fedsrch |