Corral Framework: Trustworthy and Fully Functional Data Intensive Parallel Astronomical Pipelines
Kaydedildi:
| Yayımlandı: | arXiv.org (Aug 7, 2017), p. n/a |
|---|---|
| Yazar: | |
| Diğer Yazarlar: | , , , , , |
| Baskı/Yayın Bilgisi: |
Cornell University Library, arXiv.org
|
| Konular: | |
| Online Erişim: | Citation/Abstract Full text outside of ProQuest |
| Etiketler: |
Etiket eklenmemiş, İlk siz ekleyin!
|
MARC
| LEADER | 00000nab a2200000uu 4500 | ||
|---|---|---|---|
| 001 | 2076045078 | ||
| 003 | UK-CbPIL | ||
| 022 | |a 2331-8422 | ||
| 024 | 7 | |a 10.1016/j.ascom.2017.07.003 |2 doi | |
| 035 | |a 2076045078 | ||
| 045 | 0 | |b d20170807 | |
| 100 | 1 | |a Cabral, Juan B | |
| 245 | 1 | |a Corral Framework: Trustworthy and Fully Functional Data Intensive Parallel Astronomical Pipelines | |
| 260 | |b Cornell University Library, arXiv.org |c Aug 7, 2017 | ||
| 513 | |a Working Paper | ||
| 520 | 3 | |a Data processing pipelines represent an important slice of the astronomical software library that include chains of processes that transform raw data into valuable information via data reduction and analysis. In this work we present Corral, a Python framework for astronomical pipeline generation. Corral features a Model-View-Controller design pattern on top of an SQL Relational Database capable of handling: custom data models; processing stages; and communication alerts, and also provides automatic quality and structural metrics based on unit testing. The Model-View-Controller provides concept separation between the user logic and the data models, delivering at the same time multi-processing and distributed computing capabilities. Corral represents an improvement over commonly found data processing pipelines in Astronomy since the design pattern eases the programmer from dealing with processing flow and parallelization issues, allowing them to focus on the specific algorithms needed for the successive data transformations and at the same time provides a broad measure of quality over the created pipeline. Corral and working examples of pipelines that use it are available to the community at https://github.com/toros-astro. | |
| 653 | |a Data processing | ||
| 653 | |a Data models | ||
| 653 | |a Pipelines | ||
| 653 | |a Parallel processing | ||
| 653 | |a Astronomy | ||
| 653 | |a Control systems design | ||
| 653 | |a Model testing | ||
| 653 | |a Pipelining (computers) | ||
| 653 | |a Data reduction | ||
| 653 | |a Algorithms | ||
| 653 | |a Trustworthiness | ||
| 653 | |a Controllers | ||
| 653 | |a Relational data bases | ||
| 653 | |a Computing time | ||
| 653 | |a Computer networks | ||
| 653 | |a Distributed processing | ||
| 653 | |a Query languages | ||
| 700 | 1 | |a Sánchez, Bruno | |
| 700 | 1 | |a Beroiz, Martín | |
| 700 | 1 | |a Domínguez, Mariano | |
| 700 | 1 | |a Lares, Marcelo | |
| 700 | 1 | |a Gurovich, Sebastián | |
| 700 | 1 | |a Granitto, Pablo | |
| 773 | 0 | |t arXiv.org |g (Aug 7, 2017), p. n/a | |
| 786 | 0 | |d ProQuest |t Engineering Database | |
| 856 | 4 | 1 | |3 Citation/Abstract |u https://www.proquest.com/docview/2076045078/abstract/embedded/ZKJTFFSVAI7CB62C?source=fedsrch |
| 856 | 4 | 0 | |3 Full text outside of ProQuest |u http://arxiv.org/abs/1701.05566 |