A High Performance Computing Web Search Engine Based on Big Data and Parallel Distributed Models

Enregistré dans:

Détails bibliographiques
Publié dans:	Informatica vol. 48, no. 20 (Dec 2024), p. 27
Auteur principal:	Ma, Jun
Publié:	Slovenian Society Informatika / Slovensko drustvo Informatika
Sujets:	Big Data Machine learning Search engines Computers User needs Performance evaluation Artificial intelligence Data structures Cloud computing Computational grids Data search System effectiveness Data storage High performance computing Distributed processing Efficiency Information retrieval Vector spaces
Accès en ligne:	Citation/Abstract Full Text Full Text - PDF
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Description
Résumé:	This paper presents a high-performance web search system leveraging big data technology. Utilizing a heterogeneous architecture and a parallel distributed computing model based on the MapReduce framework, the system significantly enhances efficiency, scalability, and reliability. The design includes a storage management scheme that integrates cloud storage and grid computing technologies, facilitating efficient storage and rapid access to large-scale data. Key components such as an inverted index structure, vector space model, and semantic analysis models are employed to implement functionalities across the data, logic, and display layers. An experimental environment was set up on the Microsoft Azure cloud platform using the Common Crawl dataset for testing. Performance evaluation, based on metrics including response time, accuracy, and stability, demonstrates the system's superior performance compared to two existing systems, thereby validating its effectiveness.
ISSN:	0350-5596 1854-3871
DOI:	10.31449/inf.v48i20.6776
Source:	Advanced Technologies & Aerospace Database