Real-time Big Data Monitoring System

Document identifier:
Access full text here:10.1109/SRDS47363.2019.00049
Keyword: Natural Sciences, Computer and Information Sciences, Media and Communication Technology, Naturvetenskap, Data- och informationsvetenskap, Medieteknik, Hadoop, MapReduce, Monitoring, Pervasive Mobile Computing, Distribuerade datorsystem
Publication year: 2019
Relevant Sustainable Development Goals (SDGs):
SDG 9 Industry, innovation and infrastructure
The SDG label(s) above have been assigned by


Modern big data processing systems are becoming very complex in terms of large-scale, high-concurrency and multiple talents. Thus, many failures and performance reductions only happen at run-time and are very difficult to capture. Moreover, some issues may only be triggered when some components are executed. To analyze the root cause of these types of issues, we have to capture the dependencies of each component in real-time. In this paper, we propose SmartMonit, a real-time big data monitoring system, which collects infrastructure information such as the process status of each task. At the same time, we develop a real-time stream processing framework to analyze the coordination among the tasks and the infrastructures. This coordination information is essential for troubleshooting the reasons for failures and performance reduction, especially the ones propagated from other causes.


Umit Demirbaga

Newcastle University. Bartin University
Other publications >>

Ayman Noor

Newcastle University. Taibah University
Other publications >>

Zhenyu Wen

Newcastle University
Other publications >>

Philip James

Newcastle University
Other publications >>

Karan Mitra

Luleå tekniska universitet; Datavetenskap
Other publications >>

Rajiv Ranjan

Newcastle University
Other publications >>

Record metadata

Click to view metadata