SmartMonit

Real-time Big Data Monitoring System

Document identifier: oai:DiVA.org:ltu-76617
Access full text here:10.1109/SRDS47363.2019.00049
Keyword: Natural Sciences, Computer and Information Sciences, Media and Communication Technology, Naturvetenskap, Data- och informationsvetenskap, Medieteknik, Hadoop, MapReduce, Monitoring, Pervasive Mobile Computing, Distribuerade datorsystem
Publication year: 2019
Relevant Sustainable Development Goals (SDGs):

The SDG label(s) above have been assigned by OSDG.ai

Abstract:

Modern big data processing systems are becoming very complex in terms of large-scale, high-concurrency and multiple talents. Thus, many failures and performance reductions only happen at run-time and are very difficult to capture. Moreover, some issues may only be triggered when some components are executed. To analyze the root cause of these types of issues, we have to capture the dependencies of each component in real-time. In this paper, we propose SmartMonit, a real-time big data monitoring system, which collects infrastructure information such as the process status of each task. At the same time, we develop a real-time stream processing framework to analyze the coordination among the tasks and the infrastructures. This coordination information is essential for troubleshooting the reasons for failures and performance reduction, especially the ones propagated from other causes.

Authors

Umit Demirbaga

Newcastle University. Bartin University
Other publications >>

Ayman Noor

Newcastle University. Taibah University
Other publications >>

Zhenyu Wen

Newcastle University
Other publications >>

Philip James

Newcastle University
Other publications >>

Karan Mitra

Luleå tekniska universitet; Datavetenskap
Other publications >>

Rajiv Ranjan

Newcastle University
Other publications >>

Record metadata

Click to view metadata

	
    header: 
        identifier: oai:DiVA.org:ltu-76617
        datestamp: 2021-04-19T12:43:05Z
        setSpec: SwePub-ltu
    metadata: 
        mods: 
            @attributes: 
                version: 3.7
            recordInfo: 
                recordContentSource: ltu
                recordCreationDate: 2019-11-05
            identifier: 
                http://urn.kb.se/resolve?urn=urn:nbn:se:ltu:diva-76617
                10.1109/SRDS47363.2019.00049
                2-s2.0-85084118011
            titleInfo: 
                @attributes: 
                    lang: eng
                title: SmartMonit
                subTitle: Real-time Big Data Monitoring System
            abstract: 
Modern big data processing systems are becoming very complex in terms of large-scale high-concurrency and multiple talents. Thus many failures and performance reductions only happen at run-time and are very difficult to capture. Moreover some issues may only be triggered when some components are executed. To analyze the root cause of these types of issues we have to capture the dependencies of each component in real-time. In this paper we propose SmartMonit a real-time big data monitoring system which collects infrastructure information such as the process status of each task. At the same time we develop a real-time stream processing framework to analyze the coordination among the tasks and the infrastructures. This coordination information is essential for troubleshooting the reasons for failures and performance reduction especially the ones propagated from other causes.

            subject: 
                    @attributes: 
                        lang: eng
                        authority: uka.se
                    topic: 
                        Natural Sciences
                        Computer and Information Sciences
                        Media and Communication Technology
                    @attributes: 
                        lang: swe
                        authority: uka.se
                    topic: 
                        Naturvetenskap
                        Data- och informationsvetenskap
                        Medieteknik
                    @attributes: 
                        lang: eng
                    topic: Hadoop
                    @attributes: 
                        lang: eng
                    topic: MapReduce
                    @attributes: 
                        lang: eng
                    topic: Monitoring
                    @attributes: 
                        lang: eng
                        authority: ltu
                    topic: Pervasive Mobile Computing
                    genre: Research subject
                    @attributes: 
                        lang: swe
                        authority: ltu
                    topic: Distribuerade datorsystem
                    genre: Research subject
            language: 
                languageTerm: eng
            genre: 
                conference/other
                vet
            note: 
                Published
                6
                
ISBN för värdpublikation: 978-1-7281-4222-7 978-1-7281-4223-4

            name: 
                    @attributes: 
                        type: personal
                    namePart: 
                        Demirbaga
                        Umit
                    role: 
                        roleTerm: aut
                    affiliation: Newcastle University. Bartin University
                    @attributes: 
                        type: personal
                    namePart: 
                        Noor
                        Ayman
                    role: 
                        roleTerm: aut
                    affiliation: Newcastle University. Taibah University
                    @attributes: 
                        type: personal
                    namePart: 
                        Wen
                        Zhenyu
                    role: 
                        roleTerm: aut
                    affiliation: Newcastle University
                    @attributes: 
                        type: personal
                    namePart: 
                        James
                        Philip
                    role: 
                        roleTerm: aut
                    affiliation: Newcastle University
                    @attributes: 
                        type: personal
                        authority: ltu
                    namePart: 
                        Mitra
                        Karan
                        Assistant Professor
                        1982-
                    role: 
                        roleTerm: aut
                    affiliation: 
                        Luleå tekniska universitet
                        Datavetenskap
                    nameIdentifier: 
                        karan
                        0000-0003-3489-7429
                    @attributes: 
                        type: personal
                    namePart: 
                        Ranjan
                        Rajiv
                    role: 
                        roleTerm: aut
                    affiliation: Newcastle University
            originInfo: 
                dateIssued: 2019
                publisher: IEEE
            relatedItem: 
                    @attributes: 
                        type: host
                    titleInfo: 
                        title: 2019 IEEE 38th International Symposium on Reliable Distributed Systems
                        subTitle: SRDS 2019
                    part: 
                        extent: 
                            start: 357
                            end: 359
                    @attributes: 
                        type: series
                    titleInfo: 
                        title: Symposium on Reliable Distributed Systems
                    identifier: 
                        1060-9857
                        2575-8462
            physicalDescription: 
                form: print
            typeOfResource: text