A machine learning-driven framework for error detection and operational monitoring of ATLAS job reports inside INFN-CNAF big data platform

Modern data centers provide the IT infrastructure required to efficiently deliver resources, services, monitoring systems,and data to users efficiently. Over time, they have evolved to accommodate increasing resources demands and ever-growing diverse use cases. At INFN’s CNAF, a Big Data Platform (BDP) has been developed to collect and index log reports from CNAF facilities. This ongoing project serves the Italian groups participating in high-energy physics experiments. Within this framework, for the first time, a dedicated data pipeline was implemented for the ATLAS experiment, sourcing input from the ATLAS Distributed Computing system (PanDA) and collecting reports of computational jobs executed on the INFN Tier-1 farm, thus establishing a first cooperation between the two infrastructures. The received operational logs were stored in the BDP, forming a comprehensive database of job-related information. Once a sufficiently large and consistent set of logs had been accumulated, the next step was to develop a machine learning framework,embedding two different machine learning models, the first able to perform binary classification of job outcomes (successful or failed), and the second aimed at categorizing the various types of errors which occurred during the execution of failed jobs.