ML DevOps Adoption in Practice: A Mixed-Method Study of Implementation Patterns and Organizational Benefits

Saeed Albarhami Charalampos Apostolopoulos I. Karamitsos

13 Juli 2020

This paper proposes DevOps practices for machine learning application, integrating both the development and operation environment seamlessly. The machine learning processes of development and deployment during the experimentation phase may seem easy. However, if not carefully designed, deploying and using such models may lead to a complex, time-consuming approaches which may require significant and costly efforts for maintenance, improvement, and monitoring. This paper presents how to apply continuous integration (CI) and continuous delivery (CD) principles, practices, and tools so as to minimize waste, support rapid feedback loops, explore the hidden technical debt, improve value delivery and maintenance, and improve operational functions for real-world machine learning applications.

Gopalakrishnan Sriraman S. R

6 Oktober 2023

Software and information systems have become a core competency for every business in this connected world. Any enhancement in software delivery and operations will tremendously impact businesses and society. Sustainable software development is one of the key focus areas for software organizations. The application of intelligent automation leveraging artificial intelligence and cloud computing to deliver continuous value from software is in its nascent stage across the industry and is evolving rapidly. The advent of agile methodologies with DevOps has increased software quality and accelerated its delivery. Numerous software organizations have adopted DevOps to develop and operate their software systems and improve efficiency. Software organizations try to implement DevOps activities by taking advantage of various expert services. The adoption of DevOps by software organizations is beset with multiple challenges. These issues can be overcome by understanding and structurally addressing the pain points. This paper presents the preliminary analysis of the interviews with the relevant stakeholders. Ground truths were established and applied to evaluate various machine learning algorithms to compare their accuracy and test our hypothesis. This study aims to help researchers and practitioners understand the adoption of DevOps and the contexts in which the DevOps practices are viable. The experimental results will show that machine learning can predict an organization's readiness to adopt DevOps.

Sebastian Hirschl Niklas Kühl Dominik Kreuzberger

4 Mei 2022

The final goal of all industrial machine learning (ML) projects is to develop ML products and rapidly bring them into production. However, it is highly challenging to automate and operationalize ML products and thus many ML endeavors fail to deliver on their expectations. The paradigm of Machine Learning Operations (MLOps) addresses this issue. MLOps includes several aspects, such as best practices, sets of concepts, and development culture. However, MLOps is still a vague term and its consequences for researchers and professionals are ambiguous. To address this gap, we conduct mixed-method research, including a literature review, a tool review, and expert interviews. As a result of these investigations, we contribute to the body of knowledge by providing an aggregated overview of the necessary principles, components, and roles, as well as the associated architecture and workflows. Furthermore, we provide a comprehensive definition of MLOps and highlight open challenges in the field. Finally, this work provides guidance for ML researchers and practitioners who want to automate and operate their ML products with a designated set of technologies.

C. Curino Bojan Karlas Markus Weimer + 10 lainnya

29 Juli 2022

The recent success of machine learning (ML) has led to an explosive growth of systems and applications built by an ever-growing community of system builders and data science (DS) practitioners. This quickly shifting panorama, however, is challenging for system builders and practitioners alike to follow. In this paper, we set out to capture this panorama through a wide-angle lens, performing the largest analysis of DS projects to date, focusing on questions that can advance our understanding of the field and determine investments. Specifically, we download and analyze (a) over 8M notebooks publicly available on GITHUB and (b) over 2M enterprise ML pipelines developed within Microsoft. Our analysis includes coarse-grained statistical characterizations, finegrained analysis of libraries and pipelines, and comparative studies across datasets and time. We report a large number of measurements for our readers to interpret and draw actionable conclusions on (a) what system builders should focus on to better serve practitioners and (b) what technologies should practitioners rely on.

T. Offerman Robert Blinde C. J. Stettina + 1 lainnya

19 Juni 2022

Many organizations adopt DevOps practices and tools in order to break down silos within the organization, improve software quality and delivery, and increase customer satisfaction. However, the impact of the individual practices on the performance of the organization is not well known. In this paper, we collect evidence on the effects of DevOps practices and tools on organizational performance. In an extensive literature search we identified 14 DevOps practices, consisting of 47 sub-practices. Based on these practices, we conducted a global survey to study their effects in practice, and measure DevOps maturity. Across 123 respondents, working in 11 different industries, we found that 13 of the 14 DevOps practices are adopted, determined by 50 % of the participants indicating that practices are ‘always’, ‘most of the time’, and ‘about half of the time’ applied. There is a positive correlation between the adoption of all practices and independently measured maturity. In particular, practices concerning sandboxes for minimum deployment, test-driven development, and trunk based development show the lowest correlations in our data. Effects of software delivery and organizational performance are mainly perceived positive. Yet, DevOps is also considered by some to have a negative impact such as respondents mentioning the predictability of product delivery has decreased and work is less fun. Concluding, our detailed overview of DevOps practices allows more targeted application of DevOps practices to obtain its positive effects while minimizing any negative effects.