ChatGPT in Action: Analyzing Its Use in Software Development

Peng Liang Teerath Das T. Mikkonen + 3 lainnya

20 Oktober 2023

Artificial Intelligence has demonstrated its significance in software engineering through notable improvements in productivity, accuracy, collaboration, and learning outcomes. This study examines the impact of generative AI tools, specifically ChatGPT, on the software development experiences of undergraduate students. Over a three-month project with seven students, ChatGPT was used as a support tool. The research focused on assessing ChatGPT's effectiveness, benefits, limitations, and its influence on learning. Results showed that ChatGPT significantly addresses skill gaps in software development education, enhancing efficiency, accuracy, and collaboration. It also improved participants' fundamental understanding and soft skills. The study highlights the importance of incorporating AI tools like ChatGPT in education to bridge skill gaps and increase productivity, but stresses the need for a balanced approach to technology use. Future research should focus on optimizing ChatGPT's application in various development contexts to maximize learning and address specific challenges.

Sourav Mazumdar G. Sridhara Ranjani H.G.

26 Mei 2023

ChatGPT (Chat Generative Pre-trained Transformer) is a chatbot launched by OpenAI on November 30, 2022. OpenAI's GPT-3 family of large language models serve as the foundation for ChatGPT. ChatGPT is fine-tuned with both supervised and reinforcement learning techniques and has received widespread attention for its articulate responses across diverse domains of knowledge. In this study, we explore how ChatGPT can be used to help with common software engineering tasks. Many of the ubiquitous tasks covering the breadth of software engineering such as ambiguity resolution in software requirements, method name suggestion, test case prioritization, code review, log summarization can potentially be performed using ChatGPT. In this study, we explore fifteen common software engineering tasks using ChatGPT. We juxtapose and analyze ChatGPT's answers with the respective state of the art outputs (where available) and/or human expert ground truth. Our experiments suggest that for many tasks, ChatGPT does perform credibly and the response from it is detailed and often better than the human expert output or the state of the art output. However, for a few other tasks, ChatGPT in its present form provides incorrect answers and hence is not suited for such tasks.

Tahsinur Rahman M. Akbar Muhammad Hamza + 1 lainnya

17 Desember 2023

This paper investigates the dynamics of human-AI collaboration in software engineering, focusing on the use of ChatGPT. Through a thematic analysis of a hands-on workshop in which 22 professional software engineers collaborated for three hours with ChatGPT, we explore the transition of AI from a mere tool to a collaborative partner. The study identifies key themes such as the evolving nature of human-AI interaction, the capabilities of AI in software engineering tasks, and the challenges and limitations of integrating AI in this domain. The findings show that while AI, particularly ChatGPT, improves the efficiency of code generation and optimization, human oversight remains crucial, especially in areas requiring complex problem-solving and security considerations. This research contributes to the theoretical understanding of human-AI collaboration in software engineering and provides practical insights for effectively integrating AI tools into development processes. It highlights the need for clear role allocation, effective communication, and balanced AI-human collaboration to realize the full potential of AI in software engineering.

Piyush Kulkarni Vinay Kulkarni Akanksha Somase + 1 lainnya

22 Februari 2024

The Software Development Life Cycle (SDLC) comprises multiple phases, each requiring Subject Matter Experts (SMEs) with phase-specific skills. The efficacy and quality of deliverables of each phase are skill dependent. In recent times, Generative AI techniques, including Large-scale Language Models (LLMs) like GPT, have become significant players in software engineering. These models, trained on extensive text data, can offer valuable contributions to software development. Interacting with LLMs involves feeding prompts with the context information and guiding the generation of textual responses. The quality of the response is dependent on the quality of the prompt given. This paper proposes a systematic prompting approach based on meta-model concepts for SDLC phases. The approach is validated using ChatGPT for small but complex business application development. We share the approach and our experience, learnings, benefits obtained, and the challenges encountered while applying the approach using ChatGPT. Our experience indicates that Generative AI techniques, such as ChatGPT, have the potential to reduce the skills barrier and accelerate software development substantially.

Michael J. Muller Justin D. Weisz Stephanie Houde + 2 lainnya

14 Februari 2023

Large language models (LLMs) have recently been applied in software engineering to perform tasks such as translating code between programming languages, generating code from natural language, and autocompleting code as it is being written. When used within development tools, these systems typically treat each model invocation independently from all previous invocations, and only a specific limited functionality is exposed within the user interface. This approach to user interaction misses an opportunity for users to more deeply engage with the model by having the context of their previous interactions, as well as the context of their code, inform the model’s responses. We developed a prototype system – the Programmer’s Assistant – in order to explore the utility of conversational interactions grounded in code, as well as software engineers’ receptiveness to the idea of conversing with, rather than invoking, a code-fluent LLM. Through an evaluation with 42 participants with varied levels of programming experience, we found that our system was capable of conducting extended, multi-turn discussions, and that it enabled additional knowledge and capabilities beyond code generation to emerge from the LLM. Despite skeptical initial expectations for conversational programming assistance, participants were impressed by the breadth of the assistant’s capabilities, the quality of its responses, and its potential for improving their productivity. Our work demonstrates the unique potential of conversational interactions with LLMs for co-creative processes like software development.