How Far Are We? The Triumphs and Trials of Generative AI in Learning Software Engineering
Abstrak
Conversational Generative AI (convo-genAI) is revolutionizing Software Engineering (SE) as engineers and academics embrace this technology in their work. However, there is a gap in understanding the current potential and pitfalls of this technology, specifically in supporting students in SE tasks. In this work, we evaluate through a between-subjects study (N=22) the effectiveness of ChatGPT, a convo-genAI platform, in assisting students in SE tasks. Our study did not find statistical differences in participants' productivity or self-efficacy when using ChatGPT as compared to traditional resources, but we found significantly increased frustration levels. Our study also revealed 5 distinct faults arising from violations of Human-AI interaction guidelines, which led to 7 different (negative) consequences on participants.
Artikel Ilmiah Terkait
Jennifer Brings Marian Daun
29 Juni 2023
This position paper discusses the potential for using generative AIs like ChatGPT in software engineering education. Currently, discussions center around potential threats emerging from student's use of ChatGPT. For instance, generative AI will limit the usefulness of graded homework dramatically. However, there exist potential opportunities as well. For example, ChatGPT's ability to understand and generate human language allows providing personalized feedback to students, and can thus accompany current software engineering education approaches. This paper highlights the potential for enhancing software engineering education. The availability of generative AI will improve the individualization of education approaches. In addition, we discuss the need to adapt software engineering curricula to the changed profiles of software engineers. Moreover, we point out why it is important to provide guidance for using generative AI and, thus, integrate it in courses rather than accepting the unsupervised use by students, which can negatively impact the students' learning.
Peng Liang Teerath Das T. Mikkonen + 3 lainnya
20 Oktober 2023
Artificial Intelligence has demonstrated its significance in software engineering through notable improvements in productivity, accuracy, collaboration, and learning outcomes. This study examines the impact of generative AI tools, specifically ChatGPT, on the software development experiences of undergraduate students. Over a three-month project with seven students, ChatGPT was used as a support tool. The research focused on assessing ChatGPT's effectiveness, benefits, limitations, and its influence on learning. Results showed that ChatGPT significantly addresses skill gaps in software development education, enhancing efficiency, accuracy, and collaboration. It also improved participants' fundamental understanding and soft skills. The study highlights the importance of incorporating AI tools like ChatGPT in education to bridge skill gaps and increase productivity, but stresses the need for a balanced approach to technology use. Future research should focus on optimizing ChatGPT's application in various development contexts to maximize learning and address specific challenges.
Olga Petrovska Lee Clift Rebecca Pearsall + 1 lainnya
5 Januari 2024
This paper explores how Generative AI can be incorporated into software development education. We present examples of formative and summative assessments, which explore various aspects of ChatGPT, including its coding capabilities, its ability to construct arguments as well as ethical issues of using ChatGPT and similar tools in education and the workplace. Our work is inspired by the insights from surveys that show that the learners on our Degree Apprenticeship Programme have a great interest in learning about and exploiting emerging AI technology. Similarly, our industrial partners have a clear interest for their employees to be formally prepared to use GenAI in their software engineering roles. In this vein, it is proposed that embedding the use of GenAI tools in a careful and creative way - by developing assessments which encourage learners to critically evaluate AI output - can be beneficial in helping learners understand the subject material being taught without the risk of the AI tools “doing the homework”.
Eric J. York
26 Oktober 2023
The advent of widely-accessible generative AI tools and their rapid adoption across industry and education is necessitating large-scale revisions to user experience design and web development pedagogies and curricula, a process that will take some time. This report describes a series of initial experiments using generative AI tools as a student or junior designer or web developer might, sometimes naïvely and sometimes in more sophisticated ways, to complete beginner-level and advanced projects. The report evaluates how ChatGPT performs across three categories of prompts (brainstorming, design, and coding) and assesses the quality of the outputs in order to inform the research design of a larger, ongoing interdisciplinary study in its initial phases and to document the results for instructors or senior members of design and development teams to aid them in assessing the fitness of generative AI for user experience design and web development production.
Christopher Bull Ahmed Kharrufa
24 Maret 2023
The use of Generative AI in software development is gaining traction. But what are the potentials and implications on software development education? We gathered insights on the use of Generative AI from professional software developers and make some pedagogical recommendations.
Daftar Referensi
2 referensiHow ChatGPT Will Change Software Engineering Education
Jennifer Brings Marian Daun
29 Juni 2023
This position paper discusses the potential for using generative AIs like ChatGPT in software engineering education. Currently, discussions center around potential threats emerging from student's use of ChatGPT. For instance, generative AI will limit the usefulness of graded homework dramatically. However, there exist potential opportunities as well. For example, ChatGPT's ability to understand and generate human language allows providing personalized feedback to students, and can thus accompany current software engineering education approaches. This paper highlights the potential for enhancing software engineering education. The availability of generative AI will improve the individualization of education approaches. In addition, we discuss the need to adapt software engineering curricula to the changed profiles of software engineers. Moreover, we point out why it is important to provide guidance for using generative AI and, thus, integrate it in courses rather than accepting the unsupervised use by students, which can negatively impact the students' learning.
Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming
Tovi Grossman Carl Ka To Ma + 4 lainnya
15 Februari 2023
AI code generators like OpenAI Codex have the potential to assist novice programmers by generating code from natural language descriptions, however, over-reliance might negatively impact learning and retention. To explore the implications that AI code generators have on introductory programming, we conducted a controlled experiment with 69 novices (ages 10-17). Learners worked on 45 Python code-authoring tasks, for which half of the learners had access to Codex, each followed by a code-modification task. Our results show that using Codex significantly increased code-authoring performance (1.15x increased completion rate and 1.8x higher scores) while not decreasing performance on manual code-modification tasks. Additionally, learners with access to Codex during the training phase performed slightly better on the evaluation post-tests conducted one week later, although this difference did not reach statistical significance. Of interest, learners with higher Scratch pre-test scores performed significantly better on retention post-tests, if they had prior access to Codex.
Artikel yang Mensitasi
2 sitasiTowards an Understanding of Large Language Models in Software Engineering Tasks
Kai-Chun Ning Zibin Zheng + 5 lainnya
22 Agustus 2023
Large Language Models (LLMs) have drawn widespread attention and research due to their astounding performance in text generation and reasoning tasks. Derivative products, like ChatGPT, have been extensively deployed and highly sought after. Meanwhile, the evaluation and optimization of LLMs in software engineering tasks, such as code generation, have become a research focus. However, there is still a lack of systematic research on applying and evaluating LLMs in software engineering. Therefore, this paper comprehensively investigate and collate the research and products combining LLMs with software engineering, aiming to answer two questions: (1) What are the current integrations of LLMs with software engineering? (2) Can LLMs effectively handle software engineering tasks? To find the answers, we have collected related literature as extensively as possible from seven mainstream databases and selected 123 timely papers published starting from 2022 for analysis. We have categorized these papers in detail and reviewed the current research status of LLMs from the perspective of seven major software engineering tasks, hoping this will help researchers better grasp the research trends and address the issues when applying LLMs. Meanwhile, we have also organized and presented papers with evaluation content to reveal the performance and effectiveness of LLMs in various software engineering tasks, guiding researchers and developers to optimize.
Exploring students' perspectives on Generative AI-assisted academic writing
Rita Detrick Seongryeong Yu + 2 lainnya
31 Juli 2024
The rapid development of generative artificial intelligence (GenAI), including large language models (LLM), has merged to support students in their academic writing process. Keeping pace with the technical and educational landscape requires careful consideration of the opportunities and challenges that GenAI-assisted systems create within education. This serves as a useful and necessary starting point for fully leveraging its potential for learning and teaching. Hence, it is crucial to gather insights from diverse perspectives and use cases from actual users, particularly the unique voices and needs of student-users. Therefore, this study explored and examined students' perceptions and experiences about GenAI-assisted academic writing by conducting in-depth interviews with 20 Chinese students in higher education after completing academic writing tasks using a ChatGPT4-embedded writing system developed by the research team. The study found that students expected AI to serve multiple roles, including multi-tasking writing assistant, virtual tutor, and digital peer to support multifaceted writing processes and performance. Students perceived that GenAI-assisted writing could benefit them in three areas including the writing process, performance, and their affective domain. Meanwhile, they also identified AI-related, student-related, and task-related challenges that were experienced during the GenAI-assisted writing activity. These findings contribute to a more nuanced understanding of GenAI's impact on academic writing that is inclusive of student perspectives, offering implications for educational AI design and instructional design.