1st Workshop on Evaluation of Qualitative Aspects of Intelligent Software Assistants (EQUISA)

1st Workshop on Evaluation of Qualitative Aspects

of Intelligent Software Assistants (EQUISA)

- Call for Papers -

Co-located with EASE 2025

June 17–20, 2025, Istanbul, Turkey

https://conf.researchr.org/home/ease-2025/equisa-2025

Important Dates

• Submission deadline: March 16th, 2025.
• Notification to authors: April 13th, 2025.
• Camera-ready due: April 26th, 2025.
• Early registration deadline for authors: May 5th, 2025.
• Workshop date: 20th June 2025.

Motivation

With the growing complexity of modern software systems, software engineers need to cope with the so-called information overloading along the whole development lifecycle, spanning from the requirement elicitation to the development of the actual system. In addition, fast-evolving technologies and frameworks are emerging daily. Therefore, non-expert users may struggle to express the requirements properly or select the proper third-party software libraries needed to implement a specific functionality. Such inconvenience impacts mainly the software design and construction phases, which means more than 50% of the effort made by the software engineers during the life of the project. Current software projects need to be easily scalable to reduce such maintenance costs.

To cope with these issues, intelligent software assistants have been proposed to ease the burden of choice by providing a set of automated capabilities to help developers in several tasks, e.g., debugging, testing, navigating Q&A forums, or mining information for open-source repositories. After an inference phase, the system can provide a set of valuable items, namely recommendations, according to the current task.

While traditional systems are based on a curated knowledge base that represents the main source of the recommendation process, the advent of cutting-edge AI models, or the so-called Large Language Models (LLMs) like those of the GPT family, are dramatically changing how these systems are designed, developed, and evaluated. Currently, IDEs like Visual Studio and Eclipse are being extended with LLM-based assistants, e.g., Copilot or Caret.

In this respect, a key point is to ensure a set of qualitative aspects beyond the accuracy of those assistants. Concretely, the provided items must be free from any kind of bias, ensure the user’s privacy, adhere to software licenses, and, overall, contribute to building reliable and trustworthy software projects. This objective has been recently recognized by the European Commission, which proposed the AI act, a dedicated standard for prominent AI-intensive systems providing a wide range of requirements, methodologies, and metrics focused on ensuring the mentioned qualitative aspects.

Thus, there is a need to assess the outcomes of intelligent software assistants by adhering to rigorous protocols and methodologies provided by the empirical software engineering field of study. Motivated by the above mentioned challenges, this workshop aims to provide a dedicated venue to develop and validate the new generation of intelligent software assistants to meet the most prominent quality aspects.

Topics

Topics of interest include, but are not restricted to:

• Re-usage of AI-based tools, techniques, and methodologies in developing intelligent software assistants.
• Foundational theories for software assistants to understand the underlying principles that can drive the development of more robust and generalizable recommendation systems in software engineering, with a focus on their evaluation.
• Evaluating quality aspects of software assistants, e.g., explainability, transparency, and fairness, ensuring that soft- ware assistants produce reliable results.
• New methods, tools, and frameworks to support development tasks, e.g., code-related tasks, automated classification of software artifacts, or code generation leveraging generative AI models.
• Designing specific prompt engineering techniques for intelligent software assistants based on large language models to ensure quality aspects.
• Data-driven approaches for software assistants: Leveraging large-scale data from open-source software (OSS) repositories, Q&A forums, and issue trackers to enhance the effectiveness of software assistants.
• Integration with human-in-the-loop systems: Balancing automated recommendations with human expertise to improve decision-making in complex SE scenarios.
• Adoption of advanced generative AI models, including LLMs, pre-trained models (PTMs) for software assistance, particularly emphasizing the quality effects.
• Empirical studies and controlled experiments to assess qualitative aspects of intelligent systems.
• Evolution of software systems and long-term recommendations, e.g., how software assistants can cope with the evolving nature of software systems and provide recommendations that consider long-term system maintainability and evolution.
• Cross-disciplinary applications of software assistant: Studying how techniques from other domains, e.g., human-computer interaction, natural language processing, and social network analysis, can enhance their effectiveness and usability.
• Surveys and experience reports on software assistants to support software engineering tasks both in academic and industry use cases.

Submission guidelines

The following types of submissions are solicited:

1. Full papers must be at least 5 pages and no more than 10, reporting original research on the topics.
2. Short papers must be exactly 5 pages, presenting visions, novel ideas, and experience reports on the topics.

Our review process will be following the same criteria of the main conference, namely:

• Soundness: The extent to which the paper’s contribution addresses its research questions and is supported by rigorous application of appropriate research methods.
• Significance: The extent to which the paper’s contributions can impact the field of software engineering and under which assumptions (if any).
• Novelty: The extent to which the contributions are sufficiently original with respect to the state-of-the-art.
• Verifiability and Transparency: The extent to which the paper includes sufficient information to understand how an innovation works, how data was obtained, analyzed, and interpreted, and how the paper supports independent verification or replication of the paper’s claimed contributions.
• Presentation: The extent to which the quality of writing meets the high standards, including clear descriptions, adequate use of the English language, absence of major ambiguity, clearly readable figures and tables, and adherence to the formatting instructions provided above.

All accepted papers will be part of the EASE 2025 proceedings under the copyright of the ACM digital library.

Submissions must adhere to the ACM formatting instructions, which can be found at https://www.acm.org/publications/proceedings-template for both LaTeX and Word users. Deviating from the ACM formatting instructions may lead to a desk rejection. Authors must comply with the SIGSOFT OSP
https://github.com/acmsigsoft/open-science-policies/blob/master/sigsoft-open-science-policies.md, (i.e., to archive data and artifacts in a permanent repository—e.g., Zenodo, not GitHub—and include links in the submission). By submitting to EQUISA, authors agree to the ACM Policy and Procedures on Plagiarism, Misrepresentation, and Falsification https://www.acm.org/publications/policies/plagiarism-overview. Papers submitted must not be published or under review elsewhere. The Program Chairs may use plagiarism detection software under contract to the ACM. If the research involves human subjects, the authors must adhere to the ACM Publications Policy
https://www.acm.org/publications/policies/research-involving-human-participants-and-subjects.

Workshop Organizers

• Claudio Di Sipio (University of L’Aquila, Italy)
• Valeria Pontillo (Vrije, Universiteit Brussel)
• Riccardo Rubei (University of L’Aquila, Italy)
• Pablo Gómez-Abajo (Universidad Autónoma de Madrid, Spain)

Program Committee

• Lissette Almonte, Universidad Autónoma de Madrid, Spain
• Lola Burgueño, Universidad de Málaga, Spain
• Pablo C. Cañizares, Universidad Complutense de Madrid, Spain
• Sergio Di Meglio, Department of Electrical Engineering and Information Technology Università degli Studi di Napoli Federico II
• Antonio Garmendia, Universidad Autónoma de Madrid, Spain
• Esther Guerra, Universidad Autónoma de Madrid, Spain
• Andrea Lops, Polytechnic University of Bari, Italy
• Alberto Mancino
• Alberto Núñez, Universidad Complutense de Madrid, Spain
• Gianmario Voria, University of Salerno
• Martin Weyssow, Singapore Management University
• Giordano d'Aloisio, University of L'Aquila, Italy