Logs are essential for understanding Continuous Integration (CI) behavior, particularly for diagnosing build failures and performance regressions. Yet their growing volume and verbosity make both manual inspection and automated analysis increasingly costly, time-consuming, and environmentally costly. While prior work has explored log compression, anomaly detection, and LLM-based log analysis, most efforts target structured system logs rather than the unstructured, noisy, and verbose logs typical of CI workflows.

Mobile apps face strong pressure for fast and reliable updates. Continuous Integration (CI) helps automate builds, tests, and releases, but its impact on mobile development remains underexplored. Despite the widespread use of CI, little is known about how it affects development activity, release speed, and user-facing outcomes in mobile projects. Existing studies mostly focus on CI adoption in general-purpose software, providing limited insight into mobile-specific dynamics, such as app store visibility and user engagement. In this paper, we analyze open-source Android apps to (1) compare CI adopters and non-adopters, (2) characterize adoption patterns using activity and bug metrics, and (3) assess pre/post adoption changes and user-facing outcomes. We observe that CI adopters are larger and more active, with faster and more regular releases. CI adoption is concentrated in integration- and reliability-intensive categories (e.g., finance and productivity) and is associated with higher Google Play Store engagement (more downloads and reviews) without lower ratings. Overall, CI adoption aligns with practices that support sustained delivery, higher project visibility, and stronger user engagement in mobile ecosystems.

The exponential growth of the mobile app market underscores the importance of constant innovation and rapid response to user demands. As user satisfaction is paramount to the success of a mobile application (app), developers typically rely on user reviews, which represent user feedback that includes ratings and comments to identify areas for improvement. However, the sheer volume of user reviews poses challenges in manual analysis, necessitating automated approaches. Existing automated approaches either analyze only the target app’s reviews, neglecting the comparison of similar features to competitors or fail to provide suggestions for feature enhancement. To address these gaps, we propose a Large Language Model (LLM)-based Competitive User Review Analysis for Feature Enhancement) ( LLM-Cure ), an approach powered by LLMs to automatically generate suggestions for mobile app feature improvements. More specifically, LLM-Cure identifies and categorizes features within reviews by applying LLMs. When provided with a complaint in a user review, LLM-Cure curates highly rated (4 and 5 stars) reviews in competing apps related to the complaint and proposes potential improvements tailored to the target application. We evaluate LLM-Cure on 1,056,739 reviews of 70 popular Android apps. Our evaluation demonstrates that LLM-Cure significantly outperforms the state-of-the-art approaches in assigning features to reviews by up to 13% in F1-score, 16% in recall, and 11% in precision. Additionally, LLM-Cure demonstrates its capability to provide suggestions for resolving user complaints. We verify the suggestions using the release notes that reflect the changes of features in the target mobile app. LLM-Cure achieves a promising average of 73% of the implementation of the provided suggestions, demonstrating its potential for competitive feature enhancement.

Continuous Integration and Continuous Delivery (CI/CD) is a well-established practice that automatically builds, tests, packages, and deploys software systems. To adopt CI/CD, software developers need to configure their projects using dedicated YML configuration files. Mobile apps have distinct characteristics with respect to CI/CD practices, such as testing on various emulators and deploying to app stores. However, little is known about the challenges and added value of adopting CI/CD in mobile apps and how developers maintain such a practice. In this article, we conduct an empirical study on CI/CD practices in \(2{,}557\) Android apps adopting 4 popular CI/CD services, namely GitHub Actions, Travis CI, CircleCI, and GitLab CI/CD. We also compare our findings with those reported in prior research on general CI/CD practices to situate them within broader trends. We observe a lack of commonality and standardization across CI/CD services and Android apps, leading to complex YML configurations and associated maintenance efforts. We also observe that CI/CD configurations focus primarily on the build setup, with around half of the projects performing standard testing and only 9% incorporating deployment. In addition, we find that CI/CD configurations are changed bi-monthly on average, with frequent maintenance correlating with active issue tracking, project size/age, and community engagement. Our qualitative analysis of commits uncovered 11 themes in CI/CD maintenance activities, with over a third of the changes focusing on improving workflows and fixing build issues, whereas another third involves updating the build environment, tools, and dependencies. Our study emphasizes the necessity for automation and AI-powered tools to improve CI/CD processes for mobile apps and advocates creating adaptable open source tools to efficiently manage resources, especially in testing and deployment.

The sustainability of libraries is critical for modern software development, yet many libraries face abandonment, posing significant risks to dependent projects. This study explores the prevalence and patterns of library abandonment in the Maven ecosystem. We investigate abandonment trends over the past decade, revealing that approximately one in four libraries fail to survive beyond their creation year. We also analyze the release activities of libraries, focusing on their lifespan and release speed, and analyze the evolution of these metrics within the lifespan of libraries. We find that while slow release speed and relatively long periods of inactivity are often precursors to abandonment, some abandoned libraries exhibit bursts of high frequent release activity late in their life cycle. Our findings contribute to a new understanding of library abandonment dynamics and offer insights for practitioners to identify and mitigate risks in software ecosystems.

Deep Learning (DL) frameworks play a critical role in advancing AI, and their rapid growth underscores the need for a comprehensive understanding of software quality and maintainability. DL frameworks, like other systems, are prone to code clones. Code clones refer to identical or highly similar source code fragments within the same project or even across different projects. Code cloning can have positive and negative implications for software development, influencing maintenance, readability, and bug propagation. While the existing studies focus on studying clones in DL-based applications, to our knowledge, no work has been done investigating clones, their evolution, and their impact on the maintenance of DL frameworks. In this article, we aim to address the knowledge gap concerning the evolutionary dimension of code clones in DL frameworks and the extent of code reuse across these frameworks. We empirically analyze code clones in nine popular DL frameworks, i.e., TensorFlow , Paddle , PyTorch , Aesara , Ray , MXNet , Keras , Jax , and BentoML , to investigate (1) the characteristics of the long-term code cloning evolution over releases in each framework, (2) the short-term, i.e., within-release, code cloning patterns and their influence on the long-term trends, and (3) the file-level code clones within the DL frameworks. Our findings reveal that DL frameworks adopt four distinct cloning trends: “Serpentine,” “Rise and Fall,” “Decreasing,” and “Stable” and that these trends present some common and distinct characteristics. For instance, bug-fixing activities persistently happen in clones irrespective of the clone evolutionary trend but occur more in the “Serpentine” trend. Moreover, the within-release level investigation demonstrates that short-term code cloning practices impact long-term cloning trends. The cross-framework code clone investigation reveals the presence of functional and architectural adaptation file-level cross-framework code clones across the nine studied frameworks. We provide insights that foster robust clone practices and collaborative maintenance in the development of DL frameworks.

Performance bugs are non-functional defects that can significantly reduce the performance of an application (e.g., software hanging or freezing) and lead to poor user experience. Prior studies found that each type of performance bugs follows a unique code-based performance anti-pattern and proposed different approaches to detect such anti-patterns by analyzing the source code of a program. However, each approach can only recognize one performance anti-pattern. Different approaches need to be applied separately to identify different performance anti-patterns. To predict a large variety of performance bug types using a unified approach, we propose an approach that predicts performance bugs by leveraging various historical data (e.g., source code and code change history). We collect performance bugs from 80 popular Java projects. Next, we propose performance code metrics to capture the code characteristics of performance bugs. We build performance bug predictors using machine learning models, such as Random Forest, eXtreme Gradient Boosting, and Linear Regressions. We observe that: (1) Random Forest and eXtreme Gradient Boosting are the best algorithms for predicting performance bugs at a file level with a median of 0.84 AUC, 0.21 PR-AUC, and 0.38 MCC; (2) The proposed performance code metrics have the most significant impact on the performance of our models compared to code and process metrics. In particular, the median AUC, PR-AUC, and MCC of the studied machine learning models drop by 7.7%, 25.4%, and 20.2% without using the proposed performance code metrics; and (3) Our approach can predict additional performance bugs that are not covered by the anti-patterns proposed in the prior studies.

Video gaming now represents the largest category in the entertainment industry in terms of revenue. To expand their market share, game developers are creating more cross-platform games, which are compatible with various platforms, including PCs, consoles, and smartphones. However, creating such games poses challenges as developers encounter platform-specific issues that may only surface on one of the target platforms. Consequently, many ported games fail due to careless adaptation from one exclusive platform to another. This paper presents the first empirical study on cross-platform issues by analyzing game users’ reviews for video games on both PC and game console(s). Our findings reveal that platform-related issues occur more frequently on the PC side, particularly for games that are ported from consoles. To address this challenge, we develop machine learning-based approaches to automatically identify and categorize reviews discussing platform-related issues, achieving a reasonable classification performance with 79.73% to 90.06% accuracy. Our approach would help cross-platform game developers save considerable time when analyzing user reviews.

Upon receiving a new issue report, practitioners start by investigating the defect type, the potential fixing effort needed to resolve the defect and the change impact. Moreover, issue reports contain valuable information, such as, the title, description and severity, and researchers leverage the topics of issue reports as a collective metric portraying similar characteristics of a defect. Nonetheless, none of the existing studies leverage the defect topic, i.e., a semantic cluster of defects of the same nature, such as Performance, GUI, and Database , to estimate the change impact that represents the amount of change needed in terms of code churn and the number of files changed. To this end, in this article, we conduct an empirical study on 298,548 issue reports belonging to three large-scale open-source systems, i.e., Mozilla, Apache, and Eclipse, to estimate the change impact in terms of code churn or the number of files changed while leveraging the topics of issue reports. First, we adopt the Embedded Topic Model (ETM), a state-of-the-art topic modelling algorithm, to identify the topics. Second, we investigate the feasibility of predicting the change impact using the identified topics and other information extracted from the issue reports by building eight prediction models that classify issue reports requiring small or large change impact along two dimensions, i.e., the code churn size and the number of files changed. Our results suggest that XGBoost is the best-performing algorithm for predicting the change impact, with an AUC of 0.84, 0.76, and 0.73 for the code churn and 0.82, 0.71, and 0.73 for the number of files changed metric for Mozilla, Apache, and Eclipse, respectively. Our results also demonstrate that the topics of issue reports improve the recall of the prediction model by up to 45%.

The Continuous Integration (CI) practice allows developers to build software projects automatically and more frequently. However, CI builds may undergo long build durations or frequent build breakages, which we refer to asbuild performance. Both long durations and frequent breakages of CI builds can impede developers from engaging in other development activities. Prior research has conducted independent studies on build durations or build breakages. However, there is little attention to the possible interplay between reducing build durations and build breakages. In particular, it is unclear from prior studies (i) whether and how build performance is influenced by the context of projects; (ii) whether the actions to reduce build durations would reduce or increase build breakages; and (iii) whether fixing build breakages would lead to longer or faster builds. It is important for developers to understand the practices that make both timely and passing CI builds. In this paper, we conduct experimental and survey studies on the practices that can have dual or inverse associations with twobuild performancemeasures:build durationsandbuild breakages. To this end, we extend an existing dataset calledTravisTorrentto exclude inactive projects and collect recent builds of active projects. As a result, we study 924,616 CI builds from 588GitHubprojects that are linked withTravis CI. In addition, we survey developers who contributed to the projects in our dataset to get their feedback on our experimental observations. First, we investigate project-level metrics and find that project characteristics have a significant association with build durations and breakages. In addition, we investigate how build-level metrics are associated with both build durations and breakages and observe an evident interplay between them. In particular, we observe that actions to fix build breakages (e.g., retrying or waiting for build commands) not only increase build durations but also do not guarantee passing builds. We also find that improving the build performance of a project is dependent on the current build durations and breakages of that project. Furthermore, we analyze how build performance changes over time and observe nearly a third of projects in which one performance measure is sacrificed in favor of the other, especially when not possible to achieve both together. The majority of our experimental observations are confirmed by survey results, which provide useful insights though some survey responses disagree with some of our experimental observations. Our work (a) provides developers with development and building practices to maintain timely and passing CI builds, and (b) encourages researchers to highlight any potential dual or inverse side effects when reporting actionable findings about CI builds.

The competing nature of the app market motivates us to shift our focus on apps that provide similar functionalities and directly compete with each other (i.e., peer apps). In this work, we study the ratings and the review text of 100 Android apps across 10 peer app groups. We highlight the importance of performing peer-app analysis by showing that it can provide a unique perspective over performing a global analysis of apps (i.e., mixing apps from multiple categories). First, we observe that comparing user ratings within peer groups can provide very different results from comparing user ratings from a global perspective. Then, we show that peer-app analysis provides a different perspective to spot the dominant topics in the user reviews, and to understand the impact of the topics on user ratings. Our findings suggest that future efforts may pay more attention to performing and supporting app analysis from a peer group context. For example, app store owners may consider an additional rating mechanism that normalizes app ratings within peer groups, and future research may help developers understand the characteristics of specific peer groups and prioritize their efforts.

In-app advertisements have become a major revenue source for app developers in the mobile app ecosystem. Ad libraries play an integral part in this ecosystem as app developers integrate these libraries into their apps to display ads. In this paper, we study ad library integration strategies by analyzing 35,459 updates of 1,837 top free-to-download apps of the Google Play Store. We observe that ad libraries (e.g., Google AdMob) are not always used for serving ads – 22.5% of the apps that integrate Google AdMob do not display ads. They instead depend on Google AdMob for analytical purposes. Among the apps that display ads, we observe that 57.9% of them integrate multiple ad libraries. We observe that such integration of multiple ad libraries occurs commonly in apps with a large number of downloads and ones in app categories with a high proportion of ad-displaying apps. We manually analyze a sample of apps and derive a set of rules to automatically identify four common strategies for integrating multiple ad libraries. Our analysis of the apps across the identified strategies shows that app developers prefer to manage their own integrations instead of using off-the-shelf features of ad libraries for integrating multiple ad libraries. Our findings are valuable for ad library developers who wish to learn first hand about the challenges of integrating ad libraries.

UI (User Interface) is an essential factor influencing users’ perception of an app. However, it is hard for even professional designers to determine if the UI is good or not for end-users. Users’ feedback (e.g., user reviews in the Google Play) provides a way for app owners to understand how the users perceive the UI. In this article, we conduct an in-depth empirical study to analyze the UI issues of mobile apps. In particular, we analyze more than 3M UI-related reviews from 22,199 top free-to-download apps and 9,380 top non-free apps in the Google Play Store. By comparing the rating of UI-related reviews and other reviews of an app, we observe that UI-related reviews have lower ratings than other reviews. By manually analyzing a random sample of 1,447 UI-related reviews with a 95% confidence level and a 5% interval, we identify 17 UI-related issues types that belong to four categories (i.e., “Appearance,” “Interaction,” “Experience,” and “Others” ). In these issue types, we find “Generic Review” is the most occurring one. “Comparative Review” and “Advertisement” are the most negative two UI issue types. Faced with these UI issues, we explore the patterns of interaction between app owners and users. We identify eight patterns of how app owners dialogue with users about UI issues by the review-response mechanism. We find “Apology or Appreciation” and “Information Request” are the most two frequent patterns. We find updating UI timely according to feedback is essential to satisfy users. Besides, app owners could also fix UI issues without updating UI, especially for issue types belonging to “Interaction” category. Our findings show that there exists a positive impact if app owners could actively interact with users to improve UI quality and boost users’ satisfactoriness about the UIs.

Developer chatrooms (e.g., the Gitter platform) are gaining popularity as a communication channel among developers. In developer chatrooms, a developer ( asker ) posts questions and other developers ( respondents ) respond to the posted questions. The interaction between askers and respondents results in a discussion thread . Recent studies show that developers use chatrooms to inquire about issues, discuss development ideas, and help each other. However, prior work focuses mainly on analyzing individual messages of a chatroom without analyzing the discussion thread in a chatroom. Developer chatroom discussions are context-sensitive, entangled, and include multiple participants that make it hard to accurately identify threads. Therefore, prior work has limited capability to show the interactions among developers within a chatroom by analyzing only individual messages. In this article, we perform an in-depth analysis of the Gitter platform (i.e., developer chatrooms) by analyzing 6,605,248 messages of 709 chatrooms. To analyze the characteristics of the posted questions and the impact on the response behavior (e.g., whether the posted questions get responses), we propose an approach that identifies discussion threads in chatrooms with high precision (i.e., 0.81 F-score). Our results show that inactive members responded more often and unique questions take longer discussion time than simple questions. We also find that clear and concise questions are more likely to be responded to than poorly written questions. We further manually analyze a randomly selected sample of 384 threads to examine how respondents resolve the raised questions. We observe that more than 80% of the studied threads are resolved. Advanced-level/beginner-level questions along with the edited questions are the mostly resolved questions. Our results can help the project maintainers understand the nature of the discussion threads (e.g., the topic trends). Project maintainers can also benefit from our thread identification approach to spot the common repeated threads and use these threads as frequently asked questions (FAQs) to improve the documentation of their projects.

High performance is a critical factor to achieve and maintain the success of a software system. Performance anomalies represent the performance degradation issues (e.g., slowing down in system response times) of software systems at run-time. Performance anomalies can cause a dramatically negative impact on users’ satisfaction. Prior studies propose different approaches to detect anomalies by analyzing execution logs and resource utilization metrics after the anomalies have happened. However, the prior detection approaches cannot predict the anomalies ahead of time; such limitation causes an inevitable delay in taking corrective actions to prevent performance anomalies from happening. We propose an approach that can predict performance anomalies in software systems and raise anomaly warnings in advance. Our approach uses a Long-Short Term Memory neural network to capture the normal behaviors of a software system. Then, our approach predicts performance anomalies by identifying the early deviations from the captured normal system behaviors. We conduct extensive experiments to evaluate our approach using two real-world software systems (i.e., Elasticsearch and Hadoop). We compare the performance of our approach with two baselines. The first baseline is one state-to-the-art baseline called Unsupervised Behavior Learning. The second baseline predicts performance anomalies by checking if the resource utilization exceeds pre-defined thresholds. Our results show that our approach can predict various performance anomalies with high precision (i.e., 97–100%) and recall (i.e., 80–100%), while the baselines achieve 25–97% precision and 93–100% recall. For a range of performance anomalies, our approach can achieve sufficient lead times that vary from 20 to 1,403 s (i.e., 23.4 min). We also demonstrate the ability of our approach to predict the performance anomalies that are caused by real-world performance bugs. For predicting performance anomalies that are caused by real-world performance bugs, our approach achieves 95–100% precision and 87–100% recall, while the baselines achieve 49–83% precision and 100% recall. The obtained results show that our approach outperforms the existing anomaly prediction approaches and is able to predict performance anomalies in real-world systems.

The satisfaction of users is only part of the success of a software product, since a strong competition can easily detract users from a software product/service. User churn is the jargon used to denote when a user changes from a product/service to the one offered by the competition. In this study, we empirically investigate the relationship between the issues that are present in a software product and user churn. For this purpose, we investigate a new dataset provided by the alternativeto.net platform. Alternativeto.net has a unique feature that allows users to recommend alternatives for a specific software product, which signals the intention to switch from one software product to another. Through our empirical study, we observe that (i) the intention to change software is tightly associated to the issues that are present in these software; (ii) we can predict the rate of potential churn using machine learning models; (iii) the longer the issue takes to be fixed, the higher the chances of user churn; and (iv) issues within more general software modules are more likely to be associated with user churn. Our study can provide more insights on the prioritization of issues that need to be fixed to proactively minimize the chances of user churn.

Developers always focus on delivering high-quality updates to improve, or maintain the rating of their apps. Prior work has studied user reviews by analyzing all reviews of an app. However, this app-level analysis misses the point that users post reviews to provide their feedback on a certain update. For example, two bad updates of an app with a history of good updates would not be spotted using app-level analysis. In this paper, we examine reviews at the update-level to better understand how users perceive bad updates. We focus our study on the top 250 bad updates (i.e., updates with the highest increase in the percentage of negative reviews relative to the prior updates of the app) from 26,726 updates of 2,526 top free-to-download apps in the Google Play Store. We find that feature removal and UI issues have the highest increase in the percentage of negative reviews. Bad updates with crashes and functional issues are the most likely to be fixed by a later update. However, developers often do not mention these fixes in the release notes. Our work demonstrates the necessity of an update-level analysis of reviews to capture the feelings of an app's user-base about a particular update.

The popularity of mobile apps continues to grow over the past few years. Mobile app stores, such as the Google Play Store and Apple's App Store provide a unique user feedback mechanism to app developers through app reviews. In the Google Play Store (and most recently in the Apple App Store), developers are able to respond to such user feedback.