Modern Language Models (LMs) have been successfully employed in source code generation and understanding, leading to a significant increase in research focused on learning-based code intelligence, such as automated bug repair and test case generation. Despite their great potential, LMs for code intelligence (LM4Code) are susceptible to potential pitfalls , which hinder realistic performance and further impact their reliability and applicability in real-world deployment . Such challenges drive the need for a comprehensive understanding—not just identifying these issues but delving into their possible implications and existing solutions to build more reliable LMs tailored to code intelligence. Based on a well-defined systematic research approach, we conducted an extensive literature review to uncover the pitfalls inherent in LM4Code. Finally, 121 primary studies from top-tier venues have been identified. After carefully examining these studies, we designed a taxonomy of pitfalls in LM4Code research and conducted a systematic study to summarize the issues, current solutions, implications, and challenges of different pitfalls for LM4Code systems. We developed a comprehensive classification scheme that dissects pitfalls across four crucial aspects: data collection and labeling, system design and learning, performance evaluation, and deployment and maintenance. Through this study, we aim to provide a roadmap for researchers and practitioners, facilitating their understanding and utilization of LM4Code in reliable and trustworthy ways.

Parcel Mismatch vulnerabilities in Android's Inter-Process Communication (IPC) mechanism have been a persistent security challenge for over a decade, leading to numerous privilege escalation exploits. While Google has implemented various mitigation strategies, culminating in the Lazy Bundle mechanism in Android 13, there has been no systematic analysis of these vulnerabilities and mitigations. To fill the gap, in this paper, we conduct the first comprehensive study of Parcel Mismatch vulnerabilities, proposing ParcelTaint, a new static analysis approach for detecting these issues. We develop precise models for tracking Intent and Bundle transformations across processes, enabling the discovery of new attack vectors. We reveal 10 previously unknown high-severity vulnerabilities, and 5 of them have been assigned with CVEs, including new ways to bypass existing mitigations and new attack chains in system services. All of them have been confirmed. We find that Parcel Mismatch remains a significant security concern, particularly for Android versions prior to 13 and for Original Equipment Manufacturers (OEMs) implementing custom system components. Based on our findings, Google has revised its security strategy to address core vulnerability patterns rather than relying solely on system-level mitigations. The study provides crucial insights for improving Android's IPC security and highlights the importance of systematic analysis in addressing long-standing security challenges.

In the digital era, Online Social Networks (OSNs) have become central to information dissemination, with sharing cards for link previews serving as a key feature.While these cards provide concise snapshots of shared content, their security implications have remained largely overlooked.This paper introduces the Mirage Card Attack, a novel class of attacks that exploits vulnerabilities in sharing card mechanisms across major OSNs.We identify two primary attack vectors: Proxy-Based Redirection and User-Agent-Based Cloaking.These attacks leverage design flaws in Share-SDK implementations and HTML meta tag usage, allowing attackers to bypass existing security measures and present deceptive content to users.Our systematic analysis reveals critical vulnerabilities in current sharing card systems.We demonstrate the feasibility of these attacks through comprehensive evaluations across 8 major OSNs for User-Agent-Based Cloaking and 6 OSNs for Proxy-Based Redirection.Additionally, we analyze 8 widely used card generation tools, uncovering significant security gaps.Our experiments show that some forged cards persist for over 15 days, highlighting the inadequacy of existing detection methods.To evaluate the practical impact of Mirage Card Attacks, we conduct a user study to * Both authors contributed equally to this research.

With the rapid advancement of artificial intelligence, pre-trained models (PTMs) have become fundamental building blocks in modern software systems.Model hubs, serving as centralized repositories for these components, have emerged as critical infrastructure in the AI software ecosystem.While prior research has extensively studied typosquatting attacks in traditional package management systems like NPM and PyPI, the security implications of such naming-based vulnerabilities in AI model hubs remain largely unexplored.To fill this gap, we present the first large-scale empirical study on typosquatting threats within the Hugging Face ecosystem, one of the most widely-used open-source model communities.Our research examines three key components: models, datasets, and organizations.Through a systematic analysis of 1,020,755 models (compared against the top 100 most downloaded ones), 219,812 datasets (compared against the top 100 most trending ones), and 127,011 organizations, we discovered concerning patterns of typosquatting that could compromise software supply chain security.Specifically, we identified 1,574 potentially malicious squatting models, with 10.4% exhibiting suspicious and harmful characteristics.Our investigation of datasets revealed 625 cases of typosquatting, where 42.2% showed signs of intentional impersonation based on sampling.Additionally, among the organizations studied, 302 demonstrated squatting patterns that could lead to supply chain attacks, with 4.8% showing explicit malicious intent.These findings highlight the pressing need for better naming conventions and security governance mechanisms in AI model repositories to ensure reliable and secure software development practices.We have reported all identified suspicious resources to Hugging Face for further investigation and potential mitigation measures.

As an important extension of the ChatGPT ecosystem, GPT Store has developed into an active market hosting more than 3 million customized ChatGPTs (GPTs).Despite its large scale, the current academic community still has obvious limitations in its understanding of the ecosystem of this platform.Based on a complete dataset of more than 700,000 GPTs, this paper has achieved a multidimensional analysis of GPT Store.We first systematically examined the platform operation mechanism, covering core elements such as the classification system, interaction mode, and evaluation system.We also comprehensively analyzed the security risks, such as data leakage and jailbreak in GPT Store.Finally, through a user study, this work revealed the behavioral characteristics and experience pain points in real usage scenarios.Based on these findings, we provide operational platform optimization suggestions, including functional improvement, security enhancement, and interaction improvement.This study not only constructs an analytical framework for the GPT Store ecosystem but also provides empirical evidence and optimization directions for its future development.

Android app security is a critical concern for the software industry, with companies investing significantly in protecting their intellectual property from reverse engineering attacks. While commercial protection tools exist to prevent decompilation and unauthorized code access, they pose substantial challenges for businesses: high licensing costs ranging from thousands to tens of thousands of dollars annually, significant performance overhead that impacts user experience and app ratings, and increased app size that affects download rates. These limitations particularly burden small to medium-sized enterprises and independent developers, creating an urgent industry need for cost-effective protection solutions.To address these challenges, we propose a novel file format-based anti-decompilation strategy that systematically exploits structural vulnerabilities in APK files. Building upon this strategy, we have developed APKARMOR, a lightweight and cost-effective anti-decompilation framework that exploits inherent vulnerabilities in popular reverse engineering tools. Through systematic analysis, we first identified critical weaknesses in common decompilation tools’ parsing mechanisms and structural assumptions. Based on these findings, we developed seven mutation-based protection strategies that deliberately trigger these vulnerabilities by introducing specific structural anomalies into APK files and the AndroidManifest.xml. These methods include Countermeasures against Dirty Code and Corrupted Payloads (CACoP), Pseudo-Encryption (PE), Using Unknown Compression Method (UUCM), Unavailable Magic Value (UMA), Modify the Offset Field in stringChunk (MOFS), and Dirty Bytecode Replacement of "Android" (DRA). We evaluated our exploitation strategies through extensive experiments on 100 randomly selected Android apps, testing against the latest versions of three widely used decompilation tools: JADX (v1.5.1), APKTool (v2.11.0), and Androguard (v4.1.2). Our results demonstrate that PE and DRA achieved complete protection by successfully exploiting vulnerabilities present in all tested tools. MOFS, UUCM, and UNV effectively exploited weaknesses in APKTool and Androguard’s parsing mechanisms.

HarmonyOS is Huawei’s distributed operating system designed for diverse smart devices, featuring ArkTS as its primary app development language. To enhance performance and leverage existing libraries, HarmonyOS apps can integrate native C/C++ modules through its Native Development Kit (NDK) mechanism. This creates significant challenges for static analysis, as critical data flows spanning ArkTS and native C/C++ boundaries remain invisible to existing single-language analyzers. Therefore, we present HarmoBridge, the first cross-language static analysis system for HarmonyOS that bridges this gap through novel summary based SumIR abstraction and seamless ecosystem integration. Our approach extracts dataflow summaries from native code (supporting both binary and source code analysis) and translates these summaries into intermediate representations that integrate seamlessly with the existing HarmonyOS analysis infrastructure. HarmoBridge introduces SumIR, a specialized intermediate representation that captures Node-API interaction semantics and converts them to ArkIR-compatible function bodies for downstream analysis tools. Also, we develop CrossFlowBench, a comprehensive benchmark covering representative Node-API interaction patterns, and identify potential security implications where established cross-language attack patterns widely prevalent in mobile ecosystems could adapt to HarmonyOS’s architecture. Results demonstrate that HarmoBridge achieves 81.0% accuracy in recovering cross-language data flows on CrossFlowBench, significantly outperforming baseline approaches that treat native calls as opaque operations, establishing a foundation for comprehensive cross-language analysis in the emerging HarmonyOS ecosystem.

As Large Language Models (LLMs) become integral software components in modern applications, unauthorized model derivations through fine-tuning, merging, and redistribution have emerged as critical software engineering challenges. Unlike traditional software where clone detection and license compliance are well-established, the LLM ecosystem lacks effective mechanisms to detect model lineage and enforce licensing agreements. This gap is particularly problematic when open-source model creators, such as Meta’s LLaMA, require derivative works to maintain naming conventions for attribution, yet no technical means exist to verify compliance.To fill this gap, treating LLMs as software artifacts requiring provenance tracking, we present TensorGuard, a gradient-based fingerprinting framework for LLM similarity detection and family classification. Our approach extracts model-intrinsic behavioral signatures by analyzing gradient responses to random input perturbations across tensor layers, operating independently of training data, watermarks, or specific model formats. TensorGuard supports the widely-adopted safetensors format and constructs high-dimensional fingerprints through statistical analysis of gradient features. These fingerprints enable two complementary capabilities: direct pairwise similarity assessment between arbitrary models through distance computation, and systematic family classification of unknown models via the K-Means clustering algorithm with domain-informed centroid initialization using known base models. Experimental evaluation on 58 models comprising 8 base models and 50 derivatives across five model families (Llama, Qwen, Gemma, Phi, Mistral) demonstrates 94% classification accuracy under our centroid-initialized K-Means clustering. Our work establishes a new paradigm for model similarity detection, bridging traditional software engineering practices with modern LLM distribution and compliance challenges.

Large Language Models (LLMs) have rapidly gained popularity, transforming research and industry. To support their adoption, LLM agent workflow orchestration frameworks (hereinafter referred to as LLM agent frameworks) like LangChain have become essential for building advanced applications. However, their complexity makes bugs inevitable, and these bugs can propagate to downstream applications, causing severe failures or unintended behaviors. In this paper, we first present an abstraction of the structure of mainstream LLM agent frameworks, identifying four key architectural components: data preprocessing, core schema, agent construction, and featured modules. Building on this abstraction, we conduct the first empirical study on LLM agent framework bugs, analyzing 1,026 bug instances extracted from 1,577 real-world bug-related GitHub pull requests (PRs) from three popular LLM agent frameworks: LangChain, LlamaIndex, and Haystack. For each bug, we examine its root cause, symptom, and structural component, providing a systematic taxonomy of nine root causes and six symptom categories. Finally, leveraging the framework structure abstraction and the large-scale empirical study, we perform detailed statistical analysis in terms of the distribution of bugs in different frameworks, the distribution across different framework components, and the relationship between root cause and symptom. The analysis reveals unique challenge patterns compared to traditional software, providing actionable guidance for practitioners on quality assurance.

Mini-programs, an emerging mobile application paradigm within super-apps, offer a seamless and installation-free experience. However, the adoption of the web-view component has disrupted their isolation mechanisms, exposing new attack surfaces and vulnerabilities. In this paper, we introduce a novel vulnerability called Cross Mini-program Cookie Sharing (CMCS), which arises from the shared web-view environment across mini-programs. This vulnerability allows unauthorized data exchange across mini-programs by enabling one mini-program to access cookies set by another within the same web-view context, violating isolation principles. As a preliminary step, we analyzed the web-view mechanisms of four major platforms, including WeChat, AliPay, TikTok, and Baidu, and found that all of them are affected by CMCS vulnerabilities. These findings were responsibly disclosed and acknowledged with two CVEs. Furthermore, we demonstrate the collusion attack enabled by CMCS, where privileged mini-programs exfiltrate sensitive user data via cookies accessible to unprivileged mini-programs. To measure the impact of collusion attacks enabled by CMCS vulnerabilities in the wild, we developed MiCoScan, a static analysis tool that detects mini-programs affected by CMCS vulnerabilities. MiCoScan employs web-view context modeling to identify clusters of mini-programs sharing the same web-view domain and cross-webview data flow analysis to detect sensitive data transmissions to/from web-views. Using MiCoScan, we conducted a large-scale analysis of 351,483 mini-programs, identifying 45,448 clusters sharing web-view domains, 7,965 instances of privileged data transmission, and 9,877 mini-programs vulnerable to collusion attacks. Our findings highlight the widespread prevalence and significant security risks posed by CMCS vulnerabilities, underscoring the urgent need for improved isolation mechanisms in mini-program ecosystems.

LLM app stores have seen rapid growth, leading to the proliferation of numerous custom LLM apps. However, this expansion raises security concerns. In this study, we propose a three-layer concern framework to identify the potential security risks of LLM apps, i.e., LLM apps with abusive potential, LLM apps with malicious intent, and LLM apps with backdoors. Over five months, we collected 786,036 LLM apps from six major app stores: GPT Store, FlowGPT, Poe, Coze, Cici, and Character.AI. Our research integrates static and dynamic analysis, and uses a complementary approach to detect harmful content, combining a self-refining LLM-based toxic content detector with rule-based pattern matching. Additionally, we constructed a large-scale toxic word dictionary (i.e., ToxicDict) comprising over 31,783 entries. We used these methods to uncover that 15,414 apps had misleading descriptions, 1,366 collected sensitive personal information against their privacy policies, and 15,996 generated harmful content such as hate speech, self-harm, extremism, etc. Additionally, we evaluated the potential for LLM apps to facilitate malicious activities, finding that 616 apps could be used for malware generation, phishing, etc. We reported these security risks to relevant platforms, including OpenAI and Quora, which acknowledged and appreciated our findings. The platforms are actively investigating the flagged apps; as of the submission of this paper, 1,643 apps have been removed from the GPT Store.

The customization and fragmentation of the Android ecosystem have fostered its prosperity and highlighted the growing importance of conducting security audits on these customized systems. This significance is driven by the distinct strategies that Original Equipment Manufacturers (OEMs) deploy to enhance device performance and user experience, which are important to their competitive differentiation. A key aspect of these strategies includes system-level optimizations for super apps and other widely used apps, marking a competitive trend among OEMs. Granting privileges to such apps often stems from trust in these apps. However, without proper validation of apps' identities, this can lead to severe implicit trust vulnerabilities, providing a convenient pathway for malicious apps to impersonate privileged ones and gain their access rights. For malicious developers, exploiting these vulnerabilities is both cost-effective and potentially highly rewarding. In this study, we undertook a comprehensive analysis of 686 custom Android ROMs from 46 OEMs, aimed at uncovering potential security risks associated with implicit trust vulnerabilities in apps. Our investigation identified 3,085 instances where thirdparty app package names were embedded within the ROMs. Alarmingly, only seven of these instances had implemented adequate authentication mechanisms to mitigate the associated risks, exposing 3,078 potential vulnerabilities that exhibited an increasing trend over time. We have reported 22 manually confirmed cases to seven relevant OEMs. As of the time of writing this paper, four vulnerabilities have been explicitly acknowledged by the OEMs, and one has been assigned a CVE ID.

When mobile meets LLMs, mobile app users deserve to have more intelligent usage experiences. For this to happen, we argue that there is a strong need to apply LLMs for the mobile ecosystem. We therefore provide a research roadmap for guiding our fellow researchers to achieve that as a whole. In this roadmap, we sum up six directions that we believe are urgently required for research to enable native intelligence in mobile devices. In each direction, we further summarize the current research progress and the gaps that still need to be filled by our fellow researchers.

The rapid advancement of large language models (LLMs) has revolutionized artificial intelligence, introducing unprecedented capabilities in natural language processing and multimodal content generation. However, the increasing complexity and scale of these models have given rise to a multifaceted supply chain that presents unique challenges across infrastructure, foundation models, and downstream applications. This article provides the first comprehensive research agenda of the LLM supply chain, offering a structured approach to identify critical challenges and opportunities through the dual lenses of software engineering (SE) and security and privacy (S&P). We begin by establishing a clear definition of the LLM supply chain, encompassing its components and dependencies. We then analyze each layer of the supply chain, presenting a vision for robust and secure LLM development, reviewing the current state of practices and technologies, and identifying key challenges and research opportunities. This work aims to bridge the existing research gap in systematically understanding the multifaceted issues within the LLM supply chain, offering valuable insights to guide future efforts in this rapidly evolving domain.

The rapid growth and popularity of large language model (LLM) app stores have created new opportunities and challenges for researchers, developers, users, and app store managers. As the LLM app ecosystem continues to evolve, it is crucial to understand the current landscape and identify potential areas for future research and development. This article presents a forward-looking analysis of LLM app stores, focusing on key aspects such as data mining, security risk identification, development assistance, and market dynamics. Our comprehensive examination extends to the intricate relationships between various stakeholders and the technological advancements driving the ecosystem’s growth. We explore the ethical considerations and potential societal impacts of widespread LLM app adoption, highlighting the need for responsible innovation and governance frameworks. By examining these aspects, we aim to provide a vision for future research directions and highlight the importance of collaboration among stakeholders to address the challenges and opportunities within the LLM app ecosystem. The insights and recommendations provided in this article serve as a foundation for driving innovation, ensuring responsible development, and creating a thriving, user-centric LLM app landscape.

With the growth of the Web3 domain, there has been a significant rise of third-party software packages. These range from basic blockchain interaction tools and contract development aids to front-end development packages for Decentralized Applications (DApps), with the diversity and inter-dependencies of these packages continuing to grow. However, despite these advancements, the security of these packages has not kept pace, highlighted by several significant Web3 supply chain attacks. It underscored the urgent need for a deeper understanding of the Web3 third-party library supply chain. In this paper, we present the first comprehensive analysis of the Web3 third-party package supply chain using knowledge graphs, which detail the structure, themes, and evolving dynamics within this domain. Further, we analyze the characteristics of vulnerabilities and their propagation through knowledge graphs in the Web3 ecosystem. We also extend our findings to practical applications, providing valuable insights for Web3 community stakeholders and creating an extensive dataset that includes 695,642 versions of Web3 packages and 270,217,326 dependency links. This research is aimed at reinforcing the foundation of Web3 development, promoting enhanced security practices, and supporting informed decision-making within the DApp landscape.

The Android ecosystem is significantly challenged by fragmentation, arising from diverse system versions, device specifications, and manufacturer customizations. The growing divergence among devices leads to marked variations in how a given app behaves across diverse devices. This is referred to as device-specific behaviors. Fragmentation not only complicates development processes but also impacts the overall industry by increasing maintenance costs and potentially harming user experience due to inconsistent app performance. In this work, we present the first large-scale empirical study of device-specific behaviors in real-world Android apps. We have designed a three-phase static analysis framework to accurately detect and understand the device-specific behaviors. Upon employing our tool on a dataset comprising more than 20,000 apps, we detected device-specific behaviors in 2,357 of them. By examining the distribution of device-specific behaviors, our analysis revealed that apps within the Chinese third-party app market exhibit more such behaviors compared to their counterparts in Google Play. Additionally, these behaviors are more likely to feature dominant brands that hold larger market shares. Reflecting this, we have classified these device-specific behaviors into 29 categories based on the functionalities implemented, providing a structured insight that is crucial for developers and stakeholders in the industry. Beyond the common behaviors, such as issue fixes and feature adaptations, we have observed 33 aggressive apps, including popular ones with millions of downloads. These apps abuse system properties of customized ROMs to obtain user-unresettable identifiers without requiring any permissions, posing significant privacy risks. Finally, we investigated the origins of device-specific behaviors, highlighting the significant challenges developers encounter in implementing them comprehensively. Our research aims to inform and equip industry practitioners with knowledge to enhance user experience and user privacy, marking a critical step toward addressing the less touched yet vital aspect of device-specific behaviors in the Android ecosystem.

The rapid advancements in Large Language Models (LLMs) have revolutionized natural language processing, with GPTs, customized versions of ChatGPT available on the GPT Store, emerging as a prominent technology for specific domains and tasks. To support academic research on GPTs, we introduce GPTZoo, a large-scale dataset comprising 730,420 GPT instances. Each instance includes rich metadata with 21 attributes describing its characteristics, as well as instructions, knowledge files, and third-party services utilized during its development. GPTZoo aims to provide researchers with a comprehensive and readily available resource to study the real-world applications, performance, and potential of GPTs. To facilitate efficient retrieval and analysis of GPTs, we also developed an automated command-line interface (CLI) that supports keyword-based searching of the dataset. To promote open research and innovation, the GPTZoo dataset will undergo continuous updates, and we are granting researchers public access to GPTZoo and its associated tools.

WebAssembly (abbreviated Wasm) has emerged as a cornerstone of web development, offering a compact binary format that allows high-performance applications to run at near-native speeds in web browsers. Despite its advantages, Wasm's binary nature presents significant challenges for developers and researchers, particularly regarding readability when debugging or analyzing web applications. Therefore, effective decompilation becomes crucial. Unfortunately, traditional decompilers often struggle with producing readable outputs. While some large language model (LLM)-based decompilers have shown good compatibility with general binary files, they still face specific challenges when dealing with Wasm.

The proliferation of pre-trained models (PTMs) and datasets has led to the emergence of centralized model hubs like Hugging Face, which facilitate collaborative development and reuse. However, recent security reports have uncovered vulnerabilities and instances of malicious attacks within these platforms, highlighting growing security concerns. This paper presents the first systematic study of malicious code poisoning attacks on pre-trained model hubs, focusing on the Hugging Face platform. We conduct a comprehensive threat analysis, develop a taxonomy of model formats, and perform root cause analysis of vulnerable formats. While existing tools like Fickling and ModelScan offer some protection, they face limitations in semantic-level analysis and comprehensive threat detection. To address these challenges, we propose MalHug, an end-to-end pipeline tailored for Hugging Face that combines dataset loading script extraction, model deserialization, in-depth taint analysis, and heuristic pattern matching to detect and classify malicious code poisoning attacks in datasets and models. In collaboration with Ant Group, a leading financial technology company, we have implemented and deployed MalHug on a mirrored Hugging Face instance within their infrastructure, where it has been operational for over three months. During this period, MalHug has monitored more than 705K models and 176K datasets, uncovering 91 malicious models and 9 malicious dataset loading scripts. These findings reveal a range of security threats, including reverse shell, browser credential theft, and system reconnaissance. This work not only bridges a critical gap in understanding the security of the PTM supply chain but also provides a practical, industry-tested solution for enhancing the security of pre-trained model hubs.

The exponential growth of open-source package ecosystems, particularly NPM and PyPI, has led to an alarming increase in software supply chain poisoning attacks. Existing static analysis methods struggle with high false positive rates and are easily thwarted by obfuscation and dynamic code execution techniques. While dynamic analysis approaches offer improvements, they often suffer from capturing non-package behaviors and employing simplistic testing strategies that fail to trigger sophisticated malicious behaviors. To address these challenges, we present OSCAR, a robust dynamic code poisoning detection pipeline for NPM and PyPI ecosystems. OSCAR fully executes packages in a sandbox environment, employs fuzz testing on exported functions and classes, and implements aspect-based behavior monitoring with tailored API hook points. We evaluate OSCAR against six existing tools using a comprehensive benchmark dataset of real-world malicious and benign packages. OSCAR achieves an F1 score of 0.95 in NPM and 0.91 in PyPI, confirming that OSCAR is as effective as the current state-of-the-art technologies. Furthermore, for benign packages exhibiting characteristics typical of malicious packages, OSCAR reduces the false positive rate by an average of 32.06% in NPM (from 34.63% to 2.57%) and 39.87% in PyPI (from 41.10% to 1.23%), compared to other tools, significantly reducing the workload of manual reviews in real-world deployments. In cooperation with Ant Group, a leading financial technology company, we have deployed OSCAR on its NPM and PyPI mirrors since January 2023, identifying 10,404 malicious NPM packages and 1,235 malicious PyPI packages over 18 months. This work not only bridges the gap between academic research and industrial application in code poisoning detection but also provides a robust and practical solution that has been thoroughly tested in a real-world industrial setting.

The advent of Large Language Models (LLMs) like ChatGPT has markedly transformed software development, aiding tasks from code generation to issue resolution with their human-like text generation. Nevertheless, the effectiveness of these models greatly depends on the nature of the prompts given by developers. Therefore, this study delves into the DevGPT dataset, a rich collection of developer-ChatGPT dialogues, to unearth the patterns in prompts that lead to effective problem resolutions. The underlying motivation for this research is to enhance the collaboration between human developers and AI tools, thereby improving productivity and problem-solving efficacy in software development. Utilizing a combination of textual analysis and data-driven approaches, this paper seeks to identify the attributes of prompts that are associated with successful interactions, providing crucial insights for the strategic employment of ChatGPT in software engineering environments.

Abstract Code comment generation, which attempts to summarize the functionality of source code in textual descriptions, plays an important role in automatic software development research. Currently, several structural neural networks have been exploited to preserve the syntax structure of source code based on abstract syntax trees (ASTs). However, they can not well capture both the long‐distance and local relations between nodes while retaining the overall structural information of AST. To mitigate this problem, we present a prototype tool titled PyScribe, which extends the Transformer model to a new encoder‐decoder‐based framework. Particularly, the triplet position is designed and integrated into the node‐level and edge‐level structural features of AST for producing Python code comments automatically. This paper, to the best of our knowledge, makes the first effort to model the edges of AST as an explicit component for improved code representation. By specifying triplet positions for each node and edge, the overall structural information can be well preserved in the learning process. Moreover, the captured node and edge features go through a two‐stage decoding process to yield higher qualified comments. To evaluate the effectiveness of PyScribe, we resort to a large dataset of code‐comment pairs by mining Jupyter Notebooks from GitHub, for which we have made it publicly available to support further studies. The experimental results reveal that PyScribe is indeed effective, outperforming the state‐ofthe‐art by achieving an average BLEU score (i.e., av‐BLEU) of 0.28.

Large Language Models (LLMs) have significantly impacted numerous domains, including Software Engineering (SE). Many recent publications have explored LLMs applied to various SE tasks. Nevertheless, a comprehensive understanding of the application, effects, and possible limitations of LLMs on SE is still in its early stages. To bridge this gap, we conducted a Systematic Literature Review (SLR) on LLM4SE, with a particular focus on understanding how LLMs can be exploited to optimize processes and outcomes. We selected and analyzed 395 research articles from January 2017 to January 2024 to answer four key Research Questions (RQs). In RQ1, we categorize different LLMs that have been employed in SE tasks, characterizing their distinctive features and uses. In RQ2, we analyze the methods used in data collection, pre-processing, and application, highlighting the role of well-curated datasets for successful LLM for SE implementation. RQ3 investigates the strategies employed to optimize and evaluate the performance of LLMs in SE. Finally, RQ4 examines the specific SE tasks where LLMs have shown success to date, illustrating their practical contributions to the field. From the answers to these RQs, we discuss the current state-of-the-art and trends, identifying gaps in existing research, and highlighting promising areas for future study. Our artifacts are publicly available at https://github.com/security-pride/LLM4SE_SLR .

Fragmentation is a serious problem in the Android ecosystem, which is mainly caused by the fast evolution of the system itself and the various system customizations. Many efforts have attempted to mitigate its impact via approaches to automatically pinpointing compatibility issues in Android apps. We conducted a literature review to identify all the currently available approaches to addressing this issue. Within the nine identified approaches, the four issue detection tools and one incompatible API harvesting tool could be successfully executed. We tried to reproduce them based on their original datasets and then empirically compared those approaches against common datasets. Our experimental results show that existing tool capabilities are quite distinct with only a small overlap in the compatibility issues being identified. Moreover, these detection tools commonly detect compatibility issues via two separate steps including incompatible APIs gathering and compatibility issues (induced by the incorrect invocations of the identified incompatible APIs) determination. To help developers better identify compatibility issues in Android apps, we developed a new approach, AndroMevol , to systematically spot incompatible APIs as they play a crucial role in issue detection. AndroMevol was able to pinpoint 397,678 incompatible APIs against the full history of the official Android framework and 52 customized Android frameworks spanning five popular device manufacturers. Our approach could enhance the ability of the state-of-the-art detection tools by identifying many more incompatible APIs that may cause compatibility issues in Android apps and foster more advanced approaches to pinpointing all types of compatibility issues.

HarmonyOS is an operating system boasting a substantial global user base and provides multiple versions of its SDK. Various open-source applications continue to utilize older versions, leading to compatibility issues arising from system constraints. These prevalent issues can substantially affect the user experience. Although numerous solutions have been suggested for addressing compatibility issues in Android, the subject remains largely unexplored within the context of HarmonyOS. To bridge this gap, we investigate the evolution of APIs in HarmonyOS to pinpoint those potentially causing compatibility issues. Based on these insights, we implement CiD4HMOS, a tool designed to detect and categorize compatibility issues in HarmonyOS. We evaluate the feasibility of CiD4HMOS with open-source apps and subsequently apply it to commercially released apps, highlighting its effectiveness in accurately identifying HarmonyOS compatibility issues. The experimental results uncover that CiD4HMOS is effective in detecting compatibility issues in HarmonyOS apps, achieving an accuracy rate of 86.8 % in open-source apps. And, developers of commercially released apps have significantly endorsed our reports. Our research emphasizes the necessity of continuous exploration into compatibility issues within HarmonyOS, underlining the significant role tools like CiD4HMOS play in enhancing the overall user experience.

As smartphones and mobile apps permeate every aspect of people’s lives, children are accessing mobile devices at an increasingly younger age. The inescapable exposure of advertisements in mobile apps to children has grown alarmingly. Mobile advertisements are placed by advertisers and subsequently distributed by ad SDKs, under the rare control of app developers and app markets’ content ratings. Indeed, content that is objectionable and harmful to children’s mental health has been reported to appear in advertising, such as pornography. However, few studies have yet concentrated on automatically and comprehensively identifying such kid-unsuitable mobile advertising. In this paper, we first characterize the regulations for mobile ads relating to children. We then propose our novel automated dynamic analysis framework, named AdRambler, that attempts to collect ad content throughout the lifespan of mobile ads and identify their inappropriateness for child app users. Using AdRambler, we conduct a large-scale (25,000 mobile apps) empirical investigation and reveal the non-incidental presence of inappropriate ads in apps with child-included target audiences. We collected 11,270 ad views and identified 1,289 ad violations (from 775 apps) of child user regulations, with roughly half of the app promotions not in compliance with host apps’ content ratings. Our finding indicates that even certified ad SDKs could still propagate inappropriate advertisements. We further delve into the question of accountability for the presence of inappropriate advertising and provide concrete suggestions for all stakeholders to take action for the benefit of children.

Android developers are often faced with the need to learn how to use different APIs suitable for their projects. Automated API recommendation approaches have been invented to help fill this gap, and these have been demonstrated to be useful to some extent. Unfortunately, most state-of-the-art works are not proposed for Android developers, and the ones dedicated to Android app development often suffer from high redundancy and poor run-time performance, or do not target the problem of recommending API usage patterns. To address this gap we propose to the community a new tool, namelyAPIMatchmaker, to recommend API usages by learning directly from similar real-world Android apps. Unlike existing recommendation approaches, which leverage a single context to find similar projects, we innovatively introduce a multi-dimensional, context-aware, collaborative filtering approach to better achieve the purpose. Specifically, in addition to code similarity, we also take app descriptions (or topics) into consideration to ensure that similar apps also provide similar functions. We evaluateAPIMatchmakeron a large number of real-world Android apps and observe thatAPIMatchmakeryields a high success rate in recommending APIs for Android apps under development, and it is also able to outperform the state-of-the-art.

The heavy fragmentation of the Android ecosystem has led to severe compatibility issues with apps, including those that crash at runtime or cannot be installed on certain devices but work well on other devices. To address this problem, various approaches have been proposed to detect and fix compatibility issues automatically. However, these all come with various limitations on fixing the compatibility issues, e.g., can only fix one specific type of issues, cannot deal with multi-invocation issues in a single line and issues in released apps. To overcome these limitations, we propose a generic approach that aims at fixing more types of compatibility issues in released Android apps. To this end, our prototype tool, RepairDroid, provides a generic app patch description language for users to create fix templates for compatibility issues. The created templates will then be leveraged by RepairDroid to automatically fix the corresponding issue at the bytecode level (e.g., right before users install the app). RepairDroid can support template creations for OS-induced, device-specific and inter-callback compatibility issues detected by three state-of-the-art approaches. Our experimental results show that RepairDroid can fix 7,660 out of 8,976 compatibility issues in 1,000 randomly selected Google Play apps. RepairDroid is generic to configure new compatibility issues and outperforms the state-of-the-art on effectively repairing compatibility issues in released Android apps.

Fragmentation is a serious problem in the Android ecosystem. This problem is mainly caused by the fast evolution of the system itself and the various customizations independently maintained by different smartphone manufacturers. Many efforts have attempted to mitigate its impact via approaches to automatically pinpoint compatibility issues in Android apps. Unfortunately, at this stage, it is still unknown if this objective has been fulfilled, and the existing approaches can indeed be replicated and reliably leveraged to pinpoint compatibility issues in the wild. We, therefore, propose to fill this gap by first conducting a literature review within this topic to identify all the available approaches. Among the nine identified approaches, we then try our best to reproduce them based on their original datasets. After that, we go one step further to empirically compare those approaches against common datasets with real-world apps containing compatibility issues. Experimental results show that existing tools can indeed be reproduced, but their capabilities are quite distinct, as confirmed by the fact that there is only a small overlap of the results reported by the selected tools. This evidence suggests that more efforts should be spent by our community to achieve sound compatibility issues detection.

Continuous Integration (CI) and Continuous Delivery (CD) have been demonstrated to be effective in facilitating software building, testing, and deployment. Many research studies have investigated and subsequently improved their working processes. Unfortunately, such research efforts have largely not touched on the usage of CI/CD in the development of Android apps. We fill this gap by conducting an exploratory study of CI/CD adoption in open-source Android apps. We start by collecting a set of 84,475 open-source Android apps from the most popular three online code hosting sites, namely Github, GitLab, and Bitbucket. We then look into those apps and find that (1) only around 10% of apps have leveraged CI/CD services, i.e., the majority of open-source Android apps are developed without accessing CI/CD services, (2) a small number of apps (291) has even adopted multiple CI/CD services, (3) nearly half of the apps adopted CI/CD services have not really used them, and (4) CI/CD services are useful to improve the popularity of projects.

Despite being one of the largest and most popular projects, the official Android framework has only provided test cases for less than 30% of its APIs. Such a poor test case coverage rate has led to many compatibility issues that can cause apps to crash at runtime on specific Android devices, resulting in poor user experiences for both apps and the Android ecosystem. To mitigate this impact, various approaches have been proposed to automatically detect such compatibility issues. Unfortunately, these approaches have only focused on detecting signature-induced compatibility issues (i.e., a certain API does not exist in certain Android versions), leaving other equally important types of compatibility issues unresolved. In this work, we propose a novel prototype tool, JUnitTestGen, to fill this gap by mining existing Android API usage to generate unit test cases. After locating Android API usage in given real-world Android apps, JUnitTestGen performs inter-procedural backward data-flow analysis to generate a minimal executable code snippet (i.e., test case). Experimental results on thousands of real-world Android apps show that JUnitTestGen is effective in generating valid unit test cases for Android APIs. We show that these generated test cases are indeed helpful for pinpointing compatibility issues, including ones involving semantic code changes.

Malware detection at scale in the Android realm is often carried out using machine learning techniques. State-of-the-art approaches such as DREBIN and MaMaDroid are reported to yield high detection rates when assessed against well-known datasets. Unfortunately, such datasets may include a large portion of duplicated samples, which may bias recorded experimental results and insights. In this article, we perform extensive experiments to measure the performance gap that occurs when datasets are de-duplicated. Our experimental results reveal that duplication in published datasets has a limited impact on supervised malware classification models. This observation contrasts with the finding of Allamanis on the general case of machine learning bias for big code. Our experiments, however, show that sample duplication more substantially affects unsupervised learning models (e.g., malware family clustering). Nevertheless, we argue that our fellow researchers and practitioners should always take sample duplication into consideration when performing machine-learning-based (via either supervised or unsupervised learning) Android malware detections, no matter how significant the impact might be.

It is critical for research to have an open, well-curated, representative set of apps for analysis. We present a collection of open-source Android apps collected from several sources, including Github. Our dataset, AndroZooOpen, currently contains over 45,000 app artefacts, a representative picture of Github-hosted Android apps. For apps released on Google Play, metadata including categories, ratings and user reviews, are also stored. We share this new dataset as part of our ongoing research to better support and enable new research topics involving Android app artefact analysis, and as a supplement dataset for AndroZoo, a well-known app collection of close-sourced Android apps.

Knowledge Graph, being able to connect information from a variety of sources, has become very famous in recent years since its creation in 2012 by Google. Researchers in our community have leveraged Knowledge Graph to achieve various purposes such as improving API caveats accessibilities, generating answers to developer questions, and reasoning common software weaknesses, etc. In this work, we would like to leverage the knowledge graph concept for helping developers and project managers to comprehend software repositories. To this end, we design and implement a prototype tool called GitGraph, which takes as input a Git repository and constructs automatically a knowledge graph associated with the repository. Our preliminary experimental results show that GitGraph can correctly generate knowledge graphs for Git projects and the generated graphs are also useful for users to comprehend the projects. More specifically, the knowledge graph, on one hand, provides a graphic interface that users can interactively explore the integrated artefacts such as commits and changed methods, while on the other hand, provides a convenient means for users to search for advanced relations between the different artefacts.