Dynamic Text-Attributed Graphs (DyTAGs) are a novel graph paradigm that captures evolving temporal events (edges) alongside rich textual attributes. Existing studies can be broadly categorized into TGNN-driven and LLM-driven approaches, both of which encode textual attributes and temporal structures for DyTAG representation. We observe that DyTAGs inherently comprise three distinct modalities: temporal, textual, and structural, often exhibiting completely disjoint distributions. However, the first two modalities are largely overlooked by existing studies, leading to suboptimal performance. To address this, we propose MoMent, a multi-modal network that explicitly models, integrates, and aligns each modality to learn node representations for link prediction. Given the disjoint nature of the original modality distributions, we first construct modality-specific features and encode them using individual encoders to capture correlations across temporal patterns, semantic context, and local structures. Each encoder generates modality-specific tokens, which are then fused into comprehensive node representations with a theoretical guarantee. To avoid disjoint subspaces of these heterogeneous modalities, we propose a dual-domain alignment loss that first aligns their distributions globally and then fine-tunes coherence at the instance level. This enhances coherent representations from temporal, textual, and structural views. Extensive experiments across seven datasets show that MoMent achieves up to 17.28% accuracy improvement and up to 31x speed-up against eight baselines.

Large Language Models (LLMs) have achieved impressive reasoning abilities, but struggle with temporal understanding, especially when questions involve multiple entities, compound operators, and evolving event sequences. Temporal Knowledge Graphs (TKGs), which capture vast amounts of temporal facts in a structured format, offer a reliable source for temporal reasoning. However, existing TKG-based LLM reasoning methods still struggle with four major challenges: maintaining temporal faithfulness in multi-hop reasoning, achieving multi-entity temporal synchronization, adapting retrieval to diverse temporal operators, and reusing prior reasoning experience for stability and efficiency. To address these issues, we propose MemoTime, a memory-augmented temporal knowledge graph framework that enhances LLM reasoning through structured grounding, recursive reasoning, and continual experience learning. MemoTime decomposes complex temporal questions into a hierarchical Tree of Time, enabling operator-aware reasoning that enforces monotonic timestamps and co-constrains multiple entities under unified temporal bounds. A dynamic evidence retrieval layer adaptively selects operator-specific retrieval strategies, while a self-evolving experience memory stores verified reasoning traces, toolkit decisions, and sub-question embeddings for cross-type reuse. Comprehensive experiments on multiple temporal QA benchmarks show that MemoTime achieves overall state-of-the-art results, outperforming the strong baseline by up to 24.0%. Furthermore, MemoTime enables smaller models (e.g., Qwen3-4B) to achieve reasoning performance comparable to that of GPT-4-Turbo.

The emergence of foundation models (FMs) has enabled the development of highly capable and autonomous agents, unlocking new application opportunities across a wide range of domains. Evaluating the architecture of agents is particularly important as the architectural decisions significantly impact the quality attributes of agents given their unique characteristics, including compound architecture, autonomous and non-deterministic behaviour, and continuous evolution. However, these traditional methods fall short in addressing the evaluation needs of agent architecture due to the unique characteristics of these agents. Therefore, in this paper, we present AgentArcEval, a novel agent architecture evaluation method designed specially to address the complexities of FM-based agent architecture and its evaluation. Moreover, we present a catalogue of agent-specific general scenarios, which serves as a guide for generating concrete scenarios to design and evaluate the agent architecture. We demonstrate the usefulness of AgentArcEval and the catalogue through a case study on the architecture evaluation of a real-world tax copilot, named Luna.

Implicit sentiment analysis (ISA) aims to infer sentiment that is implied rather than explicitly stated, requiring models to perform deeper reasoning over subtle contextual cues. While recent prompting-based methods using large language models (LLMs) have shown promise in ISA, they often rely on majority voting over chain-of-thought (CoT) reasoning paths without evaluating their causal validity, making them susceptible to internal biases and spurious correlations. To address this challenge, we propose CAPITAL, a causal prompting framework that incorporates front-door adjustment into CoT reasoning. CAPITAL decomposes the overall causal effect into two components: the influence of the input prompt on the reasoning chains, and the impact of those chains on the final output. These components are estimated using encoder-based clustering and the NWGM approximation, with a contrastive learning objective used to better align the encoder’s representation with the LLM’s reasoning space. Experiments on benchmark ISA datasets with three LLMs demonstrate that CAPITAL consistently outperforms strong prompting baselines in both accuracy and robustness, particularly under adversarial conditions. This work offers a principled approach to integrating causal inference into LLM prompting and highlights its benefits for bias-aware sentiment reasoning. The source code and case study are available at:https://github.com/whZ62/CAPITAL.

Multimodal Multi-hop question answering requires integrating information from diverse sources, such as images and texts, to derive answers.Existing methods typically rely on sequential retrieval and reasoning, where each step builds on the previous output.However, this single-path paradigm makes them vulnerable to errors due to misleading intermediate steps.Moreover, developing multimodal models can be computationally expensive, often requiring extensive training.To address these limitations, we propose a training-free framework guided by an Adaptive Planning Graph, which consists of planning, retrieval and reasoning modules.The planning module analyzes the current state of the Adaptive Planning Graph, determines the next action and where to expand the graph, which enables dynamic and flexible exploration of reasoning paths.To handle retrieval of text to unspecified target modalities, we devise modality-specific strategies that dynamically adapt to distinct data types.Our approach preserves the characteristics of multimodal information without costly task-specific training, enabling seamless integration with upto-date models.Finally, the experiments on MultimodalQA and WebQA show that our approach matches or outperforms existing models that rely on training.

Retrieval-augmented generation (RAG) enhances large language models (LLMs) by incorporating external knowledge.Current hybrid RAG system retrieves evidence from both knowledge graphs (KGs) and text documents to support LLM reasoning.However, it faces challenges like handling multi-hop reasoning, multi-entity questions, multi-source verification, and effective graph utilization.To address these limitations, we present HydraRAG, a training-free framework that unifies graph topology, document semantics, and source reliability to support deep, faithful reasoning in LLMs.HydraRAG handles multi-hop and multi-entity problems through agent-driven exploration that combines structured and unstructured retrieval, increasing both diversity and precision of evidence.To tackle multisource verification, HydraRAG uses a tri-factor cross-source verification (source trustworthiness assessment, cross-source corroboration, and entity-path alignment), to balance topic relevance with cross-modal agreement.By leveraging graph structure, HydraRAG fuses heterogeneous sources, guides efficient exploration, and prunes noise early.Comprehensive experiments on seven benchmark datasets show that HydraRAG achieves overall state-of-theart results on all benchmarks with GPT-3.5, outperforming the strong hybrid baseline ToG-2 by an average of 20.3% and up to 30.1%.Furthermore, HydraRAG enables smaller models (e.g., Llama-3.1-8B) to achieve reasoning performance comparable to that of GPT-4-Turbo.The source code is available on https: //stevetantan.github.io/HydraRAG/.

UI automation is an useful technique for UI testing, bug reproduction and robotic process automation. Recording the user actions with an application assists rapid development of UI automation scripts, but existing recording techniques are intrusive, rely on OS or GUI framework accessibility support or assume specific app implementations. Reverse engineering user actions from screencasts is non-intrusive, but a key reverse-engineering step is currently missing - recognize human-understandable structured user actions ([command] [widget] [location]) from action screencasts. To fill the gap, we propose a deep learning based computer vision model which can recognize 11 commands and 11 widgets, and generate location phrases from action screencasts, through joint learning and multi-task learning. We label a large dataset with 7260 video-action pairs, which record the user interactions with Word, Zoom, Firefox, Photoshop and Windows 10 Settings. Through extensive experiments, we confirm the effectiveness and generality of our model, and demonstrate the usefulness of a screencast-to-action-script tool built upon our model for bug reproduction.

Detecting driver fatigue is critical for road safety, as drowsy driving remains a leading cause of traffic accidents. Many existing solutions rely on computationally demanding deep learning models, which result in high latency and are unsuitable for embedded robotic devices with limited resources (such as intelligent vehicles/cars) where rapid detection is necessary to prevent accidents. This paper introduces LiteFat, a lightweight spatio-temporal graph learning model designed to detect driver fatigue efficiently while maintaining high accuracy and low computational demands. LiteFat involves converting streaming video data into spatio-temporal graphs (STG) using facial landmark detection, which focuses on key motion patterns and reduces unnecessary data processing. LiteFat uses MobileNet to extract facial features and create a feature matrix for the STG. A lightweight spatio-temporal graph neural network is then employed to identify signs of fatigue with minimal processing and low latency. Experimental results on benchmark datasets show that LiteFat performs competitively while significantly reduced computational complexity and latency as compared to current state-of-the-art methods. This work advances the development of real-time, resource-efficient human fatigue detection systems that can be implemented upon embedded robotic devices.

Temporal hypergraph representation learning is a concept that integrates high-order structure learning with temporal dynamics, enabling more accurate analysis of temporal and high-order interactions. To enhance model expressiveness, the latest work samples multi-hop hyperedge-centric neighbors directly from temporal hypergraphs and encodes them for high-order structure learning, achieving promising performance. Such modeling, however, incurs prohibitive computational complexity, which increases exponentially with model depth and quadratically with average hyperedge cardinality, thereby limiting model scalability. In this paper, we propose FastHeP, a fast and accurate approach for temporal hyperedge prediction, which can handle large temporal hypergraphs. The key idea is to minimize computational complexity while maintaining model expressiveness. Concretely, we design an online hyperedge-centric neighbor store, which can store time-aware and redundancy-aware neighbors for nodes with rational theoretical guarantees. Upon the neighbor store, we propose a novel hybrid message passing to model temporal high-order structures, theoretically preserving strong expressive power. This explicitly learns local high-order structures for nodes of each hyperedge via graph attention, generating the node-wise structure features. These structure features are then fused into global correlations modeling among hyperedges, with a theoretical guarantee of permutation invariance. Last, FastHeP leverages local and global high-order semantics to generate temporal hyperedge embeddings, which is efficient in a linear complexity w.r.t. model depth and average hyperedge cardinality. Extensive experiments show that FastHeP achieves up to two orders of magnitude speed-up against baselines, with an average accuracy improvement of 5.1%.

Large Language Models (LLMs) have achieved impressive results in various tasks but struggle with hallucination problems and lack of relevant knowledge, especially in deep complex reasoning and knowledge-intensive tasks.Knowledge Graphs (KGs), which capture vast amounts of facts in a structured format, offer a reliable source of knowledge for reasoning.However, existing KG-based LLM reasoning methods face challenges like handling multi-hop reasoning, multi-entity questions, and effectively utilizing graph structures.To address these issues, we propose Paths-over-Graph (PoG), a novel method that enhances LLM reasoning by integrating knowledge reasoning paths from KGs, improving the interpretability and faithfulness of LLM outputs.PoG tackles multi-hop and multi-entity questions through a three-phase dynamic multi-hop path exploration, which combines the inherent knowledge of LLMs with factual knowledge from KGs.In order to improve the efficiency, PoG prunes irrelevant information from the graph exploration first and introduces efficient three-step pruning techniques that incorporate graph structures, LLM prompting, and a pre-trained language model (e.g., SBERT) to effectively narrow down the explored candidate paths.This ensures all reasoning paths contain highly relevant information captured from KGs, making the reasoning faithful and interpretable in problem-solving.PoG innovatively utilizes graph structure to prune the irrelevant noise and represents the first method to implement multi-entity deep path detection on KGs for LLM reasoning tasks.Comprehensive experiments on five benchmark KGQA datasets demonstrate PoG outperforms the stateof-the-art method ToG across GPT-3.5-Turbo and GPT-4, achieving an average accuracy improvement of 18.9%.Notably, PoG with GPT-3.5-Turbosurpasses ToG with GPT-4 by up to 23.9%.

Utilising quantum computing technology to enhance artificial intelligence systems is expected to improve training and inference times, increase robustness against noise and adversarial attacks, and reduce the number of parameters without compromising accuracy. However, moving beyond proof-of-concept or simulations to develop practical applications of these systems while ensuring high software quality faces significant challenges due to the limitations of quantum hardware and the underdeveloped knowledge base in software engineering for such systems. In this work, we have conducted a systematic mapping study to identify the challenges and solutions associated with the software architecture of quantum-enhanced artificial intelligence systems. The results of the systematic mapping study reveal several architectural patterns that describe how quantum components can be integrated into inference engines, as well as middleware patterns that facilitate communication between classical and quantum components. Each pattern realises a trade-off between various software quality attributes, such as efficiency, scalability, trainability, simplicity, portability, and deployability. The outcomes of this work have been compiled into a catalogue of architectural patterns. Editor’s note: Open Science material was validated by the Journal of Systems and Software Open Science Board . • Quantum AI patterns focus on which tasks to delegate to quantum computers. • Quantum AI pattern catalogue provides valuable guidance for software architects. • Majority of the supporting evidence for quantum AI comes from simulations. • Key trends in quantum AI are scaling up quantum and automating architecture design. • Quantum AI can speed up training, inference, and enhance robustness.

Foundation model-enabled generative artificial intelligence facilitates the development and implementation of agents, which can leverage distinguished reasoning and language processing capabilities to takes a proactive, autonomous role to pursue users’ goals. Nevertheless, there is a lack of systematic knowledge to guide practitioners in designing the agents considering challenges of goal-seeking (including generating instrumental goals and plans), such as hallucinations inherent in foundation models, explainability of reasoning process, complex accountability, etc. To address this issue, we have performed a systematic literature review to understand the state-of-the-art foundation model-based agents and the broader ecosystem. In this paper, we present a pattern catalogue consisting of 18 architectural patterns with analyses of the context, forces, and trade-offs as the outcomes from the previous literature review. We propose a decision model for selecting the patterns. The proposed catalogue can provide holistic guidance for the effective use of patterns, and support the architecture design of foundation model-based agents by facilitating goal-seeking and plan generation. • A collection of architectural patterns for real-world agent implementations. • FM-based agent ecosystem with architectural pattern annotations as a guidance. • Curated analysis of patterns including benefits, trade-offs, and real-world uses. • A decision model for structuring the patterns and making rational design decisions.

Missing data imputation, which aims to impute the missing values in the raw datasets, is crucial for modern data-driven models like large language models (LLMs). Despite its importance, existing solutions either 1) only support numerical and categorical data or 2) show an unsatisfactory performance due to their design prioritizing text data and overlooking intrinsic characteristics of tabular data. In this paper, we propose UnIMP, a Un ified IMP utation framework that leverages LLM and high-order message passing to enhance the imputation of mixed-type data, including numerical, categorical, and text data. Specifically, we first introduce a cell-oriented hypergraph to model the table. We then propose BiHMP, an efficient Bidirectional High-order Message-Passing network to aggregate global-local and high-order information while capturing the inter-column heterogeneity and intra-column homogeneity. To align the capacity of the LLM with the information aggregated by BiHMP, we introduce Xfusion, which, together with BiHMP, acts as adapters for the LLM. We follow a pre-training and fine-tuning pipeline to train UnIMP, integrating two optimizations: chunking technique, which divides tables into smaller chunks to enhance efficiency; and progressive masking technique, which gradually adapts the model to learn more complex data patterns. Both theoretical proofs and empirical experiments on 10 real-world datasets highlight the superiority of UnIMP over existing techniques.

Real-life graphs often exhibit intricate dynamics that evolve continuously over time. To effectively represent continuous-time dynamic graphs (CTDGs), various temporal graph neural networks (TGNNs) have been developed to model their dynamics and topological structures in Euclidean space. Despite their notable achievements, the performance of Euclidean-based TGNNs is limited and bounded by the representation capabilities of Euclidean geometry, particularly for complex graphs with hierarchical and power-law structures. This is because Euclidean space does not have enough room (its volume grows polynomially with respect to radius) to learn hierarchical structures that expand exponentially. As a result, this leads to high-distortion embeddings and suboptimal temporal graph representations. To break the limitations and enhance the representation capabilities of TGNNs, in this article, we propose a scalable and effective TGNN with hyperbolic geometries for CTDG representation (called ${\mathrm { STGN}}^{h}$ ). It captures evolving behaviors and stores hierarchical structures simultaneously by integrating a memory-based module and a structure-based module into a unified framework, which can scale to billion-scale graphs. Concretely, a simple hyperbolic update gate (HuG) is designed as the memory-based module to store temporal dynamics efficiently; for the structure-based module, we propose an effective hyperbolic temporal Transformer (HyT) model to capture complex graph structures and generate up-to-date node embeddings. Extensive experimental results on a variety of medium-scale and billion-scale graphs demonstrate the superiority of the proposed ${\mathrm { STGN}}^{h}$ for CTDG representation, as it significantly outperforms baselines in various downstream tasks.

Smart contracts with excessive gas consumption can cause economic losses, such as black hole contracts. Actual gas consumption depends on runtime information and has a probability distribution under different runtime situations. However, existing static analysis tools (e.g., Solc) cannot define runtime information and only provide an approximate upper bound on gas consumption without explanation. To address the challenge, we propose a label named GCL, which describes the probability distribution of gas consumption, a code representation method containing domain features and a graph neural network (GNN) named attention-based graph isomorphism network (AGIN) oriented to domain feature, and SubgraphGas, a domain-oriented subgraph-level GNN explanation model. By combining AGIN and SubgraphGas, we have created a new explainable gas consumption prediction model (EGE). Our evaluations show that EGE outperforms prediction schemes based on Bi-LSTM. And EGE has similar explainability accuracy to general methods, but it is more efficient.

Smart contracts, which automatically execute transactions based on predefined conditions, play a crucial role in asset and money exchanges. Due to their involvement in significant financial transactions, these contracts are attractive targets for hackers, leading to substantial financial losses through exploitable vulnerabilities. While various program analysis methods such as Oyente, Mythril, and Securify have been proposed to address these security concerns, they rely on rule-based patterns that are time-consuming to develop and offer limited coverage. Deep learning methods present an alternative by automatically learning code features to detect vulnerabilities. However, existing approaches face critical challenges, including feature limitations and lack of interpretability. To address these gaps, we propose the interpretable smart contract vulnerability detector, a Graph Isomorphism Network (GIN)-based vulnerability prediction model for smart contracts, enhanced with code subgraph explanations. Our approach identifies and incorporates 43 domain-specific features, augmenting GIN with domain knowledge attention mechanisms to improve vulnerability prediction. In addition, we develop an interpreter called SubgraphV, which provides explanations for vulnerability predictions through interpreted subgraphs. Our model demonstrates superior performance over traditional tools, achieving F1 score improvements from 0.254 to 0.489 on a dataset of 103 smart contract function vulnerabilities. SubgraphV outperforms existing explainability methods like GNNexplainer, PGExplainer, and SubgraphX in pinpointing vulnerabilities, accurately reflecting vulnerability patterns, and enhancing the understanding of vulnerabilities.

Large language model (LLM) based chatbots, such as ChatGPT, have attracted huge interest in foundation models. It is widely believed that foundation models will serve as the fundamental building blocks for future AI systems. However, the architecture design of foundation model based systems has not yet been systematically explored. There is limited understanding about the impact of introducing foundation models in software architecture. Therefore, in this paper, we propose a taxonomy of foundation model based systems, which classifies and compares the characteristics of foundation models and system design options. Our taxonomy comprises three categories: the pretraining and adaptation of foundation models, the architecture design of foundation model based systems, and responsible-AI-by-design. This taxonomy can serve as concrete guidance for designing foundation model based systems.

The advent of Generative AI has marked a significant milestone in artificial intelligence, demonstrating remarkable capabilities in generating realistic images, texts, and data patterns. However, these advancements come with heightened concerns over data privacy and copyright infringement, primarily due to the reliance on vast datasets for model training. Traditional approaches like differential privacy, machine unlearning, and data poisoning only offer fragmented solutions to these complex issues. Our paper delves into the multifaceted challenges of privacy and copyright protection within the data lifecycle. We advocate for integrated approaches that combines technical innovation with ethical foresight, holistically addressing these concerns by investigating and devising solutions that are informed by the lifecycle perspective. This work aims to catalyze a broader discussion and inspire concerted efforts towards data privacy and copyright integrity in Generative AI.

The inclusion of the images opens up a security vulnerability of visually-aware recommender systems (VARSs). It can be exploited by unscrupulous parties to upload well-crafted adversarial images for certain malicious purposes (e.g., promoting their own products for profits). Some studies have focused on attacking VARSs to gain insights into their robustness, while they are still far from practical, i.e., the attacks often 1) lack diversity in perturbations, 2) are easily perceived and 3) have limited transferability, which may lead to overestimation of defenses in practice. To tackle the problems, we propose to perturb the style of the product, which is an unnoticeable but important property of visual recommendations. Specifically, we propose a novel Style perturbation-based Practical Attack Framework (SPAF). Unlike existing attacks that change pixels within l∞ -norm constraints, SPAF interferes with styles in latent feature space so that the attack becomes unbounded in the pixel space to reflect possible actual perturbations. SPAF formulates attack objectives as an optimization problem and adopts an adaptive adversarial style transfer network to solve it so that transferable and imperceptible attacks can be generated. Comprehensive experiments on real-world datasets demonstrate that SPAF significantly outperforms state-of-the-art attacks.

Microsoft's 1stgen custom Al Accelerator

Exploratory testing leverages the tester's knowledge and creativity to design test cases for effectively uncovering system-level bugs from the end user's perspective. Researchers have worked on test scenario generation to support exploratory testing based on a system knowledge graph, enriched with scenario and oracle knowledge from bug reports. Nevertheless, the adoption of this approach is hindered by difficulties in handling bug reports of inconsistent quality and varied expression styles, along with the infeasibility of the generated test scenarios. To overcome these limitations, we utilize the superior natural language understanding (NLU) capabilities of Large Language Models (LLMs) to construct a System KG of User Tasks and Failures (SysKG-UTF). Leveraging the system and bug knowledge from the KG, along with the logical reasoning capabilities of LLMs, we generate test scenarios with high feasibility and coherence. Particularly, we design chain-of-thought (CoT) reasoning to extract human-like knowledge and logical reasoning from LLMs, simulating a developer's process of validating test scenario feasibility. Our evaluation shows that our approach significantly enhances the KG construction, particularly for bug reports with low quality. Furthermore, our approach generates test scenarios with high feasibility and coherence. The user study further proves the effectiveness of our generated test scenarios in supporting exploratory testing. Specifically, 8 participants find 36 bugs from 8 seed bugs in two hours using our test scenarios, a significant improvement over the 21 bugs found by the state-of-the-art baseline.

With the widely usage of open-source software, supply-chain-based vulnerability attacks, including SolarWind and Log4Shell, have posed significant risks to software security. Currently, people rely on vulnerability advisory databases or commercial software bill of materials (SBOM) to defend against potential risks. Unfortunately, these datasets do not provide finer-grained file-level vulnerability information, compromising their effectiveness. Previous works have not adequately addressed this issue, and mainstream vulnerability detection methods have their drawbacks that hinder resolving this gap. Driven by the real needs, we propose a framework that can trace the vulnerability-relevant file for each disclosed vulnerability. Our approach uses NVD descriptions with metadata as the inputs, and employs a series of strategies with a LLM model, search engine, heuristic-based text matching method and a deep learning classifier to recommend the most likely vulnerability-relevant file, effectively enhancing the completeness of existing NVD data. Our experiments confirm that the efficiency of the proposed framework, with CodeBERT achieving 0.92 AUC and 0.85 MAP, and our user study proves our approach can help with vulnerability-relevant file detection effectively. To the best of our knowledge, our work is the first one focusing on tracing vulnerability-relevant files, laying the groundwork of building finer-grained vulnerability-aware software bill of materials.

The Python community strives to design pythonic idioms so that Python users can achieve their intent in a more concise and efficient way. According to our analysis of 154 questions about challenges of understanding pythonic idioms on Stack Overflow, we find that Python users face various challenges in comprehending pythonic idioms. And the usage of pythonic idioms in 7,577 GitHub projects reveals the prevalence of pythonic idioms. By using a statistical sampling method, we find pythonic idioms result in not only lexical conciseness but also the creation of variables and functions, which indicates it is not straightforward to map back to non-idiomatic code. And usage of pythonic idioms may even cause potential negative effects such as code redundancy, bugs and performance degradation. To alleviate such readability issues and negative effects, we develop a transforming tool, DeIdiom, to automatically transform idiomatic code into equivalent non-idiomatic code. We test and review over 7,572 idiomatic code instances of nine pythonic idioms (list/set/dict-comprehension, chain-comparison, truth-value-test, loop-else, assign-multi-targets, for-multi-targets, star), the result shows the high accuracy of DeIdiom. Our user study with 20 participants demonstrates that explanatory non-idiomatic code generated by DeIdiom is useful for Python users to understand pythonic idioms correctly and efficiently, and leads to a more positive appreciation of pythonic idioms.

Logical query answering over Knowledge Graphs (KGs) is a fundamental yet complex task. A promising approach to achieve this is to embed queries and entities jointly into the same embedding space. Research along this line suggests that using multi-modal distribution to represent answer entities is more suitable than uni-modal distribution, as a single query may contain multiple disjoint answer subsets due to the compositional nature of multi-hop queries and the varying latent semantics of relations. However, existing methods based on multi-modal distribution roughly represent each subset without capturing its accurate cardinality, or even degenerate into uni-modal distribution learning during the reasoning process due to the lack of an effective similarity measure. To better model queries with diversified answers, we propose Query2GMM for answering logical queries over knowledge graphs. In Query2GMM, we present the GMM embedding to represent each query using a univariate Gaussian Mixture Model (GMM). Each subset of a query is encoded by its cardinality, semantic center and dispersion degree, allowing for precise representation of multiple subsets. Then we design specific neural networks for each operator to handle the inherent complexity that comes with multi-modal distribution while alleviating the cascading errors. Last, we design a new similarity measure to assess the relationships between an entity and a query's multi-answer subsets, enabling effective multi-modal distribution learning for reasoning. Comprehensive experimental results show that Query2GMM outperforms the best competitor by an absolute average of 6.35%.

Giant pre-trained code models (PCMs) start coming into the developers’ daily practices. Understanding the type and amount of software knowledge in PCMs is essential for integrating PCMs into software engineering (SE) tasks and unlocking their potential. In this work, we conduct the first systematic study on the SE factual knowledge in the state-of-the-art PCM CoPilot, focusing on APIs’ Fully Qualified Names (FQNs), the fundamental knowledge for effective code analysis, search and reuse. Driven by FQNs’ data distribution properties, we design a novel lightweight in-context learning on Copilot for FQN inference, which does not require code compilation as traditional methods or gradient update by recent FQN prompt-tuning. We systematically experiment with five in-context learning design factors to identify the best configuration for practical use. With this best configuration, we investigate the impact of example prompts and FQN data properties on CoPilot's FQN inference capability. Our results confirm that CoPilot stores diverse FQN knowledge and can be applied for FQN inference due to its high accuracy and non-reliance on code analysis. Additionally, our extended study shows that the in-context learning method can be generalized to retrieve other SE factual knowledge embedded in giant PCMs. Furthermore, we find that the advanced general model GPT-4 also stores substantial SE knowledge. Comparing FQN inference between CoPilot and GPT-4, we observe that as model capabilities improve, the same prompts yield better results. Based on our experience interacting with Copilot, we discuss various opportunities to improve human-CoPilot interaction in the FQN inference task.

The emergence of foundation models, such as large language models (LLMs) GPT-4 and text-to-image models DALL-E, has opened up numerous possibilities across various domains. People can now use natural language (i.e., prompts) to communicate with AI to perform tasks. While people can use foundation models through chatbots (e.g., ChatGPT), chat, regardless of the capabilities of the underlying models, is not a production tool for building reusable AI services. APIs like LangChain allow for LLM-based application development but require substantial programming knowledge, thus posing a barrier. To mitigate this, we systematically review, summarise, refine and extend the concept of AI chain by incorporating the best principles and practices that have been accumulated in software engineering for decades into AI chain engineering, to systematize AI chain engineering methodology. We also develop a no-code integrated development environment, Prompt Sapper , which embodies these AI chain engineering principles and patterns naturally in the process of building AI chains, thereby improving the performance and quality of AI chains. With Prompt Sapper, AI chain engineers can compose prompt-based AI services on top of foundation models through chat-based requirement analysis and visual programming. Our user study evaluated and demonstrated the efficiency and correctness of Prompt Sapper.

Dataflow graphs (DFGs) capture definitions (defs) and uses across program blocks, which is a fundamental program representation for program analysis, testing and maintenance. However, dynamically typed programming languages like Python present implicit dataflow issues that make it challenging to determine def-use flow information at compile time. Static analysis methods like Soot and WALA are inadequate for handling these issues, and manually enumerating comprehensive heuristic rules is impractical. Large pre-trained language models (LLMs) offer a potential solution, as they have powerful language understanding and pattern matching abilities, allowing them to predict implicit dataflow by analyzing code context and relationships between variables, functions, and statements in code. We propose leveraging LLMs’ in-context learning ability to learn implicit rules and patterns from code representation and contextual information to solve implicit dataflow problems. To further enhance the accuracy of LLMs, we design a five-step chain of thought (CoT) and break it down into an Artificial Intelligence (AI) chain, with each step corresponding to a separate AI unit to generate accurate DFGs for Python code. Our approach’s performance is thoroughly assessed, demonstrating the effectiveness of each AI unit in the AI Chain. Compared to static analysis, our method achieves 82% higher def coverage and 58% higher use coverage in DFG generation on implicit dataflow. We also prove the indispensability of each unit in the AI Chain. Overall, our approach offers a promising direction for building software engineering tools by utilizing foundation models, eliminating significant engineering and maintenance effort, but focusing on identifying problems for AI to solve.

APIs have intricate relations that can be described in text and represented as knowledge graphs to aid software engineering tasks. Existing relation extraction methods have limitations, such as limited API text corpus, and are affected by the characteristics of the input text. To address these limitations, we propose utilizing large language models (LLMs) (e.g., GPT-3.5) as a neural knowledge base for API relation inference. This approach leverages the entire Web used to pre-train LLMs as a knowledge base and is insensitive to the context and complexity of input texts. To ensure accurate inference, we design an AI chain consisting of three AI modules: API Fully Qualified Name (FQN) Parser, API Knowledge Extractor, and API Relation Decider. The accuracy of the API FQN Parser and API Relation Decider is 0.81 and 0.83, respectively. Using the generative capacity of the LLM and our approach’s inference capability, we achieve an average F1 value of 0.76 under the three datasets, significantly higher than the state-of-the-art method’s average F1 value of 0.40. Compared to the original CoT and modularized CoT methods, our AI chain design has improved the performance of API relation inference by 71% and 49%, respectively. Meanwhile, the prompt ensembling strategy enhances the performance of our approach by 32%. The API relations inferred by our method can be further organized into structured forms to provide support for other software engineering tasks.

Extraction of Application Programming Interfaces (APIs) and their semantic relations from unstructured text (e.g., Stack Overflow) is a fundamental work for software engineering tasks (e.g., API recommendation). However, existing approaches are rule based and sequence labeling based. They must manually enumerate the rules or label data for a wide range of sentence patterns, which involves a significant amount of labor overhead and is exacerbated by morphological and common-word ambiguity. In contrast to matching or labeling API entities and relations, this article formulates heterogeneous API extraction and API relation extraction task as a sequence-to-sequence generation task and proposes the API Entity-Relation Joint Extraction framework (AERJE), an API entity-relation joint extraction model based on the large pre-trained language model. After training on a small number of ambiguous but correctly labeled data, AERJE builds a multi-task architecture that extracts API entities and relations from unstructured text using dynamic prompts. We systematically evaluate AERJE on a set of long and ambiguous sentences from Stack Overflow. The experimental results show that AERJE achieves high accuracy and discrimination ability in API entity-relation joint extraction, even with zero or few-shot fine-tuning.

Partial code usually involves non-fully-qualified type names (non-FQNs) and undeclared receiving objects. Resolving the FQNs of these non-FQN types and undeclared receiving objects (referred to as type inference) is the prerequisite to effective search and reuse of partial code. Existing dictionary-lookup based methods build a symbolic knowledge base of API names and code contexts, which involve significant compilation overhead and are sensitive to unseen API names and code context variations. In this article, we propose using a p rompt-tuned c o de m asked language mod e l (MLM) as a neural knowledge base for type inference, called POME, which is lightweight and has minimal requirements on code compilation. Unlike the existing symbol name and context matching for type inference, POME infers the FQNs syntax and usage knowledge encapsulated in prompt-tuned code MLM through a colze-style fill-in-blank strategy. POME is integrated as a plug-in into web and integrated development environments (IDE) to assist developers in inferring FQNs in the real world. We systematically evaluate POME on a large amount of source code from GitHub and Stack Overflow, and explore its generalization and hybrid capability. The results validate the effectiveness of the POME design and its applicability for partial code type inference, and they can be easily extended to different programming languages (PL). POME can also be used to generate a PL-hybrid type inference model for providing a one-for-all solution. As the first of its kind, our neural type inference method opens the door to many innovative ways of using partial code.

Vulnerable third-party libraries pose significant threats to software applications that reuse these libraries. At an industry scale of reuse, manual analysis of third-party library vulnerabilities can be easily overwhelmed by the sheer number of vulnerabilities continually collected from diverse sources for thousands of reused libraries. Our study of four large-scale, actively maintained vulnerability databases (NVD, IBM X-Force, ExploitDB, and Openwall) reveals the wide presence of information discrepancies, in terms of seven vulnerability aspects, i.e., product, version, component, vulnerability type, root cause, attack vector, and impact, between the reports for the same vulnerability from heterogeneous sources. It would be beneficial to integrate and cross-validate multi-source vulnerability information, but it demands automatic aspect extraction and aspect discrepancy detection. In this work, we experimented with a wide range of NLP methods to extract named entities (e.g., product) and free-form phrases (e.g., root cause) from textual vulnerability reports and to detect semantically different aspect mentions between the reports. Our experiments confirm the feasibility of applying NLP methods to automate aspect-level vulnerability analysis and identify the need for domain customization of general NLP methods. Based on our findings, we propose a discrepancy-aware, aspect-level vulnerability knowledge graph and a KG-based web portal that integrates diversified vulnerability key aspect information from heterogeneous vulnerability databases. Our conducted user study proves the usefulness of our web portal. Our study opens the door to new types of vulnerability integration and management, such as vulnerability portraits of a product and explainable prediction of silent vulnerabilities.

Developers’ API needs should be more pragmatic, such as seeking suggestive, explainable, and extensible APIs rather than the so-called best result. Existing API search research cannot meet these pragmatic needs because they are solely concerned with query-API relevance. This necessitates a focus on enhancing the entire query process, from query definition to query refinement through intent clarification to query results promoting divergent thinking about results. This paper designs a novel Knowledge-Aware Human-AI Dialog agent (KAHAID) which guides the developer to clarify the uncertain, under-specified query through multi-round question answering and recommends APIs for the clarified query with relevance explanation and extended suggestions (e.g., alternative, collaborating or opposite-function APIs). We systematically evaluate KAHAID. In terms of human-AI dialogue process, it achieves a high diversity of question options (the average diversity between any two options is 74.9%) and the ability to guide developers to find APIs using fewer dialogue rounds (no more than 3 rounds on average). For API recommendation, KAHAID achieves an MRR and MAP of 0.769 and 0.794, outperforming state-of-the-art API search approaches BIKER and CLEAR by at least 47% in MRR and 226.7% in MAP. For knowledge extension, KAHAID obtains an MRR and MAP of 0.815 and 0.864, surpassing state-of-the-art query clarification approaches by at least 42% in MRR and 45.2% in MAP. As the first of its kind, KAHAID opens the door to integrating the immediate response capability of API research and the interaction, clarification, explanation, and extensibility capability of social-technical information seeking.

Compared to other programming languages (e.g., Java), Python has more idioms to make Python code concise and efficient. Although Pythonic idioms are well accepted in the Python community, Python programmers are often faced with many challenges in using them, for example, being unaware of certain Pythonic idioms or not knowing how to use them properly. Based on an analysis of 7,577 Python repositories on GitHub, we find that non-idiomatic Python code that can be implemented with Pythonic idioms occurs frequently and widely. To assist Python developers in adopting Pythonic idioms, we design and implement an automatic refactoring tool named RIdiom to refactor code with Pythonic idioms. We identify twelve Pythonic idioms by systematically contrasting the abstract syntax grammar of Python and Java. Then we define the syntactic patterns for detecting non-idiomatic code for each Pythonic idiom. Finally, we devise atomic AST-rewriting operations and refactoring steps to refactor non-idiomatic code into idiomatic code. Our approach is evaluated on 1,814 code refactorings, achieving a precision of 0.99 and a recall of 0.87, underscoring its effectiveness. We further evaluate the tool's utility in helping developers refactor code with Pythonic idioms. A user study involving 14 students demonstrates a 112.9% improvement in correctness and a 35.5% speedup when referring to the tool-generated code pairs. Additionally, the 120 pull requests that refactor non-idiomatic code with Pythonic idioms, submitted to GitHub projects, resulted in 79 responses. Among these, 49 accepted and praised the refactorings, with 42 merging the refactorings into their repositories.

The field of generating recommendations within the framework of causal inference has seen a recent surge.This approach enhances insights into the influence of recommendations on user behavior and helps in identifying the underlying factors. Existing research has often leveraged propensity scores to mitigate bias, albeit at the risk of introducing additional variance. Others have explored the use of unbiased data from randomized controlled trials, although this comes with assumptions that may prove challenging in practice. In this paper, we first present the causality-aware interpretation of recommendations and reveal how the underlying exposure mechanism can bias the maximum likelihood estimation (MLE) of observational feedback. Recognizing that confounders may be elusive, we propose a contrastive self-supervised learning to minimize exposure bias, employing inverse propensity scores and expanding the positive sample set. Building on this foundation, we present a novel contrastive counterfactual learning method (CCL) that incorporates three unique positive sampling strategies grounded in estimated exposure probability or random counterfactual samples. Through extensive experiments on two real-world datasets, we demonstrate that our CCL outperforms the state-of-the-art methods.

With the increasing demand for communication between blockchains, improving the performance of cross-chain communication protocols becomes an emerging challenge. We take a first step towards analyzing the limitations of cross-chain communication protocols by comprehensively evaluating Cosmos Network's Inter-Blockchain Communication Protocol. To achieve our goal we introduce a novel framework to guide empirical evaluations of cross-chain communication protocols. We implement an instance of our framework as a tool to evaluate the IBC protocol. Our findings highlight several challenges, such as high transaction confirmation latency, bottlenecks in the blockchain's RPC implementation and concurrency issues that hinder the scalability of the cross-chain message relayer. We also demonstrate how to reduce the time required to complete cross-chain transfers by up to 70% when submitting large amounts of transfers. Finally, we discuss challenges faced during deployment with the objective of contributing to the development and advancement of cross-chain communication.

Across industries, there is an ever-increasing rate of data sharing for collaboration and innovation between organizations and their customers, partners, suppliers, and internal teams. However, many enterprises are restricted from freely sharing data due to regulatory restrictions across different regions, performance issues in moving large volume data, or requirements to maintain autonomy. In such situations, the enterprise can benefit from the concept of federated learning, in which machine learning models are constructed at various geographic sites. In this paper, we introduce a general framework, namely BDSP, to share data among enterprises based on Blockchain and federated learning techniques. Specifically, we propose a transparency contribution accounting mechanism to estimate the valuation of data and implement a proof-of-concept for further evaluation. The extensive experimental results show that the proposed BDSP has a competitive performance with higher training accuracy, an increase of over 5%, and lower communication overhead, reducing 3 times, compared to baseline approaches.

Existing NFTs confront restrictions of one-time incentive and product isolation. Creators cannot obtain benefits once having sold their NFT products due to the lack of relationships across different NFTs, which results in controversial profit sharing. This paper proposes a referable NFT solution to extend the incentive sustainability of NFTs. We construct the referable NFT (rNFT) network to increase exposure and enhance the referring relationship of inclusive items. We introduce the DAG topology to generate directed edges between each pair of NFTs with corresponding weights and labels for advanced usage. We accordingly implement and propose the scheme under Ethereum Improvement Proposal (EIP) standards, indexed in EIP-5521. Further, we provide the mathematical formation to analyze the utility for each rNFT participant. The discussion gives general guidance among multi-dimensional parameters. The solution, as a result, shape the recognition of potential values hidden in isolated NFTs and raise the interest of communities toward the discovery of NFT derivatives. To our knowledge, this is the first study to build a referable NFT network, explicitly showing the virtual connections among NFTs.

Decentralized autonomous organizations (DAOs) are critical to the blockchain ecosystem as they enable decentralized decision-making and governance, and facilitate the creation of decentralized applications (DApps) and organizations. However, despite significant importance, there is currently a lack of a comprehensive overview and detailed understanding of DAOs. To address the gap, this work presents a primary investigation of DAOs (35+). We category, examine and evaluate existing DAOs regarding their operational features, (non-)functionalities and real-world performance. In addition, we provide a consolidated exploration of DAOs by conducting a literature review [1] and an empirical study on mainstream projects, particularly Snapshot [2]. Our research contributes to a better understanding of DAOs and their potential impact on the blockchain ecosystem.

The development of effective recommender systems for Web3 assets, such as the Non-Fungible Token (NFT), requires concentration along with the growth of popularity and heterogeneity in many potential applications such as Web3 gaming and NFT rental markets, the requirements of predicting rNFT classification desire a practical solution. In this paper, we make use of the referable NFT (rNFT11In this work, rNFT mainly refers to the EIP-5521 protocol and corresponding formed network/topology [1], while NFT is used in the context of a single node, node sets, or products that align with the EIP-5521 protocol.) standard [2], indexed EIP-5521, to construct an rNFT classification framework leveraging Graph Neural Network (GNN), an emerging branch of Deep Learning (DL), which learns on the inherent topology of graph-based data. In particular, we first transform the rNFT backward and onward reference relationship to a Direct Acyclic Graph (DAG) and model appropriate node and edge features from rNFT metadata and associated token transactions. Next, a multi-layer GraphSage model is designed to include the collected features for the learning process. In this way, the model takes into account graph topology together with features to classify both the existing and incoming NFT nodes in a supervised way. We also give comprehensive elaboration on the architecture of the new GNN-based recommender system with discussions in regard to its characteristics and challenges. Furthermore, we expect to conduct extensive experiments, by presenting an initial plan, to show the feasibility and efficacy of our system.

Architectural design contexts contain a set of factors that greatly influence software application development. Among them, organizational design contexts consist of high-level company concerns and how it is structured, for example, stakeholders and development schedules heavily impacting design considerations. The Decentralized Autonomous Organization (DAO), as a vital concept in the Web3 space, represents an organization constructed by automatically executed rules, such as via smart contracts, holding features of the permissionless committee, transparent proposals, and fair contribution by participated stakeholders. In this work, we conduct a systematic literature review of existing DAO literature to summarize its structural features, benefits and challenges, and potential development directions in the context of Web3 applications.

Recent studies have shown that recommender systems (RSs) are highly vulnerable to data poisoning attacks. Understanding attack tactics helps improve the robustness of RSs. We intend to develop efficient attack methods that use limited resources to generate high-quality fake user profiles to achieve 1) transferability among black-box RSs 2) and imperceptibility among detectors. In order to achieve these goals, we introduce textual reviews of products to enhance the generation quality of the profiles. Specifically, we propose a novel attack framework named R-Trojan, which formulates the attack objectives as an optimization problem and adopts a tailored transformer-based generative adversarial network (GAN) to solve it so that high-quality attack profiles can be produced. Comprehensive experiments on real-world datasets demonstrate that R-Trojan greatly outperforms state-of-the-art attack methods on various victim RSs under black-box settings and show its good imperceptibility.

Privacy has become an increasingly essential quality to consider in a software system. Privacy practices should be adopted from the first stage of software design to safeguard personal data from unintentional information leakage. Privacy patterns have been investigated by academic and industry practitioners to address different privacy issues. The presence of these patterns is both helpful and challenging for the designer. On one hand, the privacy patterns are valuable as reusable design solutions to solve common privacy problems. On the other hand, the multitude of privacy patterns makes the choice of patterns as solutions difficult for the designer. In this paper, we propose a selection model that can assist architects in deciding suitable patterns for a software system. The selection is based on the regulatory entities and architectural characteristics implicit in the patterns. We evaluate the proposed selection model through case studies and interviews with practitioners. Our evaluation accesses the applicability and usefulness of the selection model in guiding the pattern selection for architectural design and understanding the rationale of different design decisions.

Due to convenience, open-source software is widely used. For beneficial reasons, open-source maintainers often fix the vulnerabilities silently, exposing their users unaware of the updates to threats. Previous works all focus on black-box binary detection of the silent dependency alerts that suffer from high false-positive rates. Open-source software users need to analyze and explain AI prediction themselves. Explainable AI becomes remarkable as a complementary of black-box AI models, providing details in various forms to explain AI decisions. Noticing there is still no technique that can discover silent dependency alert on time, in this work, we propose a framework using an encoder-decoder model with a binary detector to provide explainable silent dependency alert prediction. Our model generates 4 types of vulnerability key aspects including vulnerability type, root cause, attack vector, and impact to enhance the trustworthiness and users' acceptance to alert prediction. By experiments with several models and inputs, we confirm CodeBERT with both commit messages and code changes achieves the best results. Our user study shows that explainable alert predictions can help users find silent dependency alert more easily than black-box predictions. To the best of our knowledge, this is the first research work on the application of Explainable AI in silent dependency alert prediction, which opens the door of the related domains.

The usage of Python idioms is popular among Python developers in a formative study of 101 Python idiom performance related questions on Stack Overflow, we find that developers often get confused about the performance impact of Python idioms and use anecdotal toy code or rely on personal project experience which is often contradictory in performance outcomes. There has been no large-scale, systematic empirical evidence to reconcile these performance debates. In the paper, we create a large synthetic dataset with 24,126 pairs of non-idiomatic and functionally-equivalent idiomatic code for the nine unique Python idioms identified in [1], and reuse a large real-project dataset of 54,879 such code pairs provided in [1]. We develop a reliable performance measurement method to compare the speedup or slowdown by idiomatic code against non-idiomatic counterpart, and analyze the performance discrepancies between the synthetic and real-project code, the relationships between code features and performance changes, and the root causes of performance changes at the bytecode level. We summarize our findings as some actionable suggestions for using Python idioms.

Programming screencasts (e.g., video tutorials on Youtube or live coding stream on Twitch) are important knowledge source for developers to learn programming knowledge, especially the workflow of completing a programming task. Nonetheless, the image nature of programming screencasts limits the accessibility of screencast content and the workflow embedded in it, resulting in a gap to access and interact with the content and workflow in programming screencasts. Existing non-intrusive methods are limited to extract either primitive human-computer interaction (HCI) actions or coarse-grained video fragments. In this work, we leverage Computer Vision (CV) techniques to build a programming screencast analysis tool which can automatically extract code-line editing steps (enter text, delete text, edit text and select text) from screencasts. Given a programming screencast, our approach outputs a sequence of coding steps and code snippets involved in each step, which we refer to as programming workflow. The proposed method is evaluated on 41 hours of tutorial videos and live coding screencasts with diverse programming environments. The results demonstrate our tool can extract code-line editing steps accurately and the extracted workflow steps can be intuitively understood by developers.

To speed up the bug-fixing process, it is essential to triage bugs into the right components as soon as possible. Given the large number of bugs filed everyday, a reliable and effective bug-component triaging tool is needed to assist this task. LR-BKG is the state-of-the-art toolkit for doing this. However, the suboptimal performance for recommending the right component at the first position (low Top-1 accuracy) limits its usage in practice. We thoroughly investigate the limitations of LR-BKG and find out the gap between the manual feature design of LR-BKG and the characteristics of bug reports causes such suboptimal performance. Therefore, we propose an approach, DEEPTRIAG, which uses the large scale pre-trained models to extract deep features automatically from bug reports (including bug summary and description), to fill this gap. DEEPTRIAG transforms bug-component triaging into a multi-classification task (CodeBERT-Classifier) and a generation task (CodeT5-Generator). Then, we ensemble the prediction results from them to improve the performance of bug-component triaging further. Extensive experimental results demonstrate the superior performance of DEEPTRIAG on bug-component triaging over LR-BKG. In particular, the overall Top-1 accuracy is improved from 56.2% to 68.3% on Mozilla dataset and from 51.3% to 64.1% on Eclipse dataset, which verifies the effectiveness and generalization of our approach on improving the practical usage for bug-component triaging.

API recommendation methods have evolved from literal and semantic keyword matching to query expansion and query clarification. The latest query clarification method is knowledge graph (KG)-based, but limitations include out-of-vocabulary (OOV) failures and rigid question templates. To address these limitations, we propose a novel knowledge-guided query clarification approach for API recommendation that leverages a large language model (LLM) guided by KG. We utilize the LLM as a neural knowledge base to overcome OOV failures, generating fluent and appropriate clarification questions and options. We also leverage the structured API knowledge and entity relationships stored in the KG to filter out noise, and transfer the optimal clarification path from KG to the LLM, increasing the efficiency of the clarification process. Our approach is designed as an AI chain that consists of five steps, each handled by a separate LLM call, to improve accuracy, efficiency, and fluency for query clarification in API recommendation. We verify the usefulness of each unit in our AI chain, which all received high scores close to a perfect 5. When compared to the baselines, our approach shows a significant improvement in MRR, with a maximum increase of 63.9% higher when the query statement is covered in KG and 37.2% when it is not. Ablation experiments reveal that the guidance of knowledge in the KG and the knowledge-guided pathfinding strategy are crucial for our approach's performance, resulting in a 19.0% and 22.2% increase in MAP, respectively. Our approach demonstrates a way to bridge the gap between KG and LLM, effectively compensating for the strengths and weaknesses of both.

Mobile apps bring us many conveniences, such as online shopping and communication, but some use malicious designs called dark patterns to trick users into doing things that are not in their best interest. Many works have been done to summarize the taxonomy of these patterns and some have tried to mitigate the problems through various techniques. However, these techniques are either time-consuming, not generalisable or limited to specific patterns. To address these issues, we propose UIGuard, a knowledge-driven system that utilizes computer vision and natural language pattern matching to automatically detect a wide range of dark patterns in mobile UIs. Our system relieves the need for manually creating rules for each new UI/app and covers more types with superior performance. In detail, we integrated existing taxonomies into a consistent one, conducted a characteristic analysis and distilled knowledge from real-world examples and the taxonomy. Our UIGuard consists of two components, Property Extraction and Knowledge-Driven Dark Pattern Checker. We collected the first dark pattern dataset, which contains 4,999 benign UIs and 1,353 malicious UIs of 1,660 instances spanning 1,023 mobile apps. Our system achieves a superior performance in detecting dark patterns (micro averages: 0.82 in precision, 0.77 in recall, 0.79 in F1 score). A user study involving 58 participants further showed that UIGuard significantly increases users’ knowledge of dark patterns. We demonstrated potential use cases of our work, which can benefit different stakeholders, and serve as a training tool for raising awareness of dark patterns.

Federated learning is an emerging privacy-preserving AI technique where clients (i.e., organizations or devices) train models locally and formulate a global model based on the local model updates without transferring local data externally. However, federated learning systems struggle to achieve trustworthiness and embody responsible AI principles. In particular, federated learning systems face accountability and fairness challenges due to multistakeholder involvement and heterogeneity in client data distribution. To enhance the accountability and fairness of federated learning systems, we present a blockchain-based trustworthy federated learning architecture. We first design a smart contract-based data-model provenance registry to enable accountability. Additionally, we propose a weighted fair data sampler algorithm to enhance fairness in training data. We evaluate the proposed approach using a COVID-19 X-ray detection use case. The evaluation results show that the approach is feasible to enable accountability and improve fairness. The proposed algorithm can achieve better performance than the default federated learning setting in terms of the model’s generalization and accuracy.

Existing permissionless sharded-Blockchains come on the scene. However, there is a lack of systematic formulations and experiments regarding the behaviors of individual miners. In this article, we interpret block mining in a permissionless sharded-Blockchain as a repeated$M$-player noncooperative game with finite actions, and propose a new multiagent deep reinforcement learning (MADRL) framework to allow the miners to maximize their profits in a decentralized fashion by scheduling their resources across the shards without centralized coordination. We formulate the rewards, and design a two-scale action space for each miner to reduce the action space and expedite convergence. We also propose a new MADRL model, named Rainbow-WoLF-PHC, which allows each miner to learn its resource allocation online and converge fast to a mixed strategy Nash equilibrium. Extensive experiments show the superiority of the Rainbow-WoLF-PHC to its alternatives in terms of convergence, stability, and profitable actions. This work provides a prosperous design of an end-user-friendly permissionless sharded-Blockchain.

As the deployment of artificial intelligence (AI) is changing many fields and industries, there are concerns about AI systems making decisions and recommendations without adequately considering various ethical aspects, such as accountability, reliability, transparency, explainability, contestability, privacy, and fairness. While many sets of AI ethics principles have been recently proposed that acknowledge these concerns, such principles are high-level and do not provide tangible advice on how to develop ethical and responsible AI systems. To gain insight on the possible implementation of the principles, we conducted an empirical investigation involving semi-structured interviews with a cohort of AI practitioners. The salient findings cover four aspects of AI system design and development, adapting processes used in software engineering: (i) high-level view, (ii) requirements engineering, (iii) design and implementation, (iv) deployment and operation.

Although AI is transforming the world, there are serious concerns about its ability to behave and make decisions responsibly. Many ethical regulations, principles, and frameworks for responsible AI have been issued recently. However, they are high level and difficult to put into practice. On the other hand, most AI researchers focus on algorithmic solutions, while the responsible AI challenges actually crosscut the entire engineering lifecycle and components of AI systems. To close the gap in operationalizing responsible AI, this paper aims to develop a roadmap on software engineering for responsible AI. The roadmap focuses on (i) establishing multi-level governance for responsible AI systems, (ii) setting up the development processes incorporating process-oriented practices for responsible AI systems, and (iii) building responsible-AI-by-design into AI systems through system-level architectural style, patterns and techniques.

Recent blockchain-based carbon markets focus on permit-based trading requiring manual application processes to grant the right for carbon emission. A decentralized blockchain-based carbon market without relying on off-chain permits is yet to be explored. In this paper, we present a new design of blockchain-based carbon trading through the introduction of Carboncoin – a blockchain asset which tokenizes the right of energy producers to emit carbon. Instead of relying on off-chain and centralized permits, producers are allowed to freely exchange Carboncoin with each other for fiat currency. By using an on-chain asset, carbon production can be automatically expensed whenever a producer records new energy production which is certified on the blockchain. Moreover, the proposed design enables generic ESG (Environmental, Social and Governance) data to be used to provide a more holistic reputation score inclusive of ESG initiatives undertaken by market participants. We conclude that entirely blockchain-based carbon markets can be made more comprehensive using ESG data and on-chain assets, but at the cost of reduced performance.

Security databases describe characteristics of discovered vulnerabilities in text for future studying and patching. However, due to different maintainers having different perspectives about vulnerabilities, they often describe the same vulnerability in different ways, creating obstacles for gathering comprehensive information about the vulnerabilities from different databases. To mitigate this problem, Common Vulnerability and Exposures (CVE) is established to identify each publicly disclosed vulnerability with a unique CVE id, and vulnerability databases by different vendors and organizations can reference the CVE ids in their vulnerability reports. In spite of the wide adoption of CVEs, traceability issues are still prevalent. Our empirical study on vulnerability traceability across four representative security databases (NVD, IBM X-Force, ExploitDB, Openwall) shows that there was a fast-increasing amount of CVE records, traceability delay, and missing issues become severe for the vulnerability databases. To address these issues, we develop an automatic traceability recovery method for recommending related external vulnerability reports to the reports in one database. As vulnerability reports from different databases differ in content details and length, our approach does not match the reports at the document level but extracts seven distinctive vulnerability key aspects that are widely present in vulnerability descriptions. As a proof of concept, we apply our methods to recommend the reports from IBM X-Force, ExploitDB and Openwall to the NVD report. We use NVD as the target because it is a de-facto standard vulnerability database that contains the most comprehensive list of vulnerabilities. Our experiments on a wide range of NLP methods show our aspect-level matching methods can achieve high MRR and accuracy for traceability recovery across heterogeneous vulnerability databases.

Partial code usually involves non-fully-qualified type names (non-FQNs) and undeclared receiving objects. Resolving the FQNs of these non-FQN types and undeclared receiving objects (referred to as type inference) is the prerequisite to effective search and reuse of partial code. Existing dictionary-lookup based methods build a symbolic knowledge base of API names and code contexts, which involve significant compilation overhead and are sensitive to unseen API names and code context variations. In this paper, we formulate type inference as a cloze-style fill-in-blank language task. Built on source code naturalness, our approach fine-tunes a code masked language model (MLM) as a neural knowledge base of code elements with a novel “pre-train, prompt and predict” paradigm from raw source code. Our approach is lightweight and has minimum requirements on code compilation. Unlike existing symbolic name and context matching for type inference, our prompt-tuned code MLM packs FQN syntax and usage in its parameters and supports fuzzy neural type inference. We systematically evaluate our approach on a large amount of source code from GitHub and Stack Overflow. Our results confirm the effectiveness of our approach design and the practicality for partial code type inference. As the first of its kind, our neural type inference method opens the door to many innovative ways of using partial code.

Exploratory testing is an effective testing approach which leverages the tester’s knowledge and creativity to design test cases to provoke and recognize failures at the system level from the end user’s perspective. Although some principles and guidelines have been proposed to guide exploratory testing, there are no effective tools for automatic generation of exploratory test scenarios (a.k.a soap opera tests). Existing test generation techniques rely on specifications, program differences and fuzzing, which are not suitable for exploratory test generation. In this paper, we propose to leverage the scenario and oracle knowledge in bug reports to generate soap opera test scenarios. We develop open information extraction methods to construct a system knowledge graph (KG) of user tasks and failures from the steps to reproduce, expected results and observed results in bug reports. We construct a proof-of-concept KG from 25,939 bugs of the Firefox browser. Our evaluation shows the constructed KG is of high quality. Based on the KG, we create soap opera test scenarios by combining the scenarios of relevant bugs, and develop a web tool to present the created test scenarios and support exploratory testing. In our user study, 5 users find 18 bugs from 5 seed bugs in 2 hours using our tool, while the control group finds only 5 bugs based on the recommended similar bugs.

With the widespread accumulation of observational data, researchers obtain a new direction to learn counterfactual effects in many domains (e.g., health care and computational advertising) without Randomized Controlled Trials (RCTs). However, observational data suffer from inherent missing counterfactual outcomes and distribution discrepancy between treatment and control groups due to behaviour preference. Motivated by recent advances in representation learning in domain adaptation, we propose a novel framework based on Cycle-Balanced REpresentation learning for counterfactual inference (CBRE) to solve the above problems. Specifically, we realize a robust and balanced representation for different groups using adversarial training. Meanwhile, we construct an information loop that preserves original data properties cyclically, reducing information loss when transforming data into latent representation space. Experimental results on three real-world datasets demonstrate that CBRE matches/outperforms the state-of-the-art methods, and it has a great potential to be applied to counterfactual inference.

Graphical User Interface (GUI) is not merely a collection of individual and unrelated widgets, but rather partitions discrete widgets into groups by various visual cues, thus forming higher-order perceptual units such as tab, menu, card or list. The ability to automatically segment a GUI into perceptual groups of widgets constitutes a fundamental component of visual intelligence to automate GUI design, implementation and automation tasks. Although humans can partition a GUI into meaningful perceptual groups of widgets in a highly reliable way, perceptual grouping is still an open challenge for computational approaches. Existing methods rely on ad-hoc heuristics or supervised machine learning that is dependent on specific GUI implementations and runtime information. Research in psychology and biological vision has formulated a set of principles (i.e., Gestalt theory of perception) that describe how humans group elements in visual scenes based on visual cues like connectivity, similarity, proximity and continuity. These principles are domain-independent and have been widely adopted by practitioners to structure content on GUIs to improve aesthetic pleasantness and usability. Inspired by these principles, we present a novel unsupervised image-based method for inferring perceptual groups of GUI widgets. Our method requires only GUI pixel images, is independent of GUI implementation, and does not require any training data. The evaluation on a dataset of 1,091 GUIs collected from 772 mobile apps and 20 UI design mockups shows that our method significantly outperforms the state-of-the-art ad-hoc heuristics-based baseline. Our perceptual grouping method creates opportunities for improving UI-related software engineering tasks.

Compared to other programming languages (e.g., Java), Python has more idioms to make Python code concise and efficient. Although pythonic idioms are well accepted in the Python community, Python programmers are often faced with many challenges in using them, for example, being unaware of certain pythonic idioms or do not know how to use them properly. Based on an analysis of 7,638 Python repositories on GitHub, we find that non-idiomatic Python code that can be implemented with pythonic idioms occurs frequently and widely. Unfortunately, there is no tool for automatically refactoring such non-idiomatic code into idiomatic code. In this paper, we design and implement an automatic refactoring tool to make Python code idiomatic. We identify nine pythonic idioms by systematically contrasting the abstract syntax grammar of Python and Java. Then we define the syntactic patterns for detecting non-idiomatic code for each pythonic idiom. Finally, we devise atomic AST-rewriting operations and refactoring steps to refactor non-idiomatic code into idiomatic code. We test and review over 4,115 refactorings applied to 1,065 Python projects from GitHub, and submit 90 pull requests for the 90 randomly sampled refactorings to 84 projects. These evaluations confirm the high accuracy, practicality and usefulness of our refactoring tool on real-world Python code.

With recent developments in communication technologies, vehicular networks have become a reality with various applications. However, the cybersecurity aspect of vehicular networks is still an open issue that needs to be addressed with novel defence mechanisms against attacks. This paper first presents the state-of-the-art communication technologies in vehicular networks (either inter-vehicle networking or in-vehicle networking) along with their applications. Then we explore novel technologies including machine learning and blockchain as cybersecurity defence mechanisms in vehicular networks. Based on the extensive survey, we highlight some insights for future research to secure vehicular networks.

Deep reinforcement learning enables an agent to capture users' interest through dynamic interactions with the environment. It uses a reward function to learn user's interest and to control the learning process, attracting great interest in recommendation research. However, most reward functions are manually designed; they are either too unrealistic or imprecise to reflect the variety, dimensionality, and non-linearity of the recommendation problem. This impedes the agent from learning an optimal policy in highly dynamic online recommendation scenarios. To address the above issue, we propose a generative inverse reinforcement learning approach that avoids the need of defining an elaborative reward function. In particular, we model the recommendation problem as an automatic policy learning problem. We first generate policies based on observed users' preferences and then evaluate the learned policy by a measurement based on a discriminative actor-critic network. We conduct experiments on an online platform, VirtualTB, and demonstrate the feasibility and effectiveness of our proposed approach via comparisons with several state-of-the-art methods.

Deep learning frameworks generally require sufficient training data to generalize well while fail to adapt on small or few-shot datasets. Meta-learning offers an effective means of tackling few-shot scenarios and has drawn increasing attention in recent years. Meta-optimization aims to learn a shared set of parameters across tasks for meta-learning while facing challenges in determining whether an initialization condition can be generalized to tasks with diverse distributions. In this regard, we propose a meta-gradient boosting framework that can fit diverse distributions based on a base learner (which learns shared information across tasks) and a series of gradient-boosted modules (which capture task-specific information). We evaluate the model on several few-shot learning benchmarks and demonstrate the effectiveness of our model in modulating task-specific meta-learned priors and handling diverse distributions.

Blockchains have been applied in different domains to guarantee data integrity and provide a decentralized computational infrastructure for executing smart contracts. Multiple blockchain-related patterns have been summarized by academics and industry practitioners covering different aspects, such as engineering applications on top of a blockchain, structuring smart contracts, and security. The existence of these patterns is both helpful and challenging for designers. Helpful, as the existence of these patterns means that developers do not need to recreate solutions to common problems. Challenging, as the multitude of patterns leaves a designer confused about when to adopt or adapt patterns. In this paper, we propose a decision model that assists developers and architects in selecting appropriate patterns for blockchain-based applications. The selection is based on the characteristics of the use cases and trade-offs implicit in the patterns. We evaluated the proposed decision model based on expert opinion regarding its correctness and usefulness in guiding the architecture design and understanding the rationale of various design decisions.

Developing machine learning (ML) systems, just like any other system, requires architecture thinking. However, there are characteristics of ML components that create challenges and unique quality attribute (QA) concerns for software architecture and design activities, such as data-dependent behavior, detecting and responding to drift over time, and timely capture of ground truth to inform retraining. This paper presents four categories of software architecture challenges that need to be addressed to support ML system development, maintenance and evolution: software architecture practices for ML systems, architecture patterns and tactics for ML-important QAs, monitorability as a driving QA, and co-architecting and co-versioning. These challenges were collected from targeted workshops, practitioner interviews, and industry engagements. The goal of our work is to encourage further research in these areas and use the information presented in this paper to guide the development of empirically-validated practices for architecting ML systems.

Programming tutorials demonstrate programming tasks with code examples. However, our study of Stack Overflow questions reveals the low utilization of high-quality programming tutorials, which is caused task description mismatch and code information overload. Neither document search nor recently proposed activity-centric search can address these two barriers. In this work, we enrich the programming task knowledge graph with actions extracted from comments in code examples and more forms of activity sentences. To overcome the task description mismatch problem, we use code matching based task search method to find relevant programming tasks and code examples to the code under development. We integrate our knowledge graph and task search method in the IDE, and develop an observe-push based tool to prompt developers with API usage examples in explicit task contexts. To alleviate the code information overload problem, our tool highlights programming task and API information in the prompted tutorial excerpts and code examples based on the underlying knowledge graph. Our evaluation confirms the high quality of the constructed knowledge graph, and show that our code matching based task search can recommend effective code solutions to programming issues asked on Stack Overflow. Through an user study, we demonstrate that our tool is useful for assisting developers in finding and using relevant programming tutorials in their programming tasks.

Assigning bugs to the right components is the prerequisite to get the bugs analyzed and fixed. Classification-based techniques have been used in practice for assisting bug component assignments, for example, the BugBug tool developed by Mozilla. However, our study on 124,477 bugs in Mozilla products reveals that erroneous bug component assignments occur frequently and widely. Most errors are repeated errors and some errors are even misled by the BugBug tool. Our study reveals that complex component designs and misleading component names and bug report keywords confuse bug component assignment not only for bug reporters but also developers and even bug triaging tools. In this work, we propose a learning to rank framework that learns to assign components to bugs from correct, erroneous and irrelevant bug-component assignments in the history. To inform the learning, we construct a bug tossing knowledge graph which incorporates not only goal-oriented component tossing relationships but also rich information about component tossing community, component descriptions, and historical closed and tossed bugs, from which three categories and seven types of features for bug, component and bug-component relation can be derived. We evaluate our approach on a dataset of 98,587 closed bugs (including 29,100 tossed bugs) of 186 components in six Mozilla products. Our results show that our approach significantly improves bug component assignments for both tossed and non-tossed bugs over the BugBug tool and the BugBug tool enhanced with component tossing relationships, with >20% Top-k accuracies and >30% NDCG@k (k=1,3,5,10).

Application Programming Interfaces (APIs) typically come with usage constraints. The violations of these constraints (i.e. API misuses) can cause significant problems in software development. Existing methods mine frequent API usage patterns from codebase to detect API misuses. They make a naive assumption that API usage that deviates from the most-frequent API usage is a misuse. However, there is a big knowledge gap between API usage patterns and API usage constraints in terms of comprehensiveness, explainability and best practices. Inspired by this, we propose a novel approach named KGAMD (API-Misuse Detector Driven by Fine-Grained API-Constraint Knowledge Graph) that detects API misuses directly against the API constraint knowledge, rather than API usage pat-terns. We first construct a novel API-constraint knowledge graph from API reference documentation with open information extraction methods. This knowledge graph explicitly models two types of API-constraint relations (call-order and condition-checking) and enriches return and throw relations with return conditions and exception triggers. Then, we develop the KGAMD tool that utilizes the knowledge graph to detect API misuses. There are three types of frequent API misuses we can detect - missing calls, missing condition checking and missing exception handling, while existing detectors mostly focus on only missing calls. Our quantitative evaluation and user study demonstrate that our KGAMD is promising in helping developers avoid and debug API misuses

Multi-party systems are important for business processes but can be complex. Blockchain facilitates trust in multi-party systems by providing transparency, decentralised control, and immutable transaction history, to improve security and accountability between parties. The use of cryptographic hashes and the continual validation of the shared ledger in a blockchain system provides parties with data integrity for historical transactions and process integrity for smart contract execution. However, in the design of a broader system combining a blockchain with off-chain components, it is not always clear how system-level integrity is supported. This paper proposes two modelling schemes to better understand blockchain's support for integrity in multi-party blockchain-based systems. The schemes model interactions between components in an architecture as security protocols, for analysis by standard techniques and tools. We first illustrate how blockchain-based systems can be abstractly modelled directly as security protocols. Then we show how blockchain-specific issues such as consensus-based ‘forking’ (also known as ‘orphan blocks’ or ‘uncle blocks’) can be encoded. This allows transaction reordering behaviour to manifest in the model, and allows design mitigation for that problem to be checked. We illustrate our approach with analyses of three design alternatives for possible enhancements to a multi-party system for sharing trade certificates.

Device failure detection is one of most essential problems in Industrial Internet of Things (IIoT). However, in conventional IIoT device failure detection, client devices need to upload raw data to the central server for model training, which might lead to disclosure of sensitive business data. Therefore, in this article, to ensure client data privacy, we propose a blockchain-based federated learning approach for device failure detection in IIoT. First, we present a platform architecture of blockchain-based federated learning systems for failure detection in IIoT, which enables verifiable integrity of client data. In the architecture, each client periodically creates a Merkle tree in which each leaf node represents a client data record, and stores the tree root on a blockchain. Furthermore, to address the data heterogeneity issue in IIoT failure detection, we propose a novel centroid distance weighted federated averaging (CDW_FedAvg) algorithm taking into account the distance between positive class and negative class of each client data set. In addition, to motivate clients to participate in federated learning, a smart contact-based incentive mechanism is designed depending on the size and the centroid distance of client data used in local model training. A prototype of the proposed architecture is implemented with our industry partner, and evaluated in terms of feasibility, accuracy, and performance. The results show that the approach is feasible, and has satisfactory accuracy and performance.

Abstract Blockchain has attracted broad interests to build decentralised applications. A typical class of applications uses blockchain for the management of cross‐organisational business processes as well as assets. However, developing such applications without introducing vulnerabilities is hard for developers, not the least because the deployed code is immutable and can be called by anyone with access to the network. Model‐driven engineering (MDE) helps to reduce those risks, by combining proven code snippets as per the model specification, which is easier to understand than source code. Therefore, in this article, we present an approach for integrated MDE across business processes and asset management (e.g., for settlement). Our approach includes methods for fungible/non‐fungible asset registration, escrow for conditional payment, and asset swap. The proposed MDE approach is implemented in a smart contract generation tool called Lorikeet, and evaluated in terms of feasibility, functional correctness, and cost effectiveness.

According to the World Health Organization(WHO), it is estimated that approximately 1.3 billion people live with some forms of vision impairment globally, of whom 36 million are blind. Due to their disability, engaging these minority into the society is a challenging problem. The recent rise of smart mobile phones provides a new solution by enabling blind users' convenient access to the information and service for understanding the world. Users with vision impairment can adopt the screen reader embedded in the mobile operating systems to read the content of each screen within the app, and use gestures to interact with the phone. However, the prerequisite of using screen readers is that developers have to add natural-language labels to the image-based components when they are developing the app. Unfortunately, more than 77% apps have issues of missing labels, according to our analysis of 10,408 Android apps. Most of these issues are caused by developers' lack of awareness and knowledge in considering the minority. And even if developers want to add the labels to UI components, they may not come up with concise and clear description as most of them are of no visual issues. To overcome these challenges, we develop a deep-learning based model, called LabelDroid, to automatically predict the labels of image-based buttons by learning from large-scale commercial apps in Google Play. The experimental results show that our model can make accurate predictions and the generated labels are of higher quality than that from real Android developers.

GUI animations, such as card movement, menu slide in/out, snackbar display, provide appealing user experience and enhance the usability of mobile applications. These GUI animations should not violate the platform's UI design guidelines (referred to as design-don't guideline in this work) regarding component motion and interaction, content appearing and disappearing, and elevation and shadow changes. However, none of existing static code analysis, functional GUI testing and GUI image comparison techniques can "see" the GUI animations on the scree, and thus they cannot support the linting of GUI animations against design-don't guidelines. In this work, we formulate this GUI animation linting problem as a multi-class screencast classification task, but we do not have sufficient labeled GUI animations to train the classifier. Instead, we propose an unsupervised, computer-vision based adversarial autoencoder to solve this linting problem. Our autoencoder learns to group similar GUI animations by "seeing" lots of unlabeled real-application GUI animations and learning to generate them. As the first work of its kind, we build the datasets of synthetic and real-world GUI animations. Through experiments on these datasets, we systematically investigate the learning capability of our model and its effectiveness and practicality for linting GUI animations, and identify the challenges in this linting problem for future work.

API misuses cause significant problem in software development. Existing methods detect API misuses against frequent API usage patterns mined from codebase. They make a naive assumption that API usage that deviates from the most-frequent API usage is a misuse. However, there is a big knowledge gap between API usage patterns and API usage caveats in terms of comprehensiveness, explainability and best practices. In this work, we propose a novel approach that detects API misuses directly against the API caveat knowledge, rather than API usage patterns. We develop open information extraction methods to construct a novel API-constraint knowledge graph from API reference documentation. This knowledge graph explicitly models two types of API-constraint relations (call-order and condition-checking) and enriches return and throw relations with return conditions and exception triggers. It empowers the detection of three types of frequent API misuses - missing calls, missing condition checking and missing exception handling, while existing detectors mostly focus on only missing calls. As a proof-of-concept, we apply our approach to Java SDK API Specification. Our evaluation confirms the high accuracy of the extracted API-constraint relations. Our knowledge-driven API misuse detector achieves 0.60 (68/113) precision and 0.28 (68/239) recall for detecting Java API misuses in the API misuse benchmark MuBench. This performance is significantly higher than that of existing pattern-based API misused detectors. A pilot user study with 12 developers shows that our knowledge-driven API misuse detection is very promising in helping developers avoid API misuses and debug the bugs caused by API misuses.

A common challenge for most current recommender systems is the cold-start problem. Due to the lack of user-item interactions, the fine-tuned recommender systems are unable to handle situations with new users or new items. Recently, some works introduce the meta-optimization idea into the recommendation scenarios, i.e. predicting the user preference by only a few of past interacted items. The core idea is learning a global sharing initialization parameter for all users and then learning the local parameters for each user separately. However, most meta-learning based recommendation approaches adopt model-agnostic meta-learning for parameter initialization, where the global sharing parameter may lead the model into local optima for some users. In this paper, we design two memory matrices that can store task-specific memories and feature-specific memories. Specifically, the feature-specific memories are used to guide the model with personalized parameter initialization, while the task-specific memories are used to guide the model fast predicting the user preference. And we adopt a meta-optimization approach for optimizing the proposed method. We test the model on two widely used recommendation datasets and consider four cold-start situations. The experimental results show the effectiveness of the proposed methods.

Detecting Graphical User Interface (GUI) elements in GUI images is a domain-specific object detection task. It supports many software engineering tasks, such as GUI animation and testing, GUI search and code generation. Existing studies for GUI element detection directly borrow the mature methods from computer vision (CV) domain, including old fashioned ones that rely on traditional image processing features (e.g., canny edge, contours), and deep learning models that learn to detect from large-scale GUI data. Unfortunately, these CV methods are not originally designed with the awareness of the unique characteristics of GUIs and GUI elements and the high localization accuracy of the GUI element detection task. We conduct the first large-scale empirical study of seven representative GUI element detection methods on over 50k GUI images to understand the capabilities, limitations and effective designs of these methods. This study not only sheds the light on the technical challenges to be addressed but also informs the design of new GUI element detection methods. We accordingly design a new GUI-specific old-fashioned method for non-text GUI element detection which adopts a novel top-down coarse-to-fine strategy, and incorporate it with the mature deep learning model for GUI text detection.Our evaluation on 25,000 GUI images shows that our method significantly advances the start-of-the-art performance in GUI element detection.

Unlike all prior work, in this article, we investigate the notion of “unraveling metric vector spaces,” i.e., deriving meaning and low-rank structure from distance or metric space. Our new model bridges two commonly adopted paradigms for recommendations-metric learning approaches and factorization-based models, distinguishing itself accordingly. More concretely, we show that factorizing a metric vector space can be surprisingly efficacious. All in all, our proposed method, factorized metric learning, is highly effective for two classic recommendation tasks, possessing the potential of displacing many popular choices as an extremely strong baseline. We have done experiments on a number of real-world datasets, which show that our model performs better than recent state of the art largely on the rating prediction and item ranking tasks.

Integrated and collaborative manufacturing system develops as massive data are obtained by the Internet of Things (IoT) technology. However, the “trust tax” imposed on manufacturers during their countless collaborations with customers, suppliers, distributors, governments, service providers, and other manufacturers is very high. Blockchain is an emerging technology that can lead to more transparent, secure, and efficient transactions. It represents a new paradigm, as well as new thinking, of how data can be securely stored, integrated, and communicated among different stakeholders, organizations, and systems that unnecessarily trust each other. Blockchain is greatly useful for reducing the “trust tax,” especially beneficial for the small- and medium-sized enterprises that must tolerate much heavier trust tax than the established manufacturers. This paper investigates the blockchain-based security and trust mechanism and elaborates a particular application of blockchain for quality assurance, which is one of the strategic priorities of smart manufacturing. Data generated in a smart manufacturing process can be leveraged to retrieve material provenance, facilitate equipment management, increase transaction efficiency, and create a flexible pricing mechanism. The dairy industry is used to instantiate the value propositions of blockchain for quality assurance.

Rolling upgradeis a widely-used industry technique for updating software while a service provided by multiple instances of the software remains available. In cloud deployments of software, it is usual to implement the update step for rolling upgrade by replacing entire virtual machine instances. During the process of rolling upgrade, various failures may occur due to the complexity of software stack and the uncertainties of cloud platforms. Instance health checking and replacement are standard functionalities in most cloud infrastructures, though these create uncertainty in the duration of the whole upgrade procedure. In contrast, software and configuration errors are not usually detected by infrastructure functionalities, and if these happen, the entire rolling upgrade normally is unsuccessful and the system is left in an unsuitable state. In this paper, we propose an approach, named R2C, which innovates the stat of the art with our early error detection and predictability to increase the robustness of rolling upgrade on cloud platforms. We evaluate our techniques through real life testing in Amazon Web Service (AWS) and through a simulation.

Blockchain is an emerging technology for decentralized and transactional data sharing across a large network of untrusted participants. It enables new forms of distributed software architectures, where agreement on shared states can be established without trusting a central integration point. As a database and computational platform, blockchain has both advantages and disadvantages compared with conventional techniques. Blockchain may be an appropriate choice for some use cases while conventional technologies will be more appropriate for other use cases. A major difficulty for practitioners to decide whether or not to use blockchain is that limited product data or reliable technology evaluation available to assess the suitability of blockchains. In this paper, we propose an evaluation framework that comprises a list of criteria and a typical process for practitioners to assess the suitability of applying blockchain using these criteria based on the characteristics of the use cases. We then use several existing industrial trails to evaluate the feasibility of our framework.

Blockchain is of rising importance as a technology for engineering applications in cross-organizational settings, avoiding reliance on central trusted third-parties. The use of blockchain, instead of traditional databases or services, is an architectural choice in the development of a software system. The costs of execution and storage are important non-functional qualities, but as yet very little has been done to study them for blockchain-based systems. We investigate the cost of using blockchain using business process execution as a lens. Specifically, we compare the cost for computation and storage of business process execution on blockchain vs. a popular cloud service. First, we capture the cost models for both alternatives. Second, we implemented and measured the cost of business process execution on blockchain and cloud services for an example business process model from the literature. We observe two orders of magnitude difference in this cost.

Blockchain is an emerging technology for decentralised and transactional data sharing across a large network of untrusted participants. It enables new forms of distributed software architectures, where agreement on shared states can be established without trusting a central integration point. A major difficulty for architects designing applications based on blockchain is that the technology has many configurations and variants. Since blockchains are at an early stage, there is little product data or reliable technology evaluation available to compare different blockchains. In this paper, we propose how to classify and compare blockchains and blockchain-based systems to assist with the design and assessment of their impact on software architectures. Our taxonomy captures major architectural characteristics of blockchains and the impact of their principal design decisions. This taxonomy is intended to help with important architectural considerations about the performance and quality attributes of blockchain-based systems.

It is a challenge to ensure data integrity for cloud-based Internet of Things (IoT) applications because of the inherently dynamic nature of IoT data. The available frameworks of data integrity verification with public auditability cannot avoid the Third Party Auditors (TPAs). However, in a dynamic environment, such as the IoT, the reliability of the TPA-based frameworks is far from being satisfactory. In this paper, we propose a blockchain-based framework for Data Integrity Service. Under such framework, a more reliable data integrity verification can be provided for both the Data Owners and the Data Consumers, without relying on any Third Party Auditor (TPA). In this paper, the relevant protocols and a subsequent prototype system, which is implemented to evaluate the feasibility of our proposals, are presented. The performance evaluation of the implemented prototype system is conducted, and the test results are discussed. The work lays a foundation for our future work on dynamic data integrity verification in a fully decentralized environment.

Collaborative filtering (CF) has been successfully used to provide users with personalized products and services. However, dealing with the increasing sparseness of user-item matrix still remains a challenge. To tackle such issue, hybrid CF such as combining with content based filtering and leveraging side information of users and items has been extensively studied to enhance performance. However, most of these approaches depend on hand-crafted feature engineering, which is usually noise-prone and biased by different feature extraction and selection schemes. In this paper, we propose a new hybrid model by generalizing contractive auto-encoder paradigm into matrix factorization framework with good scalability and computational efficiency, which jointly models content information as representations of effectiveness and compactness, and leverage implicit user feedback to make accurate recommendations. Extensive experiments conducted over three large-scale real datasets indicate the proposed approach outperforms the compared methods for item recommendation.

Cloud-based systems get changed more frequently than traditional systems. These frequent changes involve sporadic operations such as installation and upgrade. Sporadic operations may fail due to the uncertainty of cloud platforms. Each sporadic operation manipulates a number of cloud resources. The accessibility of resources manipulated makes it possible to build an accurate process model of the correct behavior for an operation and its desired effects. This paper proposes a non-intrusive recovery approach for sporadic operations on cloud, called POD-Recovery. POD-Recovery utilizes the above-mentioned process model of the operation. When needed, it triggers recovery actions based on the model through non-intrusive means, i.e., without modifying the code which implements the sporadic operation. POD-Recovery employs an efficient artificial intelligence (AI) planning technique for generating recovery plans. We implement POD-Recovery and evaluate it by recovering from faults injected into 920 runs of five representative sporadic operations.

Blockchain is an emerging technology for decentralized and transactional data sharing across a large network of untrusted participants. It enables new forms of distributed software architectures, where components can find agreements on their shared states without trusting a central integration point or any particular participating components. Considering the blockchain as a software connector helps make explicitly important architectural considerations on the resulting performance and quality attributes (for example, security, privacy, scalability and sustainability) of the system. Based on our experience in several projects using blockchain, in this paper we provide rationales to support the architectural decision on whether to employ a decentralized blockchain as opposed to other software solutions, like traditional shared data storage. Additionally, we explore specific implications of using the blockchain as a software connector including design trade-offs regarding quality attributes.

Cloud system management is complex due to their diversity and frequent runtime changes. Cloud systems were previously managed through cloud specific management tools that focus on optimising technical metrics, such as performance. However, business users care business metrics (such as cost and revenue) more than technical metrics. To address these issues, this paper proposes a unified business-driven cloud management framework, which enables optimisation of business metrics without limiting business to a specific cloud provider. The main contributions include: (1) a taxonomy which defines a set of actions, events and metrics for unified cloud management; (2) a cloud management policy language that specifies cloud management policies from a business perspective; and (3) middleware architecture that allows business-driven management of diverse clouds. The proposed solutions are evaluated in terms of feasibility, functional correctness, generality, usefulness, and performance.

Understanding the behavior of an operations process and capturing it as an abstract process model has been shown to improve dependability significantly [1]. In particular, process context can be used for error detection, diagnosis, and even automated recovery. Creating the process model is an essential step in determining process context and, consequently, improving dependability. This paper describes two systems. The first, POD-Discovery, simplifies the creation of such an abstract process model from operations logs. An activity that previously required many manual steps can now be done largely automatically and in minutes. Using the discovered model, the second system, POD-Viz, provides operators with the ability to visualize the current state of an operations process in near-real-time and to replay a set of events to understand how the process context changed over time. This allows operators to trace the progress of an operations process easily, and helps in analyzing encountered errors.

Applications in the cloud are subject to sporadic changes due to operational activities such as upgrade, redeployment, and on-demand scaling. These operations are also subject to interferences from other simultaneous operations. Increasing the dependability of these sporadic operations is non-trivial, particularly since traditional anomaly-detection-based diagnosis techniques are less effective during sporadic operation periods. A wide range of legitimate changes confound anomaly diagnosis and make baseline establishment for "normal" operation difficult. The increasing frequency of these sporadic operations (e.g. due to continuous deployment) is exacerbating the problem. Diagnosing failures during sporadic operations relies heavily on logs, while log analysis challenges stemming from noisy, inconsistent and voluminous logs from multiple sources remain largely unsolved. In this paper, we propose Process Oriented Dependability (POD)-Diagnosis, an approach that explicitly models these sporadic operations as processes. These models allow us to (i) determine orderly execution of the process, and (ii) use the process context to filter logs, trigger assertion evaluations, visit fault trees and perform on-demand assertion evaluation for online error diagnosis and root cause analysis. We evaluated the approach on rolling upgrade operations in Amazon Web Services (A WS) while performing other simultaneous operations. During our evaluation, we correctly detected all of the 160 injected faults, as well as 46 interferences caused by concurrent operations. We did this with 91.95% precision. Of the correctly detected faults, the accuracy rate of error diagnosis is 96.55%.

Consumer-initiated sporadic operations on cloud applications, such as deployment, upgrade and reconfiguration, may fail because of the inherent uncertainty of operating in a cloud environment. For example, if a VM is not able to start, operations dependent on that VM will fail. In this paper we propose an approach for analyzing sporadic operations on cloud applications to facilitate recovery. We do this by first creating a process model of the sporadic cloud operation. The Process may not be suitable for recovery operations or the application's deployment architecture since an individual step may be at the wrong level of granularity or a failure may unnecessarily affect multiple steps. We then propose a set of process division criteria for revising an operation process to support recovery. This set of criteria contains four aspects: 1) Atomicity to support the imposition of transactions on portions of the process, 2) Idempotence to allow for re-execution of a failed portion, 3) Granularity to allow reuse of existing steps, 4) Recovery Actions Identifiable to allow proper recovery actions to be taken to recover from the errors occurring during the operation. We demonstrate the feasibility of imposing these criteria by using the rolling upgrade operation in Asgard - a popular cloud management tool for AWS EC2.

Deployment choices are critical in determining the availability of applications running in a cloud. But choosing good deployment for various software application components into virtual machines is a challenging task because of potential sharing of components among applications and potential interference from multi-tenancy. This paper presents an approach for improving the availability guarantee of software applications by optimizing the availability, performance and monetary cost trade-offs of different deployment choices. Our approach explicitly considers different classes of application requests during the decision process. The results of our experimental evaluation show that the approach can effectively improve the availability guarantees with little or negligible increase in the performance and monetary cost of the deployment choice.

Cloud consumers have a variety of deployment related techniques, such as auto-scaling policies and recovery strategies, for dealing with the uncertainties in the cloud. Uncertainties can be characterized as stochastic (such as failures, disasters, and workload spikes) and subjective (such as choice among various deployment options). Cloud consumers must consider both stochastic and subjective uncertainties. Analytic support for consumers in selecting appropriate techniques and setting the required parameters in the face of different types of uncertainty is currently limited. In this paper, we propose a set of application availability analysis models that capture subjective uncertainties in addition to stochastic uncertainties. We built and validated the models by using industry best practices on deployment, and actual commercial products for disaster recovery and live migration. Our results show that the models permit more informed and quantitative availability analysis than industry best practices under a wide range of scenarios.

A large number of cloud application failures happen during sporadic operations on cloud applications, such as upgrade, deployment reconfiguration, migration and scaling-out/in. Most of them are caused by operator and process errors [1]. From a cloud consumer's perspective, recovery from these failures relies on the limited control and visibility provided by the cloud providers. In addition, a large-scale system often has multiple operation processes happening simultaneously, which exacerbates the problem during error diagnosis and recovery. Existing built-in or infrastructure-based recovery mechanisms often assume random component failures and use checkpoint-based rollback, compensation actions [2], redundancy and rejuvenation to handle recovery [3]. These recovery mechanisms do not consider the characteristics of a specific operation process that consists of a set of steps carried out by scripts and humans interacting with fragile cloud infrastructure APIs and uncertain resources [4]. Other approaches such as FATE/DESTINI [5] look at the process implied by a system's internal protocols and rely on the built-in recovery protocol to detect and recover from bugs. The problem we target is at a different level related to the external sporadic activities operating on a hosted cloud application.

Application dependability issues depend on increasingly sophisticated activities during operation time for deployment, upgrade, scaling out/in and reactions to various failures. Traditional approaches to improving application dependability focus on artifact-oriented troubleshooting and improvements. In this paper, we present an approach using process models to represent and analyze operations with considerations of exception handlings and fault-proneness. Our goal is to reduce diagnosis and repair time for application failures that occur during operation activities such as deployment and upgrade.

The emergence of web services gives rise to service ecosystems, where core service platforms are exposed through APIs to a large number of third parties to provide new solutions. This often requires platform APIs to be able to evolve rapidly while not breaking backward compatibility in order to support a wider range of planned and unplanned use. The REST architecture style allows a service contract to be negotiated and delivered at runtime aiming to achieve higher level of evolvability. However this potential is often not fully realized due to the lack of systematic approach in architecture design to separate requirement-related contract elements from specification-related elements and integrate them into the overall API design. In this paper, we present a novel approach to designing service platform APIs in order to achieve high evolvability. We firstly apply the problem frames approach to better understand the interface coupling and the evolution of requirements. Based on the problem frame analysis, we propose a media-type centric approach to designing RESTful service API. We applied our approach to the Twitter API evolution. The result shows significant improvements in the areas of contract leaking prevention, interface consistency and adding new features.

REpresentational State Transfer (REST) is the set of design principles behind the World Wide Web (WWW). REST treats all entities in the world as link-connected resources, and supports a resource-oriented architecture (ROA) for the design of applications. REST and ROA are responsible for many of the desirable quality attributes achieved in the WWW, such as loose-coupling (better adaptability) and interoperability. However, many exiting Web-based or service-oriented applications (WSDL/SOAP-based) only use WWW/HTTP as a tunneling protocol or abuse URL and POX (Plain Old XML) by encoding method semantics in them. These applications use fine-grained remote procedure calls (RPC), breaking REST/ROA principles. We observe two kinds of challenges: 1) conceptually modelling process-intensive applications using a ROA promoted by the REST principles; and 2) practically decomposing a workflow-based business process into distributed, dynamic and RESTful process fragments. In this paper, we propose a ROA for business processes following the RESTful principles. We evaluate our approach by comparing it with current SOAP/WSDL/BPEL-driven approaches in terms of feasibility, process visibility, interoperability, and adaptability.