Skin-toned emojis are crucial for fostering personal identity and social inclusion in online communication. As AI models, particularly Large Language Models (LLMs), increasingly mediate interactions on web platforms, the risk that these systems perpetuate societal biases through their representation of such symbols is a significant concern. This paper presents the first large-scale comparative study of bias in skin-toned emoji representations across two distinct model classes. We systematically evaluate dedicated emoji embedding models (emoji2vec, emoji-sw2v) against four modern LLMs (Llama, Gemma, Qwen, and Mistral). Our analysis first reveals a critical performance gap: while LLMs demonstrate robust support for skin tone modifiers, widely-used specialized emoji models exhibit severe deficiencies. More importantly, a multi-faceted investigation into semantic consistency, representational similarity, sentiment polarity, and core biases uncovers systemic disparities. We find evidence of skewed sentiment and inconsistent meanings associated with emojis across different skin tones, highlighting latent biases within these foundational models. Our findings underscore the urgent need for developers and platforms to audit and mitigate these representational harms, ensuring that AI's role on the web promotes genuine equity rather than reinforcing societal biases.

Connected autonomous vehicles (CAVs) rely on object detection models to ensure safe and efficient navigation. Traditional centralized training approaches pose challenges related to data privacy, scalability, and communication overhead. In this study, we integrate Federated Learning (FL) with YOLO models, including YOLOv5, YOLOv8, and YOLOv11, for object detection in CAVs, utilizing the Flower framework to enable decentralized training while preserving data privacy. We design a virtual client setup that replicates a realistic scenario and apply FedAvg and FedProx aggregation strategies on the KITTI and BDD100K datasets. Our experimental results demonstrate that FL-based training outperforms traditional centralized learning, with YOLOv8 achieving a mean average precision (mAP) of 87.9% in KITTI and 61.5% in BDD100K, outperforming the baseline. Our study highlights the feasibility and effectiveness of deploying FL-based object detection models in CAVs, by conducting a comprehensive evaluation of using the Flower federated learning framework and addressing privacy concerns through decentralized training.

In this paper, we present a novel benchmark, GSOT3D, that aims at facilitating development of generic 3D single object tracking (SOT) in the wild. Specifically, GSOT3D offers 620 sequences with 123K frames, and covers a wide selection of 54 object categories. Each sequence is offered with multiple modalities, including the point cloud (PC), RGB image, and depth. This allows GSOT3D to support various 3D tracking tasks, such as single-modal 3D SOT on PC and multi-modal 3D SOT on RGB-PC or RGB-D, and thus greatly broadens research directions for 3D object tracking. To provide highquality per-frame 3D annotations, all sequences are labeled manually with multiple rounds of meticulous inspection and refinement. To our best knowledge, GSOT3D is the largest benchmark dedicated to various generic 3D object tracking tasks. To understand how existing 3D trackers perform and to provide comparisons for future research on GSOT3D, we assess eight representative point cloud-based tracking models. Our evaluation results exhibit that these models heavily degrade on GSOT3D, and more efforts are required for robust and generic 3D object tracking. Besides, to encourage future research, we present a simple yet effective generic 3D tracker, named PROT3D, that localizes the target object via a progressive spatial-temporal network and outperforms all current solutions by a large margin. By releasing GSOT3D, we expect to advance further 3D tracking in future research and applications. Our benchmark and model as well as the evaluation results will be publicly released at our webpage https://github.com/ailovejinx/GSOT3D.

The surge in scientific submissions has placed increasing strain on the traditional peer-review process, prompting the exploration of large language models (LLMs) for automated review generation. While LLMs demonstrate competence in producing structured and coherent feedback, their capacity for critical reasoning, contextual grounding, and quality sensitivity remains limited. To systematically evaluate these aspects, we propose a comprehensive evaluation framework that integrates semantic similarity analysis and structured knowledge graph metrics to assess LLM-generated reviews against human-written counterparts. We construct a large-scale benchmark of 1,683 papers and 6,495 expert reviews from ICLR and NeurIPS in multiple years, and generate reviews using five LLMs. Our findings show that LLMs perform well in descriptive and affirmational content, capturing the main contributions and methodologies of the original work, with GPT-4o highlighted as an illustrative example, generating 15.74% more entities than human reviewers in the strengths section of good papers in ICLR 2025. However, they consistently underperform in identifying weaknesses, raising substantive questions, and adjusting feedback based on paper quality. GPT-4o produces 59.42% fewer entities than real reviewers in the weaknesses and increases node count by only 5.7% from good to weak papers, compared to 50% in human reviews. Similar trends are observed across all conferences, years, and models, providing empirical foundations for understanding the merits and defects of LLM-generated reviews and informing the development of future LLM-assisted reviewing tools. Data, code, and more detailed results are publicly available at https://github.com/RichardLRC/Peer-Review.

Recent years have witnessed growing concerns over demographic biases in image-centric applications, including image search engines and generative systems. While the advent of generative AI offers a pathway to mitigate these biases by producing underrepresented images, existing solutions still fail to precisely generate images that reflect specified demographic distributions. In this paper, we propose PreciseDebias, a comprehensive end-to-end framework that can rectify demographic bias in image generation. By leveraging fine-tuned Large Language Models (LLMs) coupled with text-to-image generative models, PreciseDebias transforms generic text prompts to produce images in line with specified demographic distributions. The core component of PreciseDebias is our novel instruction-following LLM, meticulously designed with an emphasis on model bias assessment and balanced model training. Extensive experiments demonstrate the effectiveness of PreciseDebias in rectifying biases pertaining to both ethnicity and gender in images. Furthermore, when compared with two baselines, PreciseDebias illustrates its robustness and capability to capture demographic intricacies. The generalization of PreciseDebias is further illuminated by the diverse images it produces across multiple professions and demographic attributes. To ensure reproducibility, we will make PreciseDebias openly accessible to the broader research community by releasing all models and code.

Since state-of-the-art machine learning models like Transformers can not handle long text well, the quality of the summarization of legal documents is still not desirable. In order to improve the ability of machine learning models to understand the context of a long document, we introduce the keywords into the models to guide the summarization to locate and capture key information from long documents such as legal cases. Different from other works leveraging keywords to enhance the model, we further investigate how keyword quality impacts summarization. To improve the performance of the summarization, we also explore different methods for effectively encoding exceptionally lengthy documents and models for keyword extraction. The experiment results demonstrated that the keywords-based augmentation method is effective for summarization and higher-quality keywords can enhance the summarization models.

Several recently developed neural network models have shown their potential for automated text summarization. However, the evaluation results of these models on summarization of long text are fairly close in almost every major evaluation parameter. None of these models including large language models GPT-3.5 and GPT-4 can well summarize long text without manual interventions. In this paper, we report the evaluation results of several state-of-the-art neural network models on text summarization under different configurations of the input, which includes 1630 U.S. patent documents. Based on the evaluation results, we proposed a strategy for improving the text summarization in long text and demonstrated its effectiveness with new cases.

A high-quality corpus is essential for building an effective legal intelligence system. The quality of a corpus includes both the quality of original data and the quality of its corresponding labeling. The major quality dimensions of a legal corpus include comprehensiveness, freshness, and correctness. However, building a comprehensive, correct, and fresh legal corpus is a grand challenge. In this article, we propose a semiautomated machine learning framework to address the challenge. We first created an initial corpus with 4937 instances that were manually labeled. Several strategies were implemented to assure its quality. The initial results showed that class imbalance and insufficiency of training data are the two major quality issues that negatively impacted the quality of the system that was built on the data. We experimented and compared three class-imbalance-handling techniques and found that the mixed-sampling method, which combines upsampling and downsampling, was the most effective way to address the issue. In order to address the insufficiency of training data, we experimented several machine learning methods for automated data augmentation including pseudolabeling, co-training, expectation-maximization, and generative adversarial network (GAN). The results showed that GAN with deep learning models achieved the best performance. Finally, ensemble learning of different classifiers was proposed and experimented with for the construction of a legal corpus, which achieves higher quality in comprehensiveness, freshness, and correctness compared to existing work. The semiautomated machine learning framework and the data quality evaluation method developed in this research can be used for data augmentation and quality evaluation of a large dataset as well as a reference for the selection of machine learning methods for data augmentation and generation. The machine learning models, the training data, and the legal corpus are published and publicly accessible at [Online]. Available:https://github.com/haihua0913/legalArgumentmining.

Big data validation and system verification are crucial for ensuring the quality of big data applications. However, a rigorous technique for such tasks is yet to emerge. During the past decade, we have developed a big data system called CMA for investigating the classification of biological cells based on cell morphology which is captured in diffraction images. CMA includes a collection of scientific software tools, machine learning algorithms, and a large-scale cell image repository. In order to ensure the quality of big data system CMA, we developed a framework for rigorously validating the massive scale image data as well as adequately verifying both the software tools and machine learning algorithms. The validation of big data is conducted by iteratively selecting the data using a machine learning approach. An experimental approach guided by a feature selection algorithm is introduced in the framework to select an optimal feature set for improving the machine learning performance. The verification of software and algorithms is developed on the iterative metamorphic testing approach due to the non-testable property of the software and algorithms. A machine learning approach is introduced for developing test oracles iteratively to ensure the adequacy of the test coverage criteria. Performance of the machine learning algorithm is evaluated with a stratified N-fold cross validation and confusion matrix. We describe the design of the proposed big data verification and validation framework with CMA as the case study, and demonstrate its effectiveness through verifying and validating the dataset, the software and the algorithms in CMA.

Poor data quality has a direct impact on the performance of the machine learning system that is built on the data. As a demonstrated effective approach for data quality improvement, transfer learning has been widely used to improve machine learning quality. However, the “quality improvement” brought by transfer learning was rarely rigorously validated, and some of the quality improvement results were misleading. This article first exposed the hidden quality problem in the datasets used to build a machine learning system for normalizing medical concepts in social media text. The system was claimed to have achieved the best performance compared to existing work on a machine learning task. However, the results of our experiments showed that the “best performance” was due to the poor quality of the datasets and the defective validation process. To address the data quality issue and build a high-performance medical concept normalization system, we developed a transfer-learning-based strategy for data quality enhancement and system performance improvement. The results of the experiments showed a strong correlation between the quality of the datasets and the performance of the machine learning system. The results also demonstrated that a rigorous evaluation of data quality is necessary for guiding the quality improvement of machine learning. Therefore, we propose a data quality evaluation framework that includes the quality criteria and their corresponding evaluation approaches. The data validation process, the performance improvement strategy, and the data quality evaluation framework discussed in this article can be used for machine learning researchers and practitioners to build high-performance machine learning systems.

Knowledge graph has become an essential tool for semantic analysis with the development of natural language processing and deep learning. A high-quality knowledge graph is handy for building a high-performance knowledge-driven application. Despite recent advances in information extraction (IE) techniques, no suitable automated methods can be applied to constructing a domain-specific, comprehensive, and high-quality knowledge graph. However, a semi-automatic strategy, which can ensure the basic quality requirements of a knowledge graph, has been successfully implemented in the elementary science domain. This paper presents a semantic annotation system developed for building a high-quality legal knowledge graph (SALKG) using the semi-automatic strategy. We introduce its system design, architecture, algorithms, functions, and implementation. To investigate the effectiveness of SALKG, we conduct a preliminary annotation experiment with 280 legal texts which were collected from the Harvard Caselaw Access Project. The user evaluation from 32 graduate students demonstrates the high usability of SALKG in semantic annotation and the potential for building a high-quality legal knowledge graph. The system can also be adapted to other fields for constructing domain-specific knowledge graphs.

The poor quality of a dataset may produce low quality machine learning system. Therefore, transfer learning as a demonstrated effective approach for data quality improvement has been widely used for improving the quality of machine learning. However, the "quality improvement" brought by transfer learning in some studies was not rigorously validated or was even misleading. In this paper, we first investigate the quality problem of the datasets that were used for building a machine learning system. The system was claimed to have achieved the best performance comparing to existing work on a machine learning task. However, the "best performance" was due to the poor quality of the datasets as well as the incorrect validation process. Then we described an experimental study to demonstrate the effectiveness of transfer learning for improving the quality of datasets. However, the experiment results also show the quality improvement of transfer learning is not guaranteed, and a set of requirements have to be meet before applying the approach. Based on the investigation and experiment results, we propose a group of data quality criteria and evaluation approaches for quality improvement of machine learning. We investigated the research problem and explained the results through studying a machine learning system for normalizing medical concepts in social media text with open datasets.

Adequately testing scientific software is essential to the quality of the software. However, it is a grand challenge due to the oracle problem. Metamorphic testing has shown its effectiveness for alleviating the problem. But the effectiveness of metamorphic testing is highly dependent on the quality of metamorphic relations that are developed for testing the software. In this paper, we propose a framework for iteratively developing metamorphic relations for adequately testing scientific software. The basic idea is to refine metamorphic relations that are loosely defined to those that can be verified with only limited number of cases so that the relations can be accurately tested. We explain the framework through testing a scientific software system that is used for modeling light scattering of particles. Based on domain knowledge and general guidelines, a group of metamorphic relations are first identified and tested. According to testing results, the metamorphic relations are refined step by step until each of them can be accurately tested. In particular, an invariant transform is applied to the metamorphic relations for significantly reducing the number of cases that can satisfy the relations. Finally the relations are further transformed with a carefully defined hash function to ensure each of the metamorphic relations can be automatically verified. The proposed approach truly solves the oracle problem and greatly improves the effectiveness of metamorphic testing. Its effectiveness is evaluated by mutation testing and demonstrated by new problems found in the software.

During navigation, a pedestrian needs to recognize a landmark at a certain decision point. If a potential landmark located at a decision point is complicated to recognize, the complexity of the decision point is significantly increased. Thus, it is important to compute routes that avoid complicated decision points (CDPs) but still achieve optimal navigation performance. In this paper, we propose an approach for computing routes that avoid CDPs while optimizing the performance of landmark-based pedestrian navigation. The approach includes (1) a model for identifying CDPs based on the structures of pedestrian networks and landmark data in real scenes, and (2) a modified genetic algorithm for computing routes that avoid the identified CDPs and find the shortest route possible. To demonstrate the advantages and effectiveness of the proposed approach, we conducted an empirical study on the pedestrian network in a real-world scenario. The experimental results show that our approach can effectively avoid CDPs while still minimizing travel distance. Furthermore, our approach can provide the routes with the shortest travel distance if the distances of the routes without CDPs exceed a certain threshold.

There are basically two ways for improving the accuracy of machine learning: building relevant machine learning models, and providing high quality datasets for training the models. Significant efforts have been made in designing powerful machine learning models. Furthermore, many open-source datasets have been created for machine learning research. However, research on assessing the impact of the quality of a dataset on the accuracy of a machine learning system has not received attention. In this paper, we present an experimental study to show how the quality of datasets impact the accuracy of machine learning models. We discovered a common problem in datasets that could greatly impact the accuracy of machine learning. This problem could also exist in many other machine learning systems, especially those that are developed using crowd-sourced datasets. This problem is difficult to detect using traditional validation approaches. We propose a novel technique based on metamorphic testing for validating a machine learning system together with its training and testing data. The key to metamorphic testing is to create tests that will adequately test the system. We propose an approach for creating such tests. The effectiveness of the proposed approach is demonstrated through a case study of automated classification of biological cell images.

Deep learning is an important technique for extracting value from big data. However, the effectiveness of deep learning requires large volumes of high quality training data. In many cases, the size of training data is not large enough for effectively training a deep learning classifier. Data augmentation is a widely adopted approach for increasing the amount of training data. But the quality of the augmented data may be questionable. Therefore, a systematic evaluation of training data is critical. Furthermore, if the training data is noisy, it is necessary to separate out the noise data automatically. In this paper, we propose a deep learning classifier for automatically separating good training data from noisy data. To effectively train the deep learning classifier, the original training data need to be transformed to suit the input format of the classifier. Moreover, we investigate different data augmentation approaches to generate sufficient volume of training data from limited size original training data. We evaluated the quality of the training data through cross validation of the classification accuracy with different classification algorithms. We also check the pattern of each data item and compare the distributions of datasets. We demonstrate the effectiveness of the proposed approach through an experimental investigation of automated classification of massive biomedical images. Our approach is generic and is easily adaptable to other big data domains.

Detecting the groundwater runoff connectivity is important for mining and environment protection. However, traditional physical and chemical experiments based approaches are neither efficient nor effective. Experimental results have shown the bacterial community in an isolated well contains unique DNA sequences, and the bacterial communities in connected wells have common DNA sequences that are not expected in two isolated communities. In this paper, we applied a variety of clustering methods to the bacterial data of groundwater that were acquired from a group of wells, to obtain the distribution of the bacterial communities among the wells based on DNA sequences. In addition, we conducted a serials of experiments to show the distribution of the bacterial communities can indicate the groundwater runoff connectivity. The clustering results are consistent to the experimental results of the traditional physical and chemical experiments. The research shows the cluster analysis of the distribution of bacterial communities is an efficient and effective approach for detecting the groundwater runoff connectivity.

Metamorphic testing is an effective technique for testing "non-testable" programs. But the quality of metamorphic testing is highly depended on the selection of metamorphic relations and the test generation. This paper introduces an approach for iteratively developing metamorphic relations and producing adequate tests guided by testing and test evaluation results. The approach includes a framework for the development of metamorphic relations and tests, and a strategy for iteratively refining the relations and tests for generating adequate tests. The test adequacy evaluation is built on the evaluation of test coverage criteria, mutation testing, and testing of mutated metamorphic relations. The approach and its effectiveness are discussed through testing a Monte Carlo modeling program.

Absence of test oracles is the grand challenge for testing complex scientific software.Metamorphic testing is the novel technique for developing test oracles on metamorphic relations.Although it is easy to find metamorphic relations based on general guidelines and domain knowledge, the ones that can adequately test the software are difficult to be developed.This paper introduces a machine learning approach for iteratively developing metamorphic relations.The approach develops initial metamorphic relations and tests first, and then the relations and tests are refined through mining the initial test execution and evaluation results with machine learning algorithms.The approach and its effectiveness are illustrated through testing an open source discrete dipole approximation program.

Software architecture is the foundation for the development of software systems.Its correctness is important to the quality of the software systems that have been developed based on it.Formally modeling and analyzing software architecture is an effective way to ensure the correctness of software architecture.However, how to effectively verify software architecture and use the results from formal modeling and analysis is important to the application of the approach.In this paper, software architecture is modelled using high level Petri nets, and the model is then checked with a model based testing tool called MISTA, and bounded model checking tool Alloy to ensure the correctness of the model.The approach is designed as a two-phase process consisting of model-based testing and bounded model checking to ensure it is both practical and rigorous for analyzing software architecture.We illustrated the idea and procedure via modeling and analyzing the Publish-Subscribe architecture.The result has shown that combining bounded model checking with model based testing is an effective extension to ensure the development quality.

A high qualified software developer should have the ability to develop software systems following good software engineering practices. However, an integrated software engineering tool that can help students to learn the practices is absent. In this paper, we present an IDE that is able to monitor and guide students to develop software following good practices. In addition, the tool offers a set of guidelines for improving the learning process via analyzing learning activities and results.

At last year's conference, we organized a panel “Involving Undergraduates in Research: Motivations and Challenges” which was a great success with many interesting discussions. This has motivated us to develop a second iteration of this panel with an additional dimension - the Path Forward. We expect to have more discussions on how to extend what we have already learned from the past experiences to further enhance the way we involve undergraduates in doing research. Panelists will deliver a 5-minute overview of their research experiences with undergraduate students, including challenges they faced, lessons learned, and areas for improvement. The floor will then be opened for audience members to voice any concerns, questions, or comments. Student panel attendees will be given special consideration when presenting observations from their own perspective.

The North Carolina's first Master of Science in Software Engineering program was introduced at East C The North Carolina's first Master of Science in Software Engineering program was introduced at East Carolina University in spring 2008. In this paper we report on our progress in terms of successful student recruitment and retention and also course delivery methods in both face-to-face and online environments.

A mobile agent system is a special distributed system with moving programs in networks. Mobile agent systems provide a powerful and flexible paradigm for building high performance distributed systems. Due to dynamic configuration property, assuring quality of a mobile agent system is a challenge work. Formal specification and analysis of a mobile agent system provides one of the best approaches to ensure the correctness of a system design. However, it is difficult to find a formal specification tool for modeling a mobile agent system with an easy to understand and concise model. In addition, it is a challenge but also important work to provide an automatic formal analysis approach for verifying whether a system specification correctly meets certain requirements in a mobile agent system. In this paper, a framework for specification and analysis of mobile agent systems is defined. First, Predicate/Transition nets are extended with dynamic channels for modeling mobile agent systems. The formalism has the expressive power to naturally model the software architecture of a mobile agent system, and easily capture the properties especially the mobility, mobile communication and dynamic configuration of a mobile agent system. Then, model checking is instrumented to the framework for automatically verifying the correctness of the specification of a mobile agent system. In order to illustrate the capability of the formalism and the verification strategy, a medical image processing system using mobile agents is modeled using the extended Predicate/Transition nets and system properties are verified using the SPIN model checker.

The object-oriented (OO) paradigm provides several benefits during analysis and design of large-scale software systems, but scores lower in terms of testability. The low testability score for OO software is due mainly to the composition of OO systems exhibiting the characteristics of abstraction, encapsulation, genericity, inheritance, polymorphism, concurrency and exception handling. To address the difficulty of testing the features of a class, a plethora of implementation-based testing techniques (IBTTs) have been developed. However, no one IBTT has emerged as the preferred technique to test the implementation of a class. In this paper we present a technique that automatically identify those IBTTs that are most suitable for testing a class based on the characteristics of that class. Our approach uses a taxonomy of OO classes that is used to succinctly abstract the characteristics of a class under test (CUT). We have implemented a tool that automates the process of mapping IBTTs to a class. In addition to identifying the IBTTs that would be best suited for testing a class, our tool provides feedback to the tester facilitating the identification of the characteristics of the class that are not suitably tested by any of the IBTTs in the list. We provide results of a study supporting the notion that using more than on IBTT during testing improves test coverage of a CUT.

Mobile agents provide a powerful and flexible paradigm for the development of autonomic computing systems. However, due to the security concern, mobile agents are not popularly used for real-world systems. In this paper, we define a security framework that can effectively protect mobile agents and agent systems from intruder attacking. In the framework, a mobile agent finder, which is extended with a registration protocol, is used to authenticate and authorize agent systems, incoming messages, and agents. We formally model the secure mobile agent finder using predicate transition nets, and analyze the models using model checking tool Spin. The results help us to develop high confidence applications using mobile agents. In addition, the modeling and analysis approach can be easily extended to develop other complex software systems.

The process of agent migration is the major difference between logical code mobility of software agents and physical mobility of mobile nodes in ad hoc networks. Without considering agent transfer, it would make little sense to mention the modeling of strong code mobility, which aims to make a migrated agent restarted exactly from the state when it was stopped before migration. From the perspective of system's architecture, this paper proposes a two-layer approach for the formal modeling of logical agent mobility (LAM) using predicate/transition (PrT) nets. We view a mobile agent system as a set of agent spaces and agents could migrate from one space to another. Each agent space is explicitly abstracted to be a component, consisting of an environmental part and an internal connector dynamically binding agents with their environment. We use a system net, agent nets, and a connector net to model the environment, agents, and the connector, respectively. In particular, agent nets are packed up as parts of tokens in system nets, so that agent transfer and location change are naturally captured by transition firing (token game) in Petri nets. Agent nets themselves are active only at specific places and disabled at all the other places in a system net. The semantics of such a two-layer LAM model is defined by transforming it into a PrT net. This facilitates the analysis of several properties about location, state, and connection. In addition, this paper also presents a case study of modeling and analyzing an information retrieval system with mobile agents.

In the past decade, software architecture research has mainly focused on the concept formulation and the development of various architecture description languages. This field has matured enough and thus requires more emphasis on validation techniques. Symbolic model checking has been a highly successful automatic validation technique for hardware systems. We are interested in whether symbolic model checking can be effectively applied to software architecture validation. In this paper, we present our approach to apply the symbolic model checking technique to verify software architecture specifications written in SAM.