Image captioning (IC) systems aim to generate a text description of the salient objects in an image. In recent years, IC systems have been increasingly integrated into our daily lives, such as assistance for visually-impaired people and description generation in Microsoft Powerpoint. However, even the cutting-edge IC systems (e.g., Microsoft Azure Cognitive Services) and algorithms (e.g., OFA) could produce erroneous captions, leading to incorrect captioning of important objects, misunderstanding, and threats to personal safety. The existing testing approaches either fail to handle the complex form of IC system output (i.e., sentences in natural language) or generate unnatural images as test cases. To address these problems, we introduce Recursive Object MElting (ROME), a novel metamorphic testing approach for validating IC systems. Different from existing approaches that generate test cases by inserting objects, which easily make the generated images unnatural, ROME melts (i.e., remove and inpaint) objects. ROME assumes that the object set in the caption of an image includes the object set in the caption of a generated image after object melting. Given an image, ROME can recursively remove its objects to generate different pairs of images. We use ROME to test one widely-adopted image captioning API and four state-of-the-art (SOTA) algorithms. The results show that the test cases generated by ROME look much more natural than the SOTA IC testing approach and they achieve comparable naturalness to the original images. Meanwhile, by generating test pairs using 226 seed images, ROME reports a total of 9,121 erroneous issues with high precision (86.47%-92.17%). In addition, we further utilize the test cases generated by ROME to retrain the Oscar, which improves its performance across multiple evaluation metrics.

Graph representation learning has received increasing attention in recent years for many real-world applications. A major challenge in graph representation learning is the lack of labeled data. To address this challenge, Graph Neural Networks (GNNs) use message passing frameworks to combine information from unlabeled data with labeled data. However, the use of unlabeled data under the message passing framework is indirect in the training process where unlabeled data does not supervise the training process. To fully exploit the potential of unlabeled data, we propose a novel dual-view cooperative training framework for graph data where unlabeled data is involved in the training process for supervision. Specifically, we regard different views as the reasoning processes of two GNN models with which the models make predictions, integrating the understanding of different models on the underlying graph. To exchange information between models, we design a pseudo-label-based approach, where the two models mutually provide pseudo labels to each other iteratively. Moreover, to ensure the quality of pseudo labels, we propose an entropy-based pseudo-labels selection procedure and we adopt GNNExplainer to visualize different views in our framework. Our comprehensive experimental evaluation shows that our methods can boost the performance of state-of-the-art models.

A bipartite network is a network with two disjoint vertex sets and its edges only exist between vertices from different sets. It has received much interest since it can be used to model the relationship between two different sets of objects in many applications (e.g., the relationship between users and items in E-commerce). In this paper, we study the problem of efficient bi-triangle counting for a large bipartite network, where a bi-triangle is a cycle with three vertices from one vertex set and three vertices from another vertex set. Counting bi-triangles has found many real applications such as computing the transitivity coefficient and clustering coefficient for bipartite networks. To enable efficient bi-triangle counting, we first develop a baseline algorithm relying on the observation that each bi-triangle can be considered as the join of three wedges. Then, we propose a more sophisticated algorithm which regards a bi-triangle as the join of two super-wedges, where a wedge is a path with two edges while a super-wedge is a path with three edges. We further optimize the algorithm by ranking vertices according to their degrees. We have performed extensive experiments on both real and synthetic bipartite networks, where the largest one contains more than one billion edges, and the results show that the proposed solutions are up to five orders of magnitude faster than the baseline method.

Recently, the topic of truss computation has gained plenty of attention, where the k-truss of a graph is the maximum subgraph in which each edge participates in at least (k-2) triangles. Existing solutions mainly focus on homogeneous networks, where vertices are of the same type, and thus cannot be applied to heterogeneous information networks which consist of multi-typed and interconnected objects, such as the bibliographic networks and knowledge graphs. In this paper, we study the problem of truss computation over HINs, which aims to find groups of vertices that are of the same type and densely connected. To model the relationship between two vertices of the same type, we adopt the well-known concept of meta-path, which is a sequence of vertex types and edge types between two given vertex types. We then introduce two kinds of HIN triangles for three vertices, regarding a specific meta-path P. The first one requires that each pair of vertices is connected by an instance of P, while the second one also has such a connectivity constraint but further needs that the three instances of P form a circle. Based on these two kinds of triangles, we propose two HIN truss models respectively. We further develop efficient truss computation algorithms. We have performed extensive experiments on five real large HINs, and the results show that the proposed solutions are highly effective and efficient.

Recently, the topic of community search (CS) has gained plenty of attention. Given a query vertex, CS looks for a dense subgraph that contains it. Existing studies mainly focus on homogeneous graphs in which vertices are of the same type, and cannot be directly applied to heterogeneous information networks (HINs) that consist of multi-typed, interconnected objects, such as the bibliographic networks and knowledge graphs. In this paper, we study the problem of community search over large HINs; that is, given a query vertex q , find a community from an HIN containing q , in which all the vertices are with the same type of q and have close relationships. To model the relationship between two vertices of the same type, we adopt the well-known concept of meta-path , which is a sequence of relations defined between different types of vertices. We then measure the cohesiveness of the community by extending the classic minimum degree metric with a meta-path. We further propose efficient query algorithms for finding communities using these cohesiveness metrics. We have performed extensive experiments on five real large HINs, and the results show that the proposed solutions are effective for searching communities. Moreover, they are much faster than the baseline solutions.

The use of hand gestures provides an attractive alternative to cumbersome interface devices for human-computer interaction (HCI). However, the number of hand gestures has not been fully explored for HCI application. It is necessary to achieve more gestures as the command of interface. This paper proposed a method to recognize nine different hand gestures. Using camera to get images of people wearing pink gloves, and then preprocess those images by color splitting, morphological processing and edge extraction. Fourier descriptor, edge histogram and boundary moment invariants are three methods of feature extraction. At last, the template matching was used to realize the hand gesture recognition. The average recognition rate of the nine different gestures employing three different methods is 0.859.