Traditional dense video captioning predominantly focuses on edited exocentric footage. These videos are filmed from an external perspective and generally feature distinct transitions between different events, as exemplified in edited instructional videos. However, such videos do not genuinely reflect the way we perceive our real lives. Instead, we observe the world from an egocentric viewpoint and witness only continuous unedited footage. To facilitate further research, we introduce a new topic: Egocentric Vehicle Dense Video Captioning, in classic vehicle driving scenarios. This is a multi-modal, multi-task subject endeavor for a comprehensive understanding of untrimmed, egocentric driving videos. It consists of three sub-tasks that concentrate on event localization, captioning, and vehicle state estimation separately. To accomplish these tasks, it is necessary to deal with at least three challenges: extracting ego-motion relevant information, describing driving behavior and analyzing the underlying rationale, as well as resolving the boundary ambiguity problem. In response, we devise corresponding solutions, including a vehicle ego-motion learning strategy and a novel adjacent contrastive learning strategy, which effectively address the aforementioned issues. We validate our method by conducting extensive experiments on the BDD-X dataset, all of which show promising results and achieve new state-of-the-art performance on most metrics, which proves the effect of our approach.

Wi-Fi-based contactless activity recognition is of great importance to computer–human interaction, accounting for convenience concerns. However, it remains challenging to recognize activities from multiple users due to the multipath distortion and disruption of Wi-Fi signals. In this article, we propose a highly universal framework, namely, WISDOM, for Wi-Fi-based multiuser activity recognition. Specifically, we first leverage an existing model to identify the number of users from the input Wi-Fi signals. Then, we develop a subcarrier correlation and inversion-based sorting algorithm to extract the signal for each user. Finally, we design a neural network, i.e., WISDOM-Net, which is built on a bidirectional gated recurrent unit network incorporated with the attention mechanism and the one dimension convolutional neural network, to recognize the corresponding user activities. Experimental results show that our proposed WISDOM-Net outperforms the existing baselines on both the public and our own data sets. In particular, WISDOM-Net can reach an average recognition accuracy of up to 98.19% and 90.77% in 2-user and 3-user scenarios, respectively.

Referring Expression Comprehension (REC) is one important research branch in visual grounding, where the goal of REC is to localize a relevant object in the image, given an expression in the form of text to exactly describe a specific object. However, existing REC tasks aim at text content filtering and image object locating, which are evaluated based on the precision of the detection boxes. This may lead models to skip the learning process of multimodal comprehension directly and achieve good performance. In this paper, we work on how to enable an artificial agent to understand RE further and propose a more comprehensive task, called Further Comprehension on Referring Expression (FREC). In this task, we mainly focus on three sub-tasks: 1) correcting the erroneous text expression based on visual information; 2) generating the rationale of this input expression; 3) localizing the proper object based on the corrected expression. Accordingly, we make a new dataset named Further-RefCOCOs based on the RefCOCO, RefCOCO+, RefCOCOg benchmark datasets for this new task and make it publicly available. After that, we design a novel end-to-end pipeline to achieve these sub-tasks simultaneously. The experimental results demonstrate the validity of the proposed pipeline. We believe this work will motivate more researchers to explore along with this direction, and promote the development of visual grounding.

Visual Question Answering (VQA) serves as a proxy for evaluating the scene understanding of an intelligent agent by answering questions about images. Most VQA benchmarks to date are focused on those questions that can be answered through understanding visual content in the scene, such as simple counting, visual attributes, and even a little challenging questions that require extra encyclopedic knowledge. However, humans have a remarkable capacity to reason dynamic interaction on the scene, which is beyond the literal content of an image and has not been investigated so far. In this paper, we propose Agent Interaction Visual Question Answering (AI-VQA), a task investigating deep scene understanding if the agent takes a certain action. For this task, a model not only needs to answer action-related questions but also to locate the objects in which the interaction occurs for guaranteeing it truly comprehends the action. Accordingly, we make a new dataset based on Visual Genome and ATOMIC knowledge graph, including more than 19,000 manually annotated questions, and will make it publicly available. Besides, we also provide an annotation of the reasoning path while developing the answer for each question. Based on the dataset, we further propose a novel method, called ARE, that can comprehend the interaction and explain the reason based on a given event knowledge base. Experimental results show that our proposed method outperforms the baseline by a clear margin.

The consensus of multiple views on the same data will provide extra regularization, thereby improving accuracy. Based on this idea, we proposed a novel Knowledge-Supervised Learning (KSL) method for person re-identification (Re-ID), which can improve the performance without introducing extra inference cost. Firstly, we introduce isomorphic auxiliary training strategy to conduct basic multiple views that simultaneously train multiple classifier heads of the same network on the same training data. The consensus constraints aim to maximize the agreement among multiple views. To introduce this regular constraint, inspired by knowledge distillation that paired branches can be trained collaboratively through mutual imitation learning. Three novel constraints losses are proposed to distill the knowledge that needs to be transferred across different branches: similarity of predicted classification probability for cosine space constraints, distance of embedding features for euclidean space constraints, hard sample mutual mining for hard sample space constraints. From different perspectives, these losses complement each other. Experiments on four mainstream Re-ID datasets show that a standard model with KSL method trained from scratch outperforms its ImageNet pre-training results by a clear margin. With KSL method, a lightweight model without ImageNet pre-training outperforms most large models. We expect that these discoveries can attract some attention from the current de facto paradigm of "pre-training and fine-tuning" in Re-ID task to the knowledge discovery during model training.

The prevalence of heterogeneous manycore processors has been shown as a promising alternative to web-based data parallel MapReduce applications. The differences of manycore from multicore raise new challenges to designing and implementing efficient and scalable MapReduce applications, such as memory access congestion. This paper argues that it is more efficient using multiple small groups of cores than using a single group with all cores to address these challenges. We propose a group-based MapReduce implementation, called Grouped-MapReduce (GMR). It uses grouping to align tasks so that the tasks in each group are congestion-free. Further, it uses multiplexing to control the implementation of groups so that multiple groups can efficiently share the limited memory bandwidth without causing serious memory access congestion. We have implemented a prototype of GMR based on Phoenix++, an already highly optimized MapReduce runtime. Experiments on six benchmarks show that GMR implements and scales well on manycore systems and obtains an impressive improvement over Phoenix++ from 1.04x to 1.77x without artificially tuning the existing application code.

Virtualization technology can provide reusability and strong isolation between different virtual machines (VMs). However, there is no effective isolation mechanism inside a VM to solve an operating system's reliability problems, including driver faults. This paper describes Chariot, an architecture that provides effective and transparent driver isolation inside the VM, achieves fine-grained driver isolation and retains the reusability advantage of virtualization technology. First, Chariot transparently monitors an isolated driver with monitoring wrappers, and establishes an access control table (ACT) in a timely manner that records the driver write permissions. Secondly, Chariot protects the shadow page table of the VM (where the driver resides) in due time to capture its write operations. Next, the ACT examines the correctness of the write operations. Finally, if an illegal write operation is detected, Chariot recovers the faulty driver and prevents the spread of driver faults in the VM. The experimental results show that Chariot effectively isolates more than 90% of injected faults (with performance losses of |$<$|20% in most benchmarks) and effectively improves the reliability of the VM. In addition, Chariot can be easily extended to isolate new drivers and ported to other versions of OSs in the virtualization environment.

Currently, the virtualization technologies can integrate multiple operating systems into a high-performance server to maximize the utilization of the server's resources. This server can serves more users. However, the driver faults in virtual machine still seriously affect the reliability of the virtual machine, and even affect the reliability of the entire server. This paper presents Chariot, a high compatible architecture to improve virtual machine reliability. If the driver is loaded by the Chariot's isolation loading mechanism, its memory usage will be timely monitored by Chariot, and its access control table will be established. Through setting the corresponding shadow page table of the whole kernel space of the virtual machine, Chariot captures the write operations of the isolated driver. Combing the access control table, Chariot can determine the correctness of these writing operations. Chariot has an effective errors isolation capability, and is easy to develop. Also Chariot has an excellent compatibility and needs not to modify the drivers and the operation system in the virtual machine. Experimental results show that Chariot can effectively isolate the driver faults, and improve the reliability of operation system in the virtual machine environments.

With the development of virtualization technologies, the server resources are maximized by integrating multiple operating systems into a high-performance server. So the server is possible to provide services to more users simultaneously. However, the driver fault still impacts the reliability of the operating system in the virtual machine, and impacts the continuity and stability of services. This paper proposes a architecture to improve the reliability of the virtual machine environments. By monitoring the driver's memory usage, the architecture creates the authorization table. Through setting the corresponding shadow page table in the virtual machine manager of the whole kernel space of the virtual machine, the architecture captures the write operation of the virtual machine. Combing with the authorization table, the correctness of the writing operations can be determined. Our architecture needn't to modify the drivers and is easy to develop. Experimental results show that the architecture can effectively isolate the driver faults, and improve the reliability of the virtual machine environments.