Multi-core architectures have been adopted to enhance computation capability in embedded real-time systems, yet the interaction between cores affects performance, hardware complexity, and utilization of the systems. Current approaches suffer from high complexity or limited flexibility. We present a lightweight task-based multi-core interaction framework compatible with RISC-V specifications, balancing complexity and flexibility. The method proposed relies on RISC-V Core Local Interrupt Controller (CLIC) with customized extensions on the standard AMBA bus offering fully programmable capability for task offloading/trigger and shared resource protection. Compared with the interconnect between TI’s C2000 Series cores and its Control Law Accelerator(CLA) co-processors, experiments show the method reduces more than 57% of the response cycles cost by multi-core task management, but only introduces overhead of less than 1% area of a dual-issue RISC-V core with 7-stage in-order pipeline.

This paper presents optimization efforts on instruction-level parallelism in D-RTCore, a seven-stage dual-issue RISC-V processor designed for real-time control applications. To enhance instruction fetch and dispatch efficiency, we introduce four key techniques: branch prediction optimization, variable-length hardware loops, dynamic instruction fusion, and out-of-order write-back without a reorder buffer (ROB). Performance evaluations reveal that D-RTCore achieves cycle reductions of up to 79% compared to the industry-standard TI C28x. For example, it executes the complex fast fourier transform with a 44% reduction in cycles and the fast fourier transform magnitude with a remarkable 79% decrease in cycles. These results highlight D-RTCore’s advanced execution capabilities, making it a competitive solution for real-time control systems that require efficient processing.

This paper presents MULAN-WC, a novel multi-robot 3D reconstruction framework that leverages wireless signal-based coordination between robots and Neural Radiance Fields (NeRF). Our approach addresses key challenges in multi-robot 3D reconstruction, including inter-robot pose estimation, localization uncertainty quantification, and active best-next-view selection. We introduce a method for using wireless Angle-of-Arrival (AoA) and ranging measurements to estimate relative poses between robots, as well as quantifying and incorporating the uncertainty embedded in the wireless localization of these pose estimates into the NeRF training loss to mitigate the impact of inaccurate camera poses. Furthermore, we propose an active view selection approach that accounts for robot pose uncertainty when determining the best-next-views to improve the 3D reconstruction, enabling faster convergence through intelligent view selection. Extensive experiments on both synthetic and real-world datasets demonstrate the effectiveness of our framework in theory and in practice. Leveraging wireless coordination and localization uncertainty-aware training, MULAN-WC can achieve high-quality 3D reconstruction that is close to applying the ground truth camera poses. Furthermore, the quantification of the information gain from a novel view enables consistent rendering quality improvement with incrementally captured images by commanding the robot to the novel view position. Our hardware experiments showcase the practicality of deploying MULAN-WC to real robotic systems.

In this paper we propose a novel algorithm, Wi-Closure, to improve the computational efficiency and robustness of loop closure detection in multi-robot SLAM. Our approach decreases the computational overhead of classical approaches by pruning the search space of potential loop closures, prior to evaluation by a typical multi-robot SLAM pipeline. Wi-Closure achieves this by identifying candidates that are spatially close to each other measured via sensing over the wireless communication signal between robots, even when they are operating in non-line-of-sight or in remote areas of the environment from one another. We demonstrate the validity of our approach in simulation and in hardware experiments. Our results show that using Wi-closure greatly reduces computation time, by 54.1% in simulation and 76.8% in hardware experiments, compared with a multi-robot SLAM baseline. Importantly, this is achieved without sacrificing accuracy. Using Wi-closure reduces absolute trajectory estimation error by 98.0% in simulation and 89.2% in hardware experiments. This improvement is partly due to Wi-Closure's ability to avoid catastrophic optimization failure that typically occurs with classical approaches in challenging repetitive environments.

The Image Difference Captioning (IDC) task aims to describe the visual differences between two similar images with natural language. The major challenges of this task lie in two aspects: 1) fine-grained visual differences that require learning stronger vision and language association and 2) high-cost of manual annotations that leads to limited supervised data. To address these challenges, we propose a new modeling framework following the pre-training-finetuning paradigm. Specifically, we design three self-supervised tasks and contrastive learning strategies to align visual differences and text descriptions at a fine-grained level. Moreover, we propose a data expansion strategy to utilize extra cross-task supervision information, such as data for fine-grained image classification, to alleviate the limitation of available supervised IDC data. Extensive experiments on two IDC benchmark datasets, CLEVR-Change and Birds-to-Words, demonstrate the effectiveness of the proposed modeling framework. The codes and models will be released at https://github.com/yaolinli/IDC.

This paper presents the WiFi-Sensor-for-Robotics (WSR) open-source toolbox111Code: https://github.com/Harvard-REACT/WSR-Toolbox Dataset: https://github.com/Harvard-REACT/WSR-Toolbox-Dataset Demo: https://github.com/Harvard-REACT/WSR-Toolbox/wiki/Demo. It enables robots in a team to obtain relative bearing to each other, even in nonline-of-sight (NLOS) settings which is a very challenging problem in robotics. It does so by analyzing the phase of their communicated WiFi signals as the robots traverse the environment. This capability, based on the theory developed in our prior works, is made available for the first time as an open-source toolbox. It is motivated by the lack of easily deployable solutions that use robots' local resources (e.g WiFi) for sensing in NLOS. This has implications for multi-robot mapping and rendezvous, ad-hoc robot networks, and security in multi-robot teams, amongst other applications. The toolbox is designed for distributed and online deployment on robot platforms using commodity hardware and on-board sensors. We also release datasets demonstrating its performance in NLOS and line-of-sight (LOS) settings and for a multi-robot localization use case. Empirical results for hardware experiments show that the bearing estimation from our toolbox achieves accuracy with mean and standard deviation of 1.13 degrees, 11.07 degrees in LOS and 6.04 degrees, 26.4 degrees for NLOS, respectively, in an indoor office environment.

Live video interactive commenting, a.k.a. danmaku, is an emerging social feature on online video sites, which involves rich multimodal information interaction among viewers. In order to support various related research, we build a large scale video interactive comments dataset called VideoIC, which consists of 4951 videos spanning 557 hours and 5 million comments. Videos are collected from popular categories on the 'Bilibili' video streaming website. Comparing to other existing danmaku datasets, our VideoIC contains richer and denser comments information, with 1077 comments per video on average. High comment density and diverse video types make VideoIC a challenging corpus for various research such as automatic video comments generation. We also propose a novel model based on multimodal multitask learning for comment generation (MML-CG), which integrates multiple modalities to achieve effective comment generation and temporal relation prediction. A multitask loss function is designed to train both tasks jointly in the end-to-end manner. We conduct extensive experiments on both VideoIC and Livebot datasets. The results prove the effectiveness of our model and reveal some features of danmaku.

Weiying Wang, Yongcheng Wang, Shizhe Chen, Qin Jin. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.