Abstract An open-domain question answering (QA) system usually follows a retrieve-then-read paradigm, in which a retriever is used to retrieve relevant passages from a large corpus, and then a reader generates answers based on the retrieved passages and the original question. In this paper, we propose a simple and novel mutual learning framework to improve the performance of retrieve-then-read-style models via an intermediate module named the knowledge selector, which we train with reinforcement learning. The key benefits of our proposed intermediate module are: 1) no requirement for additional annotated question-passage pairs; 2) improvements in both retrieval and QA performance, as well as computational efficiency, compared to prior competitive retrieve-then-read models; 3) with no finetuning, improvement in the zero-shot performance of large-scale pre-trained language models, e.g., ChatGPT, by encapsulating the input with relevant knowledge without violating the input length constraint.

Instruction following vision-language (VL) models offer a flexible interface that supports a broad range of multimodal tasks in a zero-shot fashion. However, interfaces that operate on full images do not directly enable the user to “point to" and access specific regions within images. This capability is important not only to support reference-grounded VL benchmarks, but also, for practical applications that require precise within-image reasoning. We build Localized Visual Commonsense model which allows users to specify (multiple) regions- as-input. We train our model by sampling localized commonsense knowledge from a large language model (LLM): specifically, we prompt a LLM to collect commonsense knowledge given a global literal image description and a local literal region description automatically generated by a set of VL models. This pipeline is scalable and fully automatic, as no aligned or human-authored image and text pairs are required. With a separately trained critic model that selects high quality examples, we find that training on the localized commonsense corpus expanded solely from images can successfully distill existing VL models to support a reference-as-input interface. Empirical results and human evaluations in zero-shot settings demonstrate that our distillation method results in more precise VL models of reasoning compared to a baseline of passing a generated referring expression.

Liangke Gui, Borui Wang, Qiuyuan Huang, Alexander Hauptmann, Yonatan Bisk, Jianfeng Gao. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.

Vision-language navigation (VLN) is the task of navigating an embodied agent to carry out natural language instructions inside real 3D environments. In this paper, we study how to address three critical challenges for this task: the cross-modal grounding, the ill-posed feedback, and the generalization problems. First, we propose a novel Reinforced Cross-Modal Matching (RCM) approach that enforces cross-modal grounding both locally and globally via reinforcement learning (RL). Particularly, a matching critic is used to provide an intrinsic reward to encourage global matching between instructions and trajectories, and a reasoning navigator is employed to perform cross-modal grounding in the local visual scene. Evaluation on a VLN benchmark dataset shows that our RCM model significantly outperforms baseline methods by 10 percent on Success Rate weighted by Path Length (SPL) and achieves the state-of-the-art performance. To improve the generalizability of the learned policy, we further introduce a Self-Supervised Imitation Learning (SIL) method to explore and adapt to unseen environments by imitating its own past, good decisions. We demonstrate that SIL can approximate a better and more efficient policy, which tremendously minimizes the success rate performance gap between seen and unseen environments (from 30.7 to 11.7 percent).

Generating formal-language programs represented by relational tuples, such as Lisp programs or mathematical operations, to solve problems stated in natural language is a challenging task because it requires explicitly capturing discrete symbolic structural information implicit in the input. However, most general neural sequence models do not explicitly capture such structural information, limiting their performance on these tasks. In this paper, we propose a new encoder-decoder model based on a structured neural representation, Tensor Product Representations (TPRs), for mapping Natural-language problems to Formal-language solutions, called TP-N2F. The encoder of TP-N2F employs TPR ‘binding’ to encode natural-language symbolic structure in vector space and the decoder uses TPR ‘unbinding’ to generate, in symbolic space, a sequential program represented by relational tuples, each consisting of a relation (or operation) and a number of arguments. TP-N2F considerably outperforms LSTM-based seq2seq models on two benchmarks and creates new state-of-the-art results. Ablation studies show that improvements can be attributed to the use of structured TPRs explicitly in both the encoder and decoder. Analysis of the learned structures shows how TPRs enhance the interpretability of TP-N2F.

This paper proposes a novel neural architecture — Attentive Tensor Product Learning (ATPL) — to represent grammatical structures of natural language in deep learning models. ATPL exploits Tensor Product Representations (TPR), a structured neural-symbolic model developed in cognitive science, to integrate deep learning with explicit natural language structures and rules. The key ideas of ATPL are: 1) unsupervised learning of role-unbinding vectors of words via the TPR-based deep neural network; 2) the use of attention modules to compute TPR; and 3) the integration of TPR with typical deep learning architectures including long short-term memory and feedforward neural networks. The novelty of our approach lies in its ability to extract the grammatical structure of a sentence by using role-unbinding vectors, which are obtained in an unsupervised manner. Our ATPL approach is applied to 1) image captioning, 2) part of speech (POS) tagging, and 3) constituency parsing of a natural language sentence. The experimental results demonstrate the effectiveness of the proposed approach in all these three natural language processing tasks.

We propose a hierarchically structured reinforcement learning approach to address the challenges of planning for generating coherent multi-sentence stories for the visual storytelling task. Within our framework, the task of generating a story given a sequence of images is divided across a two-level hierarchical decoder. The high-level decoder constructs a plan by generating a semantic concept (i.e., topic) for each image in sequence. The low-level decoder generates a sentence for each image using a semantic compositional network, which effectively grounds the sentence generation conditioned on the topic. The two decoders are jointly trained end-to-end using reinforcement learning. We evaluate our model on the visual storytelling (VIST) dataset. Empirical results from both automatic and human evaluations demonstrate that the proposed hierarchically structured reinforced training achieves significantly better performance compared to a strong flat deep reinforcement learning baseline.

In this paper, we propose Object-driven Attentive Generative Adversarial Newtorks (Obj-GANs) that allow attention-driven, multi-stage refinement for synthesizing complex images from text descriptions. With a novel object-driven attentive generative network, the Obj-GAN can synthesize salient objects by paying attention to their most relevant words in the text descriptions and their pre-generated class label. In addition, a novel object-wise discriminator based on the Fast R-CNN model is proposed to provide rich object-wise discrimination signals on whether the synthesized object matches the text description and the pre-generated class label. The proposed Obj-GAN significantly outperforms the previous state of the art in various metrics on the large-scale MS-COCO benchmark, increasing the inception score by 27% and decreasing the FID score by 11%. A thorough comparison between the classic grid attention and the new object-driven attention is provided through analyzing their mechanisms and visualizing their attention layers, showing insights of how the proposed model generates complex scenes in high quality.

Vision-language navigation (VLN) is the task of navigating an embodied agent to carry out natural language instructions inside real 3D environments. In this paper, we study how to address three critical challenges for this task: the cross-modal grounding, the ill-posed feedback, and the generalization problems. First, we propose a novel Reinforced Cross-Modal Matching (RCM) approach that enforces cross-modal grounding both locally and globally via reinforcement learning (RL). Particularly, a matching critic is used to provide an intrinsic reward to encourage global matching between instructions and trajectories, and a reasoning navigator is employed to perform cross-modal grounding in the local visual scene. Evaluation on a VLN benchmark dataset shows that our RCM model significantly outperforms previous methods by 10% on SPL and achieves the new state-of-the-art performance. To improve the generalizability of the learned policy, we further introduce a Self-Supervised Imitation Learning (SIL) method to explore unseen environments by imitating its own past, good decisions. We demonstrate that SIL can approximate a better and more efficient policy, which tremendously minimizes the success rate performance gap between seen and unseen environments (from 30.7% to 11.7%).

Ming Jiang, Junjie Hu, Qiuyuan Huang, Lei Zhang, Jana Diesner, Jianfeng Gao. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

Ming Jiang, Qiuyuan Huang, Lei Zhang, Xin Wang, Pengchuan Zhang, Zhe Gan, Jana Diesner, Jianfeng Gao. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). 2019.

In this paper, we propose an Attentional Generative Adversarial Network (AttnGAN) that allows attention-driven, multi-stage refinement for fine-grained text-to-image generation. With a novel attentional generative network, the AttnGAN can synthesize fine-grained details at different sub-regions of the image by paying attentions to the relevant words in the natural language description. In addition, a deep attentional multimodal similarity model is proposed to compute a fine-grained image-text matching loss for training the generator. The proposed AttnGAN significantly outperforms the previous state of the art, boosting the best reported inception score by 14.14% on the CUB dataset and 170.25% on the more challenging COCO dataset. A detailed analysis is also performed by visualizing the attention layers of the AttnGAN. It for the first time shows that the layered attentional GAN is able to automatically select the condition at the word level for generating different parts of the image.

Qiuyuan Huang, Paul Smolensky, Xiaodong He, Li Deng, Dapeng Wu. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers). 2018.

We study in this paper the problems of both image captioning and text-to-image generation, and present a novel turbo learning approach to jointly training an image-to-text generator (a.k.a. CaptionBot) and a text-to-image generator (a.k.a. DrawingBot). The key idea behind the joint training is that image-to-text generation and text-to-image generation as dual problems can form a closed loop to provide informative feedback to each other. Based on such feedback, we introduce a new loss metric by comparing the original input with the output produced by the closed loop. In addition to the old loss metrics used in CaptionBot and DrawingBot, this extra loss metric makes the jointly trained CaptionBot and DrawingBot better than the separately trained CaptionBot and DrawingBot. Furthermore, the turbo-learning approach enables semi-supervised learning since the closed loop can provide peudo-labels for unlabeled samples. Experimental results on the COCO dataset demonstrate that the proposed turbo learning can significantly improve the performance of both CaptionBot and DrawingBot by a large margin.

Visible light communication (VLC) has been shown to have several advantages over traditional wireless communication. We envision a LED-to-smartphone VLC protocol for delivering messages to a group of unsynchronized mobile device receivers. We carefully design and implement our protocol, Martian, which allows smooth communication from the LED lights to a group of camera-enabled mobile devices. Across several phone models, Martian can achieve data rate of about 1.6kbps even with NLOS-light. Our intensive evaluations indicate that, the data rate reaches 4.2kbps on iPhone 6. This is a significant improvement compared with the 88bps data rate claimed by state-of-art design.

Joint FoUntain coding and Network coding (FUN) is proposed to boost information spreading over multi-hop lossy networks. The novelty of our FUN approach lies in combining the best features of fountain coding, intra-session network coding, and cross-next-hop network coding. This paper provides an in-depth study of FUN codes. First, we theoretically analyze the throughput of FUN codes. Second, we identify several practical issues that may undermine the actual performance, such as buffer overflow, and quantify the resulting throughput degradation. Finally, we propose a systematic design to overcome these issues. Simulation results in TDMA multi-hop networks show that our methods yield near-optimal throughput and are significantly better than fountain codes and existing network coding schemes.

We consider the problem of online dynamic channel accessing in multi-hop cognitive radio networks. Previous works on online dynamic channel accessing mainly focus on single-hop networks that assume complete conflicts among all secondary users. In the multi-hop multi-channel network settings studied here, there is more general competition among different communication pairs. A simple application of models for single-hop case to multi-hop case with N nodes and M channels leads to exponential time/space complexity O (MN), and poor theoretical guarantee on throughput performance. We thus novelly formulate the problem as a linearly combinatorial multi-armed bandits (MAB) problem that involves a maximum weighted independent set (MWIS) problem with unknown weights. To efficiently address the problem, we propose a distributed channel access algorithm that can achieve 1/ρ of the optimum averaged throughput where each node has communication complexity O (r2+D) and space complexity O (m) in the learning process, and time complexity O (D mρr) in strategy decision process for an arbitrary wireless network. Here ρ = 1 + ε is the approximation ratio to MWIS for a local r-hop network with m <; N nodes, and D is the number of mini-rounds inside each round of strategy decision.

To address the problem of information spreading over lossy communication channels, this paper proposes a joint FoUntain coding and Network coding (FUN) approach. Different from the Transmission Control Protocol (TCP), our FUN approach is a mechanism of Forward Error Correction (FEC), which does not use retransmission for recovery of lost packets. The novelty of our FUN approach lies in combining the best features of fountain coding, intra-session network coding, and cross-next-hop network coding. As such, our FUN approach is capable of achieving unprecedented high throughput over lossy channels. Experimental results demonstrate that our FUN approach achieves higher throughput than the existing schemes for multihop wireless networks.

Assume that a set of Demand Response Switch (DRS) devices are deployed in smart meters for autonomous demand side management within one house. The DRS devices are able to sense and control the activity of each appliance. We propose a set of appliance scheduling algorithms to 1) minimize the peak power consumption under a fixed delay requirement, and 2) minimize the delay under a fixed peak demand constraint. For both problems, we first prove that they are NP-Hard. Then we propose a set of approximation algorithms with constant approximation ratios. We conduct extensive simulations using both real-life appliance energy consumption data trace and synthetic data to evaluate the performance of our algorithms. Extensive evaluations verify that the schedules obtained by our methods significantly reduce the peak demand or delay compared with naive greedy algorithm or randomized algorithm.

The popularity of GPS-enabled smartphones enables a wide variety of new location-based or location-aware services and applications. However, the GPS module in a smartphone produces inaccurate position estimates and incurs high energy consumption, which inhibits the wide use of location-aware applications. To address this, we propose a social-aided cooperative location optimization (Coloc) scheme, which is capable of improving positioning accuracy and achieving low energy consumption. Specifically, our scheme enhances positioning accuracy by fusing the GPS positions of multiple co-located smartphones in a social network, or by neighborhood-based weighted least-squares estimation when relative distances between smartphones are available. The energy efficiency is achieved by sharing location information among co-located users and lower the update rate of the GPS module without sacrificing the accuracy. To validate our proposed approach, we conduct experiments in stationary and moving scenarios. Experimental results show that our proposed cooperative localization scheme can achieve sufficient performance gains in both indoor and outdoor environments.

In this work, we present SilentSense, a framework to authenticate users silently and transparently by exploiting the user touch behavior biometrics and leveraging the integrated sensors to capture the micro-movement of the device caused by user's screen-touch actions. By tracking the fine-detailed touch actions of the user, we build a "touch-based biometrics" model of the owner by extracting some principle features, and then verify whether the current user is the owner or guest/attacker. When using the smartphone, the unique operating pattern of the user is detected and learnt by collecting the sensor data and touch events silently. When users are mobile, the micro-movement of mobile devices caused by touch is suppressed by that due to the large scale user-movement which will render the touch-based biometrics ineffective. To address this, we integrate a movement-based biometrics for each user with previous touch-based biometrics. We conduct extensive evaluations of our approaches on the Android smartphone, we show that the user identification accuracy is over 99%.

As a large scale real sensor network system, GreenOrbs reveals that locating sensor nodes in the forest still faces great challenges because of volatile and fluctuating environmental factors. In this paper, we present a novel localization scheme, EARL, which provides accurate reference nodes and good ranging quality. We exam the range quality along routing paths by taking complex environmental factors into account, such as forest density, temperature and humidity. To improve localization accuracy, we use power scanning technique to judge the accuracy reference nodes and further calibrate the bad nodes through reverse-localization. To overcome the error propagation, we assign different weights to the range measurement according to the ranging quality. We implemented our localization scheme in GreenOrbs testbed, and evaluate through extensive experiments. The results demonstrate that EARL outperforms the current localization approaches with better accuracy. The localization accuracy achieved by our method is around 20% higher than best existing methods.

We propose joint channel sensing, probing, and accessing schemes for secondary users in cognitive radio networks. Our method has time and space complexity O(N·k) for a network with N channels and k secondary users, while applying classic methods requires exponential time complexity. We prove that, even when channel states are selected by adversary (thus non-stochastic), it results in a total regret uniformly upper bounded by Θ(√TN logN), w.h.p, for communication lasts for T timeslots. Our protocol can be implemented in a distributed manner due to the nonstochastic channel assumption. Our experiments show that our schemes achieve almost optimal throughput compared with an optimal static strategy, and perform significantly better than previous methods in many settings.

In this work, we address the problem of finding experts using chains of social referrals and profile matching with only local information in online social networks. By assuming that users are selfish, rational, and have privately known cost of participating in the referrals, we design a novel truthful efficient mechanism in which an expert-finding query will be relayed by intermediate users. When receiving a referral request, a participant will locally choose among her neighbors some user to relay the request. In our mechanism, several closely coupled methods are carefully designed to improve the search performance, including, profile matching, social acquaintance prediction, score function for locally choosing relay neighbors, and budget estimation. We conduct extensive experiments on several datasets of online social networks. The extensive study of our mechanism shows that the success rate of our mechanism is about 90% in finding closely matched experts using only local search and limited budget, which significantly improves the previously best rate 20%. The overall cost of finding an expert by our truthful mechanism is about 20% of the untruthful method and only about 2% of the method that always selects high-degree neighbors. The median length of social referral chains is 6 using our localized search decision, which surprisingly matches the well-known small-world phenomenon of global social structures.

We study distributed link scheduling for throughput maximization in wireless networks. The majority of results on link scheduling assume binary interference models for simplicity. While the physical interference model reflects the physical reality more precisely, the problem becomes notoriously harder under the physical interference model. There have been just a few existing results on centralized link scheduling under the physical interference model, though distributed schedulings are more practical. In this paper, by leveraging the partition and shifting strategies and the pick-and-compare scheme, we present the first distributed link scheduling algorithm that can achieve a constant fraction of the optimal capacity region subject to physical interference constraints in the linear power setting for multihop wireless networks.