We explore task-free continual learning (CL), in which a model is trained to avoid catastrophic forgetting in the absence of explicit task boundaries or identities. Among many efforts on task-free CL, a notable family of approaches are memory-based that store and replay a subset of training examples. However, the utility of stored seen examples may diminish over time since CL models are continually updated. Here, we propose Gradient based Memory EDiting (GMED), a framework for editing stored examples in continuous input space via gradient updates, in order to create more "challenging" examples for replay. GMED-edited examples remain similar to their unedited forms, but can yield increased loss in the upcoming model updates, thereby making the future replays more effective in overcoming catastrophic forgetting. By construction, GMED can be seamlessly applied in conjunction with other memory-based CL algorithms to bring further improvement. Experiments validate the effectiveness of GMED, and our best method significantly outperforms baselines and previous state-of-the-art on five out of six datasets.

Humans acquire language continually with much more limited access to data samples at a time, as compared to contemporary NLP systems.To study this human-like language acquisition ability, we present VisCOLL, a visually grounded language learning task, which simulates the continual acquisition of compositional phrases from streaming visual scenes.In the task, models are trained on a paired image-caption stream which has shifting object distribution; while being constantly evaluated by a visually-grounded masked language prediction task on held-out test sets.VisCOLL compounds the challenges of continual learning (i.e., learning from continuously shifting data distribution) and compositional generalization (i.e., generalizing to novel compositions).To facilitate research on VisCOLL, we construct two datasets, COCO-shift and Flickrshift, and benchmark them using different continual learning methods.Results reveal that SoTA continual learning approaches provide little to no improvements on VisCOLL, since storing examples of all possible compositions is infeasible.We conduct further ablations and analysis to guide future work 1 .

Deep neural models for relation extraction tend to be less reliable when perfectly labeled data is limited, despite their success in label-sufficient scenarios. Instead of seeking more instance-level labels from human annotators, here we propose to annotate frequent surface patterns to form labeling rules. These rules can be automatically mined from large text corpora and generalized via a soft rule matching mechanism. Prior works use labeling rules in an exact matching fashion, which inherently limits the coverage of sentence matching and results in the low-recall issue. In this paper, we present a neural approach to ground rules for RE, named Nero, which jointly learns a relation extraction module and a soft matching module. One can employ any neural relation extraction models as the instantiation for the RE module. The soft matching module learns to match rules with semantically similar sentences such that raw corpora can be automatically labeled and leveraged by the RE module (in a much better coverage) as augmented supervision, in addition to the exactly matched sentences. Extensive experiments and analysis on two public and widely-used datasets demonstrate the effectiveness of the proposed Nero framework, comparing with both rule-based and semi-supervised methods. Through user studies, we find that the time efficiency for a human to annotate rules and sentences are similar (0.30 vs. 0.35 min per label). In particular, Nero’s performance using 270 rules is comparable to the models trained using 3,000 labeled sentences, yielding a 9.5x speedup. Moreover, Nero can predict for unseen relations at test time and provide interpretable predictions. We release our code1 to the community for future research.

Cognitive task analysis (CTA) is a type of analysis in applied psychology aimed at eliciting and representing the knowledge and thought processes of domain experts. In CTA, often heavy human labor is involved to parse the interview transcript into structured knowledge (e.g., flowchart for different actions). To reduce human efforts and scale the process, automated CTA transcript parsing is desirable. However, this task has unique challenges as (1) it requires the understanding of long-range context information in conversational text; and (2) the amount of labeled data is limited and indirect—i.e., context-aware, noisy, and low-resource. In this paper, we propose a weakly-supervised information extraction framework for automated CTA transcript parsing. We partition the parsing process into a sequence labeling task and a text span-pair relation extraction task, with distant supervision from human-curated protocol files. To model long-range context information for extracting sentence relations, neighbor sentences are involved as a part of input. Different types of models for capturing context dependency are then applied. We manually annotate real-world CTA transcripts to facilitate the evaluation of the parsing tasks.

This paper proposes a method to design low-density parity-check (LDPC) codes for delayed bit-interleaved coded modulation (DBICM). In the method, the code variable node (VN) degree distributions and the assignments of VNs with different degrees to DBICM subchannels are optimized via two cascaded differential evolution (DE) steps. In each step, to optimize VN degree distribution or channel assignment, a parity-check matrix is constructed, and the associated decoding threshold is calculated for each element in a generation. In constructing a parity-check matrix for each channel assignment, we propose a constraint PEGlike code construction method. Protograph-EXIT is employed to calculate the decoding threshold for each parity-check matrix. We apply the proposed method to construct irregular binary LDPC codes for both 16-QAM DBICM and BICM schemes. Simulation results demonstrate that the optimized LDPC codes are within 1 dB from the associated capacity limit at a bit error rate (BER) of 10-6. Besides, the LDPC coded DBICM achieves an SNR gain of 0.5 dB to 0.1 dB over BICM counterparts at a code rate ranges from 0.25 to 0.5.

In this paper, we consider safety message transmission in a dense vehicular network. With increasing vehicular network density, the collision rate increases when multiple vehicles transmit safety messages simultaneously. To address this issue, we propose a request-transmission split time division multiple access (TDMA) scheme, referred to as RTS-TDMA. In our scheme, we divide a frame into three phases, i.e., a contention access phase, a broadcast feedback phase, and a contention-free transmission phase. Each vehicle selects a repetition rate according to a given probability distribution and repeats the transmission of its request packet to improve the reliability of the request. In addition, a roadside unit acts as the coordinator and uses a successive interference cancellation technique to resolve request collisions. RTS-TDMA also reduces the request time percentage by containing only the vehicle identity in each request packet. Both theoretical analysis and numerical results verify that the RTS-TDMA scheme can provide higher throughput than the coded slotted ALOHA scheme.

The progressive edge-growth (PEG) algorithm constructs an edge in each stage to maximize the variable node (VN) of interest's local girth in real time. Thus, this VN's local girths, after more than one edge is added to the current tanner graph (TG) setting, may not be maximized relative to that TG setting. To address this problem, we define the multi-edge local girth and edge-trial, and based on these definitions, propose a new multi-edge metric-constrained PEG algorithm (MM-PEGA) to improve the design at each VN. The MM-PEGA constructs an edge in each stage that, relative to the current TG setting, can potentially maximize the VN of interest's local girth after a certain number (up to the edge-trial) of edges are added to the TG setting. We first analyze the properties of the multi-edge local girth, and then propose an algorithm for calculating the multi-edge local girth. We also propose a method for accelerating the MM-PEGA. Moreover, we generalize the MM-PEGA for improving different PEG-like designs. According to the theoretical analysis, increasing the edge-trial of the MM-PEGA is expected to positively affect the cycle-structure and the error performance of resulting low-density parity-check (LDPC) code. This expectation is verified by simulations.

In this paper, we propose a new multi-edge metric-constrained quasi-cyclic progressive edge-growth algorithm (MM-QC-PEGA), which is suitable for constructing both single- and multi-weighted (binary) QC low-density parity-check (LDPC) codes with arbitrary lengths, rates, circulant sizes, and variable node (VN)-degree distributions. The MM-QC-PEGA is able to detect all cyclic-edge-set-minimum-virtual cycles (CMVCs), as it accurately computes the metric value of each CMVC with an a posteriori test. In addition, we propose a new greatest-common-divisor (GCD)-approximation for time efficiently approximating the metric value of a CMVC without a posteriori tests, and propose a new GCD-approximated MM-QC-PEGA (G-MM-QC-PEGA) by using the GCD-approximation to replace the a posteriori test involved in the MM-QC-PEGA. As a result, the G-MM-QC-PEGA is faster than the MM-QC-PEGA, and the CMVCs undetectable to the G-MM-QC-PEGA under multi- and single-weighted QC-LDPC code graphs have minimum lengths of eight and ten, respectively. Moreover, we propose a new masking technique that is efficient in masking the parity-check matrix consisting of an array of arbitrary circulants of the same size. Compared with the several existing works, our proposed algorithms could perform better or comparably in terms of avoiding generating small cycles and error performances. Our proposed algorithms somewhat perfect the works of QC-LDPC code construction.

We consider low-density parity-check (LDPC) code design by considering the unequal error protection property in high order modulated bit-interleaved coded modulation (BICM) schemes. The existing work mainly considered the effect of variable node edge assignments on the decoding performance. In this paper, we consider both variable node and check node edge assignments to further optimize the LDPC codes for BICM schemes. To achieve this, we derive new extrinsic information transfer (EXIT) functions for both regular and irregular LDPC code ensembles. Then we employ differential evolution to optimize the code ensembles in terms of the lowest decoding threshold. Finally, we propose a modified progressive edge growth algorithm to design regular and irregular LDPC codes based on the optimized code ensembles. The numerical results show that our designed LDPC codes have better bit error rate performance compared to the codes designed in the existing work.

We propose a new low-density parity-check (LDPC) coded binary physical-layer network coding (PNC) scheme for Gaussian two-way relay channels. In this scheme, we introduce a bit mapper between the LDPC encoder and the modulator, which considers the unequal error protections brought by the high order PSK modulations. We add a new bipartite sub-channel graph consisting of sub-channels and variable nodes (VNs) to the Tanner graph and propose a progressive edge growth (PEG) algorithm to design the bit mapper. The design paradigm is to search for the bit mapping distribution with the lowest decoding threshold by using the extrinsic information transfer (EXIT) chart, and then establish the edges progressively between VNs and sub-channels according to the distribution. The proposed PEG algorithm is employed to design the bit mappers of the schemes with 8-PSK, 16-PSK, 64-PSK and 256-PSK. Simulation results show that the proposed schemes can considerably improve the bit error performance of the PNC XOR messages, compared to the schemes without bit mappers.

In this paper, we consider the design of the bit mapping in low-density parity-check (LDPC) coded bit-interleaved coded modulation (BICM) schemes. We introduce a two-layer bipartite graph to represent the LDPC coded BICM scheme where a new bit mapping graph linking sub-channels to variable nodes (VNs) is added to the conventional Tanner graph. We propose a progressive edge growth (PEG) algorithm to design the bit mapping for the BICM scheme. The design paradigm is to provide more protections to the VNs that are allocated to the sub-channels with the lowest mutual information. We define a novel concept of unreliable depth profile to classify the reliability of the VNs. By connecting more reliable edges to the least reliable VNs, we can significantly improve the reliable edge distribution for the unreliable VNs, thus improve the extrinsic information in the iterative decoding. The proposed bit mapping algorithm is employed for the design of the LDPC coded BICM with 64-QAM and 256-QAM. Simulation results show that the proposed design can considerably improve the error performance, compared to the conventional consecutive bit mapping strategy.

To improve the performance further for the bit- interleaved coded modulation with iterative decoding (BICM-ID) system, a modified extrinsic information transfer (EXIT) model is presented, in which the decoding and the demodulation are modeled as two iterative processing units. By using the binary erasure channel (BEC) approximation for the a priori AWGN channel, the analytical formulation of mutual information for sub-channels in M-QAM are obtained. Besides, the grid depth-first searching algorithm (GDFSA) is modified for designing the optimal bit mapping distribution with minimum convergence rate. Using the DVB-S2 rate-0.5 64800-bit low-density parity-check (LDPC) code as demonstration, the simulations show that the BER performances achieve 0.225/0.650dB gains at 10-5for 16-QAM/32-QAM under AWGN channel, compared to the consecutive bit mapping.

This paper proposes two new cycle-optimized algorithms, called the multi-step progressive edge-growth (PEG) algorithm and the multi-step approximate cycle extrinsic message degree (ACE) constrained PEG algorithm, to design an LDPC code with larger girth and better cycle-connectivity. In addition, an efficient method is developed to calculate the distances between the variable nodes (VNs) and the check nodes (CNs) in the proposed algorithms. Simulation results show that compared to the conventional cycle-optimized algorithms, the proposed algorithms can achieve both better cycle-connectivity and foreseeable lower error floors.