Scoring models such as click-through rate (CTR) prediction underpin recommendation and advertising systems, but their features are highly sensitive, making plaintext cloud inference risky. Fully homomorphic encryption (FHE) enables inference directly on ciphertexts, yet homomorphic computation is expensive and bootstrapping often dominates end-to-end latency. We present HE-DeepFM, a FHE inference system for CTR prediction. We first design HE-FM, a homomorphic-friendly Factorization Machine (FM) operator that exploits CKKS SIMD packing to compute second-order interactions efficiently, thereby avoiding the naive O(F2) cost of pairwise feature interactions. Building on HE-FM, HE-DeepFM reduces bootstrapping under the same FHE budget, and can eliminate it for small model configurations. We implement HE-DeepFM with Orion and Lattigo and evaluate it on real-world datasets. On Criteo, HE-DeepFM reduces bootstrapping from 12 to 4 and achieves up to 2.85× end-to-end speedup while maintaining prediction quality comparable to the baseline.

The accuracy of pedestrian detection is susceptible to factors such as illumination and occlusion. However, the prevailing assumption that dense pedestrians do not occur in low-illumination scenarios has resulted in a paucity of research considering both low illumination and intra-class occlusion in the field of pedestrian detection. But considering them simultaneously is necessary due to the phenomenon of pedestrian aggregation that we have found. This paper proposes a low-illumination scene occluded pedestrian detection model based on domain adaptation (LOPD), suitable for low-illumination and intra-class occluded scenes. First, to address the difficulty in acquiring high-quality low-illumination domain data, we employ the semi-supervised domain adaptation method to generalise the pedestrian detection model that has been trained in the normal illumination domain to the low illumination domain. Second, to suppress the high-frequency noise in low-illumination images, we propose the cross-stage low-frequency enhancement filter (CLEF). Third, to address the issue of inaccurate positioning resulting from intra-class occlusion between pedestrians, we propose the attraction repulsion loss (AR loss). Fourth, to solve the problem that pedestrians with intra-class occlusion are easily missed in the detection, we propose the Soft DIOU Non-Maximum Suppression algorithm (SD-NMS). The experimental results demonstrate that our method exhibits superior performance in detecting pedestrians in low-illumination occlusion composite scenes compared to the prevailing mainstream models that address low illumination and occlusion independently.

Classical acoustic impedance inversion methods rely on mathematical and physical models to estimate subsurface acoustic impedance distribution. However, these methods face difficulties in accurately fitting complex impedance data. While deep learning methods have achieved higher accuracy and efficiency in impedance inversion, they remain black box models, lacking interpretability. So, it is difficult to analyze the reason why they are (or are not) effective. To address the limitations of both traditional and deep learning methods, this paper proposes a novel approach, AII-SR (Acoustic Impedance Inversion with Symbolic Regression), which discovers partial differential equations (PDEs) from impedance data to model acoustic impedance inversion. To discover these PDEs, AII-SR adopts a dual-learning framework. It employs a forward model based on the Robinson convolution principle to ensure the physical consistency and reliability of predictions. Subsequently, AII-SR creates an inversion model that combines a symbolic regression-based PDE generator with a physics-informed solving neural network (PSNN) to identify PDEs that accurately fit the impedance data. Experiments demonstrate that AII-SR outperforms deep learning methods, such as SSEI, TCN and Se-Unet, in terms of accuracy and interpretability. AII-SR generates concise, interpretable mathematical expressions in the form of PDEs, offering profound insights into the physical relationships between seismic and impedance data.

Chinese Named Entity Recognition (CNER) is extensively researched in general domains, while, in practical engineering applications, it receives more and more attention in specialized fields. However, CNER’s performance in domain-specific areas, such as in petroleum refining and entertainment, remains moderate due to a lack of annotated data. In this paper, we mainly focus on two improvements related to the problem of scarce annotated data. Firstly, we propose a novel data augmentation method named Knowledge Graph Text Alignment with BART (KGTA-BART), which, for the first time, introduces a knowledge graph extracted from structured and semi-structured data, aligns its graphic information with the semantic information of annotated text, and thus generates high-quality text from the knowledge graph using BART model. Expanding the dataset can help the model learn more entity features and improve its effectiveness when annotated data is scarce. Additionally, we develop the CNER model Virtual Adversary with BART (VA-BART), which utilizes BART as an encoder and applies the virtual adversary to CNER. This improves the capture of contextual information in the text when annotation data is scarce and enhances the model’s generalization ability. Experimental results demonstrate that VA-BART method based on KGTA-BART achieves significant improvements over the baselines when applied to domain-specific dataset in Chinese language.

Genetic programming (GP) is a commonly used approach to solve symbolic regression (SR) problems. Compared with the machine learning or deep learning methods that depend on the pre-defined model and the training dataset for solving SR problems, GP is more focused on finding the solution in a search space. Although GP has good performance on large-scale benchmarks, it randomly transforms individuals to search results without taking advantage of the characteristics of the dataset. So, the search process of GP is usually slow, and the final results could be unstable. To guide GP by these characteristics, we propose a new method for SR, called Taylor genetic programming (TaylorGP)1. TaylorGP leverages a Taylor polynomial to approximate the symbolic equation that fits the dataset. It also utilizes the Taylor polynomial to extract the features of the symbolic equation: low order polynomial discrimination, variable separability, boundary, monotonic, and parity. GP is enhanced by these Taylor polynomial techniques. Experiments are conducted on three kinds of benchmarks: classical SR, machine learning, and physics. The experimental results show that TaylorGP not only has higher accuracy than the nine baseline methods, but also is faster in finding stable results.

Many recent studies focus on developing mechanisms to explain the black-box behaviors of neural networks (NNs). However, little work has been done to extract the potential hidden semantics (mathematical representation) of a neural network. A succinct and explicit mathematical representation of a NN model could improve the understanding and interpretation of its behaviors. To address this need, we propose a novel symbolic regression method for neural works (called SRNet) to discover the mathematical expressions of a NN. SRNet creates a Cartesian genetic programming (NNCGP) to represent the hidden semantics of a single layer in a NN. It then leverages a multi-chromosome NNCGP to represent hidden semantics of all layers of the NN. The method uses a (1+λ) evolutionary strategy (called MNNCGP-ES) to extract the final mathematical expressions of all layers in the NN. Experiments on 12 symbolic regression benchmarks and 5 classification benchmarks show that SRNet not only can reveal the complex relationships between each layer of a NN but also can extract the mathematical representation of the whole NN. Compared with LIME and MAPLE, SRNet has higher interpolation accuracy and trends to approximate the real model on the practical dataset1.

Zhaojiang Lin, Bing Liu, Andrea Madotto, Seungwhan Moon, Zhenpeng Zhou, Paul Crook, Zhiguang Wang, Zhou Yu, Eunjoon Cho, Rajen Subba, Pascale Fung. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.

Andrea Madotto, Zhaojiang Lin, Zhenpeng Zhou, Seungwhan Moon, Paul Crook, Bing Liu, Zhou Yu, Eunjoon Cho, Pascale Fung, Zhiguang Wang. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.

Zhaojiang Lin, Bing Liu, Seungwhan Moon, Paul Crook, Zhenpeng Zhou, Zhiguang Wang, Zhou Yu, Andrea Madotto, Eunjoon Cho, Rajen Subba. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

Kai Sun, Seungwhan Moon, Paul Crook, Stephen Roller, Becka Silvert, Bing Liu, Zhiguang Wang, Honglei Liu, Eunjoon Cho, Claire Cardie. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2021.

Open-ended human learning and information-seeking are increasingly mediated by digital assistants. However, such systems often ignore the user's pre-existing knowledge. Assuming a correlation between engagement and user responses such as "liking" messages or asking followup questions, we design a Wizard-of-Oz dialog task that tests the hypothesis that engagement increases when users are presented with facts related to what they know. Through crowd-sourcing of this experiment, we collect and release 14K dialogs (181K utterances) where users and assistants converse about geographic topics like geopolitical entities and locations. This dataset is annotated with pre-existing user knowledge, message-level dialog acts, grounding to Wikipedia, and user reactions to messages. Responses using a user's prior knowledge increase engagement. We incorporate this knowledge into a multi-task model that reproduces human assistant policies and improves over a BERT content model by 13 mean reciprocal rank points.

We propose a simple but strong baseline for time series classification from scratch with deep neural networks. Our proposed baseline models are pure end-to-end without any heavy preprocessing on the raw data or feature crafting. The proposed Fully Convolutional Network (FCN) achieves premium performance to other state-of-the-art approaches and our exploration of the very deep neural networks with the ResNet structure is also competitive. The global average pooling in our convolutional model enables the exploitation of the Class Activation Map (CAM) to find out the contributing region in the raw data for the specific labels. Our models provides a simple choice for the real world application and a good starting point for the future research. An overall analysis is provided to discuss the generalization capability of our models, learned features, network structures and the classification semantics.

Symbolic Aggregate approximation (SAX) has been the de facto standard representation methods for knowledge discovery in time series on a number of tasks and applications. So far, very little work has been done in empirically investigating the intrinsic properties and statistical mechanics in SAX w ords. In this paper, we applied several statistical measurements and proposed a new statistical measurement, i.e. information embedding cost (IEC) to analyze the statistical behaviors of the symbolic dynamics. Our experiments on the benchmark datasets and the clinical signals demonstrate that SAX can always reduce the complexity while preserving the core information embedded in the original time series with significant embedding efficiency, as well as robust to missing values and noise. Our proposed IEC score provide a priori to determine if SAX is adequate for specific dataset, which can be generalized to evaluate other symbolic representations. Our work provides an analytical framework with several statistical tools to analyze, evaluate and further improve the symbolic dynamics for knowledge discovery in time series.

This paper proposes a set of new error criteria and a learning approach, called Adaptive Normalized Risk-Averting Training (ANRAT) to attack the non-convex optimization problem in training deep neural networks without pretraining. Theoretically, we demonstrate its effectiveness based on the expansion of the convexity region. By analyzing the gradient on the convexity index $\lambda$, we explain the reason why our learning method using gradient descent works. In practice, we show how this training method is successfully applied for improved training of deep neural networks to solve visual recognition tasks on the MNIST and CIFAR-10 datasets. Using simple experimental settings without pretraining and other tricks, we obtain results comparable or superior to those reported in recent literature on the same tasks using standard ConvNets + MSE/cross entropy. Performance on deep/shallow multilayer perceptron and Denoised Auto-encoder is also explored. ANRAT can be combined with other quasi-Newton training methods, innovative network variants, regularization techniques and other common tricks in DNNs. Other than unsupervised pretraining, it provides a new perspective to address the non-convex optimization strategy in training DNNs.

Standard Symbolic Aggregation Approximation (SAX) is at the core of many effective time series data mining algorithms. Its combination with Bag-of-Patterns (BoP) has become the standard approach with state-of-the-art performance on standard datasets. However, standard SAX with the BoP representation might neglect internal temporal correlation embedded in the raw data. In this paper, we proposed time warping SAX, which extends the standard SAX with time delay embedding vector approaches to account for temporal correlations. We test time warping SAX with the BoP representation on 12 benchmark datasets from the UCR Time Series Classification/Clustering Collection. On 9 datasets, time warping SAX overtakes the state-of-the-art performance of the standard SAX. To validate our methods in real world applications, a new dataset of vital signs data collected from patients who may require blood transfusion in the next 6 hours was tested. All the results demonstrate that, by considering the temporal internal correlation, time warping SAX combined with BoP improves classification performance.