Pre-training large Tabular Language Models (TaLMs) on tabular data has shown effectiveness for table understanding tasks. However, training proprietary large TaLMs on a company's private databases requires substantial computational resources. This paper presents an efficient multi-expert TaLM architecture and training method tailored for multi-domain databases and modest infrastructure. This architecture leverages a divide-and-conquer pretraining approach and a sparsely activated fine-tuning paradigm to reduce computation. Using this architecture, we pre-train and fine-tune a TaLM with 10 billion parameters on a banking database under simple computational infrastructures. We apply our TaLM to support various important banking applications, including risk assessment, information prediction, and profit assessment. Compared with previous baselines, our model achieves +29.3% in [email protected]% on risk assessment and +16.5% in accuracy on information prediction, showing great effectiveness and profitability of our model. This model is successfully deployed in WeBank and now supports many real business scenarios.

Temporal data distribution shift is prevalent in the financial text.How can a financial sentiment analysis system be trained in a volatile market environment that can accurately infer sentiment and be robust to temporal data distribution shifts?In this paper, we conduct an empirical study on the financial sentiment analysis system under temporal data distribution shifts using a real-world financial social media dataset that spans three years.We find that the fine-tuned models suffer from general performance degradation in the presence of temporal distribution shifts.Furthermore, motivated by the unique temporal nature of the financial text, we propose a novel method that combines out-of-distribution detection with time series modeling for temporal financial sentiment analysis.Experimental results show that the proposed method enhances the model's capability to adapt to evolving temporal shifts in a volatile financial market.

Human-like biases and undesired social stereotypes exist in large pretrained language models.Given the wide adoption of these models in real-world applications, mitigating such biases has become an emerging and important task.In this paper, we propose an automatic method to mitigate the biases in pretrained language models.Different from previous debiasing work that uses external corpora to finetune the pretrained models, we instead directly probe the biases encoded in pretrained models through prompts.Specifically, we propose a variant of the beam search method to automatically search for biased prompts such that the cloze-style completions are the most different with respect to different demographic groups.Given the identified biased prompts, we then propose a distribution alignment loss to mitigate the biases.Experiment results on standard datasets and metrics show that our proposed Auto-Debias approach can significantly reduce biases, including gender and racial bias, in pretrained language models such as BERT, RoBERTa and ALBERT.Moreover, the improvement in fairness does not decrease the language models' understanding abilities, as shown using the GLUE benchmark.