Recent years have witnessed an exponential growth of model scale in deep learning-based recommender systems---from Google's 2016 model with 1 billion parameters to the latest Facebook's model with 12 trillion parameters. Significant quality boost has come with each jump of the model capacity, which makes us believe the era of 100 trillion parameters is around the corner. However, the training of such models is challenging even within industrial scale data centers. We resolve this challenge by careful co-design of both optimization algorithm and distributed system architecture. Specifically, to ensure both the training efficiency and the training accuracy, we design a novel hybrid training algorithm, where the embedding layer and the dense neural network are handled by different synchronization mechanisms; then we build a system called Persia (short for parallel recommendation training system with hybrid acceleration) to support this hybrid training algorithm. Both theoretical demonstrations and empirical studies with up to 100 trillion parameters have been conducted to justify the system design and implementation of Persia. We make Persia publicly available (at github.com/PersiaML/Persia) so that anyone can easily train a recommender model at the scale of 100 trillion parameters.

Ground-penetrating radar (GPR) has been widely applied to the nondestructive inspection of concrete structures such as tunnel lining, bridge deck, and retaining wall, which are usually reinforced by steel bars. The scattering of electromagnetic (EM) waves caused by the dense steel rebar embedded in the concrete structures has a severe influence on the penetration capacity of GPR waves. In this letter, the scattering and penetration characteristics of EM waves propagating through rebar net are investigated via both numerical and laboratory experiments, with an aim to select the antenna nominal frequency for a different reinforcement density. The results show that the rebar, which is perpendicular to the polarization direction of GPR waves and has a very small diameter compared with the wavelength, is almost transparent to the impinged GPR waves. The scattering and interaction of GPR waves caused by the rebar that is parallel to the polarization direction result in a shielding effect, which is manifested as a blind band in the low-frequency range in the transmitted spectrum. This result violates the rule of thumb commonly used in the GPR community, i.e., the lower frequency has a deeper GPR penetration depth. In the end, a low cutoff frequency is recommended for selecting a GPR antenna with an appropriate nominal frequency when it is used in the detection of an anomaly inside and behind a reinforced concrete structure, in which the spacing of the rebar net is known.

Lifelong topic model (LTM), an emerging paradigm for never-ending topic learning, aims to yield higher-quality topics as time passes through knowledge accumulated from the past yet learned for the future. In this paper, we propose a novel lifelong topic model based on non-negative matrix factorization (NMF), called Affinity Regularized NMF for LTM (NMF-LTM), which to our best knowledge is distinctive from the popular LDA-based LTMs. NMF-LTM achieves lifelong learning by introducing word-word graph Laplacian as semantic affinity regularization. Other priors such as sparsity, diversity, and between-class affinity are incorporated as well for better performance, and a theoretical guarantee is provided for the algorithmic convergence to a local minimum. Extensive experiments on various public corpora demonstrate the effectiveness of NMF-LTM, particularly its human-like behaviors in two carefully designed learning tasks and the ability in topic modeling of big data. A further exploration of semantic relatedness in knowledge graphs and a case study on a large-scale real-world corpus exhibit the strength of NMF-LTM in discovering high-quality topics in an efficient and robust way.