Due to the immense resource demands and the involved complex techniques, it is still challenging for successfully pre-training a large language models (LLMs) with state-of-the-art performance. In this paper, we explore the key bottlenecks and designs during pre-training, and make the following contributions: (1) a comprehensive investigation into the factors contributing to training instability; (2) a robust optimization approach designed to mitigate training instability effectively; (3) an elaborate data pipeline that integrates data synthesis, data curriculum, and data selection. By integrating the above techniques, we create a rather low-cost training recipe and use it to pre-train YuLan-Mini, a fully-open base model with 2.4B parameters on 1.08T tokens. Remarkably, YuLan-Mini achieves top-tier performance among models of similar parameter scale, with comparable performance to industry-leading models that require significantly more data. To facilitate reproduction, we release the full details of training recipe and data composition. Project details can be accessed at the following link: https://anonymous.4open.science/r/YuLan-Mini/README.md.

The standardization degree of traditional Chinese medicine clinical data in the real world is low and heterogeneous data aggregation among institutions is difficult, which leads to the difficulty of sharing clinical data and scientific research data of traditional Chinese medicine. This paper designs and implements a real world traditional Chinese medicine clinical scientific research information electronic medical record sharing system. The system consists mainly of two subsystems, namely electronic medical record collection system and electronic medical record integration system. The collection system can collect, normalize and structured storage inpatient electronic medical records, outpatient electronic medical records and cloud platform electronic medical records. The integration system can integrate heterogeneous data from different traditional Chinese medicine diagnostic and treatment institutions to realize the sharing of traditional Chinese medicine clinical and research data.

Among several new technologies, such as social and cognitive mobile computing, wireless sensor networks (WSNs) constitute the founding pillar of the industrial Internet of Things. These networks are expected to play an increasingly important role in our daily lives. Social and cognitive mobile computing requires the sharing of data recorded by sensor nodes. However, the data can be vulnerable to attacks. It is of utmost importance to protect the users privacy while ensuring the security of the WSNs. This investigation is focused on the source-location privacy (SLP) of WSNs. This article proposes a dynamic multipath privacy-preserving routing (DMPPR) scheme based on multiple sinks for protecting the privacy. Different from single sink schemes, the technique of using multiple sink nodes to protect SLP is discussed in this article. Furthermore, a packet-slicing transmission scheme that generates a large number of dynamic routings based on multiple sink nodes is adopted for transmitting the packets. Local adversaries are considered, and to cope with these adversaries, a transmission loop, constructed using real and fake packets, is proposed to confuse the adversaries during the source detection process. The aim is to break the sociality between the sensor nodes. Simulations performed in MATLAB show that the proposed method outperforms similar existing schemes in terms of the secure time, adversary's capture probability, and node utilization ratio. Moreover, the DMPPR scheme also reduces energy consumption by allowing more nodes in the nonhotspot areas to participate in the packet transmission process.

The great success of supervised learning has initiated a paradigm shift from building a deterministic software system to a probabilistic artificial intelligent system throughout the industry. The historical records in enterprise domains can potentially bootstrap the traditional business into the modern data-driven approach almost everywhere. The introduction of the Deep Neural Networks (DNNs) significantly reduces the efforts of feature engineering so that supervised learning becomes even more automated. The last bottleneck is to ensure the data quality, particularly the label quality, because the performance of supervised learning is bounded by the errors present in labels. In this paper, we present a new Active Deep Denoising (ADD) approach that first builds a DNN noise model, and then adopts an active learning algorithm to identify the optimal denoising function. We prove that under the low noise condition, we only need to query the oracle with log n examples where n is the total number in the data. We apply ADD on one enterprise application and show that it can effectively reduce 1/3 of the prediction error with only 0.1% of examples verified by the oracle.

An exciting network called smart IoT has great potential to improve the level of our daily activities and the communication. Source location privacy is one of the critical problems in the wireless sensor network (WSN). Privacy protections, especially source location protection, prevent sensor nodes from revealing valuable information about targets. In this paper, we first discuss about the current security architecture and attack modes. Then we propose a scheme based on cloud for protecting source location, which is named CPSLP. This proposed CPSLP scheme transforms the location of the hotspot to cause an obvious traffic inconsistency. We adopt multiple sinks to change the destination of packet randomly in each transmission. The intermediate node makes routing path more varied. The simulation results demonstrate that our scheme can confuse the detection of adversary and reduce the capture probability.

Personalization is a long-standing problem in data mining and machine learning. Companies make personalized product recommendations to millions of users every second. In addition to the recommendation problem, with the emerging of personal devices, many conventional problems, e.g., recognition, need to be personalized as well. Moreover, as the number of users grows huge, solving personalization becomes quite challenging. In this paper, we formalize the generic personalization problem as an optimization problem. We propose several ADMM algorithms to solve this problem in a distributed way including a new Asynchronous ADMM that removes all synchronous barriers to maximize the training throughput. We provide a mathematical analysis to show that the proposed Asynchronous ADMM algorithm holds a linear convergence rate which is the best to our knowledge. The distributed personalization allows training to be performed in either a cluster or even on a user's device. This can improve the privacy protection as no personal data is uploaded, while personal models can still be shared with each other. We apply this approach to two industry problems, \emph{Facial Expression Recognition} and \emph{Job Recommendation}. Experiments demonstrate more than 30\% relative error reduction on both problems. Asynchronous ADMM allows faster training for problems with millions of users since it eliminates all network I/O waiting time to maximize the cluster CPU throughput. Experiments demonstrate 4 times faster than original synchronous ADMM algorithm.

A fundamental problem in biological and machine vision is visual invariance: How are objects perceived to be the same despite transformations such as translations, rotations, and scaling? In this letter, we describe a new, unsupervised approach to learning invariances based on Lie group theory. Unlike traditional approaches that sacrifice information about transformations to achieve invariance, the Lie group approach explicitly models the effects of transformations in images. As a result, estimates of transformations are available for other purposes, such as pose estimation and visuomotor control. Previous approaches based on first-order Taylor series expansions of images can be regarded as special cases of the Lie group approach, which utilizes a matrix-exponential-based generative model of images and can handle arbitrarily large transformations. We present an unsupervised expectation-maximization algorithm for learning Lie transformation operators directly from image data containing examples of transformations. Our experimental results show that the Lie operators learned by the algorithm from an artificial data set containing six types of affine transformations closely match the analytically predicted affine operators. We then demonstrate that the algorithm can also recover novel transformation operators from natural image sequences. We conclude by showing that the learned operators can be used to both generate and estimate transformations in images, thereby providing a basis for achieving visual invariance.