To guide consumers to mobile livestreams that involve their interested products, it is necessary to have a good understanding of livestreaming contents, and the key task is to accurately recognize the products being promoted by streamers. The mainstream cloud-based service framework is challenged by the high concurrency of requests, the high overhead of multimodal recognition, and the requirement of low latency. To break these bottlenecks, we propose a new device-cloud collaborative service framework, where each streamer's mobile device holds unimodal model that can process most of video frames and also uploads the extracted unimodal features to facilitate the cloud-side multimodal recognition of the remaining few frames. On-device unimodal model is further incrementally trained over the samples constructed by leveraging the streamers' manual labeling behaviors, leading to personalized version that can adapt to the heterogeneous and dynamic livestreaming contents of different streamers. Nevertheless, the device-side personalized unimodal features are misaligned in feature space and cannot be directly fused into the cloud-side multimodal model. We thus design a pluggable prompt generation module to transform the personalized unimodal features into prompt embeddings and prepend them to the original inputs of the multimodal backbone network, instructing feature fusion in a streamer-specific manner. We extensively evaluate on the public Fashion-Gen dataset and an industrial dataset collected from Taobao Live. We also perform online testing and practical overhead testing in Taobao Live. Evaluation results reveal the effectiveness and efficiency of our design as well as the consistent advantage over existing baselines.

Federated learning is a new distributed machine learning framework, where numerous heterogeneous clients collaboratively train a model without sharing training data. In this work, we consider a practical and ubiquitous issue when deploying federated learning in mobile environments: intermittent client availability, where the set of eligible clients may change during the training process. Such intermittent client availability would seriously deteriorate the performance of the classical federated averaging algorithm (FedAvg). Thus, we propose a simple distributed nonconvex optimization algorithm, called federated latest averaging (FedLaAvg), which leverages the latest gradients of all clients, even when the clients are not available, to jointly update the global model in each iteration. Our theoretical analysis shows that FedLaAvg achieves guaranteed convergence and a sublinear speedup with respect to the total number of clients. We implement FedLaAvg along with several baselines and evaluate them over the benchmarking MNIST and Sentiment140 data sets. The evaluation results demonstrate that FedLaAvg achieves more stable training than FedAvg in both convex and nonconvex settings and reaches a sublinear speedup. Source code and online supplement are available at the IJOC GitHub site ( http://dx.doi.org/10.1287/ijoc.2022.0057.cd , https://github.com/INFORMSJoC/2022.0057 ). History: Accepted by Ram Ramesh, Area Editor for Data Science & Machine Leaning. Funding: This work was supported by the National Key R&D Program of China [Grant 2022ZD0119100], the National Natural Science Foundation of China (NSFC) [Grants 61972252, 61972254, 62072303, 62025204, 62132018, 62202296, and 62202297], the Alibaba Innovation Research (AIR) Program, and the Tencent Rhino Bird Key Research Project. Supplemental Material: The software that supports the findings of this study is available within the paper and its Supplemental Information ( https://pubsonline.informs.org/doi/suppl/10.1287/ijoc.2022.0057 ) as well as from the IJOC GitHub software repository ( https://github.com/INFORMSJoC/2022.0057 ). The complete IJOC Software and Data Repository is available at https://informsjoc.github.io/ .

Mobile livestreaming has revolutionized the online shopping paradigm, enabling streamers to promote products to consumers with an immersive and interactive experience. To guide consumers to the livestreams that involve their interested products, it is necessary to have a good understanding of livestreaming contents with low latency, and the key task is to accurately recognize the products being promoted by the streamers. However, the mainstream cloud-based service framework is challenged by the high concurrency of service requests, the high overhead of multimodal recognition, and the requirement of low response latency. To break the bottleneck, we propose a new device-cloud collaborative learning framework, where each streamer’s mobile device holds a unimodal recognition model that can process most of frames and also uploads the extracted unimodal features to facilitate the cloud-side multimodal recognition of the remaining few frames. In addition, the on-device unimodal model is incrementally trained over the samples constructed by leveraging the streamers’ manual labeling behaviors, thereby adapting to the heterogeneous and dynamic livestreaming contents of different streamers. Nevertheless, the device-side personalized unimodal features are misaligned in feature space and cannot be directly fused into the cloud-side multimodal model. We thus design a pluggable prompt generation module to transform the personalized unimodal features into prompt embeddings, instructing the multimodal backbone network in feature fusion. Both offline and online evaluation results reveal the effectiveness and efficiency of our design as well as its consistent advantage over existing baselines.

Cloud-based learning is currently the mainstream in both academia and industry. However, the global data distribution, as a mixture of all the users' data distributions, for training a global model may deviate from each user's local distribution for inference, making the global model non-optimal for each individual user. To mitigate distribution discrepancy, on-device training over local data for model personalization is a potential solution, but suffers from serious overfitting. In this work, we propose a new device-cloud collaborative learning framework under the paradigm of domain adaption, called MPDA, to break the dilemmas of purely cloud-based learning and on-device training. From the perspective of a certain user, the general idea of MPDA is to retrieve some similar data from the cloud's global pool, which functions as large-scale source domains, to augment the user's local data as the target domain. The key principle of choosing which outside data depends on whether the model trained over these data can generalize well over the local data. We theoretically analyze that MPDA can reduce distribution discrepancy and overfitting risk. We also extensively evaluate over the public MovieLens 20M and Amazon Electronics datasets, as well as an industrial dataset collected from Mobile Taobao over a period of 30 days. We finally build a device-tunnel-cloud system pipeline, deploy MPDA in the icon area of Mobile Taobao for click-through rate prediction, and conduct online A/B testing. Both offline and online results demonstrate that MPDA outperforms the baselines of cloud-based learning and on-device training only over local data, from multiple offline and online metrics.