Machine learning (ML) has become a cornerstone of key applications at ByteDance. As model complexity and data volumes surge, data management for large-scale ML workloads faces substantial challenges, particularly with recent advances in large recommendation models (LRMs) and large multimodal models (LMMs). Traditional approaches exhibit limitations in storage efficiency, metadata scalability, update mechanisms, and integration with ML frameworks. To address these challenges, we propose Magnus, a holistic data management system built upon Apache Iceberg. Magnus integrates innovative optimizations across resource-efficient storage formats optimized for large wide tables and multimodal data, built-in support for vector and inverted indexes to accelerate data retrieval, scalable metadata planning with Git-like branching and tagging capabilities, and high-performance update/upsert based on lightweight merge-on-read (MOR) strategies. Additionally, Magnus provides native support and specialized enhancement for LRM and LMM training workloads. Experimental results demonstrate significant performance gains in real-world ML scenarios. Magnus has been deployed at ByteDance for over five years, enabling robust and efficient data infrastructure for large-scale ML workloads.

GeaFlow is a distributed dataflow system optimized for streaming graph processing, and has been widely adopted at Ant Group, serving various scenarios ranging from risk control of financial activities to analytics on social networks and knowledge graphs. It is built on top of a base with full-fledged stateful stream processing capabilities, extended with a series of graph-aware optimizations to address the space explosion and programming complexity issues of conventional join-based approaches. We propose new state backends and streaming operators that facilitate processing on dynamic graph-structured datasets, reducing space consumed by states. We develop a hybrid domain-specific language that embeds Gremlin into SQL, supporting both table and graph abstractions over streaming data. In addition to streaming workloads, GeaFlow is also extensively used for some batch processing jobs. In the largest deployments to date, GeaFlow is able to process tens of millions of events per second and manage hundreds of terabytes of states.

Network connectivity, with its significant application value for data transmission and node cooperation, has drawn a great concern in recent years. Facing the heterogeneity and complexity of the IoT system, the connectivity determination between nodes in the network is a big challenge. In view of this, this article proposes a generic and efficient connectivity determination method for IoT applications. This method first characterizes the connectivity parameters of nodes, including the direct connection probabilities between nodes, the degree centrality, and the betweenness centrality of nodes, and based on them, then constructs a node connectivity random graph (NCRG) and splits the NCRG into separate components. Furthermore, it converts the connectivity between nodes located in different components into the connectivity between these components and provides an algorithm to determine their connectivity. Specifically, three testing rules are defined in the algorithm to rank the testing priorities of these components and testing edges between these components. The simulation results show that the proposed method can efficiently achieve high accuracy with less cost.