The advancement of Large Language Models (LLMs) has remarkably pushed the boundaries towards artificial general intelligence (AGI), with their exceptional ability on understanding diverse types of information, including but not limited to images and audio. Despite this progress, a critical gap remains in empowering LLMs to proficiently understand and reason on graph data, which is ubiquitous in Big Data applications such as social networks, knowledge graphs, and molecular databases. Recent studies underscore LLMs' underwhelming performance on fundamental graph reasoning tasks. In this paper, we endeavor to unearth the obstacles that impede LLMs in graph reasoning, pinpointing the common practice of converting graphs into natural language descriptions (Graph2Text) as a fundamental bot tleneck. To overcome this impediment, we introduce GraphLLM, a pioneering end-to-end approach that synergistically integrates graph learning models with LLMs through a novel Dynamic Task Configuration System. This system employs a Hierarchical Graph Processing Pipeline that combines Local Structure Analyzers for node-level features with Global Pattern Synthesizers for graph level understanding, enabling scalable processing of large-scale graph data. Our empirical evaluations across four fundamental graph reasoning tasks validate the effectiveness of GraphLLM. The results exhibit a substantial average accuracy enhancement of 54.44%, alongside a noteworthy context reduction of 96.45% across various graph reasoning tasks, demonstrating significant potential for Big Data graph analytics.

Large Multimodal Models (LMMs) have ushered in a new era in artificial intelligence, merging capabilities in both language and vision to form highly capable \textbf{Visual Foundation Agents} that are postulated to excel across a myriad of tasks. However, existing benchmarks fail to sufficiently challenge or showcase the full potential of LMMs as visual foundation agents in complex, real-world environments. To address this gap, we introduce VisualAgentBench (VAB), a comprehensive and unified benchmark specifically designed to train and evaluate LMMs as visual foundation agents across diverse scenarios in one standard setting, including Embodied, Graphical User Interface, and Visual Design, with tasks formulated to probe the depth of LMMs' understanding and interaction capabilities. Through rigorous testing across 9 proprietary LMM APIs and 9 open models (18 in total), we demonstrate the considerable yet still developing visual agent capabilities of these models. Additionally, VAB explores the synthesizing of visual agent trajectory data through hybrid methods including Program-based Solvers, LMM Agent Bootstrapping, and Human Demonstrations, offering insights into obstacles, solutions, and trade-offs one may meet in developing open LMM agents. Our work not only aims to benchmark existing models but also provides an instrumental playground for future development into visual foundation agents. Code, train, and test data are available at \url{https://github.com/THUDM/VisualAgentBench}.

Ziwei Chai, Guoyin Wang, Jing Su, Tianjie Zhang, Xuanwen Huang, Xuwu Wang, Jingjing Xu, Jianbo Yuan, Hongxia Yang, Fei Wu, Yang Yang. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Abstract Cement hydration kinetics, characterized by heat generation in early-age concrete, poses a modeling challenge. This work proposes a physics-informed neural network (PINN) named PINN-CHK designed for cement hydration kinetics, to predict early-age temperature rises in cement paste. PINN-CHK leverages data-driven solutions to craft a high-fidelity prediction model, encompassing material properties and maturity functions in cement hydration. Trained on heated cement paste data, it simultaneously fits experimental results and underlying physics, yielding a mesh-free simulation. Incorporating governing partial differential equations (PDEs), and initial and boundary conditions into its loss function, PINN-CHK architecture undergoes rigorous benchmark testing, demonstrating unparalleled predictive accuracy compared to conventional deep-learning methods. It excels in predicting complete temperature fields during spatial–temporal cement hydration, achieving a remarkable relative L2 error as low as 0.00341. PINN-CHK achieves exceptional convergence and accuracy with only 5% of the training data, ushering in a new era in this crucial field. This innovative approach bridges the gap between theory and practice, offering an attractive alternative to conventional finite element solvers for enhanced comprehension of cement hydration kinetics and concrete maturity and strength development in cement-based materials.

The ability to perform pixel-wise segmentation on pavement cracks in real-time is paramount in road service condition assessment and maintenance decision-making practices. Recent deep learning detection models are focused on detection accuracy and require a large number of computing sources and long run times. However, highly efficient and accelerated models with acceptable accuracy in real-time pavement crack detection tasks are required but hard to achieve. In this work, we present a customized deep learning model architecture named Efficient Crack Segmentation Neural Network (ECSNet) for accelerated real-time pavement crack detection and segmentation without compromising performance. We introduce some novel parts, including small kernel convolutional layers and parallel max pooling and convolutional operation, into the architecture for crack information quickly extraction and model’s parameter reduction. We test latency and accuracy trade-offs of our proposed model using the DeepCrack Dataset. The results demonstrate strong performance in both accuracy and efficiency compared to other state-of-the-art models including DeepLabV3, FCN, LRASPP, Enet, Unet and DeepCrack. It is promising that ECSNet obtains the second place with an F1 score of (84.45%) and an Intersection over Union (IoU) of 73.08%. Furthermore, our model gains the largest Frames Per Second (FPS) and lowest training time among all the models which is 73.29 and 5011 seconds, respectively. It maintains a good balance between accuracy and efficiency metrics.

Pavement crack segmentation using deep learning methods can improve crack segmentation accuracy, but in many cases the training dataset is lacking or uneven, making it insufficient to train an accurate segmentation model. In this work, an integrated APC-GAN and AttuNet framework is proposed as an automated pavement surface crack pixel-level segmentation solution for small training datasets. First, an Automated Pavement Crack Generative Adversarial Network (APC-GAN) is designed for the pavement cracks data as an image augmentation method, which is modified and improved from a traditional Deep Convolutional Generative Adversarial Network (DCGAN). Then, a novel pixel-level semantic segmentation structure, Attention modified U-Net (AttuNet), is proposed by introducing the attention module into the convolutional network structure. In order to assess the performance of our proposed framework, an open-source dataset DeepCrack is used, which only contains 300 training images. The results show that our proposed APC-GAN could augment the datasets by producing more clear and distinct pavement images than DCGAN and the generated images could feature more diversity than traditional image augmentation methods. APC-GAN demonstrated higher accuracy than DCGAN and traditional image augmentation methods. Apparently, our proposed APC-GAN and AttuNet framework gains the highest value in the evaluation metrices, including recall, F1 score, mean Intersection over Union (mIoU) and mean Pixel Accuracy (mPA) among all models including U-Net, DeepLabv3, FCN, and LRASPP.