Graphical user interface (GUI) has become integral to modern society, making it crucial to be understood for human-centric systems. However, unlike natural images or documents, GUIs comprise artificially designed graphical elements arranged to convey specific semantic meanings. Current multi-modal large language models (MLLMs) already proficient in processing graphical and textual components suffer from hurdles in GUI understanding due to the lack of explicit spatial structure modeling. Moreover, obtaining high-quality spatial structure data is challenging due to privacy issues and noisy environments. To address these challenges, we present MP-GUI, a specially designed MLLM for GUI understanding. MP-GUI features three precisely specialized perceivers to extract graphical, textual, and spatial modalities from the screen as GUI-tailored visual clues, with spatial structure refinement strategy and adaptively combined via a fusion gate to meet the specific preferences of different GUI understanding tasks. To cope with the scarcity of training data, we also introduce a pipeline for automatically data collecting. Extensive experiments demonstrate that MP-GUI achieves impressive results on various GUI understanding tasks with limited data. Our codes and datasets are publicly available at https://github.com/BigTaige/MP-GUI.

Representation learning from a floorplan is a fundamental step for various floorplan-related applications, such as retrieval, reconstruction, and generation. Previous works often use single-layer graphs to model room categories and adjacencies as nodes and edges, which can only represent a floorplan in a coarse level without detailed information such as room shape and door/window locations. Thus, we propose SHRAG, a hierarchical semantic graph for floor-plan representation, with two hierarchical graph modules to semantically encode both coarse and detailed information of a floorplan. First, a Detail Graph Module(DGM) was designed to learn detailed contour and attachable elements embedding for each room. Then, a Global Graph Module(GGM) was applied to concatenate and encode both detailed embeddings from DGM and coarse information including room categories and room relations. Results show that the representation learned from SHRAG achieved SOTA on floorplan retrieval tasks. Further, SHRAG performs more satisfying than other floorplan similarity metrics in a comprehensive user study. We also demonstrate that SHRAG can be easily adapted to real-world complex indoor scenes even with furniture.

Learning implicit field as shape representation brings a revolution in 3D shape single view reconstruction, while it suffers from over-smoothness of corners and details. Many involve 2D supervision to solve corners and details. However, they are limited by the number of views and the accuracy of the renderer. Thus, we propose IMC-NET, an IMplicit field with Corner attention NETwork, to reconstruct details and corners of 3D shapes without any 2D supervision. In IMC-NET, we designed a corner attention branch to predict detailed corners of 3D shape under hand-crafted 3D corner points supervision. Then, an aggregation module was presented to differentiably merge the shape corners and implicit fields using a corner weight prediction. In addition, to help further shape reconstruction work on corners and details, we released a new dataset based on Shapenet models, named Shape-CornerNet, including 43784 3D shapes in 13 categories with detailed corner points. Afterward, we quantitatively and qualitatively evaluated our model on Shape-CornerNet. Our IMC-NET achieved state-of-the-art performance on Shape-CornerNet.

Three‐factor multiserver authentication protocols become a prevalence in recent years. Among these protocols, almost all of them do not involve the registration center into the authentication process. To improve the protocol’s efficiency, a common secret key is shared among all severs, which leads to a serious weakness; i.e., we find that these protocols cannot resist the passive attack from the honest‐but‐curious servers. This paper takes Wang et al.’s protocol as an example, to exhibit how an honest‐but‐curious server attacks their protocol. To remedy this weakness, a novel three‐factor multiserver authentication protocol is presented. By introducing the registration center into the authentication process, the new protocol can resist the passive attack from the honest‐but‐curious servers. Security analyses including formal and informal analyses are given, demonstrating the correctness and validity of the new protocol. Compared with related protocols, the new protocol possesses more secure properties and more practical functionalities than others at a relatively low computation cost and communication cost.