Multimodal information extraction (MIE) constitutes a set of essential tasks aimed at extracting structural information from Web texts with integrating images, to facilitate the structural construction of Web-based semantic knowledge. To address the expanding category set including newly emerging entity types or relations on websites, prior research proposed the zero-shot MIE (ZS-MIE) task which aims to extract unseen structural knowledge with textual and visual modalities. However, the ZS-MIE models are limited to recognizing the samples that fall within the unseen category set, and they struggle to deal with real-world scenarios that encompass both seen and unseen categories. The shortcomings of existing methods can be ascribed to two main aspects. On one hand, these methods construct representations of samples and categories within Euclidean space, failing to capture the hierarchical semantic relationships between the two modalities within a sample and their corresponding category prototypes. On the other hand, there is a notable gap in the distribution of semantic similarity between seen and unseen category sets, which impacts the generative capability of the ZS-MIE models. To overcome the above disadvantages, we delve into the generalized zero-shot MIE (GZS-MIE) task and propose the hyperbolic multimodal generative representation learning framework (HMGRL). The variational information bottleneck and autoencoder networks are reconstructed with hyperbolic space for modeling the multi-level hierarchical semantic correlations among samples and prototypes. Furthermore, the proposed model is trained with the unseen samples generated by the decoder, and we introduce the semantic similarity distribution alignment loss to enhance the model's generalization performance. Experimental evaluations on two benchmark datasets underscore the superiority of HMGRL compared to existing baseline methods.

Preserving semantic equivalence between source text and its translation remains a formidable challenge in machine translation. Despite recent advancements, neural machine translation (NMT) models often fall short due to their over-reliance on word-level alignment, a limitation exacerbated by the use of cross-entropy loss. This paper introduces Rosetta, a novel framework that revisits the semantic awareness of NMT Systems. Our approach builds upon the Transformer architecture, incorporating stochastic layer selection, multi-branch attention, and group fusion mechanisms to facilitate a more nuanced understanding of context and meaning. Central to our framework is the utilization of multilingual pre-trained language models (PLMs) to reinforce semantic consistency between the source text and its translation in the semantic space. We evaluate our approach on the IWSLT’14 translation dataset. Our model achieves state-of-the-art performance on the German-English translation task, surpassing existing models with a BLEU score of 39.12. To assess the semantic equivalence of translations, we employ GPT-4 as an independent evaluator. The results demonstrate that our approach surpasses literal translation, excelling in preserving semantic similarity.

Sentence embedding models are typically trained using the Contrastive Learning (CL) method, which works by pulling similar semantics closer and pushing dissimilar ones away. Recent studies have shown that utilizing a multi-teacher ranking distillation approach, which assigns fine-grained rankings to sentences, enables the generation of smoother sentence similarity representations and results in higher-quality sentence embeddings. However, the effectiveness of distillation may be limited by the capacity of the student model. A simple student model with fewer parameters may struggle to approximate a highly complex teacher model, potentially leading to overfitting on certain datasets or specific aspects of the task. To address this, we propose MaCSE, a multi-agent ranking distillation framework that dynamically selects and optimizes teacher model contributions across training stages. MaCSE employs a Centralized Training with Decentralized Execution (CTDE) paradigm, enabling collaborative agent interactions to adaptively adjust teacher fusion weights based on training dynamics. Experimental results on Semantic Textual Similarity and transfer tasks demonstrate that MaCSE outperforms most existing baselines and even rivals methods using large language models for sentence representation. Our implementation is available at GitHub1.

The task of Sign Language Production (SLP) in machine learning involves converting text-based spoken language into corresponding sign language expressions. Sign language conveys meaning through the continuous movement of multiple articulators, including manual and non-manual channels. However, most current Transformer-based SLP models convert these multi-channel sign poses into a unified feature representation, ignoring the inherent structural correlations between channels. This paper introduces a novel approach called MCST-Transformer for skeletal sign language production. It employs multi-channel spatial attention to capture correlations across various channels within each frame, and temporal attention to learn sequential dependencies for each channel over time. Additionally, the paper explores and experiments with multiple fusion techniques to combine the spatial and temporal representations into naturalistic sign sequences. To validate the effectiveness of the proposed MCST-Transformer model and its constituent components, extensive experiments were conducted on two benchmark sign language datasets from diverse cultures. The results demonstrate that this new approach outperforms state-of-the-art models on both datasets.

Abstract Recent advancements in scene text recognition have predominantly focused on leveraging textual semantics. However, an over-reliance on linguistic priors can impede a model’s ability to handle irregular text scenes, including non-standard word usage, occlusions, severe distortions, or stretching. The key challenges lie in effectively localizing occlusions, perceiving multi-scale text, and inferring text based on scene context. To address these challenges and enhance visual capabilities, we introduce the Graph Reasoning Model (GRM). The GRM employs a novel feature fusion method to align spatial context information across different scales, beginning with a feature aggregation stage that extracts rich spatial contextual information from various feature maps. Visual reasoning representations are then obtained through graph convolution. We integrate the GRM module with a language model to form a two-stream architecture called GRNet. This architecture combines pure visual predictions with joint visual-linguistic predictions to produce the final recognition results. Additionally, we propose a dynamic iteration refinement for the language model to prevent over-correction of prediction results, ensuring a balanced contribution from both visual and linguistic cues. Extensive experiments demonstrate that GRNet achieves state-of-the-art average recognition accuracy across six mainstream benchmarks. These results highlight the efficacy of our multi-modal approach in scene text recognition, particularly in challenging scenarios where visual reasoning plays a crucial role.

Abstract Malware family prediction has been mainly formulated as a multiclass classification to predict one malware family. This approach suffers from label uncertainty, which can mislead malware analysts. To render malware prediction less susceptible to uncertainty, malware family prediction, which entails predicting one or more families, is performed in this study. In this regard, an encoder–decoder malware family prediction model, EnDePMal, with label uncertainty awareness, is proposed. EnDePMal aims to predict all malware families related to samples and preserve their priorities. It comprises a residual neural network-based encoder and a long short-term memory-based decoder with an attention mechanism. The model uses a sequence of malware family names, but not a family name, as a label. Once a visualized malware image is input into EnDePMal, its encoder extracts the important features from the image. Subsequently, its decoder generates family names, where the attention mechanism allows it to focus on relevant features by attending to the encoder’s output. Experimental results show that EnDePMal can predict 77.64% of malware family sequences that preserve their priorities. Moreover, it achieves an accuracy of 93.49% and an F1-score of 0.9282 for malware families with the highest priority, rendering it comparable to the typical multiclass classification model.

Abstract Sign Language Production (SLP) refers to the task of translating textural forms of spoken language into corresponding sign language expressions. Sign languages convey meaning by means of multiple asynchronous articulators, including manual and non‐manual information channels. Recent deep learning‐based SLP models directly generate the full‐articulatory sign sequence from the text input in an end‐to‐end manner. However, these models largely down weight the importance of subtle differences in the manual articulation due to the effect of regression to the mean. To explore these neglected aspects, an efficient cascade dual‐decoder Transformer (CasDual‐Transformer) for SLP is proposed to learn, successively, two mappings SLP hand : Text → Hand pose and SLP sign : Text → Sign pose , utilising an attention‐based alignment module that fuses the hand and sign features from previous time steps to predict more expressive sign pose at the current time step. In addition, to provide more efficacious guidance, a novel spatio‐temporal loss to penalise shape dissimilarity and temporal distortions of produced sequences is introduced. Experimental studies are performed on two benchmark sign language datasets from distinct cultures to verify the performance of the proposed model. Both quantitative and qualitative results show that the authors’ model demonstrates competitive performance compared to state‐of‐the‐art models, and in some cases, achieves considerable improvements over them.

Abstract Rapidly evolving malware has become a major cybersecurity threat. Several feature‐engineering techniques have been proposed to defend against malware attacks. An entropy is a typical indicator used in identifying malware. Structural entropy is a sequence of entropy values where an entropy of a segment is calculated by the equation of the entropy itself. However, entropy‐based features are likely to be abstract and miss important information. This article proposes a feature engineering technique that involves the concept of structural entropy. This technique allows every segment to be represented as 256 entropy values for every byte value, but not as an entropy value. Our research, fine‐granularity structural entropy (FiG_SE), incorporates global patterns across all segments, local patterns across adjacent segments, and internal patterns within the segments. To extract higher‐level characteristics from our entropy feature, we use a convolutional neural network (CNN) architecture because it is effective for extracting local and global patterns, and especially for shift‐invariant patterns. Our malware classification based on CNN with the proposed feature outperforms the previous classification methods that use byte streams, entropy streams, and structural‐entropy‐based streams as inputs. Moreover, our research combined with CNN is highly resilient to obfuscation techniques and is also well suited to malware detection.

All-flash array is a popular memory device available for use in modern high-performance storage systems. Compared with other types of devices such as DRAM, NVRAM, and EEPROM, flash array combines the best features: shock resistance, low cost, low power consumption, and fast access. Moreover, the ever-increasing density of flash memory has led to a dramatic increase in the capacity, which allows the storage of large volume of data. However, flash memory is not optimal for managing a large number of small files because: 1. the small and random write operation is inefficient in flash memory; 2. massive metadata information occupies a significant portion of the namespace, which is relatively limited or scarce in big data storage systems. This paper introduces a novel approach, hash partitioning-based file compaction (HFC), to improve the efficiency of storing and accessing small files in all-flash storage systems. HFC consists of a file compaction tool and an access interface. The compaction tool merges a group (usually a directory) of small files into a set of "big files" to reduce the metadata required to be maintained in the on-chip memory. The data locality and tree structure of those small files are preserved. The access interface is designed to provide transparent access to the small files in the HFC big files. Experimental results confirm that the proposed method significantly enhances the efficiency of managing massive amounts of small files in flash memory in terms of namespace usage and access speed.

As Internet of Things (IoT) applications are getting attention these days, the importance of firmware security is also growing. However, it is not straightforward to analyze the bugs or vulnerabilities that reside in firmware. One of the major challenges is to detect information about hardware architectures of compressed firmware. Traditional analysis tools make use of static signatures embedded in the compressed binary code of firmware. However, signature extraction needs the careful elaboration of experts, and it is not always even possible. In this paper, we introduce our experience in analyzing the hardware information of compressed firmware. Since it is not possible to use the semantic information of compressed binary code, we adopt machine learning technologies for this purpose. Despite various difficulties, we have positive experimental results.

Ransomware represents a significant threat to both individuals and organizations. Moreover, the emergence of ransomware that exploits kernel vulnerabilities poses a serious detection challenge. In this paper, we propose a novel ransomware detection mechanism at a storage device, especially a flash-based storage device. To this end, we design a new buffer management policy that allows our detector to identify ransomware behaviors. Our mechanism detects a realistic ransomware sample with little negative impacts on the hit ratios of the buffers internally located in a storage device.

Generative adversarial networks (GANs) are well known both for being unstable to train and for the problem of mode collapse, particularly when trained on data collections containing a diverse set of visual objects. This study introduces an adaptive hyper-parameter learning procedure for GANs as an alternative to the existing static approach. The proposed procedure is designed to mitigate the impact of instability and saturation in the original by dynamically adjusting the ratio of the training steps of both the generator and discriminator. To accomplish this, we track and analyze stable training curves of relatively narrow datasets and use them as the target fitting lines when training more diverse data collections. Experimental results show that the proposed model improves the stability and generates more realistic images.