In this paper, we propose a Modality-aware Diffusion Distillation Network (MDDN), consisting of Diffusion-based Modality Imputation (DMI) and Margin-aware Distillation (MAD) modules, for multimodal sentiment analysis under uncertain missing modalities. More specifically, the DMI module incrementally adds Gaussian noise into the modality-specific distribution space of the available data and recovers missing modalities while adhering to their original distributions, enabling effective imputation of missing data for the network. Furthermore, the MAD module introduces the classification uncertainty of generated samples and available data to reweigh their contributions to the loss function. Consequently, the missing modalities can be reconstructed by MDDN, and be fused dynamically with the existing ones for sentiment analysis. Extensive experiments on the MOSI and MOSEI datasets demonstrate the effectiveness and robustness of the proposed network for dealing with missing modalities in sentiment analysis.

Multimodal Aspect-based Sentiment Analysis aims at identifying specific aspects like celebrities, locations, and organizations, along with their corresponding sentiments. Typically, existing methods comprehend each aspect within images and their relationship without external knowledge. To this end, we propose an Aspect-Attentioned Prompting Framework (AAPF) with Vision-language Prompting (VP) and Attention Aggregation (AA) modules to leverage the fine-grained visual semantics corresponding to the conditional sentences. More specifically, the VP constructs aspect-oriented questions from the perspectives of objects, relations, and sentence-oriented questions, leveraging chain-of-thought prompting to guide the reasoning process. Given the questions, sentences, and images, a visual language model is exploited to perceive the aspect-sensitive intricate semantics, which are further filtered by CLIP scores and condensed into an aspect-relevant summarization. Furthermore, the AA module conducts complementary cross-attention on the summarizations and the conditional sentences to model their interactions by achieving aspect-attentioned features, which can be used to make sentiment predictions. Experiments conducted on two public datasets indicate that our approach achieves superior performance against the state-of-the-art methods.

The remarkable capability of generative methods to synthesize public-domain speeches that are difficult to differentiate from natural ones has raised concerns about the proliferation of disinformation. To this end, we contribute a fake speech detection dataset, denoted as the SynSpeech dataset, which includes fully fake and partially fake speeches in multiple domains covering both GANs-based and diffusion-based generative methods, serving as a prerequisite to support fake speech detectors. For the partially fake speeches, we utilize a large language model to replace interest segments like individual names, locations, adjectives, and verbs. Additionally, these segments will be substituted with fabricated clips synthesized by generative methods to create sentences with completely opposite meanings as partially fake speech. More specifically, a total of 164,924 fully fake speeches from six distinct generators, 44,070 real speeches from 108 individuals, and 99,496 partially fake speeches have been annotated as three kinds of labels, i.e., fully fake, real, and partially fake, respectively. In the experiments, we investigate quite a few fake speech detection models and customize them for two fake speech detection tasks for comparisons.

Visual Word Sense Disambiguation (Visual-WSD), as a sub-task of fine-grained image-text retrieval, requires a high level of language-vision understanding to capture and exploit the nuanced relationships between text and visual features. However, the cross-linguistic background only with limited contextual information is considered the most significant challenges for this task. In this paper, we propose MTA, which employs a new approach for multilingual contrastive learning with self-distillation to align fine-grained textual features to fixed vision features and align non-English textual features to English textual momentum features. It is a lightweight and end-to-end model since it does not require updating the visual encoder or translation operations. Furthermore, a trilingual fine-grained image-text dataset is developed and a ChatGPT API module is integrated to enrich the word senses effectively during the testing phase. Extensive experiments show that MTA achieves state-of-the-art results on the benchmark English, Farsi, and Italian datasets in SemEval-2023 Task 1 and exhibits impressive generalization abilities when dealing with variations in text length and language.

In this paper, we propose a two-stage network with modality-specific and -shared contrastive learning (MMCL) for multimodal sentiment analysis. MMCL comprises a category-aware modality-specific contrastive (CMC) module and a self-decoupled modality-shared contrastive (SMC) module. In the first stage, the CMC module guides the encoders to extract modality-specific representations by constructing positive-negative pairs according to sample categories. In the second stage, the SMC module guides the encoders to extract modality-shared representations by constructing positive-negative pairs based on modalities and decoupling the self-contrast of all modalities. In the aforementioned modules, we leverage self-modulation factors to focus more on hard positive pairs through assigning different loss weights to positive pairs depending on their distance. In particular, we introduce a dynamic routing algorithm to cluster the inputs of the contrastive modules during training, where a gradient stopping strategy is utilized to isolate the backpropagation process of the CMC and SMC modules. Extensive experiments on the CMU-MOSI and CMU-MOSEI datasets show that MMCL achieves the state-of-the-art performance.

The cross-attention mechanism has been widely employed in the multimodal rumor detection task, which is computation-intensive and suffers from the restricted modal receptive field. In this paper, we propose a multimodal rumor detection model (MKV), which maps multimodal key semantics with discrimination into feature vectors for rumor detection. More specifically, MKV extracts high-dimensional features for each modality separately by the Multimodal Feature Extractor (MFE). The mapping mechanism learns low-dimensional mapping scheme (Map) and key semantics (Key) with discrimination from the different modal features respectively. Subsequently, the Map and Key jointly construct a state matrix (State) containing all possible permutations of modalities. In particular, a max pooling operation is performed on State and products a feature vector (Vector). The mapping mechanism is able to incrementally learn the discriminative semantics by stacking manner. Vectors from the stacking process are leveraged in the Rumor Detection module (RD). Extensive experiments on two public datasets show that the MKV achieves the state-of-the-art performance.

Misinformation detection in short videos on social media has become a pressing issue due to its popularity. However, datasets for misinformation detection are limited in terms of modality and sources, hindering the development of effective detection methods. In this article, we introduce a novel dataset denoted the multiplatform multimodal misinformation (3M) dataset. Our dataset is collected specifically to investigate and address misinformation in a multimodal context. A total of 17 352 videos were collected from two prominent social media platforms, namely TikTok and Weibo. The 3M dataset covers 30 different topics, such as sports, health, news, and art, providing a diverse range of content for analysis. We propose a novel approach named cross-modal attention misinformation detection (CAMD) for effectively detecting and addressing multimodal misinformation. CAMD leverages the cross-modal attention module to facilitate effective information exchange and fusion between modalities by learning the correlations and weights among them. The cross-modal attention module is capable of learning multilevel modality correlations, focuses primarily on the interaction between multimodal sequences across different time steps, and simultaneously adjusts the information from the source modality based on the information of the target modality. Extensive experiments on the 3M dataset show that the proposed method achieves state-of- the-art performance. Specifically, CAMD achieves accuracy, F1-score, precision, and recall values of 76.86%, 58.05%, 87.86%, and 58.70%, respectively, on the 3M dataset.

This research paper presents a comprehensive exploration of the application of Generative Artificial Intelligence (GAI) within the realm of language education. Through a systematic review of 20 empirical studies conducted within a structured three-step framework, this study elucidates the multifaceted integration of GAI in language learning. The examination encompasses various dimensions, including target languages, learners' educational levels, GAI applications, language skills, and practical outcomes. Additionally, the study critically assesses the key advantages and challenges inherent in GAI's role in language education. The paper provides valuable insights into human-technology interaction by delving into language learners' attitudes toward GAI. Notably, this research identifies three pivotal roles GAI assumes within the language learning process, and they are co-author, evaluator, and learning materials provider. The study concludes by charting a path toward future research endeavors within the evolving landscape of GAI-based language learning by implicating the future research direction of integrating the GAI into language education, including collaborations between humans and GAI, clarifying the definition of GAI-powered plagiarism, GAI-based language activities design, prompting strategies, and digital literacy.

A citation is a reference to the source of information used in an article. Citations are very useful for students and researchers to locate relevant information on a topic. Proper citation is also important in the academic ethics of article writing. Due to the rapid growth of scientific works published each year, how to automatically recommend citations to students and researchers has become an interesting but challenging research problem. In particular, a citation recommendation system can assist students to identify relevant papers and literature for academic writing. Citation recommendation can be classified into local and global citation recommendation depending on whether a specific local citation context is given; e.g., the text surrounding a citation placeholder. This article provides a systematic review on global citation recommendation models and compares the reviewed methods from the traditional topic- based models to the recent models embedded with deep neural networks, aiming to summarize this field to facilitate researchers working on citation recommendation.

Recently, graph neural networks (GNNs) have achieved promising results in session-based recommendation. Existing methods typically construct a local session graph and a global session graph to explore complex item transition patterns. However, studies have seldom investigated the repeat consumption phenomenon in a local graph. In addition, it is challenging to retrieve relevant adjacent nodes from the whole training set owing to computational complexity and space constraints. In this study, we use a GNN to jointly model intra- and inter-session item dependencies for session-based recommendations. We construct a repeat-aware local session graph to encode the intra-item dependencies and generate the session representation with positional awareness. Then, we use sessions from the current mini-batch instead of the whole training set to construct a global graph, which we refer to as the session-level global graph. Next, we aggregate the K-nearest neighbors to generate the final session representation, which enables easy and efficient neighbor searching. Extensive experiments on three real-world recommendation datasets demonstrate that RN-GNN outperforms state-of-the-art methods.

In large data centers, managing the availability of servers is often non-trivial, especially when the workload is unpredictable. Using too many servers would waste energy, while using too few would affect the performance. A recent theoretical study, which assumes the clairvoyant model where job size is known at arrival time, has successfully integrated sleep-and-wakeup management into multi-processor job scheduling and obtained a competitive tradeoff between flow time and energy [6]. This paper extends the study to the nonclairvoyant model where the size of a job is not known until the job is finished. We give a new online algorithm SATA which is, for any ε > 0, (1 + ε)-speed O( 1⁄ε2 )-competitive for the objective of minimizing the sum of flow time and energy.

This paper initiates the study of online scheduling with rejection penalty in the non-clairvoyant setting, i.e., the size (processing time) of a job is not assumed to be known at its release time. In the rejection penalty model, jobs can be rejected with a penalty, and the user cost of a job is defined as the weighted flow time of the job plus the penalty if it is rejected before completion. Previous work on minimizing the total user cost focused on the clairvoyant single-processor setting [BBC+03,CLL11] and has produced O(1)-competitive online algorithm for jobs with arbitrary weights and penalties. This paper gives the first non-clairvoyant algorithms that are O(1)-competitive for minimizing the total user cost on a single processor and multi-processors, when using slightly faster (i.e., (1+ε)-speed for any ε > 0) processors. Note that if no extra speed is allowed, no online algorithm can be O(1)-competitive even for minimizing (unweighted) flow time alone. The new user cost results can also be regarded as a generalization of previous non-clairvoyant results on minimizing weighted flow time alone (WSETF [BaD07] for a single processor; WLAPS [ZCL11] for multi-processors). The above results assume a processor running at a fixed speed. This paper shows more interesting results on extending the above study to the dynamic speed scaling model, where the processor can vary the speed dynamically and the rate of energy consumption is an arbitrary increasing function of speed. A scheduling algorithm has to decide job rejection and determine the order and speed of job execution. It is interesting to study the tradeoff between the above-mentioned user cost and energy. This paper gives two O(1)-competitive non-clairvoyant algorithms for minimizing the user cost plus energy on a single processor and multi-processors, respectively.

This paper revisits the online problem of flow-time scheduling on a single processor when jobs can be rejected at some penalty [Bansal et al. 2003]. The user cost of a job is defined as the weighted flow time of the job plus the penalty if it is rejected before completion. For jobs with arbitrary weights and arbitrary penalties, [Bansal et al. 2003] gave an online algorithm that is O((log W + log C)^2)-competitive for minimizing the total user cost when using a slightly faster processor, where W and C are the max-min ratios of job weights and job penalties, respectively. In this paper we improve this result with a new algorithm that can achieve a constant competitive ratio independent of $W$ and C when using a slightly faster processor. Note that the above results assume a processor running at a fixed speed. This paper shows more interesting results on extending the above study to the dynamic speed scaling model, where the processor can vary the speed dynamically and the rate of energy consumption is a cubic or any increasing function of speed. A scheduling algorithm has to control job admission and determine the order and speed of job execution. This paper studies the tradeoff between the above-mentioned user cost and energy, and it shows two O(1)-competitive algorithms and a lower bound result on minimizing the user cost plus energy. These algorithms can also be regarded as a generalization of the recent work on minimizing flow time plus energy when all jobs must be completed (see the survey paper [Albers 2010]).

We give three results related to online nonclairvoyant speed scaling to minimize total flow time plus energy. We give a nonclairvoyant algorithm LAPS, and show that for every power function of the form P(s)=s α , LAPS is O(1)-competitive; more precisely, the competitive ratio is 8 for α=2, 13 for α=3, and $\frac{2\alpha^{2}}{\ln\alpha}$ for α>3. We then show that there is no constant c, and no deterministic nonclairvoyant algorithm A, such that A is c-competitive for every power function of the form P(s)=s α . So necessarily the achievable competitive ratio increases as the steepness of the power function increases. Finally we show that there is a fixed, very steep, power function for which no nonclairvoyant algorithm can be O(1)-competitive.

The past decade has witnessed many interesting algorithms for maintaining statistics over a data stream. This paper initiates a theoretical study of algorithms for monitoring distributed data streams over a time-based sliding window (which contains a variable number of items and possibly out-of-order items). The concern is how to minimize the communication between individual streams and the root, while allowing the root, at any time, to be able to report the global statistics of all streams within a given error bound. This paper presents communication-efficient algorithms for three classical statistics, namely, basic counting, frequent items and quantiles. The worst-case communication cost over a window is $O(\frac{k}{\varepsilon} \log \frac{\varepsilon N}{k})$ bits for basic counting and $O(\frac{k}{\varepsilon} \log \frac{N}{k})$ words for the remainings, where $k$ is the number of distributed data streams, $N$ is the total number of items in the streams that arrive or expire in the window, and $\varepsilon < 1$ is the desired error bound. Matching and nearly matching lower bounds are also obtained.

We study online nonclairvoyant speed scaling to minimize total flow time plus energy. We first consider the traditional model where the power function is $P(s)=s^\alpha$. We give a nonclairvoyant algorithm that is shown to be $O(\alpha^3)$-competitive. We then show an $\Omega( \alpha^{1/3-\epsilon} )$ lower bound on the competitive ratio of any nonclairvoyant algorithm. We also show that there are power functions for which no nonclairvoyant algorithm can be $O(1)$-competitive.

This article extends the study of online algorithms for energy-efficient deadline scheduling to the overloaded setting. Specifically, we consider a processor that can vary its speed between 0 and a maximum speed T to minimize its energy usage (the rate is believed to be a cubic function of the speed). As the speed is upper bounded, the processor may be overloaded with jobs and no scheduling algorithms can guarantee to meet the deadlines of all jobs. An optimal schedule is expected to maximize the throughput, and furthermore, its energy usage should be the smallest among all schedules that achieve the maximum throughput. In designing a scheduling algorithm, one has to face the dilemma of selecting more jobs and being conservative in energy usage. If we ignore energy usage, the best possible online algorithm is 4-competitive on throughput [Koren and Shasha 1995]. On the other hand, existing work on energy-efficient scheduling focuses on a setting where the processor speed is unbounded and the concern is on minimizing the energy to complete all jobs; O (1)-competitive online algorithms with respect to energy usage have been known [Yao et al. 1995; Bansal et al. 2007a; Li et al. 2006]. This article presents the first online algorithm for the more realistic setting where processor speed is bounded and the system may be overloaded; the algorithm is O (1)-competitive on both throughput and energy usage. If the maximum speed of the online scheduler is relaxed slightly to (1+ϵ) T for some ϵ > 0, we can improve the competitive ratio on throughput to arbitrarily close to one, while maintaining O (1)-competitiveness on energy usage.

Energy usage has been an important concern in recent research on online scheduling. In this paper we extend the study of the tradeoff between flow time and energy from the single-processor setting [8, 6] to the multi-processor setting. Our main result is an analysis of a simple non-migratory online algorithm called CRR (classified round robin) on m ≥ 2 processors, showing that its flow time plus energy is within O(1) times of the optimal non-migratory offline algorithm, when the maximum allowable speed is slightly relaxed. This result still holds even if the comparison is made against the optimal migratory offline algorithm (the competitive ratio increases by a factor of 2.5). As a special case, our work also contributes to the traditional online flow-time scheduling. Specifically, for minimizing flow time only, CRR can yield a competitive ratio one or even arbitrarily smaller than one, when using sufficiently faster processors. Prior to our work, similar result is only known for online algorithms that needs migration [21, 23], while the best non-migratory result can achieve an O(1) competitive ratio [14].

Energy usage has been an important concern in recent research on online job scheduling, where processors are allowed to vary the speed dynamically so as to save energy whenever possible. Notice that providing good quality of service such as response time (flow time) and conserving energy are conflicting objectives. An interesting problem for scheduling is how to optimize an economic tradeoff of flow time and energy. To this end, the past two years have witnessed significant progress in the single-processor setting, and online algorithms with performance close to optimal have been obtained. In this paper we extend the study of optimizing the tradeoff between flow time and energy to the multi-processor setting. We derive and analyze a simple non-migratory online algorithm that makes use of the classified-round-robin (CRR) strategy to dispatch jobs. Even in the worst case, its performance is within O(log P) times of the optimal migratory offline algorithm, where P is the ratio of the maximum job size to the minimum job size. Technically speaking, this online result stems from a non-trivial solution to an offline problem of eliminating migration, which is also interesting by itself.

In this paper, we give a simple scheme for identifying ε-approximate frequent items over a sliding window of size n. Our scheme is deterministic and does not make any assumption on the distribution of the item frequencies. It supports O(1/ε) update and query time, and uses O(1/ε) space. It is very simple; its main data structures are just a few short queues whose entries store the position of some items in the sliding window. We also extend our scheme for variable-size window. This extended scheme uses O(1/ε log(εn)) space.