The ability to accurately interpret complex visual information is a crucial topic of multimodal large language models (MLLMs). Recent work indicates that enhanced visual perception significantly reduces hallucinations and improves performance on resolution-sensitive tasks, such as optical character recognition and document analysis. A number of recent MLLMs achieve this goal using a mixture of vision encoders. Despite their success, there is a lack of systematic comparisons and detailed ablation studies addressing critical aspects, such as expert selection and the integration of multiple vision experts. This study provides an extensive exploration of the design space for MLLMs using a mixture of vision encoders and resolutions. Our findings reveal several underlying principles common to various existing strategies, leading to a streamlined yet effective design approach. We discover that simply concatenating visual tokens from a set of complementary vision encoders is as effective as more complex mixing architectures or strategies. We additionally introduce Pre-Alignment to bridge the gap between vision-focused encoders and language tokens, enhancing model coherence. The resulting family of MLLMs, Eagle, surpasses other leading open-source models on major MLLM benchmarks.

Partial convolution weights convolutions with binary masks and renormalizes on valid pixels. It was originally proposed for image inpainting task because a corrupted image processed by a standard convolutional often leads to artifacts. Therefore, binary masks are constructed that define the valid and corrupted pixels, so that partial convolution results are only calculated based on valid pixels. It has been also used for conditional image synthesis task, so that when a scene is generated, convolution results of an instance depend only on the feature values that belong to the same instance. One of the unexplored applications for partial convolution is padding which is a critical component of modern convolutional networks. Common padding schemes make strong assumptions about how the padded data should be extrapolated. We show that these padding schemes impair model accuracy, whereas partial convolution based padding provides consistent improvements across a range of tasks. In this article, we review partial convolution applications under one framework. We conduct a comprehensive study of the partial convolution based padding on a variety of computer vision tasks, including image classification, 3D-convolution-based action recognition, and semantic segmentation. Our results suggest that partial convolution-based padding shows promising improvements over strong baselines.

Conditional image synthesis for generating photorealistic images serves various applications for content editing to content generation. Previous conditional image synthesis algorithms mostly rely on semantic maps, and often fail in complex environments where multiple instances occlude each other. We propose a panoptic aware image synthesis network to generate high fidelity and photorealistic images conditioned on panoptic maps which unify semantic and instance information. To achieve this, we efficiently use panoptic maps in convolution and upsampling layers. We show that with the proposed changes to the generator, we can improve on the previous state-of-the-art methods by generating images in complex instance interaction environments in higher fidelity and tiny objects in more details. Furthermore, our proposed method also outperforms the previous state-of-the-art methods in metrics of mean IoU (Intersection over Union), and detAP (Detection Average Precision).

Semantic segmentation requires large amounts of pixel-wise annotations to learn accurate models. In this paper, we present a video prediction-based methodology to scale up training sets by synthesizing new training samples in order to improve the accuracy of semantic segmentation networks. We exploit video prediction models' ability to predict future frames in order to also predict future labels. A joint propagation strategy is also proposed to alleviate mis-alignments in synthesized samples. We demonstrate that training segmentation models on datasets augmented by the synthesized samples leads to significant improvements in accuracy. Furthermore, we introduce a novel boundary label relaxation technique that makes training robust to annotation noise and propagation artifacts along object boundaries. Our proposed methods achieve state-of-the-art mIoUs of 83.5% on Cityscapes and 82.9% on CamVid. Our single model, without model ensembles, achieves 72.8% mIoU on the KITTI semantic segmentation test set, which surpasses the winning entry of the ROB challenge 2018.

Botnets are groups of compromised computers that botmasters (botherders) use to launch attacks over the Internet. To avoid detection, botnets use DNS fast flux to change the mapping between IP addresses and domain names periodically. Domain generation algorithms (DGAs) are employed to generate a large number of domain names. Detection techniques have been proposed to identify malicious domain names generated by DGAs. Three metrics, Kullback-Leibler (KL) distance, Edit distance (ED), and Jaccard index (JI), are used to detect botnet domains with up to 100% detection rate and 2.5% false-positive rate. In this paper, we propose two DGAs that use hidden Markov models (HMMs) and probabilistic context-free grammars (PCFGs), respectively. Experiment results show that DGA detection metrics (KL, JI, and ED) and detection systems (BotDigger and Pleiades) have difficulty detecting domain names generated using the proposed approaches. Game theory is used to optimize strategies for both botmasters and security personnel. Results show that, to optimize DGA detection, security personnel should use the ED detection technique with probability 0.78 and JI detection with probability 0.22, and botmasters should choose the HMM-based DGA with probability 0.67 and PCFG-based DGA with probability 0.33.

The pay-as-you-go service model impels cloud customers to reduce the usage cost of bandwidth. Traffic Redundancy Elimination (TRE) has been shown to be an effective solution for reducing bandwidth costs, and thus has recently captured significant attention in the cloud environment. By studying the TRE techniques in a trace driven approach, we found that both short-term (time span of seconds) and long-term (time span of hours or days) data redundancy can concurrently appear in the traffic, and solely using either sender-based TRE or receiver-based TRE cannot simultaneously capture both types of traffic redundancy. Also, the efficiency of existing receiver-based TRE solution is susceptible to the data changes compared to the historical data in the cache. In this paper, we propose a Cooperative end-to-end TRE solution (CoRE) that can detect and remove both short-term and long-term redundancy through a two-layer TRE design with cooperative operations between layers. An adaptive prediction algorithm is further proposed to improve TRE efficiency through dynamically adjusting the prediction window size based on the hit ratio of historical predictions. Besides, we enhance CoRE to adapt to different traffic redundancy characteristics of cloud applications to improve its operation cost. Extensive evaluation with several real traces show that CoRE is capable of effectively identifying both short-term and long-term redundancy with low additional cost while ensuring TRE efficiency from data changes.

Real-world applications, such as peer-to-peer (P2P) networks, e-commerce and social networks, usually employ reputation systems to provide guidance in selecting trustworthy node for high system reliability and security. A reputation system computes and publishes reputation score for each node based on a collection of opinions from others about the node. However, collusion behaviors impair the effectiveness of reputation systems in trustworthy node selection. Though many reputation calculation methods have been proposed to mitigate collusion's influence, little effort has been devoted to specifically tackling collusion. Based on the important collusion behavior characteristics in reputation evaluation and influence on reputation values, we propose a basic collusion detection method to specifically detect suspicious collusion behaviors in pairs. We further optimize the method by reducing the computing overhead. We also propose two pre-processing methods to firstly identify partial reputation raters of a node that are more likely to be colluders before applying the collusion detection method on them, thus reducing the collusion detection overhead. Extensive experimental results show that our proposed methods can significantly enhance the capability of existing reputation systems to detect collusion with low overhead. Also, the pre-processing methods are effective in reducing the collusion detection overhead without affecting the collusion detection accuracy.

Underlying link-layer protocols of well-established wireless networks that use the conventional “store-and-forward” design paradigm cannot provide highly sustainable reliability and stability in wireless communication, which introduce significant barriers and setbacks in scalability and deployments of wireless networks. In this paper, we propose a Code Embedded Distributed Adaptive and Reliable (CEDAR) link-layer framework that targets low latency and balancing en/decoding load among nodes. CEDAR is the first comprehensive theoretical framework for analyzing and designing distributed and adaptive error recovery for wireless networks. It employs a theoretically sound framework for embedding channel codes in each packet and performs the error correcting process in selected intermediate nodes in a packet's route. To identify the intermediate nodes for the decoding, we mathematically calculate the average packet delay and formalize the problem as a nonlinear integer programming problem. By minimizing the delays, we derive three propositions that: 1) can identify the intermediate nodes that minimize the propagation and transmission delay of a packet; and 2) and 3) can identify the intermediate nodes that simultaneously minimize the queuing delay and maximize the fairness of en/decoding load of all the nodes. Guided by the propositions, we then propose a scalable and distributed scheme in CEDAR to choose the intermediate en/decoding nodes in a route to achieve its objective. The results from real-world testbed “NESTbed” and simulation with MATLAB prove that CEDAR is superior to schemes using hop-by-hop decoding and destination decoding not only in packet delay and throughput but also in energy-consumption and load distribution balance.

Current reputation systems for peer-to-peer (P2P) file sharing networks suffer from high overhead on reputation querying. Also, purely relying on a threshold to detect malicious nodes may make a high-reputed node be reluctant to further increase its reputation in these reputation systems. On the other side, the social network concept of “friendship foster cooperation” can be utilized to alleviate the high overhead in reputation systems. However, the limited number of friends limits the availability of file resources in these approaches. To overcome the drawbacks, we propose a social network based reputation system, namely SocialTrust, that synergistically leverages the social network connections and traditional credit based reputation system to provide efficient reputation management for P2P file sharing. In SocialTrust, each node favors friends for service transactions, which are resulted from both real life acquaintances and online partnerships established between high-reputed and frequently-interacted nodes. When no friends are available for a request, a node chooses the server with the highest reputation. The benefits of friendship and partnership on file sharing and cost saving encourage nodes to be continuously cooperative. Further, SocialTrust considers the number of friends/partners and the reputation of a node in reputation rewarding/punishment in order to realize accurate reputation evaluation. SocialTrust can also prevent certain attacks such as deny of service and collusion. Extensive trace-driven simulation demonstrates the effectiveness of SocialTrust.

To provide robust infrastructure as a service (IaaS), clouds currently perform load balancing by migrating virtual machines (VMs) from heavily loaded physical machines (PMs) to lightly loaded PMs. Previous reactive load balancing algorithms migrate VMs upon the occurrence of load imbalance, while previous proactive load balancing algorithms predict PM overload to conduct VM migration. However, both methods cannot maintain long-term load balance and produce high overhead and delay due to migration VM selection and destination PM selection. To overcome these problems, in this paper, we propose a proactive Markov Decision Process (MDP)-based load balancing algorithm. We handle the challenges of allying MDP in virtual resource management in cloud datacenters, which allows a PM to proactively find an optimal action to transit to a lightly loaded state that will maintain for a longer period of time. We also apply the MDP to determine destination PMs to achieve long-term PM load balance state. Our algorithm reduces the numbers of Service Level Agreement (SLA) violations by long-term load balance maintenance, and also reduces the load balancing overhead (e.g., CPU time, energy) and delay by quickly identifying VMs and destination PMs to migrate. Our trace-driven experiments show that our algorithm outperforms both previous reactive and proactive load balancing algorithms in terms of SLA violation, load balancing efficiency and long-term load balance maintenance.

To provide robust infrastructure as a service (IaaS), clouds currently perform load balancing by migrating virtual machines (VMs) from heavily loaded physical machines (PMs) to lightly loaded PMs. The unique features of clouds pose formidable challenges to achieving effective and efficient load balancing. First, VMs in clouds use different resources (e.g., CPU, bandwidth, memory) to serve a variety of services (e.g., high performance computing, web services, file services), resulting in different overutilized resources in different PMs. Also, the overutilized resources in a PM may vary over time due to the time-varying heterogenous service requests. Second, there is intensive network communication between VMs. However, previous load balancing methods statically assign equal or predefined weights to different resources, which leads to degraded performance in terms of speed and cost to achieve load balance. Also, they do not strive to minimize the VM communications between PMs. We propose a Resource Intensity Aware Load balancing method (RIAL). For each PM, RIAL dynamically assigns different weights to different resources according to their usage intensity in the PM, which significantly reduces the time and cost to achieve load balance and avoids future load imbalance. It also tries to keep frequently communicating VMs in the same PM to reduce bandwidth cost, and migrate VMs to PMs with minimum VM performance degradation. Our extensive trace-driven simulation results and real-world experimental results show the superior performance of RIAL compared to other load balancing methods.

In current reputation systems for peer-to-peer (P2P) file sharing networks, reputation querying generates high overhead in file service. Also, a high-reputed node may be reluctant to further increase its reputation in these reputation systems. Recently, social network-based P2P systems that favor file sharing among friends have been proposed for reliable file sharing. However, this approach contradicts the objective of widely file sharing in P2P systems. To overcome these drawbacks, we propose a social network integrated reputation system, namely SocialTrust, that synergistically leverages the concept of "friendship fosters cooperation''. In SocialTrust, each node maintains trusted relationships for the purpose of file sharing, which are resulted from both real life acquaintance and online partnership established between high-reputed and frequently-interacted nodes. Given a number of server options, a client first chooses a friend or a partner, if available, without querying their reputations. Otherwise, it chooses the server with the highest reputation. The benefits of friendship and partnership on file sharing and cost saving encourage nodes to be continuously cooperative. Further, in order to realize accurate reputation evaluation and strong cooperation incentives, SocialTrust considers the number of friends/partners and reputation of a node in the process of reputation rewarding and punishment. Extensive trace-driven simulation demonstrates the effectiveness of SocialTrust.

Underlying link-layer protocols of wireless networks use the conventional “store and forward” design paradigm cannot provide highly sustainable reliability and stability in wireless communication, which introduce significant barriers and setbacks in scalability and deployments of wireless networks. In this paper, we propose a Code Embedded Distributed Adaptive and Reliable (CEDAR) link-layer framework that targets low latency and high throughput. CEDAR is the first comprehensive theoretical framework for analyzing and designing distributed and adaptive error recovery for wireless networks. It employs a theoretically-sound framework for embedding channel codes in each packet and performs the error correcting process in selected intermediate nodes in packet's route. To identify the intermediate nodes for the en/decoding for minimizing average packet latency, we mathematically analyze the average packet delay, using Finite State Markovian Channel model and priority queuing model, and then formalize the problem as a non-linear integer programming problem. Also, we propose a scalable and distributed scheme to solve this problem. The results from real-world testbed “NESTbed” and simulation with Matlab prove that CEDAR is superior to the schemes using hop-by-hop decoding and destination-decoding not only in packet delay but also in throughput. In addition, the simulation results show that CEDAR can achieve the optimal performance in most cases.

In peer-to-peer networks (P2Ps), many autonomous peers without preexisting trust relationships share resources with each other. Due to their open environment, the P2Ps usually employ reputation systems to provide guidance in selecting trustworthy resource providers for high reliability and security; however, node collusion impairs the effectiveness of reputation systems in trustworthy node selection. Although some reputation systems have certain mechanisms to counter collusion, the effectiveness of the mechanisms is not sufficiently high. In this paper, we leverage social networks to enhance the capability of reputation systems in combating collusion. We first analyzed real trace of the reputation system in the Overstock online auction platform, which incorporates a social network. The analysis reveals the impact of the social network on user purchasing and reputation rating patterns. We, thus, identified suspicious collusion behavior patterns and propose a social network-based mechanism, SocialTrust, to counter collusion. SocialTrust adaptively adjusts the weight of ratings based on the social distance and interest relationship between peers. Experiment results show that SocialTrust can significantly strengthen the capability of current reputation systems in combating collusion.

The pay-as-you-go service model impels cloud customers to reduce the usage cost of bandwidth. Traffic Redundancy Elimination (TRE) has been shown to be an effective solution for reducing bandwidth costs, and has recently captured significant attention in the cloud environment. By studying the TRE techniques with a trace driven approach, we found that solely using either sender-based TRE or receiver-based TRE cannot simultaneously capture traffic redundancy in both short-term (time span of seconds) and long-term (time span of hours or days) data redundancy, which concurrently appear in the traffic. Additionally, the TRE efficiency of existing receiver-based TRE solution is susceptible to data changes compared to historical data in the cache. In this paper, we propose a sender and receiver Cooperative end-to-end TRE solution (CoRE) for efficiently identifying and removing both short-term and long-term redundancy. Through a two-layer redundancy detection design and one single pass algorithm for chunking and fingerprinting, CoRE efficiently carries out cooperative operations between the sender and the receiver. By extensive evaluation with several real traces, we show that CoRE is able to identify both short-term and longterm redundancy with low additional cost, while ensuring TRE efficiency from data changes.

In peer-to-peer networks (P2Ps), many autonomous nodes without preexisting trust relationships share resources (e.g., files) between each other. Due to their open environment, P2Ps usually employ reputation systems to provide guidance in selecting trustworthy resource providers for high system reliability and security. A reputation system computes and publishes reputation score for each node based on a collection of opinions from others about the node. However, collusion behaviors impair the effectiveness of reputation systems in trustworthy node selection. Though many reputation calculation methods have been proposed to mitigate collusion's influence, little effort has been devoted to specifically tackling collusion. In this paper, we analyze transaction ratings in the Amazon and Overstock online transaction platforms during one year. The analysis of real trace confirms the existence of collusion as well as its important behavior characteristics and influence on reputation values in real reputation systems. Accordingly, we propose a collusion detection method to specifically thwart collusion behaviors. We further optimize the method by reducing the computing cost. Experimental results show that the proposed method can significantly enhance the capability of existing reputation systems to deter collusion with low cost.

In peer-to-peer networks (P2Ps), many autonomous peers without preexisting trust relationships share resources with each other. Due to their open environment, the P2Ps usually employ reputation systems to provide guidance in selecting trustworthy resource providers for high reliability and security. However, node collusion impairs the effectiveness of reputation systems in trustworthy node selection. Although some reputation systems have certain mechanisms to counter collusion, the effectiveness of the mechanisms is not sufficiently high. In this paper, we leverage social networks to enhance the capability of reputation systems in combating collusion. We first analyzed real trace of the reputation system in the Overstock online auction platform which incorporates a social network. The analysis reveals the important impact of the social network on user purchasing and reputation rating patterns. We thus identified suspicious collusion behavior patterns and propose a social network based mechanism, namely Social Trust, to counter collusion. Social Trust adaptively adjusts the weight of ratings based on the social distance and interest relationship between peers. Experimental results show that Social Trust can significantly strengthen the capability of current reputation systems in combating collusion.