Large-scale pre-training and instruction tuning have been successful at creating general-purpose language models with broad competence. However, building general-purpose vision-language models is challenging due to the rich input distributions and task diversity resulting from the additional visual input. Although vision-language pretraining has been widely studied, vision-language instruction tuning remains under-explored. In this paper, we conduct a systematic and comprehensive study on vision-language instruction tuning based on the pretrained BLIP-2 models. We gather 26 publicly available datasets, covering a wide variety of tasks and capabilities, and transform them into instruction tuning format. Additionally, we introduce an instruction-aware Query Transformer, which extracts informative features tailored to the given instruction. Trained on 13 held-in datasets, InstructBLIP attains state-of-the-art zero-shot performance across all 13 held-out datasets, substantially outperforming BLIP-2 and larger Flamingo models. Our models also lead to state-of-the-art performance when finetuned on individual downstream tasks (e.g., 90.7% accuracy on ScienceQA questions with image contexts). Furthermore, we qualitatively demonstrate the advantages of InstructBLIP over concurrent multimodal models. All InstructBLIP models are open-source.

Although many impressive works on learning-based camera ego-motion estimation methods have been proposed recently, most of them promote the accuracy of camera pose estimation by various sequential learning with loop closure optimization, while neglecting the improvement of PoseNet itself. In this paper, we focus on the coupling of rotation and translation in ego-motion estimation, and design a cascade decoupling structure to separately learn the rotation and translation of camera relative motion between adjacent frames. Meanwhile, a rigid-aware unsupervised learning framework with iterative pose refinement scheme is proposed for camera ego-motion estimation. It can disambiguate rigid motion and deformations in dynamic scenarios by jointly learning of optical flow, stereo disparity and camera pose. Validated with evaluation experiments on the public available datasets, our method is superior to the state-of-the-art unsupervised methods, and can achieve comparable results with the supervised ones.

All life on Earth is related, so that some molecular interactions are common across almost all living cells, with the number of common interactions increasing as we look at more closely related species. In particular, we expect the protein–protein interaction (PPI) networks of closely related species to share high levels of similarity. This similarity may facilitate the transfer of functional knowledge between model species and human. Multiple network alignment is the process of uncovering the connection similarity between three or more networks simultaneously. Existing algorithms for multiple network alignment rely on sequence similarities to help drive the alignments, and no comprehensive study has been done to determine the most effective ways to utilize network connectivity—network topology—to drive multiple network alignment. Here, we devise and empirically test the efficacy of several measures of topological similarity between three or more networks. To evolve the alignments toward optimal, we use simulated annealing as the search algorithm since it is agnostic to the objective being optimized. We test the measures both on the partially synthetic and highly similar PPI networks from the integrated interaction database, as well as on real PPI networks from a recent BioGRID release.

Object localization and local feature representation are key issues in fine-grained image retrieval. However, the existing unsupervised methods still need to be improved in these two aspects. For conquering these issues in a unified framework, a novel unsupervised scheme, named DSAW for short, is presented in this paper. Firstly, we proposed a dual-selection (DS) method, which achieves more accurate object localization by using an adaptive threshold method to perform feature selection on local and global activation map. Secondly, a novel and faster self-attention weights (AW) method is developed to weight local features by measuring their importance in the global context. Finally, we also evaluated the performance of the proposed method on five fine-grained image datasets and the results showed that our DSAW outperformed the existing best method.

In this paper, an improved multiperiod risk-limiting dispatch (IMRLD) is proposed as an operational method in power systems with high percentage renewables integration. The basic risk-limiting dispatch (BRLD) is chosen as an operational paradigm to address the uncertainty of renewables in this paper due to its three good features. In this paper, the BRLD is extended to the IMRLD so that it satisfies the fundamental operational requirements in the power industry. In order to solve the IMRLD problem, the convexity of the IMRLD is verified. A theorem is stated and proved to transform the IMRLD into a piece-wise linear optimization problem that can be efficiently solved. In addition, the locational marginal price of the IMRLD is derived to analyze the effect of renewables integration on the marginal operational cost. Finally, two numerical tests are conducted to validate the IMRLD.

The Z-source neutral point clamped (NPC) inverters can realize buck-boost energy conversion with all satisfying advantages of the three-level inverters. A significant problem related to this topology is the fluctuation of the neutral-point potential (NP). However, the previous modulation techniques for this topology cannot achieve the considerate stable voltage. In this paper, based on the analysis of the relationship among the shoot-through duty cycle and the NP voltage, the average NP current, switching frequency capacitor value, a new feed-forward compensator by modifying the shoot-through duty cycle which can improve to balance the NP voltage balancing is proposed. Meanwhile, in consideration of a low voltage gain caused by the NP voltage balance method, a new control method based on a maximum constant boost control method is proposed, which brings the benefits in both high voltage gain and low-frequency ripple. The considerate stable NP voltage and the high voltage gain have been proven in simulation and through experimental results from a prototype converter.