Generating natural and meaningful responses to communicate with multi-modal human inputs is a fundamental capability of Large Vision-Language Models (LVLMs). While current open-source LVLMs demonstrate promising performance in simplified scenarios such as single-turn single-image input, they fall short in real-world conversation scenarios such as following instructions in a long context history with multi-turn and multi-images. Existing LVLM benchmarks primarily focus on single-choice questions or short-form responses, which do not adequately assess the capabilities of LVLMs in real-world human-AI interaction applications. Therefore, we introduce MMDU, a comprehensive benchmark, and MMDU-45k, a large-scale instruction tuning dataset, designed to evaluate and improve LVLMs' abilities in multi-turn and multi-image conversations. We employ the clustering algorithm to find the relevant images and textual descriptions from the open-source Wikipedia and construct the question-answer pairs by human annotators with the assistance of the GPT-4o model.MMDU has a maximum of 18k image+text tokens, 20 images, and 27 turns, which is at least 5x longer than previous benchmarks and poses challenges to current LVLMs. Our in-depth analysis of 15 representative LVLMs using MMDU reveals that open-source LVLMs lag behind closed-source counterparts due to limited conversational instruction tuning data.We demonstrate that fine-tuning open-source LVLMs on MMDU-45k significantly address this gap, generating longer and more accurate conversations, and improving scores on MMDU and existing benchmarks (MMStar: +1.1%, MathVista: +1.5%, ChartQA: +1.2%). Our contributions pave the way for bridging the gap between current LVLM models and real-world application demands. The links to MMDU, and MMDU-45k are available in the supplementary material.

Understanding and modeling the 3D scene from a single image is a practical problem. A recent advance proposes a panoptic 3D scene reconstruction task that performs both 3D reconstruction and 3D panoptic segmentation from a single image. Although having made substantial progress, recent works only focus on top-down approaches that fill 2D instances into 3D voxels according to estimated depth, which hinders their performance by two ambiguities. (1) instance-channel ambiguity: The variable ids of instances in each scene lead to ambiguity during filling voxel channels with 2D information, confusing the following 3D refinement. (2) voxel-reconstruction ambiguity: 2D-to-3D lifting with estimated single view depth only propagates 2D information onto the surface of 3D regions, leading to ambiguity during the reconstruction of regions behind the frontal view surface. In this paper, we propose BUOL, a Bottom-Up framework with Occupancy-aware Lifting to address the two issues for panoptic 3D scene reconstruction from a single image. For instance-channel ambiguity, a bottom-up framework lifts 2D information to 3D voxels based on deterministic semantic assignments rather than arbitrary instance id assignments. The 3D voxels are then refined and grouped into 3D instances according to the predicted 2D instance centers. For voxel-reconstruction ambiguity, the estimated multi-plane occupancy is leveraged together with depth to fill the whole regions of things and stuff. Our method shows a tremendous performance advantage over state-of-the-art methods on synthetic dataset 3D-Front and real-world dataset Matterport3D. Code and models will be released.

Recent advances in detecting arbitrary objects in the real world are trained and evaluated on object detection datasets with a relatively restricted vocabulary. To facilitate the development of more general visual object detection, we propose V3Det, a vast vocabulary visual detection dataset with precisely annotated bounding boxes on massive images. V3Det has several appealing properties: 1) Vast Vocabulary: It contains bounding boxes of objects from 13,204 categories on real-world images, which is 10 times larger than the existing large vocabulary object detection dataset, e.g., LVIS. 2) Hierarchical Category Organization: The vast vocabulary of V3Det is organized by a hierarchical category tree which annotates the inclusion relationship among categories, encouraging the exploration of category relationships in vast and open vocabulary object detection. 3) Rich Annotations: V3Det comprises precisely annotated objects in 243k images and professional descriptions of each category written by human experts and a powerful chatbot. By offering a vast exploration space, V3Det enables extensive benchmarks on both vast and open vocabulary object detection, leading to new observations, practices, and insights for future research. It has the potential to serve as a cornerstone dataset for developing more general visual perception systems. V3Det is available at https://v3det.openxlab.org.cn/.

The National Research Council Decadal Survey identified the need for a future mission to provide accurate real-time observations of ocean wind vectors from calm to tropical cyclone wind conditions with or without the presence of rain. Tasked by the National Oceanic and Atmospheric Administration (NOAA), the Jet Propulsion Laboratory (JPL) developed a future scatterometer design that would leverage its success on the heritage of QuikSCAT but would provide more accurate measurements under all weather conditions through the use of Ku- and C-band coincident measurements of the ocean surface. To design a cost effective instrument for all weather operations from space the existing risks need to be mitigated. The work described in this paper attempts to validate results reported at hurricane strength winds in and investigate the effects of splash caused by precipitation on the scatterometer wind estimation.

IWRAP, the Imaging Wind and Rain Airborne Profiler is a conically scanning dual-band (C and Ku), dual-polarized pencil-beam airborne Doppler radar that profiles the volume reflectivity and Doppler velocity from precipitation. It also measures the ocean backscatter response especially during tropical and extra tropical cyclones. IWRAP is designed to measure troposphere winds and surface winds. Of particular importance to the storm forecasting and research community are the winds in the lower part of the hurricane boundary layer near the sea surface (lower ABL). The covariance data processing technique IWRAP traditionally employed assumes a uni-modal and symmetric Doppler spectrum and therefore, it alone is unable to account properly for the influence of the surface scattering on wind profiles in the lower ABL region. In this paper we compare covariance-based approaches and full-spectral approaches utilized to derive the true wind speed for atmospheric vertical profiles. Techniques for removal of the surface contamination and example of wind profiles retrieved from Hurricane Rita are presented.

The Microwave Remote Sensing Laboratory (MIRSL) at the University of Massachusetts developed and operates the Imaging Wind and Rain Airborne Profiler (IWRAP). With support from NASA, NOAA, and ONR, UMASS has collaborated with NOAA-NESDIS and NOAA-AOC to operate the instrument aboard the NOAA WP-3D "Hurricane Hunter" aircraft since the 2002 hurricane season and for four winter experiments sampling high-latitude storms. The goals of these experiments have been to characterize storm boundary layers and to develop improved retrievals for space-based scatterometers, specifically, NASA SeaWinds Scatterometer on QuikSCAT and Advanced Scatterometer (ASCAT) which was developed by ESA on behalf of EUMETSAT. IWRAP is a conically-scanning, dual-frequency (C-and Ku-band) coherent radar with dual-polarization capability. The incidence angles are nominally: 30, 35, 40, and 50 degrees. At times, the instrument has been configured to address specific scientific and engineering problems. Examples include: C-band measurements of high incidence angle backscatter for ASCAT validation, and C-band horizontal polarization measurements to develop a high wind speed geophysical model function (GMF). Independent measurements of surface wind speed and column integrated rain rate are made coincidently with the UMass Simultaneous Frequency Radiometer (USFMR) and AOC Stepped Frequency Microwave Radiometer (SFMR). Since 2005, IWRAP has operated in a raw data mode, recording individual radar pulses to enable the separation of near surface backscatter contributions (i.e. surface scattering versus volume scattering). The intensity of tropical cyclones ranged from tropical storm to category five hurricane, and high-latitude storms with up to hurricane force winds.

During the NOAA/NESDIS 2005 Hurricane Season (HS2005) and the 2006 Winter Experiment, the University of Massachusetts (UMass) installed two instruments on the NOAA N42RF WP-3D research aircraft: the Imaging Wind and Rain Airborne Profiler (IWRAP) and the Simultaneous Frequency Microwave Radiometer (SFMR). IWRAP is a dual-band (C- and Ku), dual-polarized pencil- beam airborne radar that profiles the volume backscatter and Doppler velocity from rain and that also measures the ocean backscatter response. It simultaneously profiles along four separate incidence angles while conically scanning at 60 RPM. SFMR is a C-band nadir viewing radiometer that measures the emission from the ocean surface and intervening atmosphere simultaneously at six frequencies. It is designed to obtain the surface wind speed and the column average rain rate. Both instruments have previously been flown during the 2002, 2003 and 2004 hurricane seasons. For the HS2005, the IWRAP system was modified to implement a raw data acquisition system. The importance of the raw data system arises when trying to profile the atmosphere all the way down to the surface with a non- nadir looking radar system. With this particular geometry, problems arise mainly from the fact that both rain and ocean provide a return echo coincident in time through the antenna's main lobe. This paper shows how this limitation has been removed and presents initial results demonstrating its new capabilities to derive the atmospheric boundary layer (ABL) wind field within the inner core of hurricanes to much lower altitudes than the ones the original system was capable of, and to analyze the spectral response of the ocean backscatter and the rain under different wind and rain conditions.