An interactive vignette is a popular and immersive visual storytelling approach that invites viewers to role-play a character and influences the narrative in an interactive environment. However, it has not been widely used by everyday storytellers yet due to authoring complexity, which conflicts with the immediacy of everyday storytelling. We introduce DiaryPlay, an AI-assisted authoring system for interactive vignette creation in everyday storytelling. It takes a natural language story as input and extracts the three core elements of an interactive vignette (environment, characters, and events), enabling authors to focus on refining these elements instead of constructing them from scratch. Then, it automatically transforms the single-branch story input into a branch-and-bottleneck structure using an LLM-powered narrative planner, which enables flexible viewer interactions while freeing the author from multi-branching. A technical evaluation (N=16) shows that DiaryPlay-generated character activities are on par with human-authored ones regarding believability. A user study (N=16) shows that DiaryPlay effectively supports authors in creating interactive vignette elements, maintains authorial intent while reacting to viewer interactions, and provides engaging viewing experiences.

One common asset of metaverse is that users can freely explore places and actions without linear procedures. Thus, it is hard yet important to understand the divergent challenges each user faces when onboarding metaverse. Our formative study (N = 16) shows that first-time users ask questions about metaverse that concern 1) a short-term spatiotemporal context, regarding the user’s current location, recent conversation, and actions, and 2) a long-term exploration context regarding the user’s experience history. Based on the findings, we present PICAN, a Large Language Model-based pipeline that generates context-aware answers to users when onboarding metaverse. An ablation study (N = 20) reveals that PICAN’s usage of context made responses more useful and immersive than those generated without contexts. Furthermore, a user study (N = 21) shows that the use of long-term exploration context promotes users’ learning about the locations and activities within the virtual environment.

Large panoramic views with high resolution have the advantage of a wide field of view over regular stereo views. However, the large size and high resolution impose difficulties on the stereo matching problem such as complexity and structure ambiguity, respectively. In this paper, effective multi-resolution disparity processing to resolve the difficulties is presented. We propose to adaptively determine the disparity search range based on the combined local structure from image intensity and initial disparity. The adaptive disparity range is able to propagate the smoothness property at low resolution to high resolution while preserving fine details. It reduces structure ambiguity as well as computational complexity. To reduce the disparity quantization error at the coarse level, we propose a reliable multiple fitting algorithm that is noticeably effective on the round surface. The spatial-multi-resolution total variation method is investigated to minimize inconsistency in space-scale dimension . The experimental results on the Middlebury datasets and real-world high-resolution images demonstrate that the proposed multi-resolution scheme produces high-quality and high-resolution disparity maps by fusing individual multi-scale disparity maps, while reducing complexity.

In this paper, we present an effective hierarchical depth processing and fusion for large stereo images. We propose the adaptive disparity search range based on the combined local structure from image and initial disparity. The adaptive search range can propagate the smoothness property at the coarse level to the fine level while preserving details and suppressing undesirable errors. The spatial-multiscale total variation method is investigated to enforce the spatial and scaling consistency of multi-scale depth estimates. The experimental results demonstrate that the proposed hierarchical scheme produces high quality and high resolution depth maps by fusing individual multi-scale depth maps, while reducing complexity.

This paper presents a new approach to estimate fog-free images from stereo foggy images. We investigate a new way to estimate transmission by computing the scattering coefficient and depth information of a scene. However, most existing visibility restoration algorithms estimate transmission independently on scattering coefficient and object distance. In the proposed method, the natural color of a foggy image is recovered using depth information from a stereo image pair even though prior knowledge or multiple images taken at different times are not required. Furthermore, we explore a new way to measure the scattering coefficient by using a stereo image pair from an image processing perspective. Experimental results verify that the proposed method outperforms the conventional defogging methods.

Multi-array camera systems have greater potential for 3-D depth-based application development compared with stereo camera systems. However, there are very few research results on multi-array-based disparity enhancement, extending standard stereo matchings to multi-array systems. In this paper, we propose to alternately use local and global fusion of multi-array disparities to maximize the disparity enhancement in array camera systems. We propose a new cascade regularization based approach, which can restore diagonal structures better than conventional techniques. The detailed analysis and experimental results verify that the cascade approach better regularizes the diagonal variations and in turn yields better image enhancement. We adapt total variation for regularization to the multi-array camera systems in order to globally combine multiple disparity estimates. A local multiple cross-filling algorithm is proposed to achieve cross consistency between array disparity estimates by effectively filling the mismatches. Experimental results show that the proposed multi-array disparity enhancement algorithm can improve the accuracy of the initial array disparity estimates up to 65% while alleviating memory limitation.

The classical local disparity methods use simple and efficient structure to reduce the computation complexity. To increase the accuracy of the disparity map, new local methods utilize additional processing steps such as iteration, segmentation, calibration and propagation, similar to global methods. In this paper, we present an efficient one-pass local method with no iteration. The proposed method is also extended to video disparity estimation by using motion information as well as imposing spatial temporal consistency. In local method, the accuracy of stereo matching depends on precise similarity measure and proper support window. For the accuracy of similarity measure, we propose a novel three-moded cross census transform with a noise buffer, which increases the robustness to image noise in flat areas. The proposed similarity measure can be used in the same form in both stereo images and videos. We further improve the reliability of the aggregation by adopting the advanced support weight and incorporating motion flow to achieve better depth map near moving edges in video scene. The experimental results show that the proposed method is the best performing local method on the Middlebury stereo benchmark test and outperforms the other state-of-the-art methods on video disparity evaluation.