This paper presents a source-aware spatial self-supervised learning (SSL) method for sound event localization and detection (SELD) based on neural blind source separation (BSS). SELD involves estimating both the temporal activations of sound classes and their directions of arrival (DOAs) from multichannel audio signals. While deep neural networks have been successful in SELD, their reliance on spatial supervision, including class labels and DOAs, makes annotation challenging and time-intensive. To address this limitation, we propose self-supervised pre-training to separate and localize moving sound sources by maximizing a log-marginal posterior probability of a neural BSS model. This model, called neural full-rank spatial covariance analysis, requires only multichannel mixture signals for training. Our BSS-based framework explicitly trains the model to distinguish multiple sources in a mixture, enabling it to predict DOAs of multiple sound events. The proposed method thus enables us to train a SELD model with a small amount of annotation. The experiments with the STARSS23 dataset show that our method outperforms baseline SELD models trained with a limited amount of supervised data.

Humans typically avert their gaze when faced with situations involving another person’s privacy, and humanoid robots should exhibit similar behaviors. Various approaches exist for privacy recognition, including an image privacy recognition model and a Large Vision-Language Model (LVLM). The former relies on datasets of labeled images, which raise ethical concerns, while the latter requires more time to recognize images accurately, making real-time responses difficult. To this end, we propose a method of automatically constructing the LLM Privacy Text Dataset (LPT Dataset), a privacy-related text dataset with privacy indicators, and a method of recognizing whether observing a scene violates privacy without ethically sensitive training images. In constructing the LPT Dataset, which consists of both private and public scenes, we use an LLM to define privacy indicators and generate texts scored for each indicator. Our model recognizes whether a given image is private or public by retrieving texts with privacy scores similar to the image in a multi-modal feature space. In our experiments, we evaluated the performance of our model on three image privacy datasets and a realistic experiment with a humanoid robot in terms of accuracy and responsibility. The experiments show that our approach identifies the private image as accurately as the highly tuned LVLM without delay.

Parallel coordinate plots (PCPs) are a prevalent method to interpret the relationship between the control parameters and metrics. PCPs deliver such an interpretation by color gradation based on a single metric. However, it is challenging to provide such a gradation when multiple metrics are present. Although a naïve approach involves calculating a single metric by linearly weighting each metric, such weighting is unclear for users. To address this problem, we first propose a principled formulation for calculating the optimal weight based on a specific preferred metric combination. Although users can simply select their preference from a two-dimensional (2D) plane for bimetric problems, multi-metric problems require intuitive visualization to allow them to select their preference. We achieved this using various radar charts to visualize the metric trade-offs on the 2D plane reduced by UMAP. In the analysis using pedestrian flow guidance planning, our method identified unique patterns of control parameter importance for each user preference, highlighting the effectiveness of our method.

The emergence of large language models (LLM) has given rise to a growing concern regarding the generation and dissemination of inaccurate information through these technologies. Addressing this issue requires a benchmark for the safety of LLM for Japanese. However, existing benchmarks are limited in that fail to adequately incorporate the unique falsehoods and erroneous information that are actively circulating on social media in Japan. This study proposes JSocialFact, a benchmark for evaluating the safety of LLM based on misleading information in Japan. The benchmark is created by manually annotating the data extracted from X posts and community notes capturing a wide range of misleading, false, or malicious information currently circulating on social media. Both manual and automatic evaluations using GPT-4 revealed discrepancies in how models handle harmful content. Although GPT-4’s evaluation showed some correlation with human judgment, notable discrepancies were observed, with GPT-4 frequently assigning higher safety scores than human evaluators. JSocialFact is the first dataset constructed from actual social media logs in Japanese, specifically designed to evaluate the safety of LLM outputs in addressing misinformation.

Data augmentation is a technique with the potential to improve model generalization performance. Automatic Augmentation, which autonomously designs suitable data augmentation strategies, has significantly improved performance. One of the effective Automatic Augmentation is RandAugment (RA), which randomly selects methods to apply from the set of available augmentations. However, its selection with equal probabilities for all available augmentations is not ideal. In this study, we propose Affinity-Weighted RandAugment which assigns weights to the selection probabilities for the available augmentations in RA. The weights are calculated based on the value of Affinity, which represents the distributional similarity between original and augmented data. Affinity provides the prediction of the effect of each augmentation according to both the model and the dataset in advance. This enables the assignment of weights to the selection probabilities with low computational cost. As an additional effort, whether Affinity could be obtained at a much lower computational cost was considered. Based on these insights we implemented Single-Pass Affinity-Weighted RA, which can achieve effective weighting for RA without additional or separate learning. Experiments in image classification using CIFAR-10/100 and Describable Textures Dataset (DTD) demonstrated that Affinity-Weighted RandAugment achieves better test accuracy in comparison with the other known RA methods.

The ability to associate the current location with previously visited places is an essential aspect of autonomous ground robots. Unstructured environments such as planetary surfaces pose a significant challenge for robots because their terrain is less distinctive. Meanwhile, traversability must be analyzed simultaneously for safe navigation. In the past, place recognition research has rarely considered traversability analysis despite its significance. This is because the structural information of terrains becomes quickly implicit during the encoding process. This paper provides a method that explicitly addresses both problems: place recognition and traversability analysis. It proposes a discrete Fourier transform (DFT) to represent the frequency components embedded in ground curvature, which underlies both concepts. Our place recognition function demonstrates excellent performance in extensive experiments using challenging planetary & urban datasets while estimating traversability that other approaches find difficult to handle.

The proximity status of the individual poses a significant infection risk since infectious diseases can be transmitted through contact with others. Implementing crowd control measures to alleviate congestion is crucial in preventing infection spread during large-scale events with numerous spectators. To assess effective crowd control, comparing pedestrian behaviors and employing appropriate methods is essential. In this study, we propose a method for visualizing the influence of pedestrian behavior on infection risk by representing proximity as a network.

Appropriate evacuation guidance for specific events and facilities can be safely and rapidly achieved by controlling gates and emergency exits. However, in the case of large-scale evacuations, such as those in cities, it is difficult to obtain an optimal control method because of the complexity of the entire city. Understanding changes in pedestrian movement and urban characteristics requires an urgent need to build a dataset based on simulation results from various scenarios. In this study, we discuss the requirements for datasets in various studies, such as agent simulation, machine learning, and artificial intelligence.

This paper proposes a low latency neural network architecture for event-based dense prediction tasks. Conventional architectures encode entire scene contents at a fixed rate regardless of their temporal characteristics. Instead, the proposed network encodes contents at a proper temporal scale depending on its movement speed. We achieve this by constructing temporal hierarchy using stacked latent memories that operate at different rates. Given low latency event steams, the multi-level memories gradually extract dynamic to static scene contents by propagating information from the fast to the slow memory modules. The architecture not only reduces the redundancy of conventional architectures but also exploits long-term dependencies. Further-more, an attention-based event representation efficiently encodes sparse event streams into the memory cells. We conduct extensive evaluations on three event-based dense prediction tasks, where the proposed approach outperforms the existing methods on accuracy and latency, while demonstrating effective event and image fusion capabilities. The code is available at https://hamarh.github.io/hmnet/.

Hyperparameter optimization (HPO) is a vital step in improving performance in deep learning (DL). Practitioners are often faced with the trade-off between multiple criteria, such as accuracy and latency. Given the high computational needs of DL and the growing demand for efficient HPO, the acceleration of multi-objective (MO) optimization becomes ever more important. Despite the significant body of work on meta-learning for HPO, existing methods are inapplicable to MO tree-structured Parzen estimator (MO-TPE), a simple yet powerful MO-HPO algorithm. In this paper, we extend TPE’s acquisition function to the meta-learning setting using a task similarity defined by the overlap of top domains between tasks. We also theoretically analyze and address the limitations of our task similarity. In the experiments, we demonstrate that our method speeds up MO-TPE on tabular HPO benchmarks and attains state-of-the-art performance. Our method was also validated externally by winning the AutoML 2022 competition on “Multiobjective Hyperparameter Optimization for Transformers”. See https://arxiv.org/abs/2212.06751 for the latest version with Appendix.

Sensor fusion has become a popular topic in robotics. However, conventional fusion methods encounter many difficulties, such as data representation differences, sensor variations, and extrinsic calibration. For example, the calibration methods used for LiDAR-camera fusion often require manual operation and auxiliary calibration targets. Implicit neural representations (INRs) have been developed for 3D scenes, and the volume density distribution involved in an INR unifies the scene information obtained by different types of sensors. Therefore, we propose implicit neural fusion (INF) for LiDAR and camera. INF first trains a neural density field of the target scene using LiDAR frames. Then, a separate neural color field is trained using camera images and the trained neural density field. Along with the training process, INF both estimates LiDAR poses and optimizes extrinsic parameters. Our experiments demonstrate the high accuracy and stable performance of the proposed method.

Infectious diseases are typically transmitted through close contact with infected persons. The effective management of overcrowding is a crucial issue for events with a large number of attendees. Since the COVID-19 outbreak, analyzing people flow to recognize pedestrian behavior and walking patterns have been attracted studies. Visualizing crowded high-risk situations for infection at large gatherings is a complex task. It requires an approach that can effectively represent both spatial and temporal features while ensuring that the visibility of walking paths is not significantly compromised. To address these issues, we propose a novel approach for visualizing proximity as a network that represents the distance relationship between pedestrians. We developed the visualization system linking three components: Proximity Network, the walking paths of selected pedestrians from the network, and the temporal statistics of pedestrian traffic. Users of this system can freely select a group of pedestrians from Proximity Network and observe the paths of the selected pedestrians. This procedure enables better visibility of walking paths and an understanding of their spatio-temporal characteristics because only a smaller number of paths are drawn. This paper presents our case study of the proposed method for visualizing pedestrian proximity using real-world people flow data collected at an event site.

Estimating individual treatment effects from observational data is a fundamental problem in causal inference. To accurately estimate treatment effects in the spatial domain, we need to address certain aspects such as how to use the spatial coordinates of covariates and treatments and how the covariates and the treatments interact spatially. We introduce a new problem of predicting treatment effects on time series outcomes from spatial graph data with a hierarchical structure. To address this problem, we propose a spatial intervention neural network (SINet) that leverages the hierarchical structure of spatial graphs to learn a rich representation of the covariates and the treatments and exploits this representation to predict a time series of treatment outcome. Using a multi-agent simulator, we synthesized a crowd movement guidance dataset and conduct experiments to estimate the conditional average treatment effect, where we considered the initial locations of the crowds as covariates, route guidance as a treatment, and number of agents reaching a goal at each time stamp as the outcome. We employed state-of-the-art spatio-temporal graph neural networks and neural network-based causal inference methods as baselines, and show that our proposed method outperformed baselines both quantitatively and qualitatively.

This paper presents an unsupervised multichannel method that can separate moving sound sources based on an amortized variational inference (AVI) of joint separation and localization. A recently proposed blind source separation (BSS) method called neural full-rank spatial covariance analysis (FCA) trains a neural separation model based on a nonlinear generative model of multichannel mixtures and can precisely separate unseen mixture signals. This method, however, assumes that the sound sources hardly move, and thus its performance is easily degraded by the source movements. In this paper, we solve this problem by introducing time-varying spatial covariance matrices and directions of arrival of sources into the nonlinear generative model of the neural FCA. This generative model is used for training a neural network to jointly separate and localize moving sources by using only multichannel mixture signals and array geometries. The training objective is derived as a lower bound on the log-marginal posterior probability in the framework of AVI. Experimental results obtained with mixture signals of moving sources show that our method outperformed an existing joint separation and localization method and standard BSS methods.

Because human movement spreads infection, and mobility is a good proxy for other social distancing measures, human mobility has been an important factor in the COVID19 epidemic. Therefore, the control of human mobility is one of the countermeasures used to suppress an epidemic.As a notable feature, COVID19 has had multiple waves (subepidemics). Understanding the causes of the start and end of each wave has important implications for a policy evaluation and the timely implementation of countermeasures. Some of the waves have been correlated with the changes in mobility, and some can be attributed to the emergence of new variants. However, the start and end of some of the waves are difficult to explain through known factors.To evaluate the effect of human mobility, we built a stochastic model incorporating individual movements of 500,000 people obtained from anonymized, user-approved location data of smartphones throughout Japan. Instead of using aggregate values of human mobility, our model tracks the movements of individuals and predicts the infection of all persons within the entire country. Although the model only has a single static parameter, it successfully reproduced the occurrence of three waves of the number of confirmed cases within the study period of March 01 to December 31, 2020 in Japan. It was previously difficult to explain the end of the second wave and the start of the third wave in the study period by human mobility alone. Our results suggest the importance of tracking individual movements instead of relaying the aggregate values of human mobility.

Hyperparameter optimization (HPO) is a compu-tationally expensive blackbox optimization problem to maximize the performance of a machine learning model by tuning the model hyperparameters. Conventionally, global search has been widely adopted rather than local search to address HPO. In this study, we investigate whether this conventional choice is reasonable by empirically comparing popular global and local search methods as applied to HPO problems. The numerical results demonstrate that local search methods consistently achieve results that are comparable to or better than those of the global search methods, i.e., local search is a more reasonable choice for HPO. We also report the findings of detailed analyses of the experimental data conducted to understand how each method functions and the objective landscapes of HPO.

Recent advances in reinforcement learning (RL) have made it possible to develop sophisticated agents that excel in a wide range of applications. Simulations using such agents can provide valuable information in scenarios that are difficult to scientifically experiment in the real world. In this paper, we examine the play-style characteristics of football RL agents and uncover how strategies may develop during training. The learnt strategies are then compared with those of real football players. We explore what can be learnt from the use of simulated environments by using aggregated statistics and social network analysis (SNA). As a result, we found that (1) there are strong correlations between the competitiveness of an agent and various SNA metrics and (2) aspects of the RL agents play style become similar to real world footballers as the agent becomes more competitive. We discuss further advances that may be necessary to improve our understanding necessary to fully utilise RL for the analysis of football.

Neural radiance fields (NeRFs) have been proposed as methods of novel view synthesis and have been used to address various problems because of its versatility. NeRF can represent colors and densities in 3D space using neural rendering assuming a straight light path. However, a medium with a different refractive index in the scene, such as a transparent medium, causes light refraction and breaks the assumption of the straight path of light. Therefore, the NeRFs cannot be learned consistently across multi-view images. To solve this problem, this study proposes a method to learn consistent radiance fields across multiple viewpoints by introducing the light refraction effect as an offset from the straight line originating from the camera center. The experimental results quantitatively and qualitatively verified that our method can interpolate viewpoints better than the conventional NeRF method when considering the refraction of transparent objects.

Perceptual Aliasing is one of the main problems in simultaneous localization and mapping (SLAM). Wrong associations between different places may lead to failure of the whole map. Research on structure information is rarely investigated among existing solutions to this problem. In cases of visual SLAM without sensors, such as LiDAR or Inertial Measurement Unit (IMU), structure information can rarely be obtained due to the sparsity of 3D points, which also makes structure analysis complex. This study provides a spherical harmonics (SH) based fast structural representation (SH-FS) in visual SLAM using sparse point clouds, which extracts the structure information from sparse points into single vector. SH-FS was applied in conventional feature-based loop closing process. Furthermore, a structure-aware loop closing method in visual SLAM was proposed to improve the robustness of SLAM systems. Moreover, our methods show a favorable performance in extensive experiments on different large-scale real world datasets.

Practitioners often encounter challenging real-world problems that involve a simultaneous optimization of multiple objectives in a complex search space. To address these problems, we propose a practical multiobjective Bayesian optimization algorithm. It is an extension of the widely used Tree-structured Parzen Estimator (TPE) algorithm, called Multiobjective Tree-structured Parzen Estimator (MOTPE). We demonstrate that MOTPE approximates the Pareto fronts of a variety of benchmark problems and a convolutional neural network design problem better than existing methods through the numerical results. We also investigate how the configuration of MOTPE affects the behavior and the performance of the method and the effectiveness of asynchronous parallelization of the method based on the empirical results.

Hyperparameter optimization (HPO), formulated as black-box optimization (BBO), is recognized as essential for automation and high performance of machine learning approaches. The CMA-ES is a promising BBO approach with a high degree of parallelism, and has been applied to HPO tasks, often under parallel implementation, and shown superior performance to other approaches including Bayesian optimization (BO). However, if the budget of hyperparameter evaluations is severely limited, which is often the case for end users who do not deserve parallel computing, the CMA-ES exhausts the budget without improving the performance due to its long adaptation phase, resulting in being outperformed by BO approaches. To address this issue, we propose to transfer prior knowledge on similar HPO tasks through the initialization of the CMA-ES, leading to significantly shortening the adaptation time. The knowledge transfer is designed based on the novel definition of task similarity, with which the correlation of the performance of the proposed approach is confirmed on synthetic problems. The proposed warm starting CMA-ES, called WS-CMA-ES, is applied to different HPO tasks where some prior knowledge is available, showing its superior performance over the original CMA-ES as well as BO approaches with or without using the prior knowledge.

This paper proposes a novel heterogeneous grid convolution that builds a graph-based image representation by exploiting heterogeneity in the image content, enabling adaptive, efficient, and controllable computations in a convolutional architecture. More concretely, the approach builds a data-adaptive graph structure from a convolutional layer by a differentiable clustering method, pools features to the graph, performs a novel direction-aware graph convolution, and unpool features back to the convolutional layer. By using the developed module, the paper proposes heterogeneous grid convolutional networks, highly efficient yet strong extension of existing architectures. We have evaluated the proposed approach on four image understanding tasks, semantic segmentation, object localization, road extraction, and salient object detection. The proposed method is effective on three of the four tasks. Especially, the method outperforms a strong baseline with more than 90% reduction in floating-point operations for semantic segmentation, and achieves the state-of-the-art result for road extraction. We will share our code, model, and data.

A pedestrian road network is a complex non-linear dynamic system, and it is prone to severe accidents due to the congestion of spatiotemporal mass gathering. This paper proposes a framework of hybrid crowd dynamics modeling and model predictive control to ease the pressure of pedestrian road networks. The framework is integrated on the basis of a mature paradigm in the field of traffic signal control; the theory is yet new to pedestrian road networks except general traffic signals on the streets. The novelty also lies in its ability to simultaneously monitor both the discrete and continuous dynamics changes in the overall network. We verify the framework through comparison with simulation of manual control commands reproduced from real cases of a large-scale outdoor event. The results indicate that the proposed framework has a superior performance in keeping the balance in the overall distribution of large-scale crowd and reducing the risk of saturation in a pedestrian road network.

Crowd-related accidents often occur in both normal and emergency situations. To prevent these problems, it is highly suggested to investigate and simulate the risks of overcrowding in a large-scale gathering by using a multi-agent system. Such simulation enables the improvement of safe and efficient pedestrian route guidance, depending on multiple scenarios with complicated environmental and traffic conditions. In this paper, for practical safety pedestrian route guidance, we propose a multi-objective evolutionary optimization method to handle multiple scenarios in a large-scale firework event. The pedestrian dataset is obtained with a multi-agent traffic simulator, CrowdWalk. As the optimization of route guidance is a multi-objective optimization problem, we modify a natural evolution strategy based multi-objective optimization algorithm by replacing the Pareto dominance relation with the scenario dominance relation. This aims for the flexibility of pedestrian route guidance in response to traffic demands. The computational results demonstrate that the method can find a well-balanced set of solution to multiple scenarios and maintain a trade-off among multiple objectives in real world applications.

Practitioners often encounter computationally expensive multiobjective optimization problems to be solved in a variety of real-world applications. On the purpose of challenging these problems, we propose a new surrogate-based multiobjective optimization algorithm that does not require a large evaluation budget. It is called Multiobjective Tree-structured Parzen Estimator (MOTPE) and is an extension of the tree-structured Parzen estimator widely used to solve expensive single-objective optimization problems. Our empirical evidences reveal that MOTPE can approximate Pareto fronts of many benchmark problems better than existing methods with a limited budget. In this paper, we discuss furthermore the influence of MOTPE configurations to understand its behavior.

Mobile stereo matching systems are becoming more important in many applications such as auto-driving and autonomous robots. However, to maintain its low power consumption, mobile platforms have only limited hardware resources. Accurate stereo matching methods require a high computational complexity, and it is difficult to maintain both acceptable accuracy and processing speed on the mobile platforms. To solve this trade-off, in this paper, we propose a novel acceleration approach for a well-known matching algorithm Zero-means Normalized Cross Correlation (ZNCC), and show its effectiveness on a Jetson TX2 embedded GPU. By combining our new approach, Z2- ZNCC, with the Semi-Global Matching (SGM) algorithm, our system achieves a low error rate of 7.76% while keeping 28 fps for 1242×375 pixels images with the maximum disparity of 128 on the KITTI 2015 dataset. This performance is higher than previous state-of-the-art system on the same hardware platform.

In deep learning, hyperparameters can severely affect the learning model performance. The Nelder-Mead (NM) method is known for showing a superior performance for hyperparameter optimization in deep learning. An initial simplex, one of the initial NM method's values, is usually determined randomly while the search performance strongly depends on the shape of the initial simplex. Therefore, it is necessary to determine a proper initial simplex as previous researchers have proposed methods to construct an initial simplex from one starting point in the bounded search space. In this study, we verified how these methods for constructing an initial simplex contribute to improving the result of hyperparameter optimization in deep learning, by using a simple model and a complicated model. A smaller initial simplex may fail to optimization by bad local minima because there are some bad local minima in both learning models. We concluded that the starting point is not necessarily located close to the origin, and that a larger initial simplex contributes to improving the result of hyperparameter optimization in deep learning.

In this paper, we propose a SAR-to-optical image translation method based on conditional generative adversarial networks (cGANs). Though cGANs have achieved great success in image translation, some problems remain in SAR-to-optical image translation. One of the problems is the colorization error owing to the lack of color information in SAR data. Since the colors of optical images are varied, while SAR images have no color information, the generator network is confused and fail to generate correctly colorized optical images. To prevent it, we introduce a region information to the image translation network. Specifically, the feature vector from the pre-trained classification network is fed to the generator and discriminator network. Experimental results with SEN1-2 dataset show the advantage of our proposed method over the baseline method that does not use any additional information.

This study proposes a self-supervised method for detecting scene changes from an image pair. For mobile cameras such as drive recorders, to alleviate the camera viewpoints' difference, image alignment and change detection must be optimized simultaneously because they depend on each other. Moreover, lighting condition makes the scene change detection more difficult because it widely varies in images taken at different times. To solve these challenges, we propose a self-supervised simultaneous alignment and change detection net-work (SACD-Net). The proposed network is robust specifically in differences of camera viewpoints and lighting conditions to simultaneously estimate warping parameters and multi-scale change probability maps while change regions are not taken into account of calculation of the feature consistency and semantic losses. Based on comparative analysis between our self-supervised and the previous supervised models as well as ablation study of the losses of SACD-Net, the results show the effectiveness of the proposed method using a synthetic dataset and our new real dataset.

Detecting sound source objects within visual observation is important for autonomous robots to comprehend surrounding environments. Since sounding objects have a large variety with different appearances in our living environments, labeling all sounding objects is impossible in practice. This calls for self-supervised learning which does not require manual labeling. Most of conventional self-supervised learning uses monaural audio signals and images and cannot distinguish sound source objects having similar appearances due to poor spatial information in audio signals. To solve this problem, this paper presents a self-supervised training method using 360° images and multichannel audio signals. By incorporating with the spatial information in multichannel audio signals, our method trains deep neural networks (DNNs) to distinguish multiple sound source objects. Our system for localizing sound source objects in the image is composed of audio and visual DNNs. The visual DNN is trained to localize sound source candidates within an input image. The audio DNN verifies whether each candidate actually produces sound or not. These DNNs are jointly trained in a self-supervised manner based on a probabilistic spatial audio model. Experimental results with simulated data showed that the DNNs trained by our method localized multiple speakers. We also demonstrate that the visual DNN detected objects including talking visitors and specific exhibits from real data recorded in a science museum.

People flow has useful knowledge in various fields including traffic, disaster prevention, and marketing. Such information can be obtained by looking for the movement patterns of each person and the features of each place.We have proposed a visualization method that summarizes the paths of a large number of pedestrians in order to discover their useful information. However, we have not been working on visualization focusing on walking states whether people are moving rapidly or slowly. This paper presents a visualization method based on the walking states of a large number of pedestrian paths measured by using cameras. The visualization tool allows users to inspect walking people based on walking states from spatial and temporal views. The spatial view applies a heatmap and the temporal view adopts a stacked line graph. In addition, the user interface has a function to simultaneously visualize changes in the walking state distribution over time and walking paths.

The relationship between human behavior and SNS (social networking service) activity is growing stronger every year as SNS websites such as Twitter and Facebook develop and expand. Accordingly, a significant number of studies related to SNS have also been reported. However, the majority of this research, such as that which examines the transmission of information via SNS websites, focuses entirely on the virtual world of SNS and few studies have established connections between virtual and real space. This paper focuses on one SNS website, Twitter, and proposes a method for analyzing the statistical relationship between the number of tweets posted on Twitter and the number of people visiting a real world location in the form of a mutual interaction model consisting of nodes and links. We conducted an experiment to verify the proposed method, in which we compared measurements of the number of passers-by in Akihabara with the number of tweets posted.

People flow information brings us useful knowledge in various industrial and social fields including traffic, disaster prevention and marketing. However, it is still an open problem to develop effective people flow analysis techniques. We suppose compression and data mining techniques are especially important for analysis and visualization of large-scale people flow datasets. This paper presents a visualization tool for large-scale people flow dataset featuring compression and data mining techniques. This tool firstly compresses the people flow datasets using UniversalSAX, an extended method of SAX (Symbolic Aggregate Approximation). Next, we apply natural language algorithms to extract movement patterns. Finally, we visualize trajectories of people flow and extracted features to represent popular walking routes and congestions.

Thanks to the recent evolution of movie- and sensor-based human tracking technologies, we can obtain and accumulate a set of walking paths ("trajectories" in this paper) of people over a long period in various places. Such people flow datasets are useful for many fields, including analyses of customer behavior, effectiveness of advertisements, and operational efficiency. This paper presents a linked visualization system to assist in the discovery of new knowledge by analyzing the accumulated people flow datasets, and a case study using this system. In this study we suppose the people flow datasets consist of a set of trajectories and temporal flow quantity. The system consists of two visualization components: classified trajectory visualization, and temporal flow quantity visualization. The former component classifies trajectories into several patterns applying the spectral clustering algorithm, and visualizes the patterns by colors on a physical space. The latter component displays temporal flow quantity of the above patterns applying a piled polygonal chart. This paper introduces a case study applying a movie-based human tracking dataset to the presented system.

In facility management, analysis of customer trajectories in office complexes is considered critical. In this paper, we propose a novel approach for the visualization of customer flow in an office complex over a long period of time. We expressed the variation in the trajectories with respect to time by using a mixture model; this was used for the visualization of the trajectory flows. The effectiveness of our approach was evaluated from the results of the customer flow analysis experiments that were conducted in an office complex.

It is important for robots that act in human-centered environments to build image processing in a bottom-up manner. This paper proposes a method to autonomously acquire image feature extraction that is suitable for motion generation while moving in unknown environment. The proposed method extracts low level features without specifying image processing for robot body and obstacles. The position of body is acquired in image by clustering of SIFT features with motion information and state transition model is generated. Based on a learning model of adaptive addition of state transition model, collision relevant features are detected. Features that emerge when the robot can not move are acquired as collision relevant features. The proposed framework is evaluated with real images of the manipulator and an obstacle in obstacle avoidance.

Human-interactive robots, such as those used for nursing, which share humans' environments and interact with them, should be covered with soft areal tactile sensors for safety and dexterous manipulation. We report the successful development of the tactile sensor system of our human-interactive robot named RI-MAN, which can lift up a dummy human.

Recently, active researches have been performed to increase a robot's intelligence so as to realize the dexterous tasks in complex environment such as in the street or homes. However, since the skillful human-like task ability is so difficult to be formulated for the robot, not only the analytical and theoretical control researches but also the direct human motion mimetic approach is necessary. In this paper, we propose that to realize the environmental interactive tasks, such as human care tasks, it is insufficient to replay the human motion along. We show a novel motion generation approach to integrate the cognitive information into the mimic of human motions so as to realize the final complex task by the robot.

Our goal is to create advanced engineering systems such as a soft human interactive robot. The robot developed here is named RI-MAN. RI-MAN exhibits the skill and ability to realize human care and welfare tasks. RI-MAN can search out a specific person in real time by fuing audio and visual information, and understand human speech based on a sound recognition function. In addition, RI-MAN's body is coverd with soft touch sensors, and RI-MAN can react to the amplitude and location of external forces. Using all these sensor functions, RI-MAN can successfully follow human commands and hold up a dummy of the same size as an adult human. RI-MAN will become an invaluable partner robot.

Ionic polymer-metal composite (IPMC), which is one of the electro-active polymer actuators, is expected as artificial muscles for robots. An interesting property of IPMC is that it requires water to work, therefore it is suitable for underwater robots. In this paper, we developed an underwater robot which mimics rajiform swimming, that is the swimming form of a ray fish. Fins are designed using sixteen IPMCs. For autonomous operation, miniaturization of the electrical devices such as a micro controllers and small amplifiers are performed. A simple traveling wave control input is employed to generate moment on the fin. In the experiment, propulsion speed is measured under various control parameters. Furthermore, incremental wave of the fin is observed although the amplitude of the control input is spatially uniform. We also discuss this phenomenon from the point of view of interaction between elasticity of the actuator and fluid dynamics

With the development of information science and robotic technology, it becomes more important to generate human interactive robots. The design platform for developing such robots should satisfy three basic conditions: (1) it can test safely the performance of the robot through the physical interaction with human, (2) human subject can estimate subjectively the outside appearance of the robot, and (3) it can simulate the dynamic human interactive robot motion within real-time. This paper proposes our immersion type dynamic simulation platform. An application to estimate the robots performance when performing cooperative object lifting task with human subject is chosen in order to show the effectiveness of our system. The analysis of the recorded data is useful to design the novel human interactive robots.

TV conference systems have been widely used recently. A participant of each site proceeds with a TV conference using video image on a screen and voice of a partner site. In this case, a fixed video camera shoots a scene of a site in general. The video image taken by a fixed camera, however, is lacking in changes. Also the fixed camera does not take shots efficiency that the participants of a partner site pay attention. As one of the candidates to avoid these defects, there is a method that the computer-controlled cameras shoots the scene. In this paper we propose an algorithm of shooting the best shot by computer-controlled cameras. The shooting algorithm is mainly decided by estimating an area of the image with high degree of attention, which is given by not only visual information but also auditory information. By using an experimental system, we confirm the effectiveness of our approach by examining a questionnaire from the participants of TV conferences.

This paper defines a spatiotemporal mutual information on the pixels of a given video image on the basis of information theory (Shannon's communication theory), which can be interpreted as the theoretical estimation of interested features for human being. As an application of this spatiotemporal mutual information, we propose a method of producing a vivid video image of the distance learning by using the computer controlled camera operation and switching of plural camera images on the basis of the video image processing. Results of the questionnaire survey for the video image produced by the method confirm the effectiveness of our approach.

We propose a method for segmentation of written regions on a blackboard in the lecture room using a video image. We first detect the static edges of which locations on the image are stationary. Next, we extract several rectangular regions in which these static edges are located densely. Finally, by use of fuzzy rules, the extracted rectangles are merges as contextual regions, where letters and figures in each contextual region explain each context. We apply our method to automatic production of lecture video, and archives segmentation of written regions on the blackboard in lecture rooms.

In distributed network systems, one of the most important problems is how to assign the files to servers in view of cost and delay. It is obvious that there is a trade-off relationship between costs and delays in these systems. In order to evaluate the optimal file allocation such that the total cost is minimized subject to the total delay, we have presented a system model that can cover a wide range of multimedia network applications like VoD (video on demand), corporation information networks, and so on. We have solved the optimization problem as 0-1 integer programming, have found the optimal file allocation, and have investigated the general characteristics in distributed systems. In this model, we have introduced many cost and delay parameters to evaluate the total cost and delay in the system more exactly. In constructing practical systems, it is necessary to investigate the weight and the contribution of each parameter to the total cost. It is very useful to show how to estimate cost and delay parameters on the basis of this analysis. In this paper, we analyze the sensitivity of these parameters and make clear the influence between principal parameters.