Public datasets, such as KITTI, nuScenes, and Waymo, have played a key role in the research and development of autonomous vehicles and advanced driver assistance systems. However, many of these datasets fail to incorporate a full range of driving conditions; some datasets only contain clear-weather conditions, underrepresenting or entirely missing colder weather conditions such as snow or autumn scenes with bright colorful foliage. In this paper, we present the Michigan State University Four Seasons (MSU-4S) Dataset, which contains real-world collections of autonomous vehicle data from varied types of driving scenarios. These scenarios were recorded throughout a full range of seasons, and capture clear, rainy, snowy, andfall weather conditions, at varying times of day. MSU-4S contains more than 100,000 two-and three-dimensional frames for camera, lidar, and radar data, as well as Global Navigation Satellite System (GNSS), wheel speed, and steering data, all annotated with weather, time-of-day, and time-of-year. Our data includes cluttered scenes that have large numbers of vehicles and pedestrians; and it also captures industrial scenes, busy traffic thoroughfare with traffic lights and nu-merous signs, and scenes with dense foliage. While pro-viding a diverse set of scenes, our data incorporate an im-portant feature: virtually every scene and its corresponding lidar, camera, and radar frames were captured in four different seasons, enabling unparalleled object detection analysis and testing of the domain shift problem across weather conditions. In that context, we present detailed analyses for 3D and 2D object detection showing a strong domain shift effect among MSU-4S data segments collected across different conditions. MSU-4S will also enable advanced mul-timodal fusion research including different combinations of camera-lidar-radar fusion, which continues to be of strong interest for the computer vision, autonomous driving and ADAS development communities. The MSU-4S dataset is available online at https://egr.msu.edu/waves/msu4s.

Object detection using LiDAR point clouds relies on a large amount of human-annotated samples when training the underlying detectors' deep neural networks. However, generating 3-D bounding box annotation for a large-scale dataset could be costly and time-consuming. Alternatively, unsupervised domain adaptation (UDA) enables a given object detector to operate on novel new data, with an unlabeled training dataset, by transferring the knowledge learned from training labeledsource domaindata to the new unlabeledtarget domain. Pseudolabel strategies, which involve training the 3-D object detector using target-domain predicted bounding boxes from a pretrained model, are commonly used in UDA. However, these pseudolabels often introduce noise, impacting performance. In this article, we introduce the domain adaptive LiDAR (DALI) object detection framework to address noise at both distribution and instance levels. First, a posttraining size normalization (PTSN) strategy is developed to mitigate bias in pseudolabel size distribution by identifying an unbiased scale after network training. To address instance-level noise between pseudolabels and corresponding point clouds, two pseudopoint clouds generation (PPCG) strategies, ray-constrained and constraint-free, are developed to generate pseudopoint clouds for each instance, ensuring the consistency between pseudolabels and pseudopoints during training. We demonstrate the effectiveness of our method on the publicly available and popular datasets KITTI, Waymo, and nuScenes. We show that the proposed DALI framework achieves state-of-the-art results and outperforms leading approaches on most of the domain adaptation tasks.

The adversarial robustness of a model is its ability to resist adversarial attacks in the form of small perturbations to input data. Universal adversarial attack methods such as Fast Sign Gradient Method (FSGM) [1] and Projected Gradient Descend (PGD) [2] are popular for LiDAR object detection, but they are often deficient compared to task-specific adversarial attacks. Additionally, these universal methods typically require unrestricted access to the model's information, which is difficult to obtain in real-world applications. To address these limitations, we present a black-box Scaling Adversarial Robustness (ScAR) method for LiDAR object detection. By analyzing the statistical characteristics of 3D object detection datasets such as KITTI, Waymo, and nuScenes, we have found that the model's prediction is sensitive to scaling of 3D instances. We propose three black-box scaling adversarial attack methods based on the available information: model-aware attack, distribution-aware attack, and blind attack. We also introduce a strategy for generating scaling adversarial examples to improve the model's robustness against these three scaling adversarial attacks. Comparison with other methods on public datasets under different 3D object detection architectures demonstrates the effectiveness of our proposed method.

Unsupervised domain adaptation (UDA) focuses on transferring knowledge learned in the labeled source domain to the unlabeled target domain. Semi-supervised learning is a proven strategy for improving UDA performance. In this paper, we propose a novel strong-weak integrated semi-supervision (SWISS) learning strategy for unsupervised domain adaptation. Under the proposed SWISSUDA framework, a strong representative set with high confidence but low diversity target domain samples and a weak representative set with low confidence but high diversity target domain samples are updated constantly during the training process. Both sets are fused randomly to generate an augmented strong-weak training batch with pseudo-labels to train the network during every iteration. Moreover, a novel adversarial logit loss is proposed to reduce the intra-class divergence between source and target domains, which is back-propagated adversarially with a gradient reverse layer between the classifier and the rest of the network. Experimental results based on two popular benchmarks, Office-Home, and DomainNet, show the effectiveness of the proposed SWISS framework with our method achieving the best performance in both Office-Home and DomainNet.

One interesting question between the Homo Sapiens and other mammal species is their protein sequence similarities. In this study, based on the global alignment algorithm with the substitution matrix of “BLOSUM62”, calculations were carried out on protein sequence similarities within Homo Sapiens and mammal species, protein families and specified proteins across species. As a result, protein sequence similarities among Homo Sapiens and other mammal species are around 19%; protein families share similarities ranges from around 20%-90%.

Dioscoreae Rhizoma, also called Shan-Yao in Chinese, is a tonic food and herbal medicine in traditional Chinese medicine (TCM). Recent biomedical researches on Shan-Yao mainly focused on cancer and immune responses that are scattered in the literature. To build a holistic view of the immunoregulation functions delivered by Shan-Yao, the dispersed experimental knowledge items were collected, integrated, analyzed, and demonstrated. As a result, a candidate regulation network was constructed towards the immunoregulation. This network was composed of Shan-Yao, bioactive compounds, target proteins, protein interactions, pathway enrichment, and functional annotations. Functional annotations demonstrated the cell killing process is tightly associated with immunoregulation scenarios under different conditions via pathway enrichment, all are participated by the same target proteins of Shan-Yao.

We present a novel method for generating panoramic street-view images which are geometrically consistent with a given satellite image. Different from existing approaches that completely rely on a deep learning architecture to generalize cross-view image distributions, our approach explicitly loops in the geometric configuration of the ground objects based on the satellite views, such that the produced ground view synthesis preserves the geometric shape and the semantics of the scene. In particular, we propose a neural network with a geo-transformation layer that turns predicted ground-height values from the satellite view to a ground view while retaining the physical satellite-to-ground relation. Our results show that the synthesized image retains well-articulated and authentic geometric shapes, as well as texture richness of the street-view in various scenarios. Both qualitative and quantitative results demonstrate that our method compares favorably to other state-of-the-art approaches that lack geometric consistency.

It is a classical task to automatically extract road networks from very high-resolution (VHR) images in remote sensing. This paper presents a novel method for extracting road networks from VHR remotely sensed images in complex urban scenes. Inspired by image segmentation, edge detection, and object skeleton extraction, we develop a multitask convolutional neural network (CNN), called RoadNet, to simultaneously predict road surfaces, edges, and centerlines, which is the first work in such field. The RoadNet solves seven important issues in this vision problem: 1) automatically learning multiscale and multilevel features [gained by the deeply supervised nets (DSN) providing integrated direct supervision] to cope with the roads in various scenes and scales; 2) holistically training the mentioned tasks in a cascaded end-to-end CNN model; 3) correlating the predictions of road surfaces, edges, and centerlines in a network model to improve the multitask prediction; 4) designing elaborate architecture and loss function, by which the well-trained model produces approximately single-pixel width road edges/centerlines without nonmaximum suppression postprocessing; 5) cropping and bilinear blending to deal with the large VHR images with finite-computing resources; 6) introducing rough and simple user interaction to obtain desired predictions in the challenging regions; and 7) establishing a benchmark data set which consists of a series of VHR remote sensing images with pixelwise annotation. Different from the previous works, we pay more attention to the challenging situations, in which there are lots of shadows and occlusions along the road regions. Experimental results on two benchmark data sets show the superiority of our proposed approaches.

The structural features in Manhattan world encode useful geometric information of parallelism, orthogonality and/or coplanarity in the scene. By fully exploiting these structural features, we propose a novel monocular SLAM system which provides accurate estimation of camera poses and 3D map. The foremost contribution of the proposed system is a structural feature-based optimization module which contains three novel optimization strategies. First, a rotation optimization strategy using the parallelism and orthogonality of 3D lines is presented. We propose a global binding method to compute an accurate estimation of the absolute rotation of the camera. Then we propose an approach for calculating the relative rotation to further refine the absolute rotation. Second, a translation optimization strategy leveraging coplanarity is proposed. Coplanar features are effectively identified, and we leverage them by a unified model handling both points and lines to calculate the relative translation, and then the optimal absolute translation. Third, a 3D line optimization strategy utilizing parallelism, orthogonality and coplanarity simultaneously is proposed to obtain an accurate 3D map consisting of structural line segments with low computational complexity. Experiments in man-made environments have demonstrated that the proposed system outperforms existing state-of-the-art monocular SLAM systems in terms of accuracy and robustness.

In this paper, we propose a unified model for camera pose estimation and a novel strategy for pose optimization by combining points and lines in monocular visual odometry. Our proposed unified model treats point and line features equivalently, which is applicable for all the minimal cases requiring the minimum number 3 of point or/and line features and can be easily extended for various circumstances with more additional observations. The core idea is to directly retrieve all stationary points of a cost function which is minimized by the first-order optimality condition without initialization or iteration. The estimated pose is reliable due to robust geometric constraints and the reliable algebraic solver. To refine the camera pose, we propose a novel optimization strategy to minimize the unconstrained Sampson error by taking specific uncertainty for each feature into account to penalize noise more reasonably. Moreover, it is simpler than the conventional bundle adjustment by avoiding the high-dimensional parameter searching. Experimental results on simulated data and real images have sufficiently demonstrated the superiority of our proposed camera pose estimation and optimization method by comparing with state-of-the-art monocular algorithms.

This paper presents a very simple and efficient algorithm to estimate 1, 2 or 3 orthogonal vanishing point(s) on a calibrated image in Manhattan world. Unlike the traditional methods which apply 1, 3, 4, or 6 line(s) to generate vanishing point hypotheses, we propose to use 2 lines to get the first vanishing point v1, then uniformly take sample of the second vanishing point v2 on the great circle of v1 on the equivalent sphere, and finally calculate the third vanishing point v3 by the cross-product of v1 and v2. There are three advantages of the proposed method over traditional multi-line method. First, the 2-line model is much more robust and reliable than the multi-line method, which can be applied in the scene with 1, 2 or 3 orthogonal vanishing point(s). Second, the probability of the 2-line model being formed of inner line segments can be calculated given the outlier ratio, which means that the number of iterations can be determined, and thus the estimation of vanishing points can be performed in a very simple exhaustive way instead of the traditional RANSAC method. Third, the real-time performance is achieved by building a polar grid for the line intersection points, which functions as a lookup table for the validation of vanishing point hypotheses. Our algorithm has been validated successfully in the YUD dataset and sets of challenging real images.

In this paper, we present an efficient edge chain detection algorithm by applying the Helmholtz principle on the gradient magnitude map of an image. An edge chain validation method is proposed which uses the “relative number of false alarms” (RNFA) instead of the traditional “number of false alarms” (NFA). The edge chains are detected first and then validated according to their RNFA values. In this way, edge chains that are weak in gradients but meaningful in vision can be detected. To evaluate the proposed edge chain detector in quantity, an edge chain detection benchmark which consists of 25 labeled images in different scenes was built. The proposed edge chain detector was tested in this benchmark, and the experimental results sufficiently demonstrate that the proposed edge chain detector outperforms the state-of-the-art methods.

In this paper, we present a robust line segment detection algorithm to efficiently detect the line segments from an input image. Firstly a parameter-free Canny operator, named as CannyPF, is proposed to robustly extract the edge map from an input image by adaptively setting the low and high thresholds for the traditional Canny operator. Secondly, both efficient edge linking and splitting techniques are proposed to collect collinear point clusters directly from the edge map, which are used to fit the initial line segments based on the least-square fitting method. Thirdly, longer and more complete line segments are produced via efficient extending and merging. Finally, all the detected line segments are validated due to the Helmholtz principle [1, 2] in which both the gradient orientation and magnitude information are considered. Experimental results on a set of representative images illustrate that our proposed line segment detector, named as CannyLines, can extract more meaningful line segments than two popularly used line segment detectors, LSD [3] and ED-Lines [4], especially on the man-made scenes.

In this paper, we present a robust method to efficiently create a globally consistent and seamless mosaic from aerial images. Firstly, a globally consistent registration strategy is proposed to align the aerial images in a common coordinate system, which combines the affine model with the homographic model effectively. To suppress the accumulation of perspective distortions induced by a sequential set of aerial images taken from a wide-range region, we proposed to initially align each image by an affine model and then perform a homographic refinement in groups to increase the global consistency. Secondly, to efficiently conceal the parallax between aligned images in overlap regions with large depth differences where it is impossible to recover a highly accurate consistent image registration, a novel optimized seamline detection algorithm in the graph cuts energy minimization framework is proposed to find optimal seamlines within overlap regions for image mosaicking through rounding visually obvious foreground objects. Finally, experimental results on several representative image sets illustrate the superiority of our proposed approaches.

Color-based vision applications face the challenge that colors are variant to illumination. In this paper we present a color classification algorithm that is adaptive to continuous variable lighting. Motivated by the dichromatic color reflectance model, we use a Gaussian mixture model (GMM) of two components to model the distribution of a color class in the YUV color space. The GMM is derived from the classified color pixels using the standard expectation-maximization (EM) algorithm, and the color model is iteratively updated over time. The novel contribution of this work is the theoretical analysis supported by experiments - that a GMM of two components is an accurate and complete representation of the color distribution of a dichromatic surface