3D point clouds are unstructured, sparse, and irregular data collected by airborne LiDAR systems over a geological region. Laser pulses emitted from the systems reflect off objects both on and above the ground, resulting in data with the longitude, latitude, and elevation of the points, and the corresponding laser pulse strengths. Ground filtering is important. The aim is to partition the points into ground and non-ground subsets. In addition, this research introduces a novel task: detecting and identifying human-made objects amidst natural tree structures. The task is performed on the non-ground subset derived given by the ground filtering stage. Marked Point Fields (MPFs) are used to these tasks. The proposed methodology consists of three stages: ground filtering, local information extraction (LIE), and clustering. In the ground filtering stage, a statistical method called One-Sided Regression (OSR) is devised to overcome the limitations of prior ground filtering methods on uneven terrains. In the LIE stage, a kernel-based method for the Hessian matrix of the MPF is developed. In the clustering stage, the Gaussian Mixture Model (GMM) is applied to the results of the LIE for partitioning the non-ground points into trees and human-made objects. The underlying assumption is that LiDAR points from trees exhibit a three-dimensional distribution, while those from human-made objects follow a two-dimensional distribution. The Hessian matrix of the MPF effectively captures the difference. Experimental results demonstrate that the proposed ground filtering method outperforms previous techniques, and the LIE method successfully distinguishes between points representing trees and human-made objects.

The Geographical Detector Model (GDM) is a popular statistical toolkit for geographical attribution analysis. Despite the striking resemblance of the q-statistic in GDM to the R-squared in linear regression models, their explicit connection has not yet been established. This study proves that the q-statistic reduces into the R-squared under a linear regression framework. Under linear regression and moderate-to-strong spatial autocorrelation, Monte Carlo simulation results show that the GDM tends to underestimate the importance of variables. In addition, an almost perfect power law relationship is present between the percentage bias and the degree of the spatial autocorrelations, indicating the presence of fast uplifting bias in response to increasing levels of spatial autocorrelations. We propose an integrated approach for variable importance quantification by bringing together the spatial econometrics model and the game theory based-Shapley value method. By applying our proposed methodology to a case study of land desertification in African, it is found human activity tends to affect land desertification both directly and indirectly. However, such effects appear to be underestimated or undistinguished in the classic GDM.

Categorical variables are nominal variables that classify observations by groups. The treatment of categorical variables in regression is a well-studied yet vital problem, with the most popular solution to perform a one hot encoding. However, challenges arise if a categorical variable has millions of levels. It will cause the memory needed for the computation far exceeds the total available memory in a given computer system or even a computer cluster. Thus, it is fair to state that one hot encoding approach has its limitations when a categorical variable has a large number of levels. The common workaround is the sparse matrix approach because it requires much fewer resources to cache the dummy variables. However, existing sparse matrix approaches are still not sufficient to handle extreme cases when a categorical variable has millions of levels. For instance, the number of subnets in network traffic analyses can easily exceeds tens of millions. In this paper, we proposed an innovative approach called sparse block regression (SBR) to address this challenge. SBR constructs a sparse block matrix using sufficient statistics. The benefits include but not limited to: 1) overcome the memory barrier issue caused by one hot encoding, 2) obtain multiple models with a single scan of data stored in the secondary storage; and 3) update the models with simple matrix operations. The study compared proposed SBR against conventional sparse matrix approaches. The experiments proved that SBR can efficiently and accurately solve the regression problem with large category number. Compared to the sparse matrix approach, SBR saved 90% memory in size during the computation.

Ridge regression is a regularization technique that can be used together with other regression algorithms to model highly correlated data. Like many other traditional techniques, ridge regression of big data version requires a large number of iterations over the dataset to converge. As the dataset cannot all be stored in memory, the dataset is split into RAM-accommodable subsets for training, however, this strategy is time-consuming for reading all subsets from hard drive to memory over and over. To overcome the memory barrier, we proposed to use working sufficient statistics to solve the problem [1]. The parameters of the working sufficient matrix is small enough to be stored in RAM all the time. They can be updated at per row level to allow online computation. This strategy only requires one iteration over the dataset. While our previous work proved its theoretical correctness, it was not clear how our innovative algorithm would work in practice. In this study, we aims to validate and evaluate the performance improvement of the algorithm we proposed in earlier work-Three sets of experiments were conducted using large data-sets published by FAA and BTS to examine the computation time, memory requirement, and the accuracy of the output. Results showed that our exact ridge aggression algorithm enjoyed many benefits, such as faster computing time, minimal memory requirements and more accurate estimates.

This research proposed a novel fitting algorithm for big data logistic regression by combining Fisher Scoring and IRWLS. The algorithm enables streamed operation to fit and update the model at per-row level, without the need to store the entire dataset in RAM. The algorithm is fully parallelizable and was implemented on Spark for this study. A set of experiments were conducted on Spark and Scikit-Learn to compare its performance against existing approaches. The results showed that the proposed method can provide exact results, rather than approximate results, with significantly fewer iterations. It also converged in merely 2 iterations in terms of prediction accuracy. More importantly, the proposed algorithm has a very small memory footprint and completely broke the memory barrier problem that debilitated conventional logistic regression approaches. Compared to the SGD sampling approach in scikit-learn, the proposed approach also demonstrated a clear advantage in accuracy.

In order to find the best linear regression model or polynomial regression model that fits the data, traditional methods have to read the whole datasets repetitively and incur many unnecessary slow I/O operations. Apache Spark can train regression models significantly more efficiently with distributed clusters due to its well-crafted in-memory computing architecture. However, if the dataset itself or the temporary data during computation is even bigger for the total physical memory space of a spark system, in-memory data has to be spilled to the secondary storage (such as hard drives or solid state disks) and read it back later if it is needed. These frequent I/O operations will negatively affect the efficiency of Spark computation. Built on top of the per-row update-able data modeling concept we proposed before, this work investigated the cases of finding the best Box-Cox transformation model on a Spark system. The major contribution of this work is that the information needed to compute a linear regression model, or a polynomial regression model can be summarized in an Information Array. The size of this information array does not grow with the datasets. Rather, it is only related to the number of features and the number of models need to be considered. Because the information array is usually very small, it can be stored in memory all the time. With the propose information array approach, the best linear or polynomial regression model could be obtained after one scan of the raw data. The experiment results proved that this approach is fast and efficient on Spark. When training 41 models, the proposed Box-Cox Information Array method is about 8 times faster than the existing Spark APIs and it has better performance of prediction than using linear regression models.