Traffic flow prediction is a fundamental task in intelligent transportation systems (ITS). Due to the influence of urban functional zones and their neighboring regions, traffic flow data exhibit complex spatiotemporal correlations, making it challenging to effectively capture temporal dependencies and spatial structures for accurate prediction. To address this issue, this article proposes a multiattention enhanced spatiotemporal fusion network (STFAENet) based on the TransUNet architecture. STFAENet consists of an encoder, a skip-connection mechanism, and a decoder, and is designed to jointly learn fine-grained local features and global spatiotemporal dependencies in dynamic traffic scenarios. Specifically, an instance pyramid spatial attention (IPSA) module is introduced in the encoder to enhance multiscale spatial feature representation through hybrid normalization and pyramid attention, enabling the extraction of high-resolution fine-grained features. In the skip-connection stage, a spatiotemporal self-attention (STSA) module is embedded to jointly model spatial and temporal dependencies and reduce the semantic gap between the encoder and decoder. The decoder further fuses local details with global contextual information to generate high-resolution prediction results. Experiments on the TaxiBJ and TaxiCQ datasets demonstrate that STFAENet achieves superior prediction performance compared with several baseline methods.

Image background recognition and reconstruction is an important research content in the field of computer vision. Instead of paying attention to all the targets in the image, this paper only focuses on the static background in the scene. Therefore, an adaptive semantic propagation method is proposed to reconstruct the complete background of static images. It executes single-connected region division and edge detection on a single frame. Experimental results demonstrate the robustness and adaptability of the method, achieving convincing performance under both simple and complex road conditions.

With the development of smart cities, video surveillance has become more prevalent in urban areas. The rapid growth of data brings challenges to video processing and analysis. Multi-object tracking (MOT), one of the most fundamental tasks in computer vision, has a wide range of applications and development prospects. MOT aims to locate multiple objects and maintain their unique identities by analyzing the video frame by frame. Most existing MOT frameworks are deployed in centralized systems, which are convenient for management but have problems such as weak algorithm adaptability, limited system scalability, and poor data security. In this paper, we propose a distributed MOT algorithm based on multi-agent reinforcement learning (DMARL-Tracker), which formulates MOT as a Markov decision process (MDP). Each object adjusts its tracking strategy during interactions with the environment. The benchmark results on MOT17 and MOT20 prove that our proposed algorithm achieves state-of-the-art (SOTA) performance. Based on this, we further integrate DMARL-Tracker into the blockchain and propose a blockchain-based collaborative MOT framework. All nodes collaborate and share information through the blockchain, achieving adaptation in different complex scenarios while ensuring data security. The simulation results show that our framework achieves good performance in terms of tracking and resource consumption.

In the current industrial informatics society, the numerous cameras deployed in the modern city promote the development of various video services, such as security monitoring and object retrieval. However, traditional methods encounter data leakage risks. Some camera owners are reluctant to share their data since the video contains confidential information. Meanwhile, domain diversities between cameras bring obstacles to practical object retrieval applications. To deal with these dilemmas, we propose a blockchain-based collaborative object retrieval (BCOR) system that can protect privacy as much as possible. BCOR includes two core components: multicamera reidentification framework (MC-ReF) and multicamera collaborative chain (M2C-Chain). Specifically, MC-ReF leverages visual relevance attention net (VRANet) to distinguish object identities in edge nodes. Through domain adaptation gradient optimization, VRANet can adapt to different cameras without the need for private camera data. M2C-Chain is responsible for maintaining the security and trust of the system. Through M2C-Chain, the collaboration among different nodes is transferred into a transaction-based manner, which is validated by a deeply integrated consensus. Finally, we implement a prototype system and deploy it into a real-world outdoor scene. The experiments indicate that BCOR achieves 30%–35% average improvement in domain adaptation on mean average precision and Rank-1 indicators. The performance analysis and security experiments also prove the efficiency and stability of BCOR.

The rapid increase in the volume of video data generated from edges in the Industrial Internet of Things, opens up new possibilities for enhancing the application of video service. Multicamera multiobject tracking (MCMT) has always been a fundamental task in video surveillance or traffic control. However, the traditional MCMT methods are limited by the communication bottleneck and computation resources of the centralized curator, and suffer from security and privacy issues. In this article, we first design multicamera multihypothesis tracking (MC-MHT) framework to achieve real-time tracking performance among edge cameras. The complex association of objects is described by multiskip trees. The tracking task is well distributed to each camera. Then, we integrate multicamera tracking chain into MC-MHT to ensure security and trust. The state transition of targets in multicamera is illustrated from the perspective of blockchain transactions. The transactions are validated by an integrated tracking consensus to counter Byzantine behavior. Numerical results derived from real-world scenarios and CAMPUS dataset show that the proposed method achieves real-time performance (24–36 FPs) and 79.0–82.4 MOTA indicator, as well as reduces identity switch errors about 71% under Byzantine attack.

Detecting the position and safe wearing of workers is an significant topic in industrial production. However, mainstream detectors aware object instances individually instead of exploring contextual information. In this article, a relation extraction module (REM) is proposed to introduce local and global contexts at the same time. It processes a set of anchors simultaneously through interaction between their appearance feature and location, thus allowing building local context and generating enhanced anchors. It can be plugged into most popular detectors without additional labeling. Experiments on public datasets and onsite surveillance video indicate that REM improves the accuracy of single-stage detectors especially small models while maintains real-time performance. A real-time intelligent surveillance system has already been established and applied in the factory, which makes great significance to the management of safety supervision departments.

It is well known that context can help object detection, but mainstream single-stage object detection algorithms still detect object instances individually. In this work, we propose a Relation Extraction Module (REM) which extracts both global context and local context at the same time. It processes a set of anchors simultaneously through interaction between their appearance and spatial features, thus allowing building local context. It gets the global context by filtering and averaging all anchor features of the current feature layer, thus allowing building background information of the image. It does not need additional manual labeling information and is easy to plug into popular detectors. Experiments on MS COCO datasets indicate that REM improves the accuracy of the popular single-stage detector while maintains real-time performance.

Abstract Multi‐object tracking is an important branch of computer vision, which is mostly used for behavior recognition and event analysis. At present, most of the research focuses on the accuracy of tracking. However, the real‐time performance is also urgently desired but there is lack of research. As a result, Deepsort, proposed 5 years ago, is still the most widely used tracker in real applications. In this paper, a Bilateral Association Tracking (BAT) framework is proposed. It uses tracklet as the basic node instead of discrete detection for tracking. Meanwhile, a Parzen density based Hierarchical Agglomerative Clustering (P‐HAC) algorithm is introduced to describe the density distribution of targets and generate tracklets with high confidence. In addition, Dual Appearance Features (DAF) is proposed which considers both spatial and temporal features of tracklets and promotes the accuracy of tracklet association. Experiments are conducted on popular benchmarks such as MOT2017, Visdrone and KITTI. BAT outperforms Deepsort on both association accuracy and trajectory integrity without obvious efficiency decline. Compared with other state‐of‐the‐art trackers, BAT shows significant advantage on computational cost while performing competitive tracking accuracy as well. It is hoped that the research can promote the applications on real‐time tracking in the near future.

Abstract Recently, zebrafish has received more and more attention due to its wide range of applications such as regeneration promoting therapeutics and drug discovery. Therefore, vision‐based trackers are utilized to record the swimming trajectory of zebrafish. In this paper, a re‐association method is introduced in the 3D reconstruction process to generate missed targets caused by occlusion. Since the variation of the overall tracking targets has the property of continuity and stability, a topology association model (TAM) is proposed by point group similarity into the tracking framework. TAM describes the movement of zebrafish from the macroscopic level and utilizes the changes of the point group structure for tracking. Experimental results show that the tracking framework enhances the overall performance and promotes the trajectory integrity. On the latest 3D‐ZeF20 benchmark, state‐of‐the‐art results are achieved. In addition, TAM tracking framework is applied to 2D general tracking to prove that the method is useful and have great advantage in other scenarios with relatively stable amount of targets as well.

For an intelligent transportation system, multiple object tracking (MOT) is more challenging from the traditional static surveillance camera to mobile devices of the Internet of Things (IoT). To cope with this problem, previous works always rely on additional information from multivision, various sensors, or precalibration. Only based on a monocular camera, we propose a hybrid motion model to improve the tracking accuracy in mobile devices. First, the model evaluates camera motion hypotheses by measuring optical flow similarity and transition smoothness to perform robust camera trajectory estimation. Second, along the camera trajectory, smooth dynamic projection is used to map objects from image to world coordinate. Third, to deal with trajectory motion inconsistency, which is caused by occlusion and interaction of long time interval, tracklet motion is described by the multimode motion filter for adaptive modeling. Fourth, in tracklets association, we propose a spatiotemporal evaluation mechanism, which achieves higher discriminability in motion measurement. Experiments on MOT15, MOT17, and KITTI benchmarks show that our proposed method improves the trajectory accuracy, especially in mobile devices and our method achieves competitive results over other state-of-the-art methods.

Recently, tracking-by-detection has become a popular paradigm in Multiple-object tracking (MOT) for its concise pipeline. Many current works first associate the detections to form track proposals and then score proposalns by manual functions to select the best. However, long-term tracking information is lost in this way due to detection failure or heavy occlusion. In this paper, the Extendable Multiple Nodes Tracking framework (EMNT) is introduced to model the association. Instead of detections, EMNT creates four basic types of nodes including correct, false, dummy and termination to generally model the tracking procedure. Further, we propose a General Recurrent Tracking Unit (RTU++) to score track proposals by capturing long-term information. In addition, we present an efficient generation method of simulated tracking data to overcome the dilemma of limited available data in MOT. The experiments show that our methods achieve state-of-the-art performance on MOT17, MOT20 and HiEve benchmarks. Meanwhile, RTU++ can be flexibly plugged into other trackers such as MHT, and bring significant improvements. The additional experiments on MOTS20 and CTMC-v1 also demonstrate the generalization ability of RTU++ trained by simulated data in various scenarios.

Recent progress in multi-object tracking (MOT) has shown great significance of a robust scoring mechanism for potential tracks. However, the lack of available data in MOT makes it difficult to learn a general scoring mechanism. Multiple cues including appearance, motion and etc., are limitedly utilized in current manual scoring functions. In this paper, we propose a Multiple Nodes Tracking (MNT) framework that adapts to most trackers. Based on this framework, a Recurrent Tracking Unit (RTU) is designed to score potential tracks through long-term information. In addition, we present a method of generating simulated tracking data without real data to overcome the defect of limited available data in MOT. The experiments demonstrate that our simulated tracking data is effective for training RTU and achieves state-of-the-art performance on both MOT17 and MOT16 benchmarks. Meanwhile, RTU can be flexibly plugged into classic trackers such as DeepSORT and MHT, and makes remarkable improvements as well.

Multiobject tracking is a basic task in video analysis. Due to the strict requirements on efficiency and resource consumption, most of the applications on edge devices are online or near-online methods. Besides motion modeling, appearance information is also widely used for tracking. However, the influence of occlusion is usually ignored. In this article, spatial-temporal co-occurrence constraints (STCCs) features are introduced to resist occlusions by exploring the rich spatial and temporal information of tracklets. In addition, a novel blockchain-based near-online framework called co-occurrence constraints tracklet tracker (CoCTs) is proposed for cross-camera tracking. It inherits the advantages of the blockchain technology in sharing information. Based on blockchain, an efficient association mechanism and a reliable information sharing method are introduced. Experimental results show that CoCT performs high computational efficiency and low resource consumption. In the edge computing environment, it achieves real-time performance on cross-camera tracking. On the MOT17 benchmark, our method shows the state-of-the-art results compared with other online trackers.

In recent years, the demand for intelligent devices related to the Internet of Things (IoT) is rapidly increasing. In the field of computer vision, many algorithms have been preinstalled in IoT devices to achieve higher efficiency, such as face recognition, area detection, target tracking, etc. Tracking is an important but complex task that needs high efficiency solutions in real applications. There is a common assumption that detection can only represent one pedestrian to describe nonoverlapping in physical space. In fact, the pixels of the image do not exactly correspond to the positions in the real world. In order to overcome the limitation of this assumption, we remove this unreasonable assumption and present a novel idea that each detector response can have multiple labels to describe different targets at the same time. Therefore, we propose a graph-based method for near-online tracking in this article. We introduce a detection multiplexing method for tracking in the monocular image and propose a multiplex labeling graph (MLG) model. Each node in MLG has the ability to represent multiple targets. In addition, we improve the shortage of graph-based trackers in using temporal features. We construct long short-term memory networks to model motion and appearance features for MLG optimization. On the public multiobject tracking challenge benchmark, our near-online method gains satisfactory efficiency and achieves state-of-the-art results without additional private detection as well.

Data association is one of the key research in tracking-by-detection framework. Due to frequent interactions among targets, there are various relationships among trajectories in crowded scenes which leads to problems in data association, such as association ambiguity, association omission, etc. To handle these problems, we propose hypothesis-testing based tracking (HTBT) framework to build potential associations between target by constructing and testing hypotheses. In addition, a spatio-temporal interaction graph (STIG) model is introduced to describe the basic interaction patterns of trajectories and test the potential hypotheses. Based on network flow optimization, we formulate offline tracking as a MAP problem. Experimental results show that our tracking framework improves the robustness of tracklet association when detection failure occurs during tracking. On the public MOT16, MOT17 and MOT20 benchmark, our method achieves competitive results compared with other state-of-the-art methods.

Recently, most multiple object tracking (MOT) algorithms adopt the idea of tracking-by-detection. Relevant research shows that the performance of the detector obviously affects the tracker, while the improvement of detector is gradually slowing down in recent years. Therefore, trackers using tracklet (short trajectory) are proposed to generate more complete trajectories. Although there are various tracklet generation algorithms, the fragmentation problem still often occurs in crowded scenes. In this paper, we introduce an iterative clustering method that generates more tracklets while maintaining high confidence. Our method shows robust performance on avoiding internal identity switch. Then we propose a deep association method for tracklet association. In terms of motion and appearance, we construct motion evaluation network (MEN) and appearance evaluation network (AEN) to learn long-term features of tracklets for association. In order to explore more robust features of tracklets, a tracklet-based training mechanism is also introduced. Tracklet groups are used as the input of the networks instead of discrete detections. Experimental results show that our training method enhances the performance of the networks. In addition, our tracking framework generates more complete trajectories while maintaining the unique identity of each target as the same time. On the latest MOT 2017 benchmark, we achieve state-of-the-art results.

Recent years have witnessed the successful marriage of finance innovations and AI techniques in various finance applications including quantitative trading (QT). Despite great research efforts devoted to leveraging deep learning (DL) methods for building better QT strategies, existing studies still face serious challenges especially from the side of finance, such as the balance of risk and return, the resistance to extreme loss, and the interpretability of strategies, which limit the application of DL-based strategies in real-life financial markets. In this work, we propose AlphaStock, a novel reinforcement learning (RL) based investment strategy enhanced by interpretable deep attention networks, to address the above challenges. Our main contributions are summarized as follows: i) We integrate deep attention networks with a Sharpe ratio-oriented reinforcement learning framework to achieve a risk-return balanced investment strategy; ii) We suggest modeling interrelationships among assets to avoid selection bias and develop a cross-asset attention mechanism; iii) To our best knowledge, this work is among the first to offer an interpretable investment strategy using deep reinforcement learning models. The experiments on long-periodic U.S. and Chinese markets demonstrate the effectiveness and robustness of AlphaStock over diverse market states. It turns out that AlphaStock tends to select the stocks as winners with high long-term growth, low volatility, high intrinsic value, and being undervalued recently.

This paper proposes a novel iterative maximum weighted independent set (MWIS) algorithm for multiple hypothesis tracking (MHT) in a tracking-by-detection framework. MHT converts the tracking problem into a series of MWIS problems across the tracking time. Previous works solve these NP-hard MWIS problems independently without the use of any prior information from each frame, and they ignore the relevance between adjacent frames. In this paper, we iteratively solve the MWIS problems by using the MWIS solution from the previous frame rather than solving the problem from scratch each time. First, we define five hypothesis categories and a hypothesis transfer model, which explicitly describes the hypothesis relationship between adjacent frames. We also propose a polynomial-time approximation algorithm for the MWIS problem in MHT. In addition to that, we present a confident short tracklet generation method and incorporate tracklet-level association into MHT, which further improves the computational efficiency. Our experiments on both MOT16 and MOT17 benchmarks show that our tracker outperforms all the previously published tracking algorithms on both MOT16 and MOT17 benchmarks. Finally, we demonstrate that the polynomial-time approximate tracker reaches nearly the same tracking performance.

Tracking-by-detection is one of the most popular approaches to tracking multiple objects in which the detector plays an important role. Sometimes, detector failures caused by occlusions or various poses are unavoidable and lead to tracking failure. To cope with this problem, we construct a heterogeneous association graph that fuses high-level detections and low-level image evidence for target association. Compared with other methods using low-level information, our proposed heterogeneous association fusion (HAF) tracker is less sensitive to particular parameters and is easier to extend and implement. We use the fused association graph to build track trees for HAF and solve them by the multiple hypotheses tracking framework, which has been proven to be competitive by introducing efficient pruning strategies. In addition, the novel idea of adaptive weights is proposed to analyze the contribution between motion and appearance. We also evaluated our results on the MOT challenge benchmarks and achieved state-of-the-art results on the MOT Challenge 2017.

Multiple object tracking is a research hotspot in the artificial intelligent field, and tracking-by-detection is one of the most popular paradigms in recent years. Among these methods, the network flow based tracker is quite popular due to its computational efficiency and optimality, but it still has one main drawback: Object detection is the processing unit, so high-order information is hard to be taken into consideration directly, and it is usually processed hierarchically, which leads to error propagation. To address this problem, we propose community evolution model for network flow based trackers. We introduce a novel community, which maintains detections and tracklets dynamically. The community allows modeling the connectivities of detections and tracklets jointly, which adaptively incorporates all-level correlations among detections and tracklets, including low-level optical flow, mid-level color histogram, and high-level ranking model. We demonstrate the validity of our method on PETS09 dataset and the MOT17 benchmark, and our method achieves competitive results. Our results on the MOT17 benchmark are available on the website.

We present a structured light measurement system to collect high accuracy surface information of the measured object with a good real-time performance. Utilizing phase-shifting method in conjunction with a matching method proposed in this paper which can significantly reduce the noisy points, we can achieve high accuracy and noiseless point cloud in a complex industrial environment. Due to the use of the heterogeneous parallel computation model, the parallelism of the algorithm is developed in a deep way. The OpenMP+CUDA hybrid computing model is then used in the system to get a better real-time performance.