With the ever increasing need for better cybersecurity, and due to the continuous growth of network traffic bandwidths, there is a continuous pursuit of faster and smarter network intrusion detection systems. Neural network-based solutions on FPGAs are very effective in detecting different types of attacks, but have problems with analyzing network traffic online at line speed. One important bottleneck that limits the throughput in raw traffic-based existing systems, is the input shape of the features that are extracted from the raw data. In this work, we propose new methods for extracting and representing features based on raw network traffic in online network intrusion detection systems. We show that feature dimensionality has a significant influence on the classification accuracy and the throughput. Our experiments are based on FPGA-based neural networks accelerated through FINN. We compare three newly proposed input shapes to the traditional 2D-based approach, and we show that two of the presented techniques greatly surpass the state-of-the-art with regards to accuracy and throughput. Our best architecture reaches a maximum bandwidth of 23.09 Gbps, while maintaining over 99% accuracy on both the UNSW-NB15 and CICIDS2017 datasets.

Motivated by an increasing interest from startups in embedded Artificial Intelligence (AI) and by their limited expertise, the AITIA Project targets the development of embedded AI techniques for industrial applications. This extended abstract presents the motivation and the solutions being developed towards four use cases: smart sensors, network intrusion detection, driver-assistance systems, and Industry 4.0.

In this paper we study the optimal distribution of CNN computations between an edge device and the cloud for a complex IoT application. We propose a pipeline in which we perform experiments with a Jetson Nano and a Raspberry Pi 3B+ as the edge device, and a T2.micro instance from Amazon EC2 as a cloud instance. To answer this generic question, we performed exhaustive experiments on a typical use case, a mobile camerabased street litter detection and mapping application based on a MobilenetV2 model. For our research, we split the computations of the CNN model and divided them over the edge and cloud instances using model partitioning, also including the edge-only and cloud-only configurations. We studied the influence of the specifications of the instances, the input size of the model, the partitioning location of the model and the available network bandwidth on the optimal split position. Depending on the choice of gaining either an economic or performance advantage, we can conclude that a balance between the choice of instances and the calculation mechanism used should be made.

Abstract Reality TV shows have gained popularity, motivating many production houses to bring new variants for us to watch. Compared to traditional TV shows, reality TV shows have spontaneous unscripted footage. Computer vision techniques could partially replace the manual labour needed to record and process this spontaneity. However, automated real-world video recording and editing is a challenging topic. In this paper, we propose a system that utilises state-of-the-art video and audio processing algorithms to, on the one hand, automatically steer cameras, replacing camera operators and on the other hand, detect all audiovisual action cues in the recorded video, to ease the job of the film editor. This publication has hence two main contributions. The first, automating the steering of multiple Pan-Tilt-Zoom PTZ cameras to take aesthetically pleasing medium shots of all the people present. These shots need to comply with the cinematographic rules and are based on the poses acquired by a pose detector. Secondly, when a huge amount of audio-visual data has been collected, it becomes labour intensive for a human editor retrieve the relevant fragments. As a second contribution, we combine state-of-the-art audio and video processing techniques for sound activity detection, action recognition, face recognition, and pose detection to decrease the required manual labour during and after recording. These techniques used during post-processing produce meta-data allowing for footage filtering, decreasing the search space. We extended our system further by producing timelines uniting generated meta-data, allowing the editor to have a quick overview. We evaluated our system on three in-the-wild reality TV recording sessions of 24 hours (× 8 cameras) each taken in real households.

In this paper, we present a room occupancy sensing solution with unique properties: (i) It is based on an omnidirectional vision camera, capturing rich scene info over a wide angle, enabling to count the number of people in a room and even their position. (ii) Although it uses a camera-input, no privacy issues arise because its extremely low image resolution, rendering people unrecognisable. (iii) The neural network inference is running entirely on a low-cost processing platform embedded in the sensor, reducing the privacy risk even further. (iv) Limited manual data annotation is needed, because of the self-training scheme we propose. Such a smart room occupancy rate sensor can be used in e.g. meeting rooms and flex-desks. Indeed, by encouraging flex-desking, the required office space can be reduced significantly. In some cases, however, a flex-desk that has been reserved remains unoccupied without an update in the reservation system. A similar problem occurs with meeting rooms, which are often under-occupied. By optimising the occupancy rate a huge reduction in costs can be achieved. Therefore, in this paper, we develop such system which determines the number of people present in office flex-desks and meeting rooms. Using an omnidirectional camera mounted in the ceiling, combined with a person detector, the company can intelligently update the reservation system based on the measured occupancy. Next to the optimisation and embedded implementation of such a self-training omnidirectional people detection algorithm, in this work we propose a novel approach that combines spatial and temporal image data, improving performance of our system on extreme low-resolution images.

Re-identification neural networks are widely used in numerous applications such as crowd control, crime investigations, safety systems and even in most smartphones to unlock the phone with a picture of the owner. These techniques are mostly used to re-identify faces or persons but in this paper we investigate the possibility to adapt these to also re-identify similar looking objects such as diamonds. Since polished diamonds are very similar to the naked eye, it is difficult to distinguish one diamond from another. We have indications that diamonds are sometimes switched by trained switchers with fake or less expensive stones, while they pretend to inspect the stone. A solution to this is diamond fingerprinting. We therefore propose a technique to generate a unique ID for each stone, which allows to re-identify the diamond solely based on an image of the gem. Since each diamond is assigned a unique ID it is even possible to keep track of the diamonds over time. This allows the seller to verify his stones before and after trading while switchers don't stand a chance. For this task we trained and adapted a classification network optimized for both speed and accuracy.

In this paper we investigate the benefit of using depth information on top of normal RGB for camera-based pedestrian detection. Indeed, depth sensing is easily acquired using depth cameras such as a Kinect or stereo setups. We investigate the best way to perform this sensor fusion with a special focus on lightweight single-pass CNN architectures, enabling real-time processing on limited hardware. We implement different network architectures, each fusing depth at different layers of our network. Our experiments show that midway fusion performs the best, outperforming a regular RGB detector substantially in accuracy. Moreover, we prove that our fusion network is better at detecting individuals in a crowd, by demonstrating that it has both a better localization of pedestrians and is better at handling occluded persons. The resulting network is computationally efficient and achieves real-time performance on both desktop and embedded GPUs.

A traditional re-identification pipeline consists of a detection and re-identification step, i.e. a person detector is run on an input image to get a cutout which is then sent to a separate re-identification system. In this work we combine detection and re-identification into one single pass neural network. We propose an architecture that can do re-identification simultaneously with detection and classification. The effect of our modification has only a negligible impact on detection accuracy, and adds the calculation of re-identification vectors at virtually no cost.The resulting re-identification vector is strong enough to be used in speed sensitive applications which can benefit from an additional re-identification vector in addition to detection. We demonstrate this by using it as detection and re-identification input for a real-time person tracker. Moreover, unlike traditional detection + re-id pipelines our single-pass network's computational cost is not dependent on the number of people in the image.

In this paper we present a framework that is able to reliably and completely autonomously detect abnormal behavior in surveillance images. As input, we rely solely on a long-wave infrared (LWIR) image sensor. Our abnormal behavior detection pipeline consists of two consecutive stages. In a first stage, we perform efficient and fast pedestrian detection and tracking. In a second step, the detected paths are fed into a semi-supervised classifier that detects abnormal behavior. As test-case we recorded a unique real-life LWIR train station dataset — which will be made publicly available — containing natural occurrences of both normal and abnormal behavior. Our experiments indicate that our proposed framework achieves excellent accuracy results at real-time processing speeds.

Hiring a professional camera crew to cover an event such as a lecture, sports game or musical performance may be prohibitively expensive. The CAMETRON project aims at drastically reducing this cost by developing an (almost) fully automated system that can produce video recordings of such events with a quality similar to that of a professional crew. This system consists of different components, including intelligent Pan-Tilt-Zoom cameras and UAVs that act as “virtual camera men”. To combine the footage of these different cameras into a single coherent and pleasant-to-watch video, a “virtual editor” is needed. This paper describes the development of such a component. We adopt a declarative approach, in which we build a model of the decision process that a human editor might follow to edit a video. To achieve a montage that obeys various cinematographic rules while at the same time retaining a natural, non-mechanical feel, we construct this model in a Probabilistic Logic Programming language. We demonstrate that the resulting system can be run in real-time and that it delivers montages that are almost indistinguishable from those made by a professional editor.

In recent years, the accuracy of pedestrian detectors significantly improved. Currently, state-of-the-art pedestrian detectors achieve high accuracy results on challenging datasets. As opposed to refining a single detector, in this paper we propose a different approach to further increase the detection accuracy: combining multiple pedestrian detectors. The most straight-forward way to combine pedestrian detectors would be a naive AND or OR combination. Here, we present a novel generic combination framework in which we exploit specific information from each pedestrian detector to determine the optimal combination parameters. Our main motivation for this approach is based on the fact that several pedestrian detection approaches are based on very different techniques (e.g. a different feature pool), and thus an efficient combination should yield higher accuracy results. Indeed, such a combination is far more powerful, and our experiments indicate that specific (that is, cleverly chosen) combinations outperform existing state-of-the-art pedestrian detection results.

In this paper we present a multi-pedestrian detection and tracking framework targeting a specific application: detecting vulnerable road users in a truck's blind spot zone. Research indicates that existing non-vision based safety solutions are not able to handle this problem completely. Therefore we aim to develop an active safety system which warns the truck driver if pedestrians are present in the truck's blind spot zone. Our system solely uses the vision input from the truck's blind spot camera to detect pedestrians. This is not a trivial task, since the application inherently requires real-time operation while at the same time attaining very high accuracy. Furthermore we need to cope with the large lens distortion and the extreme viewpoints introduced by the blind spot camera. To achieve this, we propose a fast and efficient pedestrian detection and tracking framework based on our novel perspective warping window approach. To evaluate our algorithm we recorded several realistically simulated blind spot scenarios with a genuine blind spot camera mounted on a real truck. We show that our algorithm achieves excellent accuracy results at real-time performance, using a single core CPU implementation only.

Combining new, low-cost thermal infrared and time-of-flight range sensors provides new opportunities. In this position paper we explore the possibilities of combining these sensors and using their fused data for person detection. The proposed calibration approach for this sensor combination differs from the traditional stereo camera calibration in two fundamental ways. A first distinction is that the spectral sensitivity of the two sensors differs significantly. In fact, there is no sensitivity range overlap at all. A second distinction is that their resolution is typically very low, which requires special attention. We assume a situation in which the sensors' relative position is known, but their orientation is unknown. In addition, some of the typical measurement errors are discussed, and methods to compensate for them are proposed. We discuss how the fused data could allow increased accuracy and robustness without the need for complex algorithms requiring large amounts of computational power and training data.

In this paper we investigate the integration of object detection algorithms with eye-tracking data. The emerging technology of lightweight mobile eye-trackers enables realistic in-the-wild user experience experiments. Unfortunately, mobile eye-trackers generate a large amount of video data, which up to now requires manual analysis. This time-consuming and repetitive task renders processing large datasets economically infeasible. Our main contribution is the use of object detection algorithms to perform this analysis task automatically. We compare several object detection algorithms with regard to both speed and accuracy. To prove their functionality, we have recorded an eye-tracker shopping experiment and processed the data using object detection techniques.

In this paper we present a technique to linearly recover 2D structure and motion in man made environments from three uncalibrated omnidirectional views. We use vertical lines from the scene which are projected as radial lines in the images and are automatically matched. The algorithm is based on a 1D radial trifocal tensor which encodes the relations of the three views and the projected lines. We include experiments with real images, which demonstrate the good performance of the method and its application to robotic tasks, such as robot localization based in a database of reference images or to obtain the initial values of robot and landmarks localization in SLAM algorithms

Vision sensors are attractive for autonomous robots because they are a rich source of environment information. The main challenge in using images for mobile robots is managing this wealth of information. A relatively recent approach is the use of fast wide baseline local features, which we developed and used in the novel approach to sparse visual path following described in this paper. These local features have the great advantage that they can be recognized even if the viewpoint differs significantly. This opens the door to a memory efficient description of a path by descriptors of sparse images. We propose a method for re-execution of these paths by a series of visual homing operations which yield a navigation method with unique properties: it is accurate, robust, fast, and without odometry error build-up.