Abstract Multimodal Human Action Recognition (MHAR) is an important research topic in computer vision and event recognition fields. In this work, we address the problem of MHAR by developing a novel audio-image and video fusion-based deep learning framework that we call Multimodal Audio-Image and Video Action Recognizer (MAiVAR). We extract temporal information using image representations of audio signals and spatial information from video modality with the help of Convolutional Neutral Networks (CNN)-based feature extractors and fuse these features to recognize respective action classes. We apply a high-level weights assignment algorithm for improving audio-visual interaction and convergence. This proposed fusion-based framework utilizes the influence of audio and video feature maps and uses them to classify an action. Compared with state-of-the-art audio-visual MHAR techniques, the proposed approach features a simpler yet more accurate and more generalizable architecture, one that performs better with different audio-image representations. The system achieves an accuracy 87.9% and 79.0% on UCF51 and Kinetics Sounds datasets, respectively. All code and models for this paper will be available at https://tinyurl.com/4ps2ux6n .

Due to its widespread applications, human action recognition is one of the most widely studied research problems in Computer Vision. Recent studies have shown that addressing it using multimodal data leads to superior performance as compared to relying on a single data modality. During the adoption of deep learning for visual modelling in the past decade, action recognition approaches have mainly relied on Convolutional Neural Networks (CNNs). However, the recent rise of Transformers in visual modelling is now also causing a paradigm shift for the action recognition task. This survey captures this transition while focusing on Multimodal Human Action Recognition (MHAR). Unique to the induction of multimodal computational models is the process of ‘fusing’ the features of the individual data modalities. Hence, we specifically focus on the fusion design aspects of the MHAR approaches. We analyze the classic and emerging techniques in this regard, while also highlighting the popular trends in the adaption of CNN and Transformer building blocks for the overall problem. In particular, we emphasize on recent design choices that have led to more efficient MHAR models. Unlike existing reviews, which discuss Human Action Recognition from a broad perspective, this survey is specifically aimed at pushing the boundaries of MHAR research by identifying promising architectural and fusion design choices to train practicable models. We also provide an outlook of the multimodal datasets from their scale and evaluation viewpoint. Finally, building on the reviewed literature, we discuss the challenges and future avenues for MHAR.

Currently, action recognition is predominately performed on video data as processed by CNNs. We investigate if the representation process of CNN s can also be leveraged for multimodal action recognition by incorporating image-based audio representations of actions in a task. To this end, we propose Multimodal Audio-Image and Video Action Recognizer (MAiVAR), a CNN-based audio-image to video fusion model that accounts for video and audio modalities to achieve superior action recognition performance. MAiVAR extracts meaningful image representations of audio and fuses it with video representation to achieve better performance as compared to both modalities individually on a large-scale action recognition dataset.

The complex task of vision based fruit and vegetables classification at a supermarket self-checkout poses significant challenges. These challenges include the highly variable physical features of fruit and vegetables i.e. colour, texture shape and size which are dependent upon ripeness and storage conditions in a supermarket as well as general product variation. Supermarket environments are also significantly variable with respect to lighting conditions. Attempting to build an exhaustive dataset to capture all these variations, for example a dataset of a fruit consisting of all possible colour variations, is nearly impossible. Moreover, some fruit and vegetable classes have significant similar physical features e.g. the colour and texture of cabbage and lettuce. Current state-of-the-art classification techniques such as those based on Deep Convolutional Neural Networks (DCNNs) are highly prone to errors resulting from the inter-class similarities and intra-class variations of fruit and vegetable images. The deep features of highly variable classes can invade the features of neighbouring similar classes in a learned feature space of the DCNN, resulting in confused classification hyper-planes. To overcome these limitations of current classification techniques we have proposed a class distribution-aware adaptive margins approach with cluster embedding for classification of fruit and vegetables. We have tested the proposed technique for cluster-based feature embedding and classification effectiveness. It is observed that introduction of adaptive classification margins proportional to the class distribution can achieve significant improvements in clustering and classification effectiveness. The proposed technique is tested for both clustering and classification, and promising results have been obtained.

Image classification of fruit and vegetables at supermarket self-checkouts is a complex problem. Significant variations in the size, shape and colour of objects are involved, along with potentially large variations in the environmental conditions, must be accommodated to implement such a robust and effective system. Convolution Neural Networks (CNNs) have shown promising results for object classifications. However, the scarcity of training datasets due to the diversity of varieties and applications of fruit and vegetable classification is a significant limitation to the CNN implementation for this task. To overcome this, we propose the use of transfer learning and ensemble technique. Specifically, a transfer learning based weighted average weight optimisation ensemble technique is applied to the weights of GoogleNet and MobileNet by transfer learning the pre-trained CNNs using a custom dataset. Two hyperparameter optimisation techniques have been applied in a sequential way to identify the effective weights progressively. The optimised weights are used as an input to a normalised exponential softmax layer to estimate the final probability distribution for classification. A comparative evaluation among standalone GoogleNet, MobileNet and different levels of ensemble has been presented, which supports the adoption of this technique as a solution in a real-world supermarket environment.

Recent advancements in Deep Learning have accelerated the capabilities of robotic systems in terms of visual perception, object manipulation, automated navigation, and human-robot collaboration. This paper proposes the use of a transfer learning technique with deep convolutional neural networks to learn how to visually identify the grasping configurations for a parallel plate gripper that will be used to grasp various household objects. The Red-Green-Blue-Depth (RGB-D) data from the Cornell Grasp Dataset is used to train the network model using an end-to-end learning method. With this method, we achieve a grasping configuration prediction accuracy of 93.91%.

In this paper, a panoramic image stitching algorithm is presented. The aim of image stitching is to detect several images of the same scene and merge them to create a larger image. This is achieved by first detecting the overlapping area of the acquired images, and then aligning and blending the seams of the images automatically to create a seamless panoramic image. The experimental testing into the size of the overlapping area, the use of different focal lengths, the use of tilting and panning, and the number of input images were carried out. The results demonstrated that our algorithm is able to stitch correctly with a minimum of 5% overlapping area, of images acquired by a wide angle lens, by a horizontal and vertical rotation, and also of a multitude of input images acquired at unfixed axis.

Colour 2D barcodes, such as the MMCC barcode, have been developed recently to improve the data capacity of monochrome barcodes. However, the use of colour imposes greater challenges in decoding the symbols correctly as different lighting conditions on the barcode vary the values of the colours significantly. Hence, it is desirable to have an adaptive classification of the data cells so as to adapt to various lighting conditions. In this paper, we propose a classification method that is able to adapt to different lighting conditions during the classification of the MMCC data cells in a mobile environment. To highlight the effectiveness of this method, it is compared against the classification of the data cells performed in two different colour space representations, namely RGB and YCbCr.

Two dimensional (2D) barcodes are becoming a pervasive interface for mobile devices, such as camera phones. Often, only monochrome 2D-barcodes are used due to their robustness in an uncontrolled operating environment of camera phones. Most camera phones capture and store such 2D-barcode images in the baseline JPEG format. As a lossy compression technique, JPEG does introduce a fair amount of error in the decoding of captured 2D-barcode images. In this paper, we introduce an improved JPEG compression scheme for such barcode images. By altering the JPEG compression parameters based on the DCT coefficient distribution of such barcode images, the improved compression scheme produces JPEG images with higher PSNR value as compared to the baseline implementation. We have also applied our improved scheme to a real 2D-barcode system - the QR Code and analyzed its performance against the baseline JPEG scheme.

With a camera mobile phone, which has become a "must-have" device, 2D-barcode works as an interface to bridge the physical and digital world. As the notion of ubiquitous computing has permeated, developing a new 2D-barcode and its applications has been a growing trend worldwide. A 2D-barcode symbol consists of two broad areas: data area and guide area. The components of the latter is collectively called "finder pattern" and used in locating the 2D-barcode symbol. The failure of finding the target symbol prevents a barcode reader from successfully decoding the barcode. Hence, designing a functional finder pattern is one of the key for improving the robustness of barcode reading, and thus, the entire 2D-barcode system. We have designed a novel finder pattern integrated with a color 2D-barcode for camera mobile phone applications. Through the development and evaluation of the finder pattern for effective color 2D-barcode detection, this paper discusses keys to improve the functionality and reliability of finder patterns, which should be kept in mind when designing a finder pattern for any 2D-barcode symbol.

In recent years, a trend has emerged in the use of colour to increase the data capacity of the 2D barcode. However the decoding of such colour barcode can be challenging in a mobile environment due to blurring effect that is commonly found in images captured by camera mobile phone. Blurring affects the synchronisation between the cells. It also causes the colour value of the cell to be wrongly interpreted. Hence, this paper proposes the use of border to improve the synchronisation and decoding. Three different types of border and the choice of its width will be investigated.

Today's digital cameras and camera phones allow users to capture bar codes, which are used to uniquely identify consumer product. In this paper a fast algorithm is proposed that locates a 1-D bar code in the DCT-domain of a bar code image taken by a digital camera. The algorithm uses the DCT-transform properties to distinguish bar code from other texture, morphological operations to smooth the detected bar code area and the features of the extracted area lo detect position and orientation of a bar code in the image

This paper presents a study of three important issues of the color pixel classification approach to skin segmentation: color representation, color quantization, and classification algorithm. Our analysis of several representative color spaces using the Bayesian classifier with the histogram technique shows that skin segmentation based on color pixel classification is largely unaffected by the choice of the color space. However, segmentation performance degrades when only chrominance channels are used in classification. Furthermore, we find that color quantization can be as low as 64 bins per channel, although higher histogram sizes give better segmentation performance. The Bayesian classifier with the histogram technique and the multilayer perceptron classifier are found to perform better compared to other tested classifiers, including three piecewise linear classifiers, three unimodal Gaussian classifiers, and a Gaussian mixture classifier.

Customer relationship management (CRM) initiatives have gained much attention in recent years. With the aid of data mining technology, businesses can formulate specific strategies for different customer bases more precisely. Additionally, personalisation is another important issue in CRM - especially when a company has a huge product range. This paper presents a case model and investigates the use of computational intelligent techniques for CRM. These techniques allow the complex functions of relating customer behaviour to internal business processes to be learned more easily and the industry expertise and experience from business managers to be integrated into the modelling framework directly. Hence, they can be used in the CRM framework to enhance the creation of targeted strategies for specific customer bases.

This paper presents a classifier of face and nonface patterns that is based on the naive Bayes model. Using this classifier as a tool. We analyze the effects on classification performance of preprocessing, feature extraction and classifier combination techniques. Our analysis shows that image normalization techniques that reduce the effects of different lighting conditions improve face-nonface classification significantly. In addition, techniques such as background masking and combining classifiers that use different feature vectors are shown to enhance classification performance. Over a test set of 12,000 patterns, the combined classifier using four feature vectors has correct detection rates (CDRs) of 96.2% and 99.2% at false detection rates (FDRs) of 1% and 5%, respectively.

A new skin segmentation technique for color images is proposed. The proposed technique uses a human skin color model that is based on the Bayesian decision theory and developed using a large training set of skin colors and nonskin colors. The proposed technique is novel and unique in that texture characteristics of the human skin are used to select appropriate skin color thresholds for skin segmentation. Two homogeneity measures for skin regions that take into account both global and local image features are also proposed. Experimental results showed that the proposed technique can achieve good skin segmentation performance (false detection rate of 4.5% and false rejection rate of 4.0%).

A new skin segmentation technique for color images is proposed. The proposed technique uses a human skin color model that is based on the Bayesian decision theory and developed using a large training set of skin colors and nonskin colors. The proposed technique is novel and unique in that texture characteristics of the human skin are used to select appropriate skin color thresholds for skin segmentation. Two homogeneity measures for skin regions that take into account both global and local image features are also proposed. Experimental results showed that the proposed technique can achieve good skin segmentation performance (false detection rate of 4.5% and false rejection rate of 4.0%).

This paper presents a new human skin color model in YCbCr color space and its application to human face detection. Skin colors are modeled by a set of three Gaussian clusters, each of which is characterized by a centroid and a covariance matrix. The centroids and covariance matrices are estimated from large set of training samples after a k-means clustering process. Pixels in a color input image can be classified into skin or non-skin based on the Mahalanobis distances to the three clusters. Efficient post-processing techniques namely noise removal, shape criteria, elliptic curve fitting and face/non-face classification are proposed in order to further refine skin segmentation results for the purpose of face detection.

Two bit allocation strategies, namely, maximum bit transfer (MET) and joint bit assignment (JBA) are proposed for region-of-interest coding. The MBT strategy uses a pair of quantizers to facilitate maximum bit transfer from the background to foreground image region. It assigns the highest quantization parameter to the background quantizer, and then determines the finest value of foreground quantizer that can be used without increasing the bit rate. In this approach, the background is always encoded with the coarsest quantization level, but this is not always desirable. Therefore, the JBA strategy can be used instead. It has two modes: user defined and automatic. In the user defined mode, the user can adjust bit consumption using a scale that ranges from "not coding foreground" to "coding foreground only". In the automatic mode, bit allocation is based on the characteristics of the image regions, which include size, motion and priority. The coding results showed that improved image quality was achieved at the same bit rate.

This paper addresses our proposed method to automatically segment out a person's face from a given image that consists of a head-and-shoulders view of the person and a complex background scene. The method involves a fast, reliable, and effective algorithm that exploits the spatial distribution characteristics of human skin color. A universal skin-color map is derived and used on the chrominance component of the input image to detect pixels with skin-color appearance. Then, based on the spatial distribution of the detected skin-color pixels and their corresponding luminance values, the algorithm employs a set of novel regularization processes to reinforce regions of skin-color pixels that are more likely to belong to the facial regions and eliminate those that are not. The performance of the face-segmentation algorithm is illustrated by some simulation results carried out on various head-and-shoulders test images. The use of face segmentation for video coding in applications such as videotelephony is then presented. We explain how the face-segmentation results can be used to improve the perceptual quality of a videophone sequence encoded by the H.261-compliant coder.

Recently, ITU-T established a draft standard, H.263 to he used as a possible standard for very low bit rate video coding at less than 64 kb/s. It is based on the ITU-T H.261 standard for videoconferencing. This paper provides an analysis of the performance of H.263 coder with the aim of establishing its operational parameters and limitations when operating at very low bit rates. Its performance is then optimized by selecting the desired set of operational parameters. The H.263 coder suffers from "blocking" effects when operating at very low bit rates. To improve the subjective quality of the coded images, the discrete cosine transform (DCT) coefficients are weighted using a visual quantization matrix before quantization. Simulation results showed that the proposed method increases the peak signal-to-noise ratio (PSNR) values at all bit rates tested, but more importantly, the perceived visual quality is improved significantly by reducing the annoying "blocking" effects.