Abstract Plant species identification is a fundamental process in botany and agriculture sector. In recent years, deep neural networks have become the primary approach for automating this task, providing valuable insights into biodiversity, ecological systems, and agricultural practices. Along with more discoveries in plant species, training a deep neural network becomes very challenging as the cost required to collect and annotate plant samples is expensive and impractical. Despite the lack of labelled plant samples, recent studies have explored the potential of leveraging publicly available and systematically annotated plant specimens in herbaria coupled with field images for plant species identification through cross-domain adaptation techniques. However, the accuracy of these methods remains unsatisfactory, motivating the exploration of alternative approaches. In this paper, we evaluated the feasibility of employing a pre-trained transformer-based self-distillation model (DINOv2) for cross-domain plant species identification tasks. We trained our model with the PlantCLEF2020 dataset comprised of approximately 320 k herbarium and field images representing 997 plant species. Our approach leverages the advanced feature extraction capabilities of DINOv2, which enhances cross-domain adaptation by effectively bridging the gap between herbarium and field images, achieving a 17.7% improvement over the best model proposed in previous work, that employs ensembles of Siamese network architectures with triplet loss (HFTL-ENS and OSM-ENS).

Abstract The transformation of cities into smarter and more efficient environments relies on proactive and timely detection and maintenance of city-wide infrastructure, including roadside infrastructure such as road signs and the cleaning of illegally dumped rubbish. Currently, these maintenance tasks rely predominantly on citizen reports or on-site checks by council staff. However, this approach has been shown to be time-consuming and highly costly, resulting in significant delays that negatively impact communities. This paper presents AIoT-CitySense, an AI and IoT-driven city-scale sensing framework, developed and piloted in collaboration with a local government in Australia. AIoT-CitySense has been designed to address the unique requirements of roadside infrastructure maintenance within the local government municipality. A tailored solution of AIoT-CitySense has been deployed on existing waste service trucks that cover a road network of approximately 100 kms in the municipality. Our analysis shows that proactive detection for roadside infrastructure maintenance using our solution reached an impressive 85%, surpassing the timeframes associated with manual reporting processes. AIoT-CitySense can potentially transform various domains, such as efficient detection of potholes and precise line marking for pedestrians. This paper exemplifies the power of leveraging city-wide data using AI and IoT technologies to drive tangible changes and improve the quality of city life.

Video-see-through (VST) augmented reality (AR) is widely used to present novel augmentative visual experiences by processing video frames for viewers. Among VST AR systems, assistive vision displays aim to compensate for low vision or blindness, presenting enhanced visual information to support activities of daily living for the vision impaired/deprived. Despite progress, current assistive displays suffer from a visual information bottleneck, limiting their functional outcomes compared to healthy vision. This motivates the exploration of methods to selectively enhance and augment salient visual information. Traditionally, vision processing pipelines for assistive displays rely on hand-crafted, single-modality filters, lacking adaptability to time-varying and environment-dependent needs. This article proposes the use of Deep Reinforcement Learning (DRL) and Self-attention (SA) networks as a means to learn vision processing pipelines for assistive displays. SA networks selectively attend to task-relevant features, offering a more parameter—and compute-efficient approach to RL-based task learning. We assess the feasibility of using SA networks in a simulation-trained model to generate relevant representations of real-world states for navigation with prosthetic vision displays. We explore two prosthetic vision applications, vision-to-auditory encoding, and retinal prostheses, using simulated phosphene visualisations. This article introduces SA-px, a general-purpose vision processing pipeline using self-attention networks, and SA-phos, a display-specific formulation targeting low-resolution assistive displays. We present novel scene visualisations derived from SA image patches importance rankings to support mobility with prosthetic vision devices. To the best of our knowledge, this is the first application of self-attention networks to the task of learning vision processing pipelines for prosthetic vision or assistive displays.

Timely detection of roadside assets that require maintenance is essential for improving citizen satisfaction. Currently, the process of identifying such maintenance issues is typically performed manually, which is time consuming, expensive, and slow to respond. In this paper, we present Mobile IoT-RoadBot, a mobile 5G-based Internet of Things (IoT) solution, powered by Artificial Intelligence (AI) techniques to enable opportunistic real-time identification and detection of maintenance issues with roadside assets. The Mobile IoT-RoadBot solution has been deployed on 11 bin service (waste collection) trucks in the western suburbs of Melbourne, Australia, performing real-time assessments of road-side assets as they service areas within the local government. We present the architecture of Mobile IoT-RoadBot and demonstrate its capability via an online 'points of maintenance' (PoMs) map.

There is increasing interest in using computer vision and machine learning to enhance human decision making with computer-mediated assistive vision systems. In particular, retinal implants are a rapidly advancing technology offering individuals suffering vision loss due to retinal dystrophies, an opportunity to restore partial vision. However, the visual representations achievable with current and near-term implants are severely limited in resolution and contrast, placing high importance on the selection of visual features to convey via the implant. Using vision processing algorithms on camera-captured input, functional outcomes can be enhanced with such devices. To this end, we propose a novel end-to-end vision processing pipeline for prosthetic vision that learns task-salient visual filters in simulation offline via deep reinforcement learning (DRL). Once learnt, these filters are deployable on a prosthetic vision device to process camera-captured images and produce task-guiding scene representations in real-time. We show how a set of learnt visual features enabling a virtual agent to optimally perform the task of navigation in a 3-D environment can be extracted and applied to enhance the same features in real world images. We evaluate and validate our proposed approach quantitatively and qualitatively using simulated prosthetic vision. To our knowledge, this is the first application of DRL to the derivation of scene representations for human-centric computer-mediated displays such as prosthetic vision devices.

Interest in 3D hand pose estimation is rapidly growing, offering the potential for real-time hand gesture recognition in a range of interactive VR/AR applications, and beyond. Most current 3D hand pose estimation models rely on dedicated depth-sensing cameras and/or specialised hardware support to handle both the high computation and memory requirements. However, such requirements hinder the practical application of such models on mobile devices or in other embedded computing contexts. To address this, we propose a lightweight model for hand and object pose estimation specifically targeting mobile applications. Using RGB images only, we show how our approach achieves real-time performance, comparable accuracy, and an 81% model size reduction compared with state-of-the-art methods, thereby supporting the feasibility of the model for deployment on mobile platforms.

Smart mobile devices coupled with the Internet of Things (IoT) and Artificial Intelligence (AI) have emerged as a key enabler of modern digital health applications. While cloud computing is now a well established paradigm for analysing IoT captured data in mobile health applications, on-board analysis of data using AI approaches such as Deep Learning (DL) is gaining significant momentum. This is driven primarily by advances in on-board resources enabling modern mobile devices to execute complex DL models, while also offering improved response time and accuracy for rapid decision-making, and enhanced user privacy. While the number of mobile digital health applications that use IoT and DL is increasing, progress is currently impeded by a lack of framework for profiling and evaluating the performance of DL models on mobile devices. To this end, we propose MobDL, a framework for profiling and evaluating DL models running on smart mobile devices. We present the architecture of this framework and devise a novel evaluation methodology for conducting quantitative comparisons of various DL models running on mobile devices. Three diverse digital health applications using heterogeneous data (e.g. image, time series) are introduced. We conduct extensive experimental evaluations using several DL models that have been developed using the data sets obtained for the three digital health applications to validate the effectiveness of the proposed MobDL framework.

Universally, scene text in natural images is an expressive means of communication. Texts available in an image could provide important information which can enhance the interpretation of the image, for instance texts on product packages and road signs. Recently, scene texts detection has been recognized as an important research field in computer vision. However, texts detection in natural scene images has been challenging due to the complicated scene background involving varying fonts, sizes, image resolutions etc. As to date, different methods of detecting texts in natural scene images have been proposed for horizontal texts, arbitrarily-oriented texts and curved texts, but there is a lack of detection of vertically-oriented texts in the natural scene images. Hence, this research proposed a framework for detecting the top-to-bottom, bottom-to-top and horizontally-stacked vertical texts in natural scene images. In this paper, Multi-directional Text Detector (MTD) is modelled and developed to locate the vertically-oriented texts in natural scene images, and the Vertical Scene Texts Dataset-700 (VSTD-700) is developed. The preliminary testing shows that the success rate of detection of MTD is 87% on ICDAR 2013, 73% on ICDAR 2015, 71% on MSRA-TD500 and 87% on VSTD-700 datasets. Hence, the results showed that MTD is able to detect vertically-oriented texts on top of the horizontally-oriented and arbitrarily-oriented texts in natural scene images simultaneously.

We report on physiotherapists' acceptance of a Socially Assistive Robot (SAR) as a therapeutic aid for paediatric rehabilitation. The SAR is undergoing in situ evaluation while being deployed as part of the clinical care of paediatric rehabilitation patients at the Royal Children's Hospital in Melbourne, Australia. The robot is equipped to lead rehabilitation sessions of up to 30 minutes under the guidance of a therapist, and without technician support or Wizard-of-Oz operation. In this paper we report on quantitative and qualitative data collected from 8 therapists participating in our study across 19 rehabilitation sessions. Data were collected after each therapy session. Our results show our system achieves a high degree of acceptance, particularly with respect to its perceived usefulness, and ease-of-use. Moreover, multiple sessions operating the SAR appears to strengthen positive perceptions of our system.

Socially assistive robots (SARs) offer great promise for improving outcomes in paediatric rehabilitation. However, the design of software and interactive capabilities for SARs must be carefully considered in the context of their intended clinical use. While previous work has explored specific roles and functionalities to support paediatric rehabilitation, few have considered the design of such capabilities in the context of ongoing clinical deployment. In this article, we present a two-phase in situ design process for SARs in health care, emphasising stakeholder engagement and on-site development. We explore this in the context of developing the humanoid social robot NAO as a socially assistive rehabilitation aid for children with cerebral palsy. We present and evaluate our design process, outcomes achieved, and preliminary results from ongoing clinical testing with 9 patients and 5 therapists over 14 sessions. We argue that our in situ design methodology has been central to the rapid and successful deployment of our system.

Recent work in salient object detection has considered the incorporation of depth cues from RGB-D images. In most cases, depth contrast is used as the main feature. However, areas of high contrast in background regions cause false positives for such methods, as the background frequently contains regions that are highly variable in depth. Here, we propose a novel RGB-D saliency feature. Local Background Enclosure (LBE) captures the spread of angular directions which are background with respect to the candidate region and the object that it is part of. We show that our feature improves over state-of-the-art RGB-D saliency approaches as well as RGB methods on the RGBD1000 and NJUDS2000 datasets.

Augmentations to enhance perception in prosthetic vision (also known as bionic eyes) have the potential to improve functional outcomes significantly for implantees. In current (and near-term) im-plantable electrode arrays resolution and dynamic range are highly constrained in comparison to images from modern cameras that can be head mounted. In this paper, we propose a novel, generally applicable adaptive contrast augmentation framework for prosthetic vision that addresses the specific perceptual needs of low resolution and low dynamic range displays. The scheme accepts an externally defined pixel-wise weighting of importance describing features of the image to enhance in the output dynamic range. Our approach explicitly incorporates the logarithmic scaling of enhancement required in human visual perception to ensure perceivability of all contrast augmentations. It requires no pre-existing contrast, and thus extends previous work in local contrast enhancement to a formulation for general image augmentation. We demonstrate the generality of our augmentation scheme for scene structure and looming object enhancement using simulated prosthetic vision.

We present a new visual control input from optical flow divergence enabling the design of novel, unified control laws for docking and landing. While divergence-based time-to-contact estimation is well understood, the use of divergence in visual control currently assumes knowledge of surface orientation, and/or egomotion. There exists no directly observable visual cue capable of supporting approaches to surfaces of arbitrary orientation under general motion. Central to our measure is the use of the maximum flow field divergence on the view sphere (max-div). We prove kinematic properties governing the location of max-div, and show that max-div provides a temporal measure of proximity. From this, we contribute novel control laws for regulating both approach velocity and angle of approach toward planar surfaces of arbitrary orientation, without structure-from-motion recovery. The strategy is tested in simulation, over real image sequences and in closed-loop control of docking/landing maneuvers on a mobile platform.

We investigated the use of optical flow to predict the time and location of impact of an incoming object. By examining local patterns of optical flow, we make predictions on an object's trajectory with respect to a stationary observer, and its time-to-contact with the observer's (assumed planar) body. Such a cue may serve as inputs to motor responses for reactive threat avoidance, or object interception (such as catching a ball). The presented approach is based on the observed behaviour of neurons in the F4 region of the pre-motor cortex of primates and it is part of a larger project which aims at modelling multi-sensory neurons and their contribution to reaching behaviour on the iCub humanoid platform. We present preliminary experimental results of a computation model of F4's visual responses using real image sequences acquired from the robot, and demonstrate their application in a real-time threat detection/prediction system.

We present a robust strategy for docking a mobile robot in close proximity with an upright surface using optical flow field divergence and proportional feedback control. Unlike previous approaches, we achieve this without the need for explicit segmentation of features in the image, and using complete gradient-based optical flow estimation (i.e., no affine models) in the optical flow computation. A key contribution is the development of an algorithm to compute the flow field divergence, or time-to-contact, in a manner that is robust to small rotations of the robot during ego-motion. This is done by tracking thefocusofexpansionof the flow field and using this to compensate for ego rotation of the image. The control law used is a simple proportional feedback, using the unfiltered flow field divergence as an input, for a dynamic vehicle model. Closed-loop stability analysis of docking under the proposed feedback is provided. Performance of the flow field divergence algorithm is demonstrated using offboard natural image sequences, and the performance of the closed-loop system is experimentally demonstrated by control of a mobile robot approaching a wall.

We present a strategy for generating real-time relative depth maps of an environment from optical flow, under general motion. We achieve this using an insect-inspired hemispherical fish-eye sensor with 190 degree FOV, and a de-rotated optical flow field. The de-rotation algorithm applied is based on the theoretical work of Nelson and Aloimonos (1988). From this we obtain the translational component of motion, and construct full relative depth maps on the sphere. We examine the robustness of this strategy in both simulation and real-world experiments, for a variety of environmental scenarios. To our knowledge, this is the first demonstrated implementation of the Nelson and Aloimonos algorithm working in real-time, over real image sequences. In addition, we apply this algorithm to the real-time recovery of full relative depth maps. These preliminary results demonstrate the feasibility of this approach for closed-loop control of a robot.

We present a robust strategy for docking a mobile robot in close proximity with an upright surface using optical flow field divergence. Unlike previous approaches, we achieve this without the need for explicit segmentation of the surface in the image, and using complete optical estimation (i.e. no affine models) in the control loop. A simple proportional control law is used to regulate the vehicle's velocity, using only the raw, unfiltered flow divergence as input. Central to the robustness of our approach is the derivation of a time-to-contact estimator that accounts for small rotations of the robot during ego-motion. We present both analytical and experimental results showing that through tracking of the focus of expansion to a looming surface, we may compensate for such rotations, thereby significantly improving the robustness of the time-to-contact estimate. This is demonstrated using an off-board natural image sequence, and in closed-loop control of a mobile robot

We present a comparison of four optical flow methods and three spatio-temporal filters for mobile robot navigation in corridor-like environments. Previous comparisons of optical flow methods have evaluated performance only in terms of accuracy and/or efficiency, and typically in isolation. These comparisons are inadequate for addressing applicability to continuous, real-time operation as part of a robot control loop. We emphasise the need for comparisons that consider the context of a system, and that are confirmed by in-system results. To this end, we give results for on and off-board trials of two biologically inspired behaviours: corridor centring and visual odometry. Our results show the best in-system performances are achieved using Lucas and Kanade's gradient-based method in combination with a recursive temporal filter. Results for traditionally used Gaussian filters indicate that long latencies significantly impede performance for real-time tasks in the control loop.