Map representations learned by expert demonstrations have shown promising research value. However, the field of visual navigation still faces challenges due to the lack of real-world human-navigation datasets that can support efficient, supervised, representation learning of environments. We present a Landmark-Aware Visual Navigation (LAVN) dataset to allow for supervised learning of human-centric exploration policies and map building. We collect RGBD observation and human point-click pairs as a human annotator explores virtual and real-world environments with the goal of full coverage exploration of the space. The human annotators also provide distinct landmark examples along each trajectory, which we intuit will simplify the task of map or graph building and localization. These human point-clicks serve as direct supervision for waypoint prediction when learning to explore in environments. Our dataset covers a wide spectrum of scenes, including rooms in indoor environments, as well as walkways outdoors. We releaseour dataset with detailed documentation at https://huggingface.co/datasets/visnavdataset/lavn (DOI: l0.57967/hf/2386) and a plan for long-term preservation.

We present ‘Revelio’, a real-world screen-camera communication system leveraging temporal flicker fusion in the OKLAB color space. Using spatially-adaptive flickering and encoding information in pixel region shapes, Revelio achieves visually imperceptible data embedding while remaining robust against noise, asynchronicity, and distortions in screen-camera channels, ensuring reliable decoding by standard smartphone cameras. The decoder, driven by a two-stage neural network, uses a weighted differential accumulator for precise frame detection and symbol recognition. Initial experiments demonstrate Revelio’s effectiveness in interactive television, offering an unobtrusive method for meta-information transmission.

This paper introduces Text2Net, an innovative system designed to transform plain English descriptions into dynamic, interactive network simulations within the Emulator Virtual Engine–Next Generation (Eve-NG) environment. By integrating SOTA technologies from Natural Language Processing (NLP), Large Language Models (LLMs), and proposed adaptor software, Text2Net bridges the technical knowledge gap, enabling both technical and non-technical users to effortlessly create and interact with complex network topologies within a simulation environment. The system architecture combines an intuitive chat interface, GPT4 LLM to interpret user inputs, NLP key-value extraction, a simulation adaptor, and the EVE-NG engine. TextNet democratizes network emulation, empowering educators to efficiently construct simulations for interactive learning. It also benefits industrial prototyping and testing configurations.CCS CONCEPTS•Applied computing → Interactive learning environments; Computer-assisted instruction; IT architectures;•Networks → Network design principles; Programming interfaces; Topology analysis and generation; Logical / virtual topologies; Network manageability; Programmable networks; Network management; Network monitoring.

In this work, we propose a novel re-use of neuromorphic (event) cameras for joint sensing and communications. Event cameras work on the principle of capturing changes in the light intensities, essentially capturing events that lead to such changes. This makes them operable at low power and sample events at fast rates (equivalent to about 40K frames-per-second compared to RGB cameras). We propose a system design to leverage the time-sampling nature of events for optical wireless communication and the ability to sample a collective area of physical space for imaging. In particular, we propose to address the challenges to achieve passive optical wireless (backscatter) communication as well as computer vision functions such as object and path detection using a single neuromorphic camera device. We posit that such an integrated functioning through a single low-power device opens new avenues for visible/invisible light communication and visual scene processing.CCS CONCEPTS•Networks → Mobile networks;•Hardware → Signal processing systems;•Computing methodologies → Computer vision.

In view of recent developments in underwater wireless technology and the continuous demand for deep ocean exploration, this study investigates the underwater optical wireless communication system. Underwater wireless optical communication (UWOC) has several benefits over short-distance wireless connectivity due to its significantly higher bandwidth and data rate compared to acoustic communication. This paper presents an analysis of a non-line-of-sight (NLOS) UWOC system configuration using a submerged optical intelligent reflecting surface (OIRS) that is used to support multiple users by allocating different OIRS elements to various users at the receiver end. To increase the average sum rate and preserve user fairness, techniques based on equal mirror assignment (EMA) and distance-based mirror assignment (DMA) are presented. The outcomes are evaluated against conventional orthogonal multiple access (OMA) and non-orthogonal multiple access (NOMA) schemes for underwater communications.

Machine learning (ML) based skin cancer detection tools are an example of a transformative medical technology that could potentially democratize early detection for skin cancer cases for everyone. However, due to the dependency of datasets for training, ML based skin cancer detection always suffers from a systemic racial bias. Racial communities and ethnicity not well represented within the training datasets will not be able to use these tools, leading to health disparities being amplified. Based on empirical observations we posit that skin cancer training data is biased as it's dataset represents mostly communities of lighter skin tones, despite skin cancer being far more lethal for people of color. In this paper we use domain adaptation techniques by employing CycleGANs to mitigate racial biases existing within state of the art machine learning based skin cancer detection tools by adapting minority images to appear as the majority. Using our domain adaptation techniques to augment our minority datasets, we are able to improve the accuracy, precision, recall, and F1 score of typical image classification machine learning models for skin cancer classification from the biased 50% accuracy rate to a 79% accuracy rate when testing on minority skin tone images. We evaluate and demonstrate a proof-of-concept smartphone application.

Sensing and communication technology has been used successfully in various event monitoring applications over the last two decades, especially in places where long-term manual monitoring is infeasible. However, the major applicability of this technology was mostly limited to terrestrial environments. On the other hand, underwater wireless sensor networks (UWSNs) opens a new space for the remote monitoring of underwater species and faunas, along with communicating with underwater vehicles, submarines, and so on. However, as opposed to terrestrial radio communication, underwater environment brings new challenges for reliable communication due to the high conductivity of the aqueous medium which leads to major signal absorption. In this paper, we provide a detailed technical overview of different underwater communication technologies, namely acoustic, magnetic, and visual light, along with their potentials and challenges in submarine environments. Detailed comparison among these technologies have also been laid out along with their pros and cons using real experimental results.

In this paper, we present Vi-Fi, a multi-modal system that leverages a user's smartphone WiFi Fine Timing Measurements (FTM) and inertial measurement unit (IMU) sensor data to associate the user detected on a camera footage with their corresponding smartphone identifier (e.g. WiFi MAC address). Our approach uses a recurrent multi-modal deep neural network that exploits FTM and IMU measurements along with distance between user and camera (depth information) to learn affinity matrices. As a baseline method for comparison, we also present a traditional non deep learning approach that uses bipartite graph matching. To facilitate evaluation, we collected a multi-modal dataset that comprises camera videos with depth information (RGB-D), WiFi FTM and IMU measurements for multiple participants at diverse real-world settings. Using association accuracy as the key metric for evaluating the fidelity of Vi-Fi in associating human users on camera feed with their phone IDs, we show that Vi-Fi achieves between 81% (real-time) to 91% (offline) association accuracy.

The goal of this research is to model the spatio-temporal dependencies of radon gas generation and movement underground. In this regard, we have embarked upon an interdisciplinary research effort that involves studying the dependencies of radon gas emanation with soil and environmental parameters. To this end we design, implement and deploy an innovative real-time sensor network, OpenRadon Lab, to develop a radon prediction model that maps its distribution along space and time. This network constitutes a soil-to-cloud wireless computing framework to enable machine learning assisted soil radon prediction, and optimized data offloading to conserve computing resources on the sensing devices.

In this paper, we present ViTag to associate user identities across multimodal data, particularly those obtained from cameras and smartphones. ViTag associates a sequence of vision tracker generated bounding boxes with Inertial Mea-surement Unit (IMU) data and Wi-Fi Fine Time Measurements (FTM) from smartphones. We formulate the problem as association by sequence to sequence (seq2seq) translation. In this two-step process, our system first performs cross-modal translation using a multimodal LSTM encoder-decoder network (X-Translator) that translates one modality to another, e.g. recon-structing IMU and FTM readings purely from camera bounding boxes. Second, an association module finds identity matches between camera and phone domains, where the translated modality is then matched with the observed data from the same modality. In contrast to existing works, our proposed approach can associate identities in multi-person scenarios where all users may be performing the same activity. Extensive experiments in real-world indoor and outdoor environments demonstrate that online association on camera and phone data (IMU and FTM) achieves an average Identity Precision Accuracy (IDP) of 88.39% on a 1 to 3 seconds window, outperforming the state-of-the-art Vi-Fi (82.93%). Further study on modalities within the phone domain shows the FTM can improve association performance by 12.56% on average. Finally, results from our sensitivity experiments demonstrate the robustness of ViTag under different noise and environment variations.

The paper introduces a novel, holistic approach for robust Screen-Camera Communication (SCC), where video content on a screen is visually encoded in a human-imperceptible fashion and decoded by a camera capturing images of such screen content. We first show that state-of-the-art SCC techniques have two key limitations for in-the-wild deployment: (a) the decoding accuracy drops rapidly under even modest screen extraction errors from the captured images, and (b) they generate perceptible flickers on common refresh rate screens even with minimal modulation of pixel intensity. To overcome these challenges, we introduce DeepLight, a system that incorporates machine learning (ML) models in the decoding pipeline to achieve humanly-imperceptible, moderately high SCC rates under diverse real-world conditions. DeepLight's key innovation is the design of a Deep Neural Network (DNN) based decoder that collectively decodes all the bits spatially encoded in a display frame, without attempting to precisely isolate the pixels associated with each encoded bit. In addition, DeepLight supports imperceptible encoding by selectively modulating the intensity of only the Blue channel, and provides reasonably accurate screen extraction (IoU values ≥ 83%) by using state-of-the-art object detection DNN pipelines. We show that a fully functional DeepLight system is able to robustly achieve high decoding accuracy (frame error rate < 0.2) and moderately-high data goodput (≥0.95 Kbps) using a human-held smartphone camera, even over larger screen-camera distances (ã 2m).

The concept of using visible light communication (VLC) for vehicle-to-everything (V2X) communication is attractive due to the advantages of re-purposing existing lighting (brake, tail, and headlights) for optical transmissions and light sensing elements for optical reception. We design a VLC RaceCar setup that equips a suite of sensors along with programmable driving control. We posit that such a setup can be beneficial for conducting controlled vehicular VLC experiments and a primarily teaching toolkit for hands-on learning of VLC and autonomous driving topics.

We demonstrate an application of finding target persons on a surveillance video. Each visually detected participant is tagged with a smartphone ID and the target person with the query ID is highlighted. This work is motivated by the fact that establishing associations between subjects observed in camera images and messages transmitted from their wireless devices can enable fast and reliable tagging. This is particularly helpful when target pedestrians need to be found on public surveillance footage, without the reliance on facial recognition. The underlying system uses a multi-modal approach that leverages WiFi Fine Timing Measurements (FTM) and inertial sensor (IMU) data to associate each visually detected individual with a corresponding smartphone identifier. These smartphone measurements are combined strategically with RGB-D information from the camera, to learn affinity matrices using a multi-modal deep learning network.

We demonstrate a self-training system for sports involving throwing a ball. We design a do-it-yourself (DIY) machinery that can be assembled using off-the-shelf items and integrates computer vision to visually track the ball throw accuracy. In this work, we demonstrate a system that can identify if the ball went through the hoop and approximately in which of the hoop's inner region. We envision that this preliminary design sets the foundation for a complete DIY sports IoT system that involves a hoola hoop, RaspberryPi, PiCamera and a LED strip, along with advanced ball placement and dynamics tracking.

Patients with respiratory diseases require frequent and accurate blood oxygen level monitoring. Existing techniques, however, either need a dedicated hardware or fail to predict low saturation levels. To fill in this gap, we propose a phone-based oxygen level estimation system, called PhO 2 , using camera and flashlight functions that are readily available on today’s off-the-shelf smartphones. Since the phone’s camera and flashlight were not made for this purpose, utilizing them for oxygen level estimation poses many difficulties. We introduce a cost-effective add-on together with a set of algorithms for spatial and spectral optical signal modulation to amplify the optical signal of interest while minimizing noise. A near-field-based pressure detection and feedback mechanism are also proposed to mitigate the negative impacts of user’s behavior during the measurement. We also derive a non-linear referencing model with an outlier removal technique that allows PhO 2 to accurately estimate the oxygen level from color intensity ratios produced by the smartphone’s camera. An evaluation on COTS smartphone with six subjects shows that PhO 2 can estimate the oxygen saturation within 3.5% error rate comparing to FDA-approved gold standard pulse oximetry. In addition, our evaluation in hospitals presents high correlation with ground-truth qualified by the 0.83/1.0 Kendall τ coefficient.

Contaminants can leach into drinking water in transport from water-treatment facilities or treated water storage to consumer taps. In this work, we design a low-cost mobile system that can use off-the-shelf web-cameras as an ultraviolet (UV) spectrometer. We posit to analyze and categorize impurities present in water by using fundamental image processing and computer vision techniques. In this poster paper, we qualitatively and quantitatively analyze the ultraviolet absorption and scattering through images captured of an UV light source (at 385nm wavelength) transmitted through de-ionized water containing varying contaminants and at different concentrations. We particularly explore the tests for lead, arsenic, table salt, charcoal, and coconut oil, and run a pilot-study on ground-truth analysis of tap water from five counties around Atlanta, Georgia, USA.

Internet-of-Things (IoT) paradigms have created, in addition to opportunities, a huge void in security. Although there are multiple works that explore security through device identification, cryptography and network security protocols, the question of can we trust the integrity of things to represent reality or precisely, can we trust the data and the metrics being sent by things, remains largely unanswered in distributed wireless scenarios. Given how nascent the domain is and the rapid pace at which IoT is being adopted, ensuring that the data from each of these devices is trustable is very challenging. Moreover, the problem becomes harder in wireless sensor network scenario, especially in harsh environments, due to the potential avenues for spoofing and physical attacks. To this end, this paper explores conditions or threat vectors under which a wireless network of devices may become unreliable in a fully distributed setting, and present an approach to identify potential integrity failures or threats. We present the effectiveness of our approach through a use-case analysis for precision agriculture applications. Through experimental and trace-based simulations, we show that threats can potentially be identified in real-time with 80% accuracy and at about 90% precision and recall.

The digital smart-home ecosystem comprised of the smart sensing devices and those that the user interacts with, have the ability to monitor users' activities without user awareness or consent. While the intelligence provided by these devices and systems, particularly through recommendations and reminders for user activities, are helpful, they also border on the level of intrusion to the user. The framework of the smart home devices poses a potential privacy threat once the insights gained from the data are beyond simple reminders or recommendations, and rather active inference of the person's sensitive or private information. We hypothesize that it is possible a smart-home user's privacy can be compromised with the location information and temporal sampling of the user's context by the sensors. In this work, we explore testing this hypothesis towards understanding privacy intrusion by reverse engineering a privacy intrusion model using derived insights from the sensors' data. While there can be numerous facets to privacy, we categorize our privacy model development along six attributes: personal identity, localization within the smart-space, finance activity, social interaction, genealogy history and emotional attributes. As a first step in this exploration, we set up experiments in a real-world office room environment to study privacy intrusion of a single user using off-the-shelf smart sensing devices already existing in homes today.

The proliferation of Internet-of-Things (IoT) devices has opened up plethora of opportunities for smart networking and connected applications. The large distribution of IoT devices within a finite geographical area and the pervasiveness of wireless networking presents an opportunity for such devices to collaborate. This paper proposes the idea of opportunistic collaboration among mobile IoT devices to share their services and excess computing resources. Opportunistic collaboration among devices over wireless networking requires proper coordination and agreements among the devices in a purely distributed manner. To facilitate the distributive collaboration, we propose a decentralized architecture design using blockchain technology. Through experimental evaluation of a prototype collaborative mobile-IoT system involving RaspberryPis and a Dell IoT edge gateway, we show that our proposed distributed collaborative approach is feasible and comparable to a non-collaborative edge- computing approach from a latency perspective.

Using touchscreens has largely limited user inputs to small form-factor devices. To address this constraint, we explore a novel input mechanism, dubbed PaperKey, that enables users to interact with mobile devices by performing multi-finger typing gestures on a surface where the device is placed. Using acceleration signals on the device, PaperKey infers the user's type events and then leverages a vision based technique for detecting the exact typing locations on a paper keyboard layout. Compared to single audio, image, or vibration sensing, this work accurately localizes keystrokes with faster processing speed. Additionally, this mechanism keeps the mobility of devices by working without external sensors.

Accurately measuring and monitoring patient's blood oxygen level plays a critical role in today's clinical diagnosis and healthcare practices. Existing techniques however either require a dedicated hardware or produce inaccurate measurements. To fill in this gap, we propose a phone-based oxygen level estimation system, called PhO2, using camera and flashlight functions that are readily available on today's off-the-shelf smart phones. Since phone's camera and flashlight are not made for this purpose, utilizing them for oxygen level estimation poses many challenges. We introduce a cost-effective add-on together with a set of algorithms for spatial and spectral optical signal modulation to amplify the optical signal of interest while minimizing noise. A light-based pressure detection algorithm and feedback mechanism are also proposed to mitigate the negative impacts of user's behavior during the measurement. We also derive a non-linear referencing model that allows PhO2 to estimate the oxygen level from color intensity ratios produced by smartphone's camera.

Embedded screen-camera communication techniques encode information in screen imagery that can be decoded with a camera receiver yet remains unobtrusive to the human observer. These techniques have applications in tagging content on screens similar to QR-code tagging for other objects. This paper characterizes the design space for flicker-free embedded screen-camera communication. In particular, we identify an orthogonal dimension to prior work: spatial content-adaptive encoding, and observe that it is essential to combine multiple dimensions to achieve both high capacity and minimal flicker. From these insights, we develop content-adaptive encoding techniques that exploit visual features such as edges and texture to unobtrusively communicate information. These can then be layered over existing techniques to further boost the capacity. Our experimental results show that there is potential to achieve an average goodput of about 22 kbps, significantly outperforming existing work while remaining flicker-free.

In this paper, we present the design, implementation and evaluation of a user authentication system, Headbanger, for smart head-worn devices, through monitoring the user's unique head-movement patterns in response to an external audio stimulus. Compared to today's solutions, which primarily rely on indirect authentication mechanisms via the user's smartphone, thus cumbersome and susceptible to adversary intrusions, the proposed head-movement based authentication provides an accurate, robust, light-weight and convenient solution. Through extensive experimental evaluation with 95 participants, we show that our mechanism can accurately authenticate users with an average true acceptance rate of 95.57% while keeping the average false acceptance rate of 4.43%. We also show that even simple head-movement patterns are robust against imitation attacks. Finally, we demonstrate our authentication algorithm is rather light-weight: the overall processing latency on Google Glass is around 1.9 seconds.

This paper proposes the design and implementation of low-- energy tokens for smart interaction with capacitive touch-- enabled devices by associating the token's identity with its contact, or touch. The proposed token's design features two key novel technical components: (1) a through--touch--sensor low--energy communication method for token identification and (2) a touch--sensor energy harvesting technique. The communication mechanism involves the token transmitting its identity (ID) directly through the touch--sensor by artificially modifying the effective capacitance between the touch-- sensor and token surfaces. This approach consumes significantly lower energy compared to traditional electrical signal modulation approaches. By enabling the token to harvest energy from touch--screen sensors or touch--surfaces the token is rendered battery--free. Through experimental evaluations using a prototype implementation, the proposed design is shown to achieve at least 95% identification accuracy. It is also shown to consume less energy than competitive techniques (NFC P2P and Bluetooth Low--Energy) for communicating a short ID sequence. The adoption of this technology among users is evaluated through a user study on 12 subjects.

We present a novel method for communicating between a moving camera and an electronic display by embedding and recovering hidden, dynamic information within an image. A small intensity pattern is added to alternate frames of a time-varying display. A handheld camera pointed at the display can receive not only the display image, but also an underlying message. Differencing the camera-captured alternate frames leaves the small intensity pattern, but results in errors due to photometric effects that depend on camera pose. Detecting and robustly decoding the message requires careful photometric modeling for message recovery. The key innovation of our approach is an algorithm that performs simultaneous radiometric calibration and message recovery in one convex optimization problem. By modeling the photometry of the system using a camera-display transfer function (CDTF), we derive an optimal online radiometric calibration (OORC) for robust computational messaging as demonstrated with nine different commercial cameras and displays. The online radiometric calibration algorithms described in this paper significantly reduces message recovery errors, especially for low intensity messages and oblique camera angles.

Applications on wearable personal imaging devices, or Smart-glasses as they are called, can largely benefit from accurate and energy-efficient recognition of objects that are within the user's view. Existing solutions such as optical or computer vision approaches are too energy intensive, while low-power active radio tags suffer from imprecise orientation estimates. To address this challenge, this paper presents the design, implementation, and evaluation of a radio-optical hybrid system where a radio-optical transmitter, or tag, whose radio-optical beacons are used for accurate relative orientation tracking of tagged objects by a wearable radio-optical receiver. A low-power radio link that conveys identity is used to reduce the battery drain by synchronizing the radio-optical transmitter and receiver so that extremely short optical (infrared) pulses are sufficient for orientation (angle and distance) estimation. Through extensive experiments with our prototype we show that our system can achieve orientation estimates with 1-to-2 degree accuracy and within 40 cm ranging error, with a maximum range of 9 m in typical indoor use cases. With a tag and receiver battery power consumption of 81 μW and 90 mW, respectively, our radio-optical tags and receiver are at least 1.5x energy efficient than prior works in this space.

Object recognition on wearable devices using computer vision is too energy intensive and challenging when objects are similar looking, while low-power active radio frequency identification (RFID) systems suffer from imprecise orientation (angle and distance) estimates. To address this challenge, this paper presents a novel radio-optical based recognition system where a radio-optical transmitter, or tag, that emits a beacon whose infra-red (IR) signal strength is used for accurate relative orientation tracking of tagged objects at a wearable radio-optical receiver. A low-power radio link that conveys identity is used to reduce the battery drain by synchronizing the radio- optical transmitter and receiver so that extremely short optical pulses are sufficient for precise orientation estimation. Through extensive experiments with our prototype we show that our system can achieve orientation estimates with 1-2° accuracy and within 40cm ranging error, with a maximum range of 9m in typical indoor use cases. With a tag battery power consumption of 86μW, the radio-optical tags show potential to achieve about half a decade lifetimes.

Ubiquitous light emitting devices and low-cost commercial digital cameras facilitate optical wireless communication system such as visual MIMO where handheld cameras communicate with electronic displays. While intensity-based optical communications are more prevalent in camera-display messaging, we present a novel method that uses modulated light phase for messaging and time-of-flight (ToF) cameras for receivers. With intensity-based methods, light signals can be degraded by reflections and ambient illumination. By comparison, communication using ToF cameras is more robust against challenging lighting conditions. Additionally, the concept of phase messaging can be combined with intensity messaging for a significant data rate advantage. In this work, we design and construct a phase messaging array (PMA), which is the first of its kind, to communicate to a ToF depth camera by manipulating the phase of the depth camera's infrared light signal. The array enables message variation spatially using a plane of infrared light emitting diodes and temporally by varying the induced phase shift. In this manner, the phase messaging array acts as the transmitter by electronically controlling the light signal phase. The ToF camera acts as the receiver by observing and recording a time-varying depth. We show a complete implementation of a 3×3 prototype array with custom hardware and demonstrating average bit accuracy as high as 97.8%. The prototype data rate with this approach is 1 Kbps that can be extended to approximately 10 Mbps.

Cameras are ubiquitous and increasingly being used not just for capturing images but also for communicating information. For example, the pervasive QR codes can be viewed as communicating a short code to camera-equipped sensors and recent research has explored using screen-to-camera communications for larger data transfers. Such communications could be particularly attractive in pervasive camera based applications, where such camera communications can reuse the existing camera hardware and also leverage from the large pixel array structure for high data-rate communication. While several prototypes have been constructed, the fundamental capacity limits of this novel communication channel in all but the simplest scenarios remains unknown. The visual medium differs from RF in that the information capacity of this channel largely depends on the perspective distortions while multipath becomes negligible. In this paper, we create a model of this communication system to allow predicting the capacity based on receiver perspective (distance and angle to the transmitter). We calibrate and validate this model through lab experiments wherein information is transmitted from a screen and received with a tablet camera. Our capacity estimates indicate that tens of Mbps is possible using a smartphone camera even when the short code on the screen images onto only 15% of the camera frame. Our estimates also indicate that there is room for at least 2.5x improvement in throughput of existing screen - camera communication prototypes.

Augmented Reality (AR) applications benefit from accurate detection of the objects that are within a person's view. Typically, it is not only desirable to identify what is currently within view, but also to navigate the users view to the item of interest - for example, finding a misplaced object. In this paper we demonstrate a low-power hybrid radio-optical beaconing system, where objects of interest are tagged with battery-powered RFID-like tags equipped with infrared light emitting diodes (LED) that emit periodic infrared beacons. These beacons are used for accurately estimating the angle and distance from the object to the receiver so as to locate it. The beacons are synchronized using the radio link that is also used to convey the object's unique ID.

Today's identification and authentication mechanisms for touchscreen-enabled devices are cumbersome and do not support brief usage and device sharing. To address this challenge, this work explores a novel form of "wireless" communication that exploits the capacitive touchscreens which are now used in laptops, phones, and tablets, as a signal receiver. Using a custom built hardware token, in the form of a wearable ring, we show a proof-of-concept system that transmits a user identification code to the mobile device through the touchscreen. This mechanism works without any modification to the hardware or the firmware of the mobile device.

The growing ubiquity of cameras in hand-held devices and the prevalence of electronic displays in signage creates a novel framework for wireless communications. Traditionally, the term MIMO is used for multiple-input multiple-output where the multiple-input component is a set of radio transmitters and the multiple-output component is a set of radio receivers. We employ the concept of visual MIMO where pixels are transmitters and cameras are receivers. In this manner, the techniques of computer vision can be combined with principles from wireless communications to create an optical line-of-sight communications channel. Two major challenges are addressed: (1) The message for transmission must be embedded in the observed display so that the message is hidden from the observer and the electronic display can simultaneously be used for its originally intended purpose (e.g. signage, advertisements, maps); (2) Photometric and geometric distortions during the imaging process corrupt the information channel between the transmitter display and the receiver camera. These distortions must be modeled and removed. In this paper, we present a real-time messaging paradigm and its implementation in an operational visual MIMO optical systems. As part of the system, we develop a novel algorithm for photographic message extraction which includes automatic display detection, message embedding and message retrieval. Experiments show that the system achieves an average accuracy of 94.6% at the bitrate of 6222.2 bps.

The inherent limitations in RF spectrum availability and susceptibility to interference make it difficult to meet the reliability required for automotive safety applications. To address this challenge, this work explores an alternative communication system called Visual MIMO that uses light emitting arrays as transmitters and cameras as receivers. Visual MIMO applied to vehicular communication proposes to reuse existing LED rear and headlights as transmitters and existing cameras (e.g. those used for parking assistance, rear-view cameras) as receivers. In this work we show a proof of concept based demonstration of the Visual MIMO system consisting of an LED transmitter array and a high-speed camera.

We propose a rate adaptation scheme for visual MIMO camera-based communications, wherein parallel data transmissions from light emitting arrays are received by multiple receive elements of a CCD/CMOS camera image sensor. Unlike RF MIMO, multipath fading is negligible in the visual MIMO channel. Instead, the channel is largely dependent on receiver perspective (distance and angle) and visibility issues (partial line-of-sight availability and occlusions). This allows for slower adaptation but requires the adaptation algorithm to choose among a more complex set of modes. In this paper, we define a set of operating modes for visual MIMO transmitters and propose a rate adaptation scheme to switch between these modes. Our Visual MIMO Rate Adaptation (VMRA) is a packet based rate adaptation protocol that bases its rate selection decisions on the packet error rate feedback. Using trace-based simulation results for a vehicle-to-vehicle communication scenario, we illustrate how our VMRA algorithms can adapt over distance as well as visibility variations in an optical link and achieve a higher average throughput.

Mobile optical communications has so far largely been limited to short ranges of about ten meters, since the highly directional nature of optical transmissions would require costly mechanical steering mechanisms. Advances in CCD and CMOS imaging technology along with the advent of visible and infrared (IR) light sources such as (light emitting diode) LED arrays presents an exciting and challenging concept which we call as visual-MIMO (multiple-input multiple-output) where optical transmissions by multiple transmitter elements are received by an array of photodiode elements (e.g. pixels in a camera). Visual-MIMO opens a new vista of research challenges in PHY, MAC and Network layer research and this paper brings together the networking, communications and computer vision fields to discuss the feasibility of this as well as the underlying opportunities and challenges. Example applications range from household/factory robotic to tactical to vehicular networks as well pervasive computing, where RF communications can be interference-limited and prone to eavesdropping and security lapses while the less observable nature of highly directional optical transmissions can be beneficial. The impact of the characteristics of such technologies on the medium access and network layers has so far received little consideration. Example characteristics are a strong reliance on computer vision algorithms for tracking, a form of interference cancellation that allows successfully receiving packets from multiple transmitters simultaneously, and the absence of fast fading but a high susceptibility to outages due to line-of-sight interruptions. These characteristics lead to significant challenges and opportunities for mobile networking research