This paper explores sensor-informed personal agents that can take advantage of sensor hints on wearables to enhance the personal agent's response. We demonstrate that such a sensor-in-the-loop AI agent design can be easily integrated into existing LLM agents by building a prototype named WellMax based on existing well-developed techniques such as structured prompt templates and few-shot prompting. The head-to-head comparison with a non-sensor-informed agent across five use scenarios demonstrates that this sensor-in-the-loop design can effectively improve users' needs and their overall experience. The deep-dive into agents' replies and participants' feedback further reveals that sensor-in-the-loop agents not only provide more contextually relevant responses but also exhibit a better understanding of user priorities and situational nuances. In addition, we conduct two case studies to examine the potential pitfalls and distill key insights from this sensor-in-the-loop agent. We hope this work can spawn new ideas for building more intelligent, empathetic, and effective AI-driven personal assistants.

We are witnessing a surge in the reported cases of counterfeit liquid products in the market including olive oil, honey, and alcohol. Counterfeiters often adulterate the liquid products by replacing a large portion of the authentic content with cheaper substitutes (e.g., mixing vodka with cheaper alcohol or potentially toxic methanol). Exacerbating the problem, the counterfeits are packaged and sealed to factory standards, rendering it extremely difficult for an average consumer to identify them. While solutions exist, they are often impractical for the general public as they require specialized and costly equipment. To overcome these limitations, we propose LiquidHash, a novel counterfeit liquid food product detection system. LiquidHash is a practical solution that only requires the use of a commodity smartphone to detect adulterated liquid products without opening the bottles. LiquidHash works by detecting and tracking the shape and movement of air bubbles that form inside the bottles. We implement LiquidHash and evaluate its feasibility with real-world experiments under varying conditions with a total of more than 500 minutes of video recording and observe an overall detection accuracy of up to 95%.

Counterfeit liquid food products, including olive oil, honey and alcohol, are continuing to pose severe threats to the general public as counterfeiters adulterate the authentic content with cheaper and potentially harmful substitutes, and package them in authentic bottles. Existing solutions are often impractical for the general public as they require specialized and costly equipment as well as taking liquid samples. We overcome these limitations by proposing LiquidHash, a novel detection system that only requires the use of a commodity smartphone to detect adulterated liquid products without opening the bottles. LiquidHash leverages computer vision and machine learning techniques to extract characteristics of air bubbles formed by flipping a bottle. We implement LiquidHash and evaluate its feasibility with real-world experiments and achieve an overall detection accuracy of up to 95%.