While 3D hand reconstruction from monocular images has made significant progress, generating accurate and temporally coherent motion estimates from videos remains challenging, particularly during hand-object interactions. In this paper, we present a novel 3D hand motion recovery framework that enhances image-based reconstructions through a diffusion-based and physics-augmented motion refinement model. Our model captures the distribution of refined motion estimates conditioned on initial ones, generating improved sequences through an iterative denoising process. Instead of relying on scarce annotated video data, we train our model only using motion capture data without images. We identify valuable intuitive physics knowledge during hand-object interactions, including key motion states and their associated motion constraints. We effectively integrate these physical insights into our diffusion model to improve its performance. Extensive experiments demonstrate that our approach significantly improves various frame-wise reconstruction methods, achieving state-of-the-art (SOTA) performance on existing benchmarks.

While current methods have shown promising progress on estimating 3D human motion from monocular videos, their motion estimates are often physically unrealistic be-cause they mainly consider kinematics. In this paper, we in-troduce Physics-aware Pretrained Transformer (PhysPT), which improves kinematics-based motion estimates and in-fers motion forces. PhysPT exploits a Transformer encoder-decoder backbone to effectively learn human dynamics in a self-supervised manner. Moreover, it incorporates physics principles governing human motion. Specifically, we build a physics-based body representation and contact force model. We leverage them to impose novel physics-inspired training losses (i.e., force loss, contact loss, and Euler-Lagrange loss), enabling PhysPT to capture physical properties of the human body and the forces it experiences. Experiments demonstrate that, once trained, PhysPT can be directly ap-plied to kinematics-based estimates to significantly enhance their physical plausibility and generate favourable motion forces. Furthermore, we show that these physically meaningful quantities translate into improved accuracy of an important downstream task: human action recognition.

In spite of the significant progresses in monocular or multi-view image based 3D face reconstruction research, recovering 3D faces from videos, which contains rich dynamic information of facial motions, still remains as a highly challenging problem. First, most prior works fail to generate accurate and stable 3D faces on videos, especially for recovering subtle expression details. Furthermore, existing dynamic reconstruction approaches have not fully considered the temporal dependency of facial expression transitions, which is based on the dynamic muscle activation system under a local region of the skin. To tackle the aforementioned challenges, we present a framework for dynamic 3D face reconstruction from monocular videos, which can accurately recover 3D facial geometrical representations for facial action unit (AU). Specifically, we design a coarse-to-fine framework, where the "coarse" 3D face sequences are generated by a pre-trained static reconstruction model; and the "refinement" is performed through a Transformer-based network. We design 1) a Temporal Module used for modeling temporal dependency of facial motion dynamics; 2) an Spatial Module for modeling AU spatial correlations from geometry-based AU tokens; 3) feature fusion for simultaneous dynamic facial AU recognition and 3D expression capturing. Experimental results show the superiority of our method in generating AU-aware 3D face reconstruction sequences both quantitatively and qualitatively.

A variety of real-world applications rely on accurate predictions of 3D human motion from their past observations. While existing methods have made notable progress, their predictions over subsecond horizons can still be off by many centimeters. In this paper, we argue that achieving precise human motion prediction requires characterizing the fundamental physics principles governing body movements. We introduce PhysMoP, a novel framework that incorporates Physics for human Motion Prediction. PhysMoP estimates the body configuration of the next frame by solving the Euler-Lagrange equations, a set of Ordinary Different Equations describing the physical motion rules. To limit the inherent problem of error accumulation over time, PhysMoP leverages a data-driven model and iteratively guides the physics-based prediction via a fusion model. Through extensive experiments, we demonstrate that PhysMoP significantly outperforms existing approaches at subsecond prediction horizons. For example, at a prediction horizon of 80 msec, PhysMoP outperforms traditional data-driven approaches by a factor of 10 or more.

While 3D body reconstruction methods have made remarkable progress recently, it remains difficult to acquire the sufficiently accurate and numerous 3D supervisions required for training. In this paper, we propose KNOWN, a framework that effectively utilizes body KNOWledge and uNcertainty modeling to compensate for insufficient 3D supervisions. KNOWN exploits a comprehensive set of generic body constraints derived from well-established body knowledge. These generic constraints precisely and explicitly characterize the reconstruction plausibility and enable 3D reconstruction models to be trained without any 3D data. Moreover, existing methods typically use images from multiple datasets during training, which can result in data noise (e.g., inconsistent joint annotation) and data imbalance (e.g., minority images representing unusual poses or captured from challenging camera views). KNOWN solves these problems through a novel probabilistic framework that models both aleatoric and epistemic uncertainty. Aleatoric uncertainty is encoded in a robust Negative Log-Likelihood (NLL) training loss, while epistemic uncertainty is used to guide model refinement. Experiments demonstrate that KNOWN’s body reconstruction outperforms prior weakly-supervised approaches, particularly on the challenging minority images.

The Rensselaer Mandarin Project enables a group of foreign language students to improve functional understanding, pronunciation and vocabulary in Mandarin Chinese through authentic speaking situations in a virtual visit to China. Students use speech, gestures, and combinations thereof to navigate an immersive, mixed reality, stylized realism game experience through interaction with AI agents, immersive technologies, and game mechanics. The environment was developed in a black box theater equipped with a human-scale 360◦ panoramic screen (140h, 200r), arrays of markerless motion tracking sensors, and speakers for spatial audio.

We introduce Reagent, a technology that can be used in conjunction with automated speech recognition to allow users to query and manipulate ordinary webpages via speech and pointing. Reagent can be used out-of-the-box with third-party websites, as it requires neither special instrumentation from website developers nor special domain knowledge to capture semantically-meaningful mouse interactions with structured elements such as tables and plots. When it is unable to infer mappings between domain vocabulary and visible webpage content on its own, Reagent proactively seeks help by engaging in a voice-based interaction with the user.

We demonstrate an embodied cognitive agent that helps scientists visualize and analyze exo-planets and their host stars. The prototype is situated in a room equipped with a large display, microphones, cameras, speakers, and pointing devices. Users communicate with the agent via speech, gestures, and combinations thereof, and it responds by displaying content and generating synthesized speech. Extensive use of context facilitates natural interaction with the agent.

In this demonstration, we report on the visualization capabilities of an Explainable AI Planning (XAIP) agent that can support human-in-the-loop decision-making. Imposing transparency and explainability requirements on such agents is crucial for establishing human trust and common ground with an end-to-end automated planning system. Visualizing the agent's internal decision making processes is a crucial step towards achieving this. This may include externalizing the "brain" of the agent: starting from its sensory inputs, to progressively higher order decisions made by it in order to drive its planning components. We demonstrate these functionalities in the context of a smart assistant in the Cognitive Environments Laboratory at IBM's T.J. Watson Research Center.

Economic systems are rife with heterogeneous risk events that have the potential to cause disruption. The diversity of risk types makes it challenging for companies to conduct comprehensive risk analysis for any chosen business opportunity. The current practice is laborious and expensive, involving internal risk analysts and external risk advisory services. In this paper, we present a cognitive system that augments human abilities, with the objective of drastic improvements in the productivity of risk analysis efforts. Our system is provided with a comprehensive risk taxonomy and its textual description along with an extensive corpus of textual data such as news articles. Using a series of textual analysis, knowledge extraction and machine learning techniques, the data corpus is annotated with risk-related information and indexed in a risk store for flexible query and retrieval. Our system interfaces with the risk analyst using a query orchestrator which translates analyst queries that are posed at a high level into lower level queries that are expanded to exploit the system's risk-related knowledge. It also enables formulating a graphical model and assessing the required probabilities; we introduce a particular family of models that can succinctly represent risk events modeled as stochastic processes over a long time horizon. We illustrate how a risk analyst can query the system to build a risk model with the help of a case study.

One of the main driving forces of the growing adoption of virtualization is its dramatic simplification of the provisioning and dynamic management of IT resources. By decoupling running entities from the underlying physical resources, and by providing easy-to-use controls to allocate, deallocate and migrate virtual machines (VMs) across physical boundaries, virtualization opens up new opportunities for improving overall system resource use and power efficiency. While a range of techniques for dynamic, distributed resource management of virtualized systems have been proposed and have seen their widespread adoption in enterprise systems, similar techniques for dynamic power management have seen limited acceptance. The main barrier to dynamic, power-aware virtualization management stems not from the limitations of virtualization, but rather from the underlying physical systems; and in particular, the high latency and energy cost of power state change actions suited for virtualization power management.

We describe an inexpensive autonomous robot capable of navigating previously unseen data centers and monitoring key metrics such as air temperature. The robot provides real-time navigation and sensor data to commercial IBM software, thereby enabling real-time generation of the data center layout, a thermal map and other visualizations of energy dynamics. Once it has mapped a data center, the robot can efficiently monitor it for hot spots and other anomalies using intelligent sampling. We demonstrate the robot's effectiveness via experimental studies from two production data centers.

We will demonstrate a robot for data center energy management, in action, on a simulated data center floor. We shall highlight the robot's navigation, tile and obstacle classification, event scheduling and preemption capabilities, along with its ability to discover charging docks, and successfully dock with extreme precision. We shall also show simulations on real data center layouts evincing navigational efficiency gains obtained by our latest heuristic enhancements.

An autonomic manager for enterprise server hardware management, called AMP, is described. AMP is designed to handle multiple aspects of hardware management and to work in conjunction with other management components, in particular application managers, in a way that reduces energy waste, protects server health, and preserves a high degree of autonomy both for itself and for the managers with which it works. AMP interacts with other managers in two ways: (1) exchange of nominal control over individual servers; and (2) provision of a synthetic cost function giving AMP's assessment of relative desirability of using different servers. The high-level architecture of AMP is discussed, with particular focus on the way it effects a natural decomposition of the combined hardware-and-application management problem, and on initial versions of the algorithms it uses to manage server power states and determine the cost function. AMP's viability in practice is demonstrated via prototype implementation in which it operates on real servers in collaboration with a state-of-the-art application manager. The overall system behavior is investigated via simulation.

Systems management techniques that allocate resources to running entities, such as processes and virtual machines (VMs), often require estimates of the resources required by each of these resource consumers. For example, many proposed virtual machine placement algorithms attempt to allocate VMs to physical hosts in such a way as to minimize the number of physical hosts that are occupied, while ensuring that each VM receives the CPU required to do its task adequately. The common practice is to assume that the CPU requirement is equal to the current CPU utilization, or to use a prediction of it over an appropriate time horizon. In this paper, we demonstrate that, when multiple VMs or processes co-reside on a physical host, the measured CPU utilization may provide a poor estimate of the actual requirement. We derive a simple, much more accurate alternative estimate of CPU demand, implement it, and demonstrate its superiority experimentally. Furthermore, we demonstrate that using our demand estimation framework in conjunction with dynamic resource allocation in a virtualized environment greatly improves the effectiveness of dynamic placement, resulting in one-shot convergence to optimal placement and significant improvements in the overall performance of the individual VMs.

With the continued growth of computing power and reduction in physical size of enterprise servers, the need for actively managing electrical power usage in large datacenters is becoming ever more pressing. By far the greatest savings in electrical power can be effected by dynamically consolidating workload onto the minimum number of servers needed at a given time and powering off the remainder. However, simple schemes for achieving this goal fail to cope with the complexities of realistic usage scenarios. In this paper we present a combined power-and performance-management system that builds on a state-of-the-art performance manager to achieve significant power savings without unacceptable loss of performance. In our system, the degree to which performance may be traded off against power is itself adjustable using a small number of easily-understood parameters, permitting administrators in different facilities to select the optimal tradeoff for their needs. We characterize the power saved, the effects of the tradeoff between power and performance, and the changes in behavior as the tradeoff parameters are adjusted, both in simulation and in a sample deployment of the real system.

Electrical power management in large-scale IT systems such as commercial data- centers is an application area of rapidly growing interest from both an economic and ecological perspective, with billions of dollars and millions of metric tons of CO2 emissions at stake annually. Businesses want to save power without sac- rificing performance. This paper presents a reinforcement learning approach to simultaneous online management of both performance and power consumption. We apply RL in a realistic laboratory testbed using a Blade cluster and dynam- ically varying HTTP workload running on a commercial web applications mid- dleware platform. We embed a CPU frequency controller in the Blade servers’ firmware, and we train policies for this controller using a multi-criteria reward signal depending on both application performance and CPU power consumption. Our testbed scenario posed a number of challenges to successful use of RL, in- cluding multiple disparate reward functions, limited decision sampling rates, and pathologies arising when using multiple sensor readings as state variables. We describe innovative practical solutions to these challenges, and demonstrate clear performance improvements over both hand-designed policies as well as obvious “cookbook” RL implementations.

Autonomic computing is a grand-challenge vision of the future in which computing systems will manage themselves in accordance with high-level objectives specified by humans. The IT industry recognizes that meeting this challenge is imperative; otherwise, IT systems will soon become virtually impossible to administer. But meeting this challenge is also extremely difficult, and will require a worldwide collaboration among the best minds of academia and industry. In the hope of motivating researchers in relevant areas to apply their expertise to this vitally important problem, I outline some of the main scientific and engineering challenges that collectively make up the grand challenge of autonomic computing, and provide pointers to initial efforts to address these challenges.

No abstract available.

As online markets for the exchange of goods and services become more common, the study of markets composed, at least in part, of autonomous agents has taken on increasing importance. In contrast to traditional complete–information economic scenarios, agents that are operating in an electronic marketplace often do so under considerable uncertainty. In order to reduce their uncertainty, these agents must learn about the world around them. When an agent producer is engaged in a learning task in which data collection is costly, such as learning the preferences of a consumer population, it is faced with a classic decision problem: when to explore and when to exploit. If the agent has a limited number of chances to experiment, it must explicitly consider the cost of learning (in terms of foregone profit) against the value of the information acquired. Information goods add an additional dimension to this problem; due to their flexibility, they can be bundled and priced according to a number of different price schedules. An optimizing producer should consider the profit each price schedule can extract, as well as the difficulty of learning of this schedule. In this paper, we demonstrate the tradeoff between complexity and profitability for a number of common price schedules. We begin with a one–shot decision as to which schedule to learn. Schedules with moderate complexity are preferred in the short and medium term, as they are learned quickly, yet extract a significant fraction of the available profit. We then turn to the repeated version of this one–shot decision and show that moderate complexity schedules, in particular two–part tariff, perform well when the producer must adapt to nonstationarity in the consumer population. When a producer can dynamically change schedules as it learns, it can use an explicit decision–theoretic formulation to greedily select the schedule which appears to yield the greatest profit in the next period. By explicitly considering both the learnability and the profit extracted by different price schedules, a producer can extract more profit as it learns than if it naively chose models that are accurate once learned.

We explore a scenario in which a monopolist producer of information goods seeks to maximize its profits in a market where consumer demand shifts frequently and unpredictably. The producer may set an arbitrarily complex price schedule---a function that maps the set of purchased items to a price. However, lacking direct knowledge of consumer demand, it cannotcompute the optimal schedule. Instead, it attempts to optimize profits via trial and error. By means of a simple model of consumer demand and a modified version of a simple nonlinear optimization routine, we study a variety of parametrizations of the price schedule and quantify some of the relationships among learnability, complexity, and profitability. In particular, we show that fixed pricing or simple two-parameter dynamic pricing schedules are preferred when demand shifts frequently, but that dynamic pricing based on more complex schedules tends to be most profitable when demand shifts very infrequently.

We introduce an information bundling model that addresses two important but relatively unstudied issues in real markets for information goods: automated customization of content based on categories, and competition among content providers. Using this model, we explore the strategies that sellers (or automated agents acting on their behalf) might use to set both price and bundle composition, and the market dynamics that might ensue from such strategy choices. The model incorporates different categories of information, explicitly accounts for finite production and consumption costs, and allows for possibly heterogeneous valuations by consumers. First, we determine the optimal bundle composition and price for a monopolist as a function of the seller's production costs and the consumers' preferences and consumption costs. For finite costs, finite-sized bundles are optimal. Then, we use game-theoretic analysis and simulation to explore the behavior of the market when there are multiple cont...

In an automated market for electronic goods new problems arise that have not been well studied previously. For example, information goods are very flexible. Marginal costs are negligible and nearly limitless bundling and unbundling of these items are possible, in contrast to physical goods. Consequently, producers can offer complex pricing schemes. However, the profit-maximizing design of a complex pricing schedule depends on a producer's knowledge of the distribution of consumer preferences for the available information goods. Preferences are private and can only be gradually uncovered through market experience. In this paper we compare dynamic performance across price schedules of varying complexity. We provide the producer with two machine learning methods producer that is performing a naive, knowledge-free form of leanings (function approximation and hill-climbing) which implement a strategy that balances exploitation to maximize current profits against exploration of the profit landscape to improve future profits. We find that the tradeoff between exploitation and exploration is different depending on the learning algorithms employed, and in particular depending on the complexity of the price schedule that if offered. In general, simpler price schedules are more robust and give up less profit during the learning periods even though in our stationary environment learning eventually is complete and the more complex schedules have high long-run profits. These results hold for both learning methods, even though the relative performance of the methods is quite sensitive to choice of initial conditions and differences in the smoothness of the profit landscape for different price schedules. Our results have implications for automated learning and strategic pricing in non-stationary environments, which arise when the consumer population changes, individuals change their preferences, or competing firms change their strategies.

Shopbots are software agents that automatically query multiple sellers on the Internet to gather information about prices and other attributes of consumer goods and services. Rapidly increasing in number and sophistication, shopbots are helping more and more buyers minimize expenditure and maximize satisfaction. In response at least partly to this trend, it is anticipated that sellers will come to rely on pricebots, automated agents that employ price-setting algorithms in an attempt to maximize profits. This paper reaches toward an understanding of strategic pricebot dynamics. More specifically, this paper is a comparative study of four candidate price-setting strategies that differ in informational and computational requirements: gametheoretic pricing (GT), myoptimal pricing (MY), derivative following (DF), and Q-learning (Q). In an effort to gain insights into the tradeoffs between practicality and pro tability of pricebot algorithms, the dynamic behavior that arises among homogeneous and heterogeneous collections of pricebots and shopbot-assisted buyers is analyzed and simulated. In homogeneous settings -- when all pricebots use the same pricing algorithm -- DFs outperform MYs and GTs. Investigation of heterogeneous collections of pricebots, however, reveals an incentive for individual DFs to deviate to MY or GT. The Q strategy exhibits superior performance to all the others since it learns to predict and account for the long-term consequences of its actions. Although the current implementation of Q is impractically expensive, techniques for achieving similar performance at greatly reduced computational cost are under investigation.

One scenario of the future of computation populates the Internet with vast numbers of software agents providing, trading, and using a rich variety of information goods and services in an open, free-market economy. An essential task in such an economy is the retailing or brokering of information: gathering it from the right producers and distributing it to the right consumers. This article investigates one crucial aspect of brokers' dynamical behavior, their price-setting mechanisms, in the context of a simple information-filtering economy. We consider only the simplest cases in which a broker sets its price and product parameters based solely on the system's current state, without explicit prediction of the future. Analytical and numerical results show that the system's dynamical behavior in such "myopic" cases is generally an unending cycle of disastrous competitive "wars" in price/product space. These in turn are directly attributable to the existence of multiple peaks in the brokers' profitability landscapes, a feature whose generality is likely to extend far beyond our model.

To understand the current extent of the computer virus problem and predict its future course, the authors have conducted a statistical analysis of computer virus incidents in a large, stable sample population of PCs and developed new epidemiological models of computer virus spread. Only a small fraction of all known viruses have appeared in real incidents, partly because many viruses are below the theoretical epidemic threshold. The observed sub-exponential rate of viral spread can be explained by models of localized software exchange. A surprisingly small fraction of machines in well-protected business environments are infected. This may be explained by a model in which, once a machine is found to be infected, neighboring machines are checked for viruses. This kill signal idea could be implemented in networks to greatly reduce the threat of viral spread. A similar principle has been incorporated into a cost-effective anti-virus policy for organizations which works quite well in practice.>

The authors have designed and implemented an open, market-based computational system called Spawn. The Spawn system utilizes idle computational resources in a distributed network of heterogeneous computer workstations. It supports both coarse-grain concurrent applications and the remote execution of many independent tasks. Using concurrent Monte Carlo simulations as prototypical applications, the authors explore issues of fairness in resource distribution, currency as a form of priority, price equilibria, the dynamics of transients, and scaling to large systems. In addition to serving the practical goal of harnessing idle processor time in a computer network, Spawn has proven to be a valuable experimental workbench for studying computational markets and their dynamics.>

The strong analogy between biological viruses and their computational counterparts has motivated the authors to adapt the techniques of mathematical epidemiology to the study of computer virus propagation. In order to allow for the most general patterns of program sharing, a standard epidemiological model is extended by placing it on a directed graph and a combination of analysis and simulation is used to study its behavior. The conditions under which epidemics are likely to occur are determined, and, in cases where they do, the dynamics of the expected number of infected individuals are examined as a function of time. It is concluded that an imperfect defense against computer viruses can still be highly effective in preventing their widespread proliferation, provided that the infection rate does not exceed a well-defined critical epidemic threshold.>