Dor Muhlgay, Ori Ram, Inbal Magar, Yoav Levine, Nir Ratner, Yonatan Belinkov, Omri Abend, Kevin Leyton-Brown, Amnon Shashua, Yoav Shoham. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, Yoav Shoham. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

Abstract Retrieval-Augmented Language Modeling (RALM) methods, which condition a language model (LM) on relevant documents from a grounding corpus during generation, were shown to significantly improve language modeling performance. In addition, they can mitigate the problem of factually inaccurate text generation and provide natural source attribution mechanism. Existing RALM approaches focus on modifying the LM architecture in order to facilitate the incorporation of external information, significantly complicating deployment. This paper considers a simple alternative, which we dub In-Context RALM: leaving the LM architecture unchanged and prepending grounding documents to the input, without any further training of the LM. We show that In-Context RALM that builds on off-the-shelf general purpose retrievers provides surprisingly large LM gains across model sizes and diverse corpora. We also demonstrate that the document retrieval and ranking mechanism can be specialized to the RALM setting to further boost performance. We conclude that In-Context RALM has considerable potential to increase the prevalence of LM grounding, particularly in settings where a pretrained LM must be used without modification or even via API access.1

Modern neural language models (LMs) have grabbed much attention in recent years, due in part to their massive sizes and the resources (time, money, data) required to derive them, and in part to their unprecedented performance on language understanding and generation tasks. It is clear that massive LMs are a required component of any future natural language system, are critical to improving existing applications such as search, and enable new applications previously beyond the reach of technology.

Yoav Levine, Barak Lenz, Or Dagan, Ori Ram, Dan Padnos, Or Sharir, Shai Shalev-Shwartz, Amnon Shashua, Yoav Shoham. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. 2020.

Games with continuous state and action spaces present unique challenges from an artificial intelligence (AI) viewpoint. Billiards, or pool, is one such domain that has been the focus of several research efforts aimed at designing AI agents to play successfully. Due to the continuous nature of the actions, it is natural to believe that the more time an agent has to investigate actions, the better it will perform. This paper gives a thorough description of a successful agent with a novel distributed architecture, designed for being able to grant further time for shot simulation and analysis through the utilization of many CPUs. A brief analysis of the distributed component of the agent is presented, as well as how much the extra time thus obtained contributed to its success, especially when compared to its other novel components. The described agent, CueCard, won the Computer Olympiad computational pool tournament held in 2008.

We consider the situation in which an organizer is trying to convenean event, and needs to choose whom out of a given set of agents to invite.Agents have preferences over how many attendees should be at the eventand possibly also who the attendees should be.This induces a stability requirement: All invited agents should preferattending to not attending, and all the other agents should not regretbeing not invited.The organizer's objective is to find an invitation of maximum size,subject to the stability requirement. We investigate the computational complexity of finding such an invitation when agents are truthful, as well as the mechanism design problem when agents act strategically.

We study repeated games in which players have imperfect execution skill and one player’s true skill is not common knowledge. In these settings the possibility arises of a player “hustling”, or pretending to have lower execution skill than they actually have. Focusing on repeated zero-sum games, we provide a hustle-proof strategy; this strategy maximizes a player’s payoff, regardless of the true skill level of the other player. 1

We investigated the existence of fair seeding in knockout tournaments. We define two fairness criteria, both adapted from the literature: envy-freeness and order preservation. We show how to achieve the first criterion in tournaments whose structure is unconstrained, and prove an impossibility result for balanced tournaments. For the second criterion we have a similar result for unconstrained tournaments, but not for the balanced case. We provide instead a heuristic algorithm which we show through experiments to be efficient and effective. This suggests that the criterion is achievable also in balanced tournaments. However, we prove that it again becomes impossible to achieve when we add a weak condition guarding against the phenomenon of tournament dropout.

Motivated by the prevalence of online questionnaires in electronic commerce, and of multiple-choice questions in such questionnaires, we consider the problem of eliciting truthful answers to multiple-choice questions from a knowledgeable respondent. Specifically, each question is a statement regarding an uncertain future event, and is multiple-choice -- the responder must select exactly one of the given answers. The principal offers a payment, whose amount is a function of the answer selected and the true outcome (which the principal will eventually observe). This problem significantly generalizes recent work on truthful elicitation of distribution properties, which itself generalized a long line of work in elicitation of complete distributions. We provide necessary and sufficient conditions for the existence of payments that induce truthful answers, and give a characterization of those payments. We also study in greater details the common case of questions with ordinal answers, and illustrate our results with several examples of practical interest.

While logical theories of informational attitudes, such as knowledge, certainty and belief, have flourished in the past two decades, logical formalization of other facets of rational behavior has lagged behind significantly. One intriguing line of research concerns the concept of intention. I will discuss one approach to tackling the notion within a logical framework.

Is it possible to predict how long an algorithm will take to solve a previously-unseen instance of an NP-complete problem? If so, what uses can be found for models that make such predictions? This article provides answers to these questions and evaluates the answers experimentally. We propose the use of supervised machine learning to build models that predict an algorithm's runtime given a problem instance. We discuss the construction of these models and describe techniques for interpreting them to gain understanding of the characteristics that cause instances to be hard or easy. We also present two applications of our models: building algorithm portfolios that outperform their constituent algorithms, and generating test distributions that emphasize hard problems. We demonstrate the effectiveness of our techniques in a case study of the combinatorial auction winner determination problem. Our experimental results show that we can build very accurate models of an algorithm's running time, interpret our models, build an algorithm portfolio that strongly outperforms the best single algorithm, and tune a standard benchmark suite to generate much harder problem instances.

We examine a class of wagering mechanisms designed to elicit truthful predictions from a group of people without requiring any outside subsidy. We propose a number of desirable properties for wagering mechanisms, identifying one mechanism - weighted-score wagering - that satisfies all of the properties. Moreover, we show that a single-parameter generalization of weighted-score wagering is the only mechanism that satisfies these properties. We explore some variants of the core mechanism based on practical considerations.

We investigate the problem of truthfully eliciting an expert's assessment of a property of a probability distribution, where a property is any real-valued function of the distribution such as mean or variance. We show that not all properties are elicitable; for example, the mean is elicitable and the variance is not. For those that are elicitable, we provide a representation theorem characterizing all payment (or "score") functions that induce truthful revelation. We also consider the elicitation of sets of properties. We then observe that properties can always be inferred from sets of elicitable properties. This naturally suggests the concept of elicitation complexity; the elicitation complexity of property is the minimal size of such a set implying the property. Finally we discuss applications to prediction markets.

We investigate asymptotically optimal keyword auctions, that is, auctions which maximize revenue as the number of bidders grows. We do so under two alternative behavioral assumptions. The first explicitly models the repeated nature of keyword auctions. It introduces a novel assumption on individual bidding, namely that bidders never overbid their value, and bid their actual value if shut out for long enough. Under these conditions we present a broad class of repeated auctions that are asymptotically optimal among all sequential auctions (a superset of repeated auctions). Those auctions have varying payment schemes but share the ranking method. The Google auction belongs to this class, but not the Yahoo auction, and indeed we show that the latter is not asymptotically optimal. (Nonetheless, with some additional distributional assumptions, the Yahoo auction can be shown to belong to a broad category of auctions that are asymptotically optimal among all auction mechanisms that do not rely on ad relevance.) We then look at the one-shot keyword auction, which can be taken to model repeated auctions in which relatively myopic bidders converge on the equilibrium of the full-information stage game. In this case we show that the Google auction remains asymptotically optimal and the Yahoo auction suboptimal. The distributional assumptions under which our theorems hold are quite general. We do however show that the Google auction is not asymptotically revenue-maximizing for general distributions.

At this point, restricted to rather specialized areas, and even there must be taken with a grain of salt. But at the same time you can't afford not to know it; there is currently no better underpinning for understanding multiagent systems. I will elaborate using experience in both academia and industry.

We study coalitional games where the value of cooperation among the agents are solely determined by the attributes the agents possess, with no assumption as to how these attributes jointly determine this value. This framework allows us to model diverse economic interactions by picking the right attributes. We study the computational complexity of two coalitional solution concepts for these games -- the Shapley value and the core. We show how the positive results obtained in this paper imply comparable results for other games studied in the literature.

We present a new approach to representing coalitional games based on rules that describe the marginal contributions of the agents. This representation scheme captures characteristics of the interactions among the agents in a natural and concise manner. We also develop efficient algorithms for two of the most important solution concepts, the Shapley value and the core, under this representation. The Shapley value can be computed in time linear in the size of the input. The emptiness of the core can be determined in time exponential only in the treewidth of a graphical interpretation of our representation.

We propose a new set of criteria for learning algorithms in multi-agent systems, one that is more stringent and (we argue) better justified than previous proposed criteria. Our criteria, which apply most straightfor- wardly in repeated games with average rewards, consist of three require- ments: (a) against a specified class of opponents (this class is a parameter of the criterion) the algorithm yield a payoff that approaches the payoff of the best response, (b) against other opponents the algorithm's payoff at least approach (and possibly exceed) the security level payoff (or max- imin value), and (c) subject to these requirements, the algorithm achieve a close to optimal payoff in self-play. We furthermore require that these average payoffs be achieved quickly. We then present a novel algorithm, and show that it meets these new criteria for a particular parameter class, the class of stationary opponents. Finally, we show that the algorithm is effective not only in theory, but also empirically. Using a recently introduced comprehensive game theoretic test suite, we show that the algorithm almost universally outperforms previous learning algorithms.

No abstract available.

No abstract available.

Motivated by the rise of online auctions and their relative lack of security, this paper analyzes two forms of cheating in sealed-bid auctions. The first type of cheating we consider occurs when the seller spies on the bids of a second-price auction and then inserts a fake bid in order to increase the payment of the winning bidder. In the second type, a bidder cheats in a first-price auction by examining the competing bids before deciding on his own bid. In both cases, we derive equilibrium strategies when bidders are aware of the possibility of cheating. These results provide insights into sealed-bid auctions even in the absence of cheating, including some counterintuitive results on the effects of overbidding in a first-price auction.footnotetext[1]This work was supported in part by DARPA grant F30602-00-2-0598.

We generalize the framework of non-cooperative computation (NCC), recently introduced by Shoham and Tennenholtz, to apply to cryptographic situations. We consider functions whose inputs are held by separate, self-interested agents. We consider four components of each agent's utility function: (a) the wish to know the correct value of the function, (b) the wish to prevent others from knowing it, (c) the wish to prevent others from knowing one's own private input, and (d) the wish to know other agents' private inputs. We provide an exhaustive game theoretic analysis of all 24 possible lexicographic orderings among these four considerations, for the case of Boolean functions (mercifully, these 24 cases collapse to four). In each case we identify the class of functions for which there exists an incentive-compatible mechanism for computing the function. In this article we only consider the situation in which the inputs of different agents are probabilistically independent.

Some important classical mechanisms considered in Microeconomics and Game Theory require the solution of a difficult optimization problem. This is true of mechanisms for combinatorial auctions, which have in recent years assumed practical importance, and in particular of the gold standard for combinatorial auctions, the Generalized Vickrey Auction (GVA). Traditional analysis of these mechanisms---in particular, their truth revelation properties---assumes that the optimization problems are solved precisely. In reality, these optimization problems can usually be solved only in an approximate fashion. We investigate the impact on such mechanisms of replacing exact solutions by approximate ones. Specifically, we look at a particular greedy optimization method. We show that the GVA payment scheme does not provide for a truth revealing mechanism. We introduce another scheme that does guarantee truthfulness for a restricted class of players. We demonstrate the latter property by identifying natural properties for combinatorial auctions and showing that, for our restricted class of players, they imply that truthful strategies are dominant. Those properties have applicability beyond the specific auction studied.

All in-text\treferences\tunderlined\tin\tblue\tare\tlinked\tto\tpublications\ton\tResearchGate, letting you\taccess\tand\tread\tthem\timmediately.

Article Bidding clubs: institutionalized collusion in auctions Share on Authors: Kevin Leyton-Brown Dept. of Computer Science, Stanford University, Stanford, CA Dept. of Computer Science, Stanford University, Stanford, CAView Profile , Yoav Shoham Dept. of Computer Science, Stanford University, Stanford, CA Dept. of Computer Science, Stanford University, Stanford, CAView Profile , Moshe Tennenholtz Dept. of Computer Science, Stanford University, Stanford, CA Dept. of Computer Science, Stanford University, Stanford, CAView Profile Authors Info & Claims EC '00: Proceedings of the 2nd ACM conference on Electronic commerceOctober 2000 Pages 253–259https://doi.org/10.1145/352871.352899Online:17 October 2000Publication History 24citation256DownloadsMetricsTotal Citations24Total Downloads256Last 12 Months2Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

Some important classical mechanisms considered in Microeconomics and Game Theory require the solution of a difficult optimization problem.This is true of mechanisms for combinatorial auctions, which have in recent years assumed practical importance, and in particular of the gold standard for combinatorial auctions, the Generalized Vickrey Auction (GVA).Traditional analysis of these mechanisms -in particular, their truth revelation properties -assumes that the optimization problems are solved precisely.In reality, these optimization problems can usually be solved only in an approximate fashion.We investigate the impact on such mechanisms of replacing exact solutions by approximate ones.Specifically, we look at a particular greedy optimization method, which has empirically been shown to perform well.We show that the GVA payment scheme does not provide for a truth revealing mechanism.We introduce another scheme that does guarantee truthfulness for a restricted class of players.We demonstrate the latter property by identifying sufficient conditions for a combinatorial auction to be truth-revealing, conditions which have applicability beyond the specific auction studied here.

Inspired by the success of the distributed computing community in apply logics of knowledge and time to reasoning about distributed protocols, we aim for a similarly powerful and high-level abstraction when reasoning about control problems involving uncertainty. This paper concentrates on robot motion planning with uncertainty in both control and sensing, a problem that has already been well studied within the robotics community. First, a new and natural problem in this domain is defined: does there exists a sound and complete termination condition for a motion, given initial and goal locations? If yes, how to construct it? Then we define a high-level language, a logic of time and knowledge, which we use to reason about termination conditions and to state general conditions for the existence of sound and complete termination conditions in a broad domain. Finally, we show that sound termination conditions that are optimal in a precise sense provide a natural example of knowledge-based programs with multiple implementations.

We are concerned with the utility of social laws in a computational environment, laws which guarantee the successful coexistence of multiple programs and programmers. In this paper we are interested in the off-line design of social laws, where we as designers must decide ahead of time on useful social laws. In the first part of this paper we suggest the use of social laws in the domain of mobile robots, and prove analytic results about the usefulness of this approach in that setting. In the second part of this paper we present a general model of social law in a computational system, and investigate some of its properties. This includes a definition of the basic computational problem involved with the design of multi-agent systems, and an investigation of the automatic synthesis of useful social laws in the framework of a model which refers explicitly to social laws.

We investigate logical formalization of the effects of actions in the situation calculus. We propose a formal criterion against which to evaluate theories of deterministic actions. We show how the criterion provides us a formal foundation upon which to tackle the frame problem, as well as its variant in the context of concurrent actions. Our main technical contributions are in formulating a wide class of monotonic causal theories that satisfy the criterion, and showing that each such theory can be reformulated succinctly in circumscription.

We study the process of multi-agent reinforcement learning in the context ofload balancing in a distributed system, without use of either centralcoordination or explicit communication. We first define a precise frameworkin which to study adaptive load balancing, important features of which are itsstochastic nature and the purely local information available to individualagents. Given this framework, we show illuminating results on the interplaybetween basic adaptive behavior parameters and their effect on systemefficiency. We then investigate the properties of adaptive load balancing inheterogeneous populations, and address the issue of exploration vs.exploitation in that context. Finally, we show that naive use ofcommunication may not improve, and might even harm system efficiency.

Belief revision and belief update have been considered as two distinct types of belief modification. In this paper, we show that both can be captured within a unified formalism, in which revision is encoded as update of the mental state. Specifically, we show that the framework for belief update developed by the authors can be used to capture revision as well, by extending the underlying language to include explicit representations of the agent's beliefs. The resulting framework allows for the definition of a wide variety of revision operators. These include, in particular, all the AGM operators, for which we provide a representation theorem.

A new computational framework is presented, called agent-oriented programming (AOP), which can be viewed as a specialization of object-oriented programming. The state of an agent consists of components such as beliefs, decisions, capabilities, and obligations; for this reason the state of an agent is called its mental state. The mental state of agents is described formally in an extension of standard epistemic logics: beside temporalizing the knowledge and belief operators, AOP introduces operators for obligation, decision, and capability. Agents are controlled by agent programs, which include primitives for communicating with other agents. In the spirit of speech act theory, each communication primitive is of a certain type: informing, requesting, offering, and so on. This article presents the concept of AOP, discusses the concept of mental state and its formal underpinning, defines a class of agent interpreters, and then describes in detail a specific interpreter that has been implemented.

In this paper we define the logic GK of knowledge and justified assumptions. GK is best understood as a formalization of autoepistemic reasoning processes that are more general than those in Moore's autoepistemic logic, and is formally defined via a modification of Shoham's preference semantics. We show that GK includes not only Moore's autoepistemic logic, but also Reiter's default logic. To our knowledge GK is the first complete semantic unification of the two logics. Similarly to circumscription, GK is based on the notion of logical minimization, and thus provides a bridge between circumscription and fixed-point nonmonotonic logics, an outstanding problem in nonmonotonic logics. As an application of this bridge, we propose a formalization of logic programs with negation-as-failure in circumscription.

As part of our work on agent-oriented programming,1 we are developing an agent description language. We describe an agent's “mental state” in terms of its knowledge, beliefs, desires, commitments, and abilities. Our goal is not to specify human knowledge, beliefs, etc., but rather to devise a more limited, precise language to facilitate programming agents and interagent communication. We believe that borrowing from our intuition about these commonsense terms will simplify the human designer's job. This article describes a preliminary version of our language. We define a temporal language with modal operators representing knowledge, belief, desire, commitment, and ability. This is a propositional language with quantification over time points and agents only.

: In certain areas of artificial intelligence there is need to represent continuous change and to make statements that are interpreted with respect to time intervals rather than time points. To this end we develop a modal temporal logic based on time intervals, a logic which can be viewed as a generalization of pointbased modal temporal logic. We discuss related logics, give an intuitive presentation of the new logic, and define its formal syntax and semantics. We make no assumption about the underlying nature of time, allowing it to be discrete (such as the natural numbers) or continuous (such as the rationals or the reals), linear or branching, complete (such as the reals) or not (such as the rationals). We show, however, that there are formulas in the logic that allow us to distinguish all these situations. We also give a translation of our logic into first-order logic, which allows us to apply some results on first-order logic to our modal one. Finally, we consider the difficulty o...

We offer solutions to two problems in formal temporal reasoning, the qualification problem and the extended prediction problem, which, as argued in [22], subsume the infamous frame problem. The solutions make use of nonmonotonic logics. In particular, we advocate the use of a new nonmonotonic logic, the logic of chronological ignorance. On the practical level, we demonstrate the utility of the logic in specific cases. On the more theoretical level, we identify classes of theories in the logic that have nice properties: they each have a model that is (in a certain sense) unique, and those models can be computed very easily. The first class, causal theories, is sufficiently expressive to solve the qualification problem. It also suggests a new account of the intuitive notion of causation, although we do not discuss that in depth. This class is then generalized to the class of inertial theories, which are sufficiently expressive to solve also the extended prediction problem. Inertial theories embody the concept of potential histories, which are the way the world tends to behave “naturally.”

One way to represent temporal information in a logical formalism is by associating “proposition types” with time points or time intervals. The way this is usually done in AI is by “reifying” propositions, so that what otherwise would have been formulas actually appear as arguments to some “predicate,” say TRUE, as in TRUE(t1, t2, COLOR(HOUSE, RED)). This way time is referred to explicitly, while retaining its special notational and conceptual status. We examine this method by looking closely at two of the more influential formalisms featuring reified propositions, those of Allen and McDermott. We show that these do not have completely clear semantics, and that they make some unfortunate and unnecessary ontological commitments. Finally, we present a new formalism and demonstrate that it does not suffer from these disadvantages.