As the field progresses toward Artificial General Intelligence (AGI), there is a pressing need for more comprehensive and insightful evaluation frameworks that go beyond aggregate performance metrics. This paper introduces a unified rating system that jointly models the difficulty of individual test cases and the competency of AI models (or humans) across vision, language, and action domains. Unlike existing metrics that focus solely on models, our approach allows for fine-grained, difficulty-aware evaluations through competitive interactions between models and tasks, capturing both the long-tail distribution of real-world challenges and the competency gap between current models and full task mastery. We validate the generalizability and robustness of our system through extensive experiments on multiple established datasets and models across distinct AGI domains. The resulting rating distributions offer novel perspectives and interpretable insights into task difficulty, model progression, and the outstanding challenges that remain on the path to achieving full AGI task mastery. We have made our code and results publicly available at https://ss47816.github.io/AGI-Elo/.

Scheduling is a perennial-and often challenging-problem for many groups. Existing tools are mostly static, showing an identical set of choices to everyone, regardless of the current status of attendees' inputs and preferences. In this paper, we propose Togedule, an adaptive scheduling tool that uses large language models to dynamically adjust the pool of choices and their presentation format. With the initial prototype, we conducted a formative study (N=10) and identified the potential benefits and risks of such an adaptive scheduling tool. Then, after enhancing the system, we conducted two controlled experiments, one each for attendees and organizers (total N=66). For each experiment, we compared scheduling with verbal messages, shared calendars, or Togedule. Results show that Togedule significantly reduces the cognitive load of attendees indicating their availability and improves the speed and quality of the decisions made by organizers.

Understanding the dynamics of human-AI interaction in question answering is crucial for enhancing collaborative efficiency. Extending from our initial formative study, which revealed challenges in human utilization of conversational AI support, we designed two configurations for prompt guidance: a Nudging approach, where the AI suggests potential responses for human agents, and a Highlight strategy, emphasizing crucial parts of reference documents to aid human responses. Through two controlled experiments, the first involving 31 participants and the second involving 106 participants, we compared these configurations against traditional human-only approaches, both with and without AI assistance. Our findings suggest that effective human-AI collaboration can enhance response quality, though merely combining human and AI efforts does not ensure improved outcomes. In particular, the Nudging configuration was shown to help improve the quality of the output when compared to AI alone. This paper delves into the development of these prompt guidance paradigms, offering insights for refining human-AI collaborations in conversational question-answering contexts and contributing to a broader understanding of human perceptions and expectations in AI partnerships.

Virtual academic networking is socio-technically challenging, however, fruitful for researchers' success. We introduce a system called Who2chat to tackle the challenge and facilitate connections of researchers in virtual social hours. Who2chat allows academic researchers to create a research profile and express their research interests, find researchers with similar interests, overcome social barriers, and coordinate and start video chats, all within a single interface. We engaged in an iterative design process by deploying Who2chat at academic conferences. In our preliminary deployment (N=80), we found that researchers often have difficulty finding other researchers who share similar interests, and they are shy about reaching out to other researchers. Inspired by this, we implemented social-signaling features to Who2chat and ran our first deployment (N=220). Our results highlight that the interface allowed users to find relevant researchers and helped them feel confident in joining conversations. However, this led to large group conversations where discussion topics were more superficial. In response, we developed and deployed our second interface (N=81). Key improvements were managing the size of conversations, dynamically determining and allowing individuals to join a conversation based on their relevance to the ongoing discussion, and maintaining the ratio of senior and junior members, to further enhance the quality of discussions. As a result, participants were able to meet more people and engage in more meaningful conversations. Our work demonstrates an interface design for social networking in academic settings and how to lower social barriers in virtual networking.

Even though today’s videoconferencing systems are often very useful, these systems do not provide support for one of the most important aspects of in-person meetings: the ad hoc, private conversations that happen before, after, and during the breaks of scheduled events–the proverbial hallway conversations. Here we describe our design of a simple system, called Minglr, which supports this kind of interaction by facilitating the matching of conversational partners. We describe two studies of this system’s use at two virtual conferences with over 450 total participants. Our results provide evidence for the usefulness of this capability, showing that, for example, 81% of people who used the system successfully thought that future virtual conferences should include a tool with similar functionality. We believe that similar functionality is likely to be widely implemented in many videoconferencing systems and to increase the feasibility and desirability of many kinds of remote work and socializing.

Recent research has demonstrated that (a) groups can be characterized by a collective intelligence (CI) factor that measures their ability to perform together on a wide range of different tasks, and (b) this factor can predict groups' performance on other tasks in the future. The current study examines whether these results translate into the world of teams in competitive online video games where self-organized, time-pressured, and intense collaboration occurs purely online. In this study of teams playing the online game League of Legends, we find that CI does, indeed, predict the competitive performance of teams controlling for the amount of time played as a team. We also find that CI is positively correlated with the presence of a female team member and with the team members' average social perceptiveness. Finally, unlike in prior studies, tacit coordination in this setting plays a larger role than verbal communication.

A key issue, whenever people work together to solve a complex problem, is how to divide the problem into parts done by different people and combine the parts into a solution for the whole problem. This paper presents a novel way of doing this with groups of contests called contest webs. Based on the analogy of supply chains for physical products, the method provides incentives for people to (a) reuse work done by themselves and others, (b) simultaneously explore multiple ways of combining interchangeable parts, and (c) work on parts of the problem where they can contribute the most.

On the Web, there is always a need to aggregate opinions from the crowd (as in posts, social networks, forums, etc.). Different mechanisms have been implemented to capture these opinions such as Like in Facebook, Favorite in Twitter, thumbs-up/-down, flagging, and so on. However, in more contested domains (e.g., Wikipedia, political discussion, and climate change discussion), these mechanisms are not sufficient, since they only deal with each issue independently without considering the relationships between different claims. We can view a set of conflicting arguments as a graph in which the nodes represent arguments and the arcs between these nodes represent the defeat relation. A group of people can then collectively evaluate such graphs. To do this, the group must use a rule to aggregate their individual opinions about the entire argument graph. Here we present the first experimental evaluation of different principles commonly employed by aggregation rules presented in the literature. We use randomized controlled experiments to investigate which principles people consider better at aggregating opinions under different conditions. Our analysis reveals a number of factors, not captured by traditional formal models, that play an important role in determining the efficacy of aggregation. These results help bring formal models of argumentation closer to real-world application.

Collective intelligence (CI) is a property of groups that emerges from the coordination and collaboration of members and predicts group performance on a wide range of tasks. Previous studies of CI have been conducted with lab-based groups in the USA. We introduce a new standardized online battery to measure CI and demonstrate consistent emergence of a CI factor across three different studies despite broad differences in (a) communication media (face-to-face vs online), (b) group contexts (short-term ad hoc groups vs long-term groups) and (c) cultural settings (US, Germany, and Japan). In two of the studies, we also show that CI is correlated with a group's performance on more complex tasks. Consequently, the CI metric provides a generalizable performance measure for groups that is robust to broad changes in media, context, and culture, making it useful for testing the effects of general-purpose collaboration technologies intended to improve group performance.

This article describes a series of tests of the generality of a “radically tailorable” tool for cooperative work. Users of this system can create applications by combining and modifying four kinds of building blocks: objects, views, agents, and links . We found that user-level tailoring of these primitives can provide most of the functionality found in well-known cooperative work systems such as gIBIS, Coordinator, Lotus Notes, and Information Lens. These primitives, therefore, appear to provide an elementary “tailoring language” out of which a wide variety of integrated information management and collaboration applications can be constructed by end users.

Abstract

Article Experiments with Oval: a radically tailorable tool for cooperative work Share on Authors: Thomas W. Malone View Profile , Kum-Yew Lai View Profile , Christopher Fry View Profile Authors Info & Claims CSCW '92: Proceedings of the 1992 ACM conference on Computer-supported cooperative workDecember 1992 Pages 289–297https://doi.org/10.1145/143457.143532Online:01 December 1992Publication History 87citation412DownloadsMetricsTotal Citations87Total Downloads412Last 12 Months3Last 6 weeks0 Get Citation AlertsNew Citation Alert added!This alert has been successfully added and will be sent to:You will be notified whenever a record that you have chosen has been cited.To manage your alert preferences, click on the button below.Manage my AlertsNew Citation Alert!Please log in to your account Save to BinderSave to BinderCreate a New BinderNameCancelCreateExport CitationPublisher SiteGet Access

No abstract available.

"Appears in Proceedings of the Conference on Computer Supported Cooperative Work, Los Angeles, California, October, 1990."

Many computer systems are based on various types of messages, forms, or other objects. When users of such systems need to communicate with people who use different object types, some kind of translation is necessary. In this paper, we explore the space of general solutions to this translation problem and propose a scheme that synthesizes these solutions. After first illustrating the problem in the Object Lens system, we identify two partly conflicting objectives that any translation scheme should satisfy: preservation of meaning and autonomous evolution of group languages. Then we partition the space of possible solutions to this problem in terms of the set theoretic relations between group languages and a common language. This leads to five primary solution classes and we illustrate and evaluate each one. Finally, we describe a composite scheme, called Partially Shared Views, that combines many of the best features of the other schemes. A key insight of the analysis is that partially shared type hierarchies allow “foreign” object types to be automatically translated into their nearest common “ancestor” types. The partial interoperability attained in this way makes possible flexible standards from which people can benefit from whatever agreements they do have without having to agree on everything. Even though our examples deal primarily with extensions to the Object Lens system, the analysis also suggests how other kinds of systems, such as EDI applications, might exploit specialization hierarchies of object types to simplify the translation problem.

The Information Lens provides electronic mail users with the ability to write rules that automatically sort, select, and filter their messages. This paper describes preliminary results from an eighteen-month investigation of the use of this system at a corporate test site. We report the experiences of 13 voluntary users who have each had at least three months experience with the most recent version of the system. We found that:

The costs and capabilities of information technology are improving by orders of magnitude every decade, but we are only beginning to understand the opportunities these changes provide for new ways of organizing human activity and new kinds of technology to help people work together. The new MIT Center for Coordination Science conducts multidisciplinary research to help understand these possibilities better. Research in the center draws upon many parts of MIT and ongoing projects in a variety of fields, including: computer science, organization theory, psychology, information systems, management science, and economics.

No abstract available.

Object Lens allows unsophisticated computer users to create their own cooperative work applications using a set of simple, but powerful, building blocks. By defining and modifying templates for various semistructured objects, users represent many different kinds of information. By creating semiautonomous agents, users specify rules for automatically processing this information in different situations.

Object Lens allows unsophisticated computer users to create their own cooperative work applications using a set of simple, but powerful, building blocks. By defining and modifying templates for various semistructured objects, users can represent information about people, tasks, products, messages, and many other kinds of information in a form that can be processed intelligently by both people and their computers. By collecting these objects in customizable folders, users can create their own displays which summarize selected information from the objects in table or tree formats. Finally, by creating semiautonomous agents, users can specify rules for automatically processing this information in different ways at different times. The combination of these primitives provides a single consistent interface that integrates facilities for object-oriented databases, hypertext, electronic messaging, and rule-based intelligent agents. To illustrate the power of this combined approach, we describe several simple examples of applications (such as task tracking, intelligent message routing, and database retrieval) that we have developed in this framework.

No abstract available.

The purpose of this symposium is three-fold:

Many researchers have investigated and speculated about the link between information technology and organizational structure with very mixed results. This article suggests that part of the reason for these mixed results is the coarseness of previous analyses of both technology and structure. It describes a new and much more detailed perspective for investigating this link. Using concepts of object-oriented programming from artificial intelligence, the information processing that occurs in organizations is characterized in terms of the kinds of messages people exchange and the ways they process those messages. The utility of this approach is demonstrated through the analysis of a case in which a reduction in levels of management is coupled with the introduction of a computer conferencing system. The detailed model developed for this case helps explain both macro-level data about the changes in the organizational structure, and micro-level data about individuals' use of the system.

This paper argues that using a set of semistructured message templates is surprisingly helpful in designing a variety of computer-based communication and coordination systems. Semistructured messages can help provide automatic aids for (1) composing messages to be sent, (2) selecting, sorting, and prioritizing messages that are received, (3) responding automatically to some messages, and (4) suggesting likely responses to other messages. The use of these capabilities is illustrated in a range of applications including electronic mail, computer conferencing, calendar management, and task tracking. The applications show how ideas from artificial intelligence (such as inheritance and production rules) and ideas from user interface design (such as interactive graphical editors) can be combined in novel ways for dealing with semistructured messages. The final part of the paper discusses how communities can evolve a useful set of message type definitions.

Many researchers have investigated and speculated about the link between information technology and organizational structure with very mixed results. This paper suggests that part of the reason for these mixed results is the coarseness of previous analyses of both technology and structure. The paper describes a new and much more detailed perspective for investigating this link. Using concepts of object-oriented programming from artificial intelligence, the information processing that occurs in organizations is characterized in terms of the kinds of messages people exchange and the ways they process those messages. The utility of this approach is demonstrated through the analysis of a case in which a reduction in levels of management is coupled with the introduction of a computer conferencing system. The detailed model developed for this case helps explain both macro-level data about thd changes in the organizational structure, and micro-level data about individuals' use of the system.

No abstract available.

This paper argues that using a set of semi-structured message templates is surprisingly helpful in designing a variety of computer-based communication and coordination systems. Semi-structured messages can help provide automatic aids for: (1) composing messages to be sent, (2) selecting, sorting, and prioritizing messages that are received, (3) responding automatically to some messages, and (4) suggesting likely responses to other messages. The use of these capabilities is illustrated in a range of applications including electronic mail, computer conferencing, calendar management, and task tracking. The applications show how ideas from artificial intelligence (such as inheritance and production rules) and ideas from user interface design (such as interactive graphical editors) can be combined in novel ways for dealing with semi-structured messages. The final part of the paper discusses how communities can evolve a useful set of message type definitions.

Research on human factors in computer systems has emphasized supporting individuals. This panel will discuss new issues that emerge when computer systems support groups of people and whole organizations. Malone (see following paper) will suggest a broadening of the definition of user interfaces to include “organizational interfaces” and will indicate how a theoretical base for such an endeavor might be developed. Then Cashman will describe a “coordinator tool” in use at DEC for tracking the assignment of tasks to people in activities such as software maintenance. Finally, Brown will suggest how computer systems can be designed to radically increase the bandwidth of cooperation in groups by, for example, exploiting linguistic notions of context.

This paper argues that it will become increasingly important to extend our concept of user interfaces for individual users of computers to include organizational interfaces for groups of users. A number of suggestions are given for how to develop a theoretical base for designing such interfaces. For instance, examples are used to illustrate how traditional cognitive points of view can be extended to include information processing by multiple agents in organizations. Examples of design implications from other perspectives such as motivational, economic, and political are also included.

This paper describes a series of interviews focusing on the way professional and clerical office workers organize the information in their desks and offices.A number of implications for designing "natural" and convenient computer-based information systems are discussed.Two principal claims are made: (1) A very important function of desk organization is to remind the user of things to do, not just to help the user find desired information.Failing to support this function may seriously impair the usefulness of electronic office systems, and explicitly facilitating it may provide an important advantage for automated office systems over their nonautomated predecessors.(2) The cognitive difficulty of categorizing information is an important factor in explaining how people organize their desks.Computer-based systems may help with this difficulty by (a) doing as much automatic classification as possible (e.g., based on access dates}, and (b) including untitled "piles" of information arranged by physical location as well as explicitly titled and logically arranged "files."Several other implications for the design of electronic office systems are discussed, and some differences in how people organize their desks are described.

In this paper, I will discuss two questions: (1) Why are computer games so captivating? and (2) How can the features that make computer games captivating be used to make other user interfaces interesting and enjoyable to use?

One can't deny the effectiveness of video arcade games in reachipg users! Just loop at the number of quarters pushed into the slots, the time spent by people of widely differing abilities, and the number of repeat encounters with the systems. At least part of the success is due to the ease of getting started (the first play of the game gets one comfortable with the procedures), the high degree of visualization of controls and results, and the responsiveness overall. Other factors will be taken up by the panelists.Review of the home computer market shows what can be accomplished by an easy-to-use accounting aid through advertising store demonstrations, and word of mouth. Visicalc has sold over a million dollars! Attendees will have an opportunity to try some of these impressive applications before and after the session.