Yakir Yehuda, Itzik Malkiel, Oren Barkan, Jonathan Weill, Royi Ronen, Noam Koenigstein. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2024.

Transcriptions of phone calls are of significant value across diverse fields, such as sales, customer service, healthcare, and law enforcement. Nevertheless, the analysis of these recorded conversations can be an arduous and time-intensive process, especially when dealing with long and multifaceted dialogues. In this work, we propose a novel method, which we name SegLLM, for efficient and accurate call segmentation and topic extraction. SegLLM is composed of offline and online phases. The offline phase is applied once to a given list of topics and involves generating a distribution of synthetic sentences for each topic using a large language model (LLM). The online phase is applied to every call separately and scores the similarity between the transcripted conversation and the topic anchors found in the offline phase. The proposed paradigm provides an accurate and efficient method for call segmentation and topic extraction that does not require labeled data, thus making it a versatile approach applicable to various domains.

We present a research project carried out to enable a Call Categorization Service (CCS) for Dynamics 365 Sales Conversation Intelligence. CCS identifies prevalent types of sales calls based on their transcription, for the purpose of automating manual sales processes and making informed business decisions. We sift through R&D process, and provide clear evidence that purpose-focused fine-tuned transformers out-perform GPT-3 in this text classification task. Additionally we share: an efficient, non-trivial data annotation approach suited to the problem of finding data related to rare categories in a highly unbalanced data source; Considerations regarding zero-shot and in-context learning (i.e. few-shot learning) when using LLMs for classification and cost and performance analysis that opt in favor of fine-tuned transformers as well.

Transcriptions of phone calls hold significant value in sales, customer service, healthcare, law enforcement, and more. However, analyzing recorded conversations can be a time-consuming process, especially for complex dialogues. In Microsoft Dynamics 365 Sales, a novel system, named GPT-Calls, is applied for efficient and accurate topic-based call segmentation. GPT-Calls comprises offline and online phases. In the offline phase, the system leverages a GPT model to generate synthetic sentences and extract anchor vectors for predefined topics. This phase, performed once on a given topic list, significantly reduces the computational burden. The online phase scores the similarity between the transcribed conversation and the topic anchors from the offline phase, followed by time domain analysis to group utterances into segments and tag them with topics. The GPT-Calls scheme offers an accurate and efficient approach to call segmentation and topic extraction, eliminating the need for labeled data. It is a versatile solution applicable to various industry domains. GPT-Calls operates in production under Dynamics 365 Sales Conversation Intelligence, applied to real sales conversations from diverse Dynamics 365 Sales tenants, streamlining call analysis, and saving time and resources while ensuring accuracy and effectiveness.

We developed a generic framework which enables Customer Relationship Management (CRM) organizations to deploy an automated ranking system for leads (commonly known as ‘lead scoring’). Leads are records that represent non-customers who might become customers. Lead ranking is a fundamental CRM problem with many flavors. Ranking serves as a prioritization management tool for CRM organizations, with many characteristics similar to those of recommender systems.

Collaborative filtering (CF) recommendation systems are one of the most popular and successful methods for recommending products to people. CF systems work by finding similarities between different people according to their past purchases, and using these similarities to suggest possible items of interest. In this work we show that CF systems can be enhanced using Internet browsing data and search engine query logs, both of which represent a rich profile of individuals' interests.

Although data quality has been recognized as an important factor in the broad information systems research, it has received little attention in recommender systems. Data quality matters are typically addressed in recommenders by ad-hoc cleansing methods, which prune noisy or unreliable records from the data. However, the setting of the cleansing parameters is often done arbitrarily, without thorough consideration of the data characteristics. In this work, we turn to two central data quality problems in recommender systems: sparsity and redundancy. We devise models for setting data-dependent thresholds and sampling levels, and evaluate these using a collection of public and proprietary datasets. We observe that the models accurately predict data cleansing parameters, while having minor effect on the accuracy of the generated recommendations.

We study the problem of scoring and selecting content-based features for a collaborative filtering (CF) recommender system. Content-based features play a central role in mitigating the ``cold start'' problem in commercial recommenders. They are also useful in other related tasks, such as recommendation explanation and visualization. However, traditional feature selection methods do not generalize well to recommender systems. As a result, commercial systems typically use manually crafted and selected features. This work presents a framework for automated selection of informative content-based features, that is independent of the type of recommender system or the type of features. We evaluate on recommenders from different domains: books, movies and smart-phone apps, and show effective results on each. In addition, we show how to use the proposed methods to generate meaningful features from text.

Project Sage is Microsoft's all-purpose recommender system, designed and deployed as an ultra-high scale cloud service. Sage focuses on both state of the art research and high scale robust implementation. In the research front, we demonstrate new pre-processing and cleaning techniques, a novel probabilistic matrix factorization model for implicit one-class data, and a relatively new evaluation framework. In the engineering front, we present a working service deployed on the Microsoft Azure cloud, which provides easy-to-use interfaces to integrate a recommendation service into any website.

The Query Network [12] is a model for query-based social networks automation features, motivated by the rise of social networks as a central internet application. This work generalizes the model to consist of a proposal query and an acceptance query for each participant. As a result, addition of edges is done by coordination between participants, simulating interactions between participants. We designed, implemented and experimented with evaluation algorithms for this new model. Experiments with both synthetic and real datasets show the high effectiveness of our methods.

We study a novel data management scenario, in which social networks participants use protocols in order to manage their activities and the ever-growing data available to them in the network. In particular, we study protocols which operate on a consistent network (that we define), and transform it into another consistent state by atomically performing a set of changes. Multiple protocol instances, which work on intersecting parts of the network graphs are able to operate concurrently.

We introduce and study concurrent One-Way Protocols in social networks. The model is motivated by the rise of online social networks and the fast development of automation features in them. In a One-Way architecture, used, e.g., by Twitter, participants can publish status updates and send messages only to their followers. Based on this asymmetric model, we define network consistency, and consider the scenario in which participants make consistent decisions based on their friends' decisions, like in Facebook 'Events'.

We consider a near future scenario in which users of a Web 2.0 application, such as a social network, contribute to the application not only data, but also rules which automatically query, utilize and create the data. For example, a user of a social network can define rules that automatically manage the user's friends list, the sending of various announcements, filtering of messages and more.We examine the probable case of automated addition of connections by a participant. The connections to be added are defined using a query, associated to each participant. For this, we introduce and study the Query Network model, a graph-based model in which every node models a network participant and is associated with a Datalog rule. The union of all these individual user rules constitutes a very large, recursive, Datalog program whose size is of the order of magnitude of the size of the data being queried (data whose size in a social network can easily exceed 1TB). This greatly differs from the traditional assumption that queries are small and data are large. In particular, traditional optimizers will be hard pressed to handle such queries. This is the case even if queries are 'translated' to SQL (using views) and their union is transformed to a very large SQL query.We have designed, built and experimented with evaluation algorithms for such query networks. Experiments with both synthetic and real datasets demonstrate the usefulness and high effectiveness of our methods. Extensions to the model are proposed, their implementation and testing are the subject of on-going work.

We present SoQL (social networks query language), a new language for querying and creating data in social networks. The language is designed to meet the growing need of social networks participants to efficiently manage the large, and quickly growing, amounts of data available to them, as well as automate processes of creating new data. This need is increasingly pressing as social networks gradually become an important working tool for business development and management. SoQL is a step in the direction of meeting the challenges of providing an expressive querying mechanism and automating processes in social networks.SoQL is an SQL-like language which enables the user to retrieve paths to other participants in the network, and use a retrieved path in order to attempt to create a connection with the participant at the end of the path. The language can specify complex conditions that a desired path should satisfy. The language also supports retrieving a group of participants which satisfy conditions as a group, and connecting its members to each other. SoQL uses the path and group as data types. This work presents the SoQL language and discusses implementation issues.