Abstract In this study, we explore the concept of anchoring bias in the context of sequential sentiment analysis of review corpora. We introduce a novel approach that involves using a carefully selected, limited group of reviews at the beginning of the annotation process for calibration, aiming to reduce this bias. Through an extensive set of experiments we confirm the existence of sentiment bias and demonstrate that indeed its impact can be moderated through initial calibration. We also demonstrate that the composition of the calibration set is critical, underscoring the importance of establishing sound criteria for selecting these initial reviews. By comparing the accuracy of annotators who utilized our calibration method against those who did not calibrate or used a randomly chosen set for calibration, we found that our method significantly decreases the overall annotation error. Moreover, the guidelines we develop for selecting the calibration set prove to be highly effective and adaptable, even when applied to a domain other than the one they were originally developed for. Acknowledging the overhead of labeling calibration reviews, we demonstrate that this approach is more efficient compared to eliciting multiple sentiment scores per review, a common strategy to reduce Mean Absolute Error (MAE). Our findings reveal that the proposed calibration method significantly reduces resource expenditure, compared to relying on parallel labeling, while maintaining accuracy.

The narratives expressed by patients during psychotherapy sessions provide a cognitive framework through which they process their experiences. While these narratives offer therapists valuable insights, automatically detecting them presents unique challenges due to their unstructured nature, parallel storytelling patterns, and frequent interruptions - characteristics distinct from narrative detection in traditional domains like news or social media. This paper first establishes a formal definition of therapeutic narratives as continuous segments containing characters, actions, and consequent changes, situated in specific temporal–spatial contexts. Working with domain experts, we developed and validated annotation guidelines achieving high inter-annotator agreement on a comprehensive corpus of Hebrew psychotherapy sessions. Building on this foundation, we introduce APhyND, a novel layered framework specifically designed for automatic narrative detection in psychotherapy session transcripts. Our framework makes several key innovations: (1) a clinically-informed architecture incorporating therapist-validated narrative criteria and specialized feature extraction for therapeutic discourse (2) integration of deep contextual understanding through BERT with sequence modeling via Conditional Random Fields, specifically adapted for therapeutic discourse, and (3) the first comprehensive solution for narrative detection in Hebrew psychotherapy transcripts, addressing the challenges of morphologically rich languages in therapeutic contexts. The framework employs a clinically-informed architecture incorporating therapist-validated narrative criteria, specialized feature extraction for therapeutic discourse, and adaptive threshold adjustment to handle inherent class imbalance. To evaluate the framework’s effectiveness, we conducted experiments on a dataset of 38,434 sentences from 79 psychotherapy sessions in Hebrew. Our results demonstrate significant improvements over existing approaches, achieving an f1-score of 0.804, with particularly strong performance in handling interrupted narratives and speaker transitions. The framework’s ability to process full therapy sessions in real-time while maintaining high accuracy makes it particularly valuable for clinical applications, addressing a critical gap in automated psychotherapy analysis tools. • First computational framework for automatic narrative detection in psychotherapy transcripts. • Novel BERT-CRF ensemble with smoothing achieves 0.804 f1-score on therapy transcripts. • First Hebrew narrative detection system with specialized morphological processing. • Validated narrative annotation guidelines for therapy with 0.87 inter-annotator agreement. • Evaluation on 79 therapy sessions (38,434 sentences) shows state-of-the-art performance.

This paper focuses in the inherent anchoring bias present in sequential reviews-sentiment corpora annotation processes. It proposes employing a limited subset of meticulously chosen reviews at the outset of the process, as a means of calibration, effectively mitigating the phenomenon. Through extensive experimentation we validate the phenomenon of sentiment bias in the annotation process and show that its magnitude can be influenced by pre-calibration. Furthermore, we show that the choice of the calibration set matters, hence the need for effective guidelines for choosing the reviews to be included in it. A comparison of annotators performance with the proposed calibration to annotation processes that do not use calibration or use a randomly-picked calibration set, reveals that indeed the calibration set picked is highly effective---it manages to substantially reduce the average absolute error compared to the other cases. Furthermore, the proposed selection guidelines are found to be highly robust in picking an effective calibration set also for domains different than the one based on which these rules were extracted.

Astroturfing is the practice of using a fake online social media (OSM) profile in order to influence public opinion, while giving the impression that the profile belongs to an authentic human user. The proliferation of such profiles in the political OSM scene is becoming more and more common, and as such, poses a threat to democratic processes, especially around election time. We present here a unique technique for exposing astroturfing profiles for what they are. This technique was developed and tested using a large dataset which we collected from Facebook, containing over four million comments, posted within a time span of 15 months, during which three rounds of elections took place in Israel. This political situation provided a rare opportunity to create and research what is-to the best of our knowledge-one of the largest datasets of authentic and fake profiles curated from Facebook to this day. These unique circumstances of three elections held contiguously within a short time span brought forth a set of previously unrecognized features, creating a temporal layer in the feature set, with regard to the date of elections, such as pre-election and post-election, which greatly improved classification results. By creating and selecting the right features, and using various machine learning algorithms, we are able to identify such profiles, with good precision and recall. This setup can serve as a platform for large scale astroturfing profiling in regular situations. Our unique dataset is planned to be made publicly available to the research community.

Abstract Statistical authorship attribution has a long history, culminating in the use of modern machine learning classification methods. Nevertheless, most of this work suffers from the limitation of assuming a small closed set of candidate authors and essentially unlimited training text for each. Real‐life authorship attribution problems, however, typically fall short of this ideal. Thus, following detailed discussion of previous work, three scenarios are considered here for which solutions to the basic attribution problem are inadequate. In the first variant, the profiling problem, there is no candidate set at all; in this case, the challenge is to provide as much demographic or psychological information as possible about the author. In the second variant, the needle‐in‐a‐haystack problem, there are many thousands of candidates for each of whom we might have a very limited writing sample. In the third variant, the verification problem, there is no closed candidate set but there is one suspect; in this case, the challenge is to determine if the suspect is or is not the author. For each variant, it is shown how machine learning methods can be adapted to handle the special challenges of that variant.

In this paper, we use a blog corpus to demonstrate that we can often identify the author of an anonymous text even where there are many thousands of candidate authors. Our approach combines standard information retrieval methods with a text categorization meta-learning scheme that determines when to even venture a guess.

Most research on learning to identify sentiment ignores “neutral” examples, learning only from examples of significant (positive or negative) polarity. We show that it is crucial to use neutral examples in learning polarity for a variety of reasons. Learning from negative and positive examples alone will not permit accurate classification of neutral examples. Moreover, the use of neutral training examples in learning facilitates better distinction between positive and negative examples.

In this paper, we show that stylistic text features can be exploited to determine an anonymous author's native language with high accuracy. Specifically, we first use automatic tools to ascertain frequencies of various stylistic idiosyncrasies in a text. These frequencies then serve as features for support vector machines that learn to classify texts according to author native language.

This paper describes a hybrid statistical and knowledge-based information extraction model, able to extract entities and relations at the sentence level. The model attempts to retain and improve the high accuracy levels of knowledge-based systems while drastically reducing the amount of manual labor by relying on statistics drawn from a training corpus. The implementation of the model, called TEG (Trainable Extraction Grammar), can be adapted to any IE domain by writing a suitable set of rules in a SCFG (Stochastic Context Free Grammar) based extraction language, and training them using an annotated corpus. The system does not contain any purely linguistic components, such as PoS tagger or parser. We demonstrate the performance of the system on several named entity extraction and relation extraction tasks. The experiments show that our hybrid approach outperforms both purely statistical and purely knowledge-based systems, while requiring orders of magnitude less manual rule writing and smaller amount of training data. The improvement in accuracy is slight for named entity extraction task and more pronounced for relation extraction.

In the authorship verification problem, we are given examples of the writing of a single author and are asked to determine if given long texts were or were not written by this author. We present a new learning-based method for adducing the "depth of difference" between two example sets and offer evidence that this method solves the authorship verification problem with very high accuracy. The underlying idea is to test the rate of degradation of the accuracy of learned models as the best features are iteratively dropped from the learning process.

Text-Mining is a growing area of interest within the field of Data Mining and Knowledge Discovery. Given a collection of text documents, most approaches to Text Mining perform knowledgediscovery operations either on external tags associated with each document, or on the set of all words within each document. Both approaches suffer from limitations. This paper focuses on an intermediate approach, one that we call text mining via information extraction, in which knowledge discovery takes place on focused, relevant terms, phrases and facts, as extracted from the documents.

Information extraction is one of the most important techniques used in Text Mining. One of the main problems in building information extraction (IE) systems is that the knowledge elicited from domain experts tends to be only approximately correct. In addition, the knowledge acquisition phase for building IE rules usually takes a tremendous amount of time on the part of the expert and of the linguist creating the rules. We therefore need an effective means of revising our IE rules whenever we discover such an inaccuracy. The IE revision problem is how best to go about revising a deficient IE rules using information contained in examples that expose inaccuracies. The revision process is very sensitive to implicit and explicit biases encoded in the specific revision algorithm employed. In a sense, each revision algorithm must provide two forms of biases: bias as to the place of the revision and bias as to the type of the revision that should be performed. In this paper we present a framework for writing approximate IE rules that are provided with explicit bias. The proposed framework can be used by many existing revision algorithms. The purpose of the revision bias framework is to allow the user to declare his own bias in a simple and structured way, i.e. to express the conditions placed on the domain knowledge for a given revision operator to be applied. This language extends and generalizes the work reported in [Feldman et. al. 1993]. It attacks the problem of writing IE rules from a novel perspective, one which enables a much faster development of IE systems.