The online evaluation of machine learning models is typically conducted through A/B experiments. Sequential statistical tests are valuable tools for analysing these experiments, as they enable researchers to stop data collection early without increasing the risk of false discoveries. However, existing sequential tests either limit the number of interim analyses or suffer from low statistical power. In this paper, we introduce a novel sequential test designed for the continuous monitoring of A/B experiments. We validate our method using semi-synthetic simulations and demonstrate that it outperforms current state-of-the-art sequential testing approaches. Our method is derived using a new technique that inverts a bound on the probability of threshold crossing, based on a classical maximal inequality.

Integrated development environments (IDEs) are complex applications that integrate multiple tools for creating and manipulating software project artifacts. To improve users' knowledge and the effectiveness of usage of the available functionality, the inclusion of recommender systems into IDEs has been proposed. We present a novel IDE command recommendation algorithm that, by taking into account the contexts in which a developer works and in which different commands are usually executed, is able to provide relevant recommendations. We performed an empirical comparison of the proposed algorithm with state-of-the-art IDE command recommenders on a real-world data set. The algorithms were evaluated in terms of precision, recall, F1, k-tail, and with a new evaluation metric that is specifically measuring the usefulness of contextual recommendations. The experiments revealed that in terms of the contextual relevance and usefulness of recommendations the proposed algorithm outperforms existing algorithms.

Many collaborative filtering recommender systems collect and use users' explicitly entered preferences in the form of ratings for items. However, in many real world scenarios, this form of feedback can be difficult to obtain or unavailable (e.g., news portals). In this case recommendations must be built by leveraging more abundant implicit feedback data, which only indirectly signal users' preferences or opinions. A record in such datasets is a result of an action performed by a user on an item (e.g., the item was clicked or viewed). State-of-the-art implicit feedback recommender systems predict whether the user will act on a target item and interpret this prediction as a discovered preference for the item. These models are trained by observations of user actions of one single type. For instance, they predict that a user will watch a video using a dataset of observed video watch actions. In this paper we conjecture that multiple types of user actions may be jointly exploited to predict one target type of actions. We present a general prediction model (MMF - Multiple action types Matrix Factorization) that implements this conjecture and we illustrate some practical examples. The empirical evaluation of MMF, which was conducted on a large real world dataset, shows that using multiple actions is beneficial and it can outperform a state-of-the-art implicit feedback model that uses only the target action data.