To address the variety and veracity issues of big data, data dependencies have been extended as data quality rules to adapt to various data types, ranging from (1) categorical data with equality relationships to (2) heterogeneous data with similarity relationships, and (3) numerical data with order relationships. In this survey, we briefly review the recent proposals on data dependencies categorized into the aforesaid types of data. In addition to (a) the concepts of these data dependency notations, we investigate (b) the extension relationships between data dependencies. It forms a family tree of extensions, mostly rooted in FDs. Moreover, we summarize (c) the discovery of dependencies from data, and (d) the applications of the extended data dependencies. Finally, we conclude with several directions of future studies on the emerging data.

Event data are often dirty owing to various recording conventions or simply system errors. These errors may cause serious damage to real applications, such as inaccurate provenance answers, poor profiling results, or concealing interesting patterns from event data. Cleaning dirty event data is strongly demanded. While existing event data cleaning techniques view event logs as sequences, structural information does exist among events, such as the task passing relationships between staffs in workflow or the invocation relationships among different micro-services in monitoring application performance. We argue that such structural information enhances not only the accuracy of repairing inconsistent events but also the computation efficiency. It is notable that both the structure and the names (labeling) of events could be inconsistent. In real applications, while an unsound structure is not repaired automatically (which requires manual effort from business actors to handle the structure error), it is highly desirable to repair the inconsistent event names introduced by recording mistakes. In this article, we first prove that the inconsistent label repairing problem is NP-complete. Then, we propose a graph repair approach for (1) detecting unsound structures, and (2) repairing inconsistent event names. Efficient pruning techniques together with two heuristic solutions are also presented. Extensive experiments over real and synthetic datasets demonstrate both the effectiveness and efficiency of our proposal.

Besides the conventional schema-oriented tasks, data dependencies are recently revisited for data quality applications, such as violation detection, data repairing and record matching. To address the variety and veracity issues of big data, data dependencies have been extended as data quality rules to adapt to various data types, ranging from (1) categorical data with equality relationships to (2) heterogeneous data with similarity relationships, and (3) numerical data with order relationships. In this survey, we briefly review the recent proposals on data dependencies categorized into the aforesaid types of data. In addition to (a) the concepts of these data dependency notations, we investigate (b) the extension relationships between data dependencies, e.g., conditional functional dependencies (CFDs) extend the conventional functional dependencies (FDs). It forms a family tree of extensions, mostly rooted in FDs, helping us understand the expressive power of various data dependencies. Moreover, we summarize (c) the discovery of dependencies from data, since data dependencies are often unlikely to be manually specified in a traditional way, given the huge volume and high variety of big data. We further outline (d) the applications of the extended data dependencies, in particular in data quality practice. It guides users to select proper data dependencies with sufficient expressive power and reasonable discovery cost. Finally, we conclude with several directions of future studies on the emerging data.

Multiscale design is the widely practiced use of space and scale to visually explore and articulate relationships. Free-form web curation (FFWC) is an approach to supporting multiscale design, involving creative strategies of collecting content, assembling it to juxtapose and organize, sketching, writing, shifting perspective to navigate, and exhibiting to share and collaborate. Our long term goal is to support design students with automatic, on demand feedback.

Clustering is often distracted by errors, frequently observed in almost all areas, ranging from online questionnaire to sensor reading in IoT. The dirty data values not only make themselves (the corresponding tuples) outlying, but also mislead the clustering of remaining tuples, e.g., mistakenly splitting a cluster into two or distorting the cluster center. The reason is that the traditional clustering methods either simply ignore the outliers such as DBSCAN or assign them to the closest clusters anyway, e.g., in K-Means. In this paper, we propose to save the outliers for better clustering. The idea is to adjust the erroneous values (often minimally) of the outlier in order to make it appear normally. That is, the tuples after adjusting values are no longer outlying, and thus will be clustered without distracting others. The outlier saving by value adjustment is designed to work with any clustering methods (e.g., DBSCAN or K-Means). Our technical contributions include: (1) showing NPhardness of the outlier saving problem for clustering, (2) deriving lower and upper bounds of the optimal solutions, and (3) devising approximation algorithm with performance guarantees referring to the aforesaid bounds. Experiments on datasets with real-world outliers demonstrate the higher accuracy of our proposal, compared to the state-of-the-art approaches. Remarkably, we show that the adjusted data with outlier saving indeed improve significantly clustering, as well as other applications such as classification and record matching.

Queries over event data are posed in a form of event patterns, for example, to retrieve the flights from IAH to LGA without a stopover. If the expected answer is not returned, one may ask why not, also known as explanations of non-answers. Analogous to the relational data, the explanations over event data lie in two aspects. (1) The pattern consistency explanation indicates that the patterns specified in the query are wrong (inconsistent), that is, there exists no tuple of events that can match the query. (2) The timestamp modification explanation speculates that the instance of event tuple is incorrect, for example, the timestamps of some events are imprecise and need modification. To the best of our knowledge, this is the first study on explaining non-answers over event data. We prove that both explanation problems are NP-complete. By encoding event patterns as a novel notation, we identify the special cases that can be efficiently solved or approximated. General cases are addressed by utilizing the solutions of special cases. Extensive experiments over real and synthetic datasets demonstrate both effectiveness and efficiency of our proposal.

Pattern matching is an important task in the field of Complex Event Processing (CEP). However, exact event pattern matching methods could suffer from low hit rate and loss for meaningful events identification due to the heterogeneous and dirty sources in the big data era. Since both events and patterns could be imprecise, the actual event trace may have different event names as well as structures from the pre-defined pattern. The low-quality data even intensifies the difficulty of matching. In this work, we propose to learn embedding representations for patterns and event traces separately and calculate their similarity as the scores for approximate matching.

Online reviews play a critical role in persuading or dissuading users when making purchase decisions. And yet very few users take the time to write helpful reviews. Encouragingly, recent advances in deep neural networks offer good potential to produce review-like natural language content. However, there is a lack of large, high-quality labeled data at both the aspect and sentiment level for training. Hence, toward enabling a writing assistant framework to help users post online reviews, this paper proposes a scalable labeling method for bootstrapping aspect and sentiment labels.Concretely, the proposed approach ? Aspect Dependent Online RE-views (ADORE) - leverages the underlying distribution of reviews and a small seed set of labeled data through carefully designed review segmentation and label assignment. We then show how these labels can inform a generative model to produce aspect and sentiment-aware reviews. We study the effectiveness of ADORE under various scenarios such as how end-users perceive the quality of the labels and aspect-aware generated reviews. Our experiments indicate that the proposed effective labeling process along with a regularized joint generative model lead to high quality reviews with 90% accuracy.

Structured entities are commonly abstracted, such as from XML, RDF or hidden-web databases. Direct retrieval of various structured entities is highly demanded in data lakes, e.g., given a JSON object, to find the XML entities that denote the same real-world object. Existing approaches on evaluating structured entity similarity emphasize too much the structural inconsistency. Indeed, entities from heterogeneous sources could have very distinct structures, owing to various information representation conventions. We argue that the retrieval could be more tolerant to structural differences and focus more on the contents of the entities. In this paper, we first identify the unique challenge of parent-child (containment) relationships among structured entities, which unfortunately prevent the retrieval of proper entities (returning parents or children). To solve the problem, a novel hierarchy smooth function is proposed to combine the term scores in different nodes of a structured entity. Entities sharing the same structure, namely an entity family, are employed to learn the coefficient in aggregating the scores, and thus distinguish/prune the parent or child entities. Remarkably, the proposed method could cooperate with both the bag-of-words (BOW) and word embedding models, successful in retrieving unstructured documents, for querying structured entities. Extensive experiments on real datasets demonstrate that our proposal is effective and efficient.

IoT data with timestamps are often found with outliers, such as GPS trajectories or sensor readings. While existing systems mostly focus on detecting temporal outliers without explanations, a decision maker may be more interested in the cause of the outlier appearance such that subsequent actions would be taken, e.g., cleaning unreliable readings or repairing broken devices. Such outlier detection and explanation are expected to be performed in either offline (batch) or online modes (over streaming IoT data with timestamps). In this work, we present TsOutlier, a new prototype system for detecting outliers with explanations over IoT data. The framework defines uniform profiles to explain the outliers detected by various algorithms, including the outliers with variant time intervals. Both batch and streaming processing are supported in a uniform framework. In particular, by varying the block size, it provides a tradeoff between computing the accurate results and approximating with efficient incremental computation. In this paper, we present several case studies of applying TsOutlier in industry, e.g., how this framework works in detecting outliers over the operation data of Shanghai Subway, and how to get reasonable explanations for the detected outliers in tracking excavators.