Abstract Machine learning (ML) has already fundamentally changed several businesses. More recently, it has also been profoundly impacting the computational science and engineering domains, like geoscience, climate science, and health science. In these domains, users need to perform comprehensive data analyses combining scientific data and ML models to provide for critical requirements, such as reproducibility, model explainability, and experiment data understanding. However, scientific ML is multidisciplinary, heterogeneous, and affected by the physical constraints of the domain, making such analyses even more challenging. In this work, we leverage workflow provenance techniques to build a holistic view to support the lifecycle of scientific ML. We contribute with (i) characterization of the lifecycle and taxonomy for data analyses; (ii) design decisions to build this view, with a W3C PROV compliant data representation and a reference system architecture; and (iii) lessons learned after an evaluation in an Oil & Gas case using an HPC cluster with 393 nodes and 946 GPUs. The experiments show that the decisions enable queries that integrate domain semantics with ML models while keeping low overhead (<1%), high scalability, and an order of magnitude of query acceleration under certain workloads against without our representation.

Properly modelling dynamic information that changes over time still is an open issue. Most modern knowledge bases are unable to represent relationships that are valid only during a given time interval. In this work, we revisit a previous extension to the hyperknowledge framework to deal with temporal facts and propose a temporal query language and engine. We validate our proposal by discussing a qualitative analysis of the modelling of a real-world use case in the Oil & Gas industry.

There has been many efforts in the research community to create reusable learning objects with rich metadata specification. Even with this effort, there is still a gap in the document representation of learning objects and its contents semantics. This gap makes difficult to create individual plans made to explore the learning curve and necessities of each student when consuming digital learning objects. In this paper we present a novel approach based on the hyperknowledge conceptual model to improve the understanding, reasoning and visualization of the professors about the learning process of his or her students. This approach helps professors to make queries that identify student behaviors over time. It also allows recommendation systems to be developed exploiting good user practices and past behavior patterns to suggest better ways for students to browse educational content.

In this paper, we give an overview of the semantic gap problem in multimedia and discuss how machine learning and symbolic AI can be combined to narrow this gap. We describe the gap in terms of a classical architecture for multimedia processing and discuss a structured approach to bridge it. This approach combines machine learning (for mapping signals to objects) and symbolic AI (for linking objects to meanings). Our main goal is to raise awareness and discuss the challenges involved in this structured approach to multimedia understanding, especially in the view of the latest developments in machine learning and symbolic AI.

Finding concepts considering their meaning and semantic relations in a document corpus is an important and challenging task. In this paper, we present our contributions on how to understand unstructured data present in one or multiple documents. Generally, the current literature concentrates efforts in structuring knowledge by identifying semantic entities in the data. In this paper, we test our hypothesis that hyperknowledge specifications are capable of defining rich relations among documents and extracted facts. The main evidence supporting this hypothesis is the fact that hyperknowledge was built on top of hypermedia fundamentals, easing the specification of rich relationships between different multimodal components (i.e. multimedia content and knowledge entities). The key challenge tackled in this paper is how to structure and correlate these components considering their meaning and semantic relations.

Annotation systems provide services that vary from adding simple information for signifying content of interest, to indicate patterns in documents for creating statistical models and applying machine learning techniques. In this paper, we argue that AI mechanisms should be part of the annotation process to collaborate with annotation systems' end users (i.e. annotators) as oppose to be just an outcome from annotators' work. Indeed, current systems do not delve into AI aspects to support the annotation process, lacking features that we argue as essential in annotation systems. That is, collaboration between annotators and AI, collaborative knowledge curation by extracting and structuring knowledge from annotations considering the context of annotation anchors (i.e. area that was selected to create an annotation and comprises the content of interest). To illustrate the gains of this approach, we present HAS, the Hyperknowledge Annotation System. HAS allows one to annotate multimedia content (e.g., text, image, and video) and its tight integration with AI-based services enables the extraction of additional semantic information from the annotated content. In our approach, both the annotation and the information extracted from the content are structured using the hyperknowledge conceptual model, which promotes the use of the spatiotemporal query capabilities of this model for retrieving annotations based on semantic queries. We argue that integrating AI-based services and using the hyperknowledge model for knowledge structuring leverage multimedia annotation systems, enabling the development of novel use cases.

The meaning behind a multimedia content usually vary according to its fragments, composed by spatial coordinates or intervals of time. The description of these concepts and entities in function of these multimedia fragments (anchors) is what we argue that can fulfill the multimedia meaning (semantic) description. In this paper we propose a multimedia search and retrieval approach based on semantic annotations. More precisely, our approach has two main aspects: knowledge structuring and information retrieval. Users or information extraction services define semantic annotations over anchors of multimedia content, defining their abstract meaning. Under the hood, the system uses the hyperknowledge model for structuring these semantic annotations. For approaching the second aspect, we propose a search algorithm that processes hyperknowledge graphs and retrieves multimedia anchors whose semantics matches with a given query. As proof-of-concept, we present a multimedia annotation system that uses the hyperknowledge model for structuring annotations on multimedia content. We have assessed our approach by carrying out experiments with end users regarding the effectiveness of our search algorithm. The results of this paper indicate that the hyperknowledge model is able to properly represent knowledge information from annotations on multimedia content. And they also indicate that our algorithm works for retrieving multimedia content when using this model.

Most of multimedia documents available today are agnostic to data semantics and their specification language offer little to ease authoring and mechanisms to their players so they can retrieve and present meaningful content to improve user experience. In this paper, we present the main entities of the version 3.1 of the Nested Context Model (NCM), which concentrate efforts at integrating support for enriched knowledge description to the model. This extension enables the specification of relationships between knowledge descriptions in the traditional hypermedia way, composing what we call hyperknowledge in this paper. NCM previous version (NCM 3.0) is a conceptual model for hypermedia document engineering. NCL (Nested Context Language), which is part of international standards and ITU recommendations, is an XML application language that was engineered according to NCM 3.0 definitions. The extensions discussed in this paper contribute not only for advances in the NCL specifications, but mainly as a conceptual model for hyperknowledge document engineering.

Human-Computer Interaction (HCI) area uses qualitative research methods to guide its user experience studies. These methods involve collecting and analyzing large amounts of empirical materials registered in different multimedia contents. Qualitative Data Analysis (QDA) tools support HCI qualitative research activities in some level, but the cognitive-intense work of producing structured and related data, building meaningful knowledge from it, and making decisions based on this knowledge is mainly the HCI researcher's duty. We present a case with Atlas.ti 7, a QDA tool, and in that scenario, we identify and assess key features that support HCI qualitative research. We discuss how those features could evolve to augment human capacity and understanding throughout the qualitative data analysis, considering the cognitive computing approach. We considered multimedia concepts to discuss the evolution of QDA tools as potential investigation paths for research in multimedia and cognitive systems.

Most of multimedia documents available today are agnostic to data semantics and their specification language offer little to ease authoring of meaningful content. In this paper, we present the main entities of the version 3.1 of the Nested Context Model (NCM), which concentrate efforts at integrating support for enriched knowledge description to the model. This extension enables the specification of relationships between knowledge descriptions and multimedia content in the hypermedia way, composing what we call hyperknowledge in this paper. NCM previous version (3.0) is a hypermedia conceptual model. NCL (Nested Context Language), which is part of international standards and ITU recommendations, was engineered according to NCM 3.0 definitions. The extensions discussed in this paper contribute not only for advances in the NCL, but mainly as a conceptual model for hyperknowledge document engineering.

The increasing momentum towards cognitive computing unlocks a diverse set of opportunities and challenges for the multimedia research community. In fact, with a different approach from the one present in the traditional artificial intelligence systems, cognitive computing glimpses a human-machine collaboration, promoting a more natural and symbiotic interaction. The main goal of this paper is to discuss central challenges when the multimedia research area enrolls decision-making processes in the era of cognitive computing. Aiming at supporting the multimedia research community on future investigations in this research area, we present challenges in different topics gathered in specific domains.

Different conceptual data models in the literature are either geared towards the description of low-level features of media content or to more abstract and complex conceptual representations. A lesser number attempt to cover both purposes, i.e. to support both low-level aspects and high-level concepts in a single reasoning process. However, these models lack the needed expressiveness to represent cognitive-intensive activities, which are tightly based on production and consumption of knowledge present in multimedia content and commonly relate concepts at arbitrary abstraction levels. This work aims to contribute to a better understanding of how it could be possible to model the knowledge "flow" in such activities, which may include the collaboration of different cognitive systems (human and software) and interaction with devices, software interfaces, and real objects.

The integration of multimedia frameworks and APIs for accessing cognitive computing systems into a unified programming model may not be straightforward. Current solutions use programming models that lead to problems like callback hell or synchronization of threads. In this paper, we explore the use of the reactive programming paradigm to create multimedia applications that explore cognitive computing systems features. By using CÉU as instance of reactive language, we discuss how this approach promotes a straightforward integration between these domains, while avoiding the aforementioned issues. We discuss the implementation of two case studies to illustrate the advantages of this approach.

This paper presents an architecture for monitoring the presentation of multimedia declarative applications, providing feedback about variables states, object properties, media presentation times, among others. Monitoring tools that follow the proposed architecture are able to detect if visual problems are being caused by programming errors or by player malfunctioning. The architecture presents a communication protocol designed to be independent of the declarative language used in the development of multimedia applications. The main goal is to provide an open and generic architecture that can assist multimedia application authors and presentation engine developers. As an example of the architecture use, the paper also presents a monitoring tool integrated into a graphical user interface developed for the ITU-T reference implementation of the Ginga-NCL middleware.

Multi-head setups are commonly used to enhance productivity and to increase the screen area of multimedia presentations. To support the latter, multimedia systems must have access to information about the display arrangement and about bezels and gaps between displays, to calculate the multi-head total resolution and to define the active exhibition area. Existent solutions oblige users to know the details of the display arrangement, and to map manually the logic display arrangement into the physical one. Even when they provide some features to easy this mapping, users have to pay attention on hardware details, and are limited to predefined display arrangements. In addition, users have to adhere to try-and-error calculations to achieve better results for setting bezel and gap values. This paper presents a mechanism to support multimedia presentations on multi-head setups that cracks the limitations found in the existent systems. The proposed mechanism has been integrated with the reference implementation of Ginga, the ITU-T H.761 IPTV middleware. However, the solution is generic and can be used with other multimedia systems.

This paper discusses how multimedia languages and players can handle multiple time bases in supporting inter-media synchronization with interleaved media contents transmitted as unsolicited streams (pushed data). More specifically, we are interested in inter-media synchronization defined by causal/constraint relationships. Unsolicited interleaved media-contents are usual in digital TV systems, in particular in broadcast TV, in which adverts are inserted into programs transmitted sequentially. Although some existing middleware systems address the problem, their solutions have limitations. This work proposes an alternative approach providing high-level abstractions that hide or minimize the complexity of dealing with interleaved media-content as part of hypermedia applications. The proposal has been incorporated in the Nested Context Language (NCL) and in its player, the main component of the Ginga digital TV middleware. However, despite using NCL as an example, the proposal has a broader spectrum and can be adapted and used in other languages and respective engines.

This paper presents a model for the specification of hierarchical control of focus and input events in hypermedia applications that may contain atomic objects and nested compound objects. A proposal to solve the problems of declarative languages for both Web domain and Digital TV domain is described. The model proposed by us is part of the Digital TV profiles of NCL language specification. The model, however, may be extended to be used in other declarative languages. The model was incorporated into the Ginga reference implementation. NCL and Ginga are part of ITU-T Recommendation for IPTV services and ISDB-T standards (International Standard for Digital Broadcasting — Terrestrial).

SUMMARY Ginga is the middleware of the Japanese‐Brazilian terrestrial digital TV (DTV) system, ISDB‐TB, and also the ITU Telecommunication Standardization Sector (ITU‐T) Recommendation for Internet Protocol television services. Ginga supports NCL (Nested Context Language) applications in its declarative environment (Ginga‐NCL) for which we devise an architecture for plug‐in integration. This paper focuses on this plug‐in architecture and its innovative solutions, detailing the plug‐in API, life‐cycle and control model, besides discussing how the architecture can be extended to other software systems, in particular to other DTV middleware systems. In the paper, we show how Ginga‐NCL can embed other declarative language engines. On the other hand, we also extend an HTML browser to allow it to embed Ginga‐NCL. This proof‐of‐ concept is important because NCL and HTML players are not only the most common players found in DTV middleware systems but also because they are ITU Telecommunication Standardization Sector Recommendations for terrestrial and Internet Protocol television services (NCL and Broadcast Markup Language (BML)/Lightweight Interactive Multimedia Environment (LIME)). Therefore, this proof‐of‐concept also provides a contribution to DTV middleware interoperability. Copyright © 2012 John Wiley & Sons, Ltd.

Nested Context Language (NCL) is the declarative language of the Brazilian Terrestrial Digital TV System. NCL is part of ISDB (International Standard for Digital Broadcasting) standards and also the ITU-T Recommendation H.761 for IPTV services. This paper presents, discusses, and illustrates the NCL hierarchical control model for multiple exhibition device support. Based on this model, multiple devices are orchestrated to run a DTV application in cooperation. Two types of multiple device exhibitions are distinguished. Those where the same content is shown in a set of devices under a unique control, and those where content is under each individual device control, working completely independent. In this last case, depending on viewer interactions, the resulting presented content can differ from a device to another. Examples of NCL applications using both options are presented and discussed.

In some hypermedia system applications, like interactive digital TV applications, authoring and presentation of documents may have to be done concomitantly. This is the case of live programs, where not only some contents are not known a priori, but also some temporal and spatial relationships, among program media objects, may have to be established after the unknown content definition. This paper proposes a method for hypermedia document live editing, preserving not only the presentation semantics but also the logical structure semantics defined by an author. To validate this proposal, an implementation has been done for the Brazilian Digital TV System, which is also presented.