We present CyBERT, a domain-specific Bidirectional Encoder Representations from Transformers (BERT) model, fine-tuned with a large corpus of textual cybersecurity data. State-of-the-art natural language models that can process dense, fine-grained textual threat, attack, and vulnerability information can provide numerous benefits to the cybersecurity community. The primary contribution of this paper is providing the security community with an initial fine-tuned BERT model that can perform a variety of cybersecurity-specific downstream tasks with high accuracy and efficient use of resources. We create a cybersecurity corpus from open-source unstructured and semi-unstructured Cyber Threat Intelligence (CTI) data and use it to fine-tune a base BERT model with Masked Language Modeling (MLM) to recognize specialized cybersecurity entities. We evaluate the model using various downstream tasks that can benefit modern Security Operations Centers (SOCs). The fine-tuned CyBERT model outperforms the base BERT model in the domain-specific MLM evaluation. We also provide use-cases of CyBERT application in cybersecurity based downstream tasks.

Articles about cybersecurity events like data breaches and ransomware attacks are common, both in general news and technical sources. Automatically extracting structured information from these can provide valuable information to inform both human analysts and computer systems. In this paper we describe how cybersecurity events can be described via semantic schemas, examined through an initial set of five event types. Using a collection of 1,000 news articles annotated with these event types, including their semantic roles, arguments, realis, and coreference, we detail a modular, deep-learning based information extraction (IE) pipeline, which extracts useful event information with high accuracy. We argue that the event argument set considered here can support many other cybersecurity events, facilitating the extension to new cybersecurity event types, such as distributed denial of service and SQL injection attacks.

Current language understanding approaches focus on small documents, such as newswire articles, blog posts, product reviews and discussion forum discussions. Understanding and extracting information from large documents like legal briefs, proposals, technical manuals and research articles is still a challenging task. We describe a framework that can analyze a large document and help people to locate desired information in it. We aim to automatically identify and classify different sections of documents and understand their purpose within the document. A key contribution of our research is modeling and extracting the logical structure of electronic documents using machine learning techniques, including deep learning. We also make available a dataset of information about a collection of scholarly articles from the arXiv eprints collection that includes a wide range of metadata for each article, including a table of contents, section labels, section summarizations and more. We hope that this dataset will be a useful resource for the machine learning and language understanding communities for information retrieval, content-based question answering and language modeling tasks.

We describe an approach using dynamic topic modeling to model influence and predict future trends in a scientific discipline. Our study focuses on climate change and uses assessment reports of the Intergovernmental Panel on Climate Change (IPCC) and the papers they cite. Since 1990, an IPCC report has been published every five years that includes four separate volumes, each of which has many chapters. Each report cites tens of thousands of research papers, which comprise a correlated dataset of temporally grounded documents. We use a custom dynamic topic modeling algorithm to generate topics for both datasets and apply cross-domain analytics to identify the correlations between the IPCC chapters and their cited documents. The approach reveals both the influence of the cited research on the reports and how previous research citations have evolved over time. For the IPCC use case, the report topic model used 410 documents and a vocabulary of 5911 terms while the citations topic model was based on 200K research papers and a vocabulary more than 25K terms. We show that our approach can predict the importance of its extracted topics on future IPCC assessments through the use of cross domain correlations, Jensen-Shannon divergences and cluster analytics.

In order to secure vital personal and organizational system we require timely intelligence on cybersecurity threats and vulnerabilities. Intelligence about these threats is generally available in both overt and covert sources like the National Vulnerability Database, CERT alerts, blog posts, social media, and dark web resources. Intelligence updates about cybersecurity can be viewed as temporal events that a security analyst must keep up with so as to secure a computer system. We describe CyberTwitter, a system to discover and analyze cybersecurity intelligence on Twitter and serve as a OSINT (Open-source intelligence) source. We analyze real time information updates, in form of tweets, to extract intelligence about various possible threats. We use the Semantic Web RDF to represent the intelligence gathered and SWRL rules to reason over extracted intelligence to issue alerts for security analysts.

Ensuring privacy of Big Data managed on the cloud is critical to ensure consumer confidence. Cloud providers publish privacy policy documents outlining the steps they take to ensure data and consumer privacy. These documents are available as large text documents that require manual effort and time to track and manage. We have developed a semantically rich ontology to describe the privacy policy documents and built a database of several policy documents as instances of this ontology. We next extracted rules from these policy documents based on deontic logic which can be used to automate management of data privacy. In this paper we describe our ontology in detail along with the results of our analysis of privacy policies of prominent cloud services.

Multi-relational data, like knowledge graphs, are generated from multiple data sources by extracting entities and their relationships. We often want to include inferred, implicit or likely relationships that are not explicitly stated, which can be viewed as link-prediction in a graph. Tensor decomposition models have been shown to produce state-of-the-art results in link-prediction tasks. We describe a simple but novel extension to an existing tensor decomposition model to predict missing links using similarity among tensor slices, as opposed to an existing tensor decomposition models which assumes each slice to contribute equally in predicting links. Our extended model performs better than the original tensor decomposition and the non-negative tensor decomposition variant of it in an evaluation on several datasets.

Scientists and casual users need better ways to query RDF databases or Linked Open Data. Using the SPARQL query language requires not only mastering its syntax and semantics but also understanding the RDF data model, the ontology used, and URIs for entities of interest. Natural language query systems are a powerful approach, but current techniques are brittle in addressing the ambiguity and complexity of natural language and require expensive labor to supply the extensive domain knowledge they need. We introduce a compromise in which users give a graphical "skeleton" for a query and annotates it with freely chosen words, phrases and entity names. We describe a framework for interpreting these "schema-agnostic queries" over open domain RDF data that automatically translates them to SPARQL queries. The framework uses semantic textual similarity to find mapping candidates and uses statistical approaches to learn domain knowledge for disambiguation, thus avoiding expensive human efforts required by natural language interface systems. We demonstrate the feasibility of the approach with an implementation that performs well in an evaluation on DBpedia data.

Today's scientific investigations are producing large numbers of scholarly products. These products continue to increase in diversity and complexity as researchers recognize that scholarly achievements are not only published articles but also datasets, software, and associated supporting materials. OceanLink is an online platform that addresses scholarly discovery and collaboration in the ocean sciences. The OceanLink project leverages Semantic Web technologies, web mining, and crowdsourcing to identify links between data centers, digital repositories, and professional societies to enhance discovery, enable collaboration, and begin to assess research contribution.

Pointwise mutual information (PMI) is a widely used word similarity measure, but it lacks a clear explanation of how it works. We explore how PMI differs from distributional similarity, and we introduce a novel metric, PMImax, that augments PMI with information about a word's number of senses. The coefficients of PMImaxare determined empirically by maximizing a utility function based on the performance of automatic thesaurus generation. We show that it outperforms traditional PMI in the application of automatic thesaurus generation and in two word similarity benchmark tasks: human similarity ratings and TOEFL synonym questions. PMImaxachieves a correlation coefficient comparable to the best knowledge-based approaches on the Miller-Charles similarity rating data set.

We need better ways to query large linked data collections such as DBpedia. Using the SPARQL query language requires not only mastering its syntax but also understanding the RDF data model, large ontology vocabularies and URIs for denoting entities. Natural language interface systems address the problem, but are still subjects of research. We describe a compromise in which non-experts specify a graphical query "skeleton" and annotate it with freely chosen words, phrases and entity names. The combination reduces ambiguity and allows the generation of an interpretation that can be translated into SPARQL. Key research contributions are the robust methods that combine statistical association and semantic similarity to map user terms to the most appropriate classes and properties in the underlying ontology.

Mobile Ad-hoc Networks (MANETs) are extremely susceptible to various misbehaviors and a variety of trust management schemes have been proposed to detect and mitigate them. Most schemes rely on a set of pre-defined weights to determine how the extent of each misbehavior is used to evaluate the trustworthiness. However, due to the extremely dynamic nature of MANETs, it is not possible to determine a set of weights that are appropriate for all contexts. In this paper, an Automated Trust Management (ATM) system is described for MANETs that uses a support vector machine classifier to detect malicious MANET nodes. The ATM scheme is resilient to attempts by a malicious MANET node to hide its nature by varying its misbehavior patterns over time. The performance of the ATM scheme is evaluated via an extensive simulation study and compared with existing approaches.

Nodes in Mobile Ad hoc Networks (MANETs) are required to relay data packets to enable communication between other nodes that are not in radio range with each other. However, whether for selfish or malicious reasons, a node may fail to cooperate during the network operations or even attempt to disturb them, both of which have been recognized as misbehaviors. Various trust management schemes have been studied to assess the behaviors of nodes so as to detect and mitigate node misbehaviors inMANETs. Most of existing schemes model a node's trustworthiness along a single dimension, combining all of the available evidence to calculate a single, scalar trust metric. A single measure, however, may not be expressive enough to adequately describe a node's trustworthiness in many scenarios. In this paper, we describe a multi-dimensional framework to evaluate the trustworthiness of MANET node from multiple perspectives. Our scheme evaluates trustworthiness from three perspectives: collaboration trust, behavioral trust, and reference trust. Different types of observations are used to independently derive values for these three trust dimensions. We present simulation results that illustrate the effectiveness of the proposed scheme in several scenarios.

The standard method for combating spam, either in email or on the web, is to train a classifier on manually labeled instances. As the spammers change their tactics, the performance of such classifiers tends to decrease over time. Gathering and labeling more data to periodically retrain the classifier is expensive. We present a method based on an ensemble of classifiers that can detect when its performance might be degrading and retrain itself, all without manual intervention. Experiments with a real-world dataset from the blog domain show that our methods can significantly reduce the number of times classifiers are retrained when compared to a fixed retraining schedule, and they maintain classification accuracy even in the absence of manually labeled examples.

We describe an efficient technique to weigh word-based features in binary classification tasks and show that it significantly improves classification accuracy on a range of problems. The most common text classification approach uses a document's ngrams (words and short phrases) as its features and assigns feature values equal to their frequency or TFIDF score relative to the training corpus. Our approach uses values computed as the product of an ngram's document frequency and the difference of its inverse document frequencies in the positive and negative training sets. While this technique is remarkably easy to implement, it gives a statistically significant improvement over the standard bag-of-words approaches using support vector machines on a range of classification tasks. Our results show that our technique is robust and broadly applicable. We provide an analysis of why the approach works and how it can generalize to other domains and problems.

In this article, we demonstrate the applicability of semantic techniques for detection of Conflict of Interest (COI). We explain the common challenges involved in building scalable Semantic Web applications, in particular those addressing connecting-the-dots problems. We describe in detail the challenges involved in two important aspects on building Semantic Web applications, namely, data acquisition and entity disambiguation (or reference reconciliation). We extend upon our previous work where we integrated the collaborative network of a subset of DBLP researchers with persons in a Friend-of-a-Friend social network (FOAF). Our method finds the connections between people, measures collaboration strength, and includes heuristics that use friendship/affiliation information to provide an estimate of potential COI in a peer-review scenario. Evaluations are presented by measuring what could have been the COI between accepted papers in various conference tracks and their respective program committee members. The experimental results demonstrate that scalability can be achieved by using a dataset of over 3 million entities (all bibliographic data from DBLP and a large collection of FOAF documents).

We describe our research on automatically generating rich semantic annotations of text and making it available on the Semantic Web. In particular, we discuss the challenges involved in adapting the OntoSem natural language processing system for this purpose. OntoSem, an implementation of the theory of ontological semantics under continuous development for over 15 years, uses a specially constructed NLP-oriented ontology and an ontological-semantic lexicon to translate English text into a custom ontology- motivated knowledge representation language, the language of text meaning representations (TMRs). OntoSem concentrates on a variety of ambiguity resolution tasks as well as processing unexpected input and reference. To adapt OntoSem’s representation to the Semantic Web, we developed a translation system, OntoSem2OWL, between the TMR language into the Semantic Web language OWL. We next used OntoSem and OntoSem2OWL to support SemNews, an experimental Web service that monitors RSS news sources, processes the summaries of the news stories, and publishes a structured representation of the meaning of the text in the news story.

In this paper, we describe a Semantic Web application that detects Conflict of Interest (COI) relationships among potential reviewers and authors of scientific papers. This application discovers various 'semantic associations' between the reviewers and authors in a populated ontology to determine a degree of Conflict of Interest. This ontology was created by integrating entities and relationships from two social networks, namely "knows," from a FOAF (Friend-of-a-Friend) social network and "co-author," from the underlying co-authorship network of the DBLP bibliography. We describe our experiences developing this application in the context of a class of Semantic Web applications, which have important research and engineering challenges in common. In addition, we present an evaluation of our approach for real-life COI detection.

We describe ELVIS (the Ecosystem Location Visualization and Information System), a suite of tools for constructing food webs for a given location. We express both ELVIS input and output data in OWL, thereby enabling its integration with other semantic web resources. In particular, we describe using a Triple Shop application to answer SPARQL queries from a collection of semantic web documents. This is an end-to-end case study of the semantic web's utility for ecological and environmental research.

Swoogle is a crawler-based indexing and retrieval system for the Semantic Web. It extracts metadata for each discovered document, and computes relations between documents. Discovered documents are also indexed by an information retrieval system which can use either character N-Gram or URIrefs as keywords to find relevant documents and to compute the similarity among a set of documents. One of the interesting properties we compute is ontology rank, a measure of the importance of a Semantic Web document.

The ultimate goal of service matching is to find the service provider(s) that would perform tasks of given description with the best overall degree of satisfaction. However, service description matching solves only part of the problem. Agents that match a given request may vary greatly in their actual capabilities to perform the tasks, and an agent may have strong and weak areas. In this work, we take a quantitative approach in which performance rating is considered an integral part of an agent's capability model and service distribution is taken into account in determining the degree of match. With the dynamic refinement of the agent capability model, the broker captures an agent's performance levels as well as its strong and weak areas. An experimental system has been designed and implemented within the OWL/OWL-S framework and the results statistics show significant advantage over other major levels of brokers.

This paper presents a framework to address new data management challenges introduced by data-intensive, pervasive computing environments. These challenges include a spatio-temporal variation of data and data source availability, lack of a global catalog and schema, and no guarantee of reconnection among peers due to the serendipitous nature of the environment. An important aspect of our solution is to treat devices as semi-autonomous peers guided in their interactions by profiles and context. The profiles are grounded in a semantically rich language and represent information about users, devices and data described in terms of “beliefs”, “desires”, and “intentions”. We present a prototype implementation of this framework over combined Bluetooth and Ad-Hoc 802.11 networks, and present experimental and simulation results that validate our approach and measure system performance.

The semantic Web enables intelligent agents to "outsource" knowledge, extending and enhancing their limited knowledge bases. An open question is how agents can efficiently and effectively access the vast knowledge on the inherently open and dynamic semantic Web. The problem is not that of finding a source for desired information, but deciding which among many possibly inconsistent sources is most reliable. We propose an approach to agent knowledge outsourcing inspired by the use trust in human society. Trust is a type of social knowledge and encodes evaluations about which agents can be taken as reliable sources of information or services. We focus on two important practical issues: learning trust and justifying trust. An agent can learn trust relationships by reasoning about its direct interactions with other agents and about public or private reputation information, i.e., the aggregate trust evaluations of other agents. We use the term trust justification to describe the process in which an agent integrates the beliefs of other agents, trust information, and its own beliefs to update its trust model. We describe the results of simulation experiments of the use and evolution of trust in multiagent systems. Our experiments demonstrate that the use of explicit trust knowledge can significantly improve knowledge outsourcing performance. We also describe a collaborative trust justification technique that focuses on reducing search complexity, handling inconsistent knowledge, and avoiding error propagation.

We describe an approach to retrieval of documents that contain of both free text and semantically enriched markup. In particular, we present the design and implementation prototype of a framework in which both documents and queries can be marked up with statements in the DAML+OIL semantic web language. These statements provide both structured and semi-structured information about the documents and their content. We claim that indexing text and semantic markup together will significantly improve retrieval performance. Our approach allows inferencing to be done over this information at several points: when a document is indexed, when a query is processed and when query results are evaluated.

The pervasive computing environments of the near future will involve the interactions, coordination and cooperation of numerous, casually accessible, and often invisible computing devices. These devices, whether carried on our person or embedded in our homes, businesses and classrooms, will connect via wireless and wired links to one another and to the global networking infrastructure. The result will be a networking milieu with a new level of openness. The localized and dynamic nature of their interactions raises many new issues that draw on and challenge the disciplines of agents, distributed systems, and security. This paper describes recent work by the UMBC Ebiquity research group which addresses some of these issues.

We suggest that one (or a collection) of names of Yahoo! (or any other WWW indexer's) categories can be used to describe the content of a document. Such categories offer a standardized and universal way for referring to or describing the nature of real world objects, activities, documents and so on, and may be used (we suggest) to semantically characterize the content of documents. WWW indices, like Yahoo! provide a huge hierarchy of categories (topics) that touch every aspect of human endeavors. Such topics can be used as descriptors, similarly to the way librarians use for example, the Library of Congress cataloging system to annotate and categorize books.

This paper describes the design of and experimentation with the Knowledge Query and Manipulation Language (KQML), a new language and protocol for exchanging information and knowledge. This work is part of a larger effort, the ARPA Knowledge Sharing Effort which is aimed at developing techniques and methodology for building large-scale knowledge bases which are sharable and reusable. KQML is both a message format and a message-handling protocol to support run-time knowledge sharing among agents. KQML focuses on an extensible set of performatives, which defines the permissible “speech acts” agents may use and comprise a substrate on which to develop higher-level models of interagent interaction such as contract nets and negotiation. In addition, KQML provides a basic architecture for knowledge sharing through a special class of agent called communication facilitors which coordinate the interactions of other agents. The ideas which underlie the evolving design of KQML are currently being explored through experimental prototype systems which are being used to support several testbeds in such areas as concurrent engineering, intelligent design and intelligent planning and scheduling.

We investigate the semantics for Knowledge Query Manipulation Language (KQML) and we propose a semantic framework for the language. KQML is a language and a protocol to support communication between software agents. Based on ideas from speech act theory, we propose a semantic description for KQML that associates descriptions of the cognitive states of agents with the use of the language's primitives (performatives). We use this approach to describe the semantics for the basic set of KQML performatives. We also investigate implementation issues related to our semantic approach. We suggest that KQML can offer an all purpose communication language for software agents that requires no limiting pre-commitments on the agents' structure and implementation. KQML can provide the Distributed AI, Cooperative Distributed Problem Solving and Software Agents communities with an all purpose language and environment for intelligent inter-agent communication.

For interactive systems to communicate in a cooperative manner, they must have knowledge about their users. This article explores the role of user models in such systems, with the goal of identifying when and how user models may be useful in a cooperative interactive system. User models are classified by the types of knowledge they contain, several user modelling characteristics that serve as dimension for an additional classification of user models are presented, and user model representations are discussed. These topics help to characterize the space of user modelling in cooperative interactive systems-addressing how they can be used-but do not fully address when it is appropriate to include a user model in an interactive system. Thus, a set of design considerations for user models is presented, while a final example illustrates how these topics influence the user model for a hypothetical investment consulting system.