The concern that Artificial Intelligence (AI) and Machine Learning (ML) are entering a "reproducibility crisis" has spurred significant research in the past few years. Yet with each paper, it is often unclear what someone means by "reproducibility". Our work attempts to clarify the scope of "reproducibility" as displayed by the community at large. In doing so, we propose to refine the research to eight general topic areas. In this light, we see that each of these areas contains many works that do not advertise themselves as being about "reproducibility", in part because they go back decades before the matter came to broader attention.

Software complexity decreases maintenance productivity, as do team attributes of instability and knowledge diversity. We know little about the extent to which the two team attributes interact with software complexity and shape productivity across systems of varying complexity. We address this gap by investigating whether and to what degree software complexity moderates the effects of team instability and knowledge diversity on maintenance productivity over the life of a system. We posit, given the exponential growth of code and task dependencies inherent in complex software systems, that system-level complexity has a significant nonlinear amplifying effect on the adverse effects of the two team attributes. To validate the presence of such an effect, we conduct a robust split-sample econometric analysis using three years of maintenance data from 426 mission-critical systems of a Fortune 100 company. The sampled systems vary in size (50KLOC to 2000KLOC, where 20% exceed 500KLOC), with a considerable portion of the sample manifesting “high” to “very high” software complexity. The analysis corroborates the known adverse effects of team instability, team knowledge diversity, and software complexity on maintenance productivity. More importantly, it shows—as theorized—that the adverse effects of the team attributes on maintenance productivity are significantly amplified only when software complexity grows high. We conclude with practical and research implications about how to manage software teams maintaining complex software over the life of a system.

Despite significant research progress on the problem of managing systems development risk, we are yet to see this problem addressed from an economic optimization perspective. Doing so entails answering the question: What mitigations should be planned and deployed throughout the life of a systems development project in order to control risk and maximize project value? We introduce an integrative economic optimization approach to solving this problem. The approach is integrative since it bridges two complementary research streams: one takes a traditional microlevel technical view on the software development endeavor alone, another takes a macrolevel business view on the entire life cycle of a systems project. Bridging these views requires recognizing explicitly that value-based risk management decisions pertaining to one level impact and can be impacted by decisions pertaining to the other level. The economic optimization orientation follows from reliance on real options theory in modeling risk management decisions within a dynamic stochastic optimization setting. Real options theory is well suited to formalizing the impacts of risk as well as the asymmetric and contingent economic benefits of mitigations, in a way that enables their optimal balancing. We also illustrate how the approach is applied in practice to a small realistic example.

Clancey (1992) proposed the model-construction framework as a way to explain the reasoning of knowledge-based systems (KBSs), based on his realization that all KBSs construct implicit or explicit situation-specific models (SSMs). An SSM is a rational argument that explains the solution produced for a specific problem situation pertaining to a target application task (e.g. SSMs constructed for typical diagnosis tasks are causal arguments having the structure of a proof). From a knowledge engineering perspective it makes sense that the notion of an SSM should play a major role in the modeling of tasks. Motivated by this view, we present a structured knowledge modeling methodology called SSM-directed knowledge modeling ( SSM-DKM ). In SSM-DKM, an SSM is a central structure that drives the entire modeling endeavor. In light of this fact, we explain how SSM-DKM supports three main stages in the knowledge engineering process—conceptualization, formalization and validation and instantiation—and illustrate the application of SSM-DKM to a medical diagnosis task. The knowledge model that SSM-DKM produces for a target application task has two appealing traits. First, the model embodies explicit knowledge about the ontology of SSMs that the task entails creating as solutions, thus enabling the construction of a KBS that makes these SSMs explicit. Second, the model captures strategic (or problem-solving) knowledge in declarative terms pertaining to the ontology of SSMs created for the task. Both these beneficial traits have been illustrated in the context of ACE-SSM, a KBS architecture that constructs explicit SSMs (Benaroch, 1998).

The goal of knowledge-based systems (KBSs) is not only to produce a solution to a problem that these systems face but also to construct, implicitly or explicitly, a situation-specific model (SSM) that explicates the rationale behind that solution. This paper focuses on how KBSs can benefit from the availability of explicit goal knowledge that reflects the underlying structure (ontology) of SSMs constructed for an application task. It first shows how goal knowledge can be captured. Then, it explains how ACE-SSM-an architecture for constructing explicit SSMs-uses this knowledge to direct the construction of explicit SSMs. Finally, it discusses benefits that KBSs can derive from the availability of explicit SSMs and their underlying goal knowledge. Some of these benefits pertain to ways to simplify the construction and maintenance of KBSs through reuse, while others relate to ways to endow KBSs with more robust problem solving and explanation capabilities. These benefits are illustrated using concrete examples.

Rule induction serves as an alternative knowledge acquisition method in the development of expert systems. This research presents ways to increase the information in induced decision trees, which convert to rules. Common rule induction methods do not organize data by time; however, time is a factor in many common induction applications, such as bankruptcy prediction and credit evaluation. Utilizing the time sequences in data can result in a more valuable decision support tool. This article extends rule induction algorithms to handle such data. It also illustrates the value of staged induction for a new real-world application: predicting hospital patients' length of stay. In the hospital application, the goal is to identify, as early as possible, patients likely to have an excessive length of stay, based on data gathered at admission and data that arrives gradually during the hospitalization period. In the context of this application, this article attempts to resolve conflicting results in the literature in two areas: 1) selecting attributes by the improvement in classification cost which they produce, and 2) stopping versus pruning in tree development. One conclusion is that selection and pruning by cost do not increase the average value of induced trees in this application.

An approach for designing a knowledge repository for risk management is presented. Since varied representations are used to capture the diverse types of knowledge involved in this domain, the atomic knowledge units stored in the repository are considered to be domain model (K) and inference method (M) pairs, or (K,M) pairs, which address subtasks in the domain. Such (K,M) pairs are semantically uniform. Conceptually, this allows one to view the repository as though it were a shared "database" of (K,M) pairs that has two key features. First, it serves stand-alone systems by enabling them to apply stored (K,M) pairs and share the generated results, where the applied pairs can be associated with any subtask that is part of an entire application task. Additionally, it avoids capturing redundant subtask-specific K's to the maximum possible extent by dynamically deriving them from the deep-principled K's that it stores.

Recent research suggests that the abilities of a knowledge-based system (KBS) depend in part on the amount of explicit knowledge it has about the way it is designed. This knowledge is often called design knowledge because it reflects design decisions that a KBS developer makes regarding what ontologies to embody in the system, what solution strategies to apply, what system architecture to use, etc. This paper examines one type of design knowledge pertaining to the structure underlying the solutions a KBS produces. (For example, in medical diagnosis, the output might be just a disease name, but the solution is actually a causal argument that the system implicitly constructs to find out how the disease came about.) We define this type of design knowledge, show how it can be represented, and explain how it can be used in problem solving to make the structure underlying solutions explicit. Subsequently, we also present and illustrate new avenues that the availability and use of the design knowledge discussed open with respect to the ability to build KBSs that possess strong explanation capabilities, are easier to maintain, support knowledge reuse, and offer more robustness in problem solving.

Abstract In design, inferring structure from function is a generate-and-test problem that is subject to combinatorial explosion. For certain types of economic and physical systems, it is fruitful to specify function in terms of desired behavior, and to identify sets of structurally connected components whose combined behavior under specific operating conditions matches the desired behavior. Specifically, in the financial risk management domain, behaviors of the components (stocks, bonds, options, etc.) are specified by two-dimensional piecewise linear functions called “payoff profiles,” and the goal is to identify linear combinations of these functions that produce a constrained behavior in response to uncontrollable economic events (e.g., interest rate fluctuations). Each identified combination corresponds to a configuration of investment vehicles which provide that constrained behavior. This paper presents a qualitative synthesis technique which uses this concept to construct all configurations of investment vehicles with a given desired behavior. Because the space of linear combinations of two-dimensional piecewise linear functions is subject to combinatorial explosion, the technique constrains the generation of combinations using two means. One is a goal-directed search process that relies on knowledge pertaining to the additivity of piecewise linear functions. Another is a qualitative abstraction over all piecewise linear functions with a similar shape, combined with the use of heuristic synthesis operators that rediscover information lost due to the abstraction. The technique is currently applied in a prototype expert system that supports the entire design process of risk management vehicles. The paper also discusses the possibility of using the technique in physical domains, through the configuration of analog computers.

Assembling financial instruments such as equities, bonds, options, and other derivatives into a portfolio requires a thorough understanding of how the portfolio will behave in response to changes of specific economic variables and parameters of the instruments. With more information about a more diverse set of instruments becoming available to traders, it is becoming important to limit the complexity of the analysis involved. We show how this complexity can be limited by using qualitative analysis, where the objective is to construct a few good vehicles which can then be analyzed quantitatively. We illustrate how two qualitative reasoning techniques — qualitative simulation and qualitative synthesis — are used to design investment vehicles for risk management purposes. These techniques are currently employed by a prototype expert system that aims at assisting traders solving a risk management problem called hedging.