Despite the increasing demand for safer machine learning practices, the use of Uncertainty Quantification (UQ) methods in production remains limited. This limitation is exacerbated by the challenge of validating UQ methods in absence of UQ ground truth. In classification tasks, when only a usual set of test data is at hand, several authors suggested different metrics that can be computed from such test points while assessing the quality of quantified uncertainties. This paper investigates such metrics and proves that they are theoretically well-behaved and actually tied to some uncertainty ground truth which is easily interpretable in terms of model prediction trustworthiness ranking. Equipped with those new results, and given the applicability of those metrics in the usual supervised paradigm, we argue that our contributions will help promoting a broader use of UQ in deep learning.

Software is a critical aspect of large-scale science, providing essential capabilities for making scientific discoveries. Large-scale scientific projects are vast in scope, with lifespans measured in decades and costs exceeding hundreds of millions of dollars. Successfully designing software that can exist for that span of time, at that scale, is challenging for even the most capable software companies. Yet scientific endeavors face challenges with funding, staffing, and operate in complex, poorly understood software settings. In this paper we discuss the practice of early-phase software architecture in the Square Kilometre Array Observatory’s Science Data Processor. The Science Data Processor is a critical software component in this next-generation radio astronomy instrument. We customized an existing set of processes for software architecture analysis and design to this project’s unique circumstances. We report on the series of comprehensive software architecture plans that were the result. The plans were used to obtain construction approval in a critical design review with outside stakeholders. We conclude with implications for other long-lived software architectures in the scientific domain, including potential risks and mitigations.

In practice, data augmentation is assigned a predefined budget in terms of newly created samples per epoch. When using several types of data augmentation, the budget is usually uniformly distributed over the set of augmentations but one can wonder if this budget should not be allocated to each type in a more efficient way. This paper leverages online learning to allocate on the fly this budget as part of neural network training. This meta-algorithm can be run at almost no extra cost as it exploits gradient based signals to determine which type of data augmentation should be preferred. Experiments suggest that this strategy can save computation time and thus goes in the way of greener machine learning practices.

A min max protocol offers a general method to automatically optimize steganographic algorithm against a wide class of steganalytic detectors. The quality of the resulting steganograhic algorithm depends on the ability to find an “adversarial” stego image undetectable by a set of detectors while communicating a given message. Despite min max protocol instantiated with ADV-EMB scheme leading to unexpectedly good results, we show it suffers a significant flaw and we present a theoretically sound solution called Backpack. Extensive experimental verification of min max protocol with Backpack shows superior performance to ADV-EMB, the generality of the tool by targeting a new JPEG QF100 compatibility attack and further improves the security of steganographic algorithms.

The curse of dimensionality in particle filtering can be mitigated by approximating the posterior distribution by a product of marginals on disjoint low dimensional subspaces of the state space. One such approach is known as the block particle filter in which the correction and resampling steps in particle filtering are run separately for estimating each marginal.In the linear and Gaussian case, the particle filter converges to the optimal Bayesian solution, i.e. the Kalman filter, as the number of particle increases. In this paper, we introduce the block based approach in the Kalman filter and show that the block particle filter asymptotically acts as the resulting block Kalman filter. It provides a relevant framework to disambiguate the bias incurred by the blocking step from the Monte Carlo estimation error.

The progress in steganography is hampered by a gap between non-additive distortion functions, which capture well complex dependencies in natural images, and their additive counterparts, which are efficient for data embedding. This paper proposes a theoretically justified method to approximate the former by the latter. The proposed method, called Backpack (for BACKPropagable AttaCK), combines new results in the approximation of gradients of discrete distributions with a gradient of implicit functions in order to derive a gradient w.r.t. the distortion of each JPEG coefficient. Backpack combined with the min max iterative protocol leads to a very secure steganographic algorithm. For example, the error rate of XuNet on 512 X 512 JPEG images, compressed with quality factor 100 and a payload of 0.4 bits per non-zero AC coefficient is 37.3% with Backpack, compared to a 26.5% error rate using ADV-EMB with minmax (considered state of the art in this work) and a 16.9% error rate with J-UNIWARD.

This article proposes an algorithm which allows Alice to simulate the game played between her and Eve. Under the condition that the set of detectors that Alice assumes Eve to have is sufficiently rich (e.g. CNNs), and that she has an algorithm enabling to avoid detection by a single classifier (e.g adversarial embedding, gibbs sampler, dynamic STCs), the proposed algorithm converges to an efficient steganographic algorithm. This is possible by using a min max strategy which consists at each iteration in selecting the least detectable stego image for the best classifier among the set of Eve's learned classifiers. The algorithm is extensively evaluated and compared to prior arts and results show the potential to increase the practical security of classical steganographic methods. For example the error probability Perr of XU-Net on detecting stego images with payload of 0.4 bpnzAC embedded by J-Uniward and QF 75 starts at 7.1% and is increased by +13.6% to reach 20.7% after eight iterations. For the same embedding rate and for QF 95, undetectability by XU-Net with J-Uniward embedding is 23.4%, and it jumps by +25.8% to reach 49.2% at iteration 3.

Background: The Systems Engineering and Software Engineering disciplines are highly intertwined in most modern Systems of Systems (SoS), and particularly so in industries such as defense, transportation, energy and health care. However, the combination of these disciplines during the architecting of SoS seems to be especially challenging; the literature suggests that major integration and operational issues are often linked to ambiguities and gaps between system-level and software-level architectures.

This work proposes a protocol to iteratively build a distortion function for adaptive steganography while increasing its practical security after each iteration. It relies on prior art on targeted attacks and iterative design of steganalysis schemes. It combines targeted attacks on a given detector with a \min\max strategy, which dynamically selects the most difficult stego content associated with the best classifier at each iteration. We theoretically prove the convergence, which is confirmed by the practical results. Applied on J-Uniward this new protocol increases \perr from 7% to 20% estimated by Xu-Net, and from 10% to 23% for a non-targeted steganalysis by a linear classifier with GFR features.

When Requirements Engineering(RE) models are unreasonably complex, they cannot support efficient decision making. SHORT is a tool to simplify that reasoning by exploiting the "key" decisions within RE models. These "keys" have the property that once values are assigned to them, it is very fast to reason over the remaining decisions. Using these "keys", reasoning about RE models can be greatly SHORTened by focusing stakeholder discussion on just these key decisions.This paper evaluates the SHORT tool on eight complex RE models. We find that the number of keys are typically only 12% of all decisions. Since they are so few in number, keys can be used to reason faster about models. For example, using keys, we can optimize over those models (to achieve the most goals at least cost) two to three orders of magnitude faster than standard methods. Better yet, finding those keys is not difficult: SHORT runs in low order polynomial time and terminates in a few minutes for the largest models.

The scale, heterogeneity, and pace of evolution of the storage components in big data systems makes it impractical to manually insert monitoring code for observability metric collection and aggregation. In this paper we present an architecture that automates these metric collection processes, using a model-driven approach to configure a distributed runtime observability framework. We describe and evaluate an implementation of the architecture that collects and aggregates metrics for a big data system using heterogeneous NoSQL data stores. Our scalability tests demonstrate that the implementation can monitor 20 different metrics from 10,000 database nodes with a sampling interval of 20 seconds. Below this interval, we lose metrics due to the sustained write load required in the metrics database. This indicates that observability at scale must be able to support very high write loads in a metrics collection database.

As part of the theory of belief functions, we address the problem of appraising the similarity between bodies of evidence in a relevant way using metrics. Such metrics are called evidential distances and must be computed from mathematical objects depicting the information inside bodies of evidence. Specialization matrices are such objects and, therefore, an evidential distance can be obtained by computing the norm of the difference of these matrices. Any matrix norm can be thus used to define a full metric. In this paper, we show that other matrices can be used to obtain new evidential distances. These are the α -specialization and α -generalization matrices and are closely related to the α -junctive combination rules. We prove that any L(1) norm-based distance thus defined is consistent with its corresponding α -junction. If α > 0 , these distances have in addition relevant variations induced by the poset structure of the belief function domain. Furthermore, α -junctions are meta-data dependent combination rules. The meta-data involved in α -junctions deals with the truthfulness of information sources. Consequently, the behavior of such evidential distances is analyzed in situations involving uncertain or partial meta-knowledge about information source truthfulness.

Designing massively scalable, highly available big data systems is an immense challenge for software architects. Big data applications require distributed systems design principles to create scalable solutions, and the selection and adoption of open source and commercial technologies that can provide the required quality attributes. In big data systems, the data management layer presents unique engineering problems, arising from the proliferation of new data models and distributed technologies for building scalable, available data stores. Architects must consequently compare candidate database technology features and select platforms that can satisfy application quality and cost requirements. In practice, the inevitable absence of up-to-date, reliable technology evaluation sources makes this comparison exercise a highly exploratory, unstructured task. To address these problems, we have created a detailed feature taxonomy that enables rigorous comparison and evaluation of distributed database platforms. The taxonomy captures the major architectural characteristics of distributed databases, including data model and query capabilities. In this paper we present the major elements of the feature taxonomy, and demonstrate its utility by populating the taxonomy for nine different database technologies. We also briefly describe QuABaseBD, a knowledge base that we have built to support the population and querying of database features by software architects. QuABaseBD links the taxonomy to general quality attribute scenarios and design tactics for big data systems. This creates a unique, dynamic knowledge resource for architects building big data systems.

This article presents a preceding car rear view tracking algorithm which utilizes a particle filter and belief function data fusion. Most of tracking applications resort to only one source of information, making the system dependent on the source reliability. To achieve more robust and longer tracking, multiple source data fusion is a solution. Belief functions are a powerful tool for data fusion. Using bridges between probability theory and belief function theory, data fusion information can be incorporated inside a particle filter. The efficiency of the proposed method is demonstrated on natural on-road sequences.

Architecture, business goals, and current technology (ABCs) continuously interact with each other to define and evolve the structure and function of software systems. This experience report looks at the ABC relationships in the domain of call center (CC) systems. Call centers are IT systems that provide telephone-based customer service. We look at the architecture and quality attributes of CC systems circa 1990. We then examine technology disruptions in the late 1990s which led to new business goals. Finally, we see how the system architecture has evolved to satisfy these new business goals, why this change has reprioritized the quality attributes of the system, and how the architect's skills have had to evolve to address this new type of system.

It is widely recognized that a good and appropriate architecture is critical to the success of a software product or system [5]. However, neither the system nor its architecture is static, and a good architecture anticipates and guides the evolution of the system over time. As the system evolves over time, the role of the software architect evolves as well, and skills that enabled an architect to be successful during one phase of a system’s lifetime may not enable success in later phases. This paper proposes a three-phase model to describe the evolution of software systems, and describes the contributions of the software architect which are necessary for success in each phase. This topic is of interest to practicing architects, and to software development managers responsible for selecting and hiring architects to contribute to a software system.

Software product-line engineering is one of the few approaches to software engineering that shows promise of improving software productivity by factors of 5 to 10. There are still few examples of its successful application on a large scale, partly because of the complexity of initiating a product-line engineering project and the many factors that must be addressed for such a project to be successful. This tutorial draws on experiences in introducing and sustaining product-line engineering in Lucent Technologies and in Avaya. The objective is to convey to participants the obstacles involved in transitioning to product line engineering and how to overcome such obstacles, particularly in large software development organizations. Participants will learn both technical and organizational aspects of the problem. Participants will leave the tutorial with many ideas on how to introduce product line engineering into an organization in a systematic way.