Tables, especially when having complex layouts, contain rich semantic information. However, effectively learning from tables to uncover such semantic information remains challenging. The rapid progress in natural language processing does not necessarily correspond to equivalent advancements in table parsing, which often requires joint visual and language modeling. Indeed, humans can quickly derive semantic meaning from table entries by associating them with corresponding column and/or row headers. Motivated by this observation, we propose a new heterogeneous Graph-based Table Representation Learning (GTRL) framework. GTRL combines graph-based visual modeling with sequence-based language modeling to learn granular per-cell embeddings that are sensitive to the semantic meaning of cells within their corresponding table context. We systematically evaluate the proposed GTRL framework using two datasets: a new adhesive table benchmark comprising complex tables extracted from industrial documents for learning per-entry semantics, and a publicly available large-scale dataset that enables learning header semantics from column tables. Experimental results demonstrate the competitive performance of the proposed GTRL, which often exhibits reduced computational complexity compared to state-of-the-art table representation learning models.

In this paper we propose a novel probabilistic framework for document segmentation exploiting human perceptual recognition of text regions from complicated layouts. In particular, we conceptualize text homogeneity as the Gestalt pattern displayed in text regions, characterized by proximately and symmetrically arranged units with similar morphological and texture features. We model this pattern in the local region of a connected component (CC) using an hierarchical formulation, which simulates a random walk-and-check on a graph encoding the neighborhood of the CC. The proposed formulation allows an effective computation of what we call the probabilistic local text homogeneity (PLTH) using a weighted summation of the weights of the graph, which are derived from a probabilistic description of the homogeneity between neighboring CCs and computed through Bayesian cue integration. The proposed PLTH enables a multi-aspect analysis, where various primitives such as geometrical configuration, morphological features, texture characterization and location priors are integrated in one computational probabilistic model. This enables an effective text and non-text classification of CCs preceding any grouping process, which is currently absent in document segmentation. Experimental results show that our segmentation method based on the proposed PLTH model improves upon the state-of-the-art.

Physical damages (such as torn-offs and scratches) are commonly seen in historical documents. Recognition of such damages is currently absent in digitization-and-information-extraction (DIE) systems but crucial for automatic document comprehension and exploitation. In this paper we propose a generic damage recognition (DR) method based on a joint global and local modeling of the text homogeneity (TH) pattern exhibited in document images. More specifically, a connected component (CC) based formulation is developed as a global homogeneity measure, where TH is characterized using a probabilistic graph model for a coarse recognition of damaged regions. A multi-resolution analysis (MRA) of TH is further developed for a granular within-CC recognition of damage pixels, where the disparity between damage and text pixels is characterized by exploiting neighborhood transitions. This enables the formulation of a local homogeneity measure, where the neighborhood transition around an individual pixel is modeled using the propagation of the approximation coefficients of a stationary wavelet transform (SWT). The proposed global and local homogeneity measures are integrated as a joint likelihood in a Bayesian model with a Markov random field (MRF) prior, where DR is formulated as a maximum a posterior (MAP) inference which is addressed using Markov Chain Monte Carlo (MCMC) sampling. The resulting algorithm is tested on a set of real-life historical newspaper images containing damages of varying size and shape. The performance of the algorithm is evaluated using both F-measures and the Intersection-over-Union (IoU) metric, where test results demonstrate the promising potential of the proposed method.

Document image quality assessment (DIQA) is an important process for various applications such as optical character recognition (OCR) and document restoration. In this paper we propose a no-reference DIQA model based on a deep convolutional neural network (DCNN), where the rich knowledge of natural scene image characterization of a previously-trained DCNN is exploited towards OCR accuracy oriented document image quality assessment. Following a two-stage deep transfer learning procedure, we fine-tune the knowledge base of the DCNN in the first phase and bring in a task-specific segment consisting of three fully connected (FC) layers in the second phase. Based on the fine-tuned knowledge base, the task-specific segment is trained from scratch to facilitate the application of the transferred knowledge on the new task of document quality assessment. Testing results on a benchmark dataset demonstrate that the proposed model achieves state-of-the-art performance.

Fingerprinting codes based on the marking assumption provide reliable security against collusion attacks, but lack resilience against channel errors such as symbol erasures or other forms of distortions introduced by transmission over noisy or insecure communication channels. This additional source of distortion errors can be addressed by relaxing the marking assumption. In this paper we examine the restrictions and limitations on the code construction in terms of accusation errors, alphabet size, and the distortion errors under this weaker form of the marking assumption as formulated by Guth and Pfitzmann, also deriving a theoretical minimum lower bound on the code length of q-ary fingerprinting codes. We provide the formulas for applying the already existing binary Tardos code with symmetric accusation function under the weaker form of the marking assumption along with numerical results proving the tightness of the derived bounds. We show that the code is still of asymptotically minimum code length under the relaxation of the marking assumption.

To automate quality monitoring of multimedia applications, objective quality measures for images and video content need to be designed. Objective quality measures that model the Human Visual System (HVS) have a disappointing performance, because the HVS is not sufficiently understood. Integrating machine learning (ML) techniques may increase the performance. Unfortunately, traditional ML is difficult to interpret. To this end, we developed the Locally Adaptive Fusion (LAF), for more flexible and reliable quality predictions. This manuscript proposes six interactive programs and a website that demonstrate the effectiveness of LAF, which complement the technical focus of the corresponding journal paper.

Objective measures to automatically predict the perceptual quality of images or videos can reduce the time and cost requirements of end-to-end quality monitoring. For reliable quality predictions, these objective quality measures need to respond consistently with the behavior of the human visual system (HVS). In practice, many important HVS mechanisms are too complex to be modeled directly. Instead, they can be mimicked by machine learning systems, trained on subjective quality assessment databases, and applied on predefined objective quality measures for specific content or distortion classes. On the downside, machine learning systems are often difficult to interpret and may even contradict the input objective quality measures, leading to unreliable quality predictions. To address this problem, we developed an interpretable machine learning system for objective quality assessment, namely the locally adaptive fusion (LAF). This paper describes the LAF system and compares its performance with traditional machine learning. As it turns out, the LAF system is more consistent with the input measures and can better handle heteroscedastic training data.

We propose a novel image authentication system by combining perceptual hashing and robust watermarking. An image is divided into blocks. Each block is represented by a compact hash value. The hash value is embedded in the block. The authenticity of the image can be verified by re-computing hash values and comparing them with the ones extracted from the image. The system can tolerate a wide range of incidental distortion, and locate tampered areas as small as 1/64 of an image. In order to have minimal interference, we design both the hash and the watermark algorithms in the wavelet domain. The hash is formed by the sign bits of wavelet coefficients. The lattice-based QIM watermarking algorithm ensures a high payload while maintaining the image quality. Extensive experiments confirm the good performance of the proposal, and show that our proposal significantly outperforms a state-of-the-art algorithm.

Objective characterization of jewels in paintings, especially pearls, has been a long lasting challenge for art historians. The way an artist painted pearls reflects his ability to observing nature and his knowledge of contemporary optical theory. Moreover, the painterly execution may also be considered as an individual characteristic useful in distinguishing hands. In this work, we propose a set of image analysis techniques to analyze and measure spatial characteristics of the digital images of pearls, all relying on the so called spatiogram image representation. Our experimental results demonstrate good correlation between the new metrics and the visually observed image features, and also capture the degree of realism of the visual appearance in the painting. In that sense, these results set the basis in creating a practical tool for art historical attribution and give strong motivation for further investigations in this direction.

This paper presents a novel image adaptive data hiding system using properties of the discrete wavelet transform and which is ready to use in combination with JPEG 2000. Image adaptive watermarking schemes determine the embedding samples and strength from the image statistics. We propose to use the energy of wavelet coefficients at high frequencies to measure the amount of distortion that can be tolerated by a lower frequency coefficient. The watermark decoder in image adaptive data hiding needs to estimate the same parameters used for encoding from a modified source and hence is vulnerable to desynchronization. We present a novel way to resolve these synchronization issues by employing specialized insertion, deletion and substitution codes. Given the low complexity and reduced perceptual impact of the embedding technique, it is suitable for inserting camera and/or projector information to facilitate image forensics.