No existing dataset adequately tests how well language models can incrementally update entity summaries – a crucial ability as these models rapidly advance. The Incremental Entity Summarization (IES) task is vital for maintaining accurate, up-to-date knowledge. To address this, we introduce , a fully synthetic dataset designed to expose real-world IES challenges. This dataset addresses issues like incorrect entity association and incomplete information, capturing real-world complexity by generating diverse attributes, summaries, and unstructured paragraphs with 99% alignment accuracy between generated summaries and paragraphs. Extensive experiments demonstrate the dataset’s difficulty – state-of-the-art LLMs struggle to update summaries with an F1 higher than 80.4%. We will open-source the benchmark and the evaluation metrics to help the community make progress on IES tasks.

Long-range tasks demand reasoning over long inputs.However, existing solutions are limited, e.g., long-context models require large compute budgets, parameter-efficient fine-tuning (PEFT) needs training data, and retrieval-augmented generation (RAG) entails complex task-specific designs.Though in-context approaches overcome many of these issues, methods with shortcontext LLMs are inefficient, trading context for processing more tokens.We introduce PRISM, a highly token-efficient in-context method based on structured schemas that outperforms baselines on diverse tasks with 4x shorter contexts.This approach produces concise outputs and efficiently leverages key-value (KV) caches to reduce costs by up to 54%.PRISM scales down to tiny contexts without increasing costs or sacrificing quality, and generalizes to new tasks with minimal effort by generating schemas from task descriptions.

Extracting structured data from visually rich documents like invoices, receipts, financial statements, and tax forms is key to automating many business workflows. However, building extraction models in this domain often demands a large collection of high-quality training examples. To address this challenge, we introduce FieldSwap, a novel data augmentation technique specifically designed for such extraction problems. FieldSwap generates synthetic training examples by replacing key phrases indicative of one field with those corresponding to another. Our experiments on five diverse datasets demonstrate that incorporating FieldSwap-augmented data into the training process can enhance model performance by 1–11 F1 points, particularly when dealing with limited training data (10–100 documents). Additionally, we propose algorithms for automatically inferring key phrases from the training data. Our findings indicate that FieldSwap is effective regardless of whether key phrases are manually provided by human experts or inferred automatically.

Building automatic extraction models for visually rich documents like invoices, receipts, bills, tax forms, etc. has received significant attention lately.A key bottleneck in developing extraction models for new document types is the cost of acquiring the several thousand highquality labeled documents that are needed to train a model with acceptable accuracy.In this paper, we propose selective labeling as a solution to this problem.The key insight is to simplify the labeling task to provide "yes/no" labels for candidate extractions predicted by a model trained on partially labeled documents.We combine this with a custom active learning strategy to find the predictions that the model is most uncertain about.We show through experiments on document types drawn from 3 different domains that selective labeling can reduce the cost of acquiring labeled data by 10× with a negligible loss in accuracy.

Extracting structured information from templatic documents is an important problem with the potential to automate many real-world business workflows such as payment, procurement, and payroll. The core challenge is that such documents can be laid out in virtually infinitely different ways. A good solution to this problem is one that generalizes well not only to known templates such as invoices from a known vendor, but also to unseen ones. We developed a system called Glean to tackle this problem. Given a target schema for a document type and some labeled documents of that type, Glean uses machine learning to automatically extract structured information from other documents of that type. In this paper, we describe the overall architecture of Glean, and discuss three key data management challenges : 1) managing the quality of ground truth data, 2) generating training data for the machine learning model using labeled documents, and 3) building tools that help a developer rapidly build and improve a model for a given document type. Through empirical studies on a real-world dataset, we show that these data management techniques allow us to train a model that is over 5 F1 points better than the exact same model architecture without the techniques we describe. We argue that for such information-extraction problems, designing abstractions that carefully manage the training data is at least as important as choosing a good model architecture.

We propose a novel approach using representation learning for tackling the problem of extracting structured information from form-like document images.We propose an extraction system that uses knowledge of the types of the target fields to generate extraction candidates, and a neural network architecture that learns a dense representation of each candidate based on neighboring words in the document.These learned representations are not only useful in solving the extraction task for unseen document templates from two different domains, but are also interpretable, as we show using loss cases.

Recent studies show that an overwhelming majority of emails are machine-generated and sent by businesses to consumers. Many large email services are interested in extracting structured data from such emails to enable intelligent assistants. This allows experiences like being able to answer questions such as “What is the address of my hotel in New York?” or “When does my flight leave?”. A high-quality email classifier is a critical piece in such a system. In this paper, we argue that the rich formatting used in business-to-consumer emails contains valuable information that can be used to learn better representations. Most existing methods focus only on textual content and ignore the rich HTML structure of emails. We introduce RiSER (Richly Structured Email Representation) - an approach for incorporating both the structure and content of emails. RiSER projects the email into a vector representation by jointly encoding the HTML structure and the words in the email. We then use this representation to train a classifier. To our knowledge, this is the first description of a neural technique for combining formatting information along with the content to learn improved representations for richly formatted emails. Experimenting with a large corpus of emails received by users of Gmail, we show that RiSER outperforms strong attention-based LSTM baselines. We expect that these benefits will extend to other corpora with richly formatted documents. We also demonstrate with examples where leveraging HTML structure leads to better predictions.

In emails, information abounds. Whether it be a bill reminder, a hotel confirmation, or a shipping notification, our emails contain useful bits of information that enable a number of applications. Most of this email traffic is machine-generated, sent from a business to a human. These business-to-consumer emails are typically instantiated from a set of email templates, and discovering these templates is a key step in enabling a variety of intelligent experiences. Existing email information extraction systems typically separate information extraction into two steps: an offline template discovery process (called template induction) that is periodically run on a sample of emails, and an online email annotation process that applies discovered templates to emails as they arrive. Since information extraction requires an email's template to be known, any delay in discovering a newly created template causes missed extractions, lowering the overall extraction coverage. In this paper, we present a novel system called Crusher that discovers templates completely online, reducing template discovery delay from a week (for the existing MapReduce-based batch system) to minutes. Furthermore, Crusher has a resource consumption footprint that is significantly smaller than the existing batch system. We also report on the surprising lesson we learned that conventional stream processing systems do not present a good framework on which to build Crusher. Crusher delivers an order of magnitude more throughput than a prototype built using a stream processing engine. We hope that these lessons help designers of stream processing systems accommodate a broader range of applications like online template induction in the future.

Machine generated business-to-consumer (B2C) emails such as receipts, newsletters, and promotions constitute a large portion of users' inboxes today. These emails reflect the users' interests and often are sequentially correlated, e.g., users interested in relocating may receive a sequence of messages on housing, moving, job availability, etc. We aim to infer (and eventually serve) the users' future interests by predicting the categories of their future emails. There are many useful methods, such as recurrent neural networks, that can be applied for such predictions, but in all cases the key to better performance is an effective representation of emails and users. To this end, we propose a general framework for learning embeddings for emails and users, using as input only the sequence of B2C templates users receive and open. (A template is a B2C email stripped of all transient information related to specific users.) These learned embeddings allow us to identify both sequentially correlated emails and users with similar sequential interests. We can also use the learned embeddings either as input features or embedding initializers for email category prediction tasks. Extensive experiments with millions of fully anonymized B2C emails demonstrate that the learned embeddings can significantly improve the prediction accuracy for future email categories. We hope that this effective yet simple embedding learning framework will inspire new machine intelligence applications that will improve the users' email experience.

Extracting structured data from emails can enable several assistive experiences, such as reminding the user when a bill payment is due, answering queries about the departure time of a booked flight, or proactively surfacing an emailed discount coupon while the user is at that store. This paper presents Juicer, a system for extracting information from email that is serving over a billion Gmail users daily. We describe how the design of the system was informed by three key principles: scaling to a planet-wide email service, isolating the complexity to provide a simple experience for the developer, and safeguarding the privacy of users (our team and the developers we support are not allowed to view any single email). We describe the design tradeoffs made in building this system, the challenges faced and the approaches used to tackle them. We present case studies of three extraction tasks implemented on this platform---bill reminders, commercial offers, and hotel reservations---to illustrate the effectiveness of the platform despite challenges unique to each task. Finally, we outline several areas of ongoing research in large-scale machine-learned information extraction from email.

A vast majority of the emails received by people today are machine-generated by businesses communicating with consumers. While some emails originate as a result of a transaction (e.g., hotel or restaurant reservation confirmations, online purchase receipts, shipping notifications, etc.), a large fraction are commercial emails promoting an offer (a special sale, free shipping, available for a limited time, etc.). The sheer number of these promotional emails makes it difficult for users to read all these emails and decide which ones are actually interesting and actionable. In this paper, we tackle the problem of extracting information from commercial emails promoting an offer to the user. This information enables an email platform to build several new experiences that can unlock the value in these emails without the user having to navigate and read all of them. For instance, we can highlight offers that are expiring soon, or display a notification when there»s an unexpired offer from a merchant if your phone recognizes that you are at that merchant»s store. A key challenge in extracting information from such commercial emails is that they are often image-rich and contain very little text. Training a machine learning (ML) model on a rendered image-rich email and applying it to each incoming email can be prohibitively expensive. In this paper, we describe a cost-effective approach for extracting signals from both the text and image content of commercial emails in the context of Gmail, an email platform that serves over a billion users around the world. The key insight is to leverage the template structure of emails, and use off-the-shelf OCR techniques to obtain the text from images to augment the existing text features offline. Compared to a text-only approach, we show that we are able to identify 9.12% more email templates corresponding to ~5% more emails being identified as offers. Interestingly, our analysis shows that this 5% improvement in coverage is across the board, irrespective of whether the emails were sent by large merchants or small local merchants, allowing us to deliver an improved experience for everyone.

Unsupervised template induction over email data is a central component in applications such as information extraction, document classification, and auto-reply. The benefits of automatically generating such templates are known for structured data, e.g. machine generated HTML emails. However much less work has been done in performing the same task over unstructured email data.

Machine-generated documents such as email or dynamic web pages are single instantiations of a pre-defined structural template. As such, they can be viewed as a hierarchy of template and document specific content. This hierarchical template representation has several important advantages for document clustering and classification. First, templates capture common topics among the documents, while filtering out the potentially noisy variabilities such as personal information. Second, template representations scale far better than document representations since a single template captures numerous documents. Finally, since templates group together structurally similar documents, they can propagate properties between all the documents that match the template. In this paper, we use these advantages for document classification by formulating an efficient and effective hierarchical label propagation and discovery algorithm. The labels are propagated first over a template graph (constructed based on either term-based or topic-based similarities), and then to the matching documents. We evaluate the performance of the proposed algorithm using a large donated email corpus and show that the resulting template graph is significantly more compact than the corresponding document graph and the hierarchical label propagation is both efficient and effective in increasing the coverage of the baseline document classification algorithm. We demonstrate that the template label propagation achieves more than 91% precision and 93% recall, while increasing the label coverage by more than 11%.

Modern applications, such as video and audio processing, employ many linear transforms and filters, such as the discrete cosine transform (DCT) and fast Fourier transform (FFT). There is a need to drastically reduce the energy consumption of these applications in order to enable their ultralow energy realization on mobile devices. It is well known that all of these applications can be implemented using multiple constant multiplication (MCM). As of now, there has been no effort to synthesize ultralow energy implementations of these systems, in particular, through the use of near-threshold computing (NTC). In order to synthesize MCM in NTC systems, we propose techniques that include minimizing the critical path through load reduction along with the creation of deep combinational logic in order to reduce the impact of process variation. We have demonstrated the effectiveness of our new techniques on both FFT and DCT applications and show up to an order of magnitude reduction in energy for the target delay.

Information security has emerged as an important system and application metric. Classical security solutions use algorithmic mechanisms that address a small subset of emerging security requirements, often at high-energy and performance overhead. Further, emerging side-channel and physical attacks can compromise classical security solutions. Hardware security solutions overcome many of these limitations with less energy and performance overhead. Nanoelectronics-based hardware security preserves these advantages while enabling conceptually new security primitives and applications. This tutorial paper shows how one can develop hardware security primitives by exploiting the unique characteristics such as complex device and system models, bidirectional operation, and nonvolatility of emerging nanoelectronic devices. This paper then explains the security capabilities of several emerging nanoelectronic devices: memristors, resistive random-access memory, contact-resistive random-access memory, phase change memories, spin torque-transfer random-access memory, orthogonal spin transfer random access memory, graphene, carbon nanotubes, silicon nanowire field-effect transistors, and nanoelectronic mechanical switches. Further, the paper describes hardware security primitives for authentication, key generation, data encryption, device identification, digital forensics, tamper detection, and thwarting reverse engineering. Finally, the paper summarizes the outstanding challenges in using emerging nanoelectronic devices for security.

Small form, mobile, and remote sensor network systems require secure and ultralow power data collection and communication solutions due to their energy constraints. The physical unclonable function (PUF) has emerged as a popular modern low power security primitive. However, current designs are analog in nature and susceptible to instability and difficult to integrate into existing circuitry. In this paper, we present the digital PUF which is stable in the same sense that digital logic is stable, has a very small footprint and very small timing overhead, and can be easily integrated into existing designs. We demonstrate the use of the digital PUF on two applications that are crucial for sensor networks: trusted remote sensing and logic obfuscation. We present our security analysis using standard randomness tests and confusion and diffusion analysis, and apply our new obfuscation approach on a set of standard design benchmarks.

This paper investigates and addresses the vulnerabilities of existing physical unclonable functions (PUFs). We first develop a PUF reverse engineering approach by conducting gate-level characterization (GLC). Based on the gate-level delay properties, we emulate the target PUF by designing a functionally equivalent PUF replication. Furthermore, in order to prevent such an attack, we develop a new sequential PUF architecture that is resilient to side channel-based reverse engineering. We obtain accurate results in emulating the timing behavior of the existing arbiter-based PUFs. Also, the randomness obtained from the sequential PUFs is significantly higher compared to the existing PUFs.

Foundry identification is essential for many tasks including intellectual property protection, trust, and preventing counterfeiting. In this paper, we introduce statistical techniques for foundry detection, specifically for identifying from which foundry a particular chip originates from. The key idea is to consider the distributions of channel lengths and threshold voltages after employing a variant of SAT that extracts these two metrics. We apply Kolmogorov-Smirnov and other statistical tests for comparing the two empirical distributions. Finally, we study the effects of sample size and measurement error on the correct identification rate and establish an interval of confidence using resubstitution techniques.

The physical unclonable function (PUF) has emerged as a popular and widely studied security primitive based on the randomness of the underlying physical medium. To date, most of the research emphasis has been placed on finding new ways to measure randomness, hardware realization and analysis of a few initially proposed structures, and conventional secret-key based protocols. In this work, we present our subjective analysis of the emerging and future trends in this area that aim to change the scope, widen the application domain, and make a lasting impact. We emphasize on the development of new PUF-based primitives and paradigms, robust protocols, public-key protocols, digital PUFs, new technologies, implementations, metrics and tests for evaluation/validation, as well as relevant attacks and countermeasures.

There is a great need to develop universal and robust techniques for intellectual property protection of integrated circuits. In this paper, we introduce techniques for the obfuscation of an arbitrary circuit by using physical unclonable functions (PUFs) and programmable logic. Specifically, we introduce the notion of PUF-based logic which can be configured to be functionally equivalent to any arbitrary design, as well as a new architecture for wire merging that obfuscates signal paths exponentially. We systematically apply our techniques in such a way so as to maximize obfuscation while minimizing area and delay overhead. We analyze our techniques on popular benchmark circuits and show them to be resilient against very powerful reverse engineering attacks in which the adversary has knowledge of the complete netlist along with the ability to read and write to any flip-flop in the circuit.

Computer-aided design (CAD), in its quest to facilitate new design revolutions, is again on the brink of changing its scope. Following both historical and recent technological and application trends, one can identify several emerging research and development directions in which CAD approaches and techniques may have major impacts. Among them, due to the potential to fundamentally alter everyday life as well as how science and engineering systems are designed and operated, the Internet of Things (IoT) stands out. IoT also poses an extraordinary system replete with conceptual and technical challenges. For instance, greatly reduced quantitative bounds on acceptable area and energy metrics require qualitative breakthroughs in design and optimization techniques. Most likely the most demanding of requirements for the widespread realization of many IoT visions is security. IoT security has an exceptionally wide scope in at least four dimensions. In terms of security scope it includes rarely addressed tasks such as trusted sensing, computation, communication, privacy, and digital forgetting. It also asks for new and better techniques for the protection of hardware, software, and data that considers the possibility of physical access to IoT devices. Sensors and actuators are common components of IoT devices and pose several unique security challenges including the integrity of physical signals and actuating events. Finally, during processing of collected data, one can envision many semantic attacks. Our strategic objective is to provide an impetus for the development of IoT CAD security techniques. We start by presenting a brief survey of IoT challenges and opportunities with an emphasis on security issues. Next, we discuss the potential of hardware-based IoT security approaches. Finally, we conclude with several case studies that advocate the use of stable PUFs and digital PPUFs for several IoT security protocols.

We have developed a new security hardware primitive named digital bimodal function (DBF) that enables ultra low energy security protocols. DBF allows the computation of legitimate communicating sides to be compact and low-energy while it requires any attacker exponential computational effort and energy expense. Our new approach is competitive with the energy efficiency of traditional security key cryptographic security technique (e.g., AES) while more than three orders of magnitude more energy efficient than RSA. The implementation is demonstrated using the Xilinx FPGA platform.

Localized multisensory systems for medical diagnostics are becoming increasingly important due to the prevalence of lightweight sensors and new collection, storage, and communication platforms such as mobile phones and tablets. These systems are often comprised of multisensory arrays that can be inefficient in terms of energy and cost both in sensing and transmission. We make the following two key observations: (i) the raw sensed data itself is unimportant, only those metrics relevant to diagnosis are needed, and (ii) it is often the case that the information relevant to medical diagnosis can be easily derived from the raw sensed data. Therefore, we drive our energy optimization procedure by selecting a subset of sensors that can predict these metrics well, eliminating all others and ultimately reducing the number of required sensors. We also develop a novel procedure for combining adjacent sensors to further reduce cost and sensing energy while increasing prediction strength. Finally, we present an algorithm for sub sampling the selected sensors that leverages the observations that: (i) most sensors need only be sampled after a physiological event is triggered, and (ii) such events tend to be predictable from semantic information and therefore a high sampling rate is unnecessary.

Traditional optimization methods for large multisensory networks often use sensor array reduction and sampling techniques that attempt to reduce energy while retaining full predictability of the raw sensed data. For systems such as medical sensor networks, raw data prediction is unnecessary, rather, only relevant semantics derived from the raw data are essential. We present a new method for sensor fusion, array reduction, and subsampling that reduces both energy and cost through semantics-driven system configuration. Using our method, we reduce the energy requirements of a medical shoe by a factor of 17.9 over the original system configuration while maintaining semantic relevance.