The combination of linear transformations and nonlinear activation functions forms the foundation of most modern deep neural networks, enabling them to approximate highly complex functions. This paper explores the introduction of quadratic transformations to further increase the nonlinearity of the model, with the aim of enhancing the performance of existing architectures. To minimize the additional parameters and computational burden, we propose a lightweight quadratic enhancer that leverages matrix decomposition, weight sharing, and sparsification techniques. This approach introduces only a minimal and negligible increase in parameters and forward computation, while still yielding substantial improvements in model performance. We evaluate the effectiveness of the proposed method across three tasks: text classification, image classification, and fine-tuning large language models (LLMs). In all tasks, our approach demonstrates significant performance gains.

Solving large-scale linear programming (LP) problems is an important task in various areas such as communication networks, power systems, finance and logistics. Recently, two distinct approaches have emerged to expedite LP solving: (i) First-order methods (FOMs); (ii) Learning to optimize (L2O). In this work, we propose an FOM-unrolled neural network (NN) called PDHG-Net, and propose a two-stage L2O method to solve large-scale LP problems. The new architecture PDHG-Net is designed by unrolling the recently emerged PDHG method into a neural network, combined with channel-expansion techniques borrowed from graph neural networks. We prove that the proposed PDHG-Net can recover PDHG algorithm, thus can approximate optimal solutions of LP instances with a polynomial number of neurons. We propose a two-stage inference approach: first use PDHG-Net to generate an approximate solution, and then apply PDHG algorithm to further improve the solution. Experiments show that our approach can significantly accelerate LP solving, achieving up to a 3$\times$ speedup compared to FOMs for large-scale LP problems.

This paper proposes a FPGA-based energy-efficient processor for radar-based continuous fall detection, consisting of a spectrogram generation circuit and a convolutional neural network (CNN). To reduce resource consumption and power usage of the entire processor, two designs were implemented: 1) A serial-FFT-based spectrogram generation circuit for radar signal preprocessing, and 2) An neural network (NN) accelerator based on row-stationary dataflow has been designed. By using the updated block wise computation technique, the accelerator enables the computation of only the NN’s updated inputs, resulting in an 85% reduction in multiply-accumulate (MAC) operations and a 79% reduction in intermediate result storage. Implemented on the Xilinx FPGA board ZC702, this processor consumes only 7.3k LUTs, 3.6k Flip Flops (FFs), 22 Block RAMs (BRAMs), and 10 DSPs, with a power consumption of only 0.302W. On an open-source radar fall detection dataset, this processor achieves an accuracy of 99.58%. The processor incurs a delay of only 0.431ms for a single preprocessing and NN inference, consuming just 130.16 µJ.

Integer linear programs (ILPs) are commonly employed to model diverse practical problems such as scheduling and planning. Recently, machine learning techniques have been utilized to solve ILPs. A straightforward idea is to train a model via supervised learning, with an ILP as the input and an optimal solution as the label. An ILP is symmetric if its variables can be permuted without changing the problem structure, resulting in numerous equivalent and optimal solutions. Randomly selecting an optimal solution as the label can introduce variability in the training data, which may hinder the model from learning stable patterns. In this work, we incorporate the intrinsic symmetry of ILPs and propose a novel training framework called SymILO. Specifically, we modify the learning task by introducing solution permutation along with neural network weights as learnable parameters and then design an alternating algorithm to jointly optimize the loss function. We conduct extensive experiments on ILPs involving different symmetries and the computational results demonstrate that our symmetry-aware approach significantly outperforms three existing methods----achieving $50.3\\%$, $66.5\\%$, and $45.4\\%$ average improvements, respectively.

Graph neural networks (GNNs) have recently emerged as powerful tools for addressing complex optimization problems. It has been theoretically demonstrated that GNNs can universally approximate the solution mapping functions of linear programming (LP) problems. However, these theoretical results typically require GNNs to have large parameter sizes. Conversely, empirical experiments have shown that relatively small GNNs can solve LPs effectively, revealing a significant discrepancy between theoretical predictions and practical observations. In this work, we aim to bridge this gap by providing a theoretical foundation for the effectiveness of small-size GNNs. We prove that polylogarithmic-depth, constant-width GNNs are sufficient to solve packing and covering LPs, two widely used classes of LPs. Our proof leverages the capability of GNNs to simulate a variant of the gradient descent algorithm on a carefully selected potential function. Additionally, we introduce a new GNN architecture, termed GD-Net. Experimental results demonstrate that GD-Net significantly outperforms conventional GNN structures while using fewer parameters.

In the scenarios of the Internet of Things, fall detection holds increasing significance in the health monitoring of elderly individuals. While most current research has achieved impressive performance in fall detection methods, there are limitations in deploying these methods to resource-limited edge devices. This article proposes an energy-efficient edge processor for radar-based continuous fall detection, which consists of a preprocessing module and a convolutional neural network (NN) accelerator. Multiple designs were implemented to minimize resource utilization and power consumption of the entire processor: 1) a preprocessing module based on mixed-radix FFT is utilized for radar signal preprocessing and 2) an NN accelerator is designed to support an updated blockwise (UBwise) computation technique aimed at reducing redundant calculations and intermediate result storage in continuous fall detection, along with a fully connected (FC) layer cache compression technique proposed to compress the cache required for FC layer computations. Applying these techniques results in an 80% reduction in RAM size, an 88.6% decrease in intermediate result storage, and a 92.6% reduction in multiply-accumulate operations. Implemented on an FPGA, this processor consumes merely 3.1k look up tables, 2.3k flip-flops, four block RAMs, and seven DSPs while consuming only 0.234 W of power. On an open-source radar-based fall detection data set, the processor attains an accuracy of 98.58%. Additionally, it incurs a mere 42 us delay for a single preprocessing and NN inference, consuming just 9.8 uJ. Compared to state-of-the-art works, this processor’s energy consumption is reduced by 81.2%, and the required memory is reduced by 93.3%.