We present a nonparametric interpretation for deep learning compatible modern Hopfield models and utilize this new perspective to debut efficient variants. Our key contribution stems from interpreting the memory storage and retrieval processes in modern Hopfield models as a nonparametric regression problem subject to a set of query-memory pairs. Interestingly, our framework not only recovers the known results from the original dense modern Hopfield model but also fills the void in the literature regarding efficient modern Hopfield models, by introducing sparse-structured modern Hopfield models with sub-quadratic complexity. We establish that this sparse model inherits the appealing theoretical properties of its dense analogue — connection with transformer attention, fixed point convergence and exponential memory capacity. Additionally, we showcase the versatility of our framework by constructing a family of modern Hopfield models as extensions, including linear, random masked, top-$K$ and positive random feature modern Hopfield models. Empirically, we validate our framework in both synthetic and realistic settings for memory retrieval and learning tasks.

This paper introduces the first low-power hardware accelerator for Spiking Transformers, an emerging alternative to traditional artificial neural networks. By modifying the base Spikformer model to use IAND instead of residual addition, the model exclusively utilizes spike computation. The hardware employs a fully parallel tick-batching dataflow and a time-step reconfigurable neuron architecture, addressing the delay and power challenges of multi-timestep processing in spiking neural networks. This approach processes outputs from all time steps in parallel, reducing computation delay and eliminating membrane memory, thereby lowering energy consumption. The accelerator supports 3x3 and 1x1 convolutions and matrix operations through vectorized processing, meeting model requirements. Implemented in TSMC’s 28nm process, it achieves 3.456 TSOPS (tera spike operations per second) with a power efficiency of 38.334 TSOPS/W at 500MHz, using 198.46K logic gates and 139.25KB of SRAM.

In this paper, we propose an algorithm that allows joint refinement of camera pose and scene geometry represented by decomposed low-rank tensor, using only 2D images as supervision. First, we conduct a pilot study based on a 1D signal and relate our findings to 3D scenarios, where the naive joint pose optimization on voxel-based NeRFs can easily lead to sub-optimal solutions. Moreover, based on the analysis of the frequency spectrum, we propose to apply convolutional Gaussian filters on 2D and 3D radiance fields for a coarse-to-fine training schedule that enables joint camera pose optimization. Leveraging the decomposition property in decomposed low-rank tensor, our method achieves an equivalent effect to brute-force 3D convolution with only incurring little computational overhead. To further improve the robustness and stability of joint optimization, we also propose techniques of smoothed 2D supervision, randomly scaled kernel parameters, and edge-guided loss mask. Extensive quantitative and qualitative evaluations demonstrate that our proposed framework achieves superior performance in novel view synthesis as well as rapid convergence for optimization. The source code is available at https://github.com/Nemo1999/Joint-TensoRF.

We present **STanHop-Net** (**S**parse **Tan**dem **Hop**field **Net**work) for multivariate time series prediction with memory-enhanced capabilities. At the heart of our approach is **STanHop**, a novel Hopfield-based neural network block, which sparsely learns and stores both temporal and cross-series representations in a data-dependent fashion. In essence, STanHop sequentially learns temporal representation and cross-series representation using two tandem sparse Hopfield layers. Additionally, STanHop incorporates two external memory modules: **Plug-and-Play** and **Tune-and-Play** for train-less and task-aware memory enhancements, respectively. They allow StanHop-Net to swiftly respond to sudden events. Methodologically, we construct the STanHop-Net by stacking STanHop blocks in a hierarchical fashion, enabling multi-resolution feature extraction with resolution-specific sparsity. Theoretically, we introduce a unified construction (**Generalized Sparse Modern Hopfield Model**) for both dense and sparse modern Hopfield models and show that it endows a tighter memory retrieval error compared to the dense counterpart without sacrificing memory capacity. Empirically, we validate the efficacy of STanHop-Net on many settings: time series prediction, fast test-time adaptation, and strongly correlated time series prediction.

Network robustness, which refers to a network’s ability to withstand malicious attacks on its vertices and edges, is critical across various natural and industrial domains. This paper delves into the assessment of network robustness through machine learning-based approaches, with a specific focus on structure-based representations and graph embeddings. The evaluation encompasses both synthetic and real-world networks, and three types of representation paradigms are scrutinized: 1) structure-based representations, including adjacency, incidence, and modularity matrices, 2) graph embeddings, including learning feature representation (LFR), DeepWalk, large-scale information network embedding (LINE), node2vec, structural deep network embedding (SDNE), and struc2vec, and 3) one-dimensional graph embeddings. The findings underscore the preference of convolutional neural networks (CNNs) with structure-based representations, highlighting the efficacy of adjacency and modularity matrices. While graph embeddings showcase versatility, their overall performance is comparatively lower, emphasizing the crucial role of representation complexity. This study contributes valuable insights into robustness evaluation methodologies and underscores the significance of tailored graph representations.

In recent years, there has been a significant surge in commercial demand for citation graph-based tasks, such as patent analysis, social network analysis, and recommendation systems. Graph Neural Networks (GNNs) are widely used for these tasks due to their remarkable performance in capturing topological graph information. However, GNNs’ output results are highly dependent on the composition of local neighbors within the topological structure. To address this issue, we identify two types of neighbors in a citation graph: explicit neighbors based on the topological structure and implicit neighbors based on node features. Our primary motivation is to clearly define and visualize these neighbors, emphasizing their importance in enhancing graph neural network performance. We propose a Correlation-aware Network (CNet) to re-organize the citation graph and learn more valuable informative representations by leveraging these implicit and explicit neighbors. Our approach aims to improve graph data augmentation and classification performance, with the majority of our focus on stating the importance of using these neighbors, while also introducing a new graph data augmentation method. We compare CNet with state-of-the-art (SOTA) GNNs and other graph data augmentation approaches acting on GNNs. Extensive experiments demonstrate that CNet effectively extracts more valuable informative representations from the citation graph, significantly outperforming baselines. The code is available on public GitHub. 1

We introduce the sparse modern Hopfield model as a sparse extension of the modern Hopfield model. Like its dense counterpart, the sparse modern Hopfield model equips a memory-retrieval dynamics whose one-step approximation corresponds to the sparse attention mechanism. Theoretically, our key contribution is a principled derivation of a closed-form sparse Hopfield energy using the convex conjugate of the sparse entropic regularizer. Building upon this, we derive the sparse memory retrieval dynamics from the sparse energy function and show its one-step approximation is equivalent to the sparse-structured attention. Importantly, we provide a sparsity-dependent memory retrieval error bound which is provably tighter than its dense analog. The conditions for the benefits of sparsity to arise are therefore identified and discussed. In addition, we show that the sparse modern Hopfield model maintains the robust theoretical properties of its dense counterpart, including rapid fixed point convergence and exponential memory capacity. Empirically, we use both synthetic and real-world datasets to demonstrate that the sparse Hopfield model outperforms its dense counterpart in many situations.

As the urban population rises, so does the pressure on the city’s transportation system. Most of the existing methods for passenger destination selection focus on processing the historical behaviors and travel trajectories of passengers. However, the existing methods face the generalization issue, the trained model cannot be applied to predict destinations in new metropolitan areas as the destination information is totally unseen and different from the training sets. To deal with the issue faced in IEEE BigData Cup 2022 – Trip Destination Prediction, in this work, we present a hybrid method. The main idea of our method is four-fold. The first is to implement the gravity model to capture human mobility between zones. The second contains two novel features to depict zones, including human traffic flow and feature class ratio. The third is to initialize the destinations in the new metropolitan area using the origin zones of multi-trip individuals. The last is to perform the nearest-neighbor search on both individuals and trips. The final destination prediction is produced by combining the gravity model and the nearest-neighbor search. Performance comparison reported by the competition leaderboard exhibits the superiority of our hybrid method, which also brings us to the fifth place in the competition.

A central task of a Disc Jockey (DJ) is to create a mixset of music with seamless transitions between adjacent tracks. In this paper, we explore a data-driven approach that uses a generative adversarial network to create the song transition by learning from real-world DJ mixes. The generator uses two differentiable digital signal processing components, an equalizer (EQ) and a fader, to mix two tracks selected by a data generation pipeline. The generator has to set the parameters of the EQs and fader in such a way that the resulting mix resembles real mixes created by human DJ, as judged by the discriminator counterpart. Result of a listening test shows that the model can achieve competitive results compared with a number of baselines.

The rapid development of Facebook makes a large amount of users to join the platform and share a variety of information on it. Facebook has become an innovative and effective platform for disseminating business information. It is thus interesting to investigate the issue of how to motivate the users to initiatively share the information of products or service provided by a certain brand which manages the Facebook Page. In addition, the privacy concern is one of the top issues of Facebook and would influence the sharing behavior. Therefore, this study intends to investigate the antecedents of the attitude toward sharing post on the Facebook from the perceived value perspective and how the privacy concern moderates the relationship between sharing attitude and sharing intention. According to the analysis results from 375 valid respondents, first, the utilitarian value and social value have significant effects on sharing attitude. Second, the privacy concern cannot moderate the relationship between the sharing attitude and sharing intention. In addition, we also divided the samples by gender and usage time to figure out the differences between each group. The implications of this study can provide some advices for Internet marketing decision makers to make proper marketing strategy of products and services.

In this paper, we utilize the synergy between data aggregation and fuzzy inferences to derive a new intelligent multiple-model-based estimation algorithm for robust urban mobile localization. The location estimation problem is cast into a Markov state transitioned system framework with fuzzy inferences for each base station for modeling the dynamics of a mobile station (MS), and data fusion of received time-of-arrival data and signal strength. This leads to a fuzzy-tuned extended Kalman filtering based interacting multiple-model (FT-IMM-EKF) algorithm. Due to data fusion along with an adequate fuzzy adaption of process noise variances, the proposed FT-IMM-EKF can ameliorate adverse effects of both NLOS errors and MS mobility more effectively than other relevant methods. The analytical results are substantiated by simulations.