The strong lottery ticket hypothesis (SLTH) conjectures that high-performing subnetworks, called strong lottery tickets (SLTs), are hidden in randomly initialized neural networks. Although recent theoretical studies have established the SLTH across various neural architectures, the SLTH for transformer architectures still lacks theoretical understanding. In particular, the current theory of the SLTH does not yet account for the multi-head attention (MHA) mechanism, a core component of transformers. To address this gap, we introduce a theoretical analysis of the existence of SLTs within MHAs. We prove that, if a randomly initialized MHA of H heads and input dimension d has the hidden dimension O(d log(Hd^(3/2))) for the key and value, it contains an SLT that approximates an arbitrary MHA with the same input dimension with high probability. Furthermore, by leveraging this theory for MHAs, we extend the SLTH to transformers without normalization layers. We empirically validate our theoretical findings, demonstrating that the approximation error between the SLT within a source model (MHA and transformer) and an approximate target counterpart decreases exponentially by increasing the hidden dimension of the source model.

While foundation models have been exploited for various expert tasks with their fine-tuned parameters, any foundation model will be eventually outdated due to its old knowledge or limited capability, and thus should be replaced by a new foundation model. Subsequently, to benefit from its latest knowledge or improved capability, the new foundation model should be fine-tuned on each task again, which incurs not only the additional training cost but also the maintenance cost of the task-specific data. Existing work address this problem by inference-time tuning, i.e., modifying the output probability from the new foundation model by the outputs from the old foundation model and its fine-tuned model, which involves an additional inference cost by the latter two models. In this paper, we explore a new fine-tuning principle (which we call portable reward tuning; PRT) that reduces the inference cost by its nature, based on the reformulation of fine-tuning as the reward maximization with Kullback-Leibler regularization. Specifically, instead of fine-tuning parameters of the foundation models, PRT trains the reward model explicitly through the same loss as in fine-tuning. During inference, the reward model can be used with any foundation model (with the same set of vocabularies or labels) through the formulation of reward maximization. Experimental results, including both vision and language models, show that the PRT-trained model can achieve comparable accuracy with less inference cost, in comparison to the existing work of inference-time tuning.

Active Learning is an essential method for label-efficient deep learning. As a Bayesian active learning method, Bayesian Active Learning by Disagreement (BALD) successfully selects the most representative samples by maximizing the mutual information between the model prediction and model parameters. However, when applied to a batch acquisition mode, like batch construction with greedy search, BALD suffers from poor performance, especially with noises of near-duplicate data. To address this shortcoming, we propose a diverse beam search optimized batch active learning method, which explores a graph for every batch construction by expanding the highest-scored samples of a predetermined number. To avoid near duplicate beam branches (very similar beams generated from the same root and similar samples), which is undesirable for lacking diverse representations in the feature space, we design a self-adapted constraint within candidate beams. The proposed method is able to acquire data that can better represent the distribution of the unlabeled pool, and at the same time, be significantly different from existing beams. We observe that the proposed method achieves higher batch performance than the baseline methods on three benchmark datasets.

Nowadays, the explosive growth of time series data and the idea of automatically classifying them brought chances of advanced analysis and machine cognitive processes in various domains. Hundreds of Time Series Classification (TSC) algorithms have been proposed during the last decade. Most of them need the large quantity of labeled training data for achieving good precision. However, we noticed that under most of the training scenarios, the large-scale supervised training datasets are not readily available. We thus proposed a few shot deep learning neural network framework PIN-BA (Prototypical Inception Network with Cross Branch Attention) for time series classification with only limited training data. We use CNN (Convolutional Neural Network) with branches of different reception windows for capturing the features of different time window scales. We design cross branch attention schemes based on prototypical networks to emphasize the crucial features' information during classification. A few experiments were conducted on the famous UCR Time Series Data Sets. Experimental results demonstrate that our proposed framework outperforms the other TSC models, such like 1NN-DTW and InceptionTime under the few shot training data scenarios.

Nowadays, explosive growth of ontologies are used for managing data in various domains. They usually own different vocabularies and structures following different fashions. Ontology alignment finding semantic correspondences between elements of these ontologies can effectively facilitate the data communication and novel application creation in many practical scenarios. However, we noticed that, the traditional parametric ontology mapping methods still depend on individualistic abilities for setting proper parameters for mapping. When trying to utilize artificial neural networks for the automatic ontology mapping, the training data are found insufficient in most of the cases. This paper analyzes these problems, and proposes a few-shot ontology alignment model, which can automatically learn how to map two ontologies from only a few training links between their element pairs. The proposed model applies the Siamese neural network in computer vision on ontology alignment and designs an attention detection network learning the attention weights for different ontology attributes. A few experiments that conducted on the anatomy ontology alignment show that our model achieves good performance (94.3% of F-measure) with 200 training alignments without traditional parametric setting.

Presents the introductory welcome message from the conference proceedings. May include the conference officers' congratulations to all involved with the conference event and publication of the proceedings record.

Today, most of the IoT platforms including the standardizations for them are encountering a new stage called "Semantic Interoperability" which is expected to promise the intensive analytics and high level cross-domain intelligence in future. Although multiple IoT platforms such as oneM2M and OPC UA developed the outline architectures for IoT semantic implementation, the large cost on manual semantic annotation, ontology construction and metadata transferring hinder the practical use of these architectures. Efficient data aggression and computing perception for automatic semantic annotation and ontology learning become urgent for solving this problem. This paper did an exploratory investigation into these present researches and the future of semantic-enabled IoT. Real time stream reasoning technology and hierarchical clustering technology for automatic semantic annotation and reducing the semantic meta-data traffic are analyzed. The potential ways these two kinds of technologies can be combined and the possible benefits are estimated. A software architecture for facilitating these technologies in IoT is designed with introducing the respectively analyzed related algorithms and methodologies.

Scalable messaging methods for IoT services are needed to accommodate a vast number of devices. Skip Graph based topic based pub/sub (SG-TPS), which utilizes multiple brokers composing a structured overlay network, is one of the promising candidates. In SG-TPS, the distribution delay time is widely influenced by a way to assign subscribers to each broker. There are two possible approaches: the intensive assignment by which subscribers having a same topic are accommodated on a same broker, and the extensive assignment by which the subscribers are accommodated on different brokers as far as possible. In this paper, we formulate the difference of the distribution delay time between these two approaches, and discuss about the optimization of subscriber assignment.

Distributed pub/sub messaging has become indispensable for event-driven systems. There are methods for achieving high scalability regarding topic-based pub/sub by using structured overlay networks. However, these methods waste network resources concerning "exhaust data," which have low or no value most of the time. There are at least two problems: each publisher node continues to forward data to a relay node even if there are no subscribers, and multicast trees are constructed which are excessively large for low value data, namely having a small number of subscribers. In this paper, we formulate the requirements of overlay networks by defining a property called "strong relay-free" as an expansion of relay-free property, and propose a practical method satisfying the property by using Skip Graph. The proposed method involves publishers and subscribers composing connected sub graphs to enable detecting the absence of subscribers and autonomously adjusting the tree size. Through simulation experiments, we confirmed that the proposed method can suspend publishing adaptively, and shorten the path length on multicast trees by more than 75% under an experimental condition with 100,000 nodes. The proposed method is competent for decentralized event-driven systems with encouraging the locally produced data to be consumed locally.

In the Skip Graph, which is a structured overlay network that supports range retrievals, a key is supposed to consist of a single value. Therefore, the Skip Graph cannot perform range-to-range retrievals. In the present research, we extend the Skip Graph, enabling it to retain a range as a key, and propose the Range-Key Skip Graph, which can perform range-to-range retrievals. We also implement the proposed scheme and evaluate it on the PlanetLab.

Geographical contour lines are often used for the analysis of the sensor data distribution on wide area. In this paper, we propose a method to reconstruct contour lines by collecting sensor data from widely distributed sensors. In this case, it is redundant to collect all sensors from wide area since geographically closed sensors sense similar data. We propose a sensor data collection method for peer-to-peer sensor network to realize spatial interpolation of sensor data. In our proposal, we assume hierarchical Delaunay overlay network (HDOV) to collect sensor data uniformly. In addition, using structure of HDOV, more local sensor data is collected that affect on the spatial interpolation. By this mechanism, redundant sensor data to reconstruct geographical contour lines is reduced. The simulation result shows that our proposal can reconstruct accurate geographical contour lines, reducing target node and message numbers for data collection.

This paper describes a method to achieve a vision-cued swaying task in unknown environments utilizing adaptive visual servoing. The proposed method has a hybrid structure consisting of a controller to keep the distances between feet constant (a stance servoing controller), and an adaptive visual servoing controller. Making use of the method, the motion of each joint need not be pre-programmed, but is generated by the method according to the motion of visual cues. An experimental result demonstrates how the proposed method realizes a vision-cued swaying behavior of the legged robot.