Class incremental learning (CIL) requires an agent to learn distinct tasks consecutively with knowledge retention against forgetting. Problems impeding the practice of CIL methods are twofold: (1) prompt update on non-i.i.d batch streams without boundary, namely the harsher online task-free CIL (OTCIL) scenario; (2) CIL methods suffer from heavy forgetting on learning long task streams, as shown in Fig. 1(a). To achieve efficient decision-making, the ensemble deep random vector functional link network (edRVFL) with forward regularization (-F) is proposed to replace the canonical Ridge (-R), reducing more regrets during OTCIL. Considering continuous distribution drifting on long stream, we further propose edRVFL-kF to adjust the intervention intensity of forward knowledge and derive incremental updates. edRVFL-kF can effectively avoid replay, retraining, and catastrophic forgetting while achieving lower regret over -R. Moreover, to improve robustness on non-i.i.d stream and eliminate intractable tuning of -kF, we rebuild with online Bayesian learning and propose the plug-and-play edRVFL-kF-Bayes, enabling all hard ks in multiple sub-learners to self-adapt to ever-changing distribution and optimization in OTCIL. Experiments were conducted on image datasets, including multiple evaluations, ablation tests, estimated forward, and compatibility studies, which distinctly validate the efficacy of edRVFL-kF-Bayes.

Online learning of deep neural networks faces challenges such as delayed non-incremental updating, increasing consumption, retrospective retraining, and catastrophic forgetting. To alleviate these drawbacks and achieve progressive immediate decision-making, we propose a novel Incremental Online Learning (IOL) framework of Randomized Neural Networks (Randomized NN), facilitating continuous improvements and analytics to Randomized NN performance in online scenarios. Within the framework, we further formulate IOL with ridge regularization (-R) and IOL with forward regularization (-F), both avoiding retrospective retraining and catastrophic forgetting. Moreover, the incremental algorithms for -R/-F on non-stationary batch stream are derived, featuring recursive weight updates and variable learning rates. Compared to -R, we recommend -F which improves learning performance using future unlabeled observations while further reducing online regrets to offline global experts. Additionally, we conduct a detailed analysis and theoretically derive relative cumulative regret bounds of the Randomized NN learners for -R/-F under adversarial assumptions via a novel methodology and present several corollaries, from which we observed the superiority in online learning acceleration and declined regret bounds of employing -F in IOL. Finally, our proposed methods were rigorously examined across diverse tasks, from simulation, regression, and classification tasks, to long-term time-series forecasting (LTSF) and continual learning (CL) fields, which distinctly validated the efficacy of the IOL frameworks and the advantages of forward regularization.

Forward regularization (-F) with unsupervised knowledge was advocated to replace canonical Ridge regularization (-R) in online linear learners, as it achieved a lower relative regret boundary. However, we observe that -F cannot perform as expected in practice, even possibly losing to -R for online tasks. We identify two main causes for this: (1) inappropriate intervened regularization, and (2) non-i.i.d. nature and data distribution changes in online learning (OL), both of which result in unstable posterior distribution and optima offset of the learner. To improve these, we first introduce the adjustable forward regularization (- k F), a more general -F with controllable knowledge intervention. We also derive - k F’s incremental updates with variable learning rate, and study relative regret and boundary in OL. Inspired by the regret analysis, to curb unstable penalties, we further propose - k F-Bayes style with k synchronously self-adapted to revise the intractable tuning of - k F by considering parametric posterior distribution changes in non-i.i.d. online data streams. Additionally, we integrate the - k F and - k F-Bayes into a multi-layer ensemble deep random vector functional link (edRVFL) and present two practical algorithms for batch learning, avoiding past replay and catastrophic forgetting. In experiments, we conducted tests on numerical simulation, tabular, and image datasets, where - k F-Bayes surpassed traditional -R and -F, highlighting the efficacy of ready-to-work - k F-Bayes and the great potentials of edRVFL- k F-Bayes in OL and continual learning (CL) scenarios. • - k F provides a more flexible unsupervised knowledge intervention for online learners. • We derive - k F’s incremental updates and study relative regret in OL. • We propose - k F-Bayes to consider parametric posterior changes in non-i.i.d. streams. • We integrate the - k F and - k F-Bayes into edRVFL and present two algorithms for CL.

Recent advances in style and appearance transfer are impressive, but most methods isolate global style and local appearance transfer, neglecting semantic correspondence. Additionally, image and video tasks are typically handled in isolation, with little focus on integrating them for video transfer. To address these limitations, we introduce a novel task, *Semantic Style Transfer*, which involves transferring style and appearance features from a reference image to a target visual content based on semantic correspondence. We subsequently propose a training-free method, *Semantix*, an energy-guided sampler designed for Semantic Style Transfer that simultaneously guides both style and appearance transfer based on semantic understanding capacity of pre-trained diffusion models. Additionally, as a sampler, *Semantix* can be seamlessly applied to both image and video models, enabling semantic style transfer to be generic across various visual media. Specifically, once inverting both reference and context images or videos to noise space by SDEs, *Semantix* utilizes a meticulously crafted energy function to guide the sampling process, including three key components: *Style Feature Guidance*, *Spatial Feature Guidance* and *Semantic Distance* as a regularisation term. Experimental results demonstrate that *Semantix* not only effectively accomplishes the task of semantic style transfer across images and videos, but also surpasses existing state-of-the-art solutions in both fields.

The profitability and survival of ship-owning companies in the global shipping market are deeply intertwined with accurate forecasts of ship prices and charter rates. Effective detection of market shortfalls and capitalization on temporal arbitrage opportunities are essential for maintaining a competitive edge. Traditional forecasting models, while adept at handling various multivariate time series tasks, predominantly focus on embedding synchronous time lags, often neglecting asynchronous dependencies. This paper introduces the Shipping Transformer (SFormer), a novel forecasting model designed to address this gap by integrating a discontinuous and parsimonious embedding strategy. This approach effectively captures lead-lag relationships between explanatory and target series. To further enhance forecasting performance, we introduce a cross-dimension attention module that uncovers cross-series dependencies. The SFormer sets a new benchmark for accuracy in predicting twelve time series of prices and charter rates for four ship types across multiple forecasting horizons. This research marks a significant advancement in the field of ship pricing and charter rate forecasting, providing ship-owning companies with critical insights to optimize their operations and enhance their strategic decision-making processes within the engineering management framework of the shipping industry.

The deep random vector functional link (dRVFL) and ensemble dRVFL (edRVFL) succeed in various tasks and achieve state-of-the-art performance compared with other randomized neural networks (NNs). However, existing edRVFL structures need more diversity and error correction ability in an independent network. Our work fills the gap by combining stacked deep blocks and residual learning with the edRVFL. Subsequently, we propose a novel dRVFL combined with residual learning, ResdRVFL, whose deep layers calibrate the wrong estimations from shallow layers. Additionally, we propose incorporating a scaling parameter to control the scaling of residuals from shallow layers, thus mitigating the risk of overfitting. Finally, we present an ensemble deep stacking network, SResdRVFL, based on ResdRVFL. SResdRVFL aggregates multiple blocks into a cohesive network, leveraging the benefits of deep learning and ensemble learning. We evaluate the proposed model on 28 datasets and compare it with the state-of-the-art methods. The comparative study demonstrates that the SResdRVFL is the best-performing approach in terms of average ranking and errors based on 28 datasets.

It is well known that many open-released foundational diffusion models have difficulty in generating images that substantially depart from average brightness, despite such images being present in the training data. This is due to an inconsistency: while denoising starts from pure Gaus-sian noise during inference, the training noise schedule retains residual data even in the final timestep distribution, due to difficulties in numerical conditioning in main-stream formulation, leading to unintended bias during in-ference. To mitigate this issue, certain ∊-prediction mod-els are combined with an ad-hoc offset-noise methodology. In parallel, some contemporary models have adopted zero-terminal SNR noise schedules together with v -prediction, which necessitate major alterations to pre-trained models. However, such changes risk destabilizing a large multitude of community-driven applications anchored on these pre-trained models. In light of this, our investigation revisits the fundamental causes, leading to our proposal of an inno-vative and principled remedy, called One More Step (OMS). By integrating a compact network and incorporating an ad-ditional simple yet effective step during inference, OMS ele-vates image fidelity and harmonizes the dichotomy between training and inference, while preserving original model pa-rameters. Once trained, various pre-trained diffusion mod-els with the same latent domain can share the same OMS module. Codes and models are released at here.

The electricity generation through wind energy is rapidly expanding, primarily due to its priorities of lower carbon emissions and sustainability. Precise wind speed forecasting is essential for renewable energy conversions as it mitigates the randomness of wind power, therefore aiding in more effective control and strategic planning for power system dispatch. However, the inherent fluctuation of wind speed challenges accurate and consistent time series forecasting. In this paper, we develop a novel parsimonious channel mixing ensemble deep random vector functional link (pcm-edRVFL) network to anticipate future wind speeds. The ensemble deep random vector functional link network (edRVFL) utilizes deep feature extraction and ensemble learning to improve forecasting performance. We refined the standard edRVFL model by incorporating a parsimonious channel mixing selection approach for input data, focusing on crucial historical observations, and strengthening the representation of each explanatory variable. We conduct extensive evaluations on four wind speed datasets using the proposed model, and the comparative experiment results demonstrate its superiority over other baseline models. Our proposed pcm-edRVFL network provides a practical approach for precise and efficient wind speed forecasting, proving to be an instrumental resource in wind energy design and operation systems.

The Random Vector Functional Link Network (RVFL) is a single-layer feed-forward network characterized by randomised weights in its hidden layers. However, the randomness can introduce detrimental neurons, potentially impairing the network’s performance. In response, this paper introduces multiple strategies to mitigate the noise from these randomised weights in RVFL networks. We first present a neuron normalization method that enhances latent space diversity and the network’s resilience to input features. Additionally, we develop improved approaches incorporating various feature selection and elimination techniques. Furthermore, Bayesian Optimization is utilized to optimize hyperparameters within a defined space. The efficacy of these methods is demonstrated through results from UCI classification tasks, highlighting the statistically superior performance of our Noise Eliminated edRVFL (NE-edRVFL) with neuron normalization.

Knowledge distillation (KD) is a conventional method in the field of deep learning that enables the transfer of dark knowledge from a teacher model to a student model, consequently improving the performance of the student model. In randomized neural networks, due to the simple topology of network architecture and the insignificant relationship between model performance and model size, KD is not able to improve model performance. In this work, we propose a self-distillation pipeline for randomized neural networks: the predictions of the network itself are regarded as the additional target, which are mixed with the weighted original target as a distillation target containing dark knowledge to supervise the training of the model. All the predictions during multi-generation self-distillation process can be integrated by a multi-teacher method. By induction, we have additionally arrived at the methods for infinite self-distillation (ISD) of randomized neural networks. We then provide relevant theoretical analysis about the self-distillation method for randomized neural networks. Furthermore, we demonstrated the effectiveness of the proposed method in practical applications on several benchmark datasets.

Class-incremental learning (CIL) on multivariate time series (MTS) is an important yet understudied problem. Based on practical privacy-sensitive circumstances, we propose a novel distillation-based strategy using a single-headed classifier without saving historical samples. We propose to exploit Soft-Dynamic Time Warping (Soft-DTW) for knowledge distillation, which aligns the feature maps along the temporal dimension before calculating the discrepancy. Compared with Euclidean distance, Soft-DTW shows its advantages in overcoming catastrophic forgetting and balancing the stability-plasticity dilemma. We construct two novel MTS-CIL benchmarks for comprehensive experiments. Combined with a prototype augmentation strategy, our framework demonstrates significant superiority over other prominent exemplar-free algorithms.

Text-conditional diffusion models are able to generate high-fidelity images with diverse contents. However, linguistic representations frequently exhibit ambiguous descriptions of the envisioned objective imagery, requiring the incorporation of additional control signals to bolster the efficacy of text-guided diffusion models. In this work, we propose Cocktail, a pipeline to mix various modalities into one embedding, amalgamated with a generalized ControlNet (gControlNet), a controllable normalisation (ControlNorm), and a spatial guidance sampling method, to actualize multi-modal and spatially-refined control for text-conditional diffusion models. Specifically, we introduce a hyper-network gControlNet, dedicated to the alignment and infusion of the control signals from disparate modalities into the pre-trained diffusion model. gControlNet is capable of accepting flexible modality signals, encompassing the simultaneous reception of any combination of modality signals, or the supplementary fusion of multiple modality signals. The control signals are then fused and injected into the backbone model according to our proposed ControlNorm. Furthermore, our advanced spatial guidance sampling methodology proficiently incorporates the control signal into the designated region, thereby circumventing the manifestation of undesired objects within the generated image. We demonstrate the results of our method in controlling various modalities, proving high-quality synthesis and fidelity to multiple external signals.

Deep neural networks have shown their promise in recent years with their state-of-the-art results. Yet, backpropagation-based methods may suffer from time-consuming training process and catastrophic forgetting when performing online learning. In this work we attempt to curtail them by employing the ensemble deep Random Vector Functional Link (edRVFL). As opposed to backpropagation-based neural networks that adjust weights iteratively, RVFL uses a closed-form solution method without iterative parameter learning. In addition, our approach allows the model to grow incrementally as new data is made available so that it can more resemble real-life learning scenarios. Our proposed online learning models were able to perform better on 72% of the datasets in the classification scenario and 80% of the datasets in the regression scenario, when compared to other available randomization-based online learning models in the literature. This is further supported by statistical comparisons which also show the stability of our network.

This paper proposes a three-stage online deep learning model for time series based on the ensemble deep random vector functional link (edRVFL). The edRVFL stacks multiple randomized layers to enhance the single-layer RVFL's representation ability. Each hidden layer's representation is utilized for training an output layer, and the ensemble of all output layers forms the edRVFL's output. However, the original edRVFL is not designed for online learning, and the randomized nature of the features is harmful to extracting meaningful temporal features. In order to address the limitations and extend the edRVFL to an online learning mode, this paper proposes a dynamic edRVFL consisting of three online components, the online decomposition, the online training, and the online dynamic ensemble. First, an online decomposition is utilized as a feature engineering block for the edRVFL. Then, an online learning algorithm is designed to learn the edRVFL. Finally, an online dynamic ensemble method, which can measure the change in the distribution, is proposed for aggregating all layers' outputs. This paper evaluates and compares the proposed model with state-of-the-art methods on sixteen time series.

Inspired by the ensemble strategy of machine learning, deep random vector functional link (dRVFL), and ensemble dRVFL (edRVFL) has shown state-of-the-art results on different datasets. Our present work first fills the gap of dRVFL and edRVFL work in the field of regression. We test and evaluate the performances of the dRVFLs on regression problems. Subsequently, we propose a novel regularization method boosted factor (BF), two dRVFLs variants edRVFL with skip connection (edRVFL-SC) and edRVFL with random skip connections (edRVFL-RSC) and one strategy ensemble skip connection edRVFL (esc-edRVFL) which show significant improvement over the original dRVFL. The BF is a newly introduced hyperparameter to scale the values of the activated hidden neurons to accommodate the diversity of the data, and it is also able to filter the neurons. edRVFL-SC and edRVFL-RSC are the edRVFL variants with skip connections. In edRVFL-SC, we apply dense skip connections to the edRVFL, which is inspired by the residual architecture in the deep learning area. However, due to the specificity of randomized networks, the simple skip connections are probably leading to the reuse of useless features. To address this problem, we propose a random skip connection-based edRVFL, which can keep the diversity in the latent space. esc-RVFL is an ensemble scheme that utilizes several edRVFL-RSC models trained on the different folds of the training dataset. The esc-edRVFL is identified as the best-performing algorithm through a comprehensive evaluation of 31 UCI datasets.

The integration of Vector Quantised Variational AutoEncoder (VQ-VAE) with autoregressive models as generation part has yielded high-quality results on image generation. However, the autoregressive models will strictly follow the progressive scanning order during the sampling phase. This leads the existing VQ series models to hardly escape the trap of lacking global information. Denoising Diffusion Probabilistic Models (DDPM) in the continuous domain have shown a capability to capture the global context, while generating high-quality images. In the discrete state space, some works have demonstrated the potential to perform text generation and low resolution image generation. We show that with the help of a content-rich discrete visual codebook from VQ-VAE, the discrete diffusion model can also generate high fidelity images with global context, which compensates for the deficiency of the classical autoregressive model along pixel space. Meanwhile, the integration of the discrete VAE with the diffusion model resolves the drawback of conventional autoregressive models being oversized, and the diffusion model which demands excessive time in the sampling process when generating images. It is found that the quality of the generated images is heavily dependent on the discrete visual codebook. Extensive experiments demonstrate that the proposed Vector Quantised Discrete Diffusion Model (VQ-DDM) is able to achieve comparable performance to top-tier methods with low complexity. It also demonstrates outstanding advantages over other vectors quantised with autoregressive models in terms of image inpainting tasks without additional training.

Reservoir Computing (RC) is well-suited for simpler sequential tasks which require inexpensive, rapid training, and the Echo State Network (ESN) plays a significant role in RC. In this article, we proposed variations of the Random Vector Functional Link (RVFL) network based on reservoir computing for non-sequential tasks. To commence, we present a plain echo state-based RVFL (esRVFL) that is distinguished from randomly generated input weights by the fact that esRVFL generates sparse matrices randomly to complete the initialization of the neuron weights. Following that, we extended it to a deep structure and introduced several network topologies. We also follow esRVFL and replace the single layer of echo state with a multi-layer stacked echo state network, where the entire network only needs to compute a set of output weights, which is called deep esRVFL (desRVFL). We evaluated our method on several public datasets and compared it with related methods. Experiments have shown that the proposed method can handle the classification tasks for tabular data and outperform some state-of-the-art randomized neural networks.

The randomized feed-forward neural network is a single hidden layer feed-forward neural network that enables efficient learning by optimizing only the output weights. The ensemble deep learning framework significantly improves the performance of randomized neural networks. However, the framework’s capabilities are limited by traditional hyper-parameter selection approaches. Meanwhile, different random network architectures, such as the existence or lack of a direct link and the mapping of direct links, can also strongly affect the results. We present an automated learning pipeline for the ensemble deep randomized feed-forward neural network in this paper, which integrates hyper-parameter selection and randomized network architectural search via Bayesian optimization to ensure robust performance. Experiments on 46 UCI tabular datasets show that our strategy produces state-of-the-art performance on various tabular datasets among a range of randomized networks and feed-forward neural networks. We also conduct ablation studies to investigate the impact of various hyper-parameters and network architectures.

In this paper, a sequential learning neural network, named as minimal resource allocating network (MRAN), is used to forecast monthly exchange rates between the U.S. dollar and the Deutsche mark, the British pound and the Canadian dollar. Five dominant economic structural exchange rate models are employed as the inputs of MRAN. Although the neural network cannot beat the simple random walk model without drift in out-of-sample forecast accuracy, it is better than the multilayer perceptron (MLP) neural network and the random walk model with drift in trend forecasting. The phenomena that the preferable structure of exchange rate model varies in different short periods are discovered from the simulation results. A simple model-competition methodology, purposing to choose the dominant model for next forecasting from the candidate models according to their previous short-term performance, is tested and found to improve the forecasting performance in forecast accuracy and direction accuracy.