Industrial control systems (ICSs) are widely used in various industries. These systems have become prime targets for cyber and physical attacks. The attacks, which have an impact on the physical processes of an ICS, often lead to system misbehaviors through data contamination. Recent research has shown that network-based detection methods cannot monitor the physical level activities well enough to mitigate hybrid cyber attacks and cannot entirely protect ICSs. To protect ICSs from such threats, we propose a Spatio-Temporal Autoencoder (STA) with a Dynamic Thresholding Mechanism. The STA learns the normal physical behaviors of the system by capturing deep spatio-temporal dependencies to form a unified representation of the system state. The unified representation is decoded to reconstruct the input features. Then, the dynamic threshold is used to detect and locate the anomalies. We validate the STA using data set from a real water treatment plant testbed, SWaT. Evaluation results indicate the superior performance of the STA compared with six state-of-the-art methods, achieving an average improvement of 5.8% in the F-Score.

The doubly resource constrained elementary shortest path problem (DRCESPP) has important applications in intelligent network scenarios. For solving this strong NP-hard problem, we introduce learning-based techniques and present a solution framework integrating preprocessing, graph neural networks (GNNs) and deep reinforcement learning (DRL). First, the preprocessing procedure reduces the network size and provides initial feasible paths. Then, the classifier, implemented by a graph attention network (GAT), filters out the reduced networks where better paths exist after preprocessing. Finally, a DRL-based heuristic attempts to construct the optimal path for the filtered reduced networks with an end-to-end solution paradigm. We devise a crafted reward function and a shared low-variance baseline for the reinforcement learning optimization algorithm. Our experiments suggest that the proposed framework achieves better performance compared with competitive heuristic algorithms in terms of solution quality and computational efficiency.

Identifying the nature of data flows can help improve network service and security. Most existing solutions usually simplify the traffic classification to protocol and application identification based on some uniqueness assumptions. However, in the real world these assumptions aren’t always reasonable due to the abuse of multiplexing techniques. In this work, a new scheme is proposed from a different perspective that aims to directly identify the content inside a data flow without considering the external protocols and applications. We use wavelet to obtain the time-scale signals of each data flow and develop a new hidden Markov tree (HMT) with an embedding deep neural network (DNN) to model these signals. Each hidden state of the HMT represents a specific signal generation pattern. Transition of hidden states describes the time-scale context of the signal patterns. DNN is used to describe the probabilistic relationship between the implicit patterns and the observed time-scale signals. We derive new algorithms for the model and create an instance for each type of traffic, which projects the data flows into a multi-dimensional decision space and achieves their content identification through a classifier. Numerical experiments using real datasets are presented to validate the proposed scheme. Performance-related issues and comparisons with related works are discussed.

Recurrent neural networks (RNNs) can be used to operate over sequences of vectors and have been successfully applied to a variety of problems. However, it is hard to use RNNs to model the variable dwell time of the hidden state underlying an input sequence. In this article, we interpret the typical RNNs, including original RNN, standard long short-term memory (LSTM), peephole LSTM, projected LSTM, and gated recurrent unit (GRU), using a slightly extended hidden Markov model (HMM). Based on this interpretation, we are motivated to propose a novel RNN, called explicit duration recurrent network (EDRN), analog to a hidden semi-Markov model (HSMM). It has a better performance than conventional LSTMs and can explicitly model any duration distribution function of the hidden state. The model parameters become interpretable and can be used to infer many other quantities that the conventional RNNs cannot obtain. Therefore, EDRN is expected to extend and enrich the applications of RNNs. The interpretation also suggests that the conventional RNNs, including LSTM and GRU, can be made small modifications to improve their performance without increasing the parameters of the networks.

Studies have shown that most of the flows in data centres are mice flows and last short time. However, existing green networking solutions do not consider this traffic characteristic. A mass of mice flows may be rerouted after network equipment is put into sleep mode and this can result in significant degradation of network performance. In this study, the authors propose a novel solution for green networking called energy‐efficient flow scheduling approach (EFSA). EFSA aggregates mice flow into a minimal set of links, and the largest flows that result in an overload of links in the set are rerouted in time. Instead of managing every mice flows, only a few numbers of largest flows are rerouted. Unused switches/links will be put into sleep mode for energy saving. The evaluation of EFSA shows that it can significantly reduce the number of rerouted flows and improve the performance of data centre networks while saving network energy efficiently.

Reducing energy consumption of the internet becomes more and more important. The authors propose a novel approach, called predictive energy‐saving approach (PESA), to put near‐idle nodes of a network into sleep mode to save energy. In PESA, they propose an online spatial hidden Markov model (OSHMM) to describe the distribution of the network traffic and predict the lowest‐loaded nodes. Those nodes will be put into sleep mode under the constraints that the network's full connectivity is kept and no new congested nodes are generated. To reduce the computation, they leverage on an online training technique and M‐algorithm in OSHMM. They evaluate the approach over a randomly generated topology and a real ISP backbone topology with a real traffic dataset. The results show that the proposed approach is energy efficient with low computation complexity.

Collaborative Anomaly Detection CAD is an emerging field of network security in both academia and industry. It has attracted a lot of attention, due to the limitations of traditional fortress-style defense modes. Even though a number of pioneer studies have been conducted in this area, few of them concern about the universality issue. This work focuses on two aspects of it. First, a unified collaborative detection framework is developed based on network virtualization technology. Its purpose is to provide a generic approach that can be applied to designing specific schemes for various application scenarios and objectives. Second, a general behavior perception model is proposed for the unified framework based on hidden Markov random field. Spatial Markovianity is introduced to model the spatial context of distributed network behavior and stochastic interaction among interconnected nodes. Algorithms are derived for parameter estimation, forward prediction, backward smooth, and the normality evaluation of both global network situation and local behavior. Numerical experiments using extensive simulations and several real datasets are presented to validate the proposed solution. Performance-related issues and comparison with related works are discussed.

The underflow problem of the forward-backward algorithm is a crucial issue for implementation of Hidden semi-Markov models (HsMM). A widely used solution is to scale up the forward and backward variables at each time step. We demonstrate the conventional scaling approach is not robust with several examples, then propose an improved scaling approach which is warranted to be robust and applicable to all HsMM variants. With the proposed method, all the variables are proved to be properly scaled up at the expense of acceptable computational complexity. Numerical experiments validate these claims.

Due to the limitations of the traditional port-based and payload-based traffic classification approaches, the past decade has seen extensive work on utilizing machine learning techniques to classify network traffic based on packet and flow level features. In particular, previous studies have shown that the unsupervised clustering approach is both accurate and capable of discovering previously unknown application classes. In this paper, we explore the utility of side information in the process of traffic clustering. Specifically, we focus on the flow correlation information that can be efficiently extracted from packet headers and expressed as instance-level constraints, which indicate that particular sets of flows are using the same application and thus should be put into the same cluster. To incorporate the constraints, we propose a modified constrained K-Means algorithm. A variety of real-world traffic traces are used to show that the constraints are widely available. The experimental results indicate that the constrained approach not only improves the quality of the resulted clusters, but also speeds up the convergence of the clustering process.

A new two-layer hidden Markov model is proposed to describe the arrival rate process of Web traffic. The macro state process of the first underlying layer is used to describe the large-scale trends of network traffic. The sub-state process of the second underlying layer is used to describe the small-scale fluctuations that are happening during the duration of a given macro state. Experiments are implemented to validate the proposed model.

Network traffic classification is an essential component for network management and security systems. To address the limitations of traditional port-based and payload-based methods, recent studies have been focusing on alternative approaches. One promising direction is applying machine learning techniques to classify traffic flows based on packet and flow level statistics. In particular, previous papers have illustrated that clustering can achieve high accuracy and discover unknown application classes. In this work, we present a novel semi-supervised learning method using constrained clustering algorithms. The motivation is that in network domain a lot of background information is available in addition to the data instances themselves. For example, we might know that flow f1and f2are using the same application protocol because they are visiting the same host address at the same port simultaneously. In this case, f1and f2shall be grouped into the same cluster ideally. Therefore, we describe these correlations in the form of pair-wise must-link constraints and incorporate them in the process of clustering. We have applied three constrained variants of the K-Means algorithm, which perform hard or soft constraint satisfaction and metric learning from constraints. A number of real-world traffic traces have been used to show the availability of constraints and to test the proposed approach. The experimental results indicate that by incorporating constraints in the course of clustering, the overall accuracy and cluster purity can be significantly improved.

Identifying applications and classifying network traffic flows according to their source applications are critical for a broad range of network activities. Such classifications can be based on information derived from packet header fields and payload content, or statistical characteristics of flows and communication patterns of hosts. However, most of present methods rely on some forms of priori knowledge. In this paper, an application signature based traffic classification system with a novel approach to fully automate the process of deriving signatures from unknown traffic is proposed. The key idea is to combine traffic clustering based on statistical flow properties in order to generate clusters dominated by a single application on the one hand, and application signature construction solely based on payload content from each cluster on the other hand. Evaluation using real-world traffic traces indicate that the proposed approach is highly effective.

Abstract Identifying applications and classifying network traffic flows according to their source applications are critical for a broad range of network activities. Such a decision can be based on packet header fields, packet payload content, statistical characteristics of traffic and communication patterns of network hosts. However, most present techniques rely on some sort ofa prioriknowledge, which means they require labor‐intensive preprocessing before running and cannot deal with previously unknown applications. In this paper, we propose a traffic classification system based on application signatures, with a novel approach to fully automate the process of deriving signatures from unidentified traffic. The key idea is to integrate statistics‐based flow clustering with payload‐based signature matching method, so as to eliminate the requirement of pre‐labeled training data sets. We evaluate the efficiency of our approach using real‐world traffic trace, and the results indicate that signature classifiers built from clustered data and pre‐labeled data are able to achieve similar high accuracy better than 99%. Copyright © 2010 John Wiley & Sons, Ltd.

Botnets with the centralized architecture provide a simple, low-latency, anonymous and efficient real-time communication platform for the botnet controllers. To our knowledge, most of the latest detected large-scale botnets are based on the centralized structure with HTTP or customized protocols. Therefore, centralized botnets detection helps greatly improve control of unwanted traffic. The main contribution of this study is the development of a common detection mechanism aiming at the centralized botnets. In this work we investigate the intrinsic characteristics based on the distributed yet bursting property of the centralized botnets. Our study shows that there exist great similarity and synchronization among the behaviors and the command and control (C&C) traffic of the bots, because the bots are controlled to operate according to the programmed schedule. Firstly we can determine if the groups of flows are suspectable by performing evaluation on the payload similarity and sequence correlation. Further, we will monitor and keep tracking with the collective and simultaneous behaviors of the suspicious groups of hosts. As is shown by conducting experiments, the proposed method can detect and hold back the centralized botnets effectively before they seriously influence the normal operation on the wide-scale network.

Protocol reverse engineering, the process of (re)constructing the protocol context of communication sessions by an implementation, which involves translating a sequence of packets into protocol messages, grouping them into sessions, and modeling state transitions in the protocol state machine, is well-known to be invaluable for many network security applications, including intrusion prevention and detection, traffic normalization, and penetration testing, etc. However, current practice in deriving protocol specifications is either mostly manual or focusing on automatic reverse engineering the message format only and leaving the protocol state machine inverse undone. Although regular expressions offer superior expressive ability and flexibility, application protocols are described by regular expression manually based on sufficiently understanding protocol itself. At present there is not an effect method to realize classification, recognition and control automatically for the known applications and the unknown applications in future. In this paper a novel approach is presented to model network application specification. In this work, the whole automatic protocol reverse engineering is realized through accomplishing the protocol state machine, and then the FSMs are translated to corresponding regular expressions to enrich and update the pattern database. This approach uses grammatical inference and is motivated by the observation that an implementation of the protocol is inherently a state transition process, the state machine model the essence exactly. The important significance is to describe various state protocols with a common method through modeling the protocol state transition, including known and unknown ones. This approach had been implemented in the system and evaluated using real-world implementations of three different protocols: HTTP, SMTP, FTP, and compared the extracted protocol to the corresponding other newly system, such as 17-filter.

It is difficult for the existing anomaly detection methods to distinguish the burst of normal traffic from the anomalous traffic in a large-scale Web site. This paper uses hidden semi-Markov model to describe the browsing behaviors of Web users. An efficient recursive algorithm for this model is presented for the online implementation of model update, which is used to track the Web users' browsing behaviors dynamically. An anomaly detection scheme is proposed for the application of this model. Likelihood of an observation sequence on a user browsing behaviors fitting to the model is used as a measure of normality of the user. Finally, an experiment is conducted to validate our model and algorithms, which is based on a real traffic data and an emulated distributed denial of service attack

Existing algorithms for estimating the model parameters of an explicit-duration hidden Markov model (HMM) usually require computations as large as O((MD/sup 2/ + M/sup 2/)T) or O(M/sup 2/ DT), where M is the number of states; D is the maximum possible interval between state transitions; and T is the period of observations used to estimate the model parameters. Because of such computational requirements, these algorithms are not practical when we wish to construct an HMM model with large state space and large explicit state duration and process a large amount of measurement data to obtain high accuracy. We propose a new forward-backward algorithm whose computational complexity is only O((MD + M/sup 2/)T), a reduction by almost a factor of D when D > M and whose memory requirement is O(MT). As an application example, we discuss an HMM characterization of access traffic observed at a large-scale Web site: we formulate the Web access pattern in terms of an HMM with explicit duration and estimate the model parameters using our algorithm.

The criterion that existing prefetch schemes apply in prefetching documents from the origin servers into a proxy Web server is usually the probability of each document being accessed in the near future or the popularity of the document. This criterion is not optimum in minimizing the average access latency or maximizing the average hit probability because the factors that affect the latency and the hit probability also include the response time and the updating cycle of the documents. We derive expressions for the average latency, hit probability, cache capacity and required bandwidth for a general prefetch scheme. A new prefetch scheme that combines access probability, response time and updating cycle to determine the lowest average latency or the highest hit probability is proposed. Finally, some numerical results are presented. The required parameters of the prefetch scheme can be simply derived from the log data of the cache. Thus, our scheme can be implemented in practice.