Modern large-scale computer networks generate massive amounts of log data due to their increasing size, usage, and complexity. At the same time, as cloud-based businesses continue to grow, the need for services and software dedicated to log analysis is more important than ever. Although very useful, log messages often lack the necessary details for efficient troubleshooting, requiring extensive human analysis of the source code. In this paper, we present a new architecture designed with performance in mind, capable of identifying links between software-generated logs and their logging function calls in the source code (referred to as "origins" of the logs). The system we propose uses static code analysis to generate exact log templates, which are used to match log messages efficiently using a combination of a prefix tree and regular expressions. Our implementation SCOLM can pinpoint the origin of log messages with excellent performance and success rate. SCOLM can parse nearly 1 million log lines per minute on a single thread, with a match rate of 90 to 100% on our datasets. It outperforms the speed of traditional regex-based approaches, reducing the speed by about 98.7% in our experiments. The applications of this system are numerous, including live troubleshooting and statistical event analysis.

The NTP Pool project has become a critical infrastructure for Internet time synchronization, and used by major Linux distributions, router firmware, and IoT devices by default. While IPv6 servers are inherently difficult to locate by Internet-wide scanners due to the vast address space, public IPv6 time servers in the NTP Pool can be exposed to the Internet through GeoDNS. As these servers are operated by volunteers, they may have limited security measures, making them particularly vulnerable and attractive targets for IPv6 scanning activities. In this paper, we monitor and analyze scanning activities targeting IPv6 time servers in the NTP Pool. We present an approach to distinguish between benign scans from legitimate NTP clients and suspicious scans from potentially malicious scanners. Our analysis spans several metrics including source, traffic composition, target distribution, and scanning strategies. We find that the suspicious scanners can discover newly deployed NTP servers across different regions within several hours and probe neighboring addresses for reconnaissance. In addition, our honeynet observations suggest that the scanners employ adaptive scanning strategies that focus on responsive targets. Our findings provide comprehensive insights into scanning activities targeting IPv6 NTP Pool servers, contributing to the understanding of IPv6 network security in the critical infrastructure.

Network emulation is an important component of a digital twin for verifying network behavior without impacting on the service systems. Although we need to repeatedly change network topologies and configuration settings as a part of trial and error for verification, it is not easy to reflect the change without failures because the change affects multiple devices, even if it is as simple as adding a device. We present topology-driven configuration, an idea to separate network topology and generalized configuration to make it easy to change them. Based on this idea, we aim to realize a scalable, simple, and effective configuration platform for emulation networks. We design a configuration generation method using simple and deterministic config templates with a new network parameter data model, and implement it as dot2net. We evaluate three perspectives, scalability, simplicity, and efficacy, of the proposed method using dot2net through measurement and user experiments on existing test network scenarios.

Network emulation is an effective approach to ensure sustainable and reliable network services by verifying the correctness and fault tolerance of them. However, deploying and modifying emulation networks with existing platforms is time-consuming and prone to cause configuration errors because existing emulation platforms do not provide a suitable method for scalable network configuration. To overcome this problem, we propose the design and implementation of dot2net, a template-based platform for simple, scalable, and expressive configuration of emulation networks. The key idea is to separate network configuration into network topology as a labeled graph and label definitions as config template blocks. We evaluate the performance and efficiency of config file generation and show that dot2net is particularly effective at scaling the network topologies. We also demonstrate the expressiveness of dot2net for complicated networks and advanced technologies with test emulation networks of FRR, a widely used router software.

Towards a collaborative analysis of log data obtained from multiple networks, we first need to clarify what kind of information is available as transferable knowledge between different networks. However, we cannot directly compare net-work log data from different sources because the data largely depends on the network architecture and equipment. In this paper, we focus on relational information among network log events that follow standardized network protocols regardless of network environment. We propose a comparative analysis approach relying on causality between log time-series. In this approach, we classify log messages into anonymized log time-series with log templates, reduce the number of log time-series to decrease processing time, and apply causal discovery with the PC algorithm. To decrease the processing time of causal analysis, we propose a new preprocessing method that reduces the number of log time-series without any domain knowledge (i.e., available in any ISPs). We compare log data obtained from two nation-wide ISPs to demonstrate the effectiveness of the causal approach in comparative analysis.

Data logs from network devices are primary data to understand the current status of operational networks. However, since many and heterogeneous devices generate network logs, extracting information on the network status from such logs is not an easy task in network operation, e.g., root cause analysis of network events. Though multi-variate time-series based log analyses extract correlation structure of the logs, identifying causality of the network logs is still a complex and challenging problem. The state of the art algorithm called the PC algorithm had been applied to network log analysis, but it has two fundamental limitations; (1) Generated graphs still have many undirected edges, and (2) Edges have no weight (whether plausible causality or not). To overcome these two limitations, in this paper, we rely on MixedLiNGAM to network log analysis; This algorithm produces weighted DAGs from a set of multivariate log time series. In order to show the effectiveness of the proposed method, we apply MixedLiNGAM to a set of syslog data collected at a research and education network in Japan, and then compare output causal graphs generated by MixedLiNGAM and the PC algorithm. Our result demonstrates that obtained weighted directional edges help better understand the root cause of the network events.

Configuring network devices is a main task of network operators. However, understanding and consistently updating network configuration files (config) is not an easy task especially in a large-scale and complicated networks. In this paper, we propose a semantic approach to provide better understanding of such config files, different from syntax based approaches. The key idea of the work is to extract semantics of blocks of the config files by document embedding techniques in NLP. This extraction enables us to understand context of config blocks with semantic similarity metrics instead of syntax similarity ones. Furthermore, this approach can be naturally extended to additional technical documents such as vendor’s manual documents to add more specific information on the semantics of configs. We first discuss the quality of the obtained semantics for several embedding techniques, by using clustering evaluations. We next demonstrate the effectiveness of our approach with two case studies with real network configs: (1) similar config block detection and (2) automatic labeling of config block with vendor’s documents.

One of the ways to analyze unstructured log messages from large-scale IT systems is to classify log messages with log templates generated by template generation methods. However, there is currently no shared knowledge pertained to the comparison and practical use of log template generation methods because they are implemented on the basis of diverse environments. To this end, we design and implement amulog, a general log analysis framework for diverse log template generation methods. There are three key functions of amulog: (1) parsing log messages into headers and segmented messages, (2) classifying the log messages using a scalable template-matching method, and (3) storing the structured data in a database. This framework helps us easily utilize time-series data corresponding to the log templates for further analysis. We evaluate amulog with a log dataset collected from a nation-wide academic network and demonstrate that it works in a reasonable amount of time even with over 100,000 log template candidates.

To detect root causes of failures in large-scale networks, we need to extract contextual information from operational data automatically. Correlation-based methods are widely used for this purpose, but they have a problem of spurious correlation, which buries truly important information. In this work, we propose a method for extracting contextual information in network logs by combining a graph-based causal inference algorithm and a pruning method based on domain knowledge (i.e., network protocols and topologies). Applying the proposed method to a set of log data collected from a nation-wide R & E network, we demonstrate that the pruning method reduced processing time by 74% compared with a single-handed causal analysis method, and it detected more useful information for troubleshooting compared with an existing area-based method.

Network log messages (e.g., syslog) are expected to be valuable and useful information to detect unexpected or anomalous behavior in large scale networks. However, because of the huge amount of system log data collected in daily operation, it is not easy to extract pinpoint system failures or to identify their causes. In this paper, we propose a method for extracting the pinpoint failures and identifying their causes from network syslog data. The methodology proposed in this paper relies on causal inference that reconstructs causality of network events from a set of time series of events. Causal inference can filter out accidentally correlated events, thus it outputs more plausible causal events than traditional cross-correlation-based approaches can. We apply our method to 15 months' worth of network syslog data obtained from a nationwide academic network in Japan. The proposed method significantly reduces the number of pseudo correlated events compared with the traditional methods. Also, through three case studies and comparison with trouble ticket data, we demonstrate the effectiveness of the proposed method for practical network operation.

Network log message (e.g., syslog) is valuable information to detect unexpected or anomalous behavior in a large scale network. However, pinpointing failures and their causes is not an easy problem because of a huge amount of system log data in daily operation. In this study, we propose a method extracting failures and their causes from network syslog data. The main idea of the method relies on causal inference that reconstructs causality of network events from a set of the time series of events. Causal inference allows us to reduce the number of correlated events by chance, thus it outputs more plausible causal events than a traditional cross-correlation based approach. We apply our method to 15 months network syslog data obtained in a nation-wide academic network in Japan. Our method significantly reduces the number of pseudo correlated events compared with the traditional method. Also, through two case studies and comparison with trouble ticket data, we demonstrate the effectiveness of our method for network operation.

This letter deals with the backscattering of millimeter pulsed beam waves from hydrometeors. A new approach is presented for a solution of time-dependent three-dimensional vector radiative transfer equation for the Stokes vectors to study the multiple-scattering effects of beam waves on radar echoes. General solutions for beam waves are derived in an integral form without any approximation. They are given in numerically tractable forms representing the scattering process in the space and time domain. Time-dependent second-order solutions for radar echoes of pulsed beam waves are straightforwardly obtained to predict multiple-scattering effects depending on the variation of an incident beam size. It is shown that the inhomogeneity of the radial direction of beam waves causes the mode coupling of waves between the azimuth directions in the scattering matrix, and that the mode coupling depends on the ratio of the incident beam size to the total mean free path length of the medium

In this paper, we expand the previous theory to be applied to a generic drop size distribution with spheroidal raindrops including spherical raindrops. Results will be used to discuss the multiple scattering effects on the backscatter measurements acquired by a W-band nadir-pointing radar.

This paper outlines the development of the dual-frequency precipitation radar (DPR) to be flown on the Global Precipitation Mission's spacecraft. I. INTRODUCTION In the Global Precipitation Mission (GPM), a dual- frequency precipitation radar (DPR) is planned to be flown on the spacecraft. The spacecraft serves as a high quality reference platform for training and calibrat- ing the rain retrieval algorithms used with the passive mi- crowave radiometers on the other constellation satellites. The dual-frequency radar is expected to provide accu- rate estimates of rainfall rate as well as drop size distribu- tion (DSD) parameters from the combination of Ku- and Ka-band radar returns. This paper outlines the present status of the DPR development. Following this introduc- tion, we discuss the critical issues that affect the designing of the DPR. II. DPR REQUIREMENTS A. Relevance of the DPR to GPM The relevance of the DPR to GPM lies in the radar's ca- pability of measuring storm structure, rainfall rates, drop- size distribution (DSD), path-integrated attenuation, and other useful parameters that cannot be obtained by pas- sive sensors. In the latest design, the DPR is composed of Ku-band and Ka-band channels. The Ku-band radar is approxi- mately the same as the TRMM Precipitation Radar (PR) with some improvements. The Ka-band radar provides high sensitivity to light rain and snow. The combination of data from two channels will provide accurate estimates of drop-size distribution parameters. The Ka-band radar will sample the echo data in two different modes simulta- neously. One is a high-sensitivity mode for light rain and snow detection, and the other is a matched-beam mode in which the sampling volumes of Ka- and Ku-band radar channels are matched for collecting dual-frequency echoes from the identical targets. The data collected in the lat- ter mode are used for the estimation of DSD parameters. In the matched-beam mode, a range resolution of 250 m is employed, while in the high-sensitivity mode, a range resolution of 500 m is planned. The current radar design adopts active phased array antennas in both radar chan- nels to make full use of TRMM experience. The DPR will provide three-dimensional information of hydrometeor distribution with high spatial resolution. Such data are very valuable for the study of storm struc- ture. The accurate rainfall estimates from the DPR are expected to be used for calibrating the corresponding esti- mates from the radiometer on the core satellite. The major importance of the DPR, however, lies in the fact that it can provide the regional and seasonal statistics of storm structure together with DSD parameters. Since rain re- trieval algorithms for passive microwave radiometers have to assume a vertical structure of storm either determinis- tically or statistically, reliable storm structure information is crucial for the accuracy of rain estimation. The statis- tics from the DPR can be used as a database in radiometer algorithms to reduce the uncertainties of the storm mod- els. How to utilize the information from radar data is a challenging issue. A possibility of improving the database used in a TMI rain retrieval algorithm by using TRMM's PR data is currently under examination. The DPR has three main roles in GPM. It will provide three-dimensional information of rain structure. The Ku- band radar is similar to, but not exactly the same as, the TRMM Precipitation Radar (PR). It is improved from the PR. The improvement is necessary because of three rea- sons. Firstly, the proposed orbit of the GPM core satellite is about 400 km and higher than the TRMM's orbit. This necessitates the improvement of the sensitivity to com- pensate for the increased range loss. The designed trans- mitting power is increased to 1000 W from PR's 500 W. (the actual Tx power of the PR turned out to be about 800 W.) Secondly, the orbital inclination is about 65 de- grees and larger than the TRMM's 35 degrees. Because of the oblate shape of the Earth, the altitude of the satel- lite changes more than 20 km at a 65-degree orbit which is much larger than 10 km at a 35-degree orbit. If we use a constant pulse repetition frequency (PRF) like the TRMM PR, we have to use a rather small PRF to absorb this large variation of rain echo range from the radar. The low PRF will result in a low signal-to-noise ratio. To max-

Pulse patterns are designed by adopting variable pulse repetition frequency (VPRF) for space missions involving beam scans in the cross-track direction, especially intended for the Global Precipitation Measurement Project (GPM). To cope with large variance in range from a satellite, which is caused by the beam swing and the Earth oblateness, a systematic algorithm is proposed, increasing sampling rates.