Flow Logs Pipeline (a.k.a. FLP) is an observability tool that consumes flow logs from various inputs, transforms them and exports logs to Loki and / or time series metrics to Prometheus. While flow logs encompass a lot of valuable data, observing the network from the level of flow logs is often too low. In many cases, we are interested in observing it from a higher level, the level of connections. In this work, we introduce a new processing stage in FLP that allows aggregating flow logs from the same connection - connection tracking.

Flow-Logs Pipeline is an observability tool that consumes raw network flow-logs and transforms them from their original format (e.g. NetFlow or IPFIX) into numeric metrics format. FLP allows to define transformations of data to generate condensed metrics that encapsulate network domain knowledge.

The management of large data centre (DC) network infrastructure confronts Network Reliability Engineers (NRE) with challenges. A single DC at a modern cloud services provider can host thousands of network devices. The syslog messages generated by these devices are an important type of monitoring data to detect and diagnose failures. Devices in a single DC produce millions of syslog messages per day in a variety of formats.