Abstract In this article, a new algorithm that can encrypt data on TCP data segment by using a novel chaotic random number generator was developed. First of all, dynamic analyses of the chaotic system to be used were made and the original pseudo‐random number generator (PRNG) with a new approach was designed. Key series produced by PRNG was tested on NIST and passed all the tests. The chaos‐based encryption here was performed on TCP, one of the most fundamental protocols in data communication. Following this, TCP data segments were encrypted by the sender on bit base with key series produced by PRNG, and then they were decrypted by the receiver. Security level of the chaos‐based encryption carried out here was tested with histogram, its key length, key sensitivity and entropy method. Memory and time usages were measured in performance analysis. So as to reveal the performance of the encryption, memory usage, encryption and decryption times were analyzed by comparison on different security algorithms. According to tests and analyses results, this chaos‐based encryption has a high level of performance and security; and it is an original study that can be used in real‐time implementations with TCP protocol. Copyright © 2016 John Wiley & Sons, Ltd.