Apache Kafka is a distributed data streaming platform that is widely adopted in the industry. Producers introduce messages to Kafka server nodes, and consumers fetch interested messages in real time. In Kafka, the loads of nodes that serve streaming traffic may be imbalanced because some traffic is relatively popular, resulting in load imbalance. Balancing loads among nodes enables producers and consumers to avoid performance bottleneck, shortening the latencies of sending and receiving their messages and thus improving quality of service. Partitions are the fundamental entities that serve loads in Kafka and are responsible for hosting message payloads. Kafka provides a built-in load balancer that addresses load imbalance. The built-in load balancer distributes partitions evenly to the system, which is not designed to deal with load imbalance due to the heterogeneity of loads. The state-of-the-art load balancer released by LinkedIn, namely, Cruise Control (CC), complements the Kafka built-in load balancer. The performance quality of CC is sensitive and highly depends on the ordering of performance metric constraints. We propose Yet Another (YA) load balancer for Kafka, aiming to achieve simplicity and robustness for the average case. We compare YA with the Kafka built-in load balancer and CC in a real cluster environment. Performance results validate our findings and indicate that load balancers based on gathered performance metrics can effectively reduce the end-to-end delay perceived by streaming applications up to a ratio of approximately 20. CC and our proposed load balancer clearly outperform the Kafka built-in load balancer, yet while our proposed load balancer is comparable to CC, it is more robust.

Not Only SQL (NoSQL) is a critical technology that is scalable and provides flexible schemas, thereby complementing existing relational database technologies. Although NoSQL is flourishing, present solutions lack the features required by enterprises for critical missions. In this paper, we explore solutions to the data recovery issue in NoSQL. Data recovery for any database table entails restoring the table to a prior state or replaying (insert/update) operations over the table given a time period in the past. Recovery of NoSQL database tables enables applications such as failure recovery, analysis for historical data, debugging, and auditing. In this paper, we first identify the design and implementation issues with regard to the data recovery problem for NoSQL databases, including time length of recovery, fault tolerance, scalability, memory constraint, software compatibility, and quality of recovery. Particularly, our study emphasizes on columnar NoSQL databases. We then propose and evaluate four solutions to address the data recovery problem in NoSQL; each solution has its pros and cons. We implement our solutions based on Apache HBase, a popular NoSQL database in the Hadoop ecosystem widely adopted by industry. Our implementations are extensively benchmarked with an industrial NoSQL benchmark under real environments. Specifically, our research findings and implementations in this paper have been contributed to and integrated with Apache HBase for global distribution.