Automation is desirable for a vast number of reasons. The Confluent Operator pattern attempts to capture some of those benefits by automating the activities typically completed by a human managing a service or set of services.

Typically, human DevOps teams or Cloud Engineers would oversee operations of software applications in cloud like:

• Ensuring systems operate as expected
• Defining and managing deployments
• Monitoring and handling incidents and failures
• Executing recovery plans as part of a disaster

In the interests of efficiency and consistency, the above activities led to the development of the Operator pattern. This pattern allows you to automate tasks, that are managed by manual operators.

The Operator model provided by Confluent offers the ability to deploy and manage the Confluent platform as a cloud-native, stateful container application in a Kubernetes environment. This model allows you to automate key lifecycle operations while simplifying the provisioning of the Confluent platform cluster. 

The Benefits of Confluent Operator

Automated Operations for Infrastructure Provisioning

Automating operations and provisioning gives you the advantage of consistency matched with speed and reduced manual effort.

Faster infrastructure setup

The entire process of setting up your Confluent platform can be accelerated with Operator, automating a large number of cluster provisioning activities, including:

• Setting up brokers
• Managing persistent volumes and local disks
• Broker network mapping
• Monitoring for Kafka components and Zookeeper setup
• Schema registry
• REST proxy

Environment Parity

Environment parity strives to keep tiered environments (development, staging, production, etc.) as similar as possible, minimizing gaps resulting in inconsistent testing and performance. Confluent Operator simplifies the deployment of the cluster across multiple availability zones and racks, keeping the deployment configurations very similar but allowing for slight differences in configuration parameters like persistent volume sizes and memory size.

Easy Cluster Management and Operations

Capacity planning & elastic scaling

Confluent Operator make broker capacity planning easier. Simple configuration changes accomplished through Operator lean into the dynamic nature of the architecture and the automation elements of the tool.

For example, if you wanted to expand the storage volume of your Kafka broker to 100 Gi, you could just update the dataVolumeCapacity in your Confluent component Custom Resource (CR).

kind: Kafka

metadata:

  name: kafka

Spec:

  dataVolumeCapacity: 100Gi

Smoother disaster recovery operations

Infrastructure as Code (IaC) and automation go hand in hand, making disaster recovery fast and convenient. You can automate disaster recovery (DR) activities with Confluent Operator based on your strategy, such as Warm Standby or Active-Active. Disaster recovery strategies can be further automated to trigger based on downtime monitoring alerts, which is a distinct advantage over manual, partially automated DR processes. 

Better release management

Rolling upgrades have the advantage of staggering releases to minimize downtime, but it can be burdensome to support two versions of an application during the process. By automating rolling upgrades with Operator, the release management of new versions of all the components is streamlined, eliminating the painful and time-consuming process and minimizing downtime.

Simple, Automated Security Provisioning

Cluster configuration security, too, can be automated with Operator through SASL PLAIN, SASL_SSL, or TLS with mutual authentication. We've already noted that Operator is an excellent solution for provisioning - this applies to security artifacts as well. Operator can automate the provisioning of trust stores that manage the certificates clients should trust and key stores, where the private keys for the certificates reside.

Cloud Agnostic

With the non-Kubernetes version of Kafka, different code would be required for different cloud providers. This could mean additional maintenance overhead and factor into provider shifting and lock-in decisions. However, Confluent Kafka on Kubernetes is cloud-agnostic. Therefore, the same code can be used for deployment on a multitude of infrastructures, including AWS, GCP, Azure, and even on-prem.

Conclusion

When using Confluent in a Kubernetes environment, adopting the Confluent Operator will allow you to realize all of the above benefits. That's why we recommend it over other alternatives like Terraform, Ansible Tower, Chef Cookbook, and SaltStack. If you're struggling with how to implement or incorporate Confluent Operator in your environment or would be interested in learning more about how the Operator model works, contact us. Big Compass's team is experienced in implementing Confluent Operator in your Kubernetes environment and would be happy to discuss and guide your adoption and deployment of Operator in your Kubernetes system.