3

I’m doing a POC to deploy Kafka as an application on Mesos Cluster. I came across these 2 codebases on github. One developed by apache-mesos (github page) & other developed by mesosphere and can run only on DCOS (github page).

Question: Would like to know if there are any differences between DCOS-Kafka & mesos-Kafka in terms of features and extended functionality.

Regarding Mesos-Kafka: I don’t see active participation on github (and some open issues) for mesos-kafka in the past months. Can I assume that the service is robust enough that I can use in production environment? Any Inputs on this would be helpful.

satya karthik
  • 58
  • 6

1 Answers1

0

kakfa-mesos is a package that includes a release of Kafka and a custom mesos scheduler that was meant to work around issues with running Kafka as a stateful service on Marathon. I think post but confluent is useful. It also includes a RESTful api for doing ops tasks and aims to include these features in the future (this is pulled from the article I linked)

  • Integrating Kafka commands (e.g. kafka-topics, etc) into the Scheduler so it can be used through the CLI and REST API.
  • Auto-scaling clusters (including auto reassignment of partitions) so that the resources (CPU, RAM, etc.) that brokers are using can be used elsewhere in known valleys of traffic.
  • Rack-aware partition assignment for fault tolerance.
  • Hooks so that producers and consumers can also be launched from the Scheduler and managed with the cluster.
  • Automated partition reassignment based on load and traffic

I haven't used it in a production environment myself but it has the support of Confluent which is a good sign.

DC/OS Kafka on the other hand is a DC/OS service which will probably only be useful if you are already running or plan on running services through Mesosphere's DC/OS. It also includes an API and a CLI management tool but is less ambitious with additional features. It's current feature set includes

  • Single-command installation for rapid provisioning
  • Multiple clusters for multiple tenancy with DC/OS
  • High availability runtime configuration and software updates
  • Storage volumes for enhanced data durability, known as Mesos Dynamic * * Reservations and Persistent Volumes
  • Integration with syslog-compatible logging services for diagnostics and troubleshooting
  • Integration with statsd-compatible metrics services for capacity and performance monitoring
William Hammond
  • 462
  • 3
  • 10