Support for replication using MirrorMaker 2

[Jimvin]

Run MirrorMaker 2 as a Stacklet

We do have NiFi in the stack which can be used for most data moving operations but it'd be overhead if all you need is to mirror two Kafkas.
So, this is about adding native support for MirrorMaker2 to SDP.

This is also useful for migrating an existing Kafka into Stackable.

FYI: We have internal notes from #923 in docs/modules/kafka/examples/mirror_maker/ which might or might not help but need updating with this either way.

Scope

• A CRD that runs MirrorMaker 2 in dedicated mode, replicating between Kafka clusters
  • See the open questions below
• Docs and an integration test replicating between two SDP Kafka clusters.
• Update/delete docs/modules/kafka/examples/mirror_maker/

Expected effort

10 days.
If it takes longer, stop and let me know.

Out of scope

• Kafka Connect
• Per-connector actions: restart, pause, resume, stop, offset rewind

Open questions

• CRD name (MirrorMakerCluster?)
• Do we need Discovery 2.0 for this?
• How elaborate is the CRD: do we model the connector options as real structs or is all of that done using overrides.
• dedicated.mode.enable.internal.rest defaults to false and is required above one replica. Always on, or only when replicas exceed one?
• Kerberos?
• Both directions in one CR, or one CR per direction?

Possible approach

Strimzi implements this differently than I would and Claude believes that this is because of KAFKA-9981 which was still open at the time they implemented it and which would make the dedicated mode I propose here unusable. They are building a Kafka Connect cluster with MirrorMaker config.
I propose we use the dedicated MirrorMaker mode.

Kafka Connect is purely configured using a REST API so we'd need to have the whole access delegation story and a client in the operator to talk to connect to configure it etc.
MirrorMaker can be configured using a file. That seems way more convenient.

Downside(ish): Connect exposes a web server with read & write stuff (status and pause/resume etc.). MirrorMaker does not. But we could easily patch those lines in. The patch looks simple. I don't want that for v1 though.

This is the direction I'd prefer. If you disagree tell me before implementation starts.

Notes

Note

These are notes that came out of the Claude research

• Connect and MirrorMaker 2 are already in our images
• In dedicated mode the properties file is the source of truth. MirrorHerder extends
  DistributedHerder and, on becoming leader, diffs the desired connector configs against the
  config topic and replaces them. A changed properties file therefore takes effect on restart or leader change which I believe matches our commons operator restarter pattern
• One JVM holds a Worker and a herder per replication flow, so several flows do not require
  several sets of pods.
• Connector and task state is available as a metric (connector-metrics and
  connector-task-metrics, attribute status), which covers most of what a CR status would want
  without any REST API. The failure trace is not a metric; it is in status.storage.topic.

[soenkeliebau]

That all sounds very good!

I think, we are in kinda the same boat regarding Kafka Connect right now, that Strimzi was in when implementing MirrorMaker, there is KIP-304 which suggest the same mode for Connect, but I don't think that ever went anywhere.