A message queue has one (1) job: to keep track of the already-processed message pointer.

Which Kafka doesn't do. So you either store everything forever (lol) or you write some sort of broken half-baked solution for a message queue on top of Kafka. (Broken and half-baked because you're not going to achieve fault tolerance or consistency without re-implementing the storage layer.)

Now, you're just gonna say that "Kafka isn't a message queue". Well, I don't need half of a solution that isn't even a message queue. Nobody needs that.

Then use a message queue, not a log?

Honestly, probably a lot of Kafka woes (as a bystander) come from people using it as message queue when it's not one.

There is no other use case for Kafka except using it as a message queue. That's by design. It's just an extremely poorly designed piece of software.

Considering that the core behaviour of message queue (like RabbitMQ, IBM MQ, etc) and streaming log system (like Kafka) is wildly different, I'd rather claim that the very idea of using Kafka as message queue is the core problem.

You can use a distributed log to approximate a dedicated message router (which is honestly what "message queue" systems actually are - the queue is an artifact of limited capacity not required behaviour) but such uses are going to be wrong 9 out of 10.

OTOH if you want multiple readers observe same event stream, including across time dimension, not just receive messages, then message queue systems are going to be wrong solution and systems like Kafka are going to be good options.

Both have their pros and cons, both have their uses, both are shit solution when you need the other.

The more important question is "which of those, if any, you actually need".

100%

Kafka isn't a message queue. If someone told you that it was, they were wrong. It's an event/stream processor.

I do not think it is a shit arch. I think that is an arch for some problems, but people insist on using it for uses cases it was not built for.