when I increase the number of partitions then How will kafka redistribute the existing messages among new partitions?
It will not redistribute the existing messages.
Is it possible to trigger this redistribution?
I am not aware of anything that makes this possible. Keep in mind, that placing messages into particular partitions will ensure the ordering of those messages within a partition. As this could be a quite essential requirement for your appliication it would be very dangerous to shuffle around messages from existing partitions.
If so, then How will kafka come to know about the custom partitioner as that piece of code resides in the producer part?
Exactly, Kafka does not even have the knowledge how to balance the existing messages accross old and new partitions. It could only be done on a random basis which would be quite dangerous for ordering of the messages (see answer to second question).
We can specify custom partitioner for kafka topics.
Just wanted to emphasize that a custom partitioner is always used at a producer level and you cannot specify a partitioner for a topic. Imagine the valid scenario where you have multiple producers writing to the same topic. Each producer could have an individual partitioning logic.
