Kafka consumer skipped a failing message after stop/rebalance

[Anwert]

Environment

• Managed Kafka cluster (Confluent Cloud)
• Topic with multiple partitions
• Multiple replicas in the same consumer group

---

Relevant configuration

Error policy:

policy
    .Retry(retryCount, initialDelay, delayIncrement)
    .ThenStop();

• EnableAutoCommit is left at Confluent default (true).

---

Summary

We saw a single incident where a message at offset N consistently failed, the consumer stopped, and after rebalance the group resumed from N+1 (skipping N). Before that, we had two normal stop/restart cycles where the failing message at N was retried again as expected. We cannot reproduce it so far.

---

Timeline (from logs)

For partition P, offset N:

1. P@N is consumed, processing starts.

2. Business code throws on every attempt.

3. Silverback retries multiple times; logs show the same offset P@N on each attempt.

4. After the last attempt, Silverback logs fatal stop:

   • “Fatal error occurred… The consumer will be stopped… offset: [P]@N …”

5. The very next log line is the rebalance revoke:

   • Incident: “Revoked partition [P] (offset was N+1)”
   • Normal/expected when consumer stops after failures:
     “Revoked partition [P] (offset was -1001)”

6. The partition is assigned to another consumer in the same group, and it resumes from N+1, effectively skipping P@N.

---

Questions / pointers

At the moment the only thing that stands out to us is auto-commit. Could auto-commit (EnableAutoCommit=true) realistically cause the committed group offset to advance to N+1 in this kind of “retry → stop → immediate rebalance” flow, even though the message at N never succeeded?

If auto-commit alone shouldn’t explain this, do you have any ideas what else could affect the committed offset / starting position after rebalance in Silverback/Confluent.Kafka (e.g., revoke/assign behavior, commit-on-revoke, shutdown sequence, etc.)?

---

Thanks a lot in advance for any guidance.

[BEagle1984]

Thanks for the detailed report.

Regarding EnableAutoCommit: by itself this should not explain the behavior you’re seeing.

In Silverback, EnableAutoOffsetStore is disabled. This means offsets are never advanced just because a message is polled. Offsets are explicitly stored (and therefore eligible for commit) only after successful message processing via the consumer transaction commit path.
If a message at offset N never succeeds, Silverback should never store N+1, regardless of auto-commit being enabled.

So auto-commit alone should not be able to skip a failing message.

This is the first time this issue has been reported, and I agree it’s a strange scenario. I’ll investigate this further.

For context, in Silverback v5 I made several changes to streamline the consumer loop and make the transaction handling more robust, specifically to reduce subtle race-condition edge cases.

To help narrow this down, a few questions:

• Has this happened more than once, or was it a single isolated incident?
• Are you processing messages independently per partition, or using batch processing?
• Which Silverback version are you running?

If you manage to reproduce it or notice a pattern, that would be extremely helpful. I’ll also review the stop and rebalance paths to verify that we never end up storing/committing N+1 unless N has been successfully processed.

Thanks again for the thorough report.