Sent Is Not Done: How Message Queues Really Work

November 5, 2026 · 39 min read

From message delivery to business completion: reliability, ordering, and failure recovery in distributed systems.

If you have any questions, feel free to comment below. Click the block can copy the code.
And if you think it's helpful to you, just click on the ads which can support this site. Thanks!

Message queues can be used for decoupling, asynchronous processing, and absorbing traffic spikes, but introducing them also means handling duplicate consumption, message loss, out-of-order processing, and backlogs. Knowing how to call the send and consume APIs is not enough to understand their role in a system.

This article starts with why to use MQ, then covers high availability, idempotency, reliable delivery, ordering, and backlog handling in sequence, before bringing these mechanisms together to examine message queue architecture design.

1. Why Use Message Queues? #

1.1 Why Use Message Queues #

There are three common use cases for message queues: decoupling, asynchronous processing, and absorbing traffic spikes.

Decoupling

Suppose system A produces data that systems B, C, and D all need. When sending it directly through APIs, A must maintain calls to each downstream system:

A → B
A → C
A → D

If E also needs the data, A must add another call; if D no longer needs it, A must change its code. When a downstream system times out or fails, A must also consider retries and failure handling. As more subscribers are added, the system producing the data takes on more work unrelated to its own business.

With MQ, A is responsible only for publishing messages, and systems that need the data subscribe independently:

A → MQ → B, C, D

Adding or removing subscribers no longer requires changing A’s calling logic for each one. This is the decoupling provided by the publish/subscribe model.

Direct calls: A maintains each downstream API; adding E or removing D requires changing A

Figure: With direct calls, adding or removing a downstream system requires changing A; dashed lines indicate the call relationships being changed.

Publish/subscribe: A publishes only to MQ; B and C subscribe independently, while D can leave and E can join without changing A

Figure: Publish/subscribe allows downstream systems to join or leave independently. Different business functions subscribe separately rather than competing to consume the same task.

Note that this primarily removes the dependency on direct calls. Both sides still need to agree on message fields and business semantics; coupling is not eliminated entirely. If different business functions all need to receive a message, configure independent subscriptions rather than having them compete to consume the same task.

Asynchronous Processing

Suppose that after receiving a request, A needs 3ms for a local database write and must also call B, C, and D, taking 300ms, 450ms, and 200ms respectively. If it waits for these calls sequentially, the total time is:

3 + 300 + 450 + 200 = 953ms

If B, C, and D do not need to finish processing within the current request, A can complete its local operation and then send messages for downstream systems to process asynchronously. Assuming sending the messages takes 5ms, the main request path is shortened to:

3 + 5 = 8ms

The point of this example is not the exact number of milliseconds, but moving operations that do not need to complete immediately out of the synchronous waiting path.

Synchronous calls: the user requests A, which completes its local write and waits for B, C, and D to finish before returning

Figure: A calls B, C, and D in sequence. The arrows show the order of the calls, not B calling C or C calling D. The green return path reaches the user only after D finishes; the 953ms figure assumes sequential calls.

Asynchronous calls: A publishes messages to three MQ queues, which B, C, and D consume respectively

Figure: Publishing three messages takes 5ms in total. The green curve indicates an early return to the user, while dashed lines indicate background consumption, not delivery failure. The 8ms response does not include completion of downstream business processing; node widths are not proportional to processing time.

An asynchronous response also has limits. A message having been sent indicates only that subsequent processing has been triggered, not that all downstream business operations have succeeded. The API should make clear whether its response means “accepted,” “local processing succeeded,” or “the entire workflow completed.”

Absorbing Traffic Spikes

Suppose traffic is normally just 50 requests per second but rises to 5000 per second at peak times, while the backend can sustainably process only 2000 per second.

If requests go directly to the backend, a sudden overload may slow database responses or even bring down the entire system. With MQ, requests can be buffered first and processed by consumers at a rate the database can handle.

Peak input: 5000 messages/second → MQ → Sustainable processing: 2000 messages/second

Messages accumulate during the peak; once it passes, spare processing capacity gradually drains the backlog.

Two prerequisites deserve particular attention:

• The peak is temporary; the long-term average input rate must not continually exceed processing capacity.
• Messages can wait, and MQ has enough capacity and retention time to accommodate the backlog.

At these rates, a one-hour peak adds:

(5000 - 2000) × 3600 = 10.8 million messages

If input drops to 50 messages per second after the peak while processing remains at 2000 per second, clearing this backlog takes about 92 minutes. Absorbing traffic spikes therefore does not eliminate work; it converts sudden pressure into waiting in a queue.

The 2000 operations per second here is an assumed sustainable backend processing rate, without a corresponding machine configuration or load-test record. It cannot be used to infer how many CPU cores and how much memory MySQL needs to handle 2000 operations. It means completing 2000 business operations per second, not necessarily executing 2000 SQL statements: one message may require multiple queries, updates, and transaction commits.

Even on machines with the same 4 cores, 8GB of memory, and SSD storage, processing capacity can differ greatly between updating one row by primary key and scanning large amounts of data or contending for the same row lock. An actual estimate requires fixing the machine configuration, data volume, indexes, SQL statements and transaction scope for each message, and disk-flush and replication settings, then gradually increasing concurrency to measure the sustained processing rate that meets latency and error-rate requirements. Consumer rate limits should be based on that rate rather than directly adopting 2000 messages per second.

Without MQ, peak requests pass directly through A to MySQL, exceeding the processing capacity assumed in the example

Figure: Peak requests go directly to the database at an input rate exceeding its processing capacity. The curve and threshold position illustrate only the relationship between burst traffic and capacity; they are not measured data and cannot be used to read exact rates.

With MQ, 5000 messages per second are buffered while A pulls 2000 per second and writes them to the database

Figure: MQ buffers input, and A pulls messages for consumption at a rate the backend can handle. The curve shows qualitative traffic changes; the four message cards illustrate the queued state, not an actual total of four messages. At the example’s constant rates, a one-hour peak adds 10.8 million messages; once input drops to 50 messages per second, clearing them takes about 92.3 minutes.

1.2 Advantages and Disadvantages of Message Queues #

The advantages are the decoupling, asynchronous processing, and traffic-spike absorption described above; the disadvantages are the system costs behind those benefits.

First, availability concerns. MQ becomes a new infrastructure dependency. If sending a message is a required step in the main workflow, MQ unavailability may affect the business. You must therefore consider MQ high availability and whether the business can continue accepting requests during a failure. On the other hand, MQ can also isolate brief downstream failures.

Second, increased complexity. Replacing synchronous calls with message delivery requires handling duplicate consumption, message loss, out-of-order processing, retries, and backlogs. These are precisely the topics discussed in the subsequent articles.

Third, consistency concerns. If A has completed successfully and returned a response but only some of B, C, and D succeed, the business may be temporarily or persistently inconsistent. Using MQ requires accepting the corresponding consistency model and preparing retries, compensation, and result verification.

1.3 Comparing Kafka, ActiveMQ, RabbitMQ, and RocketMQ #

Message queues can be compared in terms of throughput, the impact of topic count, latency, availability, reliability, and feature support.

The throughput and latency figures in the table are rough orders of magnitude from common historical sources, not results from a standardized load test using the same hardware, versions, and reliability settings, nor product limits; ActiveMQ here primarily refers to Classic.

2. How Do You Ensure Message Queue High Availability? #

2.1 RabbitMQ High Availability #

Single-Instance Mode

There is only one RabbitMQ instance. If it becomes unavailable, the associated messaging service is interrupted; there is no node-level fault tolerance.

Standard Cluster Mode

For a classic queue, for example, queue data resides on one node, while other nodes share the relevant metadata. Even if a client connects to another node, that node may need to forward requests to the node hosting the queue.

If the node hosting the queue goes down, the presence of other nodes in the cluster does not by itself allow the queue to continue serving requests. If the data was persisted and the disk remains intact, it may be usable after the node recovers; this is different from being available during the failure.

A standard cluster therefore addresses issues such as connectivity and queue distribution. Distributing multiple queues across different nodes can share load and improve overall cluster throughput, but an unreplicated classic queue still has a single point of failure at the queue level. Consumer connection approaches also have different costs:

• Connect to any node: A consumer can randomly choose an instance to connect to; if the queue is not on that node, requests and messages must be forwarded across nodes, adding network and data-fetching overhead. Connections are typically reused rather than randomly reestablished for every fetch.
• Connect directly to the node hosting the queue: This reduces cross-node forwarding, but processing for that queue is still concentrated on one instance and limited by its performance. Adding other nodes does not automatically distribute the load of the same unreplicated queue.

In a standard cluster, only node 2 stores queue Q’s messages; a consumer connected to node 1 reads from node 2 through forwarding

Figure: Three nodes share queue metadata, but only node 2 stores Q’s messages. When the consumer accesses the queue through node 1, requests must be forwarded and messages returned.

Mirrored Cluster Mode

Traditional mirrored queues replicate the same queue’s data to multiple nodes, improving fault tolerance through replicas.

The cost is the network, storage, and synchronization overhead of message replication. Each replica stores the complete data, so adding replicas does not distribute a single queue’s data across storage locations or directly provide linear capacity scaling.

This section primarily helps explain older systems: classic mirrored queues were removed in RabbitMQ 4.0; new deployments should not continue copying mirrored-queue policies.

Historical mirrored queues replicate the complete queue to multiple RabbitMQ nodes rather than sharding its data

Figure: The primary replica replicates the complete queue to mirrored nodes. This explains a historical mechanism, not a recommended RabbitMQ 4.x deployment.

Quorum Queues

Quorum queues^[1] are a high-availability solution that uses Raft to replicate queue state.

• A queue has a leader and followers.
• The relevant writes obtain the corresponding safety guarantees only after majority acknowledgment.
• If the leader fails, a new leader is elected when the required conditions are met.
• A three-member queue can usually tolerate one member failing; it cannot continue operating normally after losing a majority.

Specify the type with x-queue-type=quorum when declaring the queue; you cannot switch its type using an ha-mode: quorum policy. An existing queue also cannot be converted directly into a quorum queue by changing a policy.

The leader replicates messages in a three-member quorum queue, acknowledging after at least two members persist them; the queue is unavailable without a majority

Figure: A majority in a three-member queue consists of two members. Here, the leader and follower A have persisted the message, satisfying the majority condition and allowing a publisher confirm to be returned without waiting for the slower follower B. If one member fails, the remaining majority can continue operating when election and recovery conditions are met; if two members fail, the majority is lost and queue operations cannot proceed normally.

2.2 Kafka High Availability #

Basic Hierarchy

Kafka’s basic hierarchy is:

Cluster → Broker → Topic → Partition → Replica

A topic can be divided into multiple partitions distributed across different brokers. Each partition can in turn have multiple replicas:

• The leader receives writes for the partition.
• Followers fetch data from the leader to replicate the log.
• If the leader becomes unavailable, a new leader is elected from eligible replicas.

Partitions therefore allow a topic’s data to be distributed across multiple machines; replicas provide a basis for restoring service for a partition after a machine fails.

With partitions but no additional replicas, three partitions reside on three brokers, and a node failure affects its partition

Figure: Partitions distribute data, but a single replica cannot provide partition failover.

Each Kafka partition has a leader and followers on other machines; eligible replicas can take over after the leader fails

Figure: Five brokers host replicas of three partitions. Brokers 1, 2, and 3 host the leaders of P1, P2, and P3 respectively; broker 4 hosts P1’s follower, and broker 5 hosts followers of both P2 and P3. A broker can host replicas of multiple different partitions, while a given partition’s leader and followers reside on different brokers. Producers write to the corresponding leader, and consumers also fetch messages from the leader.

Writes: The Leader Receives, Followers Fetch

Consider acks=all, a replication factor of 3, and all three replicas in the ISR (In-Sync Replicas)^[2]:

1. The producer sends to the leader: Using partition information, the producer sends the message to that partition’s current leader.
2. The leader appends to its local log: The leader appends the message to the local partition log. This write usually goes into the operating system’s page cache first; it does not mean every message has already been forced to physical disk through fsync.
3. Followers fetch actively: Followers continuously send Fetch requests to the leader, retrieving and appending missing log entries, rather than the leader actively pushing messages to followers.
4. The leader checks replication progress: Followers include their fetch positions in subsequent Fetch requests, allowing the leader to determine how far they have replicated. This can be understood as confirmation of replication progress, but it is not a separate additional ack for every message.
5. Write success is returned: With acks=all, the leader waits for all replicas in the current ISR to replicate up to the corresponding position before reporting success to the producer. In this example, the ISR contains all three replicas, so the leader and both followers must finish appending the log.

If a follower falls behind for an extended period and is removed from the ISR, the leader no longer waits for it unconditionally. min.insync.replicas specifies the minimum ISR size required for acks=all writes. For example, with a replication factor of 3 and min.insync.replicas=2, an ISR with two remaining members still satisfies the write conditions; if only the leader remains, writes fail. “All” in acks=all means the current ISR, not all configured replicas.

Producers can also change the acknowledgment mode: acks=1 waits only for the leader to append to its local log; acks=0 does not wait for the broker’s write acknowledgment. These settings change the acknowledgment conditions the producer waits for; they do not disable follower replication.

Consumption: Reading Only Committed Messages

On the default read path, consumers fetch messages from the partition leader, but cannot read all data the leader has just received before the required replication has completed. Kafka uses the high watermark (HW) to mark the boundary of the committed log: ordinary consumers can read only records with offsets below the HW.

In the three-replica example above, both followers are in the ISR, so both must replicate up to the corresponding position before the leader can advance the HW accordingly and make the messages visible to consumers. If a lagging follower has been removed from the ISR, advancement depends on the current ISR rather than continually waiting for all configured replicas.

Producer acknowledgment and consumer visibility are two different boundaries: even with acks=1 and a successful response already received by the producer, the message may not yet be in the range consumers can read. Conversely, a producer not receiving a successful response does not prove that the message was not committed; the response may have been lost in transit.

For transactional messages, consumers configured with isolation.level=read_committed are also constrained by the last stable offset (LSO): unfinished transactions limit the readable boundary, and records from aborted transactions are filtered out.

High availability does not mean “replicas guarantee no loss and no interruption”: replica lag, failover policies, and multiple replicas failing together all affect those guarantees. Kafka 4.x uses KRaft to manage metadata; not every configuration restricts reads to the leader.

3. How Do You Prevent Duplicate Message Consumption? #

3.1 Why Duplicate Consumption Happens #

Consider Kafka consumer offsets:

1. The consumer reads messages and completes business processing.
2. It prepares to commit its consumer offset.
3. The consumer process stops before the offset is committed.
4. After restarting, it resumes reading from the last committed position.
5. Some messages that have already been processed are consumed again.

This is not unique to Kafka. Other message queues can similarly redeliver messages when business processing succeeds but the consumption acknowledgment does not arrive.

The central issue is that business processing being complete and MQ knowing it is complete do not necessarily happen at the same time.

Kafka offsets 152 and 153 have been processed, but a crash before committing the next position, 154, causes reading to restart at 152

Figure: The offset is stored in __consumer_offsets and indicates the next position to read. If 152 and 153 have been processed but 154 has not yet been committed, a restart may cause already processed messages to be read again.

The focus is therefore not on assuming messages will never be duplicated, but on ensuring that repeated delivery does not produce duplicate business effects.

3.2 What Idempotency Means #

For the same business operation, executing it once or repeatedly should produce the same final effect.

For example, if a message to create a record is delivered repeatedly, the database should still end up with only one corresponding record; repeated delivery of a message awarding points must not award extra points.

“No change” here means repeated operations produce no additional effects, not that the first execution cannot modify data.

3.3 Approaches to Implementing Idempotency #

Checking or Updating by Primary Key

If a message inserts data, the business primary key can be used to check whether the record already exists, or an appropriate update approach can be used.

However, “check then insert” does not by itself protect against concurrency: two consumers may both find that the record does not exist. Unique constraints or atomic writes are still required; application-level queries alone are insufficient.

Using the Idempotent Nature of Overwrites

Consider Redis SET: repeatedly setting the same value generally does not create duplicate increments as an accumulation operation would.

This depends on business conditions, however. Replaying an old message may overwrite a newer state, and repeated writes may also change the expiration time, so this cannot be generalized to “all Redis writes are idempotent.”

Deduplicating with Unique Message Identifiers

The producer assigns a unique ID to a business event, and the consumer uses it to determine whether the event has already been processed.

Two points need attention:

• Send retries should reuse the same ID; otherwise duplicates cannot be identified.
• The ID should represent one event, rather than always using the order ID, because the same order may have multiple events such as creation, payment, and cancellation.

When storing processing records in Redis, note the failure windows between “check record → perform business operation → write deduplication marker.” Writing the marker first can instead leave a successful marker for an unfinished business operation.

Using Database Unique Keys

Unique constraints prevent duplicate inserts at the storage layer and are an important method to understand when implementing idempotency.

When the business changes are also in the same database, the deduplication record and business operation can be placed in the same local transaction:

Begin transaction
  Insert this consumer's event-processing record, with a unique key preventing duplicates
  Perform the business changes
Commit transaction
Acknowledge the message

If the business operation fails, both roll back together; if it succeeds but a crash occurs before acknowledgment, redelivery encounters the deduplication constraint. If external payments, SMS, or similar operations are also involved, cooperation such as idempotent interfaces from the other party is required; a local transaction alone is not enough.

Event E1 is delivered twice: the first transaction commits a deduplication record and debit; after a crash before acknowledgment, redelivery encounters the unique key, preventing another debit and leaving the balance at 90

Figure: When the billing consumer first processes E1, it inserts a (billing, E1) deduplication record and debits 10 within the same transaction, reducing the balance from 100 to 90. A crash after commit but before acknowledgment causes MQ to deliver E1 again. The second insert encounters the unique constraint, so no further debit occurs. Only this duplicate attempt is rolled back; the first committed result is not undone. The balance remains 90, and the duplicate message is then acknowledged. If the first business operation had failed, the deduplication record and debit would have rolled back together, allowing normal processing on redelivery.

4. How Do You Ensure Reliable Message Delivery? #

Regardless of the MQ used, investigate along the chain: Did the producer send reliably, did MQ store reliably, and did the consumer actually finish processing before acknowledging? The following sections examine RabbitMQ, Kafka, and RocketMQ in turn.

4.1 RabbitMQ #

RabbitMQ’s three message-loss windows: during sending, before reliable broker storage, and premature consumer acknowledgment

Figure: Transmission, storage, and business processing each have different failure windows that need to be addressed separately.

Message Loss at the Producer

Network failures may occur while messages are being sent. Transactions or publisher confirms can be used to track sending results.

Transactions:

• Call txSelect() to begin a transaction.
• Send messages.
• Call txCommit() to commit; handle rollback and retries on failure.

Transactions impose substantial performance overhead: at commit, the producer must wait for the broker’s result, adding network interactions and synchronous waiting. If a transaction is committed for every message sent, these costs are repeatedly incurred, significantly reducing publishing throughput.

Publisher Confirms:

After the producer calls confirmSelect() to enable confirm mode, each published message is associated with a publish sequence number that increases within the current channel. This matches acknowledgments; it is not a global business message ID across channels or retries. Before sending, obtain the sequence number through getNextPublishSeqNo() and store “publish sequence number → message content, send time” in a pending-confirmation table.

With asynchronous confirms, the producer registers callbacks and updates pending-confirmation state according to the broker’s results:

• ack (basic.ack): The broker has taken responsibility for the message and met the corresponding acknowledgment conditions. After receiving the success callback, the producer can remove the corresponding pending record; this does not mean the consumer has completed business processing.
• nack (basic.nack): The broker cannot take responsibility for the message. After receiving the failure callback, the producer can schedule a retry or record the failure.
• Timeout without acknowledgment: Periodically check the pending-confirmation table and schedule resends for messages that remain unacknowledged beyond the deadline. A timeout does not necessarily mean the message never arrived; the acknowledgment may simply have been lost. Retries should therefore reuse the same business event ID, with consumers ensuring idempotency.

The delivery-tag in a confirmation corresponds to the publish sequence number; if multiple=true, it acknowledges that sequence number and all earlier unconfirmed messages, so the pending table must also be updated by range. An in-memory table is suitable for tracking live state; handling producer process crashes additionally requires recovery mechanisms such as persisting pending-send records.

The main difference from transactions is how waiting works: txCommit() waits synchronously for the result, whereas asynchronous confirms allow publishing the next message after sending, with the client invoking a callback when it receives an ack or nack, avoiding a blocking wait for every message. Applications can also explicitly wait for confirm results. There are three common approaches:

Transaction mode and confirm mode cannot be used simultaneously on the same channel.

Message Loss Within RabbitMQ

For classic queues, persistence requires two parts:

1. A durable queue to preserve queue metadata.
2. Messages marked as persistent to preserve message contents.

Setting only one is insufficient. Even if both the queue and messages are configured for persistence, a message may not yet have been written to disk after arriving at RabbitMQ. If RabbitMQ crashes at that point, messages not yet persisted may be lost.

Persistence must therefore work together with producer confirms: for persistent messages delivered to durable classic queues, RabbitMQ sends an ack to the producer only after persisting the message to disk. If it crashes before that, the producer receives no confirmation and can resend after a timeout. This lets the producer resend messages RabbitMQ lost before persistence; if the message was persisted and only the confirmation was lost, resending may create duplicates, requiring consumer idempotency.

Message Loss at the Consumer

If MQ considers consumption complete as soon as a message is delivered, but the consumer crashes before performing the business operation, that operation will be missed.

The solution is to disable automatic acknowledgment and manually ack only after business processing succeeds. Unacknowledged messages can be redelivered after failures such as a lost connection.

This order avoids premature acknowledgment but can still produce duplicates, so it must be considered together with the idempotency discussed in the previous section.

Addressing three failure locations: producer confirms and retries, broker persistence and replication, and manual consumer acknowledgment after processing

Figure: The producer tracks sending results, the broker stores messages reliably, and the consumer acknowledges after processing. Channel transactions and confirms are alternatives, not the same mode.

4.2 Kafka #

Message Loss at the Consumer

For example, a consumer puts messages into an in-memory queue after the offset has already been committed automatically. If the process then restarts, unprocessed tasks in memory are lost.

The solution is to control when offsets are committed: commit after business processing completes, not immediately after fetching successfully or placing work in a thread pool.

With parallel processing, also ensure that committed offsets do not advance past unfinished messages.

Message Loss Within Kafka

If the leader fails and its replacement replica lacks some data, messages may be lost. Configuration should consider:

• replication.factor > 1: Each partition has multiple replicas.
• min.insync.replicas > 1: Reliable writes require more than one remaining in-sync replica.
• acks=all: Each message must be written to the leader and all followers that remain in sync and participate in this confirmation before write success is returned to the producer; lagging replicas removed from the in-sync set are not waited for.
• Set producer retries to a large value, such as the Java client’s Integer.MAX_VALUE (2147483647), so retryable failures such as network errors and leader changes continue to be retried rather than abandoning sending too early because too few retries are allowed.

Kafka producers usually retry asynchronously in the background, rather than necessarily blocking the business thread. If business code calls send(...).get() to wait synchronously, it waits for sending to succeed or ultimately fail. Final failures must still be handled through callbacks or exceptions rather than ignored; retries should also be combined with producer idempotency settings to avoid duplicate writes from uncertain sending results.

For example, suppose a partition has three replicas, requires at least two in sync, and uses acks=all: when all three remain in sync, all three must write the message; if one follower falls behind and is removed from the in-sync set, the remaining two are waited for; if only the leader remains, the minimum of two replicas cannot be met, so writes fail.

The set of replicas that remain in sync is called the ISR (In-Sync Replicas). Thus, “all replicas have written” applies only when all replicas remain in sync; it does not mean unconditionally waiting for all configured replicas.

Message Loss at the Producer

After calling Kafka’s send(), the message may only have entered the producer’s memory and not yet reached Kafka. If the process crashes then, the in-memory message may be lost. Check results through callbacks or call send(...).get() to wait for the result.

If automatic retries ultimately fail, do not simply log the failure and ignore it. Pending messages can first be stored in a database and marked as sent after Kafka confirms success. After a process restart, continue scanning records not successfully sent and resend them; if Kafka received the message but the successful response was lost, resending may produce duplicates, so consumers still need idempotency.

For example, if an order has been written to the database but the service crashes before sending the “order created” message, downstream systems receive no notification. “Create order” and “save pending message” can be placed in the same database transaction: either both succeed or both roll back, after which a background task sends messages and updates their sending status.

4.3 RocketMQ #

RocketMQ reliability likewise requires considering sending, disk flushing, disk failures, consumption, and node failures separately.

Producer Sending: Transactional Messages

When sending messages, a producer can use RocketMQ’s built-in transactional message mechanism^[3] so that whether the local business operation succeeds determines whether the message is delivered downstream. The process is:

1. Send a half message first: It carries the business message content but is temporarily invisible to consumers. After receiving it, the broker returns the receipt result to the producer.
2. Confirm that the half message was sent successfully, then execute the local transaction: For example, create an order or deduct inventory. If sending the half message fails, handle that failure first rather than proceeding with this local transaction.
3. Return a status based on the local transaction result:
   • Commit: The local transaction succeeded; the broker commits the message, making it visible to consumers.
   • Rollback: The local transaction failed; the broker rolls back the half message and does not deliver it to consumers.
   • Unknown: Whether the local transaction succeeded is temporarily unclear; retain the half message and wait for a later check.
4. Check back when the status is uncertain: If Unknown is received, or the producer finishes its transaction but fails to return a status in time, the broker waits for a period and then asks the producer to check the local transaction result. For example, check whether the order was created and return Commit, Rollback, or Unknown, rather than executing the business operation again.

Why send a half message first? It first checks whether MQ can currently receive the message, avoiding a situation where the local business operation has finished before discovering that the message cannot be sent at all. The half message also retains the pending business message in MQ: even if the producer subsequently crashes, the broker knows which message needs checking; downstream systems do not process it before the outcome is clear.

Checks do not continue indefinitely. In RocketMQ 4.9.8, for example, transactionCheckMax^[4] defaults to 15 checks and can be adjusted through broker configuration. If the result remains unconfirmed when the limit is reached, the message enters the handling path for abandoned checks rather than being delivered downstream as committed; some versions move it to a dedicated system topic^[5] for investigation and manual handling instead of physically deleting it immediately.

Transactional messages do not guarantee that all downstream business operations succeed together; consumers still need retries and idempotency.

After the Broker Receives a Message: Disk Flushing

Disk flushing mainly falls into synchronous and asynchronous modes:

• Asynchronous flushing uses a cache first and persists in the background, offering better performance but leaving a corresponding failure window.
• Synchronous flushing waits until persistence requirements are met before returning, with a higher waiting cost.

Disk or Primary Node Failure: Replication and Failover

Keeping replicas on multiple nodes reduces the risk of data loss from a single disk or node failure.

DLedger provides log replication and leader election.

Consumer Processing Failures

Consumers should report consumption success only after the local business operation actually finishes. If success is reported after merely starting an asynchronous task and that task later fails, MQ may no longer retry through its failure-handling path.

Resending When a Node Is Unavailable

For example, if a RocketMQ node goes down and messages temporarily cannot be sent, store the message content and a “pending” status in a database. A scheduled background task retrieves these records and resends them, changing their status to “sent” after receiving a successful sending result. Even after a service restart, messages not yet successfully sent can then be found in the database and resent.

5. How Do You Ensure Message Ordering? #

5.1 Why Ordering Is Needed #

For database synchronization, for example, the operation order is:

Insert → Update → Delete

If the other side executes “Delete → Update → Insert,” the final state may be incorrect. Some business operations therefore require not only no message loss or duplication, but also related operations taking effect in order.

Ordering is usually required within a business entity such as the same row or order, rather than globally serializing all messages.

5.2 Out-of-Order Processing in RabbitMQ and Its Solutions #

When a queue has multiple consumers, messages may be delivered in sequence, but different consumer processing speeds can change the order in which business operations complete:

Delivery:   data1 → data2 → data3
Completion: data2 → data1 → data3

A RabbitMQ queue delivers three messages to different consumers, whose different completion times cause out-of-order database operations

Figure: Sequential delivery from a queue does not mean multiple consumers will complete database operations in the same order.

There are two approaches:

• Split work across multiple queues, placing data that requires ordering in the same queue for sequential processing by one consumer.
• Within the consumer, distribute work by business key across multiple in-memory queues, each handled by a single worker.

The key to the second approach is “always assign the same key to the same sequential worker,” not simply starting arbitrary threads. Different keys can run in parallel while processing remains sequential within the same key.

Also note that placing work in an in-memory queue does not mean processing has completed; acknowledgment should still follow business success. Redelivery after failure and consumer failover must also be included in the ordering design.

RabbitMQ queues are assigned by business key; three messages with the same key all enter Q2 for sequential processing by consumer 2

Figure: Messages with the same key enter Q2 and are processed sequentially by consumer 2; Q1 and Q3 can handle other keys, rather than distributing data for the same key round-robin across different queues.

5.3 Out-of-Order Processing in Kafka and Its Solutions #

Kafka can route related messages to the same partition based on their key. Provided that routing rules, partition count, and other relevant conditions remain unchanged, messages with the same key can retain their read order within a partition.

However, if a consumer passes fetched messages to multiple threads, business operations may still finish out of order.

Kafka messages with the same key enter P1; although the consumer reads them in order, distributing them to different threads may cause out-of-order database writes

Figure: P1’s read order has not changed; reordering occurs during concurrent processing by threads within the consumer.

Two approaches can be used:

1. Use one topic, one partition, and one consumer processing sequentially. This is easy to understand but limits parallelism.
2. Use multiple in-memory queues within the consumer, assign messages by key, and give each queue one sequential processing thread.

The solution can be summarized as:

Same business key
    → Same partition or queue
    → Same sequential processing path

This does not automatically solve every ordering problem: when multiple producers send concurrently, the broker cannot reconstruct business causality from nothing; adding partitions may change key routing; and moving failed messages to a retry queue while continuing to process later messages may also break the original order. Strict ordering requires explicit handling rules for all of these cases.

Within the consumer, business keys route messages to in-memory queues, with the same key handled by the same sequential worker

Figure: After reading P1 sequentially, messages are distributed again by key. The same key enters in-memory queue 1 for sequential execution by worker 1. In-memory queuing does not mean reliable completion; offset commits still need to be controlled.

6. How Do You Handle Message Queue Delays, Expiration, and Backlogs? #

6.1 A Large, Persistent Message Backlog #

Fundamentally, the production rate exceeds the actual business processing rate. Common causes include consumer failures, a slowing downstream database, or excessive processing time per message.

When consumption parallelism is the bottleneck and downstream systems still have spare capacity, temporary scaling can be considered:

1. Fix the consumers first, then stop the original consumer instances. Identify and resolve the causes of slow processing, such as slow database queries, external API timeouts, or consumer code errors, and verify that the repaired consumers can return to normal speed. Then pause the original instances responsible for this backlog, record the current consumption positions, and prepare for a temporary distribution program to take over, avoiding two programs competing for the same messages.
2. Create ten times as many temporary partitions or queues. For example, expand from 10 consumption shards to 100 temporarily. In Kafka, create a topic with 100 partitions; in queue-based systems, prepare the corresponding number of temporary queues. The aim is to distribute the backlog across more shards that can be processed in parallel.
3. Deploy temporary consumers that only move messages. Have them read the backlog from the existing consumption positions without time-consuming operations such as business database writes or external API calls, and distribute messages evenly round-robin to the temporary partitions or queues. Confirm successful forwarding before acknowledging old messages or committing old consumer offsets; retry failed sends rather than acknowledging old messages before forwarding them. If the business requires ordering within the same key, use fixed key-based routing instead of round-robin.
4. Temporarily add consumer machines to perform the actual business processing. For example, scale the original 10 consumer machines to 100, deploy the repaired consumer program on them, and process the temporary partitions or queues. In Kafka, use one temporary consumer group whose instances share the partitions; in queue-based systems, have each set of consumers process its corresponding queues. This increases both consumption shards and processing resources. Only with sufficient downstream capacity and evenly distributed load is it possible to approach a tenfold increase in processing speed; ten times as many machines does not necessarily mean ten times the speed.
5. Switch back to the original deployment after clearing the backlog. The temporary distribution program continues handling messages from the old entry point. When ready to switch back, briefly stop producers from writing there or pause acceptance of new requests. Wait until the old backlog has been fully forwarded and business processing in the temporary queues has finished, then stop the temporary distribution program and consumer instances. In Kafka, the distribution program should already have committed successfully forwarded offsets to the original consumer group, allowing the original consumers to resume from those positions; queue-based systems resume consuming unacknowledged messages in the old queues. Finally, restore the original consumers and production entry point, confirm that nothing was missed, and reclaim temporary machines and queues.

Recovery time must also account for new messages continuing to arrive. With a backlog B, a current processing rate of μ messages per second, and an input rate of λ messages per second, when μ > λ and rates remain stable:

Estimated time to clear = B / (μ - λ)

If μ is not greater than λ, the backlog will not clear on its own.

6.2 Message Expiration #

Consider RabbitMQ message TTL: if messages remain backlogged for too long and exceed their expiration time, they may no longer enter the normal consumption flow.

If messages have already been lost, restoring consumers alone cannot recover them. Consider “bulk reimport”: query the corresponding business data, reconstruct the messages, and gradually write them back to MQ.

This approach depends on important prerequisites:

• Missing business operations can be identified accurately.
• The original data still exists and is sufficient to reconstruct the required events.
• Resent messages have stable identifiers and do not cause duplicate business effects.
• Replay is rate-limited so it does not overwhelm downstream systems again.

If events contain intermediate states that have already been overwritten, the current business tables alone may not be enough to restore them. Configuring appropriate dead-letter and retention policies before expiration is usually more reliable than improvising data recovery afterward.

Products must also be distinguished: RabbitMQ message TTL and Kafka log retention are different mechanisms and cannot be understood using the same expiration model.

6.3 A Message Queue Nearing Capacity #

One extreme response is to consume and discard messages quickly, then reconstruct them from business data.

This should be considered only when the business permits discarding messages and complete recovery has been verified, not as a default approach. For core messages such as orders or accounting entries, blindly acknowledging and deleting them can cause irrecoverable data loss.

Prioritize limiting nonessential input, fixing consumption bottlenecks, evaluating safe scaling, and checking available archive and retention space. Discarding messages must be an explicit business trade-off, not merely a way to reduce queue length.

6.4 Optimizing RocketMQ Consumption #

Increasing Consumption Parallelism

Message consumption in business systems is often I/O-bound, involving operations such as database access or RPC calls. A consumer processing a message spends much of its time waiting for a database or external service to respond. Increasing consumption parallelism lets some threads process messages while others wait for I/O, improving total consumption throughput.

Consumption speed is ultimately also limited by database or external service capacity. Beyond a certain level of parallelism, database connection contention, lock waits, and RPC rate limits become more pronounced, and throughput may decline. Adjust parallelism gradually while observing actual throughput, processing latency, and downstream load rather than continually adding instances or threads.

For RocketMQ’s traditional Java PushConsumer in clustering consumption mode, adjustments can be made at two levels:

• Increase consumer instances within the same ConsumerGroup. Add machines or start multiple consumer processes on existing machines so more instances in the group share subscribed queues. For example, if 8 assignable queues are initially handled by 2 instances, increase to 4 or 8 instances. Increasing to 10 does not further improve parallelism because queues are assigned to specific instances, leaving the extra instances without queues.
• Increase consumption threads within a single consumer. Adjust the thread pool’s minimum and maximum sizes using consumeThreadMin and consumeThreadMax, corresponding to Java configuration methods setConsumeThreadMin(...) and setConsumeThreadMax(...). For example, change the thread range from 10–20 to 20–40 so one instance handles more messages concurrently. This mainly applies to concurrent consumption; with ordered consumption, a given queue must still be processed in sequence, and adding threads cannot make it arbitrarily parallel.

Batch Consumption

Batch processing amortizes database and network interaction costs. For example, combining multiple records into one write is usually more efficient than a separate interaction for each.

First confirm that the business supports batching and determine how partial failures should be retried.

Skipping Nonessential Messages

If the business permits losing some nonessential messages, once the backlog exceeds a threshold these messages can be reported as successful directly, skipping business processing.

This deliberately gives up some data. It applies to business operations that explicitly permit loss, not orders or financial operations that must each be completed.

Optimizing Per-Message Processing

Consider reducing database interactions:

• Originally, 4 database accesses at 5ms each, plus 5ms of computation, total 25ms.
• Reducing this to 2 database accesses plus computation gives a total of 15ms.

Per-message processing time falls by 40%, improving processing capacity under otherwise identical conditions.

7. How Would You Design a Message Queue? #

When designing a message queue^[6], start with scalability, persistence, high availability, and reliability, placing the mechanisms discussed above back into the overall architecture.

7.1 Scalability #

Consider Kafka’s hierarchy:

Broker → Topic → Partition

Divide a topic into multiple partitions and distribute them across nodes to share data storage and read/write load.

When resources are insufficient, scale out by adding nodes or adjusting partition distribution. Two distinctions matter: adding nodes does not mean data automatically migrates evenly, and adding partitions does not mean historical data is automatically resharded; migration and routing changes need separate designs.

7.2 Message Persistence #

Messages cannot be kept only in memory, or they may be lost when the process stops.

Use sequential disk writes: append messages to a log, reducing random-access overhead and providing a basis for high throughput.

7.3 High Availability #

Keep multiple replicas for each partition, with the leader coordinating external writes and followers replicating data. After a leader failure, restore service through an election.

7.4 Message Reliability #

Reliability is not a single configuration setting; the guarantees of each stage must work together:

• Producers can determine whether sending succeeded, failed, or has an unknown outcome, and retry correctly.
• Brokers acknowledge after persisting and replicating as agreed.
• Consumers acknowledge or commit progress after business processing succeeds.
• Duplicate messages do not produce duplicate business effects.

The diagrams in this article were created with logic2image^[7].

Related readings

• Message Queues and Kafka Basics
• The Main Kind of Message Queue