How do I write a queue for Adobe Commerce (Magento)?

Benefits of Using a Queue vs Cron

| Aspect               | Queue                                                                                   | Cron                                                                                 |
|----------------------|----------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------|
| Task Type            | Best for long-running, high-volume, or asynchronous tasks triggered by events           | Best for repetitive, scheduled tasks that run at specific times                      |
| Performance          | Prevents blocking user requests; tasks handled in background for better responsiveness | Can block or delay user requests if tasks are long-running                           |
| Scalability          | Easily scales horizontally; you can add more workers as load increases                 | Difficult to scale; running multiple cron jobs can lead to duplicate processing      |
| Control & Priority   | Allows prioritization and fine-grained control over task processing                    | Limited control; all scheduled tasks run at their set times without prioritization   |
| Real-Time Handling   | Suitable for near real-time processing; tasks can be handled immediately after trigger | Not real-time; tasks are only processed at the next scheduled interval               |
| Error Handling       | Can retry failed jobs automatically; better monitoring and management                  | Limited error handling; failures may go unnoticed unless explicitly logged           |
| Resource Throttling  | Can throttle and limit the number of concurrent tasks to avoid overloading resources  | Runs all scheduled tasks sequentially, which can overload resources if many tasks    |

Key Points

• Queues are ideal for tasks that are event-driven, need to be processed asynchronously, or require scaling and prioritization. They decouple task execution from user actions, improving user experience and system reliability.
• Cron jobs are simple and effective for periodic, predictable tasks but are less flexible for scaling, error handling, and real-time needs.

Queues allow for more robust management of background jobs, including retries, prioritization, and horizontal scaling, making them better suited for high-load or complex applications.

When to Use

Use queues for asynchronous, high-volume, or long-running tasks where you need control and scalability. Use cron for simple, scheduled, and low-frequency tasks.

Implementing a Queue

To write a queue for Adobe Commerce (Magento), you need to configure the message queue system using several XML files in your custom module and implement the corresponding PHP classes. Here’s a step-by-step guide:

1. Create the Required XML Configuration Files

In your module’s etc directory, add the following files:

• `communication.xml`: Defines the topic and handler.
• `queue_consumer.xml`: Links the queue to its consumer and specifies the handler method.
• `queue_topology.xml`: Declares the queue, exchange, and routing rules.
• `queue_publisher.xml`: Specifies the publisher and the exchange.

Example: `communication.xml`

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework:Communication/etc/communication.xsd">
    <topic name="vendor.module.topic" request="string">
        <handler name="processHandler" type="Vendor\Module\Model\Queue\Consumer" method="process"/>
    </topic>
</config>

• `topic name`: Unique identifier for your queue topic.
• `handler`: Class and method that will process messages from the queue.

Example: `queue_consumer.xml`

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework-message-queue:etc/consumer.xsd">
    <consumer name="vendor.module.consumer"
              queue="vendor.module.queue"
              connection="db"
              maxMessages="1000"
              consumerInstance="Magento\Framework\MessageQueue\Consumer"
              handler="Vendor\Module\Model\Queue\Consumer::process"/>
</config>

• `queue`: The queue name to listen to.
• `connection`: Use `db` for MySQL or` amqp` for RabbitMQ.
• `handler`: The class and method to process the message.

Example: `queue_topology.xml`

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework-message-queue:etc/topology.xsd">
    <exchange name="magento-db" type="topic" connection="db">
        <binding id="processHandlerBinding"
                 topic="vendor.module.topic"
                 destinationType="queue"
                 destination="vendor.module.queue"/>
    </exchange>
</config>

• Binds the topic to the queue via the `exchange`.

Example: `queue_publisher.xml`

<?xml version="1.0"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="urn:magento:framework-message-queue:etc/publisher.xsd">
    <publisher topic="vendor.module.topic">
        <connection name="db" exchange="magento-db"/>
    </publisher>
</config>

• Links the topic to the exchange for publishing.

2. Implement the Consumer Handler Class

Create a PHP class (e.g., `Vendor\Module\Model\Queue\Consumer`) with a process method:

<?php
namespace Vendor\Module\Model\Queue;

class Consumer
{
    public function process($message)
    {
        // Handle the message (e.g., process order, update status, etc.)
    }
}

• The process method receives the message from the queue and processes it.

3. Publish Messages to the Queue

Inject `Magento\Framework\MessageQueue\PublisherInterface` into your service or controller and use it to publish messages:

$publisher->publish('vendor.module.topic', $data);

• Replace `vendor.module.topic` with your topic name and `$data` with the message payload.

4. Start the Queue Consumer

Run the consumer using the Magento CLI:

bin/magento queue:consumers:start vendor.module.consumer 

• Replace `vendor.module.consumer` with the consumer name from your `queue_consumer.xml`.

You can list all consumers with:

bin/magento queue:consumers:list

5. Database vs. RabbitMQ

• The default connection is db (MySQL), suitable for small to medium workloads.
• For high-throughput or distributed systems, use RabbitMQ by setting `connection="amqp"` and configuring `RabbitMQ` in your environment.

Summary Table

| File                   | Purpose                                    |
|------------------------|--------------------------------------------|
| `communication.xml`    | Define topic and handler                   |
| `queue_consumer.xml`   | Link queue to consumer and handler         |
| `queue_topology.xml`   | Declare queue, exchange, and routing       |
| `queue_publisher.xml`  | Specify publisher and exchange             |
| Consumer PHP class     | Implements the message processing logic    |

By following these steps, you can create and manage custom queues in Adobe Commerce (Magento), enabling asynchronous processing for scalable and robust integrations.

If you need help optimizing your Adobe Comemrce (Magento) storefront performance,  contact us at x2y.dev/contact.