The Hidden Costs of Misusing Scheduled Jobs and Messaging Systems

messaging systems

azure

It's a common scenario we've all faced: you've got a table and a scheduled job designed to process this table.

Below is a simplified example of what such a script might look like:

1-- Declare a variable to store the ID of the row to be processed
2DECLARE @rowID INT
3
4-- Get the first row from the table to process
5SELECT TOP 1 @rowID = ID
6FROM TableToProcess
7ORDER BY ID ASC
8
9-- Check if a row was found
10IF @rowID IS NOT NULL
11BEGIN
12    -- Perform processing on the selected row (You can put your processing logic here)
13    -- For now, we'll add a comment to indicate where the processing happens
14
15    -- Delete the processed row from the table
16    DELETE FROM TableToProcess WHERE ID = @rowID
17END

But what happens when there's no new data to process? Without any contingency, the script will repeatedly hammer your database with fruitless calls, generating redundant traffic and potentially impacting the performance of your entire system.

Now, don't get me wrong. Scheduled jobs are not the enemy. They have a critical role in certain situations, such as scheduling regular email alerts. However, as developers, we sometimes fall into the trap of over-reliance on this tool, leading to unnecessary overhead within our systems.

So, is there a better, more efficient alternative to deal with these situations? Welcome to the realm of messaging systems.

So, What's a Messaging System?

In essence, a messaging system provides a reliable way to exchange information (in the form of messages) between applications, systems, or services. The main idea here is to enable communication between different components without them necessarily knowing about each other.

In a messaging system, producers create messages and put them onto the system. Consumers get those messages and do something with them. These systems are crucial for enabling asynchronous processing, meaning that your components don't have to wait around for other tasks to finish. This can significantly enhance performance, scalability, and reliability.

Message Flow vs Scheduled Task

Advantages

Real-time Processing: Messages are processed as they arrive, leading to faster responses.
Decoupling of Processes: Allows systems to be developed, upgraded, and scaled independently.
Improved Scalability: Messaging systems can more easily scale to handle larger loads.
Reduced Database Load: By processing data as it comes in, constant database querying can be reduced.
Resilience: Messaging systems can handle failures without losing data.

Do you need a reliable partner in tech for your next project?

Disadvantages

Complexity: Messaging systems come with their own complexities that require more advanced understanding and maintenance.
Handling Failures: Although resilient, handling failures, "poison messages," and ensuring message ordering can be challenging.
Infrastructure Requirements: Establishing and maintaining a robust messaging system might require significant resources and expertise.

Popular Messaging System Implementations

Now that you've got a taste of a messaging system let's dive into some popular implementations: Azure Service Bus, Kafka, Azure Queue System, and RabbitMQ.

Azure Service Bus

Azure Service Bus is a fully managed service from Microsoft Azure that supports queue and topic, messaging models. It's robust, provides guaranteed message delivery, and maintains the order of messages. Here's an example of sending a message with Azure Service Bus in C#:

1ServiceBusClient client = new ServiceBusClient("<connection_string>");
2ServiceBusSender sender = client.CreateSender("<queue_name>");
3
4ServiceBusMessage message = new ServiceBusMessage("Hello, World!");
5
6await sender.SendMessageAsync(message);
7
8Console.WriteLine($"Sent a single message to the queue: {sender.EntityPath}");

And here is how would consumer look like

1string connectionString = "<connection_string>";
2string queueName = "<queue_name>";
3
4await using (ServiceBusClient client = new ServiceBusClient(connectionString))
5{
6    ServiceBusProcessor processor = client.CreateProcessor(queueName, new ServiceBusProcessorOptions());
7
8    processor.ProcessMessageAsync += async args =>
9    {
10        string body = args.Message.Body.ToString();
11        Console.WriteLine($"Received message: {body}");
12        await args.CompleteMessageAsync(args.Message);
13    };
14
15    processor.ProcessErrorAsync += args =>
16    {
17        Console.WriteLine($"Error occurred: {args.Exception}");
18        return Task.CompletedTask;
19    };
20
21    await processor.StartProcessingAsync();
22
23    Console.WriteLine("Press any key to stop receiving messages...");
24    Console.ReadKey();
25
26    await processor.StopProcessingAsync();

And if you start using Azure function the implementation if even shorter and simple!

1[FunctionName("ServiceBusConsumer")]
2public static async Task Run(
3    [ServiceBusTrigger("<queue_name>", Connection = "<connection_string>")] string messageBody,
4    ILogger log)
5{
6    log.LogInformation($"Received message: {messageBody}");
7
8    // Process the message (you can replace this with your custom logic)
9
10    // Note: Unlike the previous examples, the message is automatically marked as completed (removed from the queue) by the Function after successful processing.
11}

Azure Queue System

Azure Queue System is another Microsoft offering specifically for storing large numbers of messages that may be accessed from anywhere via authenticated calls. It's simpler than Azure Service Bus and perfect for light workloads. An example of sending a message with Azure Queue in C# looks like this:

1QueueClient queueClient = new QueueClient("<connection_string>", "<queue_name>");
2string message = "Hello, World!";
3await queueClient.SendMessageAsync(message);

Kafka

Apache Kafka, an open-source system, is built to handle real-time data feeds with high throughput and low-latency. Kafka operates in a distributed manner and can handle massive amounts of data. Here's a simple example of producing a message with Kafka using Java:

1Properties props = new Properties();
2props.put("bootstrap.servers", "localhost:9092");
3props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
4props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
5
6Producer<String, String> producer = new KafkaProducer<>(props);
7ProducerRecord<String, String> record = new ProducerRecord<>("test", "Hello, World!");
8
9producer.send(record);
10producer.close();

RabbitMQ

RabbitMQ is another open-source messaging system known for its robustness. It supports multiple messaging protocols, message queuing, delivery acknowledgment, flexible queue routing, and more. Here's a Python example of sending a message with RabbitMQ:

1import pika
2
3connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
4channel = connection.channel()
5
6channel.queue_declare(queue='hello')
7
8channel.basic_publish(exchange='', routing_key='hello', body='Hello, World!')
9
10print(" [x] Sent 'Hello, World!'")
11connection.close()

There you have it. Keep your system from getting bogged down by an overload of scheduled tasks. Give messaging systems a try and watch your application's performance and scalability reach new heights. Whether you go for a fully managed service like Azure Service Bus or Azure Queue System or an open-source option like Kafka or RabbitMQ, you'll reap the benefits of reliable, scalable, and high-performance messaging.