How to use multithreading in Dynamics 365 for efficient batch processing

In the rapidly evolving landscape of enterprise resource planning (ERP) systems, efficiency drives innovation. Dynamics 365, a platform that empowers businesses to manage operations effectively with its robust architecture and extensive capabilities, is a shining example. However, performance bottlenecks often emerge as data volumes grow, and processes become more complex. This is where multithreading comes into play, offering a powerful tool to optimize execution and improve system efficiency.

In this blog, we will demystify the concept of multithreading in Dynamics 365 X++, exploring its practical applications, implementation strategies, and common challenges. Whether you’re a seasoned developer or just beginning your journey with Microsoft Dynamics 365, this guide will equip you with actionable insights to leverage multithreading effectively in your projects.

What is multithreading in Dynamics 365 Finance and Operations?

In Microsoft Dynamics 365 Finance and Operations, multithreading refers to the ability to execute multiple tasks or processes simultaneously to improve performance and efficiency, especially during batch processing or data-intensive operations.

Why use multithreading?

Multithreading provides several advantages:

• Improved performance: Tasks are distributed across multiple threads, reducing execution time and enhancing system responsiveness.
• Efficient resource utilization: Tasks are distributed across multiple threads, reducing execution time and enhancing system responsiveness.
• Scalability: By dividing workloads into smaller, independent units, multithreading makes systems more scalable and capable of handling higher transaction volumes.
• User satisfaction: Faster and smoother processes lead to better user experiences, reducing downtime and increasing productivity.

Approaches to multithreading in D365 X++

When implementing multithreading in X++, developers can choose from several approaches, depending on the nature of their tasks. Here are three commonly used methods:

1: Individual task modeling

Breaks down large tasks into smaller, independent work items that can be processed in parallel.
Use the BatchHeader and BatchJob classes to define and execute these tasks.

2: Batch bundling

Groups similar tasks into bundles and process them as a single batch. This method is ideal for scenarios where tasks share common data or dependencies.

3: Top-picking pattern

Assign tasks dynamically based on workload distribution. It’s useful for scenarios where task priorities or data availability vary over time.

Each approach offers unique advantages, and the choice depends on factors like task complexity, data volume, and system constraints.

The role of Batch Framework and SysOperation Framework

Dynamics 365 provides robust frameworks to facilitate multithreading: the Batch and SysOperation frameworks. These tools enable developers to distribute tasks across multiple threads and process them in parallel.

Batch framework

The Batch Framework is designed to schedule and execute tasks in the background. Each task within a batch job can be processed on a separate thread, making it an ideal choice for multithreaded operations. Key features include:

• Automatic load balancing across threads.
• Fault tolerance for retrying failed tasks.
• Ability to schedule jobs at specific times.

SysOperation framework

The SysOperation Framework, a newer model, builds on the Batch Framework and introduces additional benefits:

• Separation of business logic and execution logic.
• Enhanced scalability for large workloads.
• Support for parameterized task execution.

Both frameworks provide a solid foundation for multithreading in Dynamics 365, enabling developers to handle even the most demanding scenarios effectively.

Step-by-step implementation: How to create multithreaded batch jobs in Dynamics 365

Let’s examine how you, as a D365 developer, can create multithreaded batch jobs in D365. In most cases, a single-threaded batch job will be perfect. However, there are situations where additional performance may be needed, and in those cases, using multithreaded batch jobs could provide the required throughput.

The code referenced in this article follows this project structure:

A quick overview of the approach:

• We will create a batch job that will be our main job used to create tasks. This job should look to process all records in a staging table at the time of execution by dividing records between different threads.
• This job will create X number of threads. Each thread will lock the top available record in the staging table and send that record to our processing class to run business logic on that record.

Step 1: Create tasks

Create a standard single-threaded batch job. This will be our main batch job, which will divide our work into tasks and assign each task to a new thread. Most of this should be very familiar, as it follows the standard SysOps framework.

The contract class will use the user’s input to determine how many tasks to divide our work into. The service is the most interesting part, so let’s dive into that more.

• If there are records in the staging table to be processed, mark all of them as in processing. This will prevent us from attempting to process documents that get added to the staging table during the current execution.
• Then, based on the data obtained in the contract class, create the total number of threads. For each task, create an instance of our “Get work item” job.
• Finally, we can save the batch header to initiate the new batch job.

Step 2: Top picking

Here we have another standard controller class and a service class with some interesting pieces. Looking into the service class more, we see:

• We are selecting from our staging table with a pessimistic lock. This lock prevents different threads from attempting to process the same record simultaneously.
• The readPast(true) set on the staging table is needed to allow the query to skip past currently locked records and find the next available record to process.
• The select statement will grab the next available record and send it to our processing method. We could also process it in place, but this structure allows for a nice level of abstraction.
• At the end, don’t forget to update the staging table record’s status to indicate processing has completed to avoid selecting records again on a future run.

Step 3: Processing

Here is where we would run our business logic. For the sake of a high-level demonstration, this class will log our value and sleep for 5 seconds to simulate a longer processing job. However, this is where your core functionality would exist to use the data in the staging table to complete some process.

Practical application

To demonstrate this, we will load our staging table with test data. To do this, we will run our runnable job “CreateStagingRecords” by visiting the following URL: https://[D365URL].com/.cmp=[LegalEntity]&mi=SysClassRunner&cls=[ClassName]

We loaded our staging table with 100 records, and the processing job should take around 5 seconds to process each record. That would be a total of 500 seconds of execution time. However, we can run our job with 8 threads to divide that work, bringing each thread’s execution time to just over 1 minute.

Now, let’s go kick off our batch job.

We can see that the main job used to create tasks is completed quickly after the multithreaded job is spun up.

We can see our “Get work item” job completed, with all 8 threads taking approximately 1 minute each. This is exactly what we were hoping for!

Configuring maximum number of batch threads

It is important to note that users can set the maximum number of batch threads allowed to run at once.
First, go to System administration>Setup>Server configuration.

Second, set the Maximum batch threads field to 16. More than 16 can have negative performance consequences. See Microsoft’s documentation here.

Conclusion

Multithreading in Dynamics 365 X++ isn’t just a performance tweak; it’s a strategic enhancement for businesses dealing with large data volumes and complex operations. Organizations can significantly reduce processing time and unlock greater efficiency by intelligently distributing workloads across multiple threads using the Batch and SysOperation frameworks.

Are you curious about how multithreading can speed up your processes and boost system performance? Contact us at marketing@confiz.com, and let’s power up your Dynamics 365 experience together.