The Apex Async Worker Pattern to rule them all (a/k/a Threads)

Someday I will be recognized for genius or locked up for insanity

As part of my series… lets build a salesforce app… I have another utility to add to the Enterprise Utilities library. This thing is awesome. I call it a “worker framework” to attempt to bring some organization and control what is otherwise a chaotic unorganized universe of async processing in Salesforce.

Here is what I don’t like about my experience on salesforce with async processing:

1. Scheduled job in general. I find the solution extremely lacking from other CRON based tools and especially I don’t like the fact that the schedules are not able to be paused or easily created in refreshed sandboxes or scratch orgs
2. Too much boiler plate. Each batch job I end up putting in the exact same boiler plate code, especially in relation to cross cutting concerns (sensing a pattern in my blog?)
3. The user interface could use a major upgrade (Reactforce to the rescue!)
4. My obsession with api driven development requires me to attach every feature of the system to an api… including all operations related to managing async jobs
5. Common exception handling, logging, and monitoring are just the right things to do

Here are the requirements for the framework:

1. A new worker implementation can be added by creating a record in the ImplementationHandler custom metadata and implementing the enterprise.IWorker interface. Creating scalable, robust, and powerful async jobs becomes extremely easy once all of the non-functional features have been managed by the framework.
2. The implemented interface can be scheduled, unscheduled(paused), paused, and executed immediately.
3. The workers can be scheduled with CRON schedules but also the CRON schedule can be created with human readable and manageable concepts like repeat jobs per month, per day, per hour, every x number of minutes, etc. Trust me. Its awesome.
4. The api framework can be used to manage all features of the worker framework. This makes building tooling easy to build and it opens the door to front end developers (Reactforce anyone?) to manage async tasks easily from the front end.

This feature shall ever be known as feature/ENT-18__WorkerFramework

I’ll update this blog post when its done. Happy coding!

Time passes….

More time passes…

A few more reactforce based hospital portals, a complete backend library rewrite, and new clients on the reactforce bandwagon caused this feature to push pretty far out on the roadmap. But I just got my hands on the latest version of the async platform and let me tell you… better than I even thought.

We renamed it “Threads”.

Here is what it can do:

1. Define a thread which can be @future, batch, or queueable.
2. Define the apex handler that will be executed for the thread. All of the boiler plate is managed by the framework.
3. Launch a thread directly from apex, including async apex. Async operations will use the next feature to survive the async governor limitations:
4. Launch a thread via a platform event
5. Schedule, Unschedule, Pause, Unpause, and Synchronize the threads with the Apex CRON scheduler.
6. Launch a thread using the thread api endpoint.
7. Each thread captures time stamps, the batch query, and the record IDs in scope of the execution context. All of the logs are cross-referenced together so you can see exactly what ran, were there errors, and what debug logs were tied to that execution.
8. The system automatically notifies regarding success, errors, etc to a notification email. This will eventually be hooked into our messaging app to support sending SMS directly from the framework. Imagine getting a text message if your job had errors. It would feel like the distant future of 2024. Oh wait.
9. The thread monitor makes sure you are notified about when jobs did NOT run that should have. That would have saved me a few hours of my life earlier in my career!
10. Threads monitor the Platform Event DLQ, automatically retrying any failed or scheduled events. Creating scheduled system events is now as easy as setting a time/date stamp on the Platform Event and waiting for the Thread to process them in the future.
11. Easily chain threads to each other, in configuration, with no concern for the underlining platform limitations. The framework will automatically manage the thread continuations.
12. Thread priority and concurrency is managed with a simple configuration.
13. Even more cool stuff that super nerds as myself will geek out over.

Needless to say… its awesome and now writing an enterprise level async process takes nothing more than a few lines of code. We are even considering hooking this to flow just to leverage the benefits of the management and logging functions.

This was the last of the low-level libraries that we have been slowly rebuilding over the course of the last year. Because we have been so busy implementing the platform as is for clients, I have not focused to get this on the AppExchange. I think time is about up… the world needs this app.

We have a few more days of development and then I will put together a demo. You are going to be jealous… or you are going to email me at greg@cloudpremise.com to figure out how you can get ahold of this incredible framework.

This is my new favorite line of code:

new Thread('LogThread').execute('PURGE');

If you haven’t figured it out from my blog, I appreciate an app that is easy to troubleshoot in production.

Told you I would update this… I always keep my word.