Part 3: Mule Batch Processing - MUnit Testing (The Conclusion)

In earlier parts of this series, we looked at the Batch processing capability of Mule and also added some unit tests for our batch. In this last part, we will see how to test record data and message routing through steps.

For quick reference, here are links to all parts in this series:

Part 1: Mule Batch Processing - Introduction

Part 2: Mule Batch Processing - MUnit Testing

Part 3: Mule Batch Processing - MUnit Testing (The Conclusion)

1. Continuation

In part 2, we added test cases for our batch input phase and on:complete phase. That allowed us to make sure the message loading logic and processing statistic matches our expectation.

We also listed some challenges for testing the batch code and how to get around some of those. In next test cases, we will see how we can get around remaining challenges -

  1. Batch steps are scopes like ForEach, Transactional etc, and so cannot be mocked or verified for call.

  2. Record and Record Variables are valid only in Batch Context, so mocking them is not straight forward.

2. Batch Application

We will use the same batch job from Part 2. Earlier I skipped through the step flow implementation, but now is the time to add those. Here are our simple step flows that just sets the order status to either Processing, Not Processing, Processed or Failed.

Listing 2.A: Batch Job Process phase with Message routing
<batch:process-records>
   <batch:step name="Batch_Step_1">
       <flow-ref name="batch-step-1-process" doc:name="Flow batch-step-1-process"/>
   </batch:step>
   <batch:step name="Batch_Step_2"  accept-expression="#[payload.status == 'Processing']">
     <flow-ref name="batch-step-2-process" doc:name="Flow batch-step-2-process"/>
   </batch:step>
   <batch:step name="Batch_Step_3" accept-expression="#[payload.status == 'Not-Processing']">
     <flow-ref name="batch-step-3-process" doc:name="Flow batch-step-3-process"/>
   </batch:step>
</batch:process-records>
Listing 2.B: Batch Step 1 Process flow
<sub-flow name="batch-step-1-process">
    <logger message="#['Processing Step 1 for Id:' + payload.id + ' with status: ' + payload.status.trim()]" level="INFO" doc:name="Logger"/>
<batch:set-record-variable variableName="id" value="#[payload.id]" doc:name="Record Variable"/>

    <dw:transform-message doc:name="Transform Message">
        <dw:set-payload><![CDATA[%dw 1.0
%output application/java
---
(payload ++ (status: 'Processing' when payload.status == 'Ready' otherwise 'Not-Processing'))]]></dw:set-payload>
    </dw:transform-message>
</sub-flow>
Listing 2.C: Batch Step 2 Process flow
<sub-flow name="batch-step-2-process">
    <logger message="#['Processing Step 2 for Id:' + recordVars.id + ' with status: ' + payload.status]" level="INFO" doc:name="Logger"/>
            <dw:transform-message doc:name="Transform Message">
        <dw:set-payload><![CDATA[%dw 1.0
%output application/java
---
(payload ++ (status: 'Processed'))]]></dw:set-payload>
    </dw:transform-message>
</sub-flow>
Listing 2.D: Batch Step 3 Process flow
<sub-flow name="batch-step-3-process">
    <logger message="#['Processing Step 3 for Id:' + recordVars.id + ' with status: ' + payload.status]" level="INFO" doc:name="Logger"/>
            <dw:transform-message doc:name="Transform Message">
        <dw:set-payload><![CDATA[%dw 1.0
%output application/java
---
(payload ++ (status: 'Failed'))]]></dw:set-payload>
    </dw:transform-message>
</sub-flow>

3. Testing Message Routing Through Steps

Usually batch job will have more than one batch step. Each step can filter records by using either accept-expression, accept-policy or both. When we have such filter criteria, messages may be following different step routes. It is essential to test these routes.

As shown in Listing 2.A, Every message will go through step 1. But step 2 and step 3 have filter criteria. Let’s see if we can test these routes.

Challenge#4 mentions that batch components are more like scopes and cannot be mocked or verified in MUnit. Trick here is to use flow-ref in steps to call the flows/subflows that implements the actual logic. That way, we can mock/verify calls to subflows and get around our challenge.

3.1 Test Message Routing 1

In this test case, we set a single record with status 'Ready' as batch payload. This message is expected to go through Step 1 and 2 only. We will see execution details below.

Listing 3.1.A: Sample Mule Batch Job
<munit:test name="batch-test-validate-step-flow-calling-route-1"
		description="Verifies that record routing through Batch steps.">

		<dw:transform-message doc:name="Transform Message">
			<dw:set-payload><![CDATA[%dw 1.0
%output application/java
---
[
	{
		"id": 1,
		"status": 'Ready',
		"type": 'CoffeeOrder'
	}
]]]></dw:set-payload>
		</dw:transform-message>

		<synchronize:run-and-wait timeout="1000000"
			doc:name="Synchronize">
			<batch:execute name="mule-simple-batch" doc:name="Run Batch mule-simple-batch" />
		</synchronize:run-and-wait>

		<mock:verify-call messageProcessor="mule:sub-flow"
			times="1" doc:name="Verify Step 1 is called">  (1)
			<mock:with-attributes>
				<mock:with-attribute name="name"
					whereValue="#[matchContains('batch-step-1-process')]" />
			</mock:with-attributes>
		</mock:verify-call>

		<mock:verify-call messageProcessor="mule:sub-flow"
			times="1" doc:name="Verify Step 2 is called"> (2)
			<mock:with-attributes>
				<mock:with-attribute name="name"
					whereValue="#[matchContains('batch-step-2-process')]" />
			</mock:with-attributes>
		</mock:verify-call>

		<mock:verify-call messageProcessor="mule:sub-flow"
			times="0" doc:name="Verify Step 3 is NOT called"> (3)
			<mock:with-attributes>
				<mock:with-attribute name="name"
					whereValue="#[matchContains('batch-step-3-process')]" />
			</mock:with-attributes>
		</mock:verify-call>

		<mock:verify-call messageProcessor="mule:logger"
			doc:name="Verify Call"> (4)
			<mock:with-attributes>
				<mock:with-attribute name="doc:name" whereValue="#['EndLogger']" />
			</mock:with-attributes>
		</mock:verify-call>

		<logger message="#['**************** Done ******************']"
			level="INFO" doc:name="Logger" />
	</munit:test>
1 After batch is executed, verify that subflow batch-step-1-process is called exactly once.
2 Verify that subflow batch-step-2-process is also called once.
3 We do not expect, batch-step-3-process to get called, so verify that it is called 0 times.
4 As usual, to verify that batch completed successfully, verify that a logger in on:complete phase gets a call.

3.2 Test Message Routing 2

If we provide a record with staus 'Not-Ready' to our batch then it should flow through Step 1 and 3 only. We can write a test similar to Listing 3.1.A and adjust our subflow call times to our expecation.

4 Testing Flows using Record Context

So far we have verified that our Input phase is loading message correctly and messages are getting routed through steps as expected. How about testing actual sub-flows that implement the processing logic?

They appear to be simple sub-flows, so let’s write a test case to verify our batch-step-1-process.

Listing 4.A: MUnit Test For Step 1 processing flow
<munit:test name="batch-step-1-processTest-2"
  description="Verifies that status is set to Not-Processing">
  <dw:transform-message doc:name="Transform Message">
    <dw:set-payload><![CDATA[%dw 1.0
%output application/java
---
{
"id": 1,
"status": 'Not-Ready',
"type": 'CoffeeOrder'
}]]></dw:set-payload>
  </dw:transform-message>
  <flow-ref name="batch-step-1-process" doc:name="Flow-ref to batch-step-1-process" />
  <munit:assert-on-equals expectedValue="#[1]"
    actualValue="#[recordVars.id]" doc:name="Assert Id Equals" />
  <munit:assert-on-equals expectedValue="#['Not-Processing']"
    actualValue="#[payload.status]" doc:name="Assert Status Equals" />
</munit:test>
Test Failed! But why?

Remember Challenge#5? Record and Record Variables are valid only in Batch Context! Here, our flow is NOT running as a part of batch execution.

This is what failure error says -

ERROR - The test batch-step-1-processTest-1 finished with an Error.
No record could be found in payload or in flow variable BATCH_RECORD (java.lang.IllegalStateException).
org.mule.api.transformer.TransformerMessagingException: No record could be found in payload or in flow variable BATCH_RECORD (java.lang.IllegalStateException).

Error message gives us a hint that it expects a flow variable named BATCH_RECORD. But it does not say anything about what value it should have!

Here is what happens internally - When batch runs, it converts payload i.e. input record to an instance of com.mulesoft.module.batch.record.Record and sets that as BATCH_RECORD variable.

With that information, let’s try to set BATCH_RECORD and rerun the test. For that we can use an utility method from com.mulesoft.module.batch.record.BatchUtils class.

So here is our new test case with addition of BATCH_RECORD variable that we set before we call our flow under test. This should run and have an id record variable set after flow execution.

Listing 4.B: MUnit Test For Step 1 processing flow
<munit:test name="batch-step-1-processTest-1" description="Test">
		<dw:transform-message doc:name="Transform Message">
			<dw:set-payload><![CDATA[%dw 1.0
%output application/java
---
{
	"id": 1,
	"status": 'Ready',
	"type": 'CoffeeOrder'
}]]></dw:set-payload>
		</dw:transform-message>

		<set-variable variableName="BATCH_RECORD"
			value="#[com.mulesoft.module.batch.record.BatchUtils.toRecord(payload)]"
			doc:name="Variable" />   (1)

    <flow-ref name="batch-step-1-process" doc:name="Flow-ref to     batch-step-1-process" />
	<munit:assert-on-equals expectedValue="#[1]"
    actualValue="#[recordVars.id]" doc:name="Assert Id Equals" />
	<munit:assert-on-equals expectedValue="#['Processing']"
    actualValue="#[payload.status]"
    doc:name="Assert Status Equals" />
	</munit:test>
1 Initializing BATCH_RECORD variable.

How about setting a batch record variable in our test? Sometimes flow under test may require us to set a record variable. For example, batch-step-2-process and batch-step-3-process both expects a record variable id which is set by step 1.

After, we initialize the BATCH_RECORD, we can use batch:set-record-variable before calling our flow under test.

<batch:set-record-variable variableName="id"
			value="#[payload.id]" doc:name="Record Variable" />
Setting a record variable without first setting BATCH_RECORD will result in test failure for the same reason as earlier i.e. missing BATCH_RECORD.

With these concepts, we can now write test cases for all our step flows.

5. Conclusion

In this part, we learned how to test message routing through steps. Also, we learned how to get around record context in our test and verify every part involved in our batch job execution.

I hope these testing tips and tricks helps you in writing test cases for your Mule Batch Jobs and make them more error-proof.

If you have any comments on this series then feel free to let me know in comments or on twitter.

6. Source code

Mule application and all munit source code used for this series is available on github here. Feel free to take a look at more test cases in source code.

7. References and Further Reading

on twitter to get updates on new posts.
Part 2: Mule Batch Processing - MUnit Testing
Using Docker container to run Mule ESB on Raspberry Pi3

Stay updated!

On this blog, I post articles about different technologies like Java, MuleSoft, and much more.

You can get updates for new Posts in your email by subscribing to JavaStreets feed here -


{ Manik Magar }

Lives on Java Planet, Walks on Java Streets, Read/Writes in Java, JCP member, Jakarta EE enthusiast, MuleSoft Integration Architect, MuleSoft Community Ambassador, Open Source Contributor and Supporter, also writes at Unit Testers, A Family man!