DataWeave 2 - Transform with flattened keys

What if you have a map of flattened keys like address.firstName and need to transform hierarchical data set using that? Can you do that mapping in DataWeave 2? Can this be done with tail-recursion and reusable way? Let’s explore it in this post.

Forums are a great way to learn and share. You get to see a variety of use cases when helping people on the forum. Personally, I take that as an opportunity to try things that I may not get to do in everyday work. One day, I saw an interesting transformation use case that we will explore again here.

Consider that you have an employee data (a simplified example) -

Input hierarchical Employee data
[
    {
        "name": "M1",
        "department" : {
            "id": 1,
            "name": "IT",
            "address": {
                "state": "MState",
                "country": "US",
                "zip": "12345"
            }
        }
    },
    {
        "name": "M2",
        "department" : {
            "id": 1,
            "name": "IT2",
            "address": {
                "state": "M2State",
                "country": "US",
                "zip": "12345"
            }
        }
    }
]

You need to transform this to a simplified structure -

Output Structure
[
  {
    "name": "M1",
    "department": "IT",
    "state": "MState",
    "country": "US"
  },
  {
    "name": "M2",
    "department": "IT2",
    "state": "M2State",
    "country": "US"
  }
]

Solution 1: Manual Mapping

One way of doing it can be, a manual transformation mapping.

Table 1. Solution 1: Manual mapping
Payload (application/json) Script Output (application/json) 
[
    {
        "name": "M1",
        "department" : {
            "id": 1,
            "name": "IT",
            "address": {
                "state": "MState",
                "country": "US",
                "zip": "12345"
            }
        }
    },
    {
        "name": "M2",
        "department" : {
            "id": 1,
            "name": "IT2",
            "address": {
                "state": "M2State",
                "country": "US",
                "zip": "12345"
            }
        }
    }
]
 
%dw 2.0
output application/json
---
payload map ((item, index) -> {
    name: item.name,
    department: item.department.name,
    state: item.department.address.state,
    country: item.department.address.country
})
 
[
  {
    "name": "M1",
    "department": "IT",
    "state": "MState",
    "country": "US"
  },
  {
    "name": "M2",
    "department": "IT2",
    "state": "M2State",
    "country": "US"
  }
]

Solution 2: Use flattened keys mapping

In a different use case, you may have a reference key mapping that tells where to pick value for each output key. It is essentially a definition of output structure with value mapping.

Flattened key mapping
{
  "name": "name",
  "department": "department.name",
  "state": "department.address.state",
  "country": "department.address.country"
}
Table 2. Solution 2: Flattened key transformation
Payload (application/json) Script Output (application/json) 
[
    {
        "name": "M1",
        "department" : {
            "id": 1,
            "name": "IT",
            "address": {
                "state": "MState",
                "country": "US",
                "zip": "12345"
            }
        }
    },
    {
        "name": "M2",
        "department" : {
            "id": 1,
            "name": "IT2",
            "address": {
                "state": "M2State",
                "country": "US",
                "zip": "12345"
            }
        }
    }
]
 
%dw 2.0
output application/json

var mapping = { (1)
    name: "name",
    department: "department.name",
    state: "department.address.state",
    country: "department.address.country"
}

@TailRec()
fun getValue(keys:Array<String>, value) =
    if(sizeOf(keys) == 1)
        value[(keys[0])]  (5)
    else
        getValue(keys[1 to -1], value[(keys[0])])(6)
---
payload map (item, index) -> {  (2)
    (mapping mapObject  (3)
          ((value, key, index2) ->
            (key): getValue((value splitBy "."),item) (4)
    ))
}
 
[
  {
    "name": "M1",
    "department": "IT",
    "state": "MState",
    "country": "US"
  },
  {
    "name": "M2",
    "department": "IT2",
    "state": "M2State",
    "country": "US"
  }
]
1 Flattened key mapping for output structure. This can come from external variables or systems.
2 Iterate over payload with map function.
3 Use flattened mapping object for writing to output format.
4 To map the keys, call our special getValue(Array<String>,Any):Any function. The values from mapping object, eg. department.address.state are split into array of string by splitting it with period.
5 When provided key array contains only one key, return value for that key. This will handle leaf nodes like name, state etc.
6 For an array of multiple keys, we again call our getValue but this time selecting the value for first key and reducing array to exclude that key. This recursion will continue until we reach to the last key, in which case it will be a leaf node.

Properties of getValue function

  1. It is a pure-function.

  2. Its logic is independent of key or value structure, which makes it reusable for any data structure.

  3. It uses tail-recursion to avoid any stack overflow errors, in case you have nested object mess.

DataWeave functions used in this script - mapObject, map, splitBy.

Conclusion

That’s all for this short post. Hopefully, you learned something new here.

You can also see Generate Fibonacci sequence with DataWeave 2 for another example of using Tail-Recursion or other posts related to DataWeave 2.

In case you are wondering about forum question, you can find it here.

If you have any thoughts or feedback on this article, feel free to comment on this article or find me on Twitter or on manik.magar.me.

on twitter to get updates on new posts.
DataWeave 2 - Generate Fibonacci sequence
DataWeave 2 - Finding Quarter of the Year

Stay updated!

On this blog, I post articles about different technologies like Java, MuleSoft, and much more.

You can get updates for new Posts in your email by subscribing to JavaStreets feed here -


{ Manik Magar }

Lives on Java Planet, Walks on Java Streets, Read/Writes in Java, JCP member, Jakarta EE enthusiast, MuleSoft Integration Architect, MuleSoft Community Ambassador, Open Source Contributor and Supporter, also writes at Unit Testers, A Family man!