Json2Json: Template Based Json Transform

Version 0.10, Apr. 2014

Json Path based Template

Here we discuss about how Json data can be transformed into another Json data. Here are several issues.

How to read data from source Json?
How to fill data into target Json?
How to filer the data?

Since Json can be replacement of XML. One solution of XML to XML transformation is called XSL Transformations(XSLT). However NO Json transformation is popular. But the key idea is mapping and filter data from source to target. From this direction, we can get the source data, filter it, and put it into target position. To realize this,

Json Path is a nice solution for get the source data.
Json Template is a nice solution to tell the position of target data.
String based function (concatenate, split, join, index, substring, regex-match, regex-replace ) can be filter tool.

What is JsonPath?

JsonPath is totally a replacement of XPath. The idea of XPath is tree representation of the formatted data.

Here is a complete overview and a side by side comparison of the JSONPath syntax elements with its XPath counterparts.

XPath	JSONPath	Description
/	$	the root object/element
.	@	the current object/element
/	. or []	child operator
..	n/a	parent operator
//	..	recursive descent. JSONPath borrows this syntax from E4X.
*	*	wildcard. All objects/elements regardless their names.
@	n/a	attribute access. JSON structures don't have attributes.
[]	[]	subscript operator. XPath uses it to iterate over element collections and for predicates. In Javascript and JSON it is the native array operator.
\|	[,]	Union operator in XPath results in a combination of node sets. JSONPath allows alternate names or array indices as a set.
n/a	[start:end:step]	array slice operator borrowed from ES4.
[]	?()	applies a filter (script) expression.
n/a	()	script expression, using the underlying script engine.
()	n/a	grouping in Xpath

XPath has a lot more to offer (Location pathes in not abbreviated syntax, operators and functions) than listed here. Moreover there is a remarkable difference how the subscript operator works in Xpath and JSONPath.


    JSON_PATH:   ROOT ( CHILD | INDEX )*

    ROOT:                   ROOT
    CHILD:                  DOT_CHILD | ARRAY_CHILD
    INDEX:                  ARRAY_OPEN ( INDEX_CHARACTER* | WILDCARD ) ARRAY_CLOSE
    DOT_CHILD:              DOT SIMPLE_NAME
    ARRAY_CHILD:            ARRAY_OPEN ( QUOTE COMPLEX_NAME QUOTE | DOUBLE_QUOTE COMPLEX_NAME DOUBLE_QUOTE ) ARRAY_CLOSE
    SIMPLE_NAME:            SIMPLE_NAME_CHARACTER* | WILDCARD
    COMPLEX_NAME:           COMPLEX_NAME_CHARACTER* | WILDCARD

    COMPLEX_NAME_CHARACTER: LETTER | DIGIT | COMMA | HYPHEN | SPACE | UNDERSCORE
    SIMPLE_NAME_CHARACTER:  LETTER | DIGIT | UNDERSCORE
    INDEX_CHARACTER:        DIGIT | COMMA | SPACE
    LETTER:                 [A-Za-z]
    DIGIT:                  [0-9]

    ARRAY_CLOSE:            ]
    ARRAY_OPEN:             [
    COMMA:                  ,
    DOT:                    .
    DOUBLE_QUOTE:           "
    HYPHEN:                 -
    QUOTE:                  '
    ROOT:                   $
    SPACE:                  ' '
    UNDERSCORE:             _
    WILDCARD:               *

For an example, we have a Json data:

{
    "@context": {
    "name": "http://schema.org/name",
    "homepage": {
    "@id": "http://schema.org/url",
    "@type": "@id"
    },
    "image": {
    "@id": "http://schema.org/image",
    "@type": "@id"
    }
    },
    "image": "http://manu.sporny.org/images/manu.png",
    "name": "Manu Sporny",
    "homepage": "http://manu.sporny.org/"
}

and Json Path: $.@context.homepage.*

Then the results of the above Json Path will be:

[
        "http://schema.org/url",
        "@id"
]

What is Template?

Template is Json data with key-value pairs. The value could be JsonPath or normal properties.


{
    "@context": { "&$.@context.homepage.*" },
    "homepage": "&$.homepage"
}

Definition	Example	Description
&path	"&$.homepage"	Reference of JsonPath of default source Json.
&integerpath	"&2$.homepage"	Reference of JsonPath of the 2nd source Json.
&int1path1string&int2path2	"&$.name&2', '$.name"	Concatenate content of 2 Json path with string

Rule ^

Reading: If an Json value start with "&", Then this value should read from other Json object.
Replacement: If an Json object has one key, and this key starts with "%", Then this object should be executed and replaced with results.
Variable: "%alphabeta" is only allowed, When "%!alphabeta" exists, Or alphabeta is keyword in definitions ( see the above tables) .

What is String Filter?

The string filter has 7 basic functions to make it flexible. We want to implement these filter function by Java String functions.

Function	Definition	Symbol	Example	Description
function name	keywords	symbol	{ "$symbol" : [$para1, $para2, ...]} == $result	.
concatenate	%concat	%+	{"%+" : ["hello","world"]} == "helloworld"	.
split	%split	%\|	{"%\|" : ["hello.world","."]} == ["hello", "world"]	.
join	%join	%*	{"%*" : [["hello", "world"],"."]} == "hello.world"	.
index	%idx	%?	{"%?" : ["hello.world", "hello"]} == 0	.
substring	%sub	%_	{"%_" : ["hello.world", 0, 5]} == "hello"	.
length	%len	%#	{"%#" : "hello world"} == 11	.
replace	%rep	%/	{"%/" : ["hello.world", ".", " "]} == "hello world"	.
regex-match	%rmatch	%%	{"%%" : ["hello.world", "[a-z]+"]} == ["hello", "world"]	.
regex-split	%rsplit	%%\|	{"%%\|" : ["hello.world", ".l"]} == ["h", "lo.wo", "d"]	.
regex-replace	%rrep	%%/	{"%%/" : ["hello.world", ".l", "-"]} == "h-lo.wo-d"	.
jsonpath	%path	%&	{"%&" : [{"hello" : "1", "world" : "2"}, "$.hello"]} == "1"	.

What is Array Operator?

Function	Definition	Symbol	Example	Description
function name	keywords	symbol	{ "$symbol" : [$para1, $para2, ...]} == $result	.
add	%array-add	%]+	{"%]+" : [ ["hello","world"], "!" ] } == ["hello","world", "!"]	.
remove	%array-remove	%]-	{"%]-" : [ ["hello","world"], 0] } == ["world"]	.
get	%array-get	%]$	{"%]$" : [ ["hello", "world"], 0]} == "hello"	.
sublist	%array-sub	%]_	{"%]_" : [ ["hello", "world"], 0, 0]} == ["hello"]	.
index	%array-idx	%]?	{"%]?" : [ ["hello", "world"],"hello"]} == 0	.
size	%array-size	%]#	{"%]#" : ["hello", "world"]} == 2	.

What is Map Operator?

Function	Definition	Symbol	Example	Description
function name	keywords	symbol	{ "$symbol" : [$para1, $para2, ...]} == $result	.
put	%map-put	%}+	{"%}+" : [{}, {"hello":"world"}] } == {"hello":"world"}	.
get	%map-get	%}$	{"%}$" : [{"hello":"world"},"hello"]} == "world"	.
remove	%map-remove	%}-	{"%}-" : [{"hello" : "world"}, "hello"]} == {}	.
size	%map-size	%}#	{"%}#" : {"hello":"world"}} == 1	.
keys	%map-keys	%}*	{"%}*" : {"hello":"world"}} == ["hello"]	.

Process

Function	Definition	Symbol	Example	Description
Define Variable	{"%!name" : $json-object}	%!	{"%!v" : 1, "%!v2" : "hello", "%!v3" : {}, "%!v4" : [] }	define number, string, map, and array
Assignment Variable	{"%name" : $json-object}	%	{"%v2" : "hello"} == (v2 = "hello")	define number, string, map, and array
Process	{"%!proc" : { "%def" : {$initialization}, "%steps" : [{$Process}, ...], "%ret" : $return-object } }	%!+ %$ %{} %#	{"%!proc" : { "%$" : {$initialization}, "%{}" : [{$Process}, ...], "%#" : $return-object } }
If-Then-Else	{"%!if" : { "%def" : {$initialization}, "%expr" : $expression, "%if" : [$if-steps, ...] , "%else" : [$else-steps, ...], "%ret" : $return-object } }	%!? %<> %$ %if %else %#	{"%!if": { "%$" : { "%!id": "" }, "%<>" : { "==": [ "&$.name", "OpenNLP" ]}, "%if" : { "%id": "http://www.opennlp.org"}, "%else" : { "%id": "http://unknown.org" }, "%#" : "%id" } }	Result will be "http://www.opennlp.org".
For-Each	{"%!for" : { "%def" : {$initialization}, "%iter" : [$iterable, $index, $each], "%each" : [{$each-process}, ...] , "%ret" : $return-object }}	%!* %[] %$ %each %#	{"%!for": { "%$" : { "%!s": "" }, "%[]" : [ ["hello", "world"], "%i", "%e"], "%each" : {"%s": {"%+": ["%s", "%e"]}}, "%#" : "%s" } }	Result will be "helloworld".
While	{"%!while" : { "%def" : {$initialization}, "%expr" : $expression, "%do" : [{$each-process}, ...] , "%ret" : $return-object }}	%!_ %<> %$ %do %#	{"%!while": { "%$" : { "%!s": "" }, "%<>" : {"<":[ {"%#": "%s"}, 200]}, "%do" : {"%s": {"%+": ["%s", " next"]}}, "%#" : "%s" } }	Result will be " next next next next".

Project ^

We want to realize the java project by taking advantage of JsonPath Implementation, Java String Functions, and Groovy script engine.

JsonPath Implementation --> We tested 3 open source implementation (Jayway, Nebhale, Camel, Gatling). Our conclusion is that Camel is a wrapper of Jayway. Nebhale and Gatling are not stable for several cases. Thus, Jayway is the choice.
Template Replacement --> We filter all the JsonPath into Hash Map, run JsonPath result, and hold key-value pairs (JsonPath, Result). This will prevent repeating of the JsonPath.
Java String Function --> We replace the string filter mark using Java string function results.
- Realize a ProxyJavaClass to fulfil the definition through static function based on Java String Functions.
- Replace the % keywords by ProxyJavaClass.funtion().
Array Iteration --> We want to using groovy

Implementation

Github Hold
https://github.com/chunqishi/edu.brandeis.cs.json2json $ $ git clone https://github.com/chunqishi/edu.brandeis.cs.json2json.git ↵ $
Maven Dependency
 <dependency> <groupId>com.jayway.jsonpath</groupId> <artifactId>json-path</artifactId> <version>0.9.1</version> </dependency>   <dependency> <groupId>org.json</groupId> <artifactId>json</artifactId> <version>20140107</version> </dependency>   <dependency> <groupId>org.codehaus.groovy</groupId> <artifactId>groovy-console</artifactId> <version>2.0.1</version> </dependency>

Examples

In order to prove that this JSON to JSON transformation works, we use the XSLT examples in W3School. All the XML input is firstly converted into JSON use online XML to JSON converter, e.g. Utilities Online. Then we try to implement the target format using self-designed Template. Here, we find an example from online XSLT freeformatter.

Original XML is catalog lists.

<?xml version="1.0" encoding="ISO-8859-1"?>
        <catalog xmlns:foo="http://www.foo.org/" xmlns:bar="http://www.bar.org">
            <foo:cd>
                <title>Empire Burlesque</title>
                <artist>Bob Dylan</artist>
                <country>USA</country>
                <company>Columbia</company>
                <price>10.90</price>
                <bar:year>1985</bar:year>
            </foo:cd>
            <foo:cd>
                <title>Hide your heart</title>
                <artist>Bonnie Tyler</artist>
                <country>UK</country>
                <company>CBS Records</company>
                <price>9.90</price>
                <bar:year>1988</bar:year>
            </foo:cd>
            <foo:cd>
                <title>Greatest Hits</title>
                <artist>Dolly Parton</artist>
                <country>USA</country>
                <company>RCA</company>
                <price>9.90</price>
                <bar:year>1982</bar:year>
            </foo:cd>
        </catalog>

We turn it into JSON using Utilities Online.

{
  "catalog": {
    "-xmlns:foo": "http://www.foo.org/",
    "-xmlns:bar": "http://www.bar.org",
    "foo:cd": [
      {
        "title": "Empire Burlesque",
        "artist": "Bob Dylan",
        "country": "USA",
        "company": "Columbia",
        "price": "10.90",
        "bar:year": "1985"
      },
      {
        "title": "Hide your heart",
        "artist": "Bonnie Tyler",
        "country": "UK",
        "company": "CBS Records",
        "price": "9.90",
        "bar:year": "1988"
      },
      {
        "title": "Greatest Hits",
        "artist": "Dolly Parton",
        "country": "USA",
        "company": "RCA",
        "price": "9.90",
        "bar:year": "1982"
      }
    ]
  }
}

The XSLT turns the XML into HTML.

    <?xml version="1.0" encoding="ISO-8859-1"?>
    <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:foo="http://www.foo.org/" xmlns:bar="http://www.bar.org">
        <xsl:template match="/">
            <html>
            <body>
            <h2>My CD Collection</h2>
            <table border="1">
                <tr bgcolor="#9acd32">
                    <th>Title</th>
                    <th>Artist</th>
                    <th>Country</th>
                    <th>Company</th>
                    <th>Price</th>
                    <th>Year</th>
                </tr>
                <xsl:for-each select="catalog/foo:cd">
                    <tr>
                        <td><xsl:value-of select="title"/></td>
                        <td><xsl:value-of select="artist"/></td>
                        <td><xsl:value-of select="country"/></td>
                        <td><xsl:value-of select="company"/></td>
                        <td><xsl:value-of select="price"/></td>
                        <td><xsl:value-of select="bar:year"/></td>
                    </tr>
                </xsl:for-each>
            </table>
            </body>
            </html>
        </xsl:template>
    </xsl:stylesheet>

Instead of XSLT, our JSON Template will be like this.

We define a "%!s" is array with one object "th".
We want to use "for" function to process the transformation.
We iterate over JSON Path "$.catalog.foo:cd" using read ("&") function "&$.catalog.foo:cd"
We add "td" into array "%s" using array add function {"%]+", [ $array, $element ]}
We assign the new array as "%s" using assign function {"%s": $new }
We return the array "%s" using return function "%#".

{
  "html": {
    "-xmlns:bar": "http://www.bar.org",
    "-xmlns:foo": "http://www.foo.org/",
    "body": {
      "h2": "My CD Collection",
      "table": {
        "-border": "1",
        "tr": {"%!for": {
               "%$"    : {"%!s":[ { "-bgcolor": "#9acd32",
                                    "th": [
                                      "Title",
                                      "Artist",
                                      "Country",
                                      "Company",
                                      "Price",
                                      "Year" ]
                                  } ]
                         },
               "%[]"   : ["&$.catalog.foo:cd", "%i", "%e"],
               "%each" : {"%s": {"%]+": ["%s", {"td": [
                                                        {"%&":["%e", "$.title"]} ,
                                                        {"%&":["%e", "$.artist"]},
                                                        {"%&":["%e", "$.country"]},
                                                        {"%&":["%e", "$.company"]},
                                                        {"%&":["%e", "$.price"]},
                                                        {"%&":["%e", "$.bar:year"]}
                                                       ]}]}},

               "%#"    : "%s"
              }
        }
      }
    }
  }
}

We get the transformed result.

{
  "html": {
    "-xmlns:bar": "http://www.bar.org",
    "-xmlns:foo": "http://www.foo.org/",
    "body": {
      "h2": "My CD Collection",
      "table": {
        "-border": "1",
        "tr": [
          {
            "-bgcolor": "#9acd32",
            "th": [
              "Title",
              "Artist",
              "Country",
              "Company",
              "Price",
              "Year"
            ]
          },
          {
            "td": [
              "Empire Burlesque",
              "Bob Dylan",
              "USA",
              "Columbia",
              "10.90",
              "1985"
            ]
          },
          {
            "td": [
              "Hide your heart",
              "Bonnie Tyler",
              "UK",
              "CBS Records",
              "9.90",
              "1988"
            ]
          },
          {
            "td": [
              "Greatest Hits",
              "Dolly Parton",
              "USA",
              "RCA",
              "9.90",
              "1982"
            ]
          }
        ]
      }
    }
  }
}

We turn it into XML using XML to JSON converter Utilities Online. We get the XML result. It will be exactly the XSLT result.

<?xml version="1.0" encoding="UTF-8" ?>
        <html xmlns:bar="http://www.bar.org" xmlns:foo="http://www.foo.org/">
        <body>
        <h2>My CD Collection</h2>
        <table border="1">
            <tr bgcolor="#9acd32">
                <th>Title</th>
                <th>Artist</th>
                <th>Country</th>
                <th>Company</th>
                <th>Price</th>
                <th>Year</th>
            </tr>
            <tr>
                <td>Empire Burlesque</td>
                <td>Bob Dylan</td>
                <td>USA</td>
                <td>Columbia</td>
                <td>10.90</td>
                <td>1985</td>
            </tr>
            <tr>
                <td>Hide your heart</td>
                <td>Bonnie Tyler</td>
                <td>UK</td>
                <td>CBS Records</td>
                <td>9.90</td>
                <td>1988</td>
            </tr>
            <tr>
                <td>Greatest Hits</td>
                <td>Dolly Parton</td>
                <td>USA</td>
                <td>RCA</td>
                <td>9.90</td>
                <td>1982</td>
            </tr>
        </table>
        </body>
        </html>

Deployment ^


    <dependencies>
    <dependency>
    <groupId>org.lappsgrid</groupId>
    <artifactId>json2json</artifactId>
    <version>0.1.6</version>
    </dependency>
    <dependencies>