Skip to content

NIEM JSON normalization

Jump to bottom
Webb Roberts edited this page Nov 3, 2020 · 7 revisions

We want JSON data to be as simple as possible, without sacrificing the ability for anyone who understands NIEM to understand the data. JSON-LD goes a long way, but there are a couple of issues:

  1. We want to enable developers to omit "@context" on messages, when the message is identified as a message defined by a message specification that provides a context.
  2. We want to enable the use of literal (vs. object) values in messages, for simplicity, in cases where it doesn't match the NIEM data model.

In short, we want to support messages that look like this:

{
  "person": {
    "age": {
      "decimalValue": 42.5,
      "unit": "ANN"
    },
    "birth": {
      "date": "1978-01-01"
    }
  }
}

We don't want to require that messages look like this:

{
  "@context": {
    "nc": "http://release.niem.gov/niem/niem-core/5.0/#",
    "rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
    "unece": "http://release.niem.gov/niem/codes/unece_rec20/5.0/#",
    "xs": "http://www.w3.org/2001/XMLSchema#"
  },
  "nc:Person": {
    "nc:PersonAgeMeasure": {
      "unece:TimeUnitCode": {
        "rdf:value": {
          "@value": "ANN",
          "@type": "xs:token"
        }
      },
      "nc:MeasureDecimalValue": {
        "rdf:value": {
          "@value": "42.5",
          "@type": "xs:decimal"
        }
      }
    },
    "nc:PersonBirthDate": {
      "nc:Date": {
        "rdf:value": {
          "@value": "1978-01-01",
          "@type": "xs:date"
        }
      }
    }
  }
}

We make the plain NIEM JSON document fully conformant to JSON-LD and the NIEM data model through a process called NIEM JSON normalization. This process yields a NIEM JSON-LD document.

NIEM JSON normalization

We define NIEM JSON normalization which performs the following transformations

  1. Apply the context to the JSON data
  2. Convert literals to objects with typed values

This process enables plain old json objects to be converted to best-practice JSON-LD.

Question for later: is there a simpler syntax for named graphs (to support @structures:relationshipMetadata that could be applied in this process?

Note: We're using the term "JSON document" analagous to "XML document", even though it's not necessarily a document. The JSON RFC (RFC 8259) uses the term "JSON text", which is awkward. The JSON Schema specification uses "JSON document".

Step 1: Apply the context to the JSON data

We expect a JSON-LD context to be a part of a message specification for a JSON exchange. We don't want to add JSON-LD contexts inline to runtime data. We also don't really want to clutter up JSON data with @context properties.

This step applies the context from the message specification to an instance.

Step 2: Convert literals to objects

In the NIEM JSON Specification v4.0, we specified literal-to-object conversion.

This step extends that process by providing for explicit typing, applying RDF-friendly XML Schema data types.

A simple example

Take this XML data:

<?xml version="1.0" encoding="US-ASCII"?>
<nc:PersonBirthDate xmlns:nc="http://release.niem.gov/niem/niem-core/5.0/">
  <nc:Date>1978-01-01</nc:Date>
</nc:PersonBirthDate>

As an example, this Plain NIEM JSON represents the same data:

{
  "birth": {
    "date": "1978-01-01"
  }
}

This context is provided for the plain JSON:

{
  "nc": "http://release.niem.gov/niem/niem-core/5.0/#",
  "birth": "nc:PersonBirthDate",
  "date": "nc:Date"
}

After step 1, with the context applied (and using a context defining namespaces for readability), the result is:

{
  "@context": {
    "nc": "http://release.niem.gov/niem/niem-core/5.0/#"
  },
  "nc:PersonBirthDate": {
    "nc:Date": "1978-01-01"
  }
}

Step 2, literal-to-object conversion, consists of:

  • Iterate over each property that has a literal value, where the property's definition requires an object value

  • Identify the nearest base type that appears within the list of RDF-compatible XSD types provided by RDF Concepts Section 5.1, The XML Schema Built-in Datatypes

  • Replace the literal value of the property with an object that follows the template, where:

    • $literal-value is the literal value
    • $type is the nearest base type on the list
    {
  "rdf:value": {
    "@value": "$literal-value",
    "@type": "$type"
  }
}

Property nc:Date is defined by the NIEM data model to be an object type, with a simple value of type xs:date, which is on the list of RDF-compatible XSD types. Converting yields:

{
  "@context": {
    "nc": "http://release.niem.gov/niem/niem-core/5.0/#",
    "rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
    "xs": "http://www.w3.org/2001/XMLSchema#"
  },
  "nc:PersonBirthDate": {
    "nc:Date": {
      "rdf:value": {
        "@value": "1978-01-01",
        "@type": "xs:date"
      }
    }
  }
}

Example 2, more complex

Take the following XML example:

<?xml version="1.0" encoding="US-ASCII"?>
<nc:Person xmlns:nc="http://release.niem.gov/niem/niem-core/5.0/"
           xmlns:unece="http://release.niem.gov/niem/codes/unece_rec20/5.0/">
  <nc:PersonAgeMeasure>
    <nc:MeasureDecimalValue>42.5</nc:MeasureDecimalValue>
    <unece:TimeUnitCode>ANN</unece:TimeUnitCode>
  </nc:PersonAgeMeasure>
  <nc:PersonBirthDate>
    <nc:Date>1978-01-01</nc:Date>
  </nc:PersonBirthDate>
</nc:Person>

We'd like to enable a very simple JSON representation, utilizing properties preferred by the developers of the message specification:

{
  "person": {
    "age": {
      "decimalValue": 42.5,
      "unit": "ANN"
    },
    "birth": {
      "date": "1978-01-01"
    }
  }
}

This can be facilitated with a context:

{
  "nc": "http://release.niem.gov/niem/niem-core/5.0/#",
  "unece": "http://release.niem.gov/niem/codes/unece_rec20/5.0/#",
  "person": "nc:Person",
  "age": "nc:PersonAgeMeasure",
  "decimalValue": "nc:MeasureDecimalValue",
  "unit": "unece:TimeUnitCode",
  "birth": "nc:PersonBirthDate",
  "date": "nc:Date"
}

JSON-LD compaction yields the following JSON-LD instance. For maximum readability, we here use a context that only has aliases for namespaces. playground link

{
  "@context": {
    "nc": "http://release.niem.gov/niem/niem-core/5.0/#",
    "unece": "http://release.niem.gov/niem/codes/unece_rec20/5.0/#"
  },
  "nc:Person": {
    "nc:PersonAgeMeasure": {
      "unece:TimeUnitCode": "ANN",
      "nc:MeasureDecimalValue": 42.5
    },
    "nc:PersonBirthDate": {
      "nc:Date": "1978-01-01"
    }
  }
}

Within this instance, there are 3 places where the values are literals, but the NIEM data model says they should be objects:

  • unece:TimeUnitCode, which has a string literal, but should have an object with a value based on xs:token.
  • nc:MeasureDecimalValue, which has a number literal, but should have an object with a value based on xs:decimal. Note that JSON-LD by default will type a non-integer number literal as xs:double.
  • nc:Date, which has a string literal, but should have an object with a value of type xs:date.

Step 2: literal-to-object conversion

This stage converts literal values to objects holding properly-typed literal values. Each step includes examples from above.

  • Iterate over each property that has a literal value, where the property's definition requires an object value

    • In the above example, unece:TimeUnitCode, nc:MeasureDecimalValue, and nc:Date have literal values that should be objects.

  • On each property, identify the expected simple type of the simple content of the property according to the NIEM data model.

    • unece:TimeUnitCode has type unece:TimeCodeType, which has simple content of type unece:TimeCodeSimpleType.
    • nc:MeasureDecimalValue has type niem-xs:decimal, which has simple content of type xs:decimal.
    • nc:Date has type niem-xs:date, which has simple content of type xs:date.

  • Identify the nearest base type that appears within the list provided by RDF Concepts Section 5.1, The XML Schema Built-in Datatypes

    • unece:TimeCodeSimpleType restricts xs:token, which is on the list.
    • xs:decimal is on the list.
    • xs:date is on the list.

  • Replace the literal value of the property with an object that follows the template, where:

    • $literal-value is the literal value
    • $type is the nearest base type on the list
    {
  "rdf:value": {
    "@value": "$literal-value",
    "@type": "$type"
  }
}

    • For example, replace "unece:TimeCode": "ANN" with:

      "unece:TimeCode: { 
  "rdf:value": { 
    "@value": "ANN", 
    "@type": "xs:token"
  }
}

      The remaining properties appear in the full instance below.

{
  "@context": {
    "nc": "http://release.niem.gov/niem/niem-core/5.0/#",
    "rdf": "http://www.w3.org/1999/02/22-rdf-syntax-ns#",
    "unece": "http://release.niem.gov/niem/codes/unece_rec20/5.0/#",
    "xs": "http://www.w3.org/2001/XMLSchema#"
  },
  "nc:Person": {
    "nc:PersonAgeMeasure": {
      "unece:TimeUnitCode": {
        "rdf:value": {
          "@value": "ANN",
          "@type": "xs:token"
        }
      },
      "nc:MeasureDecimalValue": {
        "rdf:value": {
          "@value": "42.5",
          "@type": "xs:decimal"
        }
      }
    },
    "nc:PersonBirthDate": {
      "nc:Date": {
        "rdf:value": {
          "@value": "1978-01-01",
          "@type": "xs:date"
        }
      }
    }
  }
}

Clone this wiki locally