XML – Page 2 – Eric van der Vlist

Toward χίμαιραλ/superset

Background

Up to now, my approach to explore possible solutions to support JSON in XDM has been to evaluate the proposal to introduce maps in XSLT 3.0, leading to the χίμαιραλ proposal.

Finding out that in the current XSLT 3.0 Working Draft, JSON objects would be second class XDM citizens, I think that it’s worth considering other approaches and consider Jürgen Rennau’s UDL proposal has a very good starting point.

In their comments after Rennau’s presentation, John Cowan and Steven DeRose stressed out the need to forget about the serialization and focus on the data model, at least as a first step, when proposing alternative solutions.

Following their advise, I’d like to propose an update to core XDM 3.0 that would natively support JSON objects without introducing any new item kind.

Because this proposal is a chimera and because it’s a superset of the current XDM (any current XDM instance would be compatible with the updates I am proposing), I’ll call this proposal χίμαιραλ/superset.

The XDM carefully avoids to draw class models and prefers to specify nodes through nodes kinds and accessors. However, I think fair to say that elements are defined as nodes with three basic properties:

A mandatory name which is a QName.
A map of attributes which keys are QName.
An array of children nodes.

On the other hand, the JSON data model is composed of arrays, objects and atomic values. JSON atomic values can be numbers, strings, booleans or null. JSON key arrays can be any atomic values.

Traditional approaches to bind JSON and XML use XML element’s children to represent JSON object properties and UDL is no different in that matter.

There are obvious good reasons to do, the main one being that XML attribute values can only be atomic values while JSON object properties can also be maps or arrays.

However, the end result is that the interface mismatch resulting from binding JSON objects (which are maps) into XML children (which are arrays) is at the origin of most of the complexity of these proposals.

Proposal ( χίμαιραλ/superset)

Since XML elements already have both a map (of attributes) and an array (of children nodes), why not use these native features to bind JSON maps and arrays and just update the data model to remove the restrictions that make attribute maps and children arrays unfit for being bound respectively to JSON objects and arrays?

A JSON object would then just be bound to an XML element with attributes and without children nodes and a JSON array would be bound to an XML element with children nodes and without attribute.

This seems especially easy if we focus on the data model and postpone the (optional) definition of a serialization syntax.

First modification: elements can be anonymous

Neither JSON objects nor JSON arrays have name but XML elements have names and this name is mandatory.

To fix this issue, element names should become optional (in other words, we introduce the notion of anonymous elements).

Second modification: attribute names should also possibly be strings, booleans or null

If we bind JSON object keys on attribute names, it should be possible to use all the basic types that JSON accept for its keys.

Additionally, we may want to consider supporting other XML Schema simple types, possibly each of them.

To make this possible, the definition of the dm:node-name() accessor should be updated to return a typed values rather than a QName. This modification should concern attribute nodes at minima but for maximum homogeneity, we should probably extend that to other node types.

Third and last modification: attributes should have (optional) attributes and children

JSON object values can be objects and arrays and since objects are bound to attributes and arrays are bound to children, attributes should support both.

Mapping JSON into χίμαιραλ/superset

With these updates, binding JSON into XML become quite straightforward:

A JSON object is mapped into an anonymous element without children and one attribute per key/value pair.
A JSON array is mapped into an anonymous element without attribute and a child element per item.

Let’s take the now famous (at least on this blog) JSON snippet borrowed from the XSLT 3.0 Working Draft:

{ "accounting" : [
      { "firstName" : "John",
        "lastName"  : "Doe",
        "age"       : 23 },

      { "firstName" : "Mary",
        "lastName"  : "Smith",
        "age"       : 32 }
                 ],                                
  "sales"     : [
      { "firstName" : "Sally",
        "lastName"  : "Green",
        "age"       : 27 },

      { "firstName" : "Jim", 
        "lastName"  : "Galley",
        "age"       : 41 }
                  ]
}

Becomes:

Anonymous element without children and two attributes:
- Attribute « accounting » (as a string) with no attributes and the two following children:
  - Anonymous element with no children and the three following attributes:
    - Attribute « firstName » (as a string) and a value « John » (as a string)
    - Attribute « lastName » (as a string) and a value « Doe » (as a string)
    - Attribute « age » (as a string) and a value 23 (as a number)
  - Anonymous element with no children and the three following attributes:
    - Attribute « firstName » (as a string) and a value « Mary » (as a string)
    - Attribute « lastName » (as a string) and a value « Smith » (as a string)
    - Attribute « age » (as a string) and a value 32 (as a number)
- Attribute « sales » (as a string) with no attributes and the two following children:
  - Anonymous element with no children and the three following attributes:
    - Attribute « firstName » (as a string) and a value « Sally » (as a string)
    - Attribute « lastName » (as a string) and a value « Green » (as a string)
    - Attribute « age » (as a string) and a value 27 (as a number)
  - Anonymous element with no children and the three following attributes:
    - Attribute « firstName » (as a string) and a value « Jim » (as a string)
    - Attribute « lastName » (as a string) and a value « Galley » (as a string)
    - Attribute « age » (as a string) and a value 41 (as a number)

What do you think (comments very welcome)!

Fleshing the XDM chimera

Note: this article is derived from the presentation I have given at Balisage (precedings). See also the slides (and video) used during this presentation.

Abstract

The XQuery and XPath Data Model 3.0 (XDM) is the kernel of the XML ecosystem. XDM had been extended with foreign item types to embrace new data sources such as JSON, taking the risk to become a chimera. This talk explores some ways to move this fundamental piece of the XML stack forward.

Table of Contents

Motivation

XML Data Models

XPath/XSLT 1.0
XDML 2.0: XPath 2.0/XSLT 2.0/XQuery 1.0
XDM 3.0: XPath 3.0/XSLT 3.0/XQuery 3.0

Identity Crisis

Introducing χίμαιραλ (chimeral), the Chimera Language

Example
χίμαιραλ In a Nutshell
Remaining Issues
χίμαιραλ and the identity crisis

Moving the chimera forward

Embracing RDF
Syntactical sugar
XPath
Validation

Conclusion

Motivation

	Chimera (mythology): The Chimera (also Chimaera or Chimæra) (Greek: Χίμαιρα, Khimaira, from χίμαρος, khimaros, « she-goat ») was, according to Greek mythology, a monstrous fire-breathing female creature of Lycia in Asia Minor, composed of the parts of multiple animals: upon the body of a lioness with a tail that ended in a snake’s head, the head of a goat arose on her back at the center of her spine. The Chimera was one of the offspring of Typhon and Echidna and a sibling of such monsters as Cerberus and the Lernaean Hydra. The term chimera has also come to describe any mythical animal with parts taken from various animals and, more generally, an impossible or foolish fantasy.
	—Wikipedia

	Chimera (genetics): A chimera or chimaera is a single organism (usually an animal) that is composed of two or more different populations of genetically distinct cells that originated from different zygotes involved in sexual reproduction. If the different cells have emerged from the same zygote, the organism is called a mosaic. Chimeras are formed from at least four parent cells (two fertilized eggs or early embryos fused together). Each population of cells keeps its own character and the resulting organism is a mixture of tissues.
	—Wikipedia

During her opening keynote at XML Prague 2012, speaking about the relation between XML, HTML, JSON and RDF, Jeni Tennison warned us against the temptation to create chimeras: [chimera are usually ugly, foolish or impossible fantasies].

The next morning, Michael Kay and Jonathan Robie came to present new features in XPath/XQuery/XSLT 3.0. A lot of these features are directly based on the XQuery and XPath Data Model 3.0 (aka XDM):

The XPath Data Model is the abstraction over which XPath expressions are evaluated. Historically, all of the items in the data model could be derived directly (nodes) or indirectly (typed values, sequences) from an XML document. However, as the XPath expression language has matured, new features have been added which require additional types of items to appear in the data model. These items have no direct XML serialization, but they are never the less part of the data model.

XDM 3.0 is composed of items from a number of different technologies:

Items from the XML Infoset (nodes, attributes, …)
Datatype information borrowed from the Post Schema Validation Infoset
Sequences
Atomic values
Functions that can also be used to model JSON arrays

	Note
	The feature that will be introduced to model JSON arrays is called « maps » and it will be specified as a XSLT feature in the XSLT 3.0 recommendation (not published yet). The XSLT 3.0 editor, Michael Kay has published an early version of this feature in his blog. In this paper, XDM 3.0 will refer to the XSLT 3.0 data model (the XPath 3.0 data model augmented with maps).

XDM 3.0 being a single data model composed of items from different data models, it is fair to say that it is a chimera!

Following Jeni Tennison on stage, I have tried to show that in a world where HTML 5 on one hand and JSON on the other hand are gaining traction, XML has become an ecosystem in a competitive environment and that it’s data model is a major competitive advantage.

Among other factors, the continued success of XML will thus come from its ability to seamlessly integrate other data models such as JSON.

If we follow this conclusion, we must admit that this chimera is essential to the future of XML and do our best to make it elegant and smart.

XML Data Models

Whether it’s a bug or a feature could be debated endlessly, but a remarkable feature of the XML recommendation it’s all about syntax and parsing rule and does not really define a data model. The big advantage is that everyone can find pretty much what he wants in XML documents but for the sake of this paper we need to choose a well known -and well defined- data model to work on.

The most common XML data model is probably the data model defined by the trio XPath/XSLT/XQuery known as « XDM » since XPath version 2.0 and that’s the one we will choose.

XDM version 3.0, still work in progress, will be the third version of this data model. It’s important to understand its design and evolution to use its most advanced features and we’ll start our prospective by a short history of its versions.

XPath/XSLT 1.0

The XPath 1.0 data model is described as being composed of seven types of nodes (root, elements, text, attributes, namespaces, processing instructions and comments).

The XSLT 1.0 data model is defined as being the XPath 1.0 data model with:

Relaxed constraints on root node children to support well-formed external general parsed entities that are not well formed XML documents
An additional « base URI » property on every node.
An additional « unparsed entities » property on the root node.

It’s fair to say that these two -very close- data models are completely focused on XML, but is that all?

Not entirely and these two specifications introduce other notions that should be considered as related to the data model even if they are not described in their sections called « Data Model »…

XSLT 1.0 inadvertently mentions “the four basic XPath data-types (string, number, boolean, node-set)” to explicitly add a fifth one: “result tree fragments”.

These four basic data-types are implicitly defined in XPath 1.0 in its section about its function library but no formal description of these types is given.

XDML 2.0: XPath 2.0/XSLT 2.0/XQuery 1.0

In version 2.0, the XDM is promoted to get its own specification.

XDM 2.0 keeps the same seven types of nodes as XPath 1.0 and integrates the additions from the XSLT 1.0 data model. A number of properties are added to these nodes to capture information that had been left outside the data model by the previous version and also to support the data-type system from the PSVI (Post Schema Validation Infoset).

The term « data-type » or simply « type » being now used to refer to XML Schema data-types, a new terminology is introduced where the data model is composed of « information items » (or items) being either XML nodes or « atomic values ».

The concept of « sequences » is also introduced. Sequences are not strictly considered as items but play a very important role in XDM. They are defined as “an ordered collection of zero or more items”.

The data model is thus now composed of three different concepts:

nodes
atomic values
sequences

XDM 2.0 notes that an important difference between nodes and atomic values is that only nodes have identities:

	Each node has a unique identity. Every node in an instance of the data model is unique: identical to itself, and not identical to any other node. (Atomic values do not have identity; every instance of the value “5” as an integer is identical to every other instance of the value “5” as an integer.)
	—XQuery 1.0 and XPath 2.0 Data Model (XDM) (Second Edition)

This is a crucial distinction that divides the data model into two different kind of items (those which have an identity and those which haven’t one). Let’s take an example:

<?xml version="1.0" encoding="UTF-8"?>
<root>
    <foo>5</foo>
    <foo>5</foo>
    <bar foo="5">
        <foo>5</foo>
    </bar>
</root>

The three <foo>5</foo> look similar and can be considered « deeply equal » but they are three different elements with three different identities. This is needed because some of their properties are different: the parent of the first two is <root/> while the parent of the third one is <bar/>, the preceding sibling of the second one is the first one while the first one has no preceeding sibling, …

The three « 5 » text nodes are similar but they still are different text nodes with different identities and this is necessary because they don’t have the same parent elements.

By contrast, the atomic values of the three <foo/> element (and the atomic value of the @foo attribute) are the same atomic value, the « 5 » (assuming they have all been declared with the same datatype). Among many other things, this means that when you manipulate their values, you can’t access back to the node that is holding the value).

XDM 3.0: XPath 3.0/XSLT 3.0/XQuery 3.0

	Note
	These specifications are still work on progress, currently divided between XQuery and XPath Data Model 3.0and data model extensions described in XSL Transformations (XSLT) Version 3.0.

XDM 3.0 adds functions as a third kind of items, transforming XQuery and XSLT into functional languages.

Like atomic values, functions have no identity:

Functions have no identity, cannot be compared, and have no serialization.

—XQuery and XPath Data Model 3.0 – W3C Working Draft 13 December

2011

XSLT 3.0 adds to XDM 3.0 a fourth king of items: maps, derived from functions which, among many other use cases, can be used to model JSON objects:

Like atomic values and functions (from which they are derived), maps have no identity:

	Like sequences, maps have no identity. It is meaningful to compare the contents of two maps, but there is no way of asking whether they are « the same map »: two maps with the same content are indistinguishable.
	—XSL Transformations (XSLT) Version 3.0 – W3C Working Draft 10 July 2012

	Note
	In this statement, the specification does acknowledge that sequences have no identity either. This is understandable but didn’t seem to be clearly specified elsewhere.

Of course, XSLT 3.0 is also adding functions to create, manipulate maps and serialize/deserialize them as JSON and a syntax to define map literals. It does not any new pattern to select of match maps or map entries, though.

Identity Crisis

	Appolonius’ ship is a beautiful ship. Over the years it has been repaired so many times that there is not a single piece of the original materials remaining. The question is, therefore, is it really still Appolonius’ ship?
	—ObjectIdentity on c2.com

Object identity is often confused with mutability. The need for objects to have identities is more obvious when they are mutable, their identities being then used to track them despite their changes like Appolonius’ ship. However, XDM 3.0 gives us a good opportunity to explore the meaning and consequences of having (or not having) an identity for immutable object structures.

The definition of node identity in XDM 3.0 is directly copied from XDM 2.0:

	Each node has a unique identity. Every node in an instance of the data model is unique: identical to itself, and not identical to any other node. (Atomic values do not have identity; every instance of the value “5” as an integer is identical to every other instance of the value “5” as an integer.)
	—XQuery and XPath Data Model 3.0 – W3C Working Draft 13 December 2011

I find this definition confusing:

Why should the value “5” as an integer be instantiated and why should we care? The value “5” as an integer is… the value “5” as an integer! It’s unique and being unique, doesn’t it have an identity?
A node, with all the properties defined in XDM (including its document-uri and parent accessors) would be unique if it had « previous-sibling » or « document-order » accessors.

	Note
	To find the previous siblings of a node relying only on the accessors defined in XDM (2.0 or 3.0), you’d have to access to the node’s parent and loop over it’s children until you find the current node that you would identify as such by checking its identity.

Rather than focussing on uniqueness, which for immutable information items does not really matter, a better differentiation could be between information items which have enough context information to « know where they belong » in the data model and those which don’t.

This differentiation has the benefit of highlighting the consequences of having or not having an identity: to be able to navigate between an information item and its ancestors or sibling this item must know where it belongs. When that’s not the case, it is still be possible to navigate between the item and its descendants but axis such as ancestor:: or sibling:: are not available.

	Note
	Identity can be seen as the price to pay for the `ancestor::` and `sibling::` axis.

Let’s take back a simple example:

<?xml version="1.0" encoding="UTF-8"?>
<root>
    <foo>5</foo>
    <foo>5</foo>
    <bar> 
        <foo>5</foo>
    </bar>
</root>

In an hypothetical data model where nodes have no identity, there would be only 3 elements:

The root element
The bar element
The foo element (referred twice has children of root end once as child of bar)

If we add identity (or context information) properties, the foo elements become three information different items since they defer by these properties.

The process of adding these properties to an information item looks familiar. Depending on your background, you can compare it to:

class/object instantiation in class based Object Oriented Programming
clones in prototype based Object Oriented Programming
RDF reification.

We’ve seen that XDM 3.0 acknowledges this difference between information items which have context information and those which don’t have. I don’t want to deny that both types of data models have their use cases: there are obviously many use cases where context information is needed and use cases where lightweight structures are a better fit.

That being said, if we are serious about the support of JSON in XDM, we should offer the same features to access data whether this data is stored in maps or in XML nodes.

Let’s consider this JSON object borrowed from the XSLT 3.0 Working Draft:

{ "accounting" : [ 
      { "firstName" : "John", 
        "lastName"  : "Doe",
        "age"       : 23 },

      { "firstName" : "Mary", 
        "lastName"  : "Smith",
        "age"       : 32 }
                 ],                                 
  "sales"     : [ 
      { "firstName" : "Sally", 
        "lastName"  : "Green",
        "age"       : 27 },

      { "firstName" : "Jim",  
        "lastName"  : "Galley",
        "age"       : 41 }
                  ]
}

This object could be represented in XML by the following document:

<?xml version="1.0" encoding="UTF-8"?>
<company>
    <department name="sales">
        <employee>
            <firstName>Sally</firstName>
            <lastName>Green</lastName>
            <age>27</age>
        </employee>
        <employee>
            <firstName>Jim</firstName>
            <lastName>Galley</lastName>
            <age>41</age>
        </employee>
    </department>
    <department name="accounting">
        <employee>
            <firstName>John</firstName>
            <lastName>Doe</lastName>
            <age>23</age>
        </employee>
        <employee>
            <firstName>Mary</firstName>
            <lastName>Smith</lastName>
            <age>32</age>
        </employee>
    </department>
</company>

The features introduced in the latest XSLT 3.0 Working Draft do allow to transform rather easily from one model to the other, but these two models do not have, bar far, the same features.

In the XML flavor, when the context item is the employee « John Doe », you can easily find out what his department is because this is an element and element do carry context information.

In the map flavor by contrast when the context item is an employee map, this object has no context information and you can’t tell which is his department without looping within the containing map.

This important restriction is at a purely data model level. It is aggravated by the XPath syntax has not been extended to generalize axis so that they can work with maps. If I work with the XML version of this structure, it’s obvious to evaluate things such as the number of employees, the average age of employees, the number of departments, the number of employees by department, the average age by department, obvious to find out if there is an employee called « Mary Smith » in one of the departments, the employees who are more than 40, to get a list of employees from all the department sorted by age, … In the map flavor by contrast, I don’t have any XPath axis available and must do all these operations using a limited number of map functions (map:keys(), map:contains(), map:get()). In other words, while I can use XPath expressions with the XML version, I must use DOM like operations to access the map version!

To summarize, yes XDM 3.0 does support JSON but to do pretty much anything interesting with JSON objects, you’d better transform them into XML nodes first! XSLT 3.0 does give you the tools to do this transformation quite easily but the message to JSON users is that we don’t treat their data model as a first class citizen.

To make it worse, XPath is used by many other specifications, within and outside the W3C and the level of support for JSON provided by XDM and XPath will determine how these specifications will be able to support for JSON. Specifications that are impacted by this issue include XForms, XProc and Schematron. Supporting JSON would be really useful for these three specifications if and only if map items could have the same features than nodes.

Furthermore, the same asymmetry exists when you went to create these two structures from other sources: to create the XML structure you can use sequence constructors but to create the map structure, you have to use the map:new() and map:item() functions.

My proposal to solve this issue is:

To acknowledge the fact that any type of information item can be either « context independent » or include context information and explore the consequences of thisstatement.
To generalize XPath axis so that they can be used with map items.
To create sequence constructors for maps and map entries.

You are welcome to discuss this further:

W3C XSLT 3.0 Bug 16118, « Maps should be first class citizens »
Blog entry: XDM Maps should be first class citizens.
Blog entry: More musings on XDM 3.0

Introducing χίμαιραλ (chimeral), the Chimera Language

When I started to work on χίμαιραλ a few months ago, my first motivation was to propose an XDM serialization for maps which would turn the rather abstract prose from the specification into concrete angle brackets that you could see and read.

The exercise has been very instructive and helped me a lot to understand the spec, however a more ambitious use pattern has emerged while I was making progress. The XSLT 3.0 Working Draft is part of a batch of Working Drafts which are far more advanced. My proposals to solve the « map identity crisis » are probably too intrusive and too late to be taken into account and the batch of specifications will most probably carry on with the current proposal.

If that’s the case, we’ve seen that it makes a lot of sense to convert maps into nodes to enable to use XPath axis and χίμαιραλ provides a generic target format for these conversions.

Example

Let’s take again the JSON object borrowed from the XSLT 3.0 Working Draft:

{ "accounting" : [ 
      { "firstName" : "John", 
        "lastName"  : "Doe",
        "age"       : 23 },

      { "firstName" : "Mary", 
        "lastName"  : "Smith",
        "age"       : 32 }
                 ],                                 
  "sales"     : [ 
      { "firstName" : "Sally", 
        "lastName"  : "Green",
        "age"       : 27 },

      { "firstName" : "Jim",  
        "lastName"  : "Galley",
        "age"       : 41 }
                  ]
}

Its χίμαιραλ representation is:

<?xml version="1.0" encoding="UTF-8"?>
<χ:data-model xmlns:χ="http://χίμαιραλ.com#">
    <χ:map>
        <χ:entry key="sales" keyType="string">
            <χ:map>
                <χ:entry key="1" keyType="number">
                    <χ:map>
                        <χ:entry key="lastName" keyType="string">
                            <χ:atomic-value type="string">Green</χ:atomic-value>
                        </χ:entry>
                        <χ:entry key="age" keyType="string">
                            <χ:atomic-value type="number">27</χ:atomic-value>
                        </χ:entry>
                        <χ:entry key="firstName" keyType="string">
                            <χ:atomic-value type="string">Sally</χ:atomic-value>
                        </χ:entry>
                    </χ:map>
                </χ:entry>
                <χ:entry key="2" keyType="number">
                    <χ:map>
                        <χ:entry key="lastName" keyType="string">
                            <χ:atomic-value type="string">Galley</χ:atomic-value>
                        </χ:entry>
                        <χ:entry key="age" keyType="string">
                            <χ:atomic-value type="number">41</χ:atomic-value>
                        </χ:entry>
                        <χ:entry key="firstName" keyType="string">
                            <χ:atomic-value type="string">Jim</χ:atomic-value>
                        </χ:entry>
                    </χ:map>
                </χ:entry>
            </χ:map>
        </χ:entry>
        <χ:entry key="accounting" keyType="string">
            <χ:map>
                <χ:entry key="1" keyType="number">
                    <χ:map>
                        <χ:entry key="lastName" keyType="string">
                            <χ:atomic-value type="string">Doe</χ:atomic-value>
                        </χ:entry>
                        <χ:entry key="age" keyType="string">
                            <χ:atomic-value type="number">23</χ:atomic-value>
                        </χ:entry>
                        <χ:entry key="firstName" keyType="string">
                            <χ:atomic-value type="string">John</χ:atomic-value>
                        </χ:entry>
                    </χ:map>
                </χ:entry>
                <χ:entry key="2" keyType="number">
                    <χ:map>
                        <χ:entry key="lastName" keyType="string">
                            <χ:atomic-value type="string">Smith</χ:atomic-value>
                        </χ:entry>
                        <χ:entry key="age" keyType="string">
                            <χ:atomic-value type="number">32</χ:atomic-value>
                        </χ:entry>
                        <χ:entry key="firstName" keyType="string">
                            <χ:atomic-value type="string">Mary</χ:atomic-value>
                        </χ:entry>
                    </χ:map>
                </χ:entry>
            </χ:map>
        </χ:entry>
    </χ:map>
</χ:data-model>

Granted, it’s much more verbose than the JSON version, but it’s the exact translation of the XDM corresponding to the JSON object in XML.

χίμαιραλ In a Nutshell

The design goals are:

Be as close as possible to the XDM and its terminology
Represent XML nodes as… XML nodes
Allow round-trips (an XDM model serialized as χίμαιραλ should give a XDM model identical to the original one when de-serialized)
Be easy to process using XPath/XQuery/XSLT
Support of the PSVI is not a goal

χίμαιραλ is not the only proposal to serialize XDM as XML. Two other notable ones are:

Zorba’s XDM serializationis a straight andaccurate XDM serialization which does support PSVI annotations. As a consequence, nodes are serialized as xdm:*elements (an element is anxdm:element, an attribute an xdm:attribute element, …). This does’n meet by second requirement to represent nodes as themselves.
XDML, presented by Rennau, Hans-Jürgen, and David A. Lee atBalisage 2011 is more than just an XDM serialization and also includes manipulation and processing definitions. It introduces its own terminology and concepts and is toofar away from XDM for my design goals.

A lot of attention has been given to the first design goal: the structure of a χίμαιραλ model and the name of its elements and attributes are directly derived from the specifications.

In XDM, map entries’ values can be arrays (an array beeing nothing else than a map with integer keys) but also sequences (which is not possible in JSON). χίμαιραλ respects the fact that in XDM there is no difference between a sequence composed of a single element and represents sequences by a repetition of values.

The map map{1:= 'foo'} is serialized as:

<χ:data-model xmlns:χ="http://χίμαιραλ.com#">
   <χ:map>
      <χ:entry key="1" keyType="number">
         <χ:atomic-value type="string">foo</χ:atomic-value>
      </χ:entry>
   </χ:map>
</χ:data-model>

And the map map{1:= ('foo', 'bar')} is serialized as:

<χ:data-model xmlns:χ="http://χίμαιραλ.com#">
   <χ:map>
      <χ:entry key="1" keyType="number">
         <χ:atomic-value type="string">foo</χ:atomic-value>
         <χ:atomic-value type="string">bar</χ:atomic-value>
      </χ:entry>
   </χ:map>
</χ:data-model>

We’ve seen that XDM makes a clear distinction between nodes which have identities and other item types (atomic values, functions and maps) which haven’t. XDM allows to use nodes as map entry values. χίμαιραλ allows this feature too, but copying the nodes would create new nodes with different identities.

To avoid that, documents to which these nodes belong are copied into χ:instance elements and references between map entries values and instances are made using XPath expressions.

The following $map variable:

<xsl:variable name="a-node">
    <foo/>
</xsl:variable>
<xsl:variable name="map" select="map{'a-node':= $a-node}"/>

Is serialized as:

<χ:data-model xmlns:χ="http://χίμαιραλ.com#">
   <χ:instance id="d4" kind="document">
      <foo/>
   </χ:instance>
   <χ:map>
      <χ:entry key="a-node" keyType="string">
         <χ:node kind="document" instance="d4" path="/"/>
      </χ:entry>
   </χ:map>
</χ:data-model>

Like XSLT variable, instances do not always contain document nodes and the following $map variable:

<xsl:variable name="a-node" as="node()">
    <foo/>
</xsl:variable>
<xsl:variable name="map" select="map{'a-node':= $a-node}"/>

Is serialized as:

<χ:data-model xmlns:χ="http://χίμαιραλ.com#">
   <χ:instance id="d4e0" kind="fragment">
      <foo/>
   </χ:instance>
   <χ:map>
      <χ:entry key="a-node" keyType="string">
         <χ:node kind="element" instance="d4e0" path="root()" name="foo"/>
      </χ:entry>
   </χ:map>
</χ:data-model>

Nodes can belong to more than one instances, and this $map variable:

<xsl:variable name="a-node" as="node()*">
    <foo/>
    <bar/>
</xsl:variable>
<xsl:variable name="map" select="map{'a-node':= $a-node}"/>

Is serialized as:

<χ:data-model xmlns:χ="http://χίμαιραλ.com#">
   <χ:instance id="d4e0" kind="fragment">
      <foo/>
   </χ:instance>
   <χ:instance id="d4e3" kind="fragment">
      <bar/>
   </χ:instance>
   <χ:map>
      <χ:entry key="a-node" keyType="string">
         <χ:node kind="element" instance="d4e0" path="root()" name="foo"/>
         <χ:node kind="element" instance="d4e3" path="root()" name="bar"/>
      </χ:entry>
   </χ:map>
</χ:data-model>

Nodes can be « deep linked », a same node can be linked several times and nodes can be mixed with atomic values at wish. The following $map variable:

<xsl:variable name="doc">
    <department name="sales">
        <employee>
            <firstName>Sally</firstName>
            <lastName>Green</lastName>
            <age>27</age>
        </employee>
        <employee>
            <firstName>Jim</firstName>
            <lastName>Galley</lastName>
            <age>41</age>
        </employee>
    </department>
    <department name="accounting">
        <employee>
            <firstName>John</firstName>
            <lastName>Doe</lastName>
            <age>23</age>
        </employee>
        <employee>
            <firstName>Mary</firstName>
            <lastName>Smith</lastName>
            <age>32</age>
        </employee>
    </department>
</xsl:variable>
<xsl:variable name="map"
    select="map{
            'sales' := $doc/department[@name='sales'],
            'Sally' := $doc//employee[firstName = 'Sally'],
            'kids'  := $doc//employee[age &lt; 30],
            'dep-names-attributes' := $doc/department/@name,
            'dep-names' := for $name in $doc/department/@name return string($name)
            }"/>

Is serialized as:

<χ:data-model xmlns:χ="http://χίμαιραλ.com#">
   <χ:instance id="d4" kind="document">
      <department name="sales">
         <employee>
            <firstName>Sally</firstName>
            <lastName>Green</lastName>
            <age>27</age>
         </employee>
         <employee>
            <firstName>Jim</firstName>
            <lastName>Galley</lastName>
            <age>41</age>
         </employee>
      </department>
      <department name="accounting">
         <employee>
            <firstName>John</firstName>
            <lastName>Doe</lastName>
            <age>23</age>
         </employee>
         <employee>
            <firstName>Mary</firstName>
            <lastName>Smith</lastName>
            <age>32</age>
         </employee>
      </department>
   </χ:instance>
   <χ:map>
      <χ:entry key="sales" keyType="string">
         <χ:node kind="element"
                 instance="d4"
                 path="/&#34;&#34;:department[1]"
                 name="department"/>
      </χ:entry>
      <χ:entry key="Sally" keyType="string">
         <χ:node kind="element"
                 instance="d4"
                 path="/&#34;&#34;:department[1]/&#34;&#34;:employee[1]"
                 name="employee"/>
      </χ:entry>
      <χ:entry key="kids" keyType="string">
         <χ:node kind="element"
                 instance="d4"
                 path="/&#34;&#34;:department[1]/&#34;&#34;:employee[1]"
                 name="employee"/>
         <χ:node kind="element"
                 instance="d4"
                 path="/&#34;&#34;:department[2]/&#34;&#34;:employee[1]"
                 name="employee"/>
      </χ:entry>
      <χ:entry key="dep-names-attributes" keyType="string">
         <χ:node kind="attribute"
                 instance="d4"
                 path="/&#34;&#34;:department[1]/@name"
                 name="name">sales</χ:node>
         <χ:node kind="attribute"
                 instance="d4"
                 path="/&#34;&#34;:department[2]/@name"
                 name="name">accounting</χ:node>
      </χ:entry>
      <χ:entry key="dep-names" keyType="string">
         <χ:atomic-value type="string">sales</χ:atomic-value>
         <χ:atomic-value type="string">accounting</χ:atomic-value>
      </χ:entry>
   </χ:map>
</χ:data-model>

Remaining Issues

A collation property should be added to <χ:map/>, probably as an attribute, the transformation to serialize to χίμαιραλ should be cleaned up and the reverse transformation should be implemented.

These are pretty trivial issues and the biggest one is probably to find a way to cleanly serialize references to nodes that are not contained within an element, such as the following $map variable:

<xsl:variable name="attribute" as="node()">
    <xsl:attribute name="foo">bar</xsl:attribute>
</xsl:variable>
<xsl:variable name="map"
    select="map{
            'attribute' := $attribute
            }"/>

Support of functions should also be considered.

χίμαιραλ and the identity crisis

To some extend, χίμαιραλ can be considered as a solution to the XDM identity crisis:

Serializing an XDM model as χίμαιραλ creates elements for maps, map entries and atomic values and these elements, being nodes, have identities. The serialization istherefore also an instantiation of XDM information items as defined above.
De-serializing a χίμαιραλ to create an XDM data model is also a de-instantiation– except of course that the identity of XML nodes is not « removed ».

However, χίμαιραλ does keep a strong difference between nodes which are kept in <χ:instance> elements and maps and atomic values.

Moving the chimera forward

χίμαιραλ is a good playground to explore the new possibilities offered by XDM 3.0. Here is a (non exhaustive) list of a few directions that seem interesting…

	Note
	Don’t expect to find fully baked proposals in this section which contains, on the contrary very early drafts of ideas to follow to support XDM maps as « first class citizens »!

Embracing RDF

If you had the opportunity to enjoy the sunny weather of Orlando in December 2001, you may remember « The Syntactic Web » a provocative talk where Jonathan Robie has shown how XQuery 1.0 could be used to query normalized XML/RDF documents.

The gap between RDF triples and the versatility of its XML representation was a big issue, but the new features brought by this new version of the XPath/XQuery/XSLT package should help us.

The basic data model of RDF is based on triples, a triple being a composed of a subject, a predicate and an object. In XDM, a triple can now be represented by either a sequence, an array or a map of three items.

XDM sequences have the property that they cannot include other sequences and representing triples as sequences would mean that you couldn’t define sequences of triples. For that reason it is probably better to define triples as maps or arrays. An array being a map indexed by integers, that doesn’t make a huge difference at a conceptual level, but I find it cleaner to access to the subject of a triple using a QName (such as rdf:subject) rather than an index. Following this principle, we could define a triple as:

map {
    xs:QName('rdf:subject')   := xs:anyURI('http://www.example.org/index.html'),
    xs:QName('rdf:predicate') := xs:anyURI('http://purl.org/dc/elements/1.1/creator'),
    xs:QName('rdf:object')    := xs:anyURI('http://www.example.org/staffid/85740')
}

The χίμαιραλ serialization of this map is:

<χ:data-model xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:χ="http://χίμαιραλ.com#"
              xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
   <χ:map>
      <χ:entry key="rdf:object"
               keyType="xs:QName">
         <χ:atomic-value type="xs:anyURI">http://www.example.org/staffid/85740</χ:atomic-value>
      </χ:entry>
      <χ:entry key="rdf:predicate"
               keyType="xs:QName">
         <χ:atomic-value type="xs:anyURI">http://purl.org/dc/elements/1.1/creator</χ:atomic-value>
      </χ:entry>
      <χ:entry key="rdf:subject"
               keyType="xs:QName">
         <χ:atomic-value type="xs:anyURI">http://www.example.org/index.html</χ:atomic-value>
      </χ:entry>
   </χ:map>
</χ:data-model>

What can we do with such triples? Using higher order functions, it should not be too difficult to define triple stores with basic query features!

Is this lightweight enough? Or does RDF support deserve new information item types to be supported by XDM?

Syntactical sugar

We’ve seen that this JSON object

{ "accounting" : [ 
      { "firstName" : "John", 
        "lastName"  : "Doe",
        "age"       : 23 },

      { "firstName" : "Mary", 
        "lastName"  : "Smith",
        "age"       : 32 }
                 ],                                 
  "sales"     : [ 
      { "firstName" : "Sally", 
        "lastName"  : "Green",
        "age"       : 27 },

      { "firstName" : "Jim",  
        "lastName"  : "Galley",
        "age"       : 41 }
                  ]
}

Is serialized in χίμαιραλ as:

<?xml version="1.0" encoding="UTF-8"?>
<χ:data-model xmlns:χ="http://χίμαιραλ.com#">
    <χ:map>
        <χ:entry key="sales" keyType="string">
            <χ:map>
                <χ:entry key="1" keyType="number">
                    <χ:map>
                        <χ:entry key="lastName" keyType="string">
                            <χ:atomic-value type="string">Green</χ:atomic-value>
                        </χ:entry>
                        <χ:entry key="age" keyType="string">
                            <χ:atomic-value type="number">27</χ:atomic-value>
                        </χ:entry>
                        <χ:entry key="firstName" keyType="string">
                            <χ:atomic-value type="string">Sally</χ:atomic-value>
                        </χ:entry>
                    </χ:map>
                </χ:entry>
                <χ:entry key="2" keyType="number">
                    <χ:map>
                        <χ:entry key="lastName" keyType="string">
                            <χ:atomic-value type="string">Galley</χ:atomic-value>
                        </χ:entry>
                        <χ:entry key="age" keyType="string">
                            <χ:atomic-value type="number">41</χ:atomic-value>
                        </χ:entry>
                        <χ:entry key="firstName" keyType="string">
                            <χ:atomic-value type="string">Jim</χ:atomic-value>
                        </χ:entry>
                    </χ:map>
                </χ:entry>
            </χ:map>
        </χ:entry>
        <χ:entry key="accounting" keyType="string">
            <χ:map>
                <χ:entry key="1" keyType="number">
                    <χ:map>
                        <χ:entry key="lastName" keyType="string">
                            <χ:atomic-value type="string">Doe</χ:atomic-value>
                        </χ:entry>
                        <χ:entry key="age" keyType="string">
                            <χ:atomic-value type="number">23</χ:atomic-value>
                        </χ:entry>
                        <χ:entry key="firstName" keyType="string">
                            <χ:atomic-value type="string">John</χ:atomic-value>
                        </χ:entry>
                    </χ:map>
                </χ:entry>
                <χ:entry key="2" keyType="number">
                    <χ:map>
                        <χ:entry key="lastName" keyType="string">
                            <χ:atomic-value type="string">Smith</χ:atomic-value>
                        </χ:entry>
                        <χ:entry key="age" keyType="string">
                            <χ:atomic-value type="number">32</χ:atomic-value>
                        </χ:entry>
                        <χ:entry key="firstName" keyType="string">
                            <χ:atomic-value type="string">Mary</χ:atomic-value>
                        </χ:entry>
                    </χ:map>
                </χ:entry>
            </χ:map>
        </χ:entry>
    </χ:map>
</χ:data-model>

We can work with that, but wouldn’t it be nice if we had a native syntax that does not use XML elements and attributes to represent maps?

Depending on the requirements, many approaches are possible.

A first option would be to define pluggable notation parsers within XML and write:

<χ:notation mediatype="application/json"><![CDATA[
{ "accounting" : [ 
      { "firstName" : "John", 
        "lastName"  : "Doe",
        "age"       : 23 },

      { "firstName" : "Mary", 
        "lastName"  : "Smith",
        "age"       : 32 }
                 ],                                 
  "sales"     : [ 
      { "firstName" : "Sally", 
        "lastName"  : "Green",
        "age"       : 27 },

      { "firstName" : "Jim",  
        "lastName"  : "Galley",
        "age"       : 41 }
                  ]
}                  
]]></χ:notation>

The meaning of the <χ:notation/> element would be to trigger a parser supporting the application/json datatype. This is less verbose, more natural to JSON users, but doesn’t allow to add XML nodes in maps or sequences.

Another direction would be to extend the syntax of XML itself. To do so, again, there are many possibilities. The markup in XML is based on angle brackets and the distinction between the different XML productions is usually done through the character following the bracket in the opening tags.

This principle leaves a lot of possibilities. For instance, maps could be identified by the tags <{> and </}> to follow the characters used by XDM map literals and JSON objects:

<χ:data-model>
    <{>
        <χ:entry key="sales" keyType="string">
            <{>
                <χ:entry key="1" keyType="number">
                    <{>
                        <χ:entry key="lastName" keyType="string">
                            <χ:atomic-value type="string">Green</χ:atomic-value>
                        </χ:entry>
                        <χ:entry key="age" keyType="string">
                            <χ:atomic-value type="number">27</χ:atomic-value>
                        </χ:entry>
                        <χ:entry key="firstName" keyType="string">
                            <χ:atomic-value type="string">Sally</χ:atomic-value>
                        </χ:entry>
                    </}>
                </χ:entry>
                <χ:entry key="2" keyType="number">
                    <{>
                        <χ:entry key="lastName" keyType="string">
                            <χ:atomic-value type="string">Galley</χ:atomic-value>
                        </χ:entry>
                        <χ:entry key="age" keyType="string">
                            <χ:atomic-value type="number">41</χ:atomic-value>
                        </χ:entry>
                        <χ:entry key="firstName" keyType="string">
                            <χ:atomic-value type="string">Jim</χ:atomic-value>
                        </χ:entry>
                    </}>
                </χ:entry>
            </}>
        </χ:entry>
        <χ:entry key="accounting" keyType="string">
            <{>
                <χ:entry key="1" keyType="number">
                    <{>
                        <χ:entry key="lastName" keyType="string">
                            <χ:atomic-value type="string">Doe</χ:atomic-value>
                        </χ:entry>
                        <χ:entry key="age" keyType="string">
                            <χ:atomic-value type="number">23</χ:atomic-value>
                        </χ:entry>
                        <χ:entry key="firstName" keyType="string">
                            <χ:atomic-value type="string">John</χ:atomic-value>
                        </χ:entry>
                    </}>
                </χ:entry>
                <χ:entry key="2" keyType="number">
                    <{>
                        <χ:entry key="lastName" keyType="string">
                            <χ:atomic-value type="string">Smith</χ:atomic-value>
                        </χ:entry>
                        <χ:entry key="age" keyType="string">
                            <χ:atomic-value type="number">32</χ:atomic-value>
                        </χ:entry>
                        <χ:entry key="firstName" keyType="string">
                            <χ:atomic-value type="string">Mary</χ:atomic-value>
                        </χ:entry>
                    </}>
                </χ:entry>
            </}>
        </χ:entry>
    </}>
</χ:data-model>

Map entries are not ordered and in that respect they are similar to XML attributes. We could use this similarity and use the character @ to identify map entries:

<χ:data-model>
    <{>
        <@"sales" keyType="string">
            <{>
                <@"1" keyType="number">
                    <{>
                        <@"lastName" keyType="string">
                            <χ:atomic-value type="string">Green</χ:atomic-value>
                        </@"lastName">
                        <@"age" keyType="string">
                            <χ:atomic-value type="number">27</χ:atomic-value>
                        </@"age">
                        <@"firstName" keyType="string">
                            <χ:atomic-value type="string">Sally</χ:atomic-value>
                        </@"firstName">
                    </}>
                </@"1">
                <@"2" keyType="number">
                    <{>
                        <@"lastName" keyType="string">
                            <χ:atomic-value type="string">Galley</χ:atomic-value>
                        </@"lastName">
                        <@"age" keyType="string">
                            <χ:atomic-value type="number">41</χ:atomic-value>
                        </@"age">
                        <@"firstName" keyType="string">
                            <χ:atomic-value type="string">Jim</χ:atomic-value>
                        </@"firstName">
                    </}>
                </@"2">
            </}>
        </@"sales">
        <@"accounting" keyType="string">
            <{>
                <@"1" keyType="number">
                    <{>
                        <@"lastName" keyType="string">
                            <χ:atomic-value type="string">Doe</χ:atomic-value>
                        </@"lastName">
                        <@"age" keyType="string">
                            <χ:atomic-value type="number">23</χ:atomic-value>
                        </@"age">
                        <@"firstName" keyType="string">
                            <χ:atomic-value type="string">John</χ:atomic-value>
                        </@"firstName">
                    </}>
                </@"1">
                <@"2" keyType="number">
                    <{>
                        <@"lastName" keyType="string">
                            <χ:atomic-value type="string">Smith</χ:atomic-value>
                        </@"lastName">
                        <@"age" keyType="string">
                            <χ:atomic-value type="number">32</χ:atomic-value>
                        </@"age">
                        <@"firstName" keyType="string">
                            <χ:atomic-value type="string">Mary</χ:atomic-value>
                        </@"firstName">
                    </}>
                </@"2">
            </}>
        </@"accounting">
    </}>
</χ:data-model>

The key names have been enclosed between quotes because map keys can include any character including whitespaces, but they can be made optional when they are not needed. We could also give to the keyType a default value of « string »:

<χ:data-model>
    <{>
        <@sales>
            <{>
                <@1 keyType="number">
                    <{>
                        <@lastName>
                            <χ:atomic-value type="string">Green</χ:atomic-value>
                        </@lastName
                        <@age>
                            <χ:atomic-value type="number">27</χ:atomic-value>
                        </@age
                        <@firstName>
                            <χ:atomic-value type="string">Sally</χ:atomic-value>
                        </@firstName
                    </}>
                </@1
                <@2 keyType="number">
                    <{>
                        <@lastName>
                            <χ:atomic-value type="string">Galley</χ:atomic-value>
                        </@lastName
                        <@age>
                            <χ:atomic-value type="number">41</χ:atomic-value>
                        </@age
                        <@firstName>
                            <χ:atomic-value type="string">Jim</χ:atomic-value>
                        </@firstName
                    </}>
                </@2
            </}>
        </@sales
        <@accounting>
            <{>
                <@1 keyType="number">
                    <{>
                        <@lastName>
                            <χ:atomic-value type="string">Doe</χ:atomic-value>
                        </@lastName
                        <@age>
                            <χ:atomic-value type="number">23</χ:atomic-value>
                        </@age
                        <@firstName>
                            <χ:atomic-value type="string">John</χ:atomic-value>
                        </@firstName
                    </}>
                </@1
                <@2 keyType="number">
                    <{>
                        <@lastName>
                            <χ:atomic-value type="string">Smith</χ:atomic-value>
                        </@lastName
                        <@age>
                            <χ:atomic-value type="number">32</χ:atomic-value>
                        </@age
                        <@firstName>
                            <χ:atomic-value type="string">Mary</χ:atomic-value>
                        </@firstName
                    </}>
                </@2
            </}>
        </@accounting
    </}>
</χ:data-model>

Atomic values could be identified by <=> and </=> and the same default value applied to its type attribute:

<χ:data-model>
    <{>
        <@sales>
            <{>
                <@1 keyType="number">
                    <{>
                        <@lastName>
                            <=>Green</=>
                        </@lastName>
                        <@age>
                            <= type="number">27</=>
                        </@age>
                        <@firstName>
                            <=>Sally</=>
                        </@firstName>
                    </}>
                </@1>
                <@2 keyType="number">
                    <{>
                        <@lastName>
                            <=>Galley</=>
                        </@lastName>
                        <@age>
                            <= type="number">41</=>
                        </@age>
                        <@firstName>
                            <=>Jim</=>
                        </@firstName>
                    </}>
                </@2>
            </}>
        </@sales>
        <@accounting>
            <{>
                <@1 keyType="number">
                    <{>
                        <@lastName>
                            <=>Doe</=>
                        </@lastName>
                        <@age>
                            <= type="number">23</=>
                        </@age>
                        <@firstName>
                            <=>John</=>
                        </@firstName>
                    </}>
                </@1>
                <@2 keyType="number">
                    <{>
                        <@lastName>
                            <=>Smith</=>
                        </@lastName>
                        <@age>
                            <= type="number">32</=>
                        </@age>
                        <@firstName>
                            <=>Mary</=>
                        </@firstName>
                    </}>
                </@2>
            </}>
        </@accounting>
    </}>
</χ:data-model>

The tags that surround atomic values are useful when these values are within a sequence but look superfluous when the item has a single value. The next step could be to define that in that case as a shortcut the value and its type attribute could be directly included in the item:

<χ:data-model>
    <{>
        <@sales>
            <{>
                <@1 keyType="number">
                    <{>
                        <@lastName>Green</@lastName>
                        <@age type="number">27</@age>
                        <@firstName>Sally</@firstName>
                    </}>
                </@1>
                <@2 keyType="number">
                    <{>
                        <@lastName>Galley</@lastName>
                        <@age type="number">41</@age>
                        <@firstName>Jim</@firstName>
                    </}>
                </@2>
            </}>
        </@sales>
        <@accounting>
            <{>
                <@1 keyType="number">
                    <{>
                        <@lastName>Doe</@lastName>
                        <@age type="number">23</@age>
                        <@firstName>John</@firstName>
                    </}>
                </@1>
                <@2 keyType="number">
                    <{>
                        <@lastName>Smith</@lastName>
                        <@age type="number">32</@age>
                        <@firstName>Mary</@firstName>
                    </}>
                </@2>
            </}>
        </@accounting>
    </}>
</χ:data-model>

XPath

The χίμαιραλ serialization being XML, it is possible to use XPath path expressions to query its structure. For instance, to get a list of employees which are less than 30, we can

write:

χ:map/χ:entry/χ:map/χ:entry/χ:map[χ:entry[@key='age'][χ:atomic-value < 30]]

Or, if we’re feeling lucky:

//χ:map[χ:entry[@key='age'][χ:atomic-value < 30]]

Again, that’s good as long we work on a χίμαιραλ serialization but it would be good to be able to use path expressions directly on map data structures. To do so we would need at minima to define steps to match maps and entries.

XSLT 3.0 introduces a new map() item type which could be used as a kind test to identify maps.

If we follow the idea that map entries are similar to XML attributes, we could use the @ notation to identify them. The XPath expression would then become:

map()/@*/map()/@*/map()[@age < 30]]

Or, if we’re feeling lucky:

//map()[@age < 30]]

Validation

These data models can be complex. Wouldn’t it be useful to be able to validate them with schema languages? This would give us a way to validate JSON maps!

Of course we can already serialize them in χίμαιραλ and validate the serialization using any schema language, but again it would be good to be able to validate these structures directly.

A RELAX NG schema to validate the χίμαιραλ serialization of our example would be:

namespace χ = "http://χίμαιραλ.com#"

start = element χ:data-model { top-level-map }

# Top level map: departments
top-level-map =
    element χ:map {
        element χ:entry {
            attribute key { xsd:NMTOKEN },
            attribute keyType { "string" },
            emp-array
        }*
    }

# List of employees
emp-array =
    element χ:map {
        element χ:entry {
            attribute key { xsd:positiveInteger },
            attribute keyType { "number" },
            emp-map
        }*
    }

# Description of an employee
emp-map = element χ:map { (age | firstName | lastName) + }

age =
    element χ:entry {
        attribute key { "age" },
        attribute keyType { "string" },
        element χ:atomic-value {
            attribute type { "number" },
            xsd:positiveInteger
        }
    }

firstName =
    element χ:entry {
        attribute key { "firstName" },
        attribute keyType { "string" },
        element χ:atomic-value {
            attribute type { "string" },
            xsd:token
        }
    }

lastName =
    element χ:entry {
        attribute key { "lastName" },
        attribute keyType { "string" },
        element χ:atomic-value {
            attribute type { "string" },
            xsd:token
        }
    }

Note

In the description of the maps used to describe employees, we cannot use interleave patterns because of the restriction on interleave and the schema is approximate. In this specific case, we could

enumerate the six possible combinations but the exercise would quickly become verbose if the number of items

grew:

emp-map = element χ:map { 
      (age, firstName, lastName)  
    | (age, lastName, firstName) 
    | (firstName, age, lastName)  
    | (firstName, lastName, age) 
    | (lastName, age, firstName)  
    | (lastName, firstName, age) 
}

A Schematron schema for the χίμαιραλ serialization could be developed based on XPath expressions similar to those that have been shown in the previous section.

Again, it would be interesting to support maps directly as first class citizens in XML schema languages.

The ability to use Schematron on XDM maps depends directly on the ability to browse maps using patterns and path expressions in XPath and XSLT (see above)…

The main impact on RELAX NG would be to add map and item patterns and the schema could look like:

namespace χ = "http://χίμαιραλ.com#"

start = element χ:data-model { top-level-map }

# Top level map: departments
top-level-map =
    map  {
        entry xsd:NMTOKEN {
            emp-array
        }*
    }

# List of employees
emp-array =
    map {
        entry xsd:positiveInteger {
            emp-map
        }*
    }

# Description of an employee
emp-map = map { age, firstName, lastName  }

age =
    entry age {
            xsd:positiveInteger
        }
    }

firstName =
    entry firstName {
             xsd:token
        }
    }

lastName =
    entry lastName {
            xsd:token
        }
    }

Sequences could probably be supported without adding a new pattern but would require to relax some restrictions to allow the description of sequences mixing atomic values, maps and nodes (in Relax NG, sequences of atomic values are already possible in list datatypes, sequences of nodes are of course available to describe node contents but these two type of sequences cannot be mixed).

Conclusion

According to the definition of chimeras in genetics from Wikipedia quoted in the introduction, [chimeras are formed from at least four parent cells (two fertilized eggs or early embryos fused together). Each population of cells keeps its own character and the resulting organism is a mixture of tissues].

The current XDM proposals have added to the XML data model a foreign model to represent maps. This new model is a superset of the JSON data model. The two data models keep their own character and the resulting model is a mixture of information items.

It’s far to say that the current XDM proposal is a chimera, something described as [usually ugly, foolish or impossible fantasies] by Jeni Tennison.

I hope that the proposals sketched in this paper will help to address this situation and fully integrate these new information items in the XML echosystem.

Test driven XML development

Can test driven development be applied to XML technologies? to XSLT? to XML schema languages? XML pipelines?

This is the topic of the poster I am presenting this year at the Balisage conference.

For those of you who are not lucky enough to attend, here is a copy of this poster: Test driven development in XML Balisage 2012 poster

Schema test driven development

Note: this article is derived from the presentation I have given at the International Symposium on Quality Assurance and Quality Control in XML (precedings).

Abstract

Ever modified an XML schema? Ever broken something while fixing a bug or adding a new feature? As withany piece of engineering, the more complex a schema is, the harder it is to maintain. In other domains, unit tests dramatically reduce the number of regressions and thus provide a kind of safety net for maintainers. Can we learn from these techniques and adapt them to XML schema languages? In this workshop session, we develop a schema using unit test techniques, to illustrate their benefits in this domain.

Table of Contents

Step 1: Getting started
Step 2: Adding a schema
Step 3: Adding list title elements
Step 4: Adding todo item elements
Step 5: Modularizing the schema
Want to try it?

The workshop is run as an exchange between a customer (played by Tommie Usdin) and a schema expert (played by Eric van der Vlist).

The customer, needs a schema for her to list XML application, is puzzled by the « test first programming » technique imposed by the schema expert.

At the end of the day (or workshop), will she be converted to this well known agile or extreme programming technique adapted to the development of XML schemas?

Step 1: Getting started

	Hi Eric, can you help me to write a schema?
	—Customer

	Hi Tommie, yes, sure, what will the schema be about?
	—Expert

	I need a vocabulary for my todo lists, with todo ite…
	—Customer

	OK, you’ve told me enough, let’s get started
	—Expert (interrupting his customer)

	Get started? but I haven’t told you anything about it!
	—Customer

	Right, but it’s never too soon to write tests when you do test first programming!
	—Expert

	Note
	Test first programming (also called test driven development) developers create test case (usually unit test cases) before implementing a function. The test suite is run, code is written based on the result of these tests and the test suite and code are updated untill all the tests pass.

Test suite

(suite.xml):

<tf:suite xmlns:tf="http://xmlschemata.org/test-first/" xmlns:todo="http://balisage.net/todo#" title="Basic tests">
    <tf:case title="Root element" expected="valid" id="root">
        <todo:list/>
    </tf:case>
</tf:suite>

	Note
	The vocabulary used to define these test cases has been inspired by the SUT (XML Schema Unit Test) project. It’s a simple vocabulary (composed of only three different element) allowing to pack several XML instances together with the outcome validation result. It uses conventions that you’ll discover during the course of this workshop.

Figure 1. Test results

Test results

	Note
	The test suite is run using a simple Orbeon Forms application. The rendering relies on Orbeon Forms XForms’ implementation while the test suite is run using an Orbeon Forms’ XPLpipeline.

Step 2: Adding a schema

	You see, you can already write todo lists!
	—Expert

	Hold on, we don’t have any schema!
	—Customer

	That’s true, but you don’t have to write a schema to write XML documents.
	—Expert

I know, but you’re here to write a schema! Furthermore right now we accept anything. I don’t want

to have XML documents with anything as a root element!

—Customer

That’s a good reason to write a schema but before that we need to add a test in our suite

first.

—Expert

Test suite

(suite.xml):

<?xml version="1.0" encoding="UTF-8"?>
<tf:suite xmlns:tf="http://xmlschemata.org/test-first/" xmlns:todo="http://balisage.net/todo#" title="Basic tests">
    <tf:case title="TODO list toot element" expected="valid" id="root">
        <todo:list/>
    </tf:case>
    <tf:case title="Other root element" expected="error" id="other-root">
        <todo:title>A title</todo:title>
    </tf:case>
</tf:suite>

	Now that we’ve updated the test suite, we run it again.
	—Expert

Figure 2. Test results

Test results

	This result was expected and we can now proceed to create a schema and attach it to the test suite.
	—Expert

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
    elementFormDefault="qualified"
    targetNamespace="http://balisage.net/todo#"
    xmlns="http://balisage.net/todo#">

    <xs:element
        name="list"/>
</xs:schema>

todo.xsd

<?xml version="1.0" encoding="UTF-8"?>
<tf:suite
    xmlns:tf="http://xmlschemata.org/test-first/"
    xmlns:todo="http://balisage.net/todo#"
    title="Basic tests">
    <tf:validation
        href="todo.xsd"
        type="xsd"/>
    <tf:case
        title="TODO list toot element"
        expected="valid"
        id="root">
        <todo:list/>
    </tf:case>
    <tf:case
        title="Other root element"
        expected="error"
        id="other-root">
        <todo:title>A title</todo:title>
    </tf:case>
</tf:suite>

suite.xml

	It’s time to test again what we’ve done.
	—Expert

Figure 3. Test results

Test results

Step 3: Adding list title elements

	I am happy to see some progress, at last, but I don’t want to accept any content in the todo list element. Can you add list title elements?
	—Customer

	Sure, back to the test suite…
	—Expert

Test suite

(suite.xml):

<?xml version="1.0" encoding="UTF-8"?>
<tf:suite
    xmlns:tf="http://xmlschemata.org/test-first/"
    xmlns:todo="http://balisage.net/todo#"
    title="Basic tests">
    <tf:validation
        href="todo.xsd"
        type="xsd"/>
    <tf:case
        title="TODO list root element"
        expected="valid"
        id="root">
        <todo:list/>
    </tf:case>
    <tf:case
        title="TODO list with a title"
        expected="valid"
        id="list-title">
        <todo:list>
            <todo:title/>
        </todo:list>
    </tf:case>
    <tf:case
        title="Other root element"
        expected="error"
        id="other-root">
        <todo:title>A title</todo:title>
    </tf:case>
</tf:suite>

	Now that we’ve updated the test suite, we run it again.
	—Expert

Figure 4. Test results

Test results

	You see? We do already support list title elements!
	—Expert

	Sure, but I don’t want to accept any content in my todo list. And the title element should be mandatory. And it should not be empty by have at least one character!
	—Customer

	Back to the test suite, then…
	—Expert

Test suite

(suite.xml):

<?xml version="1.0" encoding="UTF-8"?>
<tf:suite
    xmlns:tf="http://xmlschemata.org/test-first/"
    xmlns:todo="http://balisage.net/todo#"
    title="Basic tests">
    <tf:validation
        href="todo.xsd"
        type="xsd"/>
    <todo:list>
        <tf:case
            title="Empty list element"
            expected="error"
            id="root-empty"/> 
        <todo:title>
            <tf:case title="empty title" expected="error" id="empty-title"/>
            <tf:case title="non empty title" expected="valid" id="non-empty-title">A title</tf:case>
        </todo:title>
        <tf:case
            title="Un expected element"
            expected="error"
            id="unexpected">
            <todo:foo/>
        </tf:case>
    </todo:list>
    <tf:case
        title="Other root element"
        expected="error"
        id="other-root">
        <todo:title>A title</todo:title>
    </tf:case>
</tf:suite>

Note

This is the first example with non top level tf:case elements. To understand how this works, we must look in more detail to the algorithm used by the framework to split a test suite into instances. The algorithm consists in two steps:

Loop over each tf:case element
Suppression of the tf:caseelements and of the top level elements which arenot ancestors of the current tf:case element.

This description may look complex, but the result is a rather intuitive way to define sub-trees that are common to several test cases.

	Now that we’ve updated the test suite, we run it again.
	—Expert

Figure 5. Test results

Test results

	Sounds good, now we can update the schema.
	—Expert

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
    elementFormDefault="qualified"
    targetNamespace="http://balisage.net/todo#"
    xmlns="http://balisage.net/todo#">

    <xs:element
        name="list">
        <xs:complexType>
            <xs:sequence>
                <xs:element
                    name="title">
                    <xs:simpleType>
                        <xs:restriction
                            base="xs:token">
                            <xs:minLength
                                value="1"/>
                        </xs:restriction>
                    </xs:simpleType>
                </xs:element>
            </xs:sequence>
        </xs:complexType>
    </xs:element>
</xs:schema>

todo.xsd

	And run the test suite again.
	—Expert

Figure 6. Test results

Step 4: Adding todo item elements

	Good. Now I want to add todo items. And lists should have at least one of them, by the way.
	—Customer

	Sure, back to the test suite…
	—Expert

Test suite

(suite.xml):

<?xml version="1.0" encoding="UTF-8"?>
<tf:suite
    xmlns:tf="http://xmlschemata.org/test-first/"
    xmlns:todo="http://balisage.net/todo#"
    title="Basic tests">
    <tf:validation
        href="todo.xsd"
        type="xsd"/>
    <tf:case
        title="Empty list element"
        expected="error"
        id="root-empty">
        <todo:list/>
    </tf:case>
    <todo:list>
        <!-- Testing title elements -->
        <todo:title>
            <tf:case
                title="empty title"
                expected="error"
                id="empty-title"/>
            <tf:case
                title="non empty title"
                expected="valid"
                id="non-empty-title">A title</tf:case>
        </todo:title>
        <todo:item>
            <todo:title>A title</todo:title>
        </todo:item>
        <tf:case
            title="Un expected element"
            expected="error"
            id="unexpected">
            <todo:foo/>
        </tf:case>
    </todo:list>
    <todo:list>
        <!-- Testing todo items -->
        <todo:title>Testing todo items</todo:title>
        <tf:case
            title="No todo items"
            expected="error"
            id="no-items"/>
        <todo:item>
            <tf:case
                title="empty item"
                expected="error"
                id="empty-item"/>
            <tf:case
                title="item with a title"
                expected="valid"
                id="item-title">
                <todo:title>A title</todo:title>
            </tf:case>
        </todo:item>
    </todo:list>
    <tf:case
        title="Other root element"
        expected="error"
        id="other-root">
        <todo:title>A title</todo:title>
    </tf:case>
</tf:suite>

	Let’s see what we get before any update to the schema:
	—Expert

Figure 7. Test results

Test results

	It’s time to update the schema and fix what needs to be fixed:
	—Expert

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
    elementFormDefault="qualified"
    targetNamespace="http://balisage.net/todo#"
    xmlns="http://balisage.net/todo#">

    <xs:element
        name="list">
        <xs:complexType>
            <xs:sequence>
                <xs:element
                    name="title">
                    <xs:simpleType>
                        <xs:restriction
                            base="xs:token">
                            <xs:minLength
                                value="1"/>
                        </xs:restriction>
                    </xs:simpleType>
                </xs:element>
                <xs:element
                    maxOccurs="unbounded"
                    name="item">
                    <xs:complexType>
                        <xs:sequence>
                            <xs:element
                                name="title">
                                <xs:simpleType>
                                    <xs:restriction
                                        base="xs:token">
                                        <xs:minLength
                                            value="1"/>
                                    </xs:restriction>
                                </xs:simpleType>
                            </xs:element>
                        </xs:sequence>
                    </xs:complexType>
                </xs:element>
            </xs:sequence>
        </xs:complexType>
    </xs:element>
</xs:schema>

todo.xsd

	And now we can check if we get it right.
	—Expert

Figure 8. Test results

Test results

Step 5: Modularizing the schema

	Eric, you should be ashamed, it’s a pure Russian doll schema, not modular at all! Why not define the title and list elements globally?
	—Customer

	Sure, we can do that! If we just change the structure of the schema, we don’t need to update the test suite and can work directly on the schema:
	—Expert

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
    elementFormDefault="qualified"
    targetNamespace="http://balisage.net/todo#"
    xmlns="http://balisage.net/todo#">

    <xs:element
        name="list">
        <xs:complexType>
            <xs:sequence>
                <xs:element
                    ref="title"/>
                <xs:element
                    maxOccurs="unbounded"
                    ref="item"/>
            </xs:sequence>
        </xs:complexType>
    </xs:element>
    <xs:element
        name="title">
        <xs:simpleType>
            <xs:restriction
                base="xs:token">
                <xs:minLength
                    value="1"/>
            </xs:restriction>
        </xs:simpleType>
    </xs:element>
    <xs:element
        name="item">
        <xs:complexType>
            <xs:sequence>
                <xs:element
                    ref="title"/>
            </xs:sequence>
        </xs:complexType>
    </xs:element>
</xs:schema>

todo.xsd

	But of course, each time we update the schema we must check if we’ve not introduced any bug:
	—Expert

Figure 9. Test results

Test results

	Waoo, what’s happening now?
	—Customer

	Now that our elements are global in the schema, we accept a valid title as a root element. Is that what you want?
	—Expert

	No way, a title is not a valid list!
	—Customer

	Then we have a number of options… We can go back to local elements and we can also add a schematron schema to check this constraint.
	—Expert

	Schematron is fine, we’ll probably find many other constraints to check in my todo lists anyway…
	—Customer

	OK. We still don’t have to update the test suite since we’ve not changed our requirement. Let’s write this Schematron schema then:
	—Expert

<?xml version="1.0" encoding="UTF-8"?>
<schema xmlns="http://purl.oclc.org/dsdl/schematron" queryBinding="xslt2">
    <ns uri="http://balisage.net/todo#" prefix="todo"/>
    <pattern>
        <rule context="/*">
            <assert test="self::todo:list">The root element should be a todo:list</assert>
        </rule>
    </pattern>
</schema>

todo.sch

	Saying that we don’t have to update the test suite wasn’t totally accurate because the schemas are referenced in ths test suite:
	—Expert

Test suite

(suite.xml):

<?xml version="1.0" encoding="UTF-8"?>
<tf:suite
    xmlns:tf="http://xmlschemata.org/test-first/"
    xmlns:todo="http://balisage.net/todo#"
    title="Basic tests">
    <tf:validation
        href="todo.sch"
        type="sch"/>
    <tf:validation
        href="todo.xsd"
        type="xsd"/>
    <tf:case
        title="Empty list element"
        expected="error"
        id="root-empty">
        <todo:list/>
    </tf:case>
    <todo:list>
        <todo:title>
            <tf:case
                title="empty title"
                expected="error"
                id="empty-title"/>
            <tf:case
                title="non empty title"
                expected="valid"
                id="non-empty-title">A title</tf:case>
        </todo:title>
        <todo:item>
            <todo:title>A title</todo:title>
        </todo:item>
        <tf:case
            title="Un expected element"
            expected="error"
            id="unexpected">
            <todo:foo/>
        </tf:case>
    </todo:list>
    <todo:list>
        <todo:title>Testing todo items</todo:title>
        <tf:case
            title="No todo items"
            expected="error"
            id="no-items"/>
        <todo:item>
            <tf:case
                title="empty item"
                expected="error"
                id="empty-item"/>
            <tf:case
                title="item with a title"
                expected="valid"
                id="item-title">
                <todo:title>A title</todo:title>
            </tf:case>
        </todo:item>
    </todo:list>
    <tf:case
        title="Other root element"
        expected="error"
        id="other-root">
        <todo:title>A title</todo:title>
    </tf:case>
</tf:suite>

	Time to see if we’ve fixed our issue!
	—Expert

Figure 10. Test results

Test results

	Great, we’ve made it, thanks!
	—Customer

Want to try it?

The application used to run the test suite and display its result is available at http://svn.xmlschemata.org/repository/downloads/tefisc/.

If you just want to understand how the test suite is split into XML instances, you can have a look at http://svn.xmlschemata.org/repository/downloads/tefisc/orbeon-resources/apps/tefisc/ .

In this directory:

split-tests.xslis the XSLT transformation that splits a test suite into top levelelement test cases. This transformation has no dependence on Orbeon Forms and can be manuallyrun against a test suite.
run-test.xpl is the XPL pipeline that runs a test case.
list-suites.xpl is the XPL pipeline that gives the list avaible test cases.
view.xhtml is the XForms application that displays the results.

To install this application:

InstallOrbeon Forms
copy the orbeon-resources/ directory under /WEB-INF/resources/apps/in yourorbeon webapp directory
or, alternatively, copy the tefisc/ directory wherever you want, edit web.xml.savto replace<param-value>/home/vdv/projects/tefisc/orbeon-resources</param-value>by the location of this directory on your filesystem, replace /WEB-INF/web.xml by this file and restart your application server.

Using XSLTUnit with XSLT 2.0

The latest version of XSLTUnit (v0.2) is from January 2002 and people sometimes ask me if the project is dead and when XSLTUnit will support XSLT 2.0.

The short answer is that you can already use XSLTUnit with XSLT 2.0.

The project is not dead and if I haven’t published any new version it’s just because XSLTUnit meets my needs and nobody has ever asked me any update.

I use it a lot and, no later than this afternoon, came on a new opportunity to use it to add unit tests to a function that I needed to debug for the Owark project.

Following ideas to develop a web service to create page archives, I was writing an XSLT transformation that analyses Heritrix crawl logs to determine what needs to be packaged into the archives and one of the touchy functions is to create user friendly local names that remains unique within the scope of an archive.

My first naive attempt didn’t survive real world tests and results were rather disappointing.

Using these results as a test suite was an obvious idea…

To do so, I have used log.xml, a real crawl log converted to XML as my source document and local-names.xml, the result of the transformation, as a reference.

The XSLTUnit transformation that exploit these documents, local-names.xsl, is very simple and is a nice example of what you can do with this framework and how you can use it with XSLT 2.0.

An XSLTUnit test suite is an XSLT transformation that imports both the transformation to test and xsltunit.xsl:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:exsl="http://exslt.org/common" extension-element-prefixes="exsl" xmlns:xsltu="http://xsltunit.org/0/"
  xmlns:owk="http://owark.org/xslt/" exclude-result-prefixes="exsl">
  <xsl:import href="../actions/resource-index.xslt"/>
  <xsl:import href="xsltunit.xsl"/>

.../...

</xsl:stylesheet>

Here, the test suite and the XSLT transformation to test are both XSLT 2.0 and with Saxon they just work fine with xsltunit.xsl which is XSLT 1.0.

A XSLTUnit test case is composed of comparisons, and a test case to test that the value of the owk:local-name() function on a specific log entry is equal to « index.xml » could be:

      <xsltu:test id="index">
        <xsl:call-template name="xsltu:assertEqual">
          <xsl:with-param name="id" select="'index'"/>
          <xsl:with-param name="nodes1">
            <name>
              <xsl:value-of select="owk:local-name(/log/entry[uri='https://blog.eric.van-der-vlist.com/'])"/>
            </name>
          </xsl:with-param>
          <xsl:with-param name="nodes2">
            <name>index.html</name>
          </xsl:with-param>
        </xsl:call-template>
      </xsltu:test>

This crawl log includes 292 resources and you wouldn’t want to copy and past 291 times this snippet… No problem, you can just use some XSLT power and write:

 <?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:exsl="http://exslt.org/common" extension-element-prefixes="exsl" xmlns:xsltu="http://xsltunit.org/0/"
  xmlns:owk="http://owark.org/xslt/" exclude-result-prefixes="exsl">
  <xsl:import href="../actions/resource-index.xslt"/>
  <xsl:import href="xsltunit.xsl"/>
  <xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes"/>
  <xsl:variable name="local-names" select="doc('local-names.xml')/index"/>
  <xsl:key name="log-by-uri" match="/log/entry" use="uri"/>
  <xsl:template match="/">
    <xsltu:tests>
      <xsl:for-each select="$local-names/resource">
        <xsltu:test id="{uri}">
          <xsl:call-template name="xsltu:assertEqual">
            <xsl:with-param name="id" select="uri"/>
            <xsl:with-param name="nodes1">
              <local-name>
                <xsl:value-of select="owk:unique-local-name(key('log-by-uri', current()/uri, $source ))"/>
              </local-name>
            </xsl:with-param>
            <xsl:with-param name="nodes2">
              <xsl:copy-of select="local-name"/>
            </xsl:with-param>
          </xsl:call-template>
        </xsltu:test>
      </xsl:for-each>
    </xsltu:tests>
  </xsl:template>
</xsl:stylesheet>

When you’ve done so, you can edit « local-names.xml » that contains the expected values so that they look more like what you want to, run the test suite to detect the differences, update your transformation accordingly and iterate.

More musings on XDM 3.0

Note: these are musings and should not be taken as a concrete proposal!

Balisage’s tag line is « there is nothing so practical as a good theory ». I love Balisage but my brain doesn’t really work like that and its tag line would rather be « there is nothing as theoretical as a good exercise ».

χίμαιραλ (chimeral) is the kind of exercise I needed to get into XDM 3.0 (and probably even 2.0 that I had been using without really thinking about it) and I’d like to share the musings that this exercise has inspired.

For XDM, the most generic piece of information is called an « item ».

XDM 2.0 has introduced a distinction between two kind of items:

Nodes which come directly from the XML infoset and borrow some properties to the PSVI.
Atomic values that are « pure » values living their own lives independently of their occurrences in XML documents as text or attribute node values.

Coming straight from the XML infoset, nodes are rather concrete, and you can still smell the (electronic) ink of their tags… Each node is unique and lives in its own context: when you write:

<?xml version="1.0" encoding="UTF-8"?>
<root>
  <foo>5</foo>
  <foo>5</foo>
</root>

The two <foo/> elements are two different nodes. They may look identical and have the same content, they are still two different nodes like identical twins are different persons.

Their value, by contrast, is the same value: a 5 is a 5, the fact that it’s the value of the first or of the second <foo/> (or of a @bar attribute) doesn’t make any difference.

The fact that values are shared between the places where they are used is very common among programming languages. In Fortran IV, the first programming I have ever used, these values were not well write protected (you could manage to assign the value 6 to 5), leading to dreadful bugs and I remember that I had taken the challenge to write a program that was using values as variables!

XDM 2.0 also introduced the notion of « sequences ». Sequences are really special. They are not considered as a model item and are just a kind of invisible bag to package items together and are useful because you can store them in variables and use them as parameters.

Sequences have some interesting magics…

They disappear by themselves when they pack only one item and their is no way to differentiate a sequence of one item from the item itself. XDM has invented a perfectly biodegradable bag!

They also disappear when you put them within another sequence and you can’t have sequences of sequences. You’d better think twice if you want to separate apple from oranges (or meat and cheese) before packaging them in a sequence!

You may have heard that XPath/XSLT 2.0 had abolished the difference between node sets and result tree fragments. That’s true but not always as one would have expected!

Assuming that your input document is the one mentioned above, the equivalent of an XSLT 1.0 node set could be:

<xsl:variable name="nodeset" select="/root/foo"/>

And the equivalent of an XSLT 1.0 result tree fragment would be:

<xsl:variable name="rtf">
  <xsl:copy-of select="/root/foo"/>
</xsl:variable>

Of course, the nodes in $rtf are copies of the original nodes and $nodeset = $rtf will return false().

But why does deep-equal($nodeset,$rtf) also return false()? And why does count($nodeset) return 2 when count($rtf) returns 1?

Forget that you’ve been told that there was no longer any difference between node sets and result tree fragments, these two variables are two different beasts…

$nodeset is a sequence composed of the two elements <foo/> from the input document (the elements themselves, not their copies) while $rtf is a document node with two <foo/> child elements copied from the input document.

Both are sequences of nodes, but that’s all they have in common!

XDM 3.0 adds a third type of items: functions that become first class objects transforming XPath/XQuery/XSLT into functional languages.

Michael Kay’s proposal adds as a fourth item type: maps (and arrays considered as maps with integer keys). Maps being a brand new type of item, the choice is open: they could have been considered similar to XML nodes but the current proposal is to consider them as pure values.

In my previous post I have explained the consequences of this important design decision and I’d like to take a step backward and analyze the differences between values and nodes.

Still before discovering Fortran IV, I did study mathematics and geometry and I was fascinated by the dual approach for solving problem in geometry using either Euclidean vectors or points and segments.

In the current proposal, a map is like a vector: you can reuse it as many time as you like in other map entries and it will remain unique. On the contrary, a node is like a segment. It has an identity, is bound to a parent and if you want to reuse it you need to copy and paste it.

In geometry, vectors and segments are both useful and complementary. I can understand that there is a need for both nodes and values, but why should maps always be values and nodes always be « concrete » nodes? Both are useful data structures, why should they be treated so differently?

I understand that there are (at least) to use cases for maps: to support JSON and to provide lightweight data structures. These two use cases look very different to me. Wanting to meet them both with a single feature, won’t we miss both points? Why should lightweight structures be limited to maps and why should maps always been lightweight?

In geometry the processes by which you create a vector from a segment or a segment from a vector by pinning one of its extremities are well known. Can’t we define similar processes to transform a « concrete » data structure (either node tree or map) into a pure value and vice versa?

This is not so uncommon for programmers: values can also be seen as classes (with class properties) and concrete structures as instantiations of these classes.

Now, what about chimeras mixing pure values and concrete nodes?

A handy feature of sequences and maps as currently proposed is that they can include nodes. This « inclusion » is done « by reference » and the nodes keep they identity and context. Maps using this feature are « pure values » with references to concrete nodes.

When a map entry is a node, the node keeps its context within a document and its parent remains its parent in this document. How can you do that if you want to also represent the reverse relation between the node and the map entry in which it has been added?

One option could be to define a mechanism similar to symbolic links on Linux: the link is a kind of shortcut and when you follow it you can’t always guarantee that you’ll be able to come back in the same directory. Adapted to the XDM, we wouldn’t store the parent/child relation between map entries and nodes. This would be a limitation, but we wouldn’t meet this limitation when dealing with maps de-serialized from JSON objects (JSON objects do not contain XML nodes).

Another option could be to « decorate » or « re-instantiate » the node so that the new instance has two parents, probably not with the same kind of parent/child relation but with a new kind that could be followed with a different axis. This decoration would add new context information to a tree and would be very similar to the process by which values are turned into concrete nodes. Now, would that be a practical thing to do? What about adding a map with a node as a new entry on another map? Don’t we end up with nodes that can contain an indefinite number of ancestors?

I hope that these musings can be helpful, but I should probably stick to my role of XDM user rather than giving suggestions!

To summarize: I think that we need both lightweight map structures and the full set of XPath axis on maps de-serialized from JSON objects.

Having only lightweight map structures means that users (and probably other specs and technologies) will have to continue to define custom mappings between JSON and XML to perform serious work.

XDM Maps should be first class citizens

Note: This issue has been submitted to the W3C as #16118.

The XPath/XQuery/XSLT 3.0 Data model distinguishes three types of information items:

Nodes that directly relate to the XML Infoset with some information borrowed from the PSVI.
Functions
Atomic types.

Michael Kay has recently proposed to add maps as a fourth item type derived from functions.

The main motivation for this addition is to support JSON objects that can be considered as a subset of maps items.

However, in the current proposal map items are treated very differently from XML nodes and this has deep practical consequences.

Take for instance the following simple JSON sample borrowed from Wikipedia:

{
     "firstName": "John",
     "lastName" : "Smith",
     "age"      : 25,
     "address"  :
     {
         "streetAddress": "21 2nd Street",
         "city"         : "New York",
         "state"        : "NY",
         "postalCode"   : "10021"
     },
     "phoneNumber":
     [
         {
           "type"  : "home",
           "number": "212 555-1234"
         },
         {
           "type"  : "fax",
           "number": "646 555-4567"
         }
     ]
 }

To get the postalCode from an equivalent structure expressed as XML and stored in the variable $person, one would just use the following XPath expression: $person/address/postalCode.

When the same structure is expressed in JSON and parsed into an XDM map, XPath axes can no longer be used (their purpose is to traverse documents, ie nodes) and we need to use map functions: map:get(map:get($person, 'address'), 'postcalCode').

That’s not as bad as it sounds because maps can be invoked as functions and this can be rewritten as $person('address')('postalCode') but this gives a first idea of the deep differences between maps and nodes and things would become worse if I wanted to get the postal code of persons whose first name are « John »…

Another important difference is that node items are the only ones that have a context or an identity.

When I write <foo><bar>5</bar></foo><bat></bar>5</bar></bat> each of the two bar elements happen to have the same names and values but they are considered as two different elements and even the two text nodes that are their children are two different text nodes.

When I write foo: {bar: 5}, bat: {bar: 5} the two bar entries are actually the same thing and can’t be distinguished.

This difference is important because that means that XPath axes as we know them for nodes could never be implemented on maps: if an entry in a map can’t be distinguished from en identical entry else where in another map there is no hope to be able to determine its parent for instance.

Now, why is it important to be able to define axes on maps and map entries?

I think that this is important for XSLT and XQuery users to be able to traverse maps like they traverse XML fragments (with the same level of flexibility and syntaxes that are kept as close as possible). And yes, that means being able to apply templates over maps and be able to update maps using XQuery update…

But I also think that this will be important to other technologies that rely on XPath such as (to name those I know best) XForms, pipeline languages (XPL, XPROC, …) and Schematron.

Being able to use XForms to edit JSON object is an obvious need that XForms 2.0 is trying to address through a « hack » that has been presented at XML Prague 2012.

In a longer term we can hope that XForms will abandon this hack to rely on XDM maps XForms relies a lot on the notions of nodes and axes. XForms binds controls to instance nodes and the semantics of such bindings would be quite different to be applied to XDM map entries as currently proposed.

XML pipeline languages are also good candidates to support JSON objects. Both XPL and XProc have features to loop over document fragments and choose actions depending on the results of XPath expressions and again the semantics of these features would be affected if they had to support XDM maps as currently proposed.

Schematron could be a nice answer to the issue of validating JSON objects. Schematron relies on XPath at two different levels: its rules are defined as XPath expressions and it is often very convenient to be able to use XPath axes such as ancestor and its processing model is defined in term of traversing a tree. Again, an update of Schematron to support maps would be more difficult is maps are not similar to XML nodes.

Given the place of JSON on the web, I think that it is really important to support maps and the question we have to face is: « do we want a minimal level of support that may require hard work from developers and other standards to support JSON or do we want to make it as painless as possible for them? ».

And obviously, my preference is the later: if we add maps to the XDM, we need to give them full citizenship from the beginning!

Note: The fact that map entries are unordered (and they need to be because the properties of JSON objects are unordered) is less an issue to me. We already have two node types (namespaces nodes and attributes) which relative order are « stable but implementation-dependent ».

Introducing χίμαιραλ (chimeral), the Chimera Language

In the presentation I gave at XML Prague 2012 (see my paper), one of my conclusions was that the XML data model extended by the XPath/XQuery/XSLT Working Group to embrace other data models such as JSON was an important foundation of the whole XML ecosystem.

In her amazing keynote, Jeni Tennison warned us against chimeras, “ugly, foolish or impossible fantasies” and I have thought that it would be useful to check to which extent the XPath/XQuery/XSLT 3.0 data model (aka XDM 3.0) deserves to be called a chimera.

The foundation of this data model is the XML infoset, but it also borrows informations items from the Post Schema Validation Infoset (the [in]famous PSVI) and adds its own abstract items such as sequences and, new in 3.0, functions and maps (needed to represent JSON objects).

I started to think more seriously about this, doing some researches and writing a proposal for Balisage and my plan was to wait until the conference to publish anything.

One of the things I planed to present is a simple XML serialization format for the XDM. My initial motivation to propose such a format was to have a visualization of the XDM: I find it difficult to represent it if its instances stay purely abstract and can’t be serialized and deserialized.

Working on this, I have soon discovered that this serialization can have other concrete benefits: the items that have been recently added to the XDM such as maps and even sequences are not treated as first class citizens by XPath/XQuery/XSLT and the data model can be easier to traverse using its serialization!

When for instance you have a complex map imported from JSON by the brand new parse-json() function, you can’t easily apply the templates on the map items and sub items. And of course, with a XML serialization that becomes trivial to do.

If such a serialization can be useful, there is no reason to wait until Balisage in August to discuss it and I’d like to introduce the very first version of χίμαιραλ (chimeral), the Chimera Language.

The URL itself http://χίμαιραλ.com is a chimera composed of letters from two different alphabets and merging concepts from two different civilizations!

This first version is not complete. It already supports rather complex cases, but I need to think more how to deal with maps or sequences of nodes such as namespace nodes or attributes.

So far I am really impressed by XPath 3.0 but also surprised by many limitations in term of reflexion:

No built in function to determine the basic type of an item (node, attribute, sequence, map, function, …).
The dm:node-kind()accessor to determine the type of a node is abstract and XPath 3.0 does not expose it.
The behavior of the exslt:object-type() function is surprising.

I may have missed something, but in practice I have found quite difficult when you have a variable to browse its data model.

The other aspect that I don’t like in XPath/XQuery/XSLT 3.0 is the lack of homogeneity between the way the different types of items are manipulated. This strengthen the feeling that we have a real chimera!

In XSLT for instance, I’d like to be able to apply templates and match items in the same way for any item types. Unfortunately, the features that are needed to do so (node tests, axis, …) are reserved to XML nodes. I can’t define a template that matches a map (nor a sequence by the way), I can’t apply templates over map items, …

It may be too late for version 3.0, but I really think that we should incorporate these recent additions to make them first class citizens!

Going forward, we could reconsider the way these items mix and match. Currently you can have sequences of maps, functions, nodes and atomic values, maps which values are sequences, functions, nodes and atomic values but nodes are only composed of other nodes. Even if the XML syntax doesn’t support this, I would really like to see more symmetry and be able to add sequences and maps within nodes!

In other words, I think that it would be much more coherent to treat maps and sequences like nodes…

Note: The χίμαιραλ website is currently « read only » but comments are very welcome on this blog of by mail.

XML Prague 2012: The web would be so cool without the web developers

Note: XML Prague is also a very interesting pre-conference day, a traditional dinner, posters, sponsors announcements, meals, coffee breaks, discussions and walks that I have not covered in article for lack of time.

When I was a child, I used to say that I was feeling Dutch when I was in France and French when I was in the Netherlands. That was nice to feel slightly different and I liked to analyze the differences between Dutch people who seemed to be more adult and civilized and French people who seemed to me more spontaneous and fierce.

I have found back this old feeling of being torn between two different culture very strongly this week end at XML Prague. Of course, that was no longer between French and Dutch but between the XML and Web communities.

The conference also reminded me the old joke of the Parisian visiting Corsica and saying « Corsica would be so cool without Corsicans! » and for me the tag line could have been « the web would be so cool without web developers! ».

Jeni Tennison’s amazing opening keynote was of course more subtle than that!

She started by acknowledging that the web was split into no less than four different major formats: HTML, JSON, XML and RDF.

Her presentation has been a set of clever considerations over how we can deal with these different formats and cultures, concluding that we should accept the fact than « the web is varied, complex, dynamic, beautiful ».

I was then giving my talk « XML, the eX Markup Language » (read also my paper of this blog) where I have analyzed the reasons of the failure of XML to become the one major web format and given my view on where XML should be heading.

While Jeni had explained why « chimera are usually ugly, foolish or impossible fantasies », my conclusion has been that we should focus on the data model and extend or bridge it to embrace JSON and HTML like the XPath 3.0 data model is proposing to do.

I am still thinking so, but what is such a data model if not a chimera? Is it ugly, foolish or impossible then? There is a lot to think about beyond what Hans-Jürgen Rennau and David Lee have proposed at Balisage 2011 and I think I’ll submit a proposal at Balisage 2012 on this topic!

Robin Berjon and Norman Walsh tried then to bridge the gap with their presentation « XML and HTML Cross-Pollination: A Bridge Too Far?« , an interesting talk where they’ve tried to show how interesting ideas could be shared between these two communities: « forget about angle brackets, look at the ideas ». This gave a nice list of things that do work in the browser (did you know that you could run JavaScript against any XML document?) and fascinating new ideas such as JS-SLT, a JavaScript transformation library or CSS-Schema, a CSS based schema assertion language.

Anne van Kesteren had chosen a provocative title for his talk: « What XML can learn from HTML; also known as XML5« . Working for Opera, Anne was probably the only real representative of the web community at this conference. Under that title, his presentation was an advocacy for releasing the strictness of the XML parsing rules and defining an error recovery mechanism in XML as they exist in HTML5.

His talk was followed by a panel discussion on HTML/XML convergence and this subject of error recovery has monopolized the full panel! Some of the panelists (Anne van Kesteren, Robin Berjon and myself) were less hostile but the audience did unanimously reject the idea to change anything in the well-formedness rules of the XML recommendation.

Speaking of errors may be part of the problem: errors have a bad connotation and if a syntactical construct is allowed by the error recovery mechanism with a well defined meaning, why should we still consider it an error?

However, a consensus was found to admit that it could be useful to specify an error recovery mechanism that could be used when applications need to read non well formed XML documents that may be found on the wide web. This consensus has led to the creation of the W3C XML Error Recovery Community Group.

The reaction if the room that didn’t accept to even consider a discussion on what XML well-formedness means seems rather irrational to me. Michael Sperberg-McQueen reinforced this feeling in his closing keynote when he pleaded to define this as « a separate add-on rule rather than as a spec that changes the fundamental rules of XML ».

What can be so fundamental with the definition of XML well-formedness? These reactions made me feel like we were discussing kashrut rules rather than parsing rules and the debate often looked more religious than technical!

The next talk, XProc: Beyond application/xml by Vojtěch Toman was again about bridging technologies but was less controversial, probably because the technologies to bridge with were not seen as XML competitors.

Taking a look at the workarounds used by XML pipelines to support non XML data (either encoding the data or storing it out of the pipeline), Vojtěch proposed to extend the data model flowing in the pipelines to directly support non XML content. That kind of proposal looks so obvious and simple that you wonder why it hasn’t been done before!

George Bina came next to present Understanding NVDL – the Anatomy of an Open Source XProc/XSLT implementation of NVDL. NVDL is a cool technology to bridge different schema languages and greatly facilitates the validation of compound XML documents.

Next was Jonathan Robie, presenting JSONiq: XQuery for JSON, JSON for XQuery. JSONiq is both a syntax and a set of extensions to query JSON documents in an XQuery flavor that looks like JSON. Both the syntax and the extensions look both elegant and clever.

The room was usually very quiet during the talks, waiting for the QA sessions at the end of the talks to ask questions or give comments, but as soon as Jonathan displayed the first example, Anne van Kesteren couldn’t help gasping: « what? arrays are not zero based! »

Having put JSON clothes on top of an XPath data model, JSONiq has a base index equal to one for its arrays while JavaScript and most programming languages use zero for their base indexes.

Proposing zero based arrays inside a JSONic syntax to web developers is like wearing a kippah to visit an orthodox Jew and bring him baked ham: if you want to be kosher you need to be fully kosher!

Norman Walsh came back on stage to present Corona: Managing and querying XML and JSON via REST, a project to « expose the core MarkLogic functionality—the important things developers need— as a set of services callable from other languages » in an format agnostic way (XML and JSON can be used interchangeably).

The last talk of this first day was given by Steven Pemberton, Treating JSON as a subset of XML: Using XForms to read and submit JSON. After a short introduction to XForms, Steven explained how the W3C XForms Working Group is considering supporting JSON in XForms 2.0.

While Steven was speaking, Michael Kay twitted what many of us were thinking: « Oh dear, yet another JSON-to-XML mapping coming…« . Unfortunately, until JSON finds its way into the XML Data Model, every application that wants to expose JSON to XML tool has to propose a mapping!

The first sessions of the second day were spent by Jonathan Robie and Michael Kay to present What’s New in XPath/XSLT/XQuery 3.0 and XML Schema 1.1.

A lot of good things indeed! XML Schema 1.1 in particular that will correct the biggest limitations of XML Schema 1.0 and borrow some features to Schematron, making XML Schema an almost decent schema language!

But the biggest news are for XPath/XSLT/XQuery 3.0, bringing impressive new features that will turn these languages into fully functional programming languages. And of course new types in the data model to support the JSON data model.

One of these new features are annotations and Adam Retter gave a good illustration of how these annotations can be used in his talk RESTful XQuery – Standardised XQuery 3.0 Annotations for REST. XQuery being used to power web applications, these annotations can be used to define how stored queries are associated to HTTP requests and Adam proposes to standardize them to insure interoperability between implementations.

For those of use whose head was not spinning yet, Alain Couthures came to explain how he is Compiling XQuery code into Javascript instructions using XSLT 1.0 for his XSLTForms implementation. If we can use XSLT 1.0 to compile XQuery into JavaScript, what are the next steps? XSLT 2.0?

After the lunch, Evan Lenz came to present Carrot, « an appetizing hybrid of XQuery and XSLT » which was first presented at Balisage 2011. This hybrid is not a chimera but a nice compromise for those of us who can’t really decide if they prefer XSLT or XQuery: Carrot extends the non XML syntax of XQuery to expose the templating system of XSLT.

It can be seen as yet another non XML syntax for XSLT, a templating extension for XQuery and borrows their best features to both languages!

Speaking of defining templates in XQuery, John Snelson came next to present Transform.XQ: A Transformation Library for XQuery 3.0. Taking profit of the functional programming features of XQuery 3.0, Transform.XQ is an XQuery library that implements templates in XQuery. These templates are not exactly similar to XSLT templates (the priority system is different) but like in XSLT you’ll find template definitions,apply templates methods, modes, priorities and other goodies.

Java had not been mentioned yet and Charles Foster came to propose Building Bridges from Java to XQuery. Based on XQuery API for Java (XQJ), this bridges rely on Java annotations to map Java classes and XQuery stored queries and of course POJOs are also mapped to XML to provide a very sleek integration.

The last talk was a use case by Lorenzo Bossi presenting A Wiki-based System for Schema and Data Evolution, providing a good summary of the kind of problem you have when you need to update schemas and corpus’s of documents.

Everyone was then holding their breath waiting for Michael Sperberg-McQueen’s closing keynote that has been brilliant as usual and almost impossible to summarize and should be watched on video!

Michael choose to use John Amos Comenius as an introduction for his keynote. Comenius has been the last bishop of Unity of the Brethren and became a religious refugee. That gave Michael an opportunity to call for tolerance and diversity in document formats like in real life. Comenius is also one of the earliest champions of universal education and Michael pointed out that structured markup languages were the new champions of this noble goal in his final conclusion.

Of course, there has been much more than that in his keynote, Michael taking care to mention each presentation, but this focus on Comenius confirmed my feeling of the religious feeling toward XML.

I agree with most what Michael said in his keynote except maybe when he seems to deny that the XML adoption can be considered disappointing. When he says that the original goal of XML to be able to use SGML on the web has been achieved because he, Michael Sperberg-McQueen, can use XML on his web sites, that’s true of course, but was the goal really to allow SGML experts to use SGML on the web?

It’s difficult for me to dissent because he is the one who was involved in XML at that time when I had never heard of SGML, but I would still argue that SGML was usable on the web by SGML experts and that I don’t understand the motivation of the simplification that gave birth to XML if that was not to lower the price to entry so that web developers could use XML.

The consequences of this simplifications have been very heavy: the whole stack of XML technologies had to be reinvented and SGML experts have lost a lot of time before these technologies could be considered to be at the same level as they were. And even now some features of SGML that have been stripped down could be very useful for experts on the web such as for instance DTDs powerful enough to describe wiki syntaxes.

Similarly, when discussing during lunch with Liam Quin about my talk, he said that he had always thought that XHTML would never replace HTML. I have no reason to contradict Liam, but the vision of the W3C Markup Activity was clearly to « Deliver the Web of the Future Today: Recasting HTML in XML » like it can be seen on this archive.

It’s not pleasant to admit that we’ve failed, but replacing HTML with XHTML so that XML became dominant on the browser was clearly the official vision of the W3C shared by a lot of us and this vision has failed!

We need to acknowledge that we’ve lost this battle and make peace with the web developers that have won…

Curiously, there seems to be much less aggressiveness toward JSON than toward HTML5 in the XML community as can be shown by the number of efforts to bridge XML and JSON. Can we explain this by the fact that many XML purists considered data oriented XML as less interesting and noble than document oriented XML?

Anyway, the key point is that very strong ecosystem has been created with an innovative, motivated and almost religious community and a technology stack which is both modern and mature.

XML, the eX Markup Language?

Note: this article is a copy of the paper that I have presented at XML Prague 2012.

Abstract

Revisiting the question that was the tag line of XML Prague last year: « XML as new lingua franca for the Web. Why did it never happen? », Eric tries to answer to other questions such as: « where is XML going? » or « is XML declining, becoming an eX Markup Language? ».

XML as new lingua franca for the Web. Why did it never happen?

This was the tagline of XML Prague 2011, but the question hasn’t really been answered last year and I’ll start this talk to give my view on that question.

Flashback

February 1998 is a looong time ago, a date from another century and for those of you who were not born or don’t remember, here is a small summary of what did happen in February 1998:

	February The United States Senate passes Resolution 71, urging U.S. President Bill Clinton to « take all necessary and appropriate actions to respond to the threat posed by Iraq‘s refusal to end its weapons of mass destruction programs. » February 3 – Cavalese cable car disaster: a United States Military pilot causes the deaths of 20 people near Trento, Italy, when his low-flying plane severs the cable of a cable-car. February 4 – An earthquake measuring 6.1 on the Richter scale in northeast Afghanistan kills more than 5,000 people. February 7–February 22 – The 1998 Winter Olympics are held in Nagano, Japan. February 16 – China Airlines Flight 676 crashes into a residential area near Chiang Kai-shek International Airport, killing 202 people (all 196 on board and 6 on the ground). February 20 – Iraq disarmament crisis: Iraqi President Saddam Hussein negotiates a deal with U.N. Secretary General Kofi Annan, allowing weapons inspectors to return to Baghdad, preventing military action by the United States and Britain.
	—Wikipedia

While the Iraq disarmament crisis was raging, the World Wide Web Consortium waited until the third day of the Winter Olympics held in Nagano to make the following announcement:

	Advancing its mission to lead the Web to its full potential, the World Wide Web Consortium (W3C) today announced the release of the XML 1.0 specification as a W3C Recommendation. XML 1.0 is the W3C’s first Recommendation for the Extensible Markup Language, a system for defining, validating, and sharing document formats on the Web
	—W3C Press Release (February 1998)

People curious enough to click on the second link of the announcement could easily double check that beyond the marketing bias XML was something to be used over the Internet:

	The design goals for XML are: XML shall be straightforwardly usable over the Internet. XML shall support a wide variety of applications. XML shall be compatible with SGML. It shall be easy to write programs which process XML documents. The number of optional features in XML is to be kept to the absolute minimum, ideally zero. XML documents should be human-legible and reasonably clear. The XML design should be prepared quickly. The design of XML shall be formal and concise. XML documents shall be easy to create. Terseness in XML markup is of minimal importance.
	—W3C Recommendation (February 1998)

And the point was reinforced by the man who had led the « Web SGML » initiative and is often referred to as the father of XML:

	XML arose from the recognition that key components of the original web infrastructure — HTML tagging, simple hypertext linking, and hardcoded presentation — would not scale up to meet the future needs of the web. This awareness started with people like me who were involved in industrial-strength electronic publishing before the web came into existence.
	—Jon Bosak

This has often been summarized saying that XML is about « putting SGML on the Web ».

Among the design goals the second one (« XML shall support a wide variety of applications ») has been especially successful and by the end of 98, Liora Alschuler reported that the motivations of the different players pushing XML forward were very diverse:

	The big-gun database vendors, IBM and Oracle, see XML as a pathway into and out of their data management tools. The big-gun browser vendors, Netscape and Microsoft, see XML as the e-commerce everywhere technology. The big-gun book and document publishers, for all media, are seeing a new influx of tools, integrators, and interest but the direction XML publishing will take is less well-defined and more contingent on linking and style specs still in the hands of the W3C.
	—Liora Alschuler for XML.com (December 1998)

One thing these « big-gun » players that were pushing XML to different directions did achieve has been to develop an incredible hype that rapidly covered everything and in 2001 the situation had become hardly bearable:

	Stop the XML hype, I want to get offAs editor of XML.com, I welcome the massive success XML has had. But things prized by the XML community — openness and interoperability — are getting swallowed up in a blaze of marketing hype. Is this the price of success, or something we can avoid?
	—Edd Dumbill (March 2001)

Marketers behind the hype being who they were, the image of XML that they promoted was so shiny that the XML gurus didn’t recognize their own technology and tried to fight against the hype:

	I’ve spent years learning XML / I like XML / This is why www.XmlSuck.com is here
	—PaulT (January 2001)

The attraction was high and people rushed to participate to the W3C working groups:

	Working Group size – so many people means it is difficult to gain consensus, or even know everyone’s face. Conference calls are difficult.
	—Mark Nottingham, about the SOAP W3C WG (May 2000)

Huge working groups with people pushing to different directions is not the best recipe to publish high quality standards and even though XML itself was already baked, the perception of XML depends on the full « stack »:

	This is a huge responsibility for the Schema Working Group since it means that the defects of W3C XML Schema will be perceived by most as defects of XML.
	—Eric van der Vlist on xml-dev (April 2001)

The hype was so huge that XML geeks rapidly thought that they had won the war and that XML was everywhere:

	XML is now as important for the Web as HTML was to the foundation of the Web. XML is everywhere.
	—connet.us (February 2001)

Why this hype? My guess is that the IT industry had such a desperate need for a data interchange format that any one of them could have been adopted at that time and that XML happened to be the one that went through the radar screen at the right moment:

	When the wind is strong enough, even flatirons can fly.
	—Anonymous (February 2012)

The W3C had now to maintain:

XML, a SGML subset
HTML, a SGML application that did not match the XML subset

Technically speaking, the thing to do was to refactor HTML to meet the XML requirements. Given the perceived success of XML, it seemed obvious that everyone would jump into the XML wagon and be eager to adopt XHTML.

Unfortunately from a web developer perspective the benefits of XHTML 1.0 were not that obvious:

	The problem with XHTML is :a) it’s different enough from HTML to create new compatibility problems.b) it’s not different enough from HTML to bring significant advantages.
	—Eric van der Vlist on XHTML-DEV (May 2000)

It is fair to say that Microsoft had been promoting XML since the beginning:

	XML, XML, EverywhereThere’s no avoiding XML in the .NET world. XML isn’t just used in Web applications, it’s at the heart of the way data is stored, manipulated, and exchanged in .NET systems.
	— Rob Macdonald for MSDN (February 2001)

However, despite their strong commitment to XML, Microsoft had frozen new developments on Internet Explorer. The browser has never been updated to support the XHTML media type, meaning that the few web sites using XHTML had to serve their pages as HTML!

By 2001, the landscape was set:

XML had become a dominant buzzword giving a false impression that it had been widely adopted
Under the hood, many developers were deeply upset by this hype even among the XML community
Serving XHTML web pages as such was not an option for most web sites

The landscape was set, but the hype was still high and XML was still gaining traction as a data interchange format.

In the meantime, another hype was growing…

Wikipedia has tracked the origin of the term Web 2.0 back to 1999:

The Web we know now, which loads into a browser window in essentially static screenfuls, is only an embryo of the Web to come…./…Ironically, the defining trait of Web 2.0 will be that it won’t have ant visible characteristics at all. The Web will be identified only by its underlying DNA structure– TCP/IP (the protocol that controls how files are transported across the Internet); HTTP (the protocol that rules the communication between computers on the Web), and URLs (a method for identifying files).

…/…

The Web will be understood not as screenfuls of text and graphics but as a transport mechanism, the ether through which interactivity happens.

—Darcy DiNucci (1999)

The term became widely known with the first Web 2.0 conferences in 2003 and 2004 and XML was an important piece of the Web 2.0 puzzle through Ajax (Asynchronous JavaScript and XML), coined and defined by Jesse James Garrett in 2005 as:

	Ajax isn’t a technology. It’s really several technologies, each flourishing in its own right, coming together in powerful new ways. Ajax incorporates: standards-based presentation using XHTML and CSS; dynamic display and interaction using the Document Object Model; data interchange and manipulation using XML and XSLT; asynchronous data retrieval using XMLHttpRequest; and JavaScript binding everything together.
	—Jesse James Garrett (February 2005)

This definition shows how, back in 2005, some of us still thought that XML could dominate the Web and be used both to exchange documents (in XHTML) and data.

Unfortunately, this vision defended by the W3C, has been rapidely torpedoed by Ian Hickson and Douglas Crockford.

Founded in 1994 for that purpose, the W3C had been the place where HTML had been normalized. Among other things, the W3C had been the place where the antagonists of the first browser war could meet and discuss in a neutral field.

In 2004, Netscape had disappeared, Microsoft had frozen the development of their browser and browser innovation moved into the hand of new players: Mozilla, Apple/Safari and Opera who was starting to gain traction.

Complaining that the W3C did not meet their requirements and that HTML needed to be updated urgently to meet the requirements what would be soon known as Web 2.0, they decided to fork the development of HTML:

	Software developers are increasingly using the Internet as a software platform, with Web browsers serving as front ends for server-based services. Existing W3C technologies — including HTML, CSS and the DOM — are used, together with other technologies such as JavaScript, to build user interfaces for these Web-based applications.However, the aforementioned technologies were not developed with Web Applications in mind, and these systems often have to rely on poorly documented behaviors. Furthermore, the next generation of Web Applications will add new requirements to the development environment — requirements these technologies are not prepared to fulfill alone. The new technologies being developed by the W3C and IETFcan contribute to Web Applications, but these are often designed to address other needs and only consider Web Applications in a peripheral way.The Web Hypertext Applications Technology working group therefore intends to address the need for one coherent development environment for Web Applications. To this end, the working group will create technical specifications that are intended for implementation in mass-market Web browsers, in particular Safari, Mozilla, and Opera.
	—WHATWG (June 2004)

The W3C was behind a simple choice: either push XHTML recommendations that would never be implemented in any browsers or ditch XHTML and ask the WHATWG to come back and continue their work toward HTML5 as a W3C Working Group. The later option was eventually chosen and HTML work resumed within W3C in 2007.

JSON was around since 2001. It took a few years of Douglas Crockford’s energy to popularize this JavaScript subset but around 2005, JSON rapidly became a technology of choice as a « “Fat-Free Alternative to XML” » in Ajax applications.

There is no direct link between HTML5 and JSON but the reaction against XML, its hype and its perceived complexity is a strong motivation in both cases.

Why?

A number of reasons can be found for this failure:

Bad timing between the XML and HTML specifications (see Adam Retter’s presentation at XML Amsterdam 2011).
Lack of quality of some XML recommendations (XML Namespaces, XML Schema, …).
Lack of pedagogy to explain why XML is the nicer technology on the earth.
Dumbness of Web developers who not use XML.
…

There is some truth in all these explanations, but the main reason is that from the beginning we (the XML crowd) have been arrogant, over confident and have made a significant design error.

When we read this quote:

	XML arose from the recognition that key components of the original web infrastructure — HTML tagging, simple hypertext linking, and hardcoded presentation — would not scale up to meet the future needs of the web. This awareness started with people like me who were involved in industrial-strength electronic publishing before the web came into existence.
	—Jon Bosak

We all understand what Jon Bosak meant and we probably all agree that HTML is limited and that something more extensible makes our lives easier, but we must also admit that we have been proven wrong and that HTML has been enough to scale up to the amazing applications we see today.

Of course, the timing was wrong and everything would have been easier if Tim Berners-Lee had came up with a first version of HTML that would have been a well formed XML document but on the other hand, the web had to exist before we could put SGML on the web and there had to be a prior technology.

In 1998 it was already clear that HTML was widespread and the decision to create XML as a SGML subset that would be incompatible with HTML has been a bad one:

Technically speaking because that meant that millions of existing pages would be non well formed XML (« “the first Google index in 1998 already had 26 million pages”« ).
Tactically speaking because that could be understood as « what you’ve done so far was crappy, now you must do what we tell you to do ».

To avoid this deadly risk, the first design goal of XML should have been that existing valid HTML documents were well formed XML documents. The result might have been a more complex format and specification, but this risk to create a gap between XML and HTML communities would have been minimized.

Another reason to explain this failure is that XML is about extensibility. This is both its main strength and weakness: extensibility comes at a price and XML is more complex than domain specific languages.

Remove the need for extensibility and XML will always loose against DSLs, we’ve seen a number of examples in the past:

RELAX NG compact syntax
JSON
HTML
N3
CSS
…

Is it a time to refactor XML? Converge or convert?

Hmmm… It’s time to address the questions asked this year by XML Prague!

We’ve failed to establish XML as the format to use on the web but we’ve succeeded in creating a strong toolbox which is very powerful to power websites and exchange information.

I don’t know if it’s to compensate the ecosystems that we are destructing on our planet, but one of the current buzzwords among developers is « ecosystem »: dominant programming languages such as Java and JavaScript are becoming « ecosystems » that you can use to run a number of applications that may be written using other programming languages.

What we’ve built with XML during the past 14 years is a very strong ecosystem.

The XML ecosystem is based on an (almost) universal data model that can not only represent well formed XML documents but also HTML5 documents and (with an impedance mismatch that may be reduced in future versions) JSON objects.

	Note
	Notable exceptions that cannot be represented by the XML data model include overlapping structures and graphs.

On top of this data model, we have a unique toolbox that includes:

transformation and query languages
schema languages
processing (pipeline) languages
databases
web forms
APIs for traditional programming languages
signature and encryption standards
a text based serialization syntax
binary serialization syntaxes

We can truly say that what’s important in XML is not the syntax but that:

	Angle Brackets Are a Way of Life
	—Planet XMLHack

Rather than fighting fights that we’ve already lost we need to develop our ecosystem.

The number one priority is to make sure that our data model embraces the web that is taking shape (which means HTML5 and JSON) as efficiently as possible. Rather than converge or convert we must embrace, the actual syntax is not that important after all!

To grow our ecosystem, we could also consider embracing more data models, such as graphs (RDF), name/value pairs (NOSQL), relations (SQL), overlaps (LMNL).

I am more skeptical about refactoring XML at that stage.

It’s always interesting to think about what could be done better, but refactoring a technology as widespread as XML is tough and needs to be either backward compatible or provide a huge benefit to compensate the incompatibilities.

Will we see a proposal that will prove me wrong during the conference?