Auteur/autrice : Eric van der Vlist

The latest version of XSLTUnit (v0.2) is from January 2002 and people sometimes ask me if the project is dead and when XSLTUnit will support XSLT 2.0.

The short answer is that you can already use XSLTUnit with XSLT 2.0.

The project is not dead and if I haven’t published any new version it’s just because XSLTUnit meets my needs and nobody has ever asked me any update.

I use it a lot and, no later than this afternoon, came on a new opportunity to use it to add unit tests to a function that I needed to debug for the Owark project.

Following ideas to develop a web service to create page archives, I was writing an XSLT transformation that analyses Heritrix crawl logs to determine what needs to be packaged into the archives and one of the touchy functions is to create user friendly local names that remains unique within the scope of an archive.

My first naive attempt didn’t survive real world tests and results were rather disappointing.

Using these results as a test suite was an obvious idea…

To do so, I have used log.xml, a real crawl log converted to XML as my source document and local-names.xml, the result of the transformation, as a reference.

The XSLTUnit transformation that exploit these documents, local-names.xsl, is very simple and is a nice example of what you can do with this framework and how you can use it with XSLT 2.0.

An XSLTUnit test suite is an XSLT transformation that imports both the transformation to test and xsltunit.xsl:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:exsl="http://exslt.org/common" extension-element-prefixes="exsl" xmlns:xsltu="http://xsltunit.org/0/"
  xmlns:owk="http://owark.org/xslt/" exclude-result-prefixes="exsl">
  <xsl:import href="../actions/resource-index.xslt"/>
  <xsl:import href="xsltunit.xsl"/>

.../...

</xsl:stylesheet>

Here, the test suite and the XSLT transformation to test are both XSLT 2.0 and with Saxon they just work fine with xsltunit.xsl which is XSLT 1.0.

A XSLTUnit test case is composed of comparisons, and a test case to test that the value of the owk:local-name() function on a specific log entry is equal to « index.xml » could be:

      <xsltu:test id="index">
        <xsl:call-template name="xsltu:assertEqual">
          <xsl:with-param name="id" select="'index'"/>
          <xsl:with-param name="nodes1">
            <name>
              <xsl:value-of select="owk:local-name(/log/entry[uri='https://blog.eric.van-der-vlist.com/'])"/>
            </name>
          </xsl:with-param>
          <xsl:with-param name="nodes2">
            <name>index.html</name>
          </xsl:with-param>
        </xsl:call-template>
      </xsltu:test>

This crawl log includes 292 resources and you wouldn’t want to copy and past 291 times this snippet… No problem, you can just use some XSLT power and write:

 <?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:exsl="http://exslt.org/common" extension-element-prefixes="exsl" xmlns:xsltu="http://xsltunit.org/0/"
  xmlns:owk="http://owark.org/xslt/" exclude-result-prefixes="exsl">
  <xsl:import href="../actions/resource-index.xslt"/>
  <xsl:import href="xsltunit.xsl"/>
  <xsl:output method="xml" version="1.0" encoding="UTF-8" indent="yes"/>
  <xsl:variable name="local-names" select="doc('local-names.xml')/index"/>
  <xsl:key name="log-by-uri" match="/log/entry" use="uri"/>
  <xsl:template match="/">
    <xsltu:tests>
      <xsl:for-each select="$local-names/resource">
        <xsltu:test id="{uri}">
          <xsl:call-template name="xsltu:assertEqual">
            <xsl:with-param name="id" select="uri"/>
            <xsl:with-param name="nodes1">
              <local-name>
                <xsl:value-of select="owk:unique-local-name(key('log-by-uri', current()/uri, $source ))"/>
              </local-name>
            </xsl:with-param>
            <xsl:with-param name="nodes2">
              <xsl:copy-of select="local-name"/>
            </xsl:with-param>
          </xsl:call-template>
        </xsltu:test>
      </xsl:for-each>
    </xsltu:tests>
  </xsl:template>
</xsl:stylesheet>

When you’ve done so, you can edit « local-names.xml » that contains the expected values so that they look more like what you want to, run the test suite to detect the differences, update your transformation accordingly and iterate.

Note: these are musings and should not be taken as a concrete proposal!

Balisage’s tag line is « there is nothing so practical as a good theory ». I love Balisage but my brain doesn’t really work like that and its tag line would rather be « there is nothing as theoretical as a good exercise ».

χίμαιραλ (chimeral) is the kind of exercise I needed to get into XDM 3.0 (and probably even 2.0 that I had been using without really thinking about it) and I’d like to share the musings that this exercise has inspired.

For XDM, the most generic piece of information is called an « item ».

XDM 2.0 has introduced a distinction between two kind of items:

• Nodes which come directly from the XML infoset and borrow some properties to the PSVI.
• Atomic values that are « pure » values living their own lives independently of their occurrences in XML documents as text or attribute node values.

Coming straight from the XML infoset, nodes are rather concrete, and you can still smell the (electronic) ink of their tags… Each node is unique and lives in its own context: when you write:

<?xml version="1.0" encoding="UTF-8"?>
<root>
  <foo>5</foo>
  <foo>5</foo>
</root>

The two <foo/> elements are two different nodes. They may look identical and have the same content, they are still two different nodes like identical twins are different persons.

Their value, by contrast, is the same value: a 5 is a 5, the fact that it’s the value of the first or of the second <foo/> (or of a @bar attribute) doesn’t make any difference.

XDM 2.0 also introduced the notion of « sequences ». Sequences are really special. They are not considered as a model item and are just a kind of invisible bag to package items together and are useful because you can store them in variables and use them as parameters.

Sequences have some interesting magics…

They disappear by themselves when they pack only one item and their is no way to differentiate a sequence of one item from the item itself. XDM has invented a perfectly biodegradable bag!

They also disappear when you put them within another sequence and you can’t have sequences of sequences. You’d better think twice if you want to separate apple from oranges (or meat and cheese) before packaging them in a sequence!

<xsl:variable name="nodeset" select="/root/foo"/>

<xsl:variable name="rtf">
  <xsl:copy-of select="/root/foo"/>
</xsl:variable>

XDM 3.0 adds a third type of items: functions that become first class objects transforming XPath/XQuery/XSLT into functional languages.

Michael Kay’s proposal adds as a fourth item type: maps (and arrays considered as maps with integer keys). Maps being a brand new type of item, the choice is open: they could have been considered similar to XML nodes but the current proposal is to consider them as pure values.

In my previous post I have explained the consequences of this important design decision and I’d like to take a step backward and analyze the differences between values and nodes.

Still before discovering Fortran IV, I did study mathematics and geometry and I was fascinated by the dual approach for solving problem in geometry using either Euclidean vectors or points and segments.

In the current proposal, a map is like a vector: you can reuse it as many time as you like in other map entries and it will remain unique. On the contrary, a node is like a segment. It has an identity,  is bound to a parent and if you want to reuse it you need to copy and paste it.

In geometry, vectors and segments are both useful and complementary. I can understand that there is a need for both nodes and values, but why should maps always be values and nodes always be « concrete » nodes? Both are useful data structures, why should they be treated so differently?

I understand that there are (at least) to use cases for maps: to support JSON and to provide lightweight data structures. These two use cases look very different to me. Wanting to meet them both with a single feature, won’t we miss both points? Why should lightweight structures be limited to maps and why should maps always been lightweight?

In geometry the processes by which you create a vector from a segment or a segment from a vector by pinning one of its extremities are well known. Can’t we define similar processes to transform a « concrete » data structure (either node tree or map) into a pure value and vice versa?

This is not so uncommon for programmers: values can also be seen as classes (with class properties) and concrete structures as instantiations of these classes.

Now, what about chimeras mixing pure values and concrete nodes?

A handy feature of sequences and maps as currently proposed is that they can include nodes. This « inclusion » is done « by reference » and the nodes keep they identity and context. Maps using this feature are « pure values » with references to concrete nodes.

When a map entry is a node, the node keeps its context within a document and its parent remains its parent in this document. How can you do that if you want to also represent the reverse relation between the node and the map entry in which it has been added?

One option could be to define a mechanism similar to symbolic links on Linux: the link is a kind of shortcut and when you follow it you can’t always guarantee that you’ll be able to come back in the same directory. Adapted to the XDM, we wouldn’t store the parent/child relation between map entries and nodes. This would be a limitation, but we wouldn’t meet this limitation when dealing with maps de-serialized from JSON objects (JSON objects do not contain XML nodes).

Another option could be to « decorate » or « re-instantiate » the node so that the new instance has two parents, probably not with the same kind of parent/child relation but with a new kind that could be followed with a different axis. This decoration  would add new context information to a tree and would be very similar to the process by which values are turned into concrete nodes. Now, would that be a practical thing to do? What about adding a map with a node as a new entry on another map? Don’t we end up with nodes that can contain an indefinite number of ancestors?

I hope that these musings can be helpful, but I should probably stick to my role of XDM user rather than giving suggestions!

To summarize: I think that we need both lightweight map structures and the full set of XPath axis on maps de-serialized from JSON objects.

Having only lightweight map structures means that users (and probably other specs and technologies) will have to continue to define custom mappings between JSON and XML to perform serious work.

In the presentation I gave at XML Prague 2012 (see my paper), one of my conclusions was that the XML data model extended by the XPath/XQuery/XSLT Working Group to embrace other data models such as JSON was an important foundation of the whole XML ecosystem.

In her amazing keynote, Jeni Tennison warned us against chimeras,  “ugly, foolish or impossible fantasies” and I have thought that it would be useful to check to which extent the XPath/XQuery/XSLT 3.0 data model (aka XDM 3.0) deserves to be called a chimera.

The foundation of this data model is the XML infoset, but it also borrows informations items from the Post Schema Validation Infoset (the [in]famous PSVI) and adds its own abstract items such as sequences and, new in 3.0, functions and maps (needed to represent JSON objects).

I started to think more seriously about this, doing some researches and writing a proposal for Balisage and my plan was to wait until the conference to publish anything.

One of the things I planed to present is a simple XML serialization format for the XDM. My initial motivation to propose such a format was to have a visualization of the XDM: I find it difficult to represent it if its instances stay purely abstract and can’t be serialized and deserialized.

Working on this, I have soon discovered that this serialization can have other concrete benefits: the items that have been recently added to the XDM such as maps and even sequences are not treated as first class citizens by XPath/XQuery/XSLT and the data model can be easier to traverse using its serialization!

When for instance you have a complex map imported from JSON by the brand new parse-json() function, you can’t easily apply the templates on the map items and sub items. And of course, with a XML serialization that becomes trivial to do.

If such a serialization can be useful, there is no reason to wait until Balisage in August to discuss it and I’d like to introduce the very first version of  χίμαιραλ (chimeral), the Chimera Language.

The URL itself http://χίμαιραλ.com is a chimera composed of letters from two different alphabets and merging concepts from two different civilizations!

This first version is not complete. It already supports rather complex cases, but I need to think more how to deal with maps or sequences of nodes such as namespace nodes or attributes.

So far I am really impressed by XPath 3.0 but also surprised by many limitations in term of reflexion:

• No built in function to determine the basic type of an item (node, attribute, sequence, map, function, …).
• The dm:node-kind()accessor to determine the type of a node is abstract and XPath 3.0 does not expose it.
• The behavior of the exslt:object-type() function is surprising.

I may have missed something, but in practice I have found quite difficult when you have a variable to browse its data model.

The other aspect that I don’t like in XPath/XQuery/XSLT 3.0 is the lack of homogeneity between the way the different types of items are manipulated. This strengthen the feeling that we have a real chimera!

In XSLT for instance, I’d like to be able to apply templates and match items in the same way for any item types. Unfortunately, the features that are needed to do so (node tests, axis, …) are reserved to XML nodes. I can’t define a template that matches a map (nor a sequence by the way), I can’t apply templates over map items, …

It may be too late for version 3.0, but I really think that we should incorporate these recent additions to make them first class citizens!

Going forward, we could reconsider the way these items mix and match. Currently you can have sequences of maps, functions, nodes and atomic values, maps which values are sequences, functions, nodes and atomic values but nodes are only composed of other nodes.  Even if the XML syntax doesn’t support this, I would really like to see more symmetry and be able to add sequences and maps within nodes!

In other words, I think that it would be much more coherent to treat maps and sequences like nodes…

Note: The χίμαιραλ website is currently « read only » but comments are very welcome on this blog of by mail.