This year again, XML Amsterdam did connect XML developers worldwide and it’s time to post the links to my presentations.
Many thanks to everyone involved in making this event happen!
Update: After this post I have launched http://xformsunit.org.
XForms forms can be incredibly complex and I have been thinking of an XForms unit test environment for a while. Before I start coding here are some of my thoughts on the topic…
A common way of testing XForms application is to use a generic purpose web testing tool such as Selenium.
The benefits of such tools is that you can test exactly what is displayed in the browser and simulate user actions. It’s downside is that the tests are expressed in browser related terms rather than using XForms concepts. To write these tests you need to know how XForms will be transformed into HTML and this transformation depends on the XForms implementation being used and may vary between versions.
By contrast a native XForms test environment would allow to express the tests using XForms concepts such as binds, controls and events.
The basic idea is to describe a test suite and to add actions to the form to test so that the tests are executed by the XForms engine.
Let’s say we have the following form which is a slightly adapted version of the traditional XForms « hello world »:
<?xml version="1.0" encoding="UTF-8"?>
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:xf="http://www.w3.org/2002/xforms">
<head>
<title>Hello World in XForms</title>
<xf:model id="model">
<xf:instance id="instance" xmlns="">
<data>
<PersonGivenName></PersonGivenName>
<Greetings></Greetings>
</data>
</xf:instance>
<xf:bind id="greetings" nodeset="/data/Greetings"
calculate="concat('Hello ', ../PersonGivenName, '. We hope you like XForms!')"/>
</xf:model>
</head>
<body>
<p>Type your first name in the input box. <br /> If you are running XForms, the output should be displayed in
the output area.</p>
<xf:input ref="PersonGivenName" incremental="true">
<xf:label>Please enter your first name: </xf:label>
</xf:input>
<br />
<xf:output value="Greetings">
<xf:label>Output: </xf:label>
</xf:output>
</body>
</html>
To test that the greeting are what you’d be expecting you would enter a value in the name and check the greetings.
This test can be expressed as:
<?xml version="1.0" encoding="UTF-8"?>
<suite xmlns:xh="http://www.w3.org/1999/xhtml" xmlns:xf="http://www.w3.org/2002/xforms"
xmlns:xs="http://www.w3.org/2001/XMLSchema">
<form src="hello-world.xhtml"/>
<!-- The test cases -->
<case id="test-greetings">
<action>
<setvalue ref="instance('instance')/PersonGivenName">Eric</setvalue>
</action>
<assertEqual>
<actual ref="instance('instance')/Greetings"/>
<expected>Hello Eric. We hope you like XForms!</expected>
<message>The greetings should be the concatenation of "Hello ", the given name and ". We hope you like
XForms!".</message>
</assertEqual>
</case>
</suite>
The generation of the augmented form which will perform the tests can be done in XSLT.
To implement these tests in XForms, we can add an instance initialized with the description of the test suite and updated by XForms actions to reflect the result of the tests:
<xf:instance id="xfu:instance">
<suite xmlns="">
<case id="test-greetings">
<action id="d8e9">
<setvalue ref="instance('instance')/PersonGivenName" id="d8e11">Eric</setvalue>
</action>
<assertEqual id="d8e15" passed="">
<actual ref="instance('instance')/Greetings" id="d8e17"></actual>
<expected id="d8e19">Hello Eric. We hope you like XForms!</expected>
<message id="d8e22">The greetings should be the concatenation of "Hello ", the given name
and ". We hope you like XForms!".</message>
</assertEqual>
</case>
</suite>
</xf:instance>
Note how the transformation has added id attributes to the elements of this instance which did not already carry one. These ids will be helpful while writing the actions.
These actions themselves can be coded as:
<xf:dispatch targetid="model" name="xfu:d8e9-action"/>
<xf:action ev:event="xfu:d8e9-action">
<xf:setvalue ref="instance('instance')/PersonGivenName">Eric</xf:setvalue>
<xf:recalculate/>
<xf:dispatch targetid="model" name="xfu:d8e15-action"/>
</xf:action>
<xf:action ev:event="xfu:d8e15-action">
<xf:setvalue ref="instance('xfu:instance')//*[@id = 'd8e17']" value="instance('instance')/Greetings"/>
<xf:setvalue ref="instance('xfu:instance')//*[@id = 'd8e15']/@passed" value="(instance('instance')/Greetings) = 'Hello Eric. We hope you like XForms!'"/>
</xf:action>
In this version I have chosen to encapsulate each action and test in its own XForms action and use custom events to trigger these actions one after the other. Also note the xf:recalculate which is necessary to make sure that the greetings are recalculated after we’ve changed the name.
Now that xfu:instance contains the result of the tests, it could be saved on a server and can also be displayed using XForms controls such as:
<xf:group xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xh="http://www.w3.org/1999/xhtml"
xmlns:ev="http://www.w3.org/2001/xml-events" xmlns:xfu="http://xformsunit.org/"
ref="instance('xfu:instance')">
<xh:h3>Test results</xh:h3>
<xh:dl>
<xf:repeat nodeset="case">
<xh:dt><xh:dfn><xf:output ref="@id"></xf:output></xh:dfn></xh:dt>
<xh:dd><xh:ul>
<xf:repeat nodeset="*[@passed]">
<xh:li><xf:group ref=".[@passed = 'true']">
<xh:span>passed</xh:span>
</xf:group><xf:group ref=".[@passed = 'false']">
<xh:span>failed</xh:span>
<xf:output ref="actual">
<xf:label>Actual : </xf:label>
</xf:output>
<xf:output ref="expected">
<xf:label>Expected : </xf:label>
</xf:output>
</xf:group></xh:li>
</xf:repeat>
</xh:ul></xh:dd>
</xf:repeat>
</xh:dl>
</xf:group>
The resulting forms works fine with Orbeon Forms:
It does also work with betterForm:
But it fails when run on XSLTForms:
The error raised by XSLTForms is caused by an issue with the support of the xf:recalculate action. A possible workaround is to add a delay in the dispatch action that triggers the test:
<xf:dispatch targetid="model" name="xfu:d8e15-action" delay="1">
Note that this delay causes additional client/server exchanges for server based implementations such as Orbeon Forms and the workaround should be used specifically with XSLTForms.
In our test we check that an instance value is what we expect, but can we be sure that this instance value is displayed? This is pretty obvious in our example, but what if there was a typo in the control:
<xf:output value="greetings"> <xf:label>Output: </xf:label> </xf:output>
Here the name of the element is lower case and the xf:output will display an empty string but the test will still pass.
The xf:output could also be embedded in more complex xf:group elements, such as:
<xf:group ref=".[starts-with(PersonGivenName, 'J')]"> <xf:output value="Greetings"> <xf:label>Output: </xf:label> </xf:output> </xf:group>
Here the greetings are displayed only when the name starts with a « J » but, again, the test still passes.
The visibility of the greetings could also be affected by class or style attributes using AVTs:
<div style="{if (starts-with(PersonGivenName, 'J')) then '' else 'display: none' }">
<xf:output value="Greetings">
<xf:label>Output: </xf:label>
</xf:output>
</div>
The effect would, again, be to display the greetings only if the name starts with a « J » but this would be much more complex to detect.
What’s missing here is the ability of functions to check the status of the controls themselves rather than instance values.
Orbeon Forms does provide extension functions working on controls which may be useful but are specific to this implementation.
More generally speaking there is a lack of standard introspection functions, not only for controls but also on instances and testing if an instance node is relevant, readonly, required or valid is not possible without using extension functions. Some (but not all) these needs are covered by EXForms functions but their support by actual implementations has to be checked.
Despite some limitations there seems to be a nice potential in a native XForms test framework and of course your feedback is most welcome.
Can test driven development be applied to XML technologies? to XSLT? to XML schema languages? XML pipelines?
This is the topic of the poster I am presenting this year at the Balisage conference.
For those of you who are not lucky enough to attend, here is a copy of this poster: Test driven development in XML Balisage 2012 poster
Note: this article is derived from the presentation I have given at the International Symposium on Quality Assurance and Quality Control in XML (precedings).
Abstract
Ever modified an XML schema? Ever broken something while fixing a bug or adding a new feature? As withany piece of engineering, the more complex a schema is, the harder it is to maintain. In other domains, unit tests dramatically reduce the number of regressions and thus provide a kind of safety net for maintainers. Can we learn from these techniques and adapt them to XML schema languages? In this workshop session, we develop a schema using unit test techniques, to illustrate their benefits in this domain.
Table of Contents
The workshop is run as an exchange between a customer (played by Tommie Usdin) and a schema expert (played by Eric van der Vlist).
The customer, needs a schema for her to list XML application, is puzzled by the « test first programming » technique imposed by the schema expert.
At the end of the day (or workshop), will she be converted to this well known agile or extreme programming technique adapted to the development of XML schemas?
| Hi Eric, can you help me to write a schema? | ||
| —Customer | ||
| Hi Tommie, yes, sure, what will the schema be about? | ||
| —Expert | ||
| I need a vocabulary for my todo lists, with todo ite… | ||
| —Customer | ||
| OK, you’ve told me enough, let’s get started | ||
| —Expert (interrupting his customer) | ||
| Get started? but I haven’t told you anything about it! | ||
| —Customer | ||
| Right, but it’s never too soon to write tests when you do test first programming! | ||
| —Expert | ||
 |
Note |
|---|---|
| Test first programming (also called test driven development) developers create test case (usually unit test cases) before implementing a function. The test suite is run, code is written based on the result of these tests and the test suite and code are updated untill all the tests pass. |
Test suite
(suite.xml):
<tf:suite xmlns:tf="http://xmlschemata.org/test-first/" xmlns:todo="http://balisage.net/todo#" title="Basic tests">
<tf:case title="Root element" expected="valid" id="root">
<todo:list/>
</tf:case>
</tf:suite>
|
Note |
|---|---|
| The vocabulary used to define these test cases has been inspired by the SUT (XML Schema Unit Test) project. It’s a simple vocabulary (composed of only three different element) allowing to pack several XML instances together with the outcome validation result. It uses conventions that you’ll discover during the course of this workshop. |
|
Note |
|---|---|
| The test suite is run using a simple Orbeon Forms application. The rendering relies on Orbeon Forms XForms’ implementation while the test suite is run using an Orbeon Forms’ XPLpipeline. |
| You see, you can already write todo lists! | ||
| —Expert | ||
| Hold on, we don’t have any schema! | ||
| —Customer | ||
| That’s true, but you don’t have to write a schema to write XML documents. | ||
| —Expert | ||
| I know, but you’re here to write a schema! Furthermore right now we accept anything. I don’t want
to have XML documents with anything as a root element! |
||
| —Customer | ||
| That’s a good reason to write a schema but before that we need to add a test in our suite
first. |
||
| —Expert | ||
Test suite
(suite.xml):
<?xml version="1.0" encoding="UTF-8"?>
<tf:suite xmlns:tf="http://xmlschemata.org/test-first/" xmlns:todo="http://balisage.net/todo#" title="Basic tests">
<tf:case title="TODO list toot element" expected="valid" id="root">
<todo:list/>
</tf:case>
<tf:case title="Other root element" expected="error" id="other-root">
<todo:title>A title</todo:title>
</tf:case>
</tf:suite>
| Now that we’ve updated the test suite, we run it again. | ||
| —Expert | ||
| This result was expected and we can now proceed to create a schema and attach it to the test suite. | ||
| —Expert | ||
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema
xmlns:xs="http://www.w3.org/2001/XMLSchema"
elementFormDefault="qualified"
targetNamespace="http://balisage.net/todo#"
xmlns="http://balisage.net/todo#">
<xs:element
name="list"/>
</xs:schema>
todo.xsd
<?xml version="1.0" encoding="UTF-8"?>
<tf:suite
xmlns:tf="http://xmlschemata.org/test-first/"
xmlns:todo="http://balisage.net/todo#"
title="Basic tests">
<tf:validation
href="todo.xsd"
type="xsd"/>
<tf:case
title="TODO list toot element"
expected="valid"
id="root">
<todo:list/>
</tf:case>
<tf:case
title="Other root element"
expected="error"
id="other-root">
<todo:title>A title</todo:title>
</tf:case>
</tf:suite>
suite.xml
| It’s time to test again what we’ve done. | ||
| —Expert | ||
| I am happy to see some progress, at last, but I don’t want to accept any content in the todo list element. Can you add list title elements? | ||
| —Customer | ||
| Sure, back to the test suite… | ||
| —Expert | ||
Test suite
(suite.xml):
<?xml version="1.0" encoding="UTF-8"?>
<tf:suite
xmlns:tf="http://xmlschemata.org/test-first/"
xmlns:todo="http://balisage.net/todo#"
title="Basic tests">
<tf:validation
href="todo.xsd"
type="xsd"/>
<tf:case
title="TODO list root element"
expected="valid"
id="root">
<todo:list/>
</tf:case>
<tf:case
title="TODO list with a title"
expected="valid"
id="list-title">
<todo:list>
<todo:title/>
</todo:list>
</tf:case>
<tf:case
title="Other root element"
expected="error"
id="other-root">
<todo:title>A title</todo:title>
</tf:case>
</tf:suite>
| Now that we’ve updated the test suite, we run it again. | ||
| —Expert | ||
| You see? We do already support list title elements! | ||
| —Expert | ||
| Sure, but I don’t want to accept any content in my todo list. And the title element should be mandatory. And it should not be empty by have at least one character! | ||
| —Customer | ||
| Back to the test suite, then… | ||
| —Expert | ||
Test suite
(suite.xml):
<?xml version="1.0" encoding="UTF-8"?>
<tf:suite
xmlns:tf="http://xmlschemata.org/test-first/"
xmlns:todo="http://balisage.net/todo#"
title="Basic tests">
<tf:validation
href="todo.xsd"
type="xsd"/>
<todo:list>
<tf:case
title="Empty list element"
expected="error"
id="root-empty"/>
<todo:title>
<tf:case title="empty title" expected="error" id="empty-title"/>
<tf:case title="non empty title" expected="valid" id="non-empty-title">A title</tf:case>
</todo:title>
<tf:case
title="Un expected element"
expected="error"
id="unexpected">
<todo:foo/>
</tf:case>
</todo:list>
<tf:case
title="Other root element"
expected="error"
id="other-root">
<todo:title>A title</todo:title>
</tf:case>
</tf:suite>
|
Note |
|---|---|
This is the first example with non top level tf:case elements. To understand how this works, we must look in more detail to the algorithm used by the framework to split a test suite into instances. The algorithm consists in two steps:
This description may look complex, but the result is a rather intuitive way to define sub-trees that are common to several test cases. |
| Now that we’ve updated the test suite, we run it again. | ||
| —Expert | ||
| Sounds good, now we can update the schema. | ||
| —Expert | ||
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema
xmlns:xs="http://www.w3.org/2001/XMLSchema"
elementFormDefault="qualified"
targetNamespace="http://balisage.net/todo#"
xmlns="http://balisage.net/todo#">
<xs:element
name="list">
<xs:complexType>
<xs:sequence>
<xs:element
name="title">
<xs:simpleType>
<xs:restriction
base="xs:token">
<xs:minLength
value="1"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>
todo.xsd
| And run the test suite again. | ||
| —Expert | ||
| Good. Now I want to add todo items. And lists should have at least one of them, by the way. | ||
| —Customer | ||
| Sure, back to the test suite… | ||
| —Expert | ||
Test suite
(suite.xml):
<?xml version="1.0" encoding="UTF-8"?>
<tf:suite
xmlns:tf="http://xmlschemata.org/test-first/"
xmlns:todo="http://balisage.net/todo#"
title="Basic tests">
<tf:validation
href="todo.xsd"
type="xsd"/>
<tf:case
title="Empty list element"
expected="error"
id="root-empty">
<todo:list/>
</tf:case>
<todo:list>
<!-- Testing title elements -->
<todo:title>
<tf:case
title="empty title"
expected="error"
id="empty-title"/>
<tf:case
title="non empty title"
expected="valid"
id="non-empty-title">A title</tf:case>
</todo:title>
<todo:item>
<todo:title>A title</todo:title>
</todo:item>
<tf:case
title="Un expected element"
expected="error"
id="unexpected">
<todo:foo/>
</tf:case>
</todo:list>
<todo:list>
<!-- Testing todo items -->
<todo:title>Testing todo items</todo:title>
<tf:case
title="No todo items"
expected="error"
id="no-items"/>
<todo:item>
<tf:case
title="empty item"
expected="error"
id="empty-item"/>
<tf:case
title="item with a title"
expected="valid"
id="item-title">
<todo:title>A title</todo:title>
</tf:case>
</todo:item>
</todo:list>
<tf:case
title="Other root element"
expected="error"
id="other-root">
<todo:title>A title</todo:title>
</tf:case>
</tf:suite>
| Let’s see what we get before any update to the schema: | ||
| —Expert | ||
| It’s time to update the schema and fix what needs to be fixed: | ||
| —Expert | ||
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema
xmlns:xs="http://www.w3.org/2001/XMLSchema"
elementFormDefault="qualified"
targetNamespace="http://balisage.net/todo#"
xmlns="http://balisage.net/todo#">
<xs:element
name="list">
<xs:complexType>
<xs:sequence>
<xs:element
name="title">
<xs:simpleType>
<xs:restriction
base="xs:token">
<xs:minLength
value="1"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element
maxOccurs="unbounded"
name="item">
<xs:complexType>
<xs:sequence>
<xs:element
name="title">
<xs:simpleType>
<xs:restriction
base="xs:token">
<xs:minLength
value="1"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>
todo.xsd
| And now we can check if we get it right. | ||
| —Expert | ||
| Eric, you should be ashamed, it’s a pure Russian doll schema, not modular at all! Why not define the title and list elements globally? | ||
| —Customer | ||
| Sure, we can do that! If we just change the structure of the schema, we don’t need to update the test suite and can work directly on the schema: | ||
| —Expert | ||
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema
xmlns:xs="http://www.w3.org/2001/XMLSchema"
elementFormDefault="qualified"
targetNamespace="http://balisage.net/todo#"
xmlns="http://balisage.net/todo#">
<xs:element
name="list">
<xs:complexType>
<xs:sequence>
<xs:element
ref="title"/>
<xs:element
maxOccurs="unbounded"
ref="item"/>
</xs:sequence>
</xs:complexType>
</xs:element>
<xs:element
name="title">
<xs:simpleType>
<xs:restriction
base="xs:token">
<xs:minLength
value="1"/>
</xs:restriction>
</xs:simpleType>
</xs:element>
<xs:element
name="item">
<xs:complexType>
<xs:sequence>
<xs:element
ref="title"/>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>
todo.xsd
| But of course, each time we update the schema we must check if we’ve not introduced any bug: | ||
| —Expert | ||
| Waoo, what’s happening now? | ||
| —Customer | ||
| Now that our elements are global in the schema, we accept a valid title as a root element. Is that what you want? | ||
| —Expert | ||
| No way, a title is not a valid list! | ||
| —Customer | ||
| Then we have a number of options… We can go back to local elements and we can also add a schematron schema to check this constraint. | ||
| —Expert | ||
| Schematron is fine, we’ll probably find many other constraints to check in my todo lists anyway… | ||
| —Customer | ||
| OK. We still don’t have to update the test suite since we’ve not changed our requirement. Let’s write this Schematron schema then: | ||
| —Expert | ||
<?xml version="1.0" encoding="UTF-8"?>
<schema xmlns="http://purl.oclc.org/dsdl/schematron" queryBinding="xslt2">
<ns uri="http://balisage.net/todo#" prefix="todo"/>
<pattern>
<rule context="/*">
<assert test="self::todo:list">The root element should be a todo:list</assert>
</rule>
</pattern>
</schema>
todo.sch
| Saying that we don’t have to update the test suite wasn’t totally accurate because the schemas are referenced in ths test suite: | ||
| —Expert | ||
Test suite
(suite.xml):
<?xml version="1.0" encoding="UTF-8"?>
<tf:suite
xmlns:tf="http://xmlschemata.org/test-first/"
xmlns:todo="http://balisage.net/todo#"
title="Basic tests">
<tf:validation
href="todo.sch"
type="sch"/>
<tf:validation
href="todo.xsd"
type="xsd"/>
<tf:case
title="Empty list element"
expected="error"
id="root-empty">
<todo:list/>
</tf:case>
<todo:list>
<todo:title>
<tf:case
title="empty title"
expected="error"
id="empty-title"/>
<tf:case
title="non empty title"
expected="valid"
id="non-empty-title">A title</tf:case>
</todo:title>
<todo:item>
<todo:title>A title</todo:title>
</todo:item>
<tf:case
title="Un expected element"
expected="error"
id="unexpected">
<todo:foo/>
</tf:case>
</todo:list>
<todo:list>
<todo:title>Testing todo items</todo:title>
<tf:case
title="No todo items"
expected="error"
id="no-items"/>
<todo:item>
<tf:case
title="empty item"
expected="error"
id="empty-item"/>
<tf:case
title="item with a title"
expected="valid"
id="item-title">
<todo:title>A title</todo:title>
</tf:case>
</todo:item>
</todo:list>
<tf:case
title="Other root element"
expected="error"
id="other-root">
<todo:title>A title</todo:title>
</tf:case>
</tf:suite>
| Time to see if we’ve fixed our issue! | ||
| —Expert | ||
| Great, we’ve made it, thanks! | ||
| —Customer | ||
The application used to run the test suite and display its result is available at http://svn.xmlschemata.org/repository/downloads/tefisc/.
If you just want to understand how the test suite is split into XML instances, you can have a look at http://svn.xmlschemata.org/repository/downloads/tefisc/orbeon-resources/apps/tefisc/ .
In this directory:
To install this application:
/WEB-INF/resources/apps/in yourorbeon webapp directory<param-value>/home/vdv/projects/tefisc/orbeon-resources</param-value>by the location of this directory on your filesystem, replace /WEB-INF/web.xml by this file and restart your application server.