XForms Unit

Update: After this post I have launched http://xformsunit.org.

XForms forms can be incredibly complex and I have been thinking of an XForms unit test environment for a while. Before I start coding here are some of my thoughts on the topic…

XForms test cases should be native

A common way of testing XForms application is to use a generic purpose web testing tool such as Selenium.

The benefits of such tools is that you can test exactly what is displayed in the browser and simulate user actions. It’s downside is that the tests are expressed in browser related terms rather than using XForms concepts. To write these tests you need to know how XForms will be transformed into HTML and this transformation depends on the XForms implementation being used and may vary between versions.

By contrast a native XForms test environment would allow to express the tests using XForms concepts such as binds, controls and events.

 Principles

The basic idea is to describe a test suite and to add actions to the form to test so that the tests are executed by the XForms engine.

Let’s say we have the following form which is a slightly adapted version of the traditional XForms “hello world”:

<?xml version="1.0" encoding="UTF-8"?>
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:xf="http://www.w3.org/2002/xforms">
    <head>
        <title>Hello World in XForms</title>
        <xf:model id="model">
            <xf:instance id="instance" xmlns="">
                <data>
                    <PersonGivenName></PersonGivenName>
                    <Greetings></Greetings>
                </data>
            </xf:instance>
            <xf:bind id="greetings" nodeset="/data/Greetings"
                calculate="concat('Hello ', ../PersonGivenName, '. We hope you like XForms!')"/>
        </xf:model>
    </head>
    <body>
        <p>Type your first name in the input box. <br /> If you are running XForms, the output should be displayed in
            the output area.</p>
        <xf:input ref="PersonGivenName" incremental="true">
            <xf:label>Please enter your first name: </xf:label>
        </xf:input>
        <br />
        <xf:output value="Greetings">
            <xf:label>Output: </xf:label>
        </xf:output>
    </body>
</html>

To test that the greeting are what you’d be expecting you would enter a value in the name and check the greetings.

This test can be expressed as:

<?xml version="1.0" encoding="UTF-8"?>
<suite xmlns:xh="http://www.w3.org/1999/xhtml" xmlns:xf="http://www.w3.org/2002/xforms"
    xmlns:xs="http://www.w3.org/2001/XMLSchema">
    <form src="hello-world.xhtml"/>

    <!-- The test cases -->
    <case id="test-greetings">
        <action>
            <setvalue ref="instance('instance')/PersonGivenName">Eric</setvalue>
        </action>
        <assertEqual>
            <actual ref="instance('instance')/Greetings"/>
            <expected>Hello Eric. We hope you like XForms!</expected>
            <message>The greetings should be the concatenation of "Hello ", the given name and ". We hope you like
                XForms!".</message>
        </assertEqual>
    </case>
</suite>

The generation of the augmented form which will perform the tests can be done in XSLT.

To implement these tests in XForms, we can add an instance initialized with the description of the test suite and updated by XForms actions to reflect the result of the tests:

<xf:instance id="xfu:instance">
    <suite xmlns="">
        <case id="test-greetings">
            <action id="d8e9">
                <setvalue ref="instance('instance')/PersonGivenName" id="d8e11">Eric</setvalue>
            </action>
            <assertEqual id="d8e15" passed="">
                <actual ref="instance('instance')/Greetings" id="d8e17"></actual>
                <expected id="d8e19">Hello Eric. We hope you like XForms!</expected>
                <message id="d8e22">The greetings should be the concatenation of "Hello ", the given name
                    and ". We hope you like XForms!".</message>
            </assertEqual>
        </case>
    </suite>
</xf:instance>

Note how the transformation has added id attributes to the elements of this instance which did not already carry one. These ids will be helpful while writing the actions.

These actions themselves can be coded as:

<xf:dispatch targetid="model" name="xfu:d8e9-action"/>
<xf:action ev:event="xfu:d8e9-action">
    <xf:setvalue ref="instance('instance')/PersonGivenName">Eric</xf:setvalue>
    <xf:recalculate/>
    <xf:dispatch targetid="model" name="xfu:d8e15-action"/>
</xf:action>
<xf:action ev:event="xfu:d8e15-action">
    <xf:setvalue ref="instance('xfu:instance')//*[@id = 'd8e17']" value="instance('instance')/Greetings"/>
    <xf:setvalue ref="instance('xfu:instance')//*[@id = 'd8e15']/@passed" value="(instance('instance')/Greetings) = 'Hello Eric. We hope you like XForms!'"/>
</xf:action>

In this version I have chosen to encapsulate each action and test in its own XForms action and use custom events to trigger these actions one after the other. Also note the xf:recalculate which is necessary to make sure that the greetings are recalculated after we’ve changed the  name.

Now that xfu:instance contains the result of the tests, it could be saved on a server and can also be displayed using XForms controls such as:

<xf:group xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xh="http://www.w3.org/1999/xhtml"
    xmlns:ev="http://www.w3.org/2001/xml-events" xmlns:xfu="http://xformsunit.org/"
    ref="instance('xfu:instance')">
    <xh:h3>Test results</xh:h3>
    <xh:dl>
        <xf:repeat nodeset="case">
            <xh:dt><xh:dfn><xf:output ref="@id"></xf:output></xh:dfn></xh:dt>
            <xh:dd><xh:ul>
                    <xf:repeat nodeset="*[@passed]">
                        <xh:li><xf:group ref=".[@passed = 'true']">
                                <xh:span>passed</xh:span>
                            </xf:group><xf:group ref=".[@passed = 'false']">
                                <xh:span>failed</xh:span>
                                <xf:output ref="actual">
                                    <xf:label>Actual : </xf:label>
                                </xf:output>
                                <xf:output ref="expected">
                                    <xf:label>Expected : </xf:label>
                                </xf:output>
                            </xf:group></xh:li>
                    </xf:repeat>
                </xh:ul></xh:dd>
        </xf:repeat>
    </xh:dl>
</xf:group>

The resulting forms works fine with Orbeon Forms:

Screenshot of Orbeon Forms running the tests.
Screenshot of Orbeon Forms running the tests.

It does also work with betterForm:

Screenshot of betterFORM running the tests
Screenshot of betterFORM running the tests

But it fails when run on XSLTForms:

Screenshot of XSLTForms running the tests.
Screenshot of XSLTForms running the tests.

The error raised by XSLTForms is caused by an issue with the support of the xf:recalculate action. A possible workaround is to add a delay in the dispatch action that triggers the test:

<xf:dispatch targetid="model" name="xfu:d8e15-action" delay="1">

Note that this delay causes additional client/server exchanges for server based implementations such as Orbeon Forms and the workaround should be used specifically with XSLTForms.

What you test and what you see

In our test we check that an instance value is what we expect, but can we be sure that this instance value is displayed? This is pretty obvious in our example, but what if there was a typo in the control:

        <xf:output value="greetings">
            <xf:label>Output: </xf:label>
        </xf:output>

Here  the name of the element is lower case and the xf:output will display an empty string but the test will still pass.

The xf:output could also be embedded in more complex xf:group elements, such as:

         <xf:group ref=".[starts-with(PersonGivenName, 'J')]">
            <xf:output value="Greetings">
                <xf:label>Output: </xf:label>
            </xf:output>
        </xf:group>

Here the greetings are displayed only when the name starts with a “J” but, again, the test still passes.

The visibility of the greetings could also be affected by class or style attributes using AVTs:

        <div style="{if (starts-with(PersonGivenName, 'J')) then '' else 'display: none' }">
            <xf:output value="Greetings">
                <xf:label>Output: </xf:label>
            </xf:output>
        </div>

The effect would, again, be to display the greetings only if the name starts with a “J” but this would be much more complex to detect.

What’s missing here is the ability of functions to check the status of the controls themselves rather than instance values.

Orbeon Forms does provide extension functions working on controls which may be useful but are specific to this implementation.

 Introspection

More generally speaking there is a lack of standard introspection functions, not only for controls but also on instances and testing if an instance node is relevant, readonly, required or valid is not possible without using extension functions. Some (but not all) these needs are covered by EXForms functions but their support by actual implementations has to be checked.

Conclusion

Despite some limitations there seems to be a nice potential in a native XForms test framework and of course your feedback is most welcome.

 

Share and Enjoy:
  • Identi.ca
  • StumbleUpon
  • del.icio.us
  • Facebook
  • Twitter
  • Add to favorites

12 thoughts on “XForms Unit”

  1. Sometimes an idea gets low feedback, not because it’s bad, but because it’s so well explained that few contributions can be made. And this is the case.

    I don’t see a clear improvement I could made, just a question: How to test for invalid input values?

    It’s common to test the UI with invalid values, like letters in numeric fields, invalid values for constraints, large texts in small fields, etc. Maybe in this case, an extension function could be necessary in order to check if the XForms engine marks the field as invalid and the submission is disabled.

    1. Thanks for the kind words.

      The easiest way to check for invalid values (and I agree that this is an important feature) would indeed be to use an extension function such as Orbeon’s xxf:valid. I haven’t found any similar page describing extension functions for EXSLTForms nor for betterFORM and that makes it difficult to know if equivalent functions exist for these implementations… And BTW, I am also looking for the list of functions available in XForms 2.0 which is missing from the last Working Draft!

      Another option might be to listen to the xforms-valid/xforms-invalid events for the nodes we’re testing and maintain the state (valid/invalid) somewhere in an instance. One issue is that, and I don’t really understand why, these events are dispatched to controls (and groups and switchs) rather than instance nodes. That means that if we want to be able to express these tests on instance nodes rather than controls or groups we may have to generate dummy groups to get these events. Seems doable but that requires some more thoughts and probably some hard work to get something portable!

      Thanks,

      Eric

    1. Wonderful, I had been looking for this page (the list of XForms 2.0 functions) for ages!

      I think that, more generally speaking, functions to “navigate” between the model and the view would be most useful.

      Thanks,

      Eric

  2. Hi Eric,

    Nice writeup! Three years ago I started writing a test harnas for XForms implementations. To test it I decided to add the XForms test suite, but I stopped working on it before I wrote all the tests of the XForms test suite and only made a ‘driver’ for Orbeon. You can find the project at https://github.com/nvdbleek/com.orbeon.testsuite.w3c. It is a bit different approach because it uses an processor specific selenium plugin to get access to control values, do updates and activate buttons, but maybe it is still interesting for you or others that are writing test frameworks for XForms implementations.

    I might pick up the work when we start working on a test suite for XForms 2.0, this makes it much easier for implementers to submit test reports, they don’t have to run the tests manually.

    1. Hi Nick,

      Thanks for the pointer!

      There are interesting similarities between testing the compliance of an XForms implementation with the spec and testing that a form is doing what is expected.

      The main difference is of course that in the later (which is what I have in mind for XForms Unit) you assume that you can trust the XForms implementation that you’ve chosen and define and can implement the tests using native XForms mechanisms while when you’re validating implementations you can’t escape using lower level tools.

      That being said, it would be nice if the vocabulary used to define the tests in both cases could be as close as possible and I would certainly welcome feedback from the W3C Working Group!

      Thanks,

      Eric

Leave a Reply

Your email address will not be published. Required fields are marked *

Enter your OpenID as your website to log and skip name and email validation and moderation!