# HTML4 test suite documentation

Date:
  2002.06.24
Author:
  Arthur Bobroskie (Microsoft) <artbobo@microsoft.com> created this document from the CSS Test Suite Document authored by:
  Tantek Çelik (Microsoft) <tantekc@microsoft.com>

## Table of contents

1. Introduction
2. Using the test suites
3. Test Suite Format
4. Writing a Test Suite
5. Test Suite Principles
6. Acknowledgements

## Introduction

The HTML4 Test Suite is based upon the W3C HTML 4.01 recommendation. This test suite is applicable to user agents (particularly visual user agents). The W3C Validator should be used for validation of document authoring. The test suite consists of the following:

- Test Suite Documentation (this document)
- HTML Testable Assertions from the HTML 4.01 specification including author assertions
- Test Files, self documenting tests including test purpose and testable assertion numbers.
- Test Navigation pages for ease of navigation and reference to specifications.

This document explains how to use and create the tests and test harnesses for the HTML Test Suite. The HTML Test Suite is in the TNG format used by the CSS Test Suites. This document is derived from the CSS Test Suite Documentation. Concepts from the SVG Conformance Test Manual and the XML/SOAP Test Suite are also incorporated.

Sample text and hyperlinks from test suites are presented in this document with a thin grey border like this: test suite text. test suite hyperlink. Sample hyperlinks from test suites are not underlined to avoid misrepresenting them as actual hyperlinks in this document.

## Using the test suites

### Cover page

The cover page of a test suite contains a table of contents, structured parallel to the specification being tested. Often a test suite contains a Prologue which documents among other things requirements or recommendations for a specific system configuration, or suggestions for setting your system preferences in such a way as to minimize their interference with the tests themselves.

### Getting started with the tests

Get started with running the tests in a test suite by clicking on the first numbered link in the table of contents (skipping over the Prologue), or by clicking the get started here hyperlink that many test suites provide on their cover page.

### Next and Previous

On every following page, there will be a [Next] hyperlink that will take you linearly through each section, subsection and test of the test suite, finally bringing you back to the cover page at which point you have completed the test suite. On any of these pages you can always backtrack this linear order by following the [Previous] hyperlinks.

### Test content

On every page that contains a test, you should see the test content immediately below the abovementioned hyperlinks. Instead of the test itself, if you see a Test hyperlink, then your browser doesn't properly support the `<OBJECT>` element. No matter, simply click the Test hyperlink to navigate to the test itself. After viewing the test, use the Back functionality in your web browser to return to the navigation page for that test and proceed to the next page by clicking the [Next] hyperlink.

The test content is self-describing - that is, the test content contains a textual description of what the page should look like according to the test, e.g. 1. This first item of an ordered list should be numbered

## Test Suite Format

### Test suite pages

The test suite format consists of several different kinds of pages. At the core are the test pages themselves that exercise the features of the technology. Each test page is referenced by a navigation page that also references the previous / next navigation pages, and its section page. The test suite starts with a cover page, which links directly to each of the section pages. Here is a visualization of the relatively flat conceptual hierarchical structure of the test suite:

- Index page (cover page)
  - Prologue (section page - optional)
  - Section 1 (section page)
    - Subsection 1.1 (navigation page)
      - Tests for subsection 1.1 (test page)
    - Subsection 1.2 (navigation page)
      - Tests for subsection 1.2 (test page)
    - ...
    - Subsection 1.n (navigation page)
      - Tests for subsection 1.n (test page)
  - Section 2 (section page)
    - Subsection 2.1 (navigation page)
      - Tests for subsection 2.1 (test page)
    - Subsection 2.2 (navigation page)
      - Tests for subsection 2.2 (test page)
    - ...
    - Subsection 2.n (navigation page)
      - Tests for subsection 2.n (test page)
  - ...
  - Section n (section page)
    - Subsection n.1 (navigation page)
      - Tests for subsection n.1 (test page)
    - Subsection n.2 (navigation page)
      - Tests for subsection n.2 (test page)
    - ...
    - Subsection n.n (navigation page)
      - Tests for subsection n.n (test page)
  - Acknowledgments
  - Test Suite Errata
  - Related Resources

The individual types of pages in the test suite format are described from the inside out.

### Test page format

The HTML test pages typically contain the test purpose and assertions tested stated in a `<div class="TestPurpose">` and `<div class="AssertionsTested">` block, e.g.

    Verify that ordered and unordered lists are rendered.
    Tests for assertions 10.2-8, 10.2-9

Followed by a series of tests, each of which tests one item clearly. There may also be compound tests that follow the atomic tests.

Each test contains prose which describes the correct behavior.

The test pages use valid HTML 4.0 Strict or Transitional, which is the most compatible version of HTML for present day user agents. In the future, XHTML 1.0 Transitional which follows the XHTML 1.0 Appendix C guidelines may be more compatible across more user agents. XHTML Basic, again following the XHTML 1.0 Appendix C guidelines, may also be more compatible.

Finally, the test is named by the section.subsection number of the section being tested (with periods replace by underscores e.g. 1_1), a hyphen, a test type, a hyphen, a two digit number. E.g. 1_1-BF-01. This differs somewhat from the CSS and SVG Test file naming conventions. The underscore in the section name allows better tracking to the relevant section of the specification. The test type gives an indication of the type of test. Test types include:

- BF: Basic Functionality--verify rudimentary capability in the functional area
- DF: Detailed Functionality--comprehensively probe all testable assertions for a functional area
- BFD: Basic Functionality Demo--demonstrate functionality of a section of the specification or a module.
- ER: Error condition tests

The two digit number allows otherwise identical names to be used.

Test file names should be the same as the test title with the proper suffix (almost always .html) appended. E.g. 1_1-BF-01.html Supporting files for the test should be named with an alphabet character after the test name and before the .html suffix. E.g. 1_1-BF-01a.html Generic supporting files such as image files may be named differently. Note that the test file has no navigational content whatsoever. This allows the test file to be referenced from multiple different navigational files, and allows alteration of the navigation without having to update test files which would otherwise require implementers to retest test files every time test suite navigation was changed.

### Navigation page format

Each test page is contained within a navigation page. The navigation page is titled with the name of the test suite followed by the test name `<title>`,e.g. HTML Test Suite: 10_2-BF-01. The navigation page starts with a heading documenting the test suite, followed by the test name and the section heading. e.g. HTML Test Suite: Test 10_2-BF-01 Unordered lists. A horizontal rule separates the heading from the following links:

[Previous] [Next] [Section] [Contents] [Specification]

The Previous and Next hyperlinks link to the previous and next navigation pages respectively. Previous of the first navigation page of a section links to that section page. Next of the last navigation page of a section links to the next section. And Next of the last navigation page in the suite links to the cover page.

The Section hyperlink links to the section page for all the navigation pages for this section of the specification being tested. The Contents hyperlink links to the cover page of the test suite. The Specification hyperlink links to the specific section of the specification being tested.

The navigation page transcludes the test page with an `<OBJECT>` element, which references the test file using the `data` attribute and explicitly sets the `type` of the expected data to `"text/html"`. In order to encourage a user agent to use the entire width of the document, and the remainder of the document, both the `width` and `height` attributes are set to 100%.

The `<OBJECT>` element was chosen because it is present in all "current" versions of HTML, from HTML 4.0 Transitional, to XHTML Basic, to XHTML 1.1. This permits user agents which support any one of these languages to easily utilize the navigational pages. In addition, the `<OBJECT>` element provides an excellent immediate fall back mechanism for user agents that either happen to not support `<OBJECT>`, or that have difficulties transcluding HTML content.

Inside the `<OBJECT>` element is a simple Test hyperlink which directly links to the test file. Thus user agents that do not recognize the `<OBJECT>` element and therefore ignore its markup, show the simple hyperlink instead, thereby providing access to the test file.

The navigation pages also use valid HTML 4.0 Transitional, for the same reason as the test pages. Since the navigation pages are separated from the test pages, this could easily change in the future without having to change the test pages. Similarly, the test pages could be updated to use XHTML Basic for example without having to update the navigational pages.

Lastly, the navigation pages are named "sec" followed by the name of the test file being tested with the extension ".html". E.g. "sec1_1-BF-01.html". Note that the navigation file has no test content whatsoever. This allows the test file to be updated to fix problems in tests without having to update navigation files.

Note that some HTML test will not work within the navigation pages. In those instances a link to the actual test file will be displayed.

### Section page format

For each chapter in the specification, there is a corresponding section page which hyperlinks to the navigation pages of all the tests for that chapter. The section page format very much resembles the navigation page format.

The section page contains a flat list of hyperlinks to the specific navigation pages, in specification order. E.g. Test 6_15-BF-01.html

The list is demarcated on both top and bottom by horizontal rules, and a link bar with the following links:

[Previous] [Next] [Contents] [Specification]

The Previous hyperlink links to the last navigation page of the previous section. The Previous of the first section (typically a Prologue section) simply links to the test suite cover page. The Next hyperlink links to the first navigation page of this section.

The Contents hyperlink links to the cover page, and the Specification hyperlink links to the respective section in the specification.

Similar to the navigation pages, the section pages are also titled with a heading documenting the name of the test suite, followed by the numerical section of the specification. This heading is also used for the page `<title>`,e.g. HTML Test Suite: 10 Lists. The section pages are also written in valid HTML 4.0 Transitional, and named "section" followed by the top level section number and the extension ".html". E.g. "section10.html".

### Cover page format

A test suite's cover page begins with the W3C logo and a heading stating the name of the test suite which is also used for the page `<title>`.E.g.

HTML4 Test Suite

A short paragraph of text introduces the test suite, and documents its purpose, usage, and any caveats. Another paragraph minimally explains the structure of the test suite itself. Any additional test suite specific documentation should be placed in the Prologue. A subheading labels the Table of Contents which immediately follows the introductory text.

The table of contents is a simple list of numbered hyperlinks ( e.g.: 10 Lists ) to the section pages. Following the numbered section pages for the tests, there may be one or more lettered pages for test suite appendices, such as A. Acknowledgments, B. Test Suite Errata, and C. Related Resources.

A horizontal rule separates the Table of Contents from the cover page footer. The footer contains a mailto hyperlink for test suite feedback, the proper validation stamps (HTML, CSS), and recognition of the organizations responsible for the production of the test suite, along with their respective logos.

## Writing a Test Suite

The authoring of a test suite for a particular specification begins with the identification of testable assertions in the specification grouped by section and subsection of the specification.

Try to make use of existing test pages, by reference if possible rather than duplicating. Some specifications, such as profiles, do not contain new features of their own. New versions of specifications often build upon features of older specifications. In both cases, it should be possible, and it is desirable, to reuse existing test cases.

Start by creating a new directory for your test suite, and copying the necessary support files (listed below). All test suite files for your test suite will go into this one directory.

Using the templates listed below, author navigation pages for each feature and their respective testable assertions. If possible, author navigation pages which refer to tests from previous test suites. Use a relative URL in the `data` attribute of the `<OBJECT>` tag, and `href` of the anchor `<A>`, to refer to test pages from previous test suites. Replace the heading and document `<title>` with the name of the test suite, the subsection number and title. Fix the link bar hyperlinks (Previous, Next, Section, Specification) to point to the appropriate files.

For navigation pages which require new tests, write new test pages, also starting with the template listed below. Be sure to update the document `<title>` to be the same as the corresponding navigation page.

After completing all the navigation (and necessary test) pages for a section in the specification, write the section page (again, there is a template which can be used as a starting point). Once the section pages have been completed, construct the cover page and related support pages (Prologue, Acknowledgments, Test Suite Errata (hopefully fairly empty to begin with), and Related Resources). Templates aren't included for the related support pages, but they can be easily written starting from the respective pages in the CSS1 Test Suite.

Congratulations, you've authored a test suite.

### Support Files

- base.css
- basebg.gif
- static.css
- W3C.gif
- W3C-rsrc-test.css

### Templates

These templates are derived from the latest version of the CSS1 Test Suite, with a small bit of cleanup, so there may be minor inconsequential differences. Test suite authors MUST use these templates to write new CSS Test Suites so that alternative versions (such as XML or XHTML versions) may be mechanically generated by a tool.

#### Test page

The template for a test page is shown below:

## Test Suite Principles

Some basic principles have been learned during the construction of previous test suites, applicable to both graphics suites and others:

### Valid tests

All tests must test what they purport to test and all files in the test suite, whether harness or tests themselves must at a minimum pass the W3C Validator .

### Self-documenting

The tests should be self-documenting. Test cases should describe in prose, in the test cases themselves, what is expected. Each test case should at a minimum describe correct behavior. Error handling tests should at a minimum describe incorrect, or possibly unexpected behavior.

### Simple format

Simplicity is always a good design principle. Failure of a test should be caused by the assertion being tested and not because of numerous variables introduced by a complex test. A simple format also makes it easier to author and analyze the test.

### Atomic tests

A test suite should contain atomic (meaning, testing a single feature at a time) tests. Some features have little or no effect by themselves, and therefore may require simple compound tests at a minimum.

### Based on testable statements

Tests must be based upon, and traceable to, testable statements in the specification being tested.The HTML test suite will be based on the testable assertions document.

### Reducing number of tests

Without sacrificing the principle of atomic testing, the number of test instances (files) can be reduced by having a single instance combine multiple related test purposes.

### Ease of authoring

Ease of authoring to implies that the test suite format must be easily hand authorable, that is, easily authorable with only the use of a simple text editor without a dependency upon any particular tools or any particular platform or device.

### Incremental difficulty

Test suites should begin with simpler, easier test cases, and proceed to more complex, more challenging tests. This makes sense both from a test case authoring perspective, since simpler cases are easier to author, and from a technology implementer's perspective, since simpler cases are easier to implement.

### Comprehensive

The detailed tests should try to methodically vary and test all values, plus boundary conditions and extreme conditions, of each parameter, attribute, or property.

### Something rather than nothing

The creation of a test suite is an iterative process. The creation of a conformance test suite for HTML is daunting. Start with something, then make it better. A simple incomplete test suite is far far better than none at all.

### Sooner rather than later

Corollary: a simple test suite within a few months is more useful, and thus preferred to, a more complex, more fully featured test suite a year or more from now.

### Iterate

The first version of the test suite will be imperfect and incomplete. Iterate to achieve perfection and completeness.

### Test the technology being tested, not other technologies

When writing a test suite for a particular technology (e.g. HTML), it is often necessary to utilize other technologies (e.g. CSS, ECMAScript, DOM etc.) in the construction of that test suite. Test suite authors must be careful while testing one technology, to avoid testing other technologies. Those other technologies should be authored to be maximally compatible, in a way that is most forgiving to the various user agents which will be using the test suite. Consequently, HTML test suite files should:

- Be composed primarily if not exclusively of HTML files
- use the file extension .html for HTML and XHTML files
- use the mime type "text/html" for HTML and XHTML files
- use HTML4.0 (or 4.01) Strict or Transitional document type for HTML files
- Avoid any testing for, or usage of, any particular user agent string or identifying portion thereof.
- Avoid using technologies that are not W3C standard.

### Reuse

Test suites should attempt to reuse test cases from previous test suites.

### Non-requirements

Finally, as with any set of principles and requirements, there are some non-requirements. These non-requirements are exactly that - they are features not required of a test suite. They may be desirable, or even included - yet they are not necessary to produce a test suite.

- automation - the ability to automatically run a test suite.
- use of a meta-format for generating multiple variants

### Additional principles

The SVG Test Manual documents additional Principles Applicable to Test Suite Content . Future test suite authors should consult these principles as well.

## Acknowledgments

This HTML Test Suite Documentation is mostly an edited version of the CSS Test Suite Documentation and this document would have been very difficult to create without it. The HTML Test Suite is derived from the work of many people at the W3C and other organizations and companies.

Feedback regarding the HTML4 Test Suite should be sent to www-html-testsuite@w3.org.

The HTML4 Test Suite is an effort of the World Wide Web Consortium. |