Testcases

There are two main testcase sets for Duktape:

• ECMAScript testcases (tests/ecmascript) for testing ECMAScript compliance, "real world" behavior, and Duktape specific behavior.
• API testcases (tests/api) for testing the Duktape specific C API.

Testcases are written using an "expected string" approach: a testcase file describes the expected output using a custom markup (described below) and also contains the ECMAScript or C code that is intended to produce that output. A test runner compares actual output to expected; known issue files are used to document "known bad" outputs.

This document describes the testcase formats and current test tools.

ECMAScript testcase format

Testcases are plain ECMAScript (.js) files with custom markup inside comments for providing test metadata, expected output, and include files. A testcase is "prepared" before execution using util/runtest.py:

• Testcase metadata and expected string are parsed from inside custom markup.
• A minified prologue is injected to provide a global Test object. The prologue also harmonizes the execution environment so that e.g. print() and console.log() are available so that the prepared test can be executed using Duktape, V8, etc.
• Include files are located, minified, and included into the prepared test. All utilities included must work in both strict and non-strict contexts because testcases may be either strict or non-strict programs.
• A "use strict"; declaration is prepended (even before the prologue) if test metadata indicates it is needed. This is needed when the testcase is exercising strict program code behavior.

The prologue and include files are minified to one-liners so that they don't offset the line numbers of the testcase. This is important for tests that exercise traceback line numbers for example.

Include files are specified using the following syntax:

/*@include util-buffer.js@*/

Testcase metadata is provided as JSON or YAML inside a comment block. If multiple blocks are present they are merged, with the last occurrence of a key overwriting previous occurrences:

/*---
{
    "custom": true
}
---*/

// or

/*---
custom: true
---*/

The metadata keys change over time; current keys are described below. Metadata is optional.

Finally, the expected output is specified using the following syntax:

/*===
hello world
===*/

print('hello world');

There's also a single-line shorthand:

print('hello world');  //>hello world

Full testcase example:

/*
 *  Example test.
 */

/*@include util-foo.js@*/

/*---
# Optional metadata is encoded in JSON or YAML.
slow: false
---*/

/*===
hello world
===*/

if (1) {
    print("hello world");   /* automatic newline */
} else {
    print("not quite");
}

/*===
second test
===*/

/* there can be multiple "expected" blocks (but only one metadata block) */
print("second test");

/* Shorthand can also be used. */
print("shorthand");  //>shorthand

ECMAScript testcase metadata keys

Metadata keys are added and removed as necessary so this list may be out-of-date; see util/runtest.py for current keys. All keys are optional:

---

Key Description

---

comment Optional string to comment on the testcase briefly.

slow If true, test is (very) slow and increased time limits may be necessary to avoid test timeouts.

skip If true, test is skipped without causing a test failure. Useful for unfinished tests and tests that need to be executed manually.

custom If true, some implementation dependent behavior is expected and comparison to other ECMAScript engines is not relevant. The behavior may either be entirely Duktape specific (e.g. relying on JX format) or specific behavior not required by the ECMAScript specification (e.g. additional enumeration guarantees).

nonstandard If true, expected behavior is not standards compliant but matches "real world" expectations.

endianness If set, indicates that the testcase requires a specific endianness, needed for e.g. some TypedArray testcases. Values: little, big, mixed.

use_strict Testcase is a strict mode program. When preparing the test, prepend a "use strict"; declaration as very first statement of the test, before the test prologue.

intended_uncaught Testcase intentionally fails by throwing an uncaught error (which may even be a SyntaxError). This is needed to test some program level behavior.

ECMAScript testcase known issues

Sometimes testcases fail due to known bugs or environment specific differences such as endianness. Known issue files describe the "known bad" testcase output and describes the reason for the failure. This allows a failing test to be flagged as a "known issue" rather than a failure.

Known issue files have a YAML metadata block, followed by ---, followed by the "known bad" verbatim testcase output:

summary: wurld is printed instead of world
---
hello wurld

The "known bad" output can also be provided as an MD5 hash which is useful if the full output is very large and uninteresting:

summary: wurld is printed instead of world
md5: 49a9895803ec23a6b41dd346c32203b7

Each known issue file describes a single known failure for a specific testcase. A certain testcase may have several known issue files, for different Duktape versions, different config options, different environments, etc. The current naming convention is just a numbered sequence based on the testcase name:

# For test-dev-hello-world.js:
test-dev-hello-world-1.txt
test-dev-hello-world-2.txt
test-dev-hello-world-3.txt
...

ECMAScript testcase best practices

Indentation

Indent with 4 spaces, no tabs.

Verifying exception type

Since ECMAScript doesn't require specific error messages for errors thrown, the messages should not be inspected or printed out in test cases. ECMAScript does require specific error types though (such as TypeError. These can be verified by printing the name property of an error object.

For instance:

try {
    null.foo = 1;
} catch (e) {
    print(e.name);
}

prints:

TypeError

When an error is not supposed to occur in a successful test run, the exception message can (and should) be printed, as it makes it easier to resolve a failing testcase. This can be done most easily as:

try {
    null.foo = 1;
} catch (e) {
    print(e.stack || e);
}

This is portable and prints a stack trace when available.

Printing tracebacks, pointers, etc

While it should be generally avoided, in some testcases it's necessary to print out tracebacks, JX-serialize pointers, etc. When doing so:

• Replace filenames and line numbers in tracebacks with e.g. FILE:LINE. Otherwise the test output will include temporary file names and it won't be possible to describe a stable expected output.
• Replace pointers with e.g. PTR. Pointer format is platform dependent and can include 0x12345678, 0x123456789abcdef, and 12345678.

There are utility includes to perform these replacements.

API testcase format

Testcase files are C files with a test() function. The test function gets as its argument an already initialized duk_context * and print out text to stdout. The testcase can assume duktape.h and common headers like stdio.h have been included. There are also some predefined macros (like TEST_SAFE_CALL() and TEST_PCALL()) to minimize duplication in testcase code.

Expected output and metadata is defined as for ECMAScript testcases. However, the expected output shorthand syntax (//>output) cannot be used because it's not portable C89.

Example:

/*===
Hello world from ECMAScript!
Hello world from C!
===*/

void test(duk_context *ctx) {
    duk_push_string("print('Hello world from ECMAScript!');");
    duk_eval(ctx);
    printf("Hello world from C!\n");
}

API testcase known issues

As for ECMAScript testcases, known issues are documented using known issue files providing the "known bad" output. The format is the same as for ECMAScript tests.

Test tools

• util/runtest.py: prepares and executes a single testcase, and prints out a readable result summary. Optionally writes JSON test result file, prepared testcase, and various other outputs to specified files. The tool can also be used to just prepare a test. The runtest.py tool can be used both manually and as part of running a test suite.
• util/prep_test.py: earlier version of runtest.py, used by runtests.js and likely be to deprecated.
• runtests/runtests.js: original Node.js based test runner which is likely to be rewritten as a Python program.
• testrunner/: distributed test runner jobs for GitHub commit/pull webhook tests. The testrunner client/server code is in its own repo https://github.com/svaarala/duktape-testrunner.

Future work

• Put testcases in a directory hierarchy instead (test/stmt/trycatch.js), perhaps scales better (at the expense of adding hassle to e.g. grepping).