Validation setup

Generalities

Setup files

Like profiles, setup configurations have nodes to describe different steps of the process. These nodes are splitted into subnodes to describe the course of the run.

The validation configuration is specified using setup files. These files can be in the yml format, or be an executable files generating a yml configuration in stdout. The informations of this configuration are crossed with the profile informations to

When PCVS is launched in a directory, it browses every subdirectory to find any pcvs.yml or pcvs.setup file and launches itself with the corresponding configuration.

example

exampletree/
├── subdir1
│   └── pcvs.yml
└── subdir2
    └── pcvs.yml

Launching `pcvs run exampletree will generate tests for subdir1/pcvs.yml and for subdir2/pcvs.yml. There is no need to put a setup configuration in the root of exampletree, but it is possible to add a setup here.

Structure

The yml input must have one node per test. Each test can describe the following configurations :

  • build

  • run

  • validate

  • group

  • tag

  • artifact

Build

The build node describe how a binary file should be built depending on its sources. It contains the following subndoes :

build:
    files: path/to/the/file/to/build
    sources:
        binary: name of the binary to be built (if necessary)
    depends_on: ["list of test names it depends on"]

    cflags: extra cflags
    ldflags: extra ldflags
    cwd: directory where the binary should be built
    variants: [list of variants (CF Configuration basic blocks -> compiler
    node)]

    autotools:
        params: [list of options for autotools]
    cmake:
        params: [list of options for cmake]
    make:
        target: target for make command

Run

The run node describes how a binary file should be launched. It contains the following nodes :

run:
    cwd: path to build directory
    depends_on:
        test: [list of tests on which it depends]
        spack: [list of spack dependencies used by this test]
        module: [list of installed modules this test needs]
    program: name of the binary file

The run node owns the iterate subnode which can contain custom iterators desribed in the criterion node in the selected profile. Moreover, the run.iterate node can define custom iterators without defining them in criterion by writing them in the run.iterate.program node.

run:
    iterate:
        iterator_described_in_profile.runtime.criterion:
            values: [list of values for the corresponding iterator]
        program:
            custom_iterator:
                numeric: true/false
                type: "argument" or "environment"
                values: [list of values taken by the iterator]
                subtitle: string chosen to identify this iterator

Validate

The validate node describes the expected test behaviour, including exit, time and matching output.

validate:
    expect_exit: expected exit code (integer)
    time:
        mean: expected time to compute the test (seconds / float) tolerance:
        standard deviation for expected time (seconds / float)
        kill_after: maximum time after which process has to be killed
        (seconds / float)
    match:
        label:
            expr:
            expect:
    script:
        path: Path to a validating script

Group

Groups are described in profiles. They can contain build, run, tag, validate, and artifact subnodes. Once a group is defined in the used profile it can be called in the validation setup file.

group: name of the group defined in the profile

Tag

Tags get in the results and tests can be sorted tag-wise. A test can have multiple tags and tags do not have to be defined upstream.

tag:
    - tag1
    - tag2

Artifact

The artifact node contains anything the output should have in addition to the results of tests.

artifact:
    obj1: "path/to/obj1"
    obj2: "path/to/obj2"