Effective Practices of Artifact Evaluation for TACAS

The following "effective practices" are not hard requirements, but according to experience from the community, they should be followed as closely as possible.

Artifact Archiving

The following requirements were developed by the ACM Task Force on Data, Software, and Reproducibility in Publication [3, 6]:

Paper Pictograms/Badges and Data Availability Statement

Papers should indicate the availability of data and the result of the artifact-evaluation process. The badge on the first page of the paper (and ideally the metadata in the digital library) indicates the availability of an artifact. The explicit Data Availability Statement makes it easy to identify the artifact.

The ACM, together with a broad research community, has developed standard paper badges and a policy [2]. The notions were inspired by the International Vocabulary of Metrology [1] and further refined and developed by a broad NISO working group [7]. Note that there was a change of definitions in order to make notions more compatible [8].

The pictograms can be used also by conferences that are not sponsored by ACM [5].

There are three categories of paper badges:

A section in the paper describes the availability of data and software. (For grant proposals, it is already a standard practice to provide such a declaration.)

Deadlines

There are four important deadlines:

Checklist for Camera-Ready Papers

The AEC/PC chairs should

Do not strictly enforce, but a mild pushing and pointing authors to the literature is helpful to convince unexperienced artifact authors.

References

Author: Dirk Beyer - please contact me in case of questions.