Effective Practices of Artifact Evaluation for TACAS
The following "effective practices" are not hard requirements, but according to experience from the community, they should be followed as closely as possible.
Artifact Archiving
The following requirements were developed by the ACM Task Force on Data, Software, and Reproducibility in Publication [3, 6]:
- Identification: Using DOIs to identify published objects is standard. The DOI should point to the version of the artifact that was used to obtain the results in the paper. Many researchers use Zenodo to obtain a DOI. Second-best: Where DOIs are not possible, hashes (such as SHA-256) could be used.
- Long-Term Availability: It is necessary that the artifacts are archived in an archive that hosts the artifacts on a long-term basis, such as Zenodo, etc. [4] (Version repositories do not fulfill this requirement, as the hosting company could decide at any point in time to discontinue the service. Example: Google Code)
- Immutability: It is necessary that the artifact cannot be changed after publication, because the reader needs to use the material exactly as the authors did to obtain their result.
- License: Readers must be able to find out how they can reuse the artifact.
Paper Pictograms/Badges and Data-Availability Statement
Papers should indicate the availability of data and the result of the artifact-evaluation process. The badge on the first page of the paper (and ideally the metadata in the digital library) indicates the availability of an artifact. The explicit Data Availability Statement makes it easy to identify the artifact.
The ACM, together with a broad research community, has developed standard paper badges and a policy [2]. The notions were inspired by the International Vocabulary of Metrology [1] and further refined and developed by a broad NISO working group [7]. Note that there was a change of definitions in order to make notions more compatible [8].
The pictograms can be used also by conferences that are not sponsored by ACM [5].
There are three categories of paper badges:
- Availability of Artifacts (green badge): The artifact is available for others to validate the results in the article. The hosting digital library provides a DOI to identify and cite the artifact, ensures long-term availability and immutability.
- Evaluation of Artifacts (red badges): The artifact was evaluated by an artifact-evaluation committee. There are two badges "Functional" and "Reusable" that can be assigned. The conditions for the "Reusable" badge imply the conditions for the "Functional" badge. These badges target the goal of repeatability.
- Evaluation of Results in Article (blue badges): The results of the article were validated by researchers that are independent from the authors. The badge "Results Validated --- Reproduced" is assigned if the results from the article were sucessfully reproduced using the artifact from the authors, and the badge "Results Validated --- Replicated" is assigned if the experiments from the article were sucessfully replicated without the artifact from the authors.
A section in the paper describes the availability of data and software. (For grant proposals, it is already a standard practice to provide such a declaration.)
- Section "Data-Availability Statement": The article contains a section before the references
that makes a declaration of availability of data and software.
If data cannot be made available, this section explains how the data and software
necessary to repeat the experiments can be obtained.
In particular, the authors should describe what should be used for reproduction (pointing to a specific version of an archive, e.g., at Zenodo) and what for reuse (a URL of, e.g., a GitLab or project home page).
Deadlines
There are four important deadlines:
- artifact-submission deadline (authors submit a DOI that points to the artifact to be evaluated)
- notification deadline (chairs send results to the authors)
- artifact-publication deadline for revised versions (authors might want (or are required) to revise the artifact)
- delivery of final meta data and DOIs to an artifact repository (chairs make the relations between article and artifact available)
Checklist for Camera-Ready Papers
The AEC/PC chairs should
- check that badge is present on paper (if granted)
- check that a Data Availability Statement is present after conclusion and before references
- check that a DOI is assigned to the artifact and used in the paper and it points to a specific version
- check that the artifact with DOI is contained in the references of the paper (important to have it in the CrossRef data)
- (check that DOIs are provided for all references where possible)
- compile a table with article-artifact relationships (to extend the TACAS List of Artifacts)
Do not strictly enforce, but a mild pushing and pointing authors to the literature is helpful to convince unexperienced artifact authors.
References
- [1] JCGM Working Group 2. International Vocabulary of Metrology – Basic and General Concepts and Associated Terms (VIM). 3rd edition, JCGM 200:2012, BIPM, 2012. http://www.bipm.org/en/publications/guides/vim.html
- [2] ACM. Policy on Artifact Review and Badging. ACM, 2018. https://www.acm.org/publications/policies/artifact-review-and-badging-current
- [3] Simon Adar, Dirk Beyer, Patricia Cruse, Gustavo Durand, Wayne Graves, Christopher Heid, Lundon Holmes, Chuck Koscher, Meredith Morovatis, Joshua Pyle, Bernard Rous, Wes Royer, and Dan Valen. Best Practices on Artifact Integration. ACM Task Force on Data, Software, and Reproducibility in Publication. December 2017. https://www.sosy-lab.org/research/prs/2017-12-07_ACM-WS-Repro_BestPracticesOnArtifactIntegration_Dirk.pdf
- [4] Christian S. Collberg and Todd A. Proebsting. 2016. Repeatability in computer systems research. Commun. ACM 59, 3 (2016), 62–69. https://doi.org/10.1145/2812803
- [5] ACM. The Blue Diamond: ACM's Publications Newsletter. ACM, May 2021. https://www.acm.org/articles/pubs-newsletter/2021/blue-diamond-may-2021#Badging
- [6] Bernard Rous. The ACM Task Force on Data, Software, and Reproducibility in Publication. December 2017. https://www.acm.org/publications/task-force-on-data-software-and-reproducibility
- [7] NISO. Reproducibility Badging and Definitions: A Recommended Practice of the National Information Standards Organization. NISO RP-31-2021, January 2021. https://doi.org/10.3789/niso-rp-31-2021
- [8] Heroux, Michael A., Lorena A. Barba, Victoria Stodden, and Michaela Taufer. Toward a Compatible Reproducibility Taxonomy for Computational and Computing Sciences. Sandia Report SAND2018-11186, October 2018. https://doi.org/10.2172/1481626