Artifact Evaluation for TACAS 2024

TACAS 2024 will include an artifact evaluation (AE).

In either case, authors must indicate at paper submission whether an artifact will be submitted. Authors who did not indicate that they intend to submit an artifact will not have their artifact evaluated.

Artifacts and Evaluation Criteria

An artifact is any additional material (software, data sets, machine-checkable proofs, etc.) that substantiates claims made in the paper and ideally renders them fully replicable. For example, an artifact might consist of a tool and its documentation, input files used for tool evaluation in the paper, and configuration files or documents describing parameters used in the experiments. The Artifact Evaluation Committee (AEC) will read the corresponding paper and evaluate the artifact according to the following criteria:

The evaluation will be based on the EAPLS guidelines, and the AEC will decide which of the badges — among Functional, Reusable, and Available — will be assigned to a given artifact and added to the title page of the paper in case of acceptance.

Compulsory AE for Tool and Tool Demonstration Papers

Regular tool papers and tool demonstration papers are required to submit an artifact for evaluation by October 26th, 2023. These artifacts will be expected to satisfy the requirements for the "Functional" and "Available" badges. Results of AE will be taken into consideration in the paper reviewing and rebuttal phase of TACAS 2024. Exemption from some aspects of AE is possible in exceptional circumstances, see Exemption.

Optional AE for Research and Case Study Papers

Authors of research and case study papers are invited to submit an artifact no later than December 28th 2023, anywhere on Earth. Artifact submission is optional, and the artifact will be reviewed only after paper acceptance.

Artifact Submission

Artifact submission is handled via EasyChair. After the paper submission deadline, the paper submission will be copied by us from the main track to the artifact evaluation track. In this track, you should supply your submission's artifact abstract and upload a ZIP archive as supplementary material. Do not attempt to create a new submission, nor change your submission's other details such as authors or title. Because of this copying process, you can upload or update your artefact either in the main track before the paper submission deadline, or in the AE track during a window before the deadline. The TACAS PC or AEC chairs will inform authors when the AE track is open for different paper categories.

An artifact submission must contain:

The artifact hyperlink should point to a self-contained archive that allows the AEC to evaluate the artifact on the TACAS virtual machine. Authors should test their artifact on TACAS-23 virtual machine prior to the submission and include all relevant instructions. Instructions should be clear and specify every step required to build and run the artifact, assuming no specific knowledge and including steps that the authors might consider trivial.

Guidelines for Artifacts

We expect artifact authors to package their artifact and write instructions such that AEC members can evaluate the artifact using the TACAS 2023 Artifact Evaluation Virtual Machine (hereafter VM) for VirtualBox. Note that this is not a misprint: we use the TACAS 2023 VM as the AEC chairs consider it serviceable for TACAS 2024. The virtual machine is based on an Ubuntu 22.04 LTS image with the following additional packages: build-essential, cmake, clang, mono-complete, openjdk-8-jdk, python3.10, pip3, ruby, and a 32-bit libc. VirtualBox guest additions are installed on the VM; it is therefore possible to connect a shared folder from the host computer.

This year, the TACAS AEC chairs have decided to allow reasonable network access from the VM, following a wider trend for software builds requiring network access. Authors can therefore decide whether to supply a completely self-contained artifact, rely on network access for external resources, or combine both. However, the main artifact described in the paper must be included in the archive, and use of external resources is at the author's own risk. We anticipate typical usage of this relaxation to be for the installation of third-party dependencies. The AEC chairs welcome feedback on this aspect, in particular questions about atypical artifacts. All the same, if the artifact requires additional software or libraries that are not part of the virtual machine, the instructions must include all necessary steps for their installation and setup on a "clean" VM. In particular, authors should not rely on instructions provided by external tools.

It is to the advantage of authors to prepare an artifact that is easy to evaluate by the AEC. Some guidelines:

Members of the AEC will use the submitted artifact for the sole purpose of AE. We do, however, encourage authors to make their artifacts publicly and permanently available.

Please note that the reviewers will only have limited time to reproduce the experiments and they will likely use a machine that is different to yours. Again, if your experiments need a significant amount of time (more than a few hours), please prepare a representative subset of experiments that could be run in a shorter amount of time. It may be wise to test your artifact in the virtual machine running on other platforms available to you.

Exemption

Under particular conditions tool papers and tool demonstration papers may be exempted from submitting the artifact, using the provided VM, or acquiring both the "Functional" and "Available" badges. Possible reasons for such exemptions include the need for special hardware (GPUs, compute clusters, robots, etc.), extreme resource requirements, or licensing issues. Artifacts should nonetheless be as complete as possible. Contact the AEC chairs as soon as possible if you anticipate the need for exemption.

Rebuttal

There will be no formal rebuttal procedure for AE. However, there will be an early light review of artifacts to ensure basic functionality: in case of technical issues at this stage and at the discretion of the AEC chairs, a single round of rebuttal may be applied to some artifacts shortly after submission. A single set of questions from reviewers will be put to artifact authors, who may reply with a single set of answers to address issues and facilitate further evaluation. Update of the submitted files or further communication will not be allowed.

Artifact Evaluation Chairs

Artifact Evaluation Committee