Artifact Evaluation for TACAS 2023

TACAS 2023 will include an artifact evaluation (AE). There are two separate deadlines for the artifact submission, depending on the paper category:

Artifacts and Evaluation Criteria

An artifact is any additional material (software, data sets, machine-checkable proofs, etc.) that substantiates the claims made in the paper and ideally makes them fully replicable. As an example, a typical artifact would consist of the tool (in binary or source code form) and its documentation, the input files (e.g., models analyzed or programs verified) used for the tool evaluation in the paper, and a configuration file or document describing the parameters used in the experiments. The Artifact Evaluation Committee will read the corresponding paper and evaluate the submitted artifact w.r.t. the following criteria:

The evaluation will be based on the guidelines at this link, and the AEC will decide which of the badges, among Functional, Reusable, and Available, will be assigned to a given artifact and added to the title page of the paper in case of acceptance.

Compulsory AE for Tool and Tool Demonstration Papers

Regular tool papers and tool demonstration papers are required to submit an artifact for evaluation by November 10, 2022. These artifacts will in general be expected to satisfy the requirements for the "Functional" and "Available" badges. The results of the artifact evaluation will be taken into consideration in the paper reviewing and rebuttal phase of TACAS 2023. The fact that not all experiments may be reproducible (e.g., due to high computational demands) or that the tool cannot be made available (e.g. due to proprietary restrictions) does not mean automatic rejection of the paper. However, the authors have to clarify if any of the above conditions apply when submitting the artifact (or require for an exemption from the AE, see later).

Optional Artifact Evaluation for Accepted Research and Case Study Papers

Authors of accepted research papers and case study papers are also invited to submit an artifact. In this case, the submission is voluntary. Authors of the accepted papers will be invited to submit an artifact two weeks after notification.

Artifact Submission

The artifact submission is handled via EasyChair. Artifacts have to be submitted in the "TACAS 2023 - Artifact Evaluation" track, with title "Artifact for Paper (title of the original paper)", and the same authors as the submitted paper. An artifact submission consists of

Guidelines for Artifacts

We expect artifact submissions to package their artifact and write their instructions such that Artifact Evaluation Committee (AEC) members can evaluate the artifact using the TACAS 2023 Artifact Evaluation Virtual Machine for VirtualBox available via Zenodo.

The virtual machine is based on an Ubuntu 22.04 LTS GNU/Linux operating system with the following additional packages: build-essential, cmake, clang, mono-complete, openjdk-8-jdk, python3.10, pip3, ruby, and a 32-bit libc. Moreover, VirtualBox guest additions are installed on the VM; it is therefore possible to connect a shared folder from the host computer. If the artifact requires additional software or libraries that are not part of the virtual machine, the instructions must include all necessary steps for their installation and setup. Any software that is not already part of the virtual machine must be included in the .zip file. AEC members will not download software or data from external sources, and the artifact must work without a network connection. In case you feel that this VM will not allow an adequate replication of the results in your paper, please contact the AEC chairs prior to artifact submission.

It is to the advantage of authors to prepare an artifact that is easy to evaluate by the AEC. Some guidelines:

Members of the AEC will use the submitted artifact for the sole purpose of artifact evaluation. We do, however, encourage authors to make their artifacts publicly and permanently available.

Please note that the reviewers will only have a limited time to reproduce the experiments and they will likely use a machine that is different from your machine. Thus, again, if your experiments need a significant amount of time (e.g., longer than a few days), please prepare a representative subset of experiments that could be run in a shorter amount of time (ideally, several hours). Lastly, test your virtual machine on other platforms.

Possibility for exemption

Under particular conditions tool papers and tool demonstration papers may be exempted from either submitting the artifact, using the provided virtual machine, or acquiring both the "Functional" and "Available" badges.

Possible reasons for one of such exemptions include the need for special hardware (GPUs, compute clusters, bluetooth devices, robots, etc.), software licensing issues or the need to access the internet. Note that, even if your experiments need special resources, you are encouraged to submit a complete artifact (and also an artifact that even replicates a subset of the experiments). This way, the reviewers have the option to replicate the experiments in the event they have access to the required resources.

Important Dates

All dates refer to 23:59 "anywhere on Earth" (UTC-12) on that day.

Deadlines for tool papers and tool demonstration papers:

Deadlines for accepted regular and case study papers:

Artifact Evaluation Chairs

Artifact Evaluation Committee