TACAS Artifact Evaluation

• For regular tool papers and tool demonstration papers, AE is mandatory and artifacts must be submitted by the end of October 29, 2026, “Anywhere on Earth” (AoE, UTC-12).
• For accepted research and case study papers, AE is optional and artifacts may be submitted by the end of January 11, 2027, “Anywhere on Earth” (AoE, UTC-12).

Tool and Tool Demonstration Papers

• Artifact submission deadline: October 29, 2026
• Smoke test notification: November 9, 2026
• Open rebuttal: November 28–December 4, 2026
• Paper and artifact notification: December 22, 2026

Research and Case Study Papers

• Artifact submission deadline: January 11, 2027
• Smoke test notification: TBD
• Open rebuttal: TBD
• Paper and artifact notification: February 11, 2027

Artifacts and Evaluation Criteria

An artifact is any additional material (software, data sets, machine-checkable proofs, etc.) that substantiates claims made in the paper and ideally renders them fully replicable. For example, an artifact might consist of a tool and its documentation, input files used for tool evaluation in the paper, and configuration files or documents describing parameters used in the experiments. The Artifact Evaluation Committee (AEC) will read the corresponding paper and evaluate the artifact according to the following criteria:

• Consistency with and replicability of results presented in the paper
• Completeness
• Documentation and ease of use
• Availability in a permanent online repository (e.g., Zenodo)

The evaluation will be based on the ETAPS Artifact Badge guidelines, and the AEC will decide which of the badges — among Available, Functional, and Reusable — will be assigned to a given artifact and added to the title page of the paper in case of acceptance.

We summarize the main evaluation criteria for each badge in the following.

Available

The artifact is permanently archived with a persistent identifier (DOI) and publicly accessible.

Example: The tool’s source code and its dependencies are publicly available on Zenodo with a permanent DOI (it can also be hosted on e.g. GitHub in addition to the permanently available repository, but only hosting it on GitHub is not sufficient.)

Functional

The artifact works as described and can reproduce all (or almost all) key results from the paper.

Example: A tool submission compiles, runs, and reproduces the results from the paper (e.g. correctly verifies all benchmark models from the result tables).

Reusable

The artifact is well-documented, modular, and easy to adapt for future research.

Example: A tool is accompanied with clear APIs, code documentation, example scripts, and instructions for adding and running new benchmarks (i.e., it can easily be run on additional inputs).

• Regular tool papers and tool demonstration papers are required to submit an artifact for evaluation by October 29, 2026. These artifacts will be expected to satisfy the requirements for the Functional and Available badges. Results of the AE will be taken into consideration in the paper reviewing and rebuttal phase of TACAS 2027. Exemption from some aspects of AE is possible in exceptional circumstances.

• Authors of accepted research and case study papers are invited to submit an artifact no later than January 11, 2027. Artifact submission is optional, and the artifact will be reviewed only after paper acceptance.

An artifact submission must contain:

• the paper submission number as stated in the EasyChair submission (in a designated field).
• the paper as uploaded to the TACAS PC for review in EasyChair.
• a ZIP archive (uploaded to OpenReview) containing:
  • a license document (LICENSE) for the artifact. It is required that the license at least allows the AEC to evaluate the artifact.
  • instructions (README) including:
    • a hyperlink to the artifact
    • additional requirements for the artifact, such as installation of proprietary software or particular hardware resources
    • detailed instruction for an early smoke test review that allows reviewers to: (1) verify that the artifact can properly run; and (2) perform a short evaluation of the artifact before the full evaluation and detect any difficulties (see Smoke Test)
    • detailed instructions for use of the artifact and replication of results in the paper, including estimated resource use and required time if non-trivial
    • list of badges you are applying for in your submission. In particular, if you are not applying for the Reusable badge, please indicate it
    • platforms you tested this on (in particular: the host architecture: arm64 vs amd64, the operating system, etc.)

The artifact hyperlink should point to a self-contained archive that allows the AEC to evaluate the artifact on either the TACAS virtual machine or the Docker image (see below for more information). Authors should test their artifact prior to the submission and include all relevant instructions. The instructions should be clear and specify every step required to build and run the artifact, assuming no specific knowledge and including steps that the authors might consider trivial. Ideally, there will be a single command to build the tool and a single command to run the experiments.

If further clarifications will be required after the deadline (e.g., a requested author response to the smoke test), you will be asked to answer in OpenReview.

TACAS Virtual Machine

We expect artifact authors to package their artifact and write instructions such that AEC members can evaluate the artifact using the TACAS 2027 Artifact Evaluation Virtual Machine (hereafter VM) for VirtualBox. The 2027 VM runs on Ubuntu 26.04 and supports Python, C++, Java, Mono, Ruby, and Rust; and has the following installations: build-essential, git, curl, cmake, clang, mono-complete, default-jdk, python3-full, python3-pip, and ruby. Notice that there are two VM image versions (amd64 and arm64). Furthermore, VirtualBox guest additions are installed on the VM; it is therefore possible to connect a shared folder from the host computer. Artifacts may also be evaluated using the TACAS 2026 VM (https://doi.org/10.5281/zenodo.17171929).

Docker Image

Authors can alternatively provide a Docker image that can be run on the AEC’s machine. Do not assume that all reviewers are familiar with Docker. Please provide explicit instructions on how to run the Docker image, including complete command line arguments.

Network Access

Your artifact is allowed reasonable network access, following a wider trend for software builds requiring network access. Authors can therefore decide whether to (a) supply a completely self-contained artifact, (b) rely on network access for external resources, or (c) combine both. However, the main artifact described in the paper must be included in the archive, and use of external resources is at the author’s own risk. If you are using external resources, please ensure that they are version pinned to ensure long-term replicability. We anticipate typical usage of this relaxation to be for the installation of third-party dependencies. If the artifact requires additional software or libraries that are not part of the virtual machine, the instructions must include all necessary steps for their installation and setup on a “clean” VM. In particular, authors should not rely on instructions provided by external tools. Again, the ideal setup is a single command to install the tool and all dependencies and a single command to run the experiments.

• Your artifact need not be anonymized. In particular, you need not rename your tool in both paper and artifact, even if it yields a potential loss of anonymity. For instance, if you submit a paper/artifact on a new tool called XYZ5 which has a clear connection to a tool called XYZ4 (and the authors of XYZ4 are publicly known) then you should keep calling your tool XYZ5 and do not hide a relationship with XYZ4. Your paper/artifact will not be rejected because of this.
• Keep the evaluation process simple (e.g. through the use of scripts), and provide detailed documentation assuming minimum user expertise.
• Document in detail how to replicate most, or ideally all, of the experimental results of the paper using the artifact.
• State resource requirements and/or the environment in which you successfully tested the artifact.
• For experiments requiring a large amount of resources, we strongly recommend providing a way to replicate a representative subset of the results such that results can be reproduced on various hardware platforms in reasonable time (within about 8 hours on a modern notebook). Do include the full set of experiments for those reviewers with sufficient hardware or time, alongside an explanation on how to reproduce individual results.
• If you use the virtual machine, do not submit a virtual machine; only submit your files, which AEC members will copy into the provided virtual machine.
• Reviewers will likely use a machine that is different to yours. It may be wise to test your artifact in the virtual machine / Docker running on other platforms and other architectures available to you.
• For the Available badge, you must upload your artifact to a permanent repository (e.g. Zenodo, figshare, or Dryad) that provides a Digital Object Identifier (DOI), and use that link in your submission. This excludes other repositories such as an institutional or personal website, source code hosting services, or cloud storage.
• Members of the AEC will use the submitted artifact for the sole purpose of AE. We do, however, encourage authors to make their artifacts publicly and permanently available.

Exemption

Under particular conditions tool papers and tool demonstration papers may be exempted from submitting the artifact, using the provided VM, or acquiring both the Functional and Available badges. Possible reasons for such exemptions include the need for special hardware (GPUs, compute clusters, robots, etc.), extreme resource requirements, or licensing issues. Artifacts should nonetheless be as complete as possible. Contact the AEC chairs as soon as possible if you anticipate the need for exemption.

Smoke Test

Shortly after submission, reviewers run a smoke test to check that each artifact executes and that its basic functionality works. If technical issues arise, the AEC chairs may, at their discretion, send the authors a single round of questions. Authors can respond with clarifications only; no changes to the artifact are accepted at this stage.

Open Rebuttal

Once reviewers have shared their feedback, authors enter an open rebuttal period (see Important Dates) in which they can discuss the review directly with the reviewers. This exchange gives authors the chance to clarify issues and propose minor fixes or configuration changes. Reviewers decide, at their discretion, whether to accept such changes, provided they do not qualitatively alter the evaluation.