ETAPS Artifact Badges

A badge in the “Artifacts Evaluated” category is applied to papers whose associated artifacts have successfully passed the AE. At most one of the following two badges can be awarded to any given paper, with the “Reusable” badge subsuming the “Functional” one:

Functional

The artifacts associated with the paper are documented, consistent, complete, exercisable, and include appropriate evidence of verification and validation. They have been made available to the AE committee with permissions/under a license allowing their evaluation.

• Documented: An inventory of the artifacts and a sufficient description to enable the artifacts to be exercised are included.
• Consistent: The artifacts are relevant to the associated paper, and significantly contribute to the generation of its main results.
• Complete: As far as possible and reasonable, all components relevant to the associated paper are included.
• Exercisable: The included software (including scripts) used to generate the results in the associated paper can be successfully executed, and included data can be accessed and appropriately manipulated.

Reusable

The artifacts associated with the paper are well-suited for reuse and repurposing beyond their concrete purpose in relation to the associated paper. That is, they

• have all the qualities of the “Functional” badge,
• are very carefully documented and well-structured to the extent that reuse and repurposing is facilitated, and
• strictly adhere to norms and standards of the research community for artifacts of this type.

The single badge in this category is awarded to papers whose associated artifacts are permanently and publicly available, with minimal requirements as to the qualities of the artifacts themselves:

The artifacts associated with the paper have been made permanently available for retrieval on a publicly accessible archival repository which has a declared plan to enable permanent accessibility and assigns DOIs to its entries. The artifacts’ DOIs are referenced in a data-availability statement at the end of the paper.

Artifacts do not need to have received any other badge in order to receive this one. In particular, they need not be functional as long as they are relevant to the paper and add value beyond the paper’s text. Such artifacts could be something as simple as the data from which the figures are drawn, or as complex as a complete software system under study. Even artifacts that (partially) contradict the paper’s results in a relevant way can in principle receive this badge.

Notes: Any repository—be it open, institutional, or commercial—that satisfies the criteria of having a declared plan for permanent accessibility and assigning DOIs is acceptable. ETAPS recommends making artifacts available via the open Zenodo repository.

A badge in the “Results Validated” category is applied to papers whose main results have been successfully obtained by a person or team other than the authors, with the requirements that

• the independently obtained results and those of the paper must be in agreement to a tolerance deemed acceptable for experiments of the given type, and
• differences in the results should not change the main claims made in the paper.

Results Reproduced

The main results reported in the paper have been obtained in a subsequent study by a person or team other than the authors, using (in part) artifacts provided by the authors.

Results Replicated

The main results reported in the paper have been independently obtained in a subsequent study by a person or team other than the authors, without the use of authors-supplied artifacts.

Notes: A standard AE procedure cannot result in the “Replicated” badge being awarded because the AE, by its nature, uses the artifacts supplied by the authors. We recommend AE committees to exercise care when setting up an evaluation procedure that is intended to result in “Reproduced” badges being awarded: the requirement of “a subsequent study” may go beyond the scope of the time-limited AE process.