Criteria for the evaluation of TACAS 2012 tool papers

Tool papers are often hard to evaluate. Thus, in order to increase the level of consistency of the evaluation process, a set of common criteria for the evaluation of tool papers has been prepared for the members of the TACAS Program Committee. In order to ensure fairness of the process, the criteria are here listed for potential authors of tool papers.

First, we strongly suggest to the authors to make their tools available via the web, even if only for the evaluation process. The reviewers are asked to visit the tool web page, to download the tool and to make an effort to install it and to try some examples, which were provided by the tool developers. The reviewers are required to state in their report which experiences they made.

The reviewers are asked to comment on the following aspects:

• degree of maturity of the tool as a software product (e.g implementation, documentation, development process)
• the user basis (e.g. whether the tool is widely used or simply used by the developers)
• its availability (e.g. source vs binary, licensing restrictions)
• robustness (e.g. whether it runs on three examples vs it is integrated within some industrial development flow)

The tool papers we are accepting as submissions fall into two categories (see also the Call for Papers on the TACAS 2012 web page):

Regular tool papers present a new tool, a new tool component, or novel extensions to an existing tool. They focus primarily on engineering aspects, with special emphasis on important design and implementation concerns. A thorough discussion of theoretical foundations is not required, although the paper should provide a summary of such, with relevant citations. A tool paper should describe the tool's software architecture and core data structures and algorithms, and also give a clear account of its functionality. The paper should discuss the tool's practical capabilities with reference to the type and size of problems it can handle, and experience with realistic case studies. Papers that present extensions to existing tools should clearly focus on the improvements or extensions with respect to previously published versions of the tool, preferably substantiated by data on enhancements in terms of resources and capabilities. Tool papers are evaluated by the TACAS Tool Chair with the help of the Programme Committee. Tool papers have a maximum of 15 pages.

Tool demonstration papers present tools based on aforementioned technologies (e.g., theorem-proving, model-checking, static analysis, or other formal methods) or falling into relevant application areas (e.g., system construction and transformation, testing, analysis of real-time, hybrid or biological systems, etc.) and focus on the usage aspects of the tool. Tool demonstration papers are evaluated by the TACAS Tool Chair with the help of the Programme Committee. Tools presented in tool demonstration papers must be publicly available. Tool demo papers have a maximum of 6 pages.
For tool demonstration papers it is not required to give an overview of the theoretical foundations or to present experimental evaluation. However, we urge authors to motivate why their tool is interesting and significant.