Abstract Motivation: Genome-wide functional annotation either by manual or automatic means has raised considerable concerns regarding the accuracy of assignments and the reproducibility of methodologies. In addition, a performance evaluation of automated systems that attempt to tackle sequence analyses rapidly and reproducibly is generally missing. In order to quantify the accuracy and reproducibility of function assignments on a genome-wide scale, we have re-annotated the entire genome sequence of Chlamydia trachomatis (serovar D), in a collaborative manner. Results: We have encoded all annotations in a structured format to allow further comparison and data exchange and have used a scale that records the different levels of potential annotation errors according to their propensity to propagate in the database due to transitive function assignments. We conclude that genome annotation may entail a considerable amount of errors, ranging from simple typographical errors to complex sequence analysis problems. The most surprising result of this comparative study is that automatic systems might perform as well as the teams of experts annotating genome sequences. Availability and supplementary information: http://www.ebi.ac.uk/research/cgg/annotation/cteval/ Contact: [email protected] * To whom correspondence should be addressed. † INA-EKETA, GR-57001 Thessaloniki, Greece ‡ Computational Biology Center, Memorial Sloan-Kettering Cancer Center, New York, NY 10021, USA § Aetion Technologies LLC, Worthington, OH 43085, USA ¶ Institut Curie, F-75248 Paris, France ∥ CNRS, UMR6543, F-06108 Nice, France ** Alma Bioinformatics, E-28760 Madrid, Spain †† Cap Gemini Ernst & Young, London SW1X 7LX, UK ‡‡ Univ. of Rome ‘La Sapienza’, I-00185 Rome, Italy §§ MWG-Biotech AG, Ebersberg, D-85560 Berlin, Germany ¶¶ Wellcome Trust Biocentre, Univ. of Dundee, Dundee DD1 5HN, UK