CAMEO | Continuous Automated Model EvaluatiOn

The goal of CAMEO is to continuously evaluate the accuracy and reliability of various protein structure prediction services in a fully automated manner

CAMEO supports users of structure prediction services:

• Retrospective evaluation of prediction accuracy allows to select the most suitable tool for a given modeling problem.
• CAMEO provides continuous real-time data on the reliability of different prediction services, up-time and responsiveness.

CAMEO supports developers of structure prediction services:

• CAMEO provides continuous real-time data on the performance of new methods. Thus, direct comparison with other state-of-the-art methods helps debugging and benchmarking of new developments on a large number of evaluated prediction targets to provide statistically significant results on the same targets at the same point in time.
• CAMEO evaluates pre-diction accuracy not "post-dictions". Hence, CAMEO data can be also used in publications to demonstrate a method’s performance in an independent blind direct comparative benchmark to document the performance of new algorithms.
• CAMEO monitors a server's technical reliability, up-time and responsiveness, and can notify the server developer in case of unexpected errors due to technical problems (e.g. network problems or server configuration issues).
• CAMEO monitors a server's prediction accuracy in comparison to other methods, and notifies the server developer in case of unexpected prediction failures, e.g. due to outdated template libraries or bugs in the prediction pipelines.

CAMEO is a community project:

• CAMEO applies quality assessment criteria established by the protein structure prediction community. Since the accuracy requirements for different scientific applications vary, there is no "one fits all" score. CAMEO therefore offers a variety of scores - assessing different aspects of a prediction (coverage, local accuracy, completeness, etc.) to reflect these requirements.
• CAMEO is a community project - please feel free to suggest additional/alternative ways how CAMEO can support users and developers of structure prediction.

Join CAMEO today...

We invite developers of prediction methods to participate by registering their servers (REGISTER). We also invite developers of scoring and evaluation methods to suggest alternative scoring schemes. Please contact us directly.
Servers of the following groups are registered so far:
Schwede Lab, Sali Lab, Research Group McGuffin, Soeding Lab, Baker Lab, Fiser Lab, the Sternberg lab and the Zhang Lab.

CAMEO is developed by the Computational Structural Biology Group at the Swiss Institute of Bioinformatics (SIB) and the Biozentrum of the University of Basel. CAMEO is partially supported by the SIB Swiss Institute of Bioinformatics and by grant U01 GM093324-01 from the National Institute of General Medical Sciences.