Welcome! EvaluatIR.org aims to be a valuable resource for researchers in the field of Information Retrieval, providing an online reference and analytical tool for retrieval effectiveness results. IR Test Collections developed through efforts such as the Text REtrieval Conference (TREC) have enabled large scale empirical evaluation of IR systems. however finding current and comparable benchmarks for these collections is often difficult, especially when many results are only reported in summarised form in journal or conference papers. A small team at the University of Melbourne has been developing this site to address this issue.

With EvaluatIR.org you can:

• See what the state of the art in retrieval access is, using our database of evaluation results. It includes past retrieval runs submitted to TREC, benchmarks of out-of-the box IR systems and runs published by other researchers. Information about how the results were obtained will be collected and documented wherever possible.
• Privately upload your own runs in TREC runfile format.
• Compare any results in our database with each other or your private uploaded runs - in a range of different ways. A range of different evaluation metrics are available , and results can be compared graphically query-by-query, using statistical significance tests, ranked against other runs, using score standardisation, and more.
• Share your uploaded runs with other researchers, and link to back to relevant publications.

Feel free to browse through the publicly accessible portions of our site: you can look at rankings and overall scores of runs for all test collections as well as use our search. Please fill out our very short signup form to be granted access to the rest of the site (and there is a lot more: all of the analytical features, as well as run uploading is available to registered users only).

Please keep in mind that the site is very much as work in progress, there will be glitches and missing features.

Further links:
EvaluatIR: An Online Tool for Evaluating and Comparing IR Systems on William Webber's blog.
The 32nd Annual ACM SIGIR Conference, where we will be presenting a demo of the system.

---