The project aims at investigating the following web services:
Web Service 1: implementations of several reputation systems, so instructors can feed their rating data to the web service, and have it return the scores that any particular reputation system would have assigned to each student. This will facilitate research in reputation systems, or in combining reputation and calibration to improve the quality of assessment. It is also important to be able to visualize the calibration and reputation results, and to visualize them across time, so that an instructor can see how many students are having trouble, and how many have improved from earlier in the course.
Web Service 2: a real-time visual representation or “heat map” of the performance of students in the class over time, taking into account students’ performance on assignments, and performance as reviewers.
Web Service 3 is an automated meta-review module that computes a set of metrics on the reviews, and gives feedback to the reviewer on how to improve the review. Calibration, reputation systems, and meta-reviewing are three techniques for improving the quality of reviewing and evaluation. Our web services make it easy for researchers to experiment with them, singly and in combination with each other.
Web Service 4 will implement different reviewer-assignment algorithms. Reviewer-competency information can be computed by, or passed to, the web service. Each time the review system needs to assign another reviewer, it will query the web service, which will run the selected algorithm and tell the review system which reviewer to assign. This will allow instructors to experiment with different strategies for reviewer assignment, and will give the project team the data they need to design better approaches for assigning reviewers.
Web service 5 will provide the ability for reviewers to annotate document types such as wikis and Google docs, and interact with the author and other reviewers about the annotations.
Web service 6 helps the instructor improve the reliability of rubric criteria. It keeps track of scores assigned on each rubric criterion, and shows the instructor which have the greatest reliability between raters (“inter-rater reliability”). It also puts the criteria and their reliabilities into a single database so that the system can “learn” characteristics of a reliable criterion.
Web service 7 allows us to produce and experiment with summarizations and visualizations of peer text, peer ratings, and peer rankings built for both formative and evaluative purposes.