PeerLogic is an NSF-funded research project that provides services for educational peer-review (aka. peer-assessment) systems. The project has two main parts.
- A suite of web services that provide functionality useful to peer-assessment systems, and can be used to the extent desired by the developers of those systems.
- A data warehouse, an open repository of anonymized data from millions of reviews performed in several different peer-assessment systems. This is intended as a resource for researchers in peer assessment that is free of the institutional bias that might be present in a corpus drawn from a single system.
We provide web services that can easily be incorporated into existing peer-review systems. As a point of reference, Google maps is a well known web service. A web developer who wants to embed a map of a business’s location into its web site simply makes a call to Google maps, specifying parameters such as the location of the business, the text for the flag, and the scale of the map to be shown. Google charts is another web service that, given a data set, affords any web site the ability to display a graph of it.
Peer-assessment systems, too, can benefit from web services. Regardless of their design or implementation, they have common needs, like the need to determine the credibility of reviews and the need efficiently to visualize large amounts of summary data on the reviews performed by a particular class.
A web-service approach has three major advantages. It offers an easy way for researchers to replicate each other’s studies: they need not rewrite the code for their system, but just specify the parameters for the web service to use. By aggregating the results from the same experiments on different peer-review systems, researchers can gain the statistical power to reach stronger conclusions about the effects of the interventions on student learning. And finally, it avoids subtle differences in implementation could make research results difficult to compare and slow the technology’s diffusion. By evaluating the services in multiple contexts, we are developing features that will be useful to a broad range of peer-review systems.
A web-services approach is independent of any particular peer-review system. Each web service is available to all peer-review systems. As a synopsis of what can be done, here are some existing and planned web services.
- Reputation algorithms. What constitutes a good peer reviewer? It is typically someone who scores artifacts in a way that corresponds closely with the way an expert scores them. If expert scores are available, we can compare them with a particular reviewer’s score to get a measure of the competency of that reviewer. In the more common case where expert scores are not available, we can compare scores of one reviewer with another. A good reviewer will assign different scores to different work (rather than high-five all the work (s)he is given to review), and assign scores that are not too different from the scores assigned by other reviewers. Many algorithms have been developed for determining reviewer competency. We make several of them available by a web-services call, so they needn’t be implemented in each system.
- Metareview metrics. In addition to determining reviewer competency, we would also like to be able to determine the effectiveness of a review. We can measure the textual feedback of a review in different ways: Is its tone positive or negative? How many suggestions does it contain? Does it use the same words over and over, or did the reviewer give different feedback on different points? Do the words used in the review seem relevant to the content of the artifact?
- Summarization. When an author receives many peer reviews, it is hard to take them all into account. Rather than flip back and forth between the reviews, it would be helpful for an author to see a concise summary of what all of them say. Summarization is a problem that has received a lot of attention from natural language processing (NLP) researchers. We adapt a set of their algorithms to produce summaries of reviews.
- Visualizations. How many students in the class have completed their reviews, and how many reviews were completed each day? What is the distribution of scores given by the students? On which criteria do the students rate work most highly? Most negatively? Which rubric criteria have the best inter-rater reliability? There are any number of statistics that a student or instructor might want to see, and we intend to provide ways for any review system to show them.
To see what is available in any of these areas, click on the icons below.
NC State research team is submitting three new research papers: 1. Improving Formation of Student Teams: a Clustering Approach Shoaib Akbar, Yang Song, Zhewei Hu, Ed Gehringer Today’s courses in engineering and other... READ MORE
The team members manage to get three papers accepted at the 46th Annual Frontiers in Education (FIE) Conference: “Five Years of Extra Credit in a Studio-Based Course: An Effort to Incentivize Socially Useful... READ MORE
The project members from NC State, Ed Gehringer, Ferry Pramudianto and Yang Song organize the 2016 CSPRED workshop, which will be held on Wednesday, June 29, 2016 in conjunction with the 9th International... READ MORE
Dr. Gehringer the lead PI of the project gave a talk on TLT Friday Live on May 13, 2016. He presented interesting insights on How computer-supported grading and reviewing of students’ work can... READ MORE