CHAIR HOW-TO: TPMS

OVERVIEW

The Toronto Paper Matching System is a service to match papers with reviewers, developed by researchers at the University of Toronto.
Email 'reviewer-paper-matching@cs.toronto.edu' to obtain a chair account to use this service.

Chairs can also ask their reviewers to register their reviewer email accounts at TPMS and follow TPMS instruction to update their profiles there.

It is advised to enable TPMS before you enable submissions as authors need to agree to the terms of TPMS. Once TPMS is enabled, the TPMS agreement checkbox appears on the Submission page when the authors upload their papers. TPMS must be enabled for each track that uses TPMS scores for paper assignments. TPMS scores are between 0 and 1. A higher score indicates a better match between a paper and a reviewer.

Papers must be in pdf format for TPMS matching.

Large conferences could use TPMS scores to prepare a suggested list of reviewers for each paper. Some conferences use a large number of suggestions for each paper, e.g. 1000. Others may use less than 100. Small conferences can use TPMS scores directly when automatically assigning reviewers and meta-reviewers.

ENABLING TPMS

• In the Settings tab, select Features>TPMS. Check the ‘Enable TPMS’ checkbox.

• Enter your credentials and click ‘Submit.’

• From the ’Deposit Authors’ dropdown, select Only Primary Contact Author, then click ‘Save changes.’

AUTHOR AGREEMENT

• Once TPMS is enabled for all tracks, Authors will then need to agree to the use of TPMS for paper matching. This is done by checking the required checkbox in the TPMS agreement section on the submission form.

USING TPMS

TPMS workflow consists of the following.

• Send Papers
• Submit Matching Request
• Retrieve Scores
• Download Scores

TPMS workflow pages can be accessed from the Chair Console.

• Click ‘Actions’ then select “TPMS>Send Papers”.

Send Papers

Important

While sending papers to TPMS, Chairs should not change any paper's status.

There are 2 tabs in the Send Papers section of TPMS: ‘Send All Papers’ and ‘Send One Paper’. The ‘Send One Paper’ tab is used to resend a paper that may have encountered an issue during the initial sending of all papers. The issue could be due to transient connection errors or the file being deleted while the transfer was in progress. It is also possible that the paper was updated after the initial file was sent to TPMS.

In a multi-track conference, you may select one or more tracks to send the papers to TPMS.

In the scenario below, we will send 4 ready papers from Track 1 to TPMS.

• Click ‘Send’ to send all the papers.

You will see the progress of the papers sent to TPMS.

• Click ok when it reaches 100%.

 

Submit Matching Request

This will ask TPMS to match the Reviewers and/or Meta-Reviewers to the papers that were sent.

• Select Track, Reviewers and/or Meta-Reviewers, then click ‘Submit’.

Retrieve Scores

It may take some time for TPMS to process a matching request. The more papers and reviewers, the longer it takes for scores to be ready for retrieval.

• Click the Retrieve Scores tab, then click ‘Retrieve Scores’ link. After scores are retrieved, the TPMS ranks that were derived from the scores will automatically show in a column on various pages.

Note

In a multi-track conference, there will be a 'Track' column between 'Created (Pacific Time)' and 'Reviewers Selected.'

Download Scores

This will give scores in a CSV file for offline use. The scores can be used in various ways such as preparing a suggestion list of papers for reviewers to bid on.

• From the Chair Console, select Actions>TPMS>Download Scores. Select to download scores for reviewers or meta-reviewers.

• Save the file locally.

• When you open the CSV file with Excel you see the Paper ID, the Email of the reviewer and the TPMS score.

VARIOUS PAGES WHERE TPMS SCORES ARE USED

Reviewer Console – Bidding

The TPMS rank column will now show in the Reviewer Console to aid in placing their bids on papers. The TPMS rank of 1 has the highest matching score.

Meta-Reviewer Console – Bidding

The TPMS rank column will now show in the Meta-Reviewer Console to aid in placing their bids on papers. The TPMS rank of 1 has the highest matching score.

Edit Suggestions page

The TPMS rank column will now show in the ‘Edit Suggestions’ page to aid in suggesting reviewers for papers. The TPMS rank of 1 has the highest matching score.

Edit Reviewer Assignments page

• From the ‘Manage Reviewers’ page, scroll to the right, click ‘more’ at the end of the Row of a Reviewer, then select ‘Edit Assignments’.

The ‘Edit Reviewer Assignments’ page shows the TPMS column with ranking. The TPMS rank of 1 has the highest matching score.

Edit Meta-Reviewer Assignments page

• From the ‘Manage Meta-Reviewers’ page, scroll to the right, click ‘more’ at the end of the Row of a Meta-Reviewer, then select ‘Edit Assignments’.

The ‘Edit Meta-Reviewer Assignments’ page shows the TPMS column with ranking. The TPMS rank of 1 has the highest matching score.

Automatic Assignment—Meta-Reviewer

• From the Chair Console click Actions>Automatic Assignment>Meta-Reviewer

After selecting the Papers and Meta-Reviewers, you will come to the Configure options step.

• Click the ‘Custom’ radio button. The ‘TPMS scores’ field is now present. Here you would add scores (values between 0 and 1) in one or more fields making sure the total equals 1.

Automatic Assignment—Reviewer

• From the Chair Console click Actions>Automatic Assignment>Reviewer

After selecting the Papers and Reviewers, you will come to the Configure options step.

• Click the ‘Custom’ radio button. The ‘TPMS scores’ field is now present. Here you would add scores (values between 0 and 1) in one or more fields making sure the total equals 1.