FAQ-Chair: Manage Reviewers

Why do invited Reviewers get "invitation has expired" when clicking the accept link even though the date of expiration is set in the future?

"Invitation has expired" appears either when the link actually expires or because the invitation was resent. The user is clicking on the previous invitation's links. It is also important to know that if a user is imported into the conference, there should be no invitation sent to the same user. Look at the Reviewer Invites page.

Reviewers accepted invite so why can't they see the conference?

The Status column of the 'Manage Reviewer Invites' page will show you if the email you invited is a CMT account or not. If the reviewer says the invitation was accepted but the status is 'Not Responded' and not a link, then the email the reviewer used is incorrect. It is not the same email the invite was sent to.

Why can't the Reviewer get verification after accepting invite and creating account?

There are a couple of reasons the reviewer didn't get a verification. One is because the invite was sent to a different address than the one they used to create the account. It must be spelled exactly the way it was sent, for example the invite was sent to '@xxmail.school.edu.cn' but they registered with '@xxschool.edu.cn'. The other issue is that it is not uncommon for universities to block emails. We suggest you contact your university IT department and have them whitelist '@msr-cmt.org.'

When a reviewer clicks accept in an invitation, they are accepting to be a reviewer in the conference, not accepting the paper they were assigned.
If reviewers are to have a choice to accept a paper assigned to them or not, the best thing to do is implement 'Reviewer Note.'

Is there any way to bulk invite reviewers to multiple tracks at once?

Currently the bulk invites will need to be sent per track. However, bulk importing can be done for all tracks. Here is the difference between importing and inviting

Why do email template placeholders show in the body of the email?

Placeholder names show because you are sending the email from a different page than where the template was created. Each page that you email from, may have different placeholders. Not all placeholders will work in all pages.

Why do Reviewers have no where to enter subject areas?

In the Review settings, enable 'Require reviewers to select subject area'. This will give them the red bar on top of their Reviewer Console. They can subsequently edit the subject areas under their name dropdown>My Subject Areas. Please note, if the Reviewer entered subject areas, then the 'Require reviewers to select subject area' setting is deselected, they will not be able to edit their subject areas.

Why can't Reviewers find their conferences after they log in?

This is more than likely because they are logging in with an incorrect email. Please note that all conferences from 2018 and prior have been removed from CMT. The data is irretrievable.

Why can Reviewers save a draft but cannot submit the Review?

Make sure that all required (*) questions have been answered. If they have all been answered and there is still a problem, check that the review file setting doesn't have maximum number of files set less than minimum number of files.

How did a Reviewer get assigned his/her own paper?

The reviewer has two different CMT account email addresses - one in a reviewer role, the other as a co-author. Find out which account they want to keep and remove the other from the conference.
To change the reviewer's CMT email, add the author-email to the reviewer role, do not invite it. Explained here.
To change the co-author, edit the submission, remove the incorrect email and add the correct one.

A Reviewer entered a review, which I saw previously, but why does it now appear to have been removed?

There could be a few reasons.

Reviewer now has a conflict with that paper
The paper has has changed the track after review was submitted
The paper could have been withdrawn or deleted
The author(s) may have deleted their CMT account(s)
The Review phase has completed or been disabled

Why can't Reviewers actually enter a response to one of the questions on their assigned papers?

Go to the Review Forms, find the question they cannot answer and click 'Edit.'
Make sure the 'Locked for edit' box is unchecked.

How can I fix the wrong name of a reviewer I mistakenly entered against a correct email?

This can only happen when inviting a reviewer where the user does not have a CMT account. The Reviewer can accept the invitation and enter the correct name when creaing the account. No one is allowed to modify a user's profile but the owner of that profile. The chair can also resend the invitation with a note saying 'the user name is mistakenly entered, please enter the correct name when creating your account.'

Reviewers can edit reviews when discussion is enabled. How do we stop this?

In the Chair Console, set reviews as ready only using “Actions -> Review Read Only -> Bulk Enable”.

How do I set a default view for the Reviewer Console?"

Console cutomization is only available for Meta-Reviewers and Senior-Meta-Reviewers.

How do we make all "Accepted" papers visible to reviews?

Bulk disable all papers for discussion
Bulk enable only accepted papers for discussion
Enable review setting “Allow all reviewers to discuss non-conflicting papers"
Enable reviewing discussion activity

Are Reviewers automatically notified of their assignments?

No, there are no automatic emails sent out. They must be sent manually. However, if an assignment is manually created or edited, and the Reviewer setting 'Allow chair to notify reviewer of new manual assignment' is enabled, the Chair will see a popup allowing the system to send and email to that reviewer notifying them of the assignment.

I accidentally unassigned a reviewer from a submission and lost their review. Is it possible to restore a review?

Just re-assign reviewer to the submission. The submitted review will be restored automatically.

Is there anyway that I can unassign the reviewers that have not submitted a review?

From “Manage Reviewers” page, filter reviewers with less than 100 percent review completion.
Select “More -> Edit Assignment”
Find papers with “Review Submitted? = No” and uncheck “Assign” checkbox.

Why are Reviewers are unable to view other reviews?

For Reviewers to see others' reviews, the Chair needs to enable papers for discussion in the 'Chair Console' and 'Review Discussion' activity needs to be enabled with a proper future date. Also, Chairs need to go to the Review Form settings page and enable 'Visible to other reviewers during discussion' on all questions that are to be seen by other Reviewers.

Why are Reviewers seeing Desk Reject papers?

CMT does not automatically filter out desk reject papers from reviewers. In the Review settings page, select 'Do not show desk reject papers'.

'Do not show desk reject/withdrawn papers' are enabled. After sending an email with the placeholder {Review.AssignedDetail}, why are those papers still included in the email?

The reviewer setting “Do not show ...” does not change how many papers are assigned to the reviewer. It just hides those papers in the Reviewer Console.
If you want to use {Review.AssignedDetail} in your emails, an option would be to unassign reviewers from all withdrawn and desk rejected papers.

How do I get Reviewers to see a paper's status?

Go to Review Settings and check 'Allow reviewers to see paper status before author notification'.

Do you have default Review forms?

Each conference is unique. All forms, including the Review forms, will need to be created.

Why can't Reviewers see the Compare Reviews button when it has been enabled in the settings?

The Chair must also enable the Review Submission Activity and set the deadline to a date in the future in order for Reviewers to be able to compare papers.

How does a Reviewer select his/her own User Type?

This must be done by CMT Support. Just send an email to CMT Support.

In the Automatic Assignment Wizard, why can't I assign maximum number of papers per reviewer?

Make sure you have Reviewers in your conference before using the wizard.

I invited a Reviewer and it was accepted, so why do I just see a string of alpha-numeric characters in all of the fields?

This is because after the invitation was accepted, the user deleted their CMT account.

Why can't the External Reviewer see Author Feedback?

Author Feedback needs to be submitted for the External Reviewer and Reviewer to be able to access it.

What is the 'Do not use review snapshot during author feedback' setting?

Review snapshot is automatically enabled. This means that reviewers can continue to edit their reviews while authors see the snapshot. Checking the box disables the review snapshot.

How do I modify the parameter of the placeholder {Invite.Expiration}, for Reviewer invitations?

The time span is adjusted in the Settings tab>Features>Reviewer. Enter the amount of days in the field next to 'Reviewer invite expiration time span (in days).' Save the changes. This is not retroactive to invitations already sent.

Are Submission file names visible to the Reviewers?

The original name will only be visible to chairs and track chairs. Reviewers will see '[PaperID] Submission.pdf'. However, CMT does not scan pdfs. If there is identifiable text in the pdf itself, it will be visible to the reviewer.

How do I make only some of the reviews visible to authors?

Deselect the 'Visible to authors during author feedback' and ‘Visible to authors after author notification' checkboxes for any reviews not to be viewed by authors.

Senior Meta-Reviewers can always see identity of Reviewers and Meta-Reviewers.
Meta-Reviewer-related visibility settings are on the Meta-Review Settings page:

Show meta-reviewer identity to reviewer
Show reviewer identity to meta-reviewer
Show meta-reviewer identity to other meta-reviewer

Disable this Reviewer setting:

Show Reviewer Identity to Other Reviewer

How do Reviewers email Meta-Reviewers?

The Chair needs to enable 'Meta-Review Submission' setting with a proper end date, then an email link will be in the 'Reviewer Console' in the 'Review and Discussion' column for the paper about which they need to email.

Why are Authors unable to see Reviews and decisions?

Follow the steps in this Cheat Sheet to make reviews visible to authors

Why are Reviewers seeing all papers when they are already assigned papers?

This means 'Reviewer Bidding' is still set as 'Enabled.' Go to Settings>Activity Timeline and set the 'Reviewer Bidding' status to 'Completed.' Don't forget to save the changes.

How can Reviewers see their assigned papers?

Reviewers will need to login to the site to access their assigned papers. From their Reviewer Console, they can click on "Download" link corresponding to the Paper Id. As Chair, you can impersonate any reviewer and view their experience using "Users -> Conference User" and clicking on "More -> Impersonate" corresponding to the user name.

How do I remove Reviewers?

To remove a user from the Reviewer role, Chairs can use the menu option "Users -> Conference User". Apply a filter to find the Reviewer. Use the "More -> Manage Roles" menu corresponding to the user name to add or remove that user’s role (Reviewer).

How do we download Reviewers' subject area-based relevance scores?

Chairs can use "Submissions -> Actions -> Export to Tab Delimited -> Reviewer Relevance Scores" menu to download scores.

How do we get a snapshot of all Reviews just before the rebuttal (Author Feedback) phase?

From the Chair console, go to “Actions -> Export to Excel -> Reviews Snapshot” to download the snapshot created before rebuttal.

How do late Reviews interact with the 'Review Snapshot' setting?

If the 'Snapshot' option is used, late Reviews (or updated reviews) will not be available to authors. For example, if there are two Reviews for a paper (using the 'Snapshot' option) and later a third Review is added, the Authors will still see only two reviews.

If we get late Reviews with snapshot turned on, how do we update the snapshot so Authors can see those reviews?

Re-enabling 'Author Feedback' activity will update the snapshot.

How do we have discussion posts/emails sent/visible to only those Reviewers who have submitted their reviews?

Check the Review setting 'Allow access to reviewing data only after submitting own review.'

Will CMT send automatic reminders to non-responsive Reviewer and Meta-Reviewer invites?

No, CMT does not automatically send reminders. The resending of invites to Reviewers and Meta-Reviewers will need to be done manually.

In Discussion phase, why is Author Feedback not visible to the Reviewers?

The Author Feedback status needs to be marked as 'Completed' with a date set in the past. Go to the Settings tab, select Activity Timeline>Deadline and mark the status as 'Completed.'

I deleted the Reviewer from the conference, do I have to remove the Reviewer invite, also?

If the Reviewer is removed from the Reviewer role in the conference, there is no need to remove the invite.

I deleted a Reviewer from our conference. Will they still receive conference emails?

Once the Reviewer is deleted from the conference, they will not receive any reviewer emails or reviewer notifications from the conference. If that user is also an Author with a submitted paper to the same conference, then that user will continue to receive emails pertaining to Authors.

How do I change the expiration of the Reviewer invitation?

In the Reviewer settings tab you can change the duration of the invitation, but this has to be saved before you send the invites. Changing the invitation expiration date does not affect invitations that have already been sent.
The expiration will be what the initial invitation had.

My Reviewer got an error 'You are currently not allowed to perform this operation.'

Please make sure the Reviewer does not have dual roles of both a Reviewer and an External Reviewer for the same paper.

Can I change a reviewer question from 'option' to 'option with value' without losing any reviews?

The reviews will not be affected as long as the question is not deleted.

Why does the Reviewer only see a few assigned papers, when the others that are not visible have no conflicts at all?

If the Reviewer sees some papers and conflicts are not an issue, look at the paper status of the missing ones to see if they are withdrawn, desk-rejected or rejected.

Why can't Reviewers view their assignments?

There are a number of reasons why Reviewers cannot see their assignments.

Chairs need to enable Review Submission Activity and set an appropriate deadline (for each track separately).
Make sure no conflicts have arisen.
Make sure papers are assigned.
Make sure papers are not withdrawn, desk-rejected or rejected.
Make sure the 'Review: Read Only' column in the Chair Console is set to 'No'.
Make sure there are Review questions for Reviewers to answer otherwise there will be no 'Enter Review' link.
Make sure the 'Allow only revision papers for reviewing' is not checked in Settings>Features>Review.'

Why does Reviewer see assignments but Chair does not?

Make sure Reviewer is logged into the correct conference.
Screenshots of Paper Titles from Reviewer Console will also confirm incorrect conference.

Reviewers download revision papers but zip files are empty

This will happen when the 'Review Submission' deadline has passed. Chairs need to ensure the 'Review Submission' deadline is set in the future.

Why do I get an error copying Review questions to another track?

It is because you have the same question(s) in the destination track.

Why can't Reviewers submit their reviews?

Follow all the steps here in the quick guide.
It is likely an improper deadline or a review setting that shouldn't be enabled.

How do I stop Reviewers from changing their Reviews?

If the Reviewer gave a response, yet more Reviewing is to be done, enable 'Lock for edit' in the Review questions form. This will lock the question so it cannot be edited by the Reviewer.

I disabled reviewer bidding but it is still on

Make sure you click the save button after changing the status
For multi-track conferences, you must enable/disable and save bidding for each track separately.

Reviewers said there are no links to accept in the invite email

If invited properly, each invitation email has its own accept/decline links. If there are no links, the invitation was either modified or not sent from the proper place within CMT. The Chairs will need to rectify this.

How can I bulk delete Reviewer assignments?

From Chair Console, click Actions>Bulk Delete Assignment>Reviewer.

Is there any way to adjust coefficients for each Review criterion and calculate score for each paper?

You can export reviews to Excel (Actions -> Export to Excel -> Review) to compute the final scores using appropriate weights.

How do I enable automatic confirmation emails to reviewer when reviewer submits a review?

After submitting the review, a confirmation email will automatically be sent to the Reviewer (and the Chair, if enabled for notification of reviews) stating Reviewer ABC submitted a review for paper XXX in conference YYY.

Can I change Review questions after Reviews made?

We do not recommend this, however if you are going from a Review question Type of 'Option' and want to have it as 'Option with Value', this is acceptable and there will be no issues as long as the choices are not updated (added or removed).

Why do I get an error inviting a Reviewer to a submission?

As noted in the guide, You should not invite existing Reviewers to a paper; you should just assign them. The error clearly states, "The user can't be invited as reviewer for paper xx as the user is already a regular reviewer."

How do I save a Reviewer Invite template?

Invitation emails cannot be saved as template like other emails.

Why can't Reviewers see papers to bid on?

Presuming the bidding phase is enabled properly, if 'Allow bidding only for suggested submissions'is enabled, Reviewers will only see what papers Meta-Reviewers suggest. If there are no suggestions, there will be no papers to bid on.

Why can't reviewers see all papers' reviews when I set it correctly?

It is likley the Review setting, ‘Allow access to reviewing data only after submitting own review’ will need to be disabled.

How do Reviewers see author's answers to submission form?

Provided the questions in the submission form are marked 'Visible to reviewers,' when the review phase is open, the assigned reviewer can see those answers in the Submission Summary by clicking the Paper ID number in their Reviewer Console.