CHAIR HOW-TO: Create and Manage Review Questions
IMPORTANT
For Mult-Track Conferences a track must be chosen before being worked on first.
OVERVIEW
Review questions are unique to each conference.
This document will show how to configure the Review Form with generic questions/tasks/descriptions/calls-to-action.
There are four types of Review Questions:
- Comment - The Reviewer enters text to comment on the paper
- Agreement - The Reviewer agrees to the posted statement/question
- Options - The Reviewer has a choice of answers. It can be restricted to a single answer or be multiple choice.
- Options with Value - The Reviewer has a choice of answers (usually restricted to a single answer), each with its own number value. This allows the Chair to quickly see the Reviewer's opinion of the quality of the paper and aids in the decision-making.
Click here to see how to copy questions from one track to another in a multi-track conference.
IMPORTANT
It is not advisable to change the Review Questions after they have been answered by Reviewers. The completion percentage in the Chair Console will not change, and the Reviewers will not know they need to answer more/different questions. Also the question numbering and order may change creating unwanted issues.
Note
Everything with a red asterisk is required.
REVIEW FORM
- Go to Settings>Forms>Review and click ‘+Add new question.’
CREATE NEW QUESTION (COMMENT)
The Create New Question dialog box appears. This incorporates the text field, the checkboxes and the type of question. The ‘Note’ field is ancillary.
Text
- The required ‘Text’ field is where the question/task/description/call-to-action goes. In this case we enter ‘Comments to authors.’
Checkboxes
- Required: Check this if the question is required to be answered by the Reviewer.
- Locked for Edit: If the Reviewer gave a response yet more Reviewing is to be done, this will lock the question so it cannot be edited by the Reviewer.
- Visible to other reviewers during discussion: This makes this question visible to other Reviewers in the Discussion phase.
- Visible to authors during author feedback: Authors will be able to view Reviewers’ answers to this question in the Author Feedback phase.
- Visible to authors after author notification: Enabling this will show the Reviewers’ answers to this question after the Authors are notified of the decision.
- Visible to meta-reviewers: This enables Meta-Reviewers to see the Reviewer's answer to this question.
- Visible to senior meta-reviewers: This enables Senior Meta-Reviewers to see the Reviewer's answer to this question.
Note
If all the checkboxes (except 'Required') are deselected, then only the Chairs can view the answers. This can be useful for confidential comments from the Reviewer to be seen by only the Chairs.
Type
- Since this is a comment the Reviewer would need to type in, click the dropdown next to ‘Type’ and select ‘Comment.’
- Click the ‘Maximum Length’ dropdown to set the maximum length (in characters) of the Reviewer’s comment. Then click ‘Save changes.’
REVIEW QUESTIONS PAGE
It is now saved on the ‘Review Questions’ page.
CREATE NEW QUESTION (AGREEMENT)
Text
- From the ‘Create New Question’ dialog, enter an agreement-type text. For example, we entered ‘While performing my duties as a reviewer, I have and will continue to abide by the xxx code of conduct.’
Note
The checkboxes were covered in the previous section.
Type
- Select ‘Agreement’ from the ‘Type’ dropdown and click ‘Save changes.’
REVIEW QUESTIONS PAGE
It is now saved on the ‘Review Questions’ page.
CREATE NEW QUESTION (OPTIONS)
Text
- From the ‘Create New Question’ dialog, enter an option-type text. For example, we entered ‘Have you seen this submission online (e.g., arXiv, personal website, social media)?’
Type
- Select ‘Options’ from the ‘Type’ dropdown.
Show as
- Click the ‘Show as’ dropdown and there will be 4 choices:
- List with single choice (radio button)
- List with multiple choice (checkbox list)
- Drop down list with single choice (drop down box)
- List box with multiple choice (use Ctrl/Shift to select multiple items) In this case we will use the ‘List with single choice (radio button)’ selection
Choices
- Click ‘+Add New Choice.’
- In the text box, enter the first-choice answer. In this case, ‘Yes.’
- Click ‘+Add New Choice’ again. Then enter the second-choice answer. In this case, ‘No.’ Click ‘Save changes.’
REVIEW QUESTIONS PAGE
It is now saved on the ‘Review Questions’ page.
CREATE NEW QUESTION (OPTIONS WITH VALUE)
NOTE
If configured, the Discussion page will show values of Reviewer questions answered by Reviewers for that particular paper.
This is perhaps the most important question type. From the ‘Create New Question’ dialog, enter an option-type text. Each answer will have a value.
For example, there will be 5 answers to this question: ‘Overall score’.
Each answer will have a unique value so the Chairs can make a more informed decision on the paper. The answers and values will be:
- Strong Accept 4
- Accept 3
- Weak Accept 2
- Weak Reject 1
- Reject 0
Text
- Enter ‘Overall score’ in the text box.
Type
- In the ‘Type’ dropdown select ‘Options with value.’
Show as
- For this example we do not want multiple answers for this question. Select ‘List with a single choice (radio button list).
Choices
- Click ‘+Add New Choice.’
- Enter ‘Strong Accept’ and change the value to ‘4.’
- Click ‘+Add New Choice’ 4 more times.
- Enter the text and values for each as outlined above. Click ‘Save changes.’
REVIEW QUESTIONS PAGE
It is now saved on the ‘Review Questions’ page.
- Click the ‘Preview’ button.
REVIEW QUESTIONS PREVIEW
The ‘Review Questions Preview’ page is what the Reviewers will see when they Review their assigned papers.
Note
Everything in blue italics are the results of the enabled checkboxes in each question.
VISIBLE TO OTHER REVIEWERS
Important
If the checkbox is selected and a Reviewer enters a review, the 'visible to...' blue italics will be visible, but the reviews will not be visible until the actual phase of the conference is enabled. For example, it may say 'visible to other reviewers', but it is not visible until Discussion is enabled.
Reviewers can only see other reviews in Discussion and if the 'visible to other reviewers during discussion' is enabled.
- The Reviewer clicks on the 'All Reviews' link.
All the other reviews are visible to the Reviewer
EDIT QUESTION
IMPORTANT
It is not advisable to change the Review Questions after they have been answered by Reviewers. The completion percentage in the Chair Console will not change, and the Reviewers will not know they need to answer more/different questions.
Chairs may find the need to edit a question. In this example we will edit Question #3 to allow visibility to only the Chair. No one else will be able to view this question/answer.
- From the ‘Review Questions page, click on the edit icon for Question #3.
Edit Question Page
- The ‘Edit Question’ page appears. Deselect the checkboxes except for ‘Required.’ Then click ‘Save changes.’
REVIEW QUESTIONS PAGE
- Click the ‘Preview’ button.
REVIEW QUESTIONS PREVIEW
- Verify there is no visibility to anyone but the Chair (by virtue of no blue italics).
COPY QUESTIONS TO ANOTHER TRACK
Important
If the target track already has questions, copying questions from the source track to that track will not override those pre-existing questions. The new questions will be added to the target track Form after the last pre-existing question.
In a Multi-Track Conference, the Chairs can copy questions from one track to another making it a much less time-consuming task.
In this case we will copy the questions from the 'MAIN' track to the 'Supp' track.
The 'Supp' track is showing no questions.
- After the questions for the 'MAIN' track have been created, click 'Copy questions to other track.'
- In the 'Copy Questions to Other Track' pop-up, click the 'Supp' track and then click the 'Copy' button once. To highlight more, simply [CTRL] click on the tracks desired.
IMPORTANT
Do not click 'Copy' more than once or the questions will be added to the new track again, thus doubling them.
A green alert with 'Questions copied!' will appear at the bottom of the page.
- Go to the 'Supp' track and verify the questions appear.
CHANGING QUESTIONS
Once the Review phase has started we strongly recommend no changes are made to Reviewer questions.
However, if aggregate columns are wanted, the Reviewer questions can be changed to 'Options with Values' if the original question type is 'Options.'
Changing ‘Options’ to ‘Options with Values’ = OK. Any other changes in question tpye will not work and remove whatever the review is for that question.