CHAIR HOW-TO: Manage Meta-Review Aggregates

OVERVIEW

The Chairs may want to see an aggregate of answers to valued-meta-review questions in the Chair Console. A valued-meta-review question is a meta-review question with a weight assigned to multiple choice answers.
The Meta-Reviewer answers these questions when giving their review of their assigned paper. The answers, in turn, show up as a numeric value in the Chair Console.

Single-Track Conference

Meta-Review Settings

  • Go to the Meta-Review settings page and select 'Show aggregate columns on consoles.' Save the changes.

Meta-Review Aggregate

Meta-Review Questions

Create review questions with a value.
Related: Meta-Review Questions

Meta-Reviews Completed

Once the Meta-Reviewer completes the review of a paper, the Chair will be able to see the meta-review values in the Chair Console. In this example we have three valued-questions.
All three questions show in the Chair Console.

Meta-Review Aggregate

Limitations

There is a maximum of eight meta-review-aggregated columns that are able to be shown in the Chair Console. These columns are shown in alphabetical order according to the question.
This means if you have more than 8 value added questions, you will need to decide which ones will be shown on the Chair Console. Additionally, if the 10th question you want is at the end of the alphabet, even if it is question number 2 in the Review, it will not be visible in the Chair Console.

Workaround

The workaround is quite easy. You can use grouping symbols to mitigate the column-order. Grouping symbols are parentheses, brackets and braces and are used as they have a hierarchy.
For example, you have 10 value-questions with the last question, alphabetically, titled ‘Years Active.’
Since there are only 8 columns, this will not be seen in the Chair Console.
If you want this question to appear in the columns, wrap the title in parentheses so it looks like this – (Years Active).

Here is the hierarchy of the grouping symbols for CMT:

  1. Parenthesis ( ) come first: The title will appear in the beginning columns, in alphabetical order.
  2. Brackets [ ] will be second: All titles with brackets will appear after all titles with parentheses, but before braces, again in alphabetical order.
  3. Braces { } will be third: This will put the questions after the first two grouping symbols, but ahead of ones without any grouping symbols – also in alphabetical order.
Note

If you use brackets and no parentheses, then those titles with brackets will be in the beginning columns.

You can see the hierarchy in these 8 questions from left to right. Also, each grouping is in alphabetical order. Meta-Review Aggregate

Multi-Track Conference

Follow the same steps for a Multi-Track Conference as a Single-Track Conference.

Scenarios

The below scenarios are for viewing aggregates in the Chair Console.

  1. If you have six meta-review aggregate questions in Track 1, and two in Track 2, and there are no duplicate questions in both tracks, then they all will show in the Chair Console.

  2. If you have six meta-review aggregate questions in Track 1, and five in Track 2, again all different questions, then the last three questions at the end of the alphabet will not be visible. It doesn't matter if the last three questions are in Track 1, Track 2, or spread across both.

  3. If you have six meta-review aggregate questions in Track 1, and five in Track 2, but all of them are the same in both Tracks, except Track 2 is missing one, then six questions will show.

  4. If you have six meta-review aggregate questions in Track 1, and five in Track 2, with three questions the same in both Tracks, then you will have eight columns shown. This will be all of the questions.

  5. If you have six meta-review aggregate questions in Track 1, and six in Track 2, with three questions the same in both Tracks, then the question that was at the end of the alphabet will be the one not visible, regardless of Track.

For Track Chairs

In Scenarios 2 and 5, the Track Chair can see those questions not visible to the main Chair Console.
This is because the main Chair Console shows everything for every Track, while the Track Chairs only see what is in their particular Track.