F.A.Q.

Technical Track FAQ

The goal of this set of FAQs is to provide the committees, the authors, and the community as a whole with information about how the technical research track reviewing process of ICSE 2015 operates. We believe that documenting and sharing this information will increase ICSE operating transparency, help everyone better understand the process and expectations, and provide a baseline for future conferences and track organizers.

If you have a question about the process that is not answered here, please email the chairs.

What is the reviewing model for the technical track of ICSE 2015?
The reviewing model for ICSE 2015 is depicted in the following figure. It involves two program co-chairs and two committees, the Program Committee (PC) and the Reviewing Committee (RC) working in concert. The process starts with a cursory submission review by the chairs for format and scope compliance. It is followed by a first reviewing phase where PC and RC members will provide a total of two reviews per paper (every submission will receive at least one review from a PC member in this phase). Papers with at least one supportive review will advance to the second phase. During the second phase, PC members will be assigned to perform a third review. This will be followed by an online discussion to clarify reviewers’ positions, assigning an additional reviewer if necessary, and a physical meeting with the PC members and chairs to decide the final list of accepted papers.
Why are we changing the reviewing model to the PC+RC?
1. The short answer:
  To scale up to a larger number of submissions, to increase the reviewing expertise per submission, and to increase community participation.
2. The long answer:
  ICSE has received an ever-increasing number of submissions (from ~405 in 2009 to ~495 in 2014) and various practices have been put in place over the years to accommodate those increases while maintaining the reviewing quality (e.g., 2 phase reviewing). With close to 500 submissions, depending on the adopted reviewing model, recent ICSE required anywhere from 1200 to 1500 reviews. Assuming a 50 member PC, that is 24-30 reviews per PC member, a very heavy reviewing load. Additionally, with the increasing range of software engineering topics, having enough expertise in the PC to match the submissions has become harder. Increasing the PC size, however, is not feasible as we intend to keep a physical PC meeting to discuss the submissions.
  There are several ways to mitigate this challenge, all of them with tradeoffs.
  Our choice to address these challenges was to adopt the PC+RC model. This model, which has been used in various ways at other venues (.e.g., PLDI, OOPSLA, ASE), helps us address the challenges without radically changing the existing model and timeline that has worked well for ICSE in the past. The incorporation of the new RC committee allows us to involve a greater part of the community, providing additional expertise and reviewing power. This model also lets us keep most of the reviewers of the submissions involved throughout the decision process, something that we believe is important to make the better decisions.
What are the responsibilities of the PC or RC members (and why should the community thank them for their work)?
1. PC members will be requested to:
  1. Review ~20 papers
  2. Lead the online discussion of subset of assigned submissions and participate in the discussion of assigned submissions that are undecided (the chairs will appoint one discussion leader, among the reviewers of each submission).
  3. Prepare a summary of the online discussion (following the format provided by the chairs). This summary will be provided to the authors for additional feedback and will be used at the PC meeting to guide the discussion.
  4. Attend PC meeting
2. RC members will be requested to:
  1. Review ~10 papers
  2. Participate in the online discussion.
How many and what reviews can we expect from ICSE?
1. Each submission will receive at least 2 reviews. Submissions that move to the second reviewing phase will receive 3 reviews (and in cases when further expertise is needed, they will receive 4 reviews). At most, one of the reviews will come from the RC, and the rest from the PC.
2. PC and RC reviewers are carefully selected because of their expertise and professionalism, and the reviews should reflect these traits. Reviewers are also instructed on what the chairs expect from the reviews and what negative reviewing behaviors and patterns should be avoided. Reviews should provide useful feedback to the authors, including strengths and perceived weaknesses of the work.
How are Committees members submissions evaluated?
1. Chairs may not submit papers
2. Committees’ members can submit papers. Their submissions are evaluated with the same rigor as any other submission. Furthermore, submissions from the PC and RC committee members do not receive the benefit of early notification.
3. Throughout the process committees’ members will not be allowed to know the status of their submission and the reviews. And the identity of the reviewers will not be disclosed. (Just like with any other submission that represents a conflict of interest as defined by Sigsoft policies: http://www.sigsoft.org/about/policies/pc-policy.htm )
What are the benefits of being a Chair, PC or RC member?
1. Program Chairs, PC members, and RC members are volunteers and receive no compensation of any kind.
2. That being said, being a chair or member of the ICSE PC or RC committees, is an honor in our community. It recognizes a high level of research expertise and a high level of trust in their ability to assess submissions.
What submissions make it to each phase?
1. Submissions that meet the format, scope, and criteria of the conference will make it to the first reviewing phase.
2. Submissions that have at least one supportive reviewer or that lack enough reviewing expertise in the first reviewing phase will make it to the second reviewing phase.
3. Submissions with at least one supportive reviewer with high expertise will make it to the online discussion.
4. Submissions that merit further discussion will make it to the PC meeting. In general, submissions for which the reviews are consistently negative or positive won’t be discussed since their assessment is unlikely to change. We will use a lower and an upper scoring threshold for a submission to be discussed. Submissions ranked above the threshold and that received no major concerns for acceptance after the online discussion will be accepted without discussion at the PC meeting. Similarly, submissions below the threshold will be rejected.
Will the RC members be “heard” at the PC meeting?
This is one of the potential negative tradeoffs of the PC+RC model as RC members are not at the PC meeting. We will mitigate this challenge by having:
1. An online discussion where RC members can voice their opinion more strongly. The chairs will encourage and follow these closely.
2. For submissions where the RC member is the dissenting voice (as a champion or as a detractor), we have allocated time for another PC member to review the paper, the reviews, and the discussion before the PC meeting.
Why are we sending “early” rejection notices?
To provide quicker feedback. There is no reason to make authors wait for more than 100 days, when the outcome is known after 50 days.
One exception: to avoid potential evaluation bias by PC+RC members due to early feedback, PC+RC members will not benefit from the early notices.
Why not use rebuttal and/or shepherding?
Since the cost-effectiveness of these practices is questionable, we decided to keep the process simple, focusing the reviewing process and the reviewers’ time in evaluating the papers as submitted. The committees’ responsibility will then be to make an assessment if the flaws and omissions in a submission are minor and can be addressed in time for the camera ready, or require such major revision that cannot be accepted in the submitted form.
What is the difference between Categories and Topics?
Topics are the areas covered by the conference. When a paper is submitted, authors are asked to select the topics that are most relevant to their submission. This information is then used by the program chairs to better match submissions to reviewers. The five Categories (add link) (Analytical, Empirical, Methodological, Perspective, Technological) are also provided by the authors at submission time. Categories are meant to help position the intended contribution of each submission, as well as set the expectations on how this contribution should be evaluated by the reviewers.
What instructions do PC + RC members receive?
Program Chairs communicate with PC and RC members many times throughout the reviewing process with a stream of emails, personal communications, and presentations.
Here are are some of them.
How would we know how well the reviewing process worked?
The program chairs will continue the data-driven assessment approach performed by the chairs of ICSE 2014 http://orbilu.uni.lu/handle/10993/16345 to quantify the performance of the reviewing process from the perspectives of authors and reviewers in terms of reviewing and paper quality, reviewing workload, and community engagement.
How many papers can I submit to ICSE?
There is no limit to the number of papers you can submit to ICSE. However, keep in mind that the review of every submission consumes precious community resources (expert reviewers’ time) so a “shotgun” approach may be detrimental to the community and to your reputation. Focusing on less and higher quality submissions benefits us all.
How are reviewers assigned to submitted papers?
The assignment of reviewers to submitted papers is done by the program chairs, taking into consideration each submission topic and category, and the reviewers expertise, interest, load, and potential for conflict of interest.
New to ICSE2015: we will tap on the reviewers expertise to recommend other reviewers in the PC that might be able to provide additional insights by reviewing the submission.

Technical Track FAQ

NOTE! This site uses cookies and similar technologies.