Science Peer Review For The 21st Century

By creating an environment whereby experts can share their opinions under psychologically safe conditions, SciPinion has created a safe haven for experts to share their true opinions without influence and we are putting this to work to help inform the most important science issues facing society.

Far too often, important debates about important scientific issues are inflamed by activists, competing research groups or sometimes simply because the science on a specific issue has some uncertainty. This environment can lead to reactionary decisions by regulators or fuel litigation. In some situations, experts who share their opinions on these controversial topics are attacked.

SciPinion strives to:

Help companies, governments and consumers make better decisions by understanding the collective wisdom of the world’s experts.
Increase trust in science by reducing sources of bias and groupthink.
Fostering engagement amongst the scientific community.

Engage SciPinion About Us

SciPinion Certified Peer Reviews

When the SciPinion Peer Review Process is followed in its entirety, it is considered a SciPinion Certified Peer Review. These peer reviews adhere to the following standards:

SciPinion has full control over selection of experts after receipt of input on expertise and demographic needs, inclusion/exclusion criteria
SciPinion has final say on question composition
Panel size is of sufficient size to cover expertise needs for the topic area (Min. 5)
Inclusion of a modified Delphi, multi-round review format
The sponsor and all review materials shall be blinded to the experts
Experts will remain anonymous before and during all deliberations to reduce groupthink and participation bias

We stand behind the results of these panels, certifying their objectivity, integrity, and reproducibility. Read on to learn more about our proprietary peer review process.

The SciPinion Peer Review Process

We have employed our process to engage the world’s experts, conducting peer reviews, providing companies and governments better insight on how scientific data should be interpreted and applied to decision making. We have recruited thousands of experts in toxicology, epidemiology, and exposure assessment to serve on peer review panels and conduct consulting efforts to date.

As part of our peer review practice, we have developed a formal approach to identifying, recruiting, selecting, and engaging panels of experts to minimize potential bias (Figure 1: Kirman et al., 2019). We have also developed a robust web app that facilitates collections of opinions and debates amongst panels of experts, all while remaining anonymous to each other during the engagement.

Our approach to asking detailed questions with quantitative response options supports quantitative measures of opinions and statistical measures of outliers and consensus amongst panels of experts. This integrated approach has been shown to yield reproducible results (Figure 2: Kirman et al., 2019). No other company or government body has tested or proven that their approach to assembling and conducting peer review panels yields reproducible results/outcomes. The fact that SciPinion has proven our approach is reproducible lends extensive credibility to the outcomes of our reviews and position statements and will carry more weight than the opinion of a single expert or consulting firm.

Figure 1 – The SciPinion Peer Review Process

Figure 2 – Reproducible Results

Background: Design Features of SciPis

Any engagement with SciPinion can be designed to best meet the project objectives.

A SciPi is defined as a collection of Scientific oPinions. There are various design options to each stage of a project. Each panel engagement with SciPinion involve three basic phases: Pre-SciPi, SciPi and Post-SciPi.

Figure 3 – Pool Of Ideal Reviewers

Pre-SciPi Stage

Within the Pre-SciPi stage, the sponsor can choose whether to assemble the review package or have SciPinion prepare it. The process of selecting experts can have an additional trust factor implemented by engaging a third-party auditor who helps vet the expertise of the volunteers (this is usually recommended on topics that are highly controversial and charged) (Figure 3: Kirman et al., 2019). And the sponsor can choose to prepare the charge questions or rely on SciPinion to prepare those. On those controversial topics, we can also employ a third-party editor to review the questions and verify no leading or biased questions are asked.

SciPi Stage

All SciPis are conducted in a manner in which all experts are anonymous to each other during the interaction. This helps to minimize the effects of group think (e.g., social pressures associated with face-to-face group deliberations that can mislead groups towards conformity around an incorrect position, rather than consensus around a correct position). Likewise, all SciPis are designed to facilitate independent input from experts. During Round 1 of a SciPi, all experts log in, review the SciPi instructions and download the review material. They all answer the SciPi questions independently and cannot see answers from any other experts on the panel. If a sponsor desires, multiple rounds of review can be incorporated to help facilitate collective wisdom and learning amongst the experts and/or to include follow-up questions. Round 2 often includes a debate round in which all experts log back into the web app and review the results from all experts and debate each other. This also allows the experts to vote (thumbs up or down) on fellow experts’ debate entries. Round 3 often includes: A) experts answer follow-up (clarifying or new) questions posed by either the sponsor or ones suggested by the experts, and B) allows the experts to re-answer questions from Round 1 in case the debate informed the expert to change their answer.

Post SciPi Stage

Sharing of results can vary from:

PDF of results from directly from the SciPinion App
Short report that includes analytics (outliers, consensus) if sufficient numbers of experts are engaged (e.g., 10+). This report is most appropriate for internal purposes.
Formal report that is appropriate for regulatory submissions (additional cost) or a manuscript to be submitted to a journal for publication.

The form of the report can be matched to the objectives of the project and can be done in phases (e.g., decisions about formal report and/or manuscript can be made upon completion of the SciPi).

Managing Bias

The impacts of bias on face-to-face peer review panels

When experts, or any group of people for that matter, are placed in a room to conduct a face-to-face deliberation, there are numerous and well known sources of negative heuristics that can significantly impact the outcome of the deliberations. These include:

Group think
Deference to the perceived expert
Amplification
Overbearing panel members

In addition, in today’s adversarial environment, experts have been attacked for speaking their opinions on controversial topics. As a result, many experts simply ‘stay on the sidelines’ despite being some of the most knowledgeable experts on a given topic. For all these reasons, SciPinion designed a process and methodology for finding, selecting, and engaging panels of experts that eliminates the negative heuristics.

What about bias?

Many regulatory agencies pretend to try and design panels that eliminate bias. At SciPinion, we contend that everyone has bias and some of those sources of bias may be unknown. For instance, we have found that the year of obtaining an advanced degree can impact how an expert interprets scientific evidence from cutting-edge technologies.

Can we ‘balance’ the sources of bias?

Other agencies will claim to establish panels that ‘balance’ sources of bias or opinions. When agencies do this, what they are admitting is that they already know the opinions of certain experts and will try and ‘balance’ a panel to give them an even split on opinions. This in itself is introducing bias, because those selecting the panel are ‘assuming’ they know the preconceived opinions of each expert. While this will likely yield a mix of opinions, it tells you nothing about how prevailing one opinion is over another amongst the broader community of experts. So, if truly the community of experts is 95/5, establishing a panel with a 50/50 split will yield un-representative findings.

Why we do things the way we do

Panel selection

When we select a panel of experts for a peer review, we use an objective and quantitative metric of expertise and a model picks the panel. This process if completely objective, quantitative, transparent and reproducible. We do not assume or make any prejudgments about what the opinions of any particular expert will be. If a sponsor desires or requires certain diversity in demographics (e.g., region of residence, gender, sector of employment), the model can account for those requirements.

Panel Engagement

All engagements with experts occur online. We do this to eliminate the negative influences that occur with face to face meetings. While experts are working online within the SciPinion web app, the experts identity is not disclosed. Each expert is labeled sequentially Expert 1, Expert 2, and so on. This eliminates any undue influences by recognizing your peers and affords greater psychological safety. Upon completion of a peer review, disclosing of the identities of the experts can occur via three options (to be disclosed upon recruitment of experts); experts are identified but a key as to which expert is which is NOT provided (preferred approach), experts are identified and a key is provided to disclose which expert is which. On few occasions, particularly when the topic is so controversial that most experts would never want to be named, we can allow experts to remain anonymous in perpetuity. This is an extremely rare case.