Scientific challenges seek to promote the understanding of a problem, solution or outcome using empirical or measurable evidence-based practices. Scientific challenges encourage critical thinking to gain insight into natural, physical, computational, social and all other sciences. These challenges use a focused problem-statement approach that outlines specific criteria used for evaluation without specifying or suggesting how to solve the problem. Solutions require clarity and substance with respect to experimental design features to promote scientific rigor and reproducibility. It's essential to provide adequate details to your solver pool so that solutions can be verified and validated as required in the evaluation and judging criteria for these types of challenges. Interestingly, incentive prizes began squarely in the scientific world. Scientific societies in 18th and 19th century Europe regularly posted lists of challenges; David Hilbert in 1900 famously listed 23 math problems of the century, some of which remain unsolved.
The design of a scientific challenge often overlaps with any of the six other challenge types: ideation; software and apps; technology demonstration; analytics, visualizations and algorithms; creative design; and business plans. Scientific challenges solicit hypotheses from the crowd and advance scientific research through hypothesis testing, discovery based research, computational approaches and predictive modeling using huge data sets. These challenges also apply scientific discoveries and knowledge to other problem areas.
Scientific challenges can take many forms, including:
- Small ideation challenges which solicit new thoughts, theories, proposals, concepts and hypotheses that might address a broad problem
- Theoretical challenges requiring a well-thought-out idea with detailed descriptions, specifications and requirements
- Full demonstration challenges with prototype development, proof of concept or reduction-to-practice simulation that proves a solution or idea and can be tested by others
Scientific challenges can be structured in various ways, including as single-phase, multi-phase and linked competitions. Single-phase challenges can deal with ideation or algorithms as in the . Multi-phased challenges seek theories leading to proof-of-concepts within one competition as in the Environmental Protection Agency (EPA) and National Institutes of Health (NIH) , the Food and Drug Administration Food Safety Challenge, and the. NIH . Linked challenges are treated as separate challenges for administrative purposes but build upon earlier successes. The Seizure Detection Challenge and Seizure Prediction Challenge were two separate but linked challenges for which solvers used the same data sets. The NIH Follow that Cell Challenge could have been two separate, linked challenges if the total amount of the prize was not available for obligation in one fiscal year. Sometimes scientific challenges are also combined with citizen science projects like the Smithsonian Conservation Biology Institute's project to track cats and coyotes in order to engage volunteers in data collection.
1. Consider structuring a scientific challenge in multiple phases so there are several opportunities to evaluate, test, refine and improve the solutions.
An example of a single-phased scientific challenge is the Defense Threat Reduction Agency Algorithm Challenge, for which participants were asked to develop bioinformatic algorithms to analyze a specific metagenomic dataset. In this case, the results were objective, quantifiable and easily measured, and the challenge could be developed as an ongoing leaderboard using one or two specific performance metrics.
Scientific challenges structured in a multi-phased or linked competition may be more likely to result in a successful outcome, but you must plan the administrative requirements, evaluation aspects and timeframe of your challenge accordingly. Depending on the complexity and nature of the scientific challenge you propose and the specific objectives that you hope to accomplish, a multi-phased competition structure enables your technical review panel to test or verify specific claims from your submissions and ensures that your challenge can accommodate unconventional solutions and responses. It's often difficult to assess at face value whether a submission has fully met the criteria. The results and performance claims for each submission must be evaluated and validated before a final award determination can be made. This evaluation and validation process may need at least one additional round of competition and technical assessment beyond the initial submission package.
Scientific challenges most often provide for quantitative, as opposed to qualitative, evaluation and judging. In order for a scientific challenge to be successful it is imperative that specific, clear and unambiguous evaluation criteria be identified during challenge development and included in the announcement and other communications. The goal is to use a quantifiable measurement process that reduces subjectivity during evaluation and judging. Use quantitative measurements such as fastest, shortest and smallest whenever possible. Using a point system to assign weights to the different evaluation criteria can help both judges and solvers. The solvers can assess their own progress, and judges can use the criteria in a clear and forthright manner. These types of measurements can also reduce the burden on technical evaluators and judges.
3. Prizes should look different than grant programs! Sell your scientific challenge to others in the agency.
Scientific challenges can be more difficult to sell within your agency since they differ dramatically from traditional acquisition approaches. Much of the scientific research supported by the federal government is through grants and contracts. Challenges offer an opportunity to solicit new ideas into the mix. The process of running a scientific challenge itself generates a lot of ideas. Even when they don't win, solvers can follow up on ideas, and agencies can suggest potential partners or other resources to help solvers further develop their solutions. Challenges allow you to engage individuals at different career levels and from different academic backgrounds.
You can sell this approach to your agency leadership by using testimonies from participants in scientific challenges. For example, finalists from the first phase of the NIH Follow that Cell Challenge cited many benefits to the prize approach for stimulating research. In their own words:
- It generated a level of excitement.
- We had fun putting together the proposal.
- It pushed our thinking.
- The proposal was easier without having to think about budget and personnel justification; we could devote all the thinking to the science.
- It was a novel approach by presenting the need and asking for solutions without constraints.
- Challenges can tap resources across the world.
- It generated collaborations.
- People from all over the world want to work with you.
- It attracts folks from industry to garage tinkerers.
- It can incentivize business.
- It's open to everyone, not just NIH researchers or funded researchers.
- It generated creative ideas.
- You can submit proposals without strict NIH or National Science Foundation guidelines.
- We enjoyed working on the solution.
It's also important to remember that a challenge presents an opportunity to design a more flexible, results-driven research approach that should look dramatically different from a traditional grant opportunity. Don't give in to the urge to structure a prize competition like a traditional grant call! This could limit your pool of solvers to the usual suspects.
Challenges seek to engage new and old crowds differently. The innovation you're looking for could lie in a mathematical or technological approach from a different discipline. To reach solvers from new and different disciplines or fields, you want to use language appropriate for the crowd you're targeting. However, don't use too much insider speak when describing a challenge. Minimize the domain-specific language in the challenge description so you don't discourage participation from other crowds. Encourage your internal stakeholders to talk about the challenge broadly, both within their respective communities and at scientific or professional association meetings. This will help you reach more potential solvers and lend credibility to the challenge approach.
Challenges are a relatively new way for federal agencies to spur innovation. Educate and encourage potential solvers and their organizations to learn about challenges and the opportunities they offer. You can take certain actions to ensure solvers feel encouraged to participate. Emphasize to solvers the the nonmonetary value of participating in scientific challenges. Some challenges, like the Food and Drug Administration (FDA) Food Safety Challenge, provide an opportunity for solvers to work with otherwise inaccessible scientific experts to advance their solution or technology. The faster timeline of challenges attracts individuals such as millennials, who have good ideas but expect results faster than the 12- to 18-month process of submitting a federal research grant proposal. Ensure your solvers engage the administrative staff at their institution or organization early in the process so that intellectual property negotiations between federal agencies and a solver organization can be more seamless.
Be clear about eligibility requirements according to the legal authority you use to run your challenge.The America COMPETES Act requires individuals to be U.S. citizens or permanent residents to win a prize. For scientific challenges this requirement can be a roadblock to success, since science is a global enterprise. Some agencies allow recognition, but not a cash prize, for foreign participants as part of a team with U.S. citizens, while other agencies do not allow foreign participants as part of the team. Do your best to include everyone in your solver pool for the greatest chance of success.
Scientific challenges typically don't fund all of the research on a given topic, but they supplement grants and contracts and generate new ideas and solutions. Technology development issues associated with science can be much more involved. Milestone payments along the way can be critical to sustaining the participation of solvers in a challenge.
Although cash prizes are very attractive to potential solvers, often nonmonetary incentives may be just as attractive. Examples of nonmonetary incentives for scientific challenges include promoting mentorship with experts not normally accessible to solvers, awarding the publication of photos in a scientific journal or lunch with a high-ranking agency official. Other reasons people will compete: recognition among peers, networking opportunities, improving skills and benchmarking technologies against others.
Subject matter experts can significantly enhance your challenge, but managing conflicts of interest can be tricky. Consult your agency's guidelines on managing conflicts of interest of outside experts who provide advice in developing the challenge or in evaluating solutions. Be prepared to deal with possible issues, especially when the competition involves an area of science where judges and potential solvers may have current or past interactions. Dealing with these conflicts may impact the judging process and the time needed for its completion.
A good way to engage outside experts in planning a challenge is through participation at scientific workshops or through public requests for information on a general matter. You can then limit external expert involvement in development of the specific challenge. Use of outside experts with brand name recognition may help attract solvers. Marvel Comics creator Stan Lee participated in the National Science Foundation's Generation Nano competition, which asked students to create superheroes with gear inspired by nanotechnology.