In this phase you'll analyze and document the results, outcomes and impact of your challenge. You'll explore ways to stay engaged with solvers. You'll also consider what to do next with high-potential solutions, whether it's moving them into an "accelerator" or exploring other avenues to transition solutions from "prizes to procurement."
Document Metrics, Results and Outcomes
When preparing for your challenge, you thought through the intended goals and outcomes of your project. When developing your challenge, you defined metrics to measure your progress. Now you'll capture how well the challenge achieved those goals and outcomes.
You're required to report certain metrics, results and outcomes every other year under the American Innovation and Competitiveness Act. But even if you ran your challenge under another legal authority, the White House Office of Science and Technology Policy encourages you to contribute data about your challenge for that report. You'll probably need to quantify your success at some level for your own agency as well.
Metrics might include:
- documentation of the incentive structure (e.g., number and amounts of prizes and non-monetary incentives)
- fees and management costs
- labor hours
- registration and submission statistics
- participation demographics
- judging structure and scoring
Results might include:
- performance impacts
- cost-benefit analyses
- subjective benefits
- ancillary impacts
Outcomes might include:
- new ideas
- actionable data about a problem
- a better-educated constituency
If you hired a vendor to manage your challenge, you may need to gather this information from that entity.
Costs and benefits go beyond prize money and solutions.
Examine the complete set of inputs and outputs to your challenge. Look at the obvious direct costs as well as indirect costs such as staff time, opportunity costs and in-kind support.
When it comes to benefits, think about advantages to your agency, the participants and the broader public. For example, a challenge may lead to changes in policy, gained skills and experience for participants and greater public awareness of an issue.
Looking at all of these aspects will help you generate a more complete cost-benefit analysis that resonates with a broad range of stakeholders.
Meeting minimum or standard reporting requirements for your agency and the required annual report to Congress keeps you compliant, but it likely won't give you the unique insight you are looking for that is tailored to your project or agency mission. Think of the lessons you'll want to have learned from the prize or challenge process in advance, so you can collect information and document the process in ways that will respond to the questions you want answered.
When you first designed your challenge you had a clear goal in mind. You also probably thought about all of the different ways you would want to look back on your program. Now is your chance to use the data you collected earlier to reflect on your process and inform future action. Data may help answer questions such as these:
- How many people participated?
- How many participants had not interacted with your organization before?
- Are you attracting solvers from specific demographics or different disciplines?
- What does the geographic distribution of participation and winners look like?
- Are there other ways you can leverage the community that has responded to the challenge?
- How much better is the winning solution than your current solution? Quantify this. Is it 10 percent better than the baseline? Two times better?
- How many people were reached by your challenge? What were the total social media impressions? Page views? Votes in an online voting activity? You should be keeping accurate counts and links of any press your prize received.
- Are there other projects or programs that could benefit from the challenge?
Imagine the prize process is like a book with many different potential endings. What information might be needed in the future? Is follow-up required? Will someone have something they expect to measure a year after the award? Determine exactly what the last chapter of your prize journey will be now and be sure to communicate that with all stakeholders and partners so assumptions about future activities do not go undiscovered. You may want to set a calendar reminder to follow up with winners in three months, six months, one year and two years to keep track of them and their solutions. It can take a year or more to see real impact for more ambitious prizes, especially those with a market stimulation element.
If you're engaging in survey activity, remember that Paperwork Reduction Act rules apply to 10 or more people. You want to use this engagement to continue to collect data and information, and you will need to find ways that this will benefit the participants.