In this phase you will analyze and document the results, outcomes and impact of your incentivized competition. You will evaluate avenues for remaining engaged with your solvers as well as and next steps for high-potential solutions, whether moving them into an "accelerator" or exploring other channels for transitioning prize solutions to procurement.
Document the Challenge
Document the results of your challenge by wrapping all of what you've learned into a meaningful form. Collect impact stories, document lessons learned, consider commissioning independent assessments of the challenge and analyze metrics from the previous step. It is important to capture much of this information as quickly as possible while it is still fresh in participants' and implementers' minds. Products could include case studies for this toolkit; other communication materials that summarize the impact of the challenge and spotlight your solvers; and independent assessments or research papers that assess the impact of the challenge and even compare it against more traditional problem-solving approaches. Challenge documentation should also include any federal records that were generated as a result of the competition. A record of the evaluation and selection process should also be generated.
Conducting an "after-action" meeting once a challenge concludes is a best practice. You should have this scheduled and on your team's calendar before the challenge concludes. This helps capture institutional knowledge so the agency can improve challenges in the future and also helps the challenge manager complete required annual reporting processes. Discuss lessons learned related to questions like:
- What could have been improved in judging, communications and operations?
- Did the evaluation process result in the selection of the best submissions?
- How might agency clearance and coordination go more smoothly next time?
- What would you have done differently in challenge design looking back?
- What worked well?
- What were any unintended consequences, both positive and negative?
- If you used the toolkit to help develop the challenge, are there additional insights you would add to the content? Where was the toolkit not helpful?
These lessons shouldn't be written up and shared with your agency's prize point of contact. Consider packaging these lessons as a case study for inclusion in this toolkit. You could also consider sharing these lessons in additional ways with other communities as discussed in the "Share Best Practices and Results" step.
Ask for feedback whenever it is relevant. Collect quotes, insights and metrics from your partners, judges, winners, stakeholders and others in the challenge process. Doing this consistently and consciously means that you will have more to work with when developing an assessment of the impact of your challenge. It is also worth noting that qualitative feedback often is critical. For example, did your senior executive think the prize hit the mark or evolved how your organization does business? Capture a quote or feedback around those insights that you can use inside and outside of your organization.
By reaching out to a new, ambitious and excited community of solvers, you may receive plenty of feedback about other related opportunities your agency may wish to explore. Be sure you share the feedback with those who could take action on it. For example, the Intelligence Advanced Research Projects Activity continued to receive interest in using the data after the INSTINCT Challenge was over, which has led to a discussion of how to make data publicly available in the future.
Documenting the impact of your challenge is about way more than data. In fact, images, videos, stories and infographics can be much more powerful subjective descriptions of impact. This is a moment to gather and write down the impact stories that you'll later tell about your challenge. What quotes about impact do you have from participants in the challenge? From users of the wining solution? From the agency customer? Take a moment to ask partners, winners, participants and stakeholders questions to understand and validate the perception and insights. NASA keeps up with challenge winners and participants through its "after the challenge" content. Challenge.gov will provide another way to highlight solvers in your challenge through its success stories page by interviewing winners. Make sure to tell these stories from your challenge in any required annual reporting you do in the next step.
You just spent a lot of time and effort pursuing an exciting approach to addressing issues for your organization in new and creative ways. In that process you created numerous program and implementation plans, communication materials, evaluations, stories and other documents. You should compile these resources on a shared drive or in another location for future challenge managers. Make sure that all of the descriptive documents, press releases, one pagers, social media tool kits and analytics are collected and organized. Keep your communications plan, checklists, tweet-sheets, website links, press releases and other elements somewhere easy to access. A record of the evaluation and selection process should also be kept here. Be sure to include copies of documents that your partners used and created.
As a federal employee, you may be asked to participate in an internal or external audit focused on the activity, financials and performance of your challenge. If you are thorough now in collecting and maintaining your information and telling a clear story, it will be easier to work with a team that wants to review your information later.
Challenge design is still a craft that that is being learned by challenge managers across sectors. It's important to take an objective, retrospective look at the impact of a challenge in order to make recommendations for process improvements the next time. Agencies like the Department of Health and Human Services and the Department of Housing and Urban Development have supported the creation of third-party reports that analyze the impact of challenges. For example, AgResults is a $118 million multi-donor, multi-lateral initiative that uses pull mechanisms like prizes to incentivize and reward high-impact agricultural innovations that promote global food security, health and nutrition. AgResults has contracted with an external evaluator that provides lessons learned throughout the multi-year implementation of this program from an evaluator's perspective.
Also, some academics are studying the science of open innovation. This field looks at how and when open innovation methods can be best applied to improve results. These academic institutions will study one or more challenges to understand what worked and what didn't, how the approach differed from more traditional models and more. Academic research into the conduct and outcomes of your challenge may take months or years to produce a report, but it moves the knowledge of the entire field forward.