In this phase you'll put your plans into action. You'll roll out your communications plan, accept submissions and interact with solvers to generate interest and enthusiasm throughout. After your submission deadline, you'll begin to evaluate entries, select winners and verify their eligibility. Successful execution of this phase is critical to maintain credibility of the challenge and your agency.
Manage the Judging Process and Select Winners
A challenge can seem like a feel-good, win-win situation for both your agency and participants. At the end of the day, though, it's still a competition. And competitors expect and deserve a fair, transparent process for deciding winners.
It's crucial that you select and appoint only qualified judges to ensure an environment where quality and ingenuity are recognized above all else.
The judging process can take place in person or remotely. Judges may gather on a conference call or via webinar. No matter the setting, you should document key decisions and stick to the evaluation guidelines you defined when you developed submission requirements.
An evaluation process can be objective, subjective or a combination of the two. Competitors should know how they'll be evaluated before they submit solutions. It's useful to provide all participants with feedback from judges.
You can even divide the judging process into stages. This makes sense if there's a large number of submissions to prioritize before a final panel reviews them.
Note: You may need to have judges sign forms, such as nondisclosure agreements and disclosures of conflicts of interest.
Also, your selection officials may or may not be your judges. It all depends on how your agency administers challenges.
Your judges can make or break your challenge.
You worked hard to attract solvers with different skill sets and backgrounds, and you don't want their innovations to be overlooked or dismissed during the judging process.
When selecting judges, consider their background, expertise and ability to work on a team. Recognize them as members of a team even if they work remotely and never meet in person. They should understand the goals of the challenge and be open to new ideas proposed in solver submissions.
Judges should take part in constructive debate about the submissions. It depends on your challenge, but you may want your team of judges to reflect a range of expertise and interests. This way, submissions are considered from a variety of perspectives. If your panel consists of five engineers and one social scientist, it's likely the latter will see something the others don't.
Pay attention to any restrictions your agency or chosen legal authority places on who can serve as a judge. Some legal authorities require judges to be from outside your agency. Others require them to be federal staff or special government employees.
You also can allow members of the public to vote on submissions. This can generate excitement and strengthen the sense of community around your challenge and topic area. You can use public voting to complement or even in place of a more formal judging team.
Keep in mind that the number of submissions you receive will dictate to some extent the number of judges you need. This is especially true for paper submissions.
Put together a list of backup judges who can help if you get more submissions than anticipated. Call these backup judges into action if you sense your main judging panel is overwhelmed.
Your judges might be outside experts. You might pay them or they might volunteer. Whatever the situation, your challenge could require them to spend significant amounts of time evaluating submissions.
Communicate upfront the time and effort expected. And then think about how you can make it worth their while.
Just as you incentivize your solvers, you want to provide reasons for why an expert would want to join your judging panel. Consider non-monetary benefits, especially if you're looking for volunteer judges.
Will your judges receive significant publicity or widespread acceptance and recognition in their fields? Will they make professional connections?
Know the answers to these questions before you approach potential judges. This will help you retain them. You don't want judges to commit and then drop out, especially if you're publicizing their involvement in the challenge from the get-go.
And that's something you'll want to do if you have judges with name recognition and who wield a great deal of influence. Having high-quality judges will attract more participants to your challenge.
The National Science Foundation's Generation Nano competition asked students to create super heroes based on actual research in science and technology. It's safe to say that the middle and high schoolers who entered the competition took note of Marvel Comics creator Stan Lee's involvement.
A fair and transparent judging process is crucial to maintaining the integrity of your challenge. It's also they key to defending the results of the competition in the face of any dispute.
Familiarize your judges with the process you developed in Phase 2. They need to recognize the power they have in deciding who advances in and wins your challenge.
Issues and topics can be debated ad nauseam, and there will be differences of opinion. But judges ultimately have to make a recommendation or score based on the process you designed. Be prepared to step in and mediate any issues that arise during difficult or tense periods in the judging process.
Strike a balance between structure and openness. You want to provide a framework in which your judges can make solid recommendations, but you also want to be prepared for submissions that come out of left field. Just because they weren't on anybody's radar doesn't mean they won't solve your problem.
Again, follow your previously defined judging protocol as you may need to defend the challenge if participants dispute award decisions. The Federal Trade Commission recently won a case where a participant who didn't win appealed the decision to both the Government Accountability Office and the U.S. Court of Federal Claims seeking damages.
Finally, people spend a lot of time, energy and even money to participate in challenges. Even if they don't win, make it worth their while. When judges and experts offer comments and feedback on submissions, pass that information onto the individual solvers so they can continue to refine and improve their solutions.
Your rules may call for submissions all at once or break the process into different parts. Sometimes judges have a small amount of material to examine. Other times, they have a great deal to go through.
Allow your judges enough time to evaluate submissions and meet deadlines established in your rules.
Develop workflow or Gantt charts to know when judges need to have certain tasks completed. Block time on their calendars for evaluation and panel meetings well in advance.
A challenge differs greatly from a traditional grant. Your agency leadership may have little to no role in deciding who advances in and wins your challenge. That decision could rest entirely with the judges as is the case with many NASA Centennial Challenges.
The American Innovation and Competitiveness Act states: "For each competition, the head of an agency, either directly or through an agreement … shall appoint one or more qualified judges to select the winner or winners of the prize competition."
Because the judging process can occur completely outside of your agency's normal operating procedures, you need to educate leaders on the authority your judges have in selecting winners for the challenge.
Still, some legal authorities and agencies require a government official to make the final decision about who wins a competition. Make sure you know your agency's policies for who must approve the winners.
If the approver is at the secretary level, allow yourself enough time to get on his or her calendar to obtain approval. You don't want to hold an award ceremony or make a winner announcement without that approval.
Document how decisions were made during the judging process so you can explain and justify to agency leaders.
We want to reiterate the point we just made.
Depending on your agency, the judges may not be making the ultimate decision about who wins your challenge. This scenario is more likely if your judges come from outside your agency. In some cases, a federal employee may be required to make the final winner selections based on the judges' evaluations.
Make sure you know the protocol you need to follow regardless of who has the final say. And even if they aren't making the final decision, it's still a good idea to brief agency leaders about the winners.
Standardize Your Challenge Performance Evaluation Process: The judging of challenge performance must be consistent with the evaluation factors and terms and conditions you developed in Phase 2.
Consider using common templates and a repeatable process for evaluation and documentation of solution performance. This process should take into account strengths, weaknesses, deficiencies and risks of each solution to support a technical evaluation.
Consult with your legal and contracting offices to determine if this evaluation process can be used to support the past performance requirement for a follow-on Federal Acquisition Regulation (FAR)-based acquisition. For this, it would have to establish the "currency and relevance" of the solver to meet the government's need and clearly indicate the "offeror's ability to perform."