In this phase you'll determine the challenge structure and implementation timeline. During this critical step, you'll work with internal groups to establish eligibility and submission requirements, terms and conditions, and judging criteria. You'll connect with your communications team to outline your announcement and ongoing outreach strategy to engage potential solvers.
Define Evaluation and Judging Process
The appropriate evaluation and judging process can vary widely depending on the challenge type. Your particular problem might lend itself to solutions that can be judged objectively (e.g., the fastest possible algorithm) or with subjective criteria based on expert opinion. If a challenge is designed to achieve multiple outcomes (e.g., technological performance and usability), it may include both subjective and objective evaluation.
In this step, you'll establish the process to evaluate submissions. This includes defining specific criteria; identifying judges, subject matter experts and reviewers; and writing instructions to ensure a fair, transparent process and methodologically sound outcome.
Depending on the complexity of the challenge and criteria, you may need to develop a scoring function that uses performance data generated during the evaluation processes. Make sure your scoring function actually supports the type of innovation you want. Depending on the legal authority under which you're running the challenge, you may be able to use judges from other government agencies, foundations, academia or even industry.
The judging process refers to the evaluation of the challenge criteria. This can be done in person or virtually depending on the challenge type. Ideation challenges generally are judged virtually using a review template with specific questions related to the evaluation criteria. Technology solution challenges may involve virtual assessment to select the best ideas followed by in-person judging to assess performance.
Include your judges early in the development of selection criteria. Their technical expertise can lead to an easier and more transparent selection process, both for them and the participants.
When developing judging criteria, first decide on an appropriate judging mechanism. This is determined by the competition topic and the policies and practices of the sponsoring agency.
Consult with appropriate agency staff (legal, operations, finance, subject matter experts, etc.) to determine if there are agency restrictions on the use of internal or external mechanisms. Use the challenge format to guide the specific system selection (subjective, quantitative, algorithm, etc.) Establish and adhere strictly to a well-defined judging mechanism. This gives you a more legally defensible basis for judging the competition and better safeguards your agency from potential litigation.
Your agency and legal authority both may have specific requirements on who can be a judge. Some agencies require outside judges. Others require judges to be federal staff or special government employees.
Fair and transparent judging criteria are essential. State all criteria clearly in your challenge announcement and apply it to each submission. You can't use any criteria not included in the challenge announcement to judge submissions.
Your criteria can't create an unfair advantage for any group or individual. For example, don't require the use of resources that aren't public or that all participants can't equally access.
The judging process is also shaped by what kinds of criteria the judges use to evaluate submissions. These criteria could be quantitative or qualitative, analytical or semi-empirical, objective or subjective.
Craft each evaluation criterion so it can be measured. The judge or measurement system must be able to determine if each criterion has been met.
Solutions will have many different assets with varying degrees of effectiveness. Calibrate the measurement scale for each criterion to reward exceptional results in a way that demonstrably links to the stated problem.
Depending on the criteria, judges may be required to reach a consensus or majority vote. Also, you may require your judging panel to meet in person depending on the process.
If your challenge requires subjective judgment, develop an assessment protocol. Train all judges to use the protocol before judging begins. The protocol must be used when judging every submission. Determine the number of reviews to be conducted for each submission and define a procedure for handling anomalies, such as a tie. Consider designating one judge as the final selecting official in a tie-break scenario.
It's critical to review procedures and expectations for in-person judging scenarios. Establish any roles for the process, such as judge panel foreman, liaison with the challenge manager, etc.
Follow the judging protocol you develop in this step to ensure fairness and transparency. It also will help you defend your challenge should participants question the validity of awards. The Federal Trade Commission had to do just that in 2015 when a participant who didn't win its Robocall Challenge appealed the decision to both the Government Accountability Office and the U.S. Court of Federal Claims seeking damages.
Be prepared to deal with possible conflicts of interest, especially when your challenge is in a field where judges may have interacted with potential solvers.
Seek input from your legal counsel and address potential conflicts before formally selecting your judges. The nature of the conflicts and how you deal with them can have an impact on how you structure the judging process and time needed to evaluate submissions.
Make sure judges sign any necessary non-disclosure agreements. Those coming from government agencies may already be bound by agreements specifying governmental responsibilities. But you may need different kinds of non-disclosure agreements for judges employed by other entities. Consult your legal counsel to determine what agreements are necessary.
Ask your legal counsel also about managing conflicts of interest involving outside experts who provide advice in helping you develop the challenge.
2.4 Define Evaluation and Judging Process (current page)