Match Contest Rules
(The NIST Official Rules are explained below the Match Contest Rules.)
These rules pertain to the competition hosted on the Topcoder website.
In the event of any discrepancy or inconsistency between the terms and conditions of the official rules located at Challenge.gov and these Guidelines or any other Competition materials, the terms and conditions of the NIST Official Rules on Challenge.gov as specified within the Challenge Specific Agreement shall control.
1. This Challenge will tentatively run from October 31 2018 to May 6 2019 and be divided into a sequence of three matches. Each match will introduce new constraints to the scoring function, described later.
2. Upon registration for the challenge, the participants will receive one (1) or more datasets bifurcated into training and testing data in addition to instructions or pseudo code to generate the ground truth for the training set for all matches to date.
3. For match 1 the ground truth will be generated using only Clustering Analysis.
4. For match 2 the ground truth will be generated using Clustering and Classification.
5. For match 3 the ground truth will be generated using Clustering, Classification and Regression Analysis.
6. Inside the JSON submission payload, participants will receive a value of delta from NIST/Topcoder and will submit their synthetic datasets at three different values of epsilon: 0.1, 1, 10. A test harness (shell script and or Jar file) will be provided to sequentially run the participant solutions with different epsilon values that will concatenate and format the output for submission to Topcoder. Participants with algorithms that produce high quality privatized synthetic data on smaller values of epsilon, will receive higher scores.
7. Upon proper submission, participants will be redirected to the leaderboard where they will see their handle and the new score.
8. Certification Process: All participants will be required to submit a complete written explanation of their algorithm, and a clear, correct mathematical proof that their solution satisfies differential privacy, to a dedicated mailbox. This document will be reviewed by NIST staff or their delegates (who may request clarification or rewriting if documents are unclear or underspecified). Participants will receive “certification” that they have a basic understanding of differential privacy, or a brief explanation why their algorithm or proof are incorrect. An indicator will display on the leaderboard to all participants who have had their approach certified.
- For participants with higher than normal scores for their specified epsilon and delta, the participant algorithm and code must be provided as often as requested by NIST staff or their delegates who will review for non-intentional mistakes. This is a courtesy to ensure the participant still qualifies for prize eligibility.
- Certification status will be used in determining the provisional score for entries.
- Only certified approaches are eligible for cash prizes (see rule #9, Final Review).
9. Final Review: Before awarding cash prizes to participants who place in prize-winning ranks at the end of each Marathon Match, these participants must submit a report containing their complete algorithm description and mathematical privacy proof, along with their source code. They will have 7 days notification to provide this report. This report will be subject to a final review to verify that their algorithm correctly satisfies differential privacy at the stated values of delta and epsilon, and that the code correctly implements the stated algorithm. The review will be performed by Topcoder and NIST staff or their delegates, which may include NIST certified subject matter experts from outside NIST. If a participant is placed in a prize winning rank but fails to provide the report, or the review determines that their solution does not satisfy differential privacy, then they will not receive a prize, and it will be awarded to the participant with the next best performance who successfully completes the final review.
10. An indicator will be present on the leaderboard to identify submissions with intention to publicly share their solutions at the end of the final match. Having this indicator will not increase or decrease participants’ chances of winning a prize.
11. Relinquish - Topcoder/NIST is allowing registered participants or teams to “relinquish”. Government employees, students or any persons who are not allowed to take monetary prizes or feel they may have a conflict of interest are welcome to participate in the challenge while agreeing to forego the monetary prizes (by responding to the Relinquish post in the challenge forum). Relinquish means the member will compete, and their solutions scored, but they will not be eligible for a prize. Once a person or team relinquishes, we post their name to a forum thread labeled “Relinquished Competitors”. Members who relinquish must submit their implementation code and methods to maintain leaderboard status. Winners who are identified as relinquished will still be publicly recognized in the final winner announcements based on their placement on the leaderboard and the prize award, normally allotted to the placement of the relinquished team, will be given to the next certified team on the leaderboard. Throughout the challenge, the challenge’s online leaderboard will display rankings and accomplishments, giving them various opportunities to have their work viewed and appreciated by stakeholders from industry, government and academic communities.
12. All participants must register on the challenge platform, hosted by Topcoder as members. If they are already members, they must be in good standing (not banned or suspended).
13. Teaming is allowed, participants are permitted to form teams for this competition. After forming a team, participants of the same team are permitted to collaborate with each other. To form a team, a Topcoder member may recruit other Topcoder members, and register the team by completing the Topcoder Teaming Form for this challenge. Each team must declare a Captain. All participants in a team must individually register for this Competition and accept Topcoder's Terms and Conditions prior to joining the team. Team Captains must apportion prize distribution percentages for each teammate on the Teaming Form. The sum of all prize portions must equal 100%. The minimum permitted size of a team is 1 member, the maximum permitted team size is 5 members. Only team Captains may submit a solution to the Competition. Topcoder members participating in a team will not receive a rating for this Competition. Notwithstanding Topcoder rules and conditions to the contrary, solutions submitted by any Topcoder member who is a member of a team but is not the Captain of the team may be deleted and is ineligible for award. The deadline for forming teams is 11:59pm ET on the 21st day following the date that Registration and Submission opens as shown on the Challenge Details page for each match. Topcoder will prepare a Teaming Agreement for each team that has completed the Topcoder Teaming Form and distribute it to each member of the team. Teaming Agreements must be electronically signed by each team member to be considered valid. All Teaming Agreements are void, unless electronically signed by all team members by 11:59pm ET of the 28th day following the date that Registration & Submission opens as shown on the Challenge Details page. Any Teaming Agreement received after this period is void. Teaming Agreements may not be changed in any way after signature. The registered teams will be listed in the contest forum thread titled “Registered Teams”.
14. Organizations such as companies may compete as one competitor if they are registered as a team and follow all challenge and platform rules.
15. In this match participants may use any programming language and libraries, including commercial solutions, provided Topcoder/NIST are able to run it free of any charge. You may use open source languages and libraries provided they are equally free for your use, use by another competitor, or use by NIST. If your solution requires licenses, you must have these licenses and be able to legally install them in a testing VM (see “Ways to Win: Final Prizes” section). Submissions will be deleted/destroyed after they are confirmed. Topcoder/NIST will not purchase licenses to run your code. Prior to submission, please make absolutely sure your submission can be run free of cost, and with all necessary licenses pre-installed in your solution. Topcoder/NIST are not required to contact submitters for additional instructions if the code does not run. If we are unable to run your solution due to license problems, including any requirement to download a license, your submission might be rejected. Be sure to contact us right away if you have concerns about this requirement.
16. If the solution includes licensed software (e.g. commercial software, open source software, etc), participants must include the full license agreements with the submission. Include licenses in a folder labeled “Licenses”. Within the same folder, include a text file labeled “README” that explains the purpose of each licensed software package as it is used in your solution.
17. External datasets and pre-trained models are allowed for use in the competition provided the following are satisfied: (a) The external data and pre-trained models are unencumbered with legal restrictions that conflict with its use in the competition. And (b) The data source or data used to train the pre-trained models is defined in the submission description.
NIST Official Rules (Challenge-specific Agreement)
PLEASE READ THIS CAREFULLY! You ("Innovator” or “Participant”) and NIST ("Challenge Sponsor”) are entering into this Challenge-Specific Agreement ("CSA”) for this particular incentive-based competition ("Challenge”) only. In order to participate in this Challenge, Innovator must accept these terms, and therefore should take the time to understand them. This CSA includes the NIST Official Rules.
In the event of any discrepancy or inconsistency between the terms and conditions of the official rules and disclosures or other statements contained in any Competition materials, including but not limited to the Competition submission form, Competition website and use terms, Topcoder terms of participation, advertising (including but not limited to television, print, radio or online ads), the terms and conditions of the NIST Official Rules on Challenge.gov website as specified within this Challenge Specific Agreement shall control.
2. America COMPETES Reauthorization Act of 2010: All challenge and prize competitions shall be performed in accordance with the America COMPETES Reauthorization Act of 2010, Pub. Law 111-358, title I, § 105(a), Jan. 4, 2011, as amended, codified at 15 U.S.C. § 3719 (hereinafter “America COMPETES Act”).
3. Eligibility: Each Competition Participant (individual, team, or legal entity) is required to register on the Topcoder NIST Differential Privacy Synthetic Data Challlenge [Unlinkable Data Challenge-Stage 2] website. There shall be one Official Representative for each Competition Participant. The Official Representative must provide a username (which may serve as a team or affiliation name), email address, and affirm that he/she has read and consents to be governed by the Competition Rules. At NIST’s discretion, any violation of this rule will be grounds for disqualification from the Competition. Multiple individuals and/or legal entities may collaborate as a team to submit a single entry, in which case the designated Official Representative will be responsible for meeting all entry and evaluation requirements. Participation is subject to all U.S. federal, state and local laws and regulations. Participants, including individuals and private entities, must not have been convicted of a felony criminal violation under any Federal law within the preceding 24 months and must not have any unpaid Federal tax liability that has been assessed, for which all judicial and administrative remedies have been exhausted or have lapsed, and that is not being paid in a timely manner pursuant to an agreement with the authority responsible for collecting the tax liability. Participants must not be suspended, debarred, or otherwise excluded from doing business with the Federal Government. Individuals entering on behalf of or representing a company, institution or other legal entity are responsible for confirming that their entry does not violate any policies of that company, institution or legal entity. Any other individuals or legal entities involved with the design, production, execution, distribution or evaluation of the NIST Differential Privacy Synthetic Data Challenge are not eligible to participate.
To be eligible for a cash prize:
a. A Participant (whether an individual, team, or legal entity) must have registered to participate and complied with all the requirements under section 3719 of title 15, United States Code as contained herein.
b. At the time of Entry, the Official Representative (individual or team lead, in the case of a group project) must be age 18 or older and a U.S. citizen or permanent resident of the United States or its territories.
c. In the case of a private entity, the business shall be incorporated in and maintain a primary place of business in the United States or its territories.
d. Participants may not be a Federal entity or Federal employee acting within the scope of their employment. NIST employees are not eligible to participate. Non-NIST Federal employees acting in their personal capacities should consult with their respective agency ethics officials to determine whether their participation in this Competition is permissible.
e. A Participant shall not be deemed ineligible because the Participant consulted with Federal employees or used Federal facilities in preparing its submission to the NIST Differential Privacy Synthetic Data Challenge prize competition if the Federal employees and facilities are made available to all Participants on an equitable basis.
4. Submissions: By participating in this Challenge, Innovator may submit to Challenge Sponsor submission materials ("Submission”), as outlined in these NIST Official Rules on Challenge.gov and the Challenge Guidelines specific to this Challenge on Topcoder.com. By submitting a Submission, Innovator thereby agrees to provide reasonable assistance and additional information concerning the Submission to Challenge Sponsor, if requested.
5. Warranties: By submitting an Entry, the Participant represents and warrants that all information submitted is true and complete to the best of the Participant’s knowledge, that the Participant has the right and authority to submit the Entry on the Participant’s own behalf or on behalf of the persons and entities that the Participant specifies within the Entry, and that the Entry (both the information and materials submitted in the Entry and the underlying technology/method/idea/treatment protocol/solution described in the Entry):
a. Is the Participant’s own original work, or is submitted by permission with full and proper credit given within the Entry;
b. Does not contain trade secrets (the Participant’s or anyone else’s);
c. Does not knowingly violate or infringe upon the patent rights, industrial design rights, copyrights, trademarks, rights of privacy, publicity or other intellectual property or other rights of any person or entity;
d. Does not contain malicious code, such as viruses, malware, timebombs, cancelbots, worms, Trojan horses or other potentially harmful programs or other material or information;
e. Does not and will not violate any applicable law, statute, ordinance, rule or regulation, including, without limitation, United States export laws and regulations, including but not limited to, the International Traffic in Arms Regulations and the Department of Commerce Export Regulations; and
f. Does not trigger any reporting or royalty or other obligation to any third party.
6. Intellectual Property: Any applicable intellectual property rights to an Entry will remain with the Participant. By participating in the prize competition, the Participant is not granting any rights in any patents, pending patent applications, or copyrights related to the technology described in the Entry. However, by submitting an Entry, the Participant is granting NIST, NASA, and any parties acting on their behalf certain limited rights as set forth herein.
a. By submitting an Entry, the Participant grants to NIST, NASA, and any parties acting on their behalf the right to review the Entry, to describe the Entry in any materials created in connection with this competition, and to screen and evaluate the Entry. NIST and NASA, and any parties acting on their behalf will also have the right to publicize Participant’s name and, as applicable, the names of Participant’s team members and/or Organization which participated in submitting the Entry following the conclusion of the Competition.
b. As part of its submission, the Participant must provide written consent granting NIST, NASA, and any parties acting on their behalf, a royalty-free, non-exclusive, irrevocable, worldwide license to display publicly and use for promotional purposes the Participant’s entry (“demonstration license”). This demonstration license includes posting or linking to the Participant’s entry on NIST and NASA’s websites, including the Competition Website, and partner websites, and inclusion of the Participant’s Entry in any other media, worldwide.
7. Trade Secret Information: By making a submission to this prize competition, the Participant agrees that no part of its submission includes any Trade Secret information, ideas or products. All submissions to this prize competition are deemed non-proprietary. Since NIST does not wish to receive or hold any submitted materials “in confidence” it is agreed that, with respect to the Participant’s Entry, no confidential or fiduciary relationship or obligation of secrecy is established between NIST, NASA, or any parties acting on their behalf and the Participant, the Participant’s team, or the company or institution the Participant represents when submitting an Entry, or any other person or entity associated with any part of the Participant’s Entry.
8. Liability: Participants shall agree to assume any and all risks and waive claims against the Federal Government and its related entities, except in the case of willful misconduct, for any injury, death, damage, or loss of property, revenue, or profits, whether direct, indirect, or consequential, arising from participation in this prize competition, whether the injury, death, damage, or loss arises through negligence or otherwise.
9. Insurance: Participants are not required to obtain liability insurance for this Competition.
10. Indemnification: Participants shall agree to indemnify the Federal Government against third party claims for damages arising from or related to Challenge activities.
11. Changes and Cancellation: Challenge Sponsor has the right to make updates and/or make any changes at any time during the Challenge. Innovators are responsible for regularly reviewing the official rules on Challenge.gov website and updates on the Topcoder site to ensure they are meeting all rules and requirements of the Challenge. Challenge Sponsor has the right to cancel the Challenge at any time, without warning or explanation, and to subsequently remove the Prize completely.
12. Payments: The prize competition winners will be paid prizes directly from NIST. Prior to payment, winners will be required to verify eligibility. The verification process with the agency includes providing the full legal name, tax identification number or social security number, routing number and banking account to which the prize money can be deposited directly.
13. Existing Laws: The Federal Government shall not, by virtue of conducting this prize competition, be responsible for compliance by Participants in the prize competition with Federal law, including licensing, export control, and nonproliferation laws, and related regulations.
Participation is subject to all U.S. federal, state and local laws and regulations. Participants are responsible for checking applicable laws and regulations in their jurisdiction(s) before participating in the prize competition to ensure that their participation is legal. Individuals entering on behalf of or representing a company, institution or other legal entity are responsible for confirming that their entry does not violate any policies of that company, institution or legal entity.
14. Registration and Submissions: Submissions must be made online (only), via upload to the Topcoder website, at the specific dates and times listed in the Summary of Important Dates on Challenge.gov website and Topcoder site. No late submissions will be accepted.
15. Selection of Winners: Based on the winning criteria, prizes will be awarded per the Judging Criteria section in the Challenge Guidelines. In the case of a tie, the winner(s) will be selected based on the highest votes from the judges.
16. Judging: The final determination of the winners will be made at the sole discretion of NIST. Scores and feedback from NIST will not be shared.
17. Progressive Prize Awards: There are multiple ways for Challenge leaders to win progressive prizes halfway through each marathon match and prior to the final round. Progressive prizes are announced when the Challenge is publicly launched and awarded based on the participants placement on the provisional leaderboard at an exact time (approximately half way through the challenge), and their provisional testing score (not the sequestered results). These prizes are awarded to the Top 4 pre-certified participants from the provisional leaderboard once their code has been inspected. There are 4 progressive prizes available for each marathon match at $1000 each.
a. A Marathon Match is defined as a long-running crowdsourcing challenge, using the Topcoder website, where participants solve complex algorithms on large data sets. Participants submit their software code to create and optimize their algorithm.
b. A Leaderboard is an online, public display of the competition on the Topcoder website. Each marathon match will have a leaderboard where participants will receive immediate feedback about the quality of their submission. Participants can repeatedly improve their score and validate the capabilities of their algorithm by viewing the leaderboard. The leaderboard will display the participants handle, ranking and provisional score, at any given point in time. Additionally, the leaderboard will identify those teams or individuals who agree to make their submission (code) publicly available at the conclusion of this challenge; this is an option, not required to participate.
18. Additional Prize Awards: Additional funds are available to the NIST Judge panel, in the form of Additional Prizes, up to $20,000 and assessed at the end of the final marathon match. An additional prize of $4000 may be awarded to each of the top 5 award winning teams who agree to provide and do provide their full code solution in an open source repository for use by all interested parties.
19. Privacy Advisory: The Topcoder.com website is hosted by a private entity and is not a service of NIST. The solicitation and collection of your personal or individually identifiable information is subject to the host’s privacy and security policies and will not be shared with NIST unless you win the Challenge. Challenge winners’ personally identifiable information must be made available to NIST in order to collect an award.