The event website and official rules are posted at http://www.nist.gov/ariac.
Eligibility to Participate
Participation in the ARIAC Prize Competition is open to ALL; however, not all participants are eligible to win cash prizes as explained in the next section.
Each Competition Participant (individual, team, or legal entity) is required to register on the ARIAC website. There shall be one Official Representative for each Competition Participant. The Official Representative must provide a name, username (which may serve as a team or affiliation name), email address, and affirm that he/she has read and consents to be governed by the Competition Rules. Multiple registrations per Participant are NOT allowed. At NIST’s discretion, any violation of this rule will be grounds for disqualification from the Competition. Multiple individuals and/or legal entities may collaborate as a team to submit a single entry (Participant’s code submission to a qualifier or final), in which case the designated Official Representative will be responsible for meeting all entry and evaluation requirements. Participation is subject to all U.S. federal, state and local laws and regulations. Participants must not be suspended, debarred, or otherwise excluded from doing business with the Federal Government. Individuals entering on behalf of or representing a company, institution or other legal entity are responsible for confirming that their entry does not violate any policies of that company, institution or legal entity. Any other individuals or legal entities involved with the design, production, execution, distribution or evaluation of ARIAC are not eligible to participate.
Once registered, Participants will have access to the interfaces needed to participate in the competition through the ARIAC website. NIST will create an account to allow the Participants to upload their submissions to the qualifier. Modifications and improvements to a Participant’s control system during the qualifying period are to be expected, but the Participant should have one final control system ready for the final competition.
Elibility to Win a Cash Prize
To be eligible for a cash prize:
- A Participant (whether an individual, team, or legal entity) must have registered to participate and complied with all of the requirements under section 3719 of title 15, United States Code as contained herein.
- At the time of Entry, the Official Representative (individual or team lead, in the case of a group project) must be age 18 or older and a U.S. citizen or permanent resident of the United States or its territories.
- In the case of a private entity, the business shall be incorporated in and maintain a primary place of business in the United States or its territories.
- Participants may not be a Federal entity or Federal employee acting within the scope of their employment. NIST employees are not eligible to participate. Non-NIST Federal employees acting in their personal capacities should consult with their respective agency ethics officials to determine whether their participation in this Competition is permissible.
- A Participant shall not be deemed ineligible because the Participant consulted with Federal employees or used Federal facilities in preparing its submission to the ARIAC Prize Competition if the Federal employees and facilities are made available to all Participants on an equitable basis.
In addition, interested Participants who do not meet the eligibility requirements to win a prize (i.e., individuals who are neither a US citizen nor a permanent resident of the United States or non-US-based entities) are encouraged to participate in the Competition. They are invited to register on the ARIAC website and download the training material. The performance obtained by these Participants will be displayed on the ARIAC website in the same manner as the performance obtained by Participants who are eligible to win cash prizes.
The NIST Director will appoint a panel of three qualified judges. The Judges will be robot agility and/or simulation experts from inside and/or outside of NIST. The Judges will determine winners according to the Judging Criteria described herein. The Judges may not have personal or financial interests in, or be an employee, officer, director, or agent of, any entity that is a registered Participant in the Competition and may not have a familial or financial relationship with an individual who is a registered Participant. In the event of such a conflict, a Judge must recuse himself or herself and a new Judge may be appointed.
Basis on which Finalists and Winners Will Be Selected
A portion of the scores will be automatically calculated, shared with the Participants, and publicly posted for each Trial at the conclusion of the Finals (after all Trials are run) as a combination of cost and performance metrics, described below in the Trial Score Calculation section. A Participant’s final automated score is the sum of scores for each of the 15 competition Trials.
The simulation environment is a representation of an industrial kitting work cell with two robot arms, a conveyor belt, part bins, and trays.
The conveyor belt is a 1 m wide plane that transports objects across the work environment at a fixed speed of roughly 0.2 m/s. Parts continuously appear on the belt for the duration of the trial. When parts reach the end of the conveyor belt they are automatically removed. Teams can control the conveyor belt during development, but not during the final competition.
There are eight part bins that may be used for building kits. Parts in these bins will not be replaced once used.
There are two robot arms mounted on a linear actuator that operates parallel to the conveyor belt. The linear actuator measures 4 m.
Two automated guided vehicles (AGV) are located at either end of the linear actuator. Kits are built on top of these AGVs. A team will programmatically signal the AGVs when the kits are ready to be taken away. The signaled AGV will depart for a short period and then return with an empty tray.
The trays used for assembling kits are flat trays measuring 0.5 x 0.7 m.
The two robot arms used in each trial will be Universal Robots UR10.
The robot arms’ position is controlled through the linear actuator on which it is mounted.
The end of the arms are equipped with a vacuum gripper. The vacuum gripper is controlled in a binary manner and reports whether or not it is successfully gripping an object.
A team can place sensors around the environment. Each sensor has a cost that factors into the final score. Available sensors are:
- Break beam: reports when a beam is broken by an object. It does not provide distance information.
- Laser scanner: provides an array of distances to a sensed object.
- Cognex logical camera: provides information about the pose and type of all models within its field of view.
- Proximity: detects the range to an object.
An order is an instruction containing kits for the robot system to complete. Each order will specify the kit to be assembled, i.e. the list of parts to be put in the kit. Each specified part has the following structure:
- The type of part.
- The position and orientation of the part on the tray.
Above each AGV is a quality control sensor that detects faulty parts. If faulty parts are detected while teams are filling trays, those parts should be removed from the tray and replaced with another part of the same type. Faulty parts are considered unwanted parts: they will not count for any points when the kit is submitted, and they will cost teams the all-parts bonus if left in trays.
Scores will be automatically calculated for each trial as a combination of performance metrics and costs.
Each trial will consist of the following steps:
- The robots programmatically signal that they are able to begin accepting orders.
- The first Order (Order 1) is sent to the robots.
- A fixed amount of time is allowed to complete the order.
- In the case of the Dropped Part testing method, up to three parts will be forcibly dropped from the gripper.
- In the case of the In-Process Kit Change testing method, a new Order (Order 2) will be issued that is of higher priority than the previously issued Order 1. When Order 2 is complete, building of Order 1 is to resume.
- The robots signal programmatically when a kit is complete and ready for quality control.
- The robot system will be notified that the trial is over. The trial is over when time runs out or all Orders have been fulfilled.
The following values are calculated for a Participant’s system setup. As Participants are to use the same system setup for all Trials, the values will remain unchanged between Trials.
The choices made by the competitors (choices of sensors) will have a specific cost associated with them. The final cost for individual sensors will be made available to the Participants at least two weeks before the Finals. For planning purposes, Participants should use:
- $500 for each logical camera used.
- $200 for each depth camera used.
- $100 for each other sensor used (e.g., break beam, proximity, laser scanner).
The sum of these cost choices made by each team will be divided by a baseline cost of $1700 and made into a cost factor for the scoring. The team costs are designated as TC, and the baseline cost is designated as BC. The cost factor, is then calculated as:
Performance metrics that cover both completion and efficiency are calculated for each Trial separately. Completion captures the quality of the orders fulfilled (that the boxes contain the correct products in the correct position/orientations); efficiency captures the responsiveness in fulfilling orders (that the orders were filled quickly).
Each trial, since it has the option to have more than one order, will have a certain set of scores for each order. Each order consists of a list of products needed and a prescribed position and orientation for each product within the box. The completion score, CS, is a combined score for each order, starting with 1 point for each product that is placed in the box. For each product that is within 3 cm of the correct position, and within 0.1 radians of the correct orientation will result in an additional point. Next, since the main focus of the competition is to have fulfilled orders, any order with all of the products in the confines of the box and in the correct position/orientation receives an additional point for each product. So, for an order sj with i products in it would have a maximum score of:
The Efficiency of the approaches chosen by the teams are accounted for in the scoring by using the times that each team takes to complete each trial. The times are counted in seconds from the start of the trial until the first kit is completed and is designated as T01. In the trials with multiple orders, the time, Tj is counted in seconds from the time that the jth order is sent to the team. For each trial, an average of the times for each team (ATj) is compared to the individual team time to calculate the Efficiency Factor:
If a team’s system times out for a trial, the efficiency factor for the trial is set to 0 and the trial time is not used to calculate the average times for that trial.
For the multi-order trials where one of the orders is a high priority order, an additional multiplier, h, is applied with the efficiency factor to give a higher proportion of the score to that order, which is initially set to a value of 3. The final h value will be made available to the Participants at least two weeks before the finals.
Trial Score Calculation
All of the scoring factors described above are combined as shown below to calculate the score for each trial:
Final Score Calculation
The final overall score is the equally weighted sum of all trial scores. As mentioned in the Competition Scenarios section, trials are expected to focus on the Baseline Kit Building, Dropped Part, and In-Process Kit Change scenarios, among others.
The Winners of cash prizes will be determined by three Judges appointed by the NIST Director, using the Judging Criteria outlined herein.
NIST will announce the Finalists via the Event Website as well as those Finalists that have been awarded a cash award (each, an “Award”). The anticipated number and amount of the Awards that will be awarded for this Competition is set forth in these rules; however, NIST is not obligated to make all or any Awards, and reserves the right to award fewer than the anticipated number of Awards in the event an insufficient number of eligible Participants meet any one or more of the Judging Criteria for this Competition, based on NIST’s sole discretion. Using the Judging Criteria described herein, three qualified Judges (appointed by the NIST Director) will determine the winners of the ARIAC Prize Competition.
The winner verification process for being eligible to receive an Award includes providing the full legal name, tax identification number or social security number, routing number and banking account to which the prize money can be deposited directly. Return of any notification as “undeliverable” will result in disqualification. The verification form must be returned within seven (7) calendar days of receipt. After verification of eligibility, Awards will be distributed in the form of electronic funds transfer addressed to the Official Representatives specified in the winning Entries. That Official Representative will have sole responsibility for further distribution of any Award among Participants in a group Entry or within a company or institution that has submitted an Entry through that representative. The list of Entries receiving Awards for the Competition, including the names of all members of a team, will be made public according to the timeline outlined on the Event Website.
Winners are responsible for all taxes and reporting related to any Award received as part of the Competition.
All costs incurred in the preparation of Competition Entries are to be borne by Participants.
Presentation of Winning Approaches at a To-Be-Determined Workshop in a Robotics Conference
Up to three Winners of the ARIAC Competition may be invited by NIST to attend a workshop dedicated to the ARIAC Competition at a future TBD robotics conference. The Winners will be required to make a 25-minute presentation on their approach in the ARIAC Competition followed by audience question and answer.
NIST may arrange and pay for the travel for one team-selected representative from each of the winning Participants to cover eligible travel costs such as airfare, conference registration fee, and lodging.
Point of Contact for the Competition
Questions about the ARIAC Prize Competition can be directed to William Harrison, firstname.lastname@example.org.