Announcement of Requirements and Registration for “Pill Image Recognition Challenge”

Authority: 15 U.S.C. 3719

SUMMARY:

The Pill Image Recognition Challenge is a National Institutes of Health (NIH) Challenge under the America COMPETES (Creating Opportunities to Meaningfully Promote Excellence in Technology, Education, and Science) Reauthorization Act of 2010 (Pub. L. 111-358). Through this Challenge, the National Library of Medicine (NLM), part of NIH, seeks algorithms and software to match images of prescription oral solid-dose pharmaceutical medications (pills, including capsules and tablets). The objective of the Challenge is the development and discovery of high-quality algorithms and software that rank how well consumer images of prescription pills match reference images of pills in the authoritative NLM RxIMAGE database. NLM may use all or part of any Challenge entry (i.e., algorithm and software) to create a future software system and a future API (Application Programming Interface) for pill image recognition; the system will be freely usable and the API will be freely accessible.

DATES:

NLM will make a set of consumer-quality images and a companion set of reference images publicly available on January 15, 2016.

The Challenge begins January 19, 2016.

Submission period: April 4, 2016 to May 31, 2016.

Judging period: June 6, 2016 to July 15, 2016.

Winners announced: August 1, 2016.

Submissions received by NLM after the submission period ends will not be considered. A submission is considered to meet the submission deadline if it is received by May 31, 2016, 5:00 p.m. EDT. While NLM plans to acknowledge receipt of each Challenge submission, the Government is under no obligation to acknowledge receipt of the information received or provide feedback to respondents with respect to any information submitted. NLM will amend this Federal Register notice if the timeline or the rules for the Challenge are modified. In addition, NLM will notify registered Challenge participants by email of any amendments and will include the modified Challenge showing the changes.

ADDRESSES:

Notifications of any amendment to this Federal Register notice and answers to frequently asked questions about it will be posted at http://pir.nlm.nih.gov/challenge/notifications-and-FAQs. Submissions must be mailed to: Pill Image Recognition Challenge, Computational Photography Project for Pill Identification (C3PI), National Library of Medicine, Building 38A, Room B1-N30, 8600 Rockville Pike, Bethesda, MD 20894.

FOR FURTHER INFORMATION CONTACT:

Michael J. Ackerman, Ph.D. at (301) 402-4100 or PIR@nlm.nih.gov.

SUPPLEMENTARY INFORMATION:

The IC's Statutory Authority To Conduct the Challenge

What has become today's National Library of Medicine began in 1836 as a small collection of medical books and journals in the office of the U.S. Army Surgeon General. A 1956 act of Congress (Pub. L. 84-941) transferred the library to the Public Health Service and gave it its current name. That law authorizes NLM to “assist the advancement of medical and related sciences and to aid the dissemination and exchange of scientific and other information important to the progress of medicine and to the public health” and to “promote the use of computers and telecommunications by health professionals (including health professionals in rural areas) for the purpose of improving access to biomedical information for health care delivery and medical research.” In addition to its subject-matter authority, NLM is conducting this competition under the America COMPETES Reauthorization Act of 2010 (Pub. L. 111-358).

Subject of Challenge

Unidentified and misidentified prescription pills present challenges for patients and professionals. Unidentified pills can be found by family members, health professionals, educators, and law enforcement. The nine out of 10 U.S. citizens over age 65 who take more than one prescription pill can be prone to misidentifying those pills. Taking such pills can result in adverse drug events that affect health or cause death. To reduce such errors, any person should easily be able to confirm that a prescription pill or a refill is correct. For example, a person should be able to easily verify—or not—that a refill that has a different color, shape, or text imprinted on the pill is a different generic version of equivalent drugs he or she was already taking.

To help address these problems, the NLM Computational Photography Project for Pill Identification (C3PI) is developing infrastructure and tools for identifying prescription pills. The infrastructure includes photographs of such pills taken under laboratory lighting conditions, from a camera directly above the front and the back faces of the pill, and at high resolution. Specialized digital macro-photography techniques were then used to capture JPEG pill images. The NLM RxIMAGE database contains these high-quality images and associated pill data such as appearance (color, shape, size, text imprinted on the pill, etc.), ingredients, and identifiers such as its National Drug Code (NDC) [ http://www.fda.gov/Drugs/InformationOnDrugs/ucm142438.htm ]. RxIMAGE images and data are freely available. The freely accessible RxIMAGE API provides text-based search and retrieval of images and data from the RxIMAGE database. By contributing their algorithm and software, Challenge participants will take part in a broader NLM effort to develop a freely usable software system and a freely accessible API for image-based search and retrieval from a mobile device.

In a typical scenario for a future NLM mobile app, a person will download the app and use it to photograph a prescription pill, possibly under poor lighting conditions, from an angle, or at low resolution. The future app will communicate with the future pill image recognition software system, which may use all or part of any Challenge entry, to compare that photo to reference images in the RxIMAGE database, and will return one or more reference images that most likely match the photographed pill along with their associated pill data.

NLM provides two directories of images to Challenge participants for use in preparing their submissions:

• Directory DR contains 2000 JPEG reference images of 1000 pills. For each pill there are two reference images, one of the front of the pill and one of the back of the pill. These pill images are the same as in the RxIMAGE database.
• Directory DC contains 5000 JPEG consumer-quality images of the same 1000 pills that were photographed for DR. However, they were taken with a variety of digital cameras, under various lighting conditions, and at camera angles not necessarily perpendicular to the faces of the pills. They are akin to photos of prescription pills that the general public might take.

For a pill for which there is at least one consumer-quality image in DC, DR has two reference images of that pill, one of the front of the pill and one of the back of the pill. Conversely, for a pill for which there are two reference images in DR (one of the front of the pill and one of the back of the pill), DC has two or more consumer-quality images of that pill, taken under different conditions.

DR and DC come with a “ground truth table” that is a two-column table with column headers ref_images and cons_images. Each row of the table gives in the first column the name of a reference image and in the second column the name of a consumer-quality image corresponding to that reference image. There is a separate row in the table for each (reference image, consumer-quality image) pair, even when multiple reference and consumer-quality images are all photos of the same pill. Respondents can use the images in DR and DC, the ground truth table, and the RxIMAGE database in developing, training, and validating their algorithms and software. They can also supplement these data.

Rules for Participating in the Challenge

Teams of one or more members can participate in this Challenge. There is no maximum team size. Each team must have a captain. Individual team members and team captains must register in accordance with the Registration Process for Participants below. The role of the team captain is to serve as the corresponding participant with NLM about the Challenge and to submit the team's Challenge entry. While NLM will notify all registered Challenge participants by email of any amendments to the Challenge, the team captain is expected to keep the team members informed about matters germane to the Challenge.

(1) To be eligible to win the Challenge prize, a team—

a. Shall have registered to participate in the Challenge under the rules promulgated by the NIH as published in this Notice;

b. Shall have complied with all the requirements set forth in this Notice;

c. In the case of a private entity, shall be incorporated in and maintain a primary place of business in the United States, and in the case of an individual, whether participating singly or in a group, shall be a citizen or permanent resident of the United States. However, non-U.S. citizens and non-permanent residents can participate as a member of a team that otherwise satisfies the eligibility criteria. Non-U.S. citizens and non-permanent residents are not eligible to win a monetary prize (in whole or in part). Their participation as part of a winning team, if applicable, may be recognized when the results are announced.

d. May not be a Federal entity;

e. May not be a Federal employee acting within the scope of the employee's employment and further, in the case of HHS employees, may not work on their submission(s) during assigned duty hours. Note: Federal ethical conduct rules may restrict or prohibit Federal employees from engaging in certain outside activities, so any Federal employee seeking to participate in this Challenge outside the scope of employment should consult his/her agency's ethics official prior to developing an submission;

f. May not be an employee of the NIH, a judge of the challenge, or any other party involved with the design, production, execution, or distribution of the Challenge or the immediate family of such a party (i.e., spouse, parent, step-parent, child, or step-child).

g. All team members must be at least 18 years old at the time of submission.

(2) Federal grantees may not use Federal funds to develop their Challenge submissions unless use of such funds is consistent with the purpose of their grant award and specifically requested to do so due to the Challenge design, and as announced in the Federal Register.

(3) Federal contractors may not use Federal funds from a contract to develop their Challenge submissions or to fund efforts in support of their Challenge submission.

(4) By participating in this Challenge, each individual (whether competing singly or in a group) and entity agrees to assume any and all risks and waive claims against the Federal government and its related entities (as defined in the COMPETES Act), except in the case of willful misconduct, for any injury, death, damage, or loss of property, revenue, or profits, whether direct, indirect, or consequential, arising from participation in this Challenge, whether the injury, death, damage, or loss arises through negligence or otherwise.

(5) Based on the subject matter of the Challenge, the type of work that it will possibly require, as well as an analysis of the likelihood of any claims for death, bodily injury, property damage, or loss potentially resulting from Challenge participation, no individual (whether competing singly or in a group) or entity participating in the Challenge is required to obtain liability insurance or demonstrate financial responsibility in order to participate in this Challenge.

(6) By participating in this Challenge, each individual (whether competing singly or in a group) and entity agrees to indemnify the Federal government against third party claims for damages arising from or related to Challenge activities.

(7) An individual or entity shall not be deemed ineligible because the individual or entity used Federal facilities or consulted with Federal employees during the Challenge if the facilities and employees are made available to all individuals and entities participating in the Challenge on an equitable basis.

(8) By participating in this Challenge, each individual (whether participating singly or in a group) and entity grants to the NIH, in any existing or inchoate copyright or patent rights owned by the individual or entity, an irrevocable, paid-up, royalty-free, nonexclusive worldwide license to use, reproduce, post, link to, share, and display publicly on the Web the submission, except for source code. This license includes without limitation posting or linking to the submission, except for source code, on the NLM Pill Image Recognition Web site [ http://pir.nlm.nih.gov/challenge ]. In developing its future software system and future API, NLM may include algorithms and software from Challenge entries and may consult with individuals or teams that submitted entries. Thus, the license also permits NLM to develop the future software system and the future API, independently or with others, using any algorithms or software from Challenge entries, including those obtained from other Challenges or solicitations, and NLM may freely use, reproduce, modify and distribute the resulting future software system and API without restriction. NLM may work with individuals or teams that submitted entries to write articles about pill image recognition and submit them to peer-reviewed journals. Each participant will retain all other intellectual property rights in their submissions, as applicable.

(9) NIH reserves the right, in its sole discretion, to (a) cancel, suspend, or modify the Challenge through amendment to this Federal Register notice, and/or (b) not award any prizes if no entries are deemed worthy. In addition, NLM reserves the right to disqualify any Challenge participants or entries in instances where cheating or other misconduct is identified.

(10) Each individual (whether participating singly or in a group) or entity agrees to follow all applicable federal, state, and local laws, regulations, and policies.

(11) Each individual (whether participating singly or in a group) and entity participating in this Challenge must comply with all terms and conditions of these rules, and participation in this Challenge constitutes each such participant's full and unconditional agreement to abide by these rules. Winning is contingent upon fulfilling all requirements herein.

(12) Each individual (whether participating singly or in a group) and entity grants to NLM and NLM contractors assisting NLM with C3PI the right to review the submission, study the algorithms and the code, and run the software on other sets of images.

(13) Submissions must not infringe upon any copyright, patent, trade secrets, or any other rights of any third party. Each individual (whether participating singly or in a group) or entity warrants that he/she or the team is the sole author and owner of any copyrightable work that the submission comprises, that the submission is wholly original with the participant or is an improved version of an existing work that the participant has sufficient rights to use and improve. In addition, the submission must not trigger any reporting or royalty obligation to any third party. A submission must not include proprietary, classified, confidential, or sensitive information.

(14) The submission does not contain malicious code such as viruses, timebombs, cancelbots, worms, trojan horses, or other potentially harmful programs or other material or information.

(15) Notwithstanding the above and consistent with the principal objective of the Challenge to make results widely available to the public. If the submitter distributes their executable code or source code NLM encourages every individual and team to distribute their submission's executable code and preferably also its source code to the public under an Apache 2.0 License that permits the public to benefit from and improve upon the submission. In this case and by mutual agreement, NLM may also post or link to the code from the Pill Image Recognition Web site. The entry must include a description of how and under what license terms it intends to make any code that is part of the entry available to the public.

(16) Challenge participants are free to discuss their submission and the ideas and technologies that it contains with other parties, except as stated in #7 above and are free to contract with any third parties so long as they do not sign any agreement or undertake any obligation that conflicts with any agreement that they have entered into, such as with any team members, or do enter into regarding their submission for the Challenge. For the purpose of clarity, Challenge participants acknowledge that the intent of the Challenge is to encourage people to collaborate and share ideas and innovations.

Registration Process for Participants

To participate in this Challenge, team captains must register their teams, including providing the names and email addresses of all team members, at http://pir.nlm.nih.gov/challenge/register . Early registration is encouraged in order to be able to receive email notifications if this Federal Register notice is amended to change the timeline or the rules of this Challenge.

Submission Requirements

Participants must provide a complete submission as defined below to be considered for the prize. The submission must be saved to a USB storage device containing a virtual machine and must be received (not simply post-marked) by NLM by May 31, 2016, 5 p.m. EDT. Submissions must be mailed to: Pill Image Recognition Challenge, Computational Photography Project for Pill Identification (C3PI), National Library of Medicine, Building 38A, Room B1-N30, 8600 Rockville Pike, Bethesda, MD 20894.

The Submission is defined to include:

1. Executable software for ranking how well consumer images of pills taken by digital cameras match reference images. The software shall be a batch-mode program or a script whose input consists of a directory of consumer images and a directory of reference images. The output shall be a comma-separated-value (csv) M-by-N matrix MR of ranks that for i = 1,...,M compares consumer image i with reference images j = 1,...,N. For each consumer image, no rank can appear more than once. The software does not need to identify pills by name.

2. Source code for the executable that is both human- and machine-readable. The source code can be written in any programming language(s).

3. A .csv file containing the matrix MRC (C for Challenge) of ranks that is the output from executing the executable using DC and DR as input. In this case MRC is a matrix that has at least 5000 rows and has 2000 columns. For each row i, MRC(i,j) will rank how well reference image j matches consumer-quality image i, for j = 1,...,2000. If reference image J best matches consumer-quality image I then MRC(I,J) = 1, and if reference image K is the worst match to consume-quality image I then MRC(I,K) = 2000.

4. A text file written in English and containing the algorithm in pseudo-code that the source code implements, and a description of how it works and any tools or packages that it uses. The pseudo-code is to have the complete pipeline from the input directories to a matrix of ranks, and also include any code that implements features or does offline training.

5. A one-page text file written in English that contains the following:

a. Title of entry

b. Names and email addresses of the team captain and all team members

c. A five or more character identifier for the entry that is used as a prefix in the names of all of the team's submitted files

d. A brief description of the submission

Amount of the Prize; Award Approving Official

Up to five monetary prizes may be awarded: $25,000 for 1st Place, $15,000 for 2nd Place, $5,000 for 3rd Place, and two $2,500 prizes for Honorable Mention for a total prize award pool of up to $50,000. The names of the winners and the titles of their entries will be posted on NLM Web sites. The Award Approving Official is the Director of the National Library of Medicine.

Payment of the Prize

Prizes awarded under this Challenge will be paid by electronic funds transfer and may be subject to Federal income taxes. HHS/NIH will comply with IRS (U.S. Internal Revenue Service) withholding and reporting requirements, where applicable.

Basis Upon Which Winners Will Be Selected

NLM will first review submissions to determine their suitability for judging and eligibility to win the prize. An eligible submission (a) complies with the rules in this Federal Register notice, (b) follows the detailed submission instructions at http://pir.nlm.nih.gov/challenge/ , (c) is complete (e.g., meets the Submission Requirements above), and (d) is confirmed by NLM that the executable is an implementation of the submitted source code, can be run on the submitted virtual machine using directories DC and DR as input, and the output matrix is the same as the submission's matrix MRC.

Eligible submissions will proceed to the judging process, and will be evaluated based on how well the submissions can match pill images across a large number of queries. NLM has developed evaluation software and created directories of images to use in selecting the Challenge winners. For example, directories DRJ and DCJ (J for Judging) contain reference and consumer-quality images similar to those in directories DR and DC provided to potential Challenge participants, and the ground truth matrix for judging MGTJ has MGTJ(i,j)=1 if DCJ(i) and DRJ(j) are photos of the same pill, else MGTJ(i,j)=0. The evaluation software calculates the mean average precision (MAP) for directories and ground truth matrix such as DRJ, DRC, and MGTJ. Mean average precision is a widely used measure for evaluating how well information retrieval systems (for example, search engines) retrieve results across a large number of queries. The MAP formula and examples of MAP calculations for this Challenge are at http://pir.nlm.nih.gov/challenge/MAP_example.

The eligible submissions will be submitted to the evaluation software. The submission with the highest MAP score will be recommended as the first place winner, with the second, third, fourth, and fifth best MAPs, respectively being recommended to earn second place, third place, and two honorable mentions. In the event of tied scores, the tied submissions will be tested against additional DRJ and DCJ directories until a winner is determined. Dr. Terry Yoo will serve as the judge for the competition, thereby overseeing the evaluation software process and being responsible for its proper application.

Additional Information

The NLM Computational Photography Project for Pill Identification (C3PI) is a research and development project in the Office of High Performance Computing and Communications (OHPCC) within the NLM Lister Hill National Center for Biomedical Communications (LHNCBC). C3PI computer scientists conduct computer vision R&D in text- and image-based search and retrieval. C3PI's overall goal is to help improve the prescription drug information made available to health professionals and consumers. The NLM “Pill Image Recognition Request for Information” (PIR RFI) ( https://www.fbo.gov/index?s=opportunity&mode=form&id=a1a694718366ea7cbaf8f715047d63e1&tab=core&_cview=0 ) was a pilot for this Challenge. The RFI was announced in February 2015 and responses were due in May 2015. RFI responses were used to test, evaluate, and as needed refine the components of the Challenge, including its instructions. NLM appreciates the work done by the parties that responded to the RFI.