Silicon chips and saving people's eyesight are ORNL researcher's legacy

Research involving artificial intelligence (AI) - performed in the late 1990s by Ken Tobin and others at Oak Ridge National Laboratory - is behind technology now being used at a drug store chain's health care clinics to provide sight-saving telemedical diagnoses to hundreds of thousands of diabetic patients.

Imagine this: A patient who learned recently from a health care provider at a drug store clinic that he has diabetes returned to the clinic for an eye exam.

In this scenario, the drug store provider took photos of both his eyes with a small camera. She uploaded the images from a computer to an internet site viewed remotely by an ophthalmologist who treats eye diseases and disorders. The diagnosis will be emailed to her within a day or so and she will notify the patient of the results.

She might have said to the patient, “You should get this exam every year. A serious complication can be diabetic retinopathy. That’s when high blood sugar levels can damage small blood vessels in the retina in the back of your eyes.

“The disease can eventually make you blind. But if we catch it early enough before you notice you’re losing your vision, you can get laser surgery to save your eyesight.”

The ophthalmologist making the diagnosis uses a content-based image retrieval (CBIR) system that automatically searches for and matches the patient’s eye pictures with similar eye images in a large database. The archive includes images that feature signatures of healthy eyes, of eyes showing the first signs of diabetic retinopathy (that is correctable by laser surgery) and of eyes so severely affected by the disease that blindness is inevitable.

What’s the origin of this technology that has led to the convenient use of the services of CVS MinuteClinic and endocrinologists? It stems from research involving AI by Tobin and others at ORNL.

Tobin, chief research and university partnerships officer at Oak Ridge Associated Universities since 2020, described CBIR systems he and ORNL colleagues developed and patented during a recent talk on his 35 years of innovation and leadership to Friends of ORNL. He said the CBIR development was “a predecessor for AI advances that we read about in the news lately, such as ChatGPT.”

“When I think back about my career, what makes me sleep well at night is the telemedical retina diagnosis achievement because it saves or maintains thousands of patients’ vision,” said Tobin, who has described himself as “an imaging guy who used AI.”

He added that he was also proud of the technology development he spearheaded that has helped the U.S. semiconductor industry “improve product yield and reduce waste” to achieve high-quality device products and increase industry profits. Semiconductor devices, otherwise known as “chips,” are essential components in the electronic circuits of our cars, laptops, smart tablets and smartphones.

ORNL’s automated semiconductor image retrieval technology is ranked ninth among the laboratory’s top 10 technology transfer successes. These top 10 technologies account for 48% of the lab’s royalties.

Tobin’s initial work with Shaun Gleason, Tom Karnowski and others for SEMATECH, a consortium of semiconductor companies like Hewlett Packard, IBM and Motorola, led to their development of a CBIR system for manufacturers, thanks to Laboratory Directed Research & Development (LDRD) funding in 1998-99 from ORNL.

Semiconductor companies produce silicon wafers, which create integrated circuits in electronic devices. A wafer, ranging from 8 to 11 inches across, can contain up to 600 semiconductor chips, each of which has potentially millions or billions of transistors. A single wafer, which may have up to three million components, costs thousands of dollars.

“Chip fabricators want to know how the manufacturing process can make bad or faulty chips,” Tobin said. To improve the number of good chips and minimize the number of bad ones, the fabricator needs to determine quickly which of the complicated semiconductor manufacturing processes is faulty and needs fixing.

Tobin and his colleagues developed a classifier of digital images of individual wafers provided by semiconductor companies that had scratches and other defects in different patterns. These maps were labeled as signatures of a machine not working right or producing particle contamination during a coating or chemical deposition process forming a wafer’s layers.

By matching images of newly formed wafers with images in a database, “our classifier quickly generated results on what processes needed to be fixed and automatically sent reports hourly, daily or weekly to yield engineers,” Tobin said. “A number of companies that worked with SEMATECH wanted our technology in their fab” because it would help them reduce waste, increase efficiency and boost profits.

Unfortunately, since ORNL had only a work-for-others agreement with SEMATECH, it could not collect royalties from SEMATECH’s licensing of the lab’s technology to semiconductor companies outside the consortium. So, UT-Battelle, the lab manager, developed “background intellectual property” that led to patents initially licensed to Applied Materials and later licensed in 2004 to Rudolph Technologies. ORNL received royalties from 2004 to 2019 for its automated defect signature analysis that enabled improvements in semiconductor manufacturing processes.

ORNL researchers have developed CBIR systems with large pictorial data repositories for medical imaging of organs in small animals to determine disease states and for geospatial imaging by satellites for identifying locations of power plants and air force bases.

Tobin became involved in the automated diagnosis of eye diseases when Lee Riedinger, who spent many years at the University of Tennessee at Knoxville as a physicist and research director, worked early this century as the deputy director of science and technology at ORNL. Riedinger invited health care professionals to the lab from the UT Health Science Center in Memphis. That’s how Tobin met Edward Chaum of the center’s Hamilton Eye Institute.

“He was working with patients who had diabetic retinopathy and age-related macular degeneration,” Tobin said. “He wanted to know if he gave me a big repository of eye images if I could automate the diagnosis of eye diseases and catch diabetic retinopathy early enough to prevent blindness.”

An estimated 100 million people in the world are affected by diabetic retinopathy.

Tobin, Karnowski and Chaum received a patent for “a method for the diagnosis of blinding eye disease using image content and an archive of diagnosed human patient data.”

They assembled a Telemedical Retinal Image Analysis and Diagnosis (TRIAD) network at nine sites in three states. The test network proved the concept could be successful.

However, UT-Battelle’s initial attempts to license TRIAD to a company failed because of concerns about the Food and Drug Administration’s reluctance to approve diagnostic techniques using AI. So, TRIAD was licensed in 2009 to Tobin’s start-up company, Hubble Telemedical Inc. in Knoxville.

In 2010 his company received venture capital investment from Memphis Biomed Ventures, enabling the hiring of a CEO, computer programmers and marketing specialists.

"We had subscriptions to our remote diabetic retinopathy screening service,” Tobin said.

In January 2015, Hubble was acquired by a New York company, Welch Allyn Inc., a leading medical diagnostic device company that delivered practical innovation at the point of care. In June 2015, Welch Allyn was acquired by Hill-Rom, Inc.

In 2022 Hill-Rom made an agreement with CVS MinuteClinic to screen customers for blinding diseases with the help of a certified reader. So, Tobin’s vision to use computer vision techniques to automate the search for images that can be tied to errant manufacturing processes culminated in a technology that is saving thousands of people’s vision,

This article originally appeared on Oakridger: Silicon chips and saving people's eyesight are OR researcher's legacy