Product information

Haleon leverages Microsoft’s Seeing AI technology to make healthcare product information more accessible to people who are blind or visually impaired

For Marc Powell, a seemingly simple trip to the store to buy drugs or other consumer health products is often an expensive and complicated undertaking.

Powell, who is blind, usually has to ask a store employee to help him find what he needs and then read the package information to him, which can seem intrusive. At home, he will seek help from his partner, who has partial vision, but the print is sometimes too small for her to read. In a pinch, he will browse the Internet or wait for a family member to stop and read aloud to him the information he needs.

“I’m hugely dependent on someone else to let me know about the product – what I’m supposed to do with it, how many pills I can take, or what’s in it,” says Powell, chief innovation officer. accessibility for the UK. based at the Royal National Institute of Blind People (RNIB), who lives in Cambridge, England. “It’s crazy to say that, isn’t it?”

Mark Powell.

A new collaboration between Microsoft and global consumer health company Haleon aims to make health product information more accessible to people who are blind or partially sighted, or those with low literacy. The companies have worked together to extend the functionality of Microsoft’s Seeing AI app to provide detailed audio information for more than 1,500 Haleon products in the US and UK, including brands such as Centrum, Sensodyne and Emergen-C. The initiative was launched in recognition of World Sight Day on October 13.

Using Seeing AI, users can now scan the barcode of Haleon products and hear the same information provided on the packaging, such as product name, usage instructions and ingredients. When users point a phone at a Haleon product, the Seeing AI app guides them to the barcode with a series of intensified beeps. After scanning the code, users can navigate between sections to get the specific information they want, a feature that was welcomed by visually impaired people who tried the feature before its launch.

“We’ve had great feedback from people who are blind and partially sighted,” says Tamara Rogers, Global Marketing Director at Haleon. “They really appreciate the increased independence that comes with being able to access our product information themselves.”

Black and white head shot of Tamara Rogers, Chief Marketing Officer at Haleon.
Tamara Rogers.

The new feature was developed not only for people who are blind or visually impaired, but also for those with literacy issues, Rogers says. “There’s a classic healthcare industry saying that’s, ‘Always read the label, always read the directions,’ and for some that’s just not possible,” she says.

“This initiative will make Haleon products more accessible to people who are blind or visually impaired. It will also offer more autonomy to people with low literacy skills. It’s a great way to be able to communicate in a different way, audibly, rather than asking people to read.

Launched in 2017, Microsoft’s Seeing AI app is an AI camera-based app that can read text such as documents or signs, describe scenes, recognize currency, and even identify friends. Available in 19 languages, the app has become a versatile tool that helps people who are blind or partially sighted navigate their daily lives. The new and improved feature helps consumers in the US and UK read packaging details in English. Previously, there was no easy way for Seeing AI to read detailed packaging information without a user moving a camera around the text.

The collaboration with Haleon “is really important and exciting, because Seeing AI has always been able to recognize the products, but the challenge has been to have access to really rich data, and which comes from the manufacturer”, explains Saqib Shaikh, a engineer. manager at Microsoft who led the team that developed Seeing AI.

“Now you just need to scan the barcode and hear exactly what was written on the packaging. This is directly what the manufacturer wanted you to hear. The information is really accurate and you get it all in one place. You don’t have to scan all the different sides of the package to find the pieces you want. It’s all there once you scan the barcode.

“Now you can give more information and reliable information,” says Shaikh. “Haleon is leading the way in doing this.”

Shaikh hopes more companies will follow Haleon’s lead and use the technology to provide audio information to people who are blind or visually impaired, or those with literacy issues. This could provide important information not only for health products, he says, but also for food products and countless other things people use in their daily lives.

“I’d like to make every product accessible so you can just scan it, whether it’s barcode or some other future technology, and find out what it is,” he says. “That data is out there. It’s just that all the data is in silos. What we have been able to do here is, through the companies that had the will, to break down these silos.

Close-up of hands holding a box of Centrum in one hand and a phone displaying the Seeing AI app in the other.
Using the Seeing AI app, detailed audio information will be available for over 1,500 Haleon products in the US and UK

Over 3 million people in the US and over 2 million in the UK live with vision loss, and millions more struggle with low literacy. A study commissioned by Haleon found that 93% of respondents did not think health products were sufficiently accessible, and nearly one in five said they had taken the incorrect dosage of a product because they could not read. packaging.

Haleon saw the technology as a tool to solve these problems, Rogers says, but wanted to build on an existing platform rather than create new technology that people who are blind or visually impaired should adopt. Haleon approached Microsoft about using the Seeing AI app to create audio guides for its products, and the two companies worked closely to develop Haleon’s new barcode functionality. Microsoft has developed an end-to-end pipeline that allows Haleon to monitor its consumer product data, which is stored in Azure, and update information or add new items as needed.

Powell says to his knowledge, Haleon – a GlaxoSmithKline spin-off created in July 2022 to focus on consumer health products – is the first consumer health company to provide accessible information via Seeing AI for its products.

“I think it will be really helpful,” he says. “It allows us, as blind people, to have the same level of knowledge and understanding as someone who can read the packaging. So it’s a very good step. »