Azure AI Vision app helps blind, low-vision adults in 354 tasks

AI system

System: Cornell Tech study scene-description app (unnamed) built on the Microsoft Azure AI Vision image description API
Model: Not assessed
Release: Not assessed
Application developer: Cornell Tech (app); Microsoft (Azure AI Vision image description API)
Role: Not assessed
Configuration: Not assessed
Inputs and outputs: Not assessed
Autonomy: Not assessed
Workflow: An iPhone app sending photos to the Azure AI Vision image description API. Blind and low-vision users photographing objects they did not want to touch or ask others about. A minimum of one photo per day for two weeks with a short questionnaire per entry.
Human oversight: Not assessed
Relation to AI: Azure AI Vision generated every description. The researchers' app only submitted photos and collected ratings. The paper documents individual successful uses and an overall judgment that descriptions “still need significant improvements”. Participants were allowed to use other tools during the study, so the diary covers only what they chose to photograph with this app.
AI contribution: Azure AI Vision generated every description. The researchers' app only submitted photos and collected ratings. The paper documents individual successful uses and an overall judgment that descriptions “still need significant improvements”. Participants were allowed to use other tools during the study, so the diary covers only what they chose to photograph with this app.

Outcome identifiers: Not assessed
Benefit: 2.76 out of 5 for satisfaction and 2.43 out of 4 for trust on average
Category: Improved Accessibility
Status: Not assessed
Beneficiary groups: People.
Beneficiaries: Not assessed
Reach: Not assessed
Distribution and access: Not assessed
Duration: Not assessed
Expected?: Yes
Costs and harms: Not assessed

Studies: Not assessed
Study design: Not assessed
Measurement: Not assessed
Setting and sample: Not assessed
Measures: 2.76 out of 5 for satisfaction and 2.43 out of 4 for trust on average
Comparison: asking a sighted family member (P1's husband)
Results: Result: She was happy to get an answer without having to call her husband.
Additional value from AI: Not assessed
Limitations: Not assessed
Denominators: Not assessed
Materials for replication: Not assessed

Participants: Developer: Cornell Tech (app); Microsoft (Azure AI Vision image description API). Operator: Study participants (16 blind and low-vision adults).
Jurisdictions: Benefit: United States. Developer: United States.
Ownership and control: Not assessed
Dependencies: Not assessed
Cross-border connections: Not assessed
Geopolitical implications: Not assessed

Conditions for replication: The description is accurate enough for the user to infer what they need. Users sometimes drew answers from partially correct output. The task does not require reading text, which the app could not do.
Replication attempts: 1 follow-up entries. Benefit recurred: 1.
Later findings: 7 March 2025. The same Cornell Tech group's follow-up two-week diary study (CHI EA 2025), with 20 BLV people using a newer app they built on a multimodal large language model, collected 553 entries; a preliminary analysis of 60 entries found trust 3.75/5 and satisfaction 4.15/5, higher than the 2024 scores.
Availability: Not assessed
Institutional decision: Not assessed
Grounds: Not assessed
Action and result: Not assessed
Success criterion: Not assessed

Assessed benefits: People.
Depth: People: 1.
Reasons: People: Participants completed small daily identifications without asking family. They rated descriptions 2.76/5 for satisfaction and 2.43/4 for trust on average. The app could not read text.
Observed or intended: Not assessed
Rubric: Not assessed
Assessor and date: Not assessed
Alternative assessments: Not assessed
Fill: The makers' claim

Kind: Not assessed
Related entries: Not assessed
Field: Accessibility / visual assistance. Blind and low-vision users.
Purpose: The study app was built to describe photographed scenes for blind and low-vision users, emulating Seeing AI's scene channel. The documented identifications were that purpose.
Dates: Date basis unassessed: 2024 (precision unassessed). Updated: 22 October 2026.
Stage: Not assessed
Version: SHA-256: 1b862e63ef09…

Peer-reviewed study Developer

Gonzalez Penuela RE, Collins J, Bennett CL, Azenkot S (2024). Investigating Use Cases of AI-Powered Scene Description Applications for Blind and Low Vision People. Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '24). doi:10.1145/3613904.3642211; arXiv:2403.15604.
Peer-reviewed study Developer

Gonzalez Penuela et al. (2024), arXiv HTML version (P1 example).
Preprint Developer

Gonzalez Penuela RE, Hu R, Lin S, Shende T, Azenkot S (2025). Towards Understanding the Use of MLLM-Enabled Applications for Visual Interpretation by Blind and Low Vision People. arXiv:2503.05899, submitted 7 March 2025; to appear at CHI 2025.
Other Independent author

Semantic Scholar record for doi 10.1145/3613904.3642211 (venue, authors and open-access link).