Places
Real-world entities where users need practical visit-planning information, including food venues, cultural venues, offices, and public-facing services.
Challenge
Multimodal accessibility planning with AI models
The MAP evaluation assesses multimodal AI systems as assistants for users with accessibility requirements when planning visits to real places. We enable evaluation in an open world setting and allow for dynamic changes in the real world by running assessments within specified windows.
Challenge Scope
The first version of MAP focuses on the city of Zurich, Switzerland. Systems are assessed on providing useful and accurate information for requests with specifications related to accessibility in relation to real locations such as restaurants, cafes, museums, local government, offices, and other places of interest.
Accessibility Features
Features include provisions for mobility, blind or low-vision, hearing, and cognitive requirements, including step-free entrance, signage, hearing-access provisions, coordinates, and related accessibility information.
Evaluation Window
Assessments run within specified windows so that model outputs can be judged against changing real-world conditions.
Evaluation Design
MAP is designed around places in the real world and aims to take account of changes over time.
1
Prepare
Entrants use challenge details and validation data to prepare model systems.
2
Respond
Systems respond to samples from a set of prompt types.
3
Score
Responses are scored against ground truth data on places and features.
4
Report
Approved results are published through a public leaderboard following evaluation phases.
Project - Phase One
These pages provide information on the benchmark and workshop - with access to the leaderboard.
Enter the Challenge
Our first iteration of the challenge covers the two types of questions in the claim verification for accessibility planning (Feature check/Search-by-feature). The entrant portal supports registration, validation response uploads, and test-window model/API endpoint guidance.