Places
Real-world entities where users need practical visit-planning information, including food venues, cultural venues, offices, and public-facing services.
Public challenge
Multimodal accessibility planning with AI models
The MAP evaluation assesses multimodal AI systems as assistants for users with accessibility requirements when planning visits to real places. We enable evaluation in an open world setting and allow for dynamic changes in the real world by running assessments within specified windows.
Challenge Scope
The first version of MAP focuses on the city of Zurich, Switzerland. Systems are assessed on providing useful and accurate information for requests with specifications related to accessibility in relation to real locations such as restaurants, cafes, museums, local government, offices, and other places of interest.
Accessibility Features
Features include provisions for mobility, blind or low-vision, hearing, and cognitive requirements, including step-free entrance, toilet-door width, hearing-access provisions, coordinates, and related accessibility information.
Evaluation Window
Assessments run within specified windows so that model outputs can be judged against changing real-world conditions.
Evaluation Design
MAP is designed around realistic requests from the community and aims to take account of real-world changes.
1
Prepare
Entrants use challenge details and validation data to prepare model systems.
2
Respond
Systems respond to samples from a set of prompt types.
3
Score
Responses are scored against ground truth data on places and features.
4
Report
Approved results will be published through a public leaderboard following evaluation phases.