Google
File appearance is part of the test
Announced on November 10, Multimodal Knowledge Agent measures professional tasks where non-textual information matters. It includes 200 tasks created by experts from 10 industries and 83 professions.
The tasks require multi-step work with files such as drawings, reports, presentations and spreadsheets. Evaluation criteria assign a 75% weight to factual accuracy and 25% to visual design.
How the models are compared
Models run with the same open-source Stirrup agent environment, tools and turn budget. Results are available through Artificial Analysis. An MKA score is not directly equivalent to a score from another benchmark using a different setup.
Hugging Face
OpenAI