The realm of artificial intelligence is witnessing a pivotal moment with the advent of SolidGeo, a benchmarking test that is redefining the capabilities of multimodal large language models (MLLM). Pioneered by a team of researchers from the Institute of Automation, Chinese Academy of Sciences, SolidGeo sets a new standard for evaluating AI's prowess in three-dimensional spatial reasoning, focusing specifically on the intricacies of solid geometry.

SolidGeo's challenge to AI models transcends the traditional realm of planar geometry by demanding a comprehensive grasp of three-dimensional structures and their spatial relationships. This requires not only a high level of spatial reasoning but also the integration of visual and textual information. The benchmark's dataset, which boasts 3113 high-quality solid geometry problems, is drawn from K-12 education and high school math competitions. Each problem is accompanied by an image and a detailed solution, ensuring the authenticity and reliability of the data.

In an experiment conducted on 26 mainstream multimodal models, the results were revealing. The current state-of-the-art OpenAI-o1 model achieved an accuracy of only 49.5% on the SolidGeo test, significantly lagging behind human performance, which stands at 77.5%. The performance of other models was equally disheartening, with many open-source models scoring below 30%. Particularly in complex solid geometry tasks, such as those involving the folding and unfolding of planes, the OpenAI-o1 model's accuracy plummeted to 36.1%. Interestingly, some models unexpectedly excelled in tasks of specific difficulty, suggesting a potential lack of generalization ability when dealing with simpler problems.

The study delves deeper into the performance disparities among models across different prompting strategies, problem difficulties, and reasoning efficiencies. It was found that most models experienced a significant drop in accuracy as task difficulty increased. Reasoning efficiency often suffered due to excessively long outputs, leading to a phenomenon of "overthinking," posing challenges for the practical application of AI.

The introduction of SolidGeo not only establishes a new benchmark for AI models in solid geometry reasoning but also propels the exploration of multimodal models in the field of spatial intelligence. As the capabilities of large models continue to evolve, achieving breakthroughs in complex domains such as solid geometry will become a critical task for researchers in the future.