Alibaba Launches Qwen2-VL: Outperforms GPT-4o and Claude 3.5 Sonnet

Alibaba has released Qwen2-VL, a vision-language model that outperforms competitors like GPT-4o and Claude 3.5 Sonnet, with top performance on visual and multimodal tasks. Read more, and also learn how to install it locally on your computer.

Share
Alibaba Launches Qwen2-VL: Outperforms GPT-4o and Claude 3.5 Sonnet

Alibaba has recently released Qwen2-VL, the latest model in its vision-language series, offering a range of innovative features. Qwen2-VL can communicate via camera, play card games, and act as an agent to control mobile phones and robots. The model comes in three versions: open-source models with 2 billion and 7 billion parameters, and an advanced 72-billion-parameter model available via API.

Top Results in Benchmarking and Document Understanding

The advanced 72-billion-parameter version of Qwen2-VL has achieved state-of-the-art performance in visual understanding across 20 different benchmark tests. "Our 72B model demonstrates top-tier performance across most metrics, often surpassing even closed models like GPT-4o and Claude 3.5-Sonnet," the company's blog states, underscoring the model's significant advantages in document understanding.

Qwen2-VL performs exceptionally well on tests like MathVista (for mathematical reasoning), DocVQA (for document understanding), and RealWorldQA (for answering real-world questions using visual information).

Ability to Handle Long Videos and Advanced Control

The model can analyze videos longer than 20 minutes, create detailed summaries, and answer questions about the content. Qwen2-VL can also function as a control agent, meaning it can operate devices like mobile phones and robots through visual signals and text commands.

Qwen2-VL also has the ability to recognize and understand text in images across multiple languages, including European languages, Japanese, Korean, and Arabic, making it accessible to a global audience.

How to Install the Model Locally on Your Computer

Architectural Improvements

One of the key architectural improvements in Qwen2-VL is the implementation of Naive Dynamic Resolution support. The model can adapt to and process images of varying sizes and clarity. "Unlike its predecessor, Qwen2-VL can handle arbitrary image resolutions, ensuring consistency between the model's input and the inherent information in images," says Binyuan Hui, creator of OpenDevin and lead of the Qwen project.

Hui explains that this approaches human visual perception, enabling the model to process images regardless of clarity or size. Another important upgrade is the introduction of Multimodal Rotary Position Embedding (M-ROPE). "By decomposing the original rotary embedding into three parts representing temporal and spatial (height and width) information, M-ROPE enables the LLM to simultaneously capture and integrate 1D textual, 2D visual, and 3D video positional information," Hui explains.

Many Use Cases

Qwen2-VL's use cases are many and varied. William J.B. Mattingly, a digital nomad, praised the model as his new favorite for Handwritten Text Recognition (HTR) when trying to convert handwritten text into digital format. Ashutosh Shrivastava, another user, used the model to solve a calculus problem and reported successful results, confirming the model's validity for problem-solving.

Although GitHub flagged the organization for unknown reasons, the update is available on Hugging Face.