Sep 19, 2023 · In this work, we introduce the Qwen-VL series, a set of large-scale vision-language models (LVLMs) designed to perceive and understand both texts and images. Starting from the Qwen-LM as a foundation, we endow it with visual capacity by the meticulously designed (i) visual receptor, (ii) input-output interface, (iii) 3-stage training pipeline ...

Sep 26, 2024 · Additionally, we evaluate three categories of LLMs against this benchmark: 1) proprietary LLM services such as GPT-4; 2) public LLMs like QWen; and 3) specialized medical LLMs, like HuatuoGPT2. Through this extensive evaluation, we aim to provide a better understanding of LLMs' performance in the medical domain, ultimately contributing to their ...

The overall network architecture of Qwen-VL consists of three components and the details of model parameters are shown in Table 1: Large Language Model: Qwen-VL adopts a large language model as its foundation component. The model is initialized with pre-trained weights from Qwen-7B (Qwen, 2023).

Qwen-1.5-7B Qwen-1.5-32B Qwen-1.5-72B Claude-3.5-sonnet Gemma-2-9b Gemma-2-27b Llama-3.1-8B Llama-3.1-70B Llama-3.1-405B Safe Over-Refusal Claude Gemini GPT-3.5 GPT-4 Llama-2 Llama-3 Mistral Qwen Gemini-1.5-pro* Figure 1: Over refusal rate vs toxic prompts rejection rate on OR-Bench-Hard-1K and OR-Bench-Toxic. Results are measured with ...

Published as a conference paper at ICLR 2024 3.1 LOSS OF HIGH FREQUENCY INFORMATION - "NTK-AWARE" INTERPOLATION If we look at rotary position embeddings (RoPE) only from an information encoding perspective,

visual-language understanding. Qwen-VL [2] uses three-stage training to convert QwenLM to Qwen-VL. LLaVA series [46, 44, 45] adopt visual instruction tuning that uses instruction-following data to convert LLM into multimodal LLM. ShareGPT4V [7] collects detailed image caption data from GPT4V to augment the LLaVA models.

ness and privacy awareness simultaneously, e.g., improving Qwen-2-7B-Instruct’s fairness awareness by 12.2% and privacy awareness by 14.0%. More crucially, DEAN remains robust and effective with limited annotated data or even when only malicious fine-tuning data is available, whereas SFT methods may fail to perform properly in such scenarios.

Apr 24, 2023 · Recent years have witnessed a big convergence of language, vision, and multi-modal pretraining. In this work, we present mPLUG-2, a new unified paradigm with modularized design for multi-modal pretraining, which can benefit from modality collaboration while addressing the problem of modality entanglement.

Welcome to the OpenReview homepage for ICLR 2024

tuning phase. To lift the OCR accuracy and support other languages, e.g., Chinese, Qwen-VL Bai et al. (2023b) unfreezes its image encoder (a CLIP-G) and uses lots of OCR data in its stage-two training. Innovatively, Vary Wei et al. (2023) generates a new …