QwenCloudBRASIL
Início›Biblioteca›Documentação
Tradução de apoio para consulta. Código e identificadores técnicos preservados. Consultar referência original ↗

Compreensão de áudio

Fonte: https://docs.qwencloud.com/developer-guides/text-generation/audio-understanding - extraído do HTML público arquivado.

Texto + imagem/input de áudio

​ Começando.

? Pré-requisitos ?

import os
import base64
import soundfile as sf
import numpy as np
from openai import OpenAI

# 1. Initialize the client
client = OpenAI(
  api_key=os.getenv("DASHSCOPE_API_KEY"),  # Make sure the environment variable is configured
  base_url="https://maas.qwencloudapi.com/compatible-mode/v1",
)

# 2. Initiate the request
try:
  completion = client.chat.completions.create(
    model="qwen3.5-omni-plus",
    messages=[{"role": "user", "content": "Who are you?"}],
    modalities=["text", "audio"],  # Specify text and audio output
    audio={"voice": "Tina", "format": "wav"},
    stream=True,  # Must be set to True
    stream_options={"include_usage": True},
  )

  # 3. Process the streaming response and decode the audio
  print ("Model response:")
  audio_base64_string = ""
  for chunk in completion:
    # Process the text part
    if chunk.choices and chunk.choices[0].delta.content:
      print (chunk.choices[0].delta.content, end="")

    # Collect the audio part
    if chunk.choices and hasattr (chunk.choices[0].delta, "audio") and chunk.choices[0].delta.audio:
      audio_base64_string += chunk.choices[0].delta.audio.get("data", "")

  # 4. Save the audio file
  if audio_base64_string:
    wav_bytes = base64.b64decode(audio_base64_string)
    audio_np = np.frombuffer(wav_bytes, dtype=np.int16)
    sf.write("audio_assistant.wav", audio_np, samplerate=24000)
    print ("\n Audio file saved to: audio_assistant.wav")

except Exception as e:
  print (f"Request failed: {e}")

Resposta Depois que você correr o Python Ou Node.js código, a resposta do texto é devolvida e um arquivo de áudio chamado audio_assistant.wav é salvo no mesmo diretório que seu arquivo de código. Entendido.

Model response:
I am a large language model developed by Alibaba Cloud. My name is Qwen. How can I help you?

Correndo o HTTP O código retorna texto e dados de áudio codificados pela Base64 diretamente no audio Campo. Entendido.

data: {"choices":[{"delta":{"content":"I"},"finish_reason":null,"index":0,"logprobs":null}],"object":"chat.completion.chunk","usage":null,"created":1757647879,"system_fingerprint":null,"model":"qwen3.5-omni-plus","id":"chatcmpl-a68eca3b-c67e-4666-a72f-73c0b4919860"}
data: {"choices":[{"delta":{"content":" am"},"finish_reason":null,"index":0,"logprobs":null}],"object":"chat.completion.chunk","usage":null,"created":1757647879,"system_fingerprint":null,"model":"qwen3.5-omni-plus","id":"chatcmpl-a68eca3b-c67e-4666-a72f-73c0b4919860"}
......
data: {"choices":[{"delta":{"audio":{"data":"/v8AAAAAAAAAAAAAAA...","expires_at":1757647879,"id":"audio_a68eca3b-c67e-4666-a72f-73c0b4919860"}},"finish_reason":null,"index":0,"logprobs":null}],"object":"chat.completion.chunk","usage":null,"created":1757647879,"system_fingerprint":null,"model":"qwen3.5-omni-plus","id":"chatcmpl-a68eca3b-c67e-4666-a72f-73c0b4919860"}
data: {"choices":[{"finish_reason":"stop","delta":{"content":""},"index":0,"logprobs":null}],"object":"chat.completion.chunk","usage":null,"created":1764763585,"system_fingerprint":null,"model":"qwen3.5-omni-plus","id":"chatcmpl-e8c82e9e-073e-4289-a786-a20eb444ac9c"}
data: {"choices":[],"object":"chat.completion.chunk","usage":{"prompt_tokens":207,"completion_tokens":103,"total_tokens":310,"completion_tokens_details":{"audio_tokens":83,"text_tokens":20},"prompt_tokens_details":{"text_tokens":207}},"created":1757940330,"system_fingerprint":null,"model":"qwen3.5-omni-plus","id":"chatcmpl-9cdd5a26-f9e9-4eff-9dcc-93a878165afc"}

​ Linguagens apoiadas

Qwen3.5-Omni idiomas suportados ** Idiomas de entrada (74 línguas):** Chinês, Inglês, Alemão, Francês, Italiano, Tcheco, Indonésio, Tailandês, Coreano, Polonês, Japonês, Vietnamita, Finlandês, Português, Espanhol, Holandês, Russo, Malaio, Catalão, Sueco, Turco, Ucraniano, Romeno, Eslovaco, Dinamarquês, Islandês, Norueguês (Bokmal), Macedônio, Grego, Húngaro, Galego, Filipino, Croata, Bósnia, Esloveno, Búlgaro, Cazaque, Bielorrusso, Letão, Estoniano, Azerbaijão, Uyghur, Swahili, Hindi, Esperanto, Quirgyz, Tajik, Cebuano, Afrikaans, Árabe, Lituano, Javanese, Bengali, Persa, Hebraico, Punjabi, Gujarati, Mongol, Asturiano, Kannada, Marathi, Interlingua, Malayalam, Maltês, Norueguês Nynorsk, Telugu, Urdu, Georgiano, Basco, Tamil, Odia, Sérvio, MaoriDialetos de entrada (39 dialetos):Norte de Mandarim, Dialeto Guizhou, Cantonês, Dialeto Henan, Hong Kong Cantonês, Xangainês, Dialeto Shaanxi, Dialeto Tianjin, Hokkien Taiwanês, Dialeto Yunnan, Anhui, Dialeto Fujian, Dialeto Gansu, Dialeto Guangdong, Dialeto Hubei, Dialeto Hunan, Dialeto Jiangxi, Dialeto Shandong, Dialeto Shanxi, Sichuan dialeto, dialeto de Guangxi, dialeto de Hainan, dialeto de Chongqing, dialeto de Changsha, dialeto de Hangzhou, dialeto de Hefei, dialeto de Yinchuan, dialeto de Zhengzhou, dialeto de Shenyang, dialeto de Wenzhou, dialeto de Wuhan, dialeto de Kunming, dialeto de Taiyuan, dialeto de Nanchang, dialeto de Jinan, dialeto de Lanzhou, dialeto de Nanjing, Hakka, Mim do Sul Linguagens de áudio de saída (29 línguas): Chinês, Inglês, Alemão, Italiano, Português, Espanhol, Japonês, Coreano, Francês, Russo, Tailandês, Indonésio, Árabe, Vietnamita, Turco, Finlandês, Polonês, Hindi, Holandês, Tcheco, Urdu, Tagalog, sueco, dinamarquês, hebraico, islandês, malaio, norueguês, persaDialetos áudio de saída (7 dialetos):Dialeto sichuano, dialeto de Pequim, dialeto tianjin, Nanjing Dialeto, dialeto Shaanxi, cantonês, Min sulista Qwen3-Omni-Flash idiomas suportados ** Idiomas de entrada/saída (11 línguas): Chinês, Inglês, Alemão, Francês, Italiano, Tailandês, Coreano, Japonês, Russo, Espanhol, Português Dialeto de entrada/saída (8 dialetos):** Dialeto de Sichuan, Xangai, Cantonês, Sul Min, Dialeto de Shaanxi, Dialeto de Nanjing, Dialeto de Tianjin, Dialeto de Pequim Para nomes de modelos, contexto, preços e versões instantâneas, veja [lista de modelos] (https://www.qwencloud.com/developer-guides/getting-started/model-selection). Para limites de solicitação simultânea, veja Limites de taxa.

​ Desempenho do modelo

​ Análise de conteúdo de áudio e vídeo

Ver a tabela estruturada nos quadros de referência desta página ↓

​ Compreensão multidialética

Ver a tabela estruturada nos quadros de referência desta página ↓

​ Geração de legendas líricas

Ver a tabela estruturada nos quadros de referência desta página ↓

​ Programação de áudio e vídeo

Ver a tabela estruturada nos quadros de referência desta página ↓

​ Uso

​ Saída de transmissão

Todos os pedidos para Qwen-Omni Deve definir stream=True.

​ Configuração do modelo

Configure parâmetros, avisos e comprimentos de áudio-vídeo baseado no seu caso de uso para equilibrar custo, velocidade e qualidade.

Audio-video compreensão

Audiocompreensão

Ver a tabela estruturada nos quadros de referência desta página ↓

Recomendado prompt estruturado para compreensão de áudio-vídeo Entendido.

Provide a detailed description of the video.
It should explicitly include three sections:
1. A structured chronological storyline of **every noticeable audio and visual details**
2. A structured list of all visible text. For each text element, include start timestamp, end timestamp, the exact text content, the appearance characteristics. If no text appears, explicitly state so.
3. A structured speech-to-text transcription, include speaker（Corresponding to the character or voice‑over in Section 1, including their accent and tone）, exact spoken content, start timestamp, end timestamp, and speaking state (prosody, emotion, and style). If no speech appears, explicitly state so.
Aside from these three required sections, you are free to organize any additional content in any way you find helpful. This additional content can include global information about the entire video or localized information about specific moments. You may choose the topic of this extra content freely.
Output Format:

História

xx:xx.xxx - xx:xx.xxx <an unstructured long paragraph in natural language describing what happened during this period, blending both audio and video details.> xx:xx.xxx - xx:xx.xxx <an unstructured long paragraph in natural language describing what happened during this period, blending both audio and video details.> xx:xx.xxx - xx:xx.xxx <an unstructured long paragraph in natural language describing what happened during this period, blending both audio and video details.> ...

Texto visível

xx:xx.xxx - xx:xx.xxx "<element>": <appearance> "<element>": <appearance> xx:xx.xxx - xx:xx.xxx "<element>": <appearance> "<element>": <appearance> "<element>": <appearance> xx:xx.xxx - xx:xx.xxx "<element>": <appearance> ...

Palestrantes e Transcrição

Perfis de falante: <speaker> - <profile> <speaker> - <profile> <speaker> - <profile> ... xx:xx.xxx - xx:xx.xxx

<another section>

<paragraphs>

<another section>

<paragraphs> ...

Para descrições de vídeos longos, segmente-os primeiro. Equilibrar custo e qualidade controlando o comprimento do áudio e a complexidade.

Ver a tabela estruturada nos quadros de referência desta página ↓

Recomendado prompt estruturado para compreensão de áudio Entendido.

Provide a detailed description of the audio.

It should explicitly include two sections:

1. A structured chronological storyline of **every noticeable audio details**
2. A structured speech-to-text transcription, include speaker（Corresponding to the character or voice‑over in Section 1, including their accent and tone）, exact spoken content, start timestamp, end timestamp, and speaking state (prosody, emotion, and style). If no speech appears, explicitly state so.

Aside from these two required components, you are free to organize any additional content in any way you find helpful. This additional content can include global information about the entire audio or localized information about specific moments. You may choose the topic of this extra content freely.

Output Format:

História

xx:xx.xxx - xx:xx.xxx <an unstructured long paragraph in natural language describing what happened during this period, blending both audio details.>

xx:xx.xxx - xx:xx.xxx <an unstructured long paragraph in natural language describing what happened during this period, blending both audio details.>

xx:xx.xxx - xx:xx.xxx <an unstructured long paragraph in natural language describing what happened during this period, blending both audio details.>

...

...

Palestrantes e Transcrição

Perfis de falante: <speaker> - <profile> <speaker> - <profile> <speaker> - <profile> ...

xx:xx.xxx - xx:xx.xxx

xx:xx.xxx - xx:xx.xxx

xx:xx.xxx - xx:xx.xxx

...

<another section>

<paragraphs>

<another section>

<paragraphs>

...

Para descrições de áudio longo, segmente-o primeiro.

​ Modo de pensar

Para ativar/desativar, saída de streaming, e thinking_budget Veja, veja.https://www.qwencloud.com/developer-guides/text-generation/thinking). Qwen3-Omni-Flash é um modelo de pensamento híbrido (enable_thinking predefinições para false). No modo de pensar, definir modalities: ["text"] - A saída de áudio não é suportada quando o pensamento é ativado.

​ Pesquisa na Web

O Qwen3.5-Omni Séries suportam busca na web para recuperar informações em tempo real e executar raciocínio. Activar a pesquisa na Web usando o enable_search parâmetro e conjunto search_strategy para agent. Python Node.js Curva Entendido.

import os
from openai import OpenAI

client = OpenAI(
  api_key=os.getenv("DASHSCOPE_API_KEY"),
  base_url="https://maas.qwencloudapi.com/compatible-mode/v1",
)

try:
  completion = client.chat.completions.create(
    model="qwen3.5-omni-plus",
    messages=[{
      "role": "user",
      "content": "Please look up today's date and day of the week, and tell me what major holidays fall on this date."
    }],
    stream=True,
    stream_options={"include_usage": True},
    extra_body={
      "enable_search": True,
      "search_options": {
        "search_strategy": "agent"
      }
    }
  )

  print ("Model response (includes real-time information):")
  for chunk in completion:
    if chunk.choices and chunk.choices[0].delta.content:
      print (chunk.choices[0].delta.content, end="")
  print ()

except Exception as e:
  print (f"Request failed: {e}")

​ Entrada multimodal

​ Vídeo e texto de entrada

Você pode inserir vídeo como uma lista de imagens ou como um arquivo de vídeo. Se você inserir um arquivo de vídeo, o modelo também pode entender o áudio no vídeo. O seguinte código de exemplo usa um URL de vídeo da internet como exemplo. Para inserir um vídeo local, veja os arquivos locais codificados pela Base de Entrada64. A saída de transmissão é necessária para todas as chamadas.

​ Formato de arquivo de vídeo (pode entender áudio no vídeo)

import os
from openai import OpenAI

client = OpenAI(
  api_key=os.getenv("DASHSCOPE_API_KEY"),
  base_url="https://maas.qwencloudapi.com/compatible-mode/v1",
)

completion = client.chat.completions.create(
  model="qwen3.5-omni-plus",
  messages=[
    {
      "role": "user",
      "content": [
        {
          "type": "video_url",
          "video_url": {
            "url": "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20241115/cqqkru/1.mp4"
          },
        },
        {"type": "text", "text": "What is the video about?"},
      ],
    },
  ],
  modalities=["text", "audio"],
  audio={"voice": "Tina", "format": "wav"},
  stream=True,
  stream_options={"include_usage": True},
)

for chunk in completion:
  if chunk.choices:
    print (chunk.choices[0].delta)
  else:
    print (chunk.usage)

​ Formato da lista de imagens

Número de imagens

import os
from openai import OpenAI

client = OpenAI(
  api_key=os.getenv("DASHSCOPE_API_KEY"),
  base_url="https://maas.qwencloudapi.com/compatible-mode/v1",
)

completion = client.chat.completions.create(
  model="qwen3.5-omni-plus",
  messages=[
    {
      "role": "user",
      "content": [
        {
          "type": "video",
          "video": [
            "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20241108/xzsgiz/football1.jpg",
            "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20241108/tdescd/football2.jpg",
            "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20241108/zefdja/football3.jpg",
            "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20241108/aedbqh/football4.jpg",
          ],
        },
        {"type": "text", "text": "Describe the process shown in this video"},
      ],
    }
  ],
  modalities=["text", "audio"],
  audio={"voice": "Tina", "format": "wav"},
  stream=True,
  stream_options={"include_usage": True},
)

for chunk in completion:
  if chunk.choices:
    print (chunk.choices[0].delta)
  else:
    print (chunk.usage)

​ Áudio e texto

import os
from openai import OpenAI

client = OpenAI(
  api_key=os.getenv("DASHSCOPE_API_KEY"),
  base_url="https://maas.qwencloudapi.com/compatible-mode/v1",
)

completion = client.chat.completions.create(
  model="qwen3.5-omni-plus",
  messages=[
    {
      "role": "user",
      "content": [
        {
          "type": "input_audio",
          "input_audio": {
            "data": "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20250211/tixcef/cherry.wav",
            "format": "wav",
          },
        },
        {"type": "text", "text": "What is this audio about"},
      ],
    },
  ],
  modalities=["text", "audio"],
  audio={"voice": "Tina", "format": "wav"},
  stream=True,
  stream_options={"include_usage": True},
)

for chunk in completion:
  if chunk.choices:
    print (chunk.choices[0].delta)
  else:
    print (chunk.usage)

​ Imagem e texto

Qwen-Omni Modelos suportam múltiplas entradas de imagem. Os requisitos para as imagens de entrada são os seguintes:

import os
from openai import OpenAI

client = OpenAI(
  api_key=os.getenv("DASHSCOPE_API_KEY"),
  base_url="https://maas.qwencloudapi.com/compatible-mode/v1",
)

completion = client.chat.completions.create(
  model="qwen3.5-omni-plus",
  messages=[
    {
      "role": "user",
      "content": [
        {
          "type": "image_url",
          "image_url": {
            "url": "https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/zh-CN/20241022/emyrja/dog_and_girl.jpeg"
          },
        },
        {"type": "text", "text": "What scene is depicted in the image?"},
      ],
    },
  ],
  # Set the output data modality. Two are currently supported: ["text","audio"] and ["text"].
  modalities=["text", "audio"],
  audio={"voice": "Tina", "format": "wav"},
  # stream must be set to True to avoid errors.
  stream=True,
  stream_options={
    "include_usage": True
  }
)

for chunk in completion:
  if chunk.choices:
    print (chunk.choices[0].delta)
  else:
    print (chunk.usage)

​ Conversa de várias voltas

Quando você usa o recurso de conversação multi-turnos de Qwen-Omni Modelos, notem o seguinte:

import os
from openai import OpenAI

client = OpenAI(
  api_key=os.getenv("DASHSCOPE_API_KEY"),
  base_url="https://maas.qwencloudapi.com/compatible-mode/v1",
)

completion = client.chat.completions.create(
  model="qwen3.5-omni-plus",
  messages=[
    {
      "role": "user",
      "content": [
        {
          "type": "input_audio",
          "input_audio": {
            "data": "https://dashscope.oss-cn-beijing.aliyuncs.com/audios/welcome.mp3",
            "format": "mp3",
          },
        },
        {"type": "text", "text": "What is this audio about"},
      ],
    },
    {
      "role": "assistant",
      "content": [{"type": "text", "text": "This audio says: Welcome to QwenCloud"}],
    },
    {
      "role": "user",
      "content": [{"type": "text", "text": "Can you tell me about this company?"}],
    },
  ],
  # Set the output data modality. Two are currently supported: ["text","audio"] and ["text"].
  modalities=["text"],
  # stream must be set to True to avoid errors.
  stream=True,
  stream_options={"include_usage": True},
)

for chunk in completion:
  if chunk.choices:
    print (chunk.choices[0].delta)
  else:
    print (chunk.usage)

​ Parse Base64-codificado saída de dados de áudio

A saída de áudio de Qwen-Omni Modelos são dados codificados da Base64 entregues em um fluxo. Você pode usar uma variável de cordas para acumular os dados da Base64 de cada fragmento quando chegar. Depois que o fluxo estiver completo, decodifique a string final para criar o arquivo de áudio. Alternativamente, decodificar e reproduzir cada fragmento em tempo real como é recebido. Python Node.js Entendido.

# Installation instructions for pyaudio:
# APPLE Mac OS X
#   brew install portaudio
#   pip install pyaudio
# Debian/Ubuntu
#   sudo apt-get install python-pyaudio python3-pyaudio
#   or
#   pip install pyaudio
# CentOS
#   sudo yum install -y portaudio portaudio-devel && pip install pyaudio
# Microsoft Windows
#   python -m pip install pyaudio

import os
from openai import OpenAI
import base64
import numpy as np
import soundfile as sf

client = OpenAI(
  api_key=os.getenv("DASHSCOPE_API_KEY"),
  base_url="https://maas.qwencloudapi.com/compatible-mode/v1",
)

completion = client.chat.completions.create(
  model="qwen3.5-omni-plus",
  messages=[{"role": "user", "content": "Who are you"}],
  # Set the output data modality. Two are currently supported: ["text","audio"] and ["text"].
  modalities=["text", "audio"],
  audio={"voice": "Tina", "format": "wav"},
  # stream must be set to True to avoid errors.
  stream=True,
  stream_options={"include_usage": True},
)

# Method 1: Decode after the generation is complete
audio_string = ""
for chunk in completion:
  if chunk.choices:
    if hasattr (chunk.choices[0].delta, "audio"):
      try:
        audio_string += chunk.choices[0].delta.audio["data"]
      except Exception as e:
        print (chunk.choices[0].delta.content)
  else:
    print (chunk.usage)

wav_bytes = base64.b64decode(audio_string)
audio_np = np.frombuffer(wav_bytes, dtype=np.int16)
sf.write("audio_assistant_py.wav", audio_np, samplerate=24000)

# Method 2: Decode while generating (comment out the code for Method 1 to use Method 2)
# # Initialize PyAudio
# import pyaudio
# import time
# p = pyaudio.PyAudio()
# # Create an audio stream
# stream = p.open(format=pyaudio.paInt16,
#                 channels=1,
#                 rate=24000,
#                 output=True)

# for chunk in completion:
#     if chunk.choices:
#         if hasattr(chunk.choices[0].delta, "audio"):
#             try:
#                 audio_string = chunk.choices[0].delta.audio["data"]
#                 wav_bytes = base64.b64decode(audio_string)
#                 audio_np = np.frombuffer(wav_bytes, dtype=np.int16)
#                 # Play the audio data directly
#                 stream.write(audio_np.tobytes())
#             except Exception as e:
#                 print(chunk.choices[0].delta.content)

# time.sleep(0.8)
# # Clean up resources
# stream.stop_stream()
# stream.close()
# p.terminate()

​ Arquivos locais codificados pela base de entrada 64

Imagens

Áudio

Arquivo de vídeo

Lista de imagens (como vídeo) Este exemplo usa o arquivo localmente salvo [eagle.png] (https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/en-US/20250415/zgfeuf/eagle.png). Python Node.js Entendido.

import os
from openai import OpenAI
import base64

client = OpenAI(
  api_key=os.getenv("DASHSCOPE_API_KEY"),
  base_url="https://maas.qwencloudapi.com/compatible-mode/v1",
)

#  Base64 encoding format
def encode_image (image_path):
  with open (image_path, "rb") as image_file:
    return base64.b64encode(image_file.read()).decode("utf-8")

base64_image = encode_image("eagle.png")

completion = client.chat.completions.create(
  model="qwen3.5-omni-plus",
  messages=[
    {
      "role": "user",
      "content": [
        {
          "type": "image_url",
          "image_url": {"url": f"data:image/png;base64,{base64_image}"},
        },
        {"type": "text", "text": "What scene is depicted in the image?"},
      ],
    },
  ],
  # Set the output data modality. Two are currently supported: ["text","audio"] and ["text"].
  modalities=["text", "audio"],
  audio={"voice": "Tina", "format": "wav"},
  # stream must be set to True to avoid errors.
  stream=True,
  stream_options={"include_usage": True},
)

for chunk in completion:
  if chunk.choices:
    print (chunk.choices[0].delta)
  else:
    print (chunk.usage)

Este exemplo usa o arquivo localmente salvo [welcome.mp3] (https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/en-US/20250430/vicmsy/welcome.mp3). Python Node.js Entendido.

import os
from openai import OpenAI
import base64
import numpy as np
import soundfile as sf
import requests

client = OpenAI(
  api_key=os.getenv("DASHSCOPE_API_KEY"),
  base_url="https://maas.qwencloudapi.com/compatible-mode/v1",
)

def encode_audio (audio_path):
  with open (audio_path, "rb") as audio_file:
    return base64.b64encode(audio_file.read()).decode("utf-8")

base64_audio = encode_audio("welcome.mp3")

completion = client.chat.completions.create(
  model="qwen3.5-omni-plus",
  messages=[
    {
      "role": "user",
      "content": [
        {
          "type": "input_audio",
          "input_audio": {
            "data": f"data:;base64,{base64_audio}",
            "format": "mp3",
          },
        },
        {"type": "text", "text": "What is this audio about"},
      ],
    },
  ],
  # Set the output data modality. Two are currently supported: ["text","audio"] and ["text"].
  modalities=["text", "audio"],
  audio={"voice": "Tina", "format": "wav"},
  # stream must be set to True to avoid errors.
  stream=True,
  stream_options={"include_usage": True},
)

for chunk in completion:
  if chunk.choices:
    print (chunk.choices[0].delta)
  else:
    print (chunk.usage)

Este exemplo usa o arquivo local [spring_mountain.mp4] (https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/en-US/20250430/ngisno/spring_mountain.mp4). Python Node.js Entendido.

import os
from openai import OpenAI
import base64
import numpy as np
import soundfile as sf

client = OpenAI(
  api_key=os.getenv("DASHSCOPE_API_KEY"),
  base_url="https://maas.qwencloudapi.com/compatible-mode/v1",
)

#  Base64 encoding format
def encode_video (video_path):
  with open (video_path, "rb") as video_file:
    return base64.b64encode(video_file.read()).decode("utf-8")

base64_video = encode_video("spring_mountain.mp4")

completion = client.chat.completions.create(
  model="qwen3.5-omni-plus",
  messages=[
    {
      "role": "user",
      "content": [
        {
          "type": "video_url",
          "video_url": {"url": f"data:;base64,{base64_video}"},
        },
        {"type": "text", "text": "What is she singing?"},
      ],
    },
  ],
  # Set the output data modality. Supported modalities are ["text", "audio"] and ["text"].
  modalities=["text", "audio"],
  audio={"voice": "Tina", "format": "wav"},
  # stream must be set to True. Otherwise, an error occurs.
  stream=True,
  stream_options={"include_usage": True},
)

for chunk in completion:
  if chunk.choices:
    print (chunk.choices[0].delta)
  else:
    print (chunk.usage)

Este exemplo usa os arquivos locais [football1.jpg] (https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/en-US/20250415/spqrrx/football1.jpg), [football2.jpg] (https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/en-US/20250415/vtnhyr/football2.jpg), [football3.jpg] (https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/en-US/20250430/enmhbp/football3.jpg), e [football4.jpg] (https://help-static-aliyun-doc.aliyuncs.com/file-manage-files/en-US/20250430/ognmyc/football4.jpg). Python Node.js Entendido.

import os
from openai import OpenAI
import base64
import numpy as np
import soundfile as sf

client = OpenAI(
  api_key=os.getenv("DASHSCOPE_API_KEY"),
  base_url="https://maas.qwencloudapi.com/compatible-mode/v1",
)

#  Base64 encoding format
def encode_image (image_path):
  with open (image_path, "rb") as image_file:
    return base64.b64encode(image_file.read()).decode("utf-8")

base64_image_1 = encode_image("football1.jpg")
base64_image_2 = encode_image("football2.jpg")
base64_image_3 = encode_image("football3.jpg")
base64_image_4 = encode_image("football4.jpg")

completion = client.chat.completions.create(
  model="qwen3.5-omni-plus",
  messages=[
    {
      "role": "user",
      "content": [
        {
          "type": "video",
          "video": [
            f"data:image/jpeg;base64,{base64_image_1}",
            f"data:image/jpeg;base64,{base64_image_2}",
            f"data:image/jpeg;base64,{base64_image_3}",
            f"data:image/jpeg;base64,{base64_image_4}",
          ],
        },
        {"type": "text", "text": "Describe the procedure in this video."},
      ],
    }
  ],
  # Set the output data modality. Supported modalities are ["text", "audio"] and ["text"].
  modalities=["text", "audio"],
  audio={"voice": "Tina", "format": "wav"},
  # stream must be set to True. Otherwise, an error occurs.
  stream=True,
  stream_options={"include_usage": True},
)

for chunk in completion:
  if chunk.choices:
    print (chunk.choices[0].delta)
  else:
    print (chunk.usage)

​ Referência da API

Para os parâmetros de entrada e saída de Qwen-Omni, veja [Completação de bate-papo API] (https://www.qwencloud.com/api-reference/chat/openai-chat).

​ Limites de faturamento e taxa

Regras de cobrança Qwen-Omni é faturado com base no número de fichas para diferentes modalidades, como áudio, imagem e vídeo. Veja [Pricing] (https://www.qwencloud.com/developer-guides/getting-started/pricing) para detalhes de preços. Regras para converter áudio, imagens e vídeos em fichas ?Audio ?

import math
from PIL import Image  # pip install Pillow

# ============ Model parameter configuration (modify as needed) ============

# Image factor: 32 for Qwen3.5-Omni series and Qwen3-Omni-Flash
IMAGE_FACTOR = 32

# Token minimum: 24 for Qwen3.5-Omni series; 4 for Qwen3-Omni-Flash
MIN_TOKENS = 24

# High-resolution mode (only supported by Qwen3.5-Omni series, not by Qwen3-Omni-Flash)
# True  → Token maximum 16384
# False → Token maximum 1280 (default)
VL_HIGH_RESOLUTION_IMAGES = False

# ============ Pixel range (auto-calculated from above) ============

MIN_PIXELS = MIN_TOKENS * IMAGE_FACTOR * IMAGE_FACTOR
MAX_PIXELS = (16384 if VL_HIGH_RESOLUTION_IMAGES else 1280) * IMAGE_FACTOR * IMAGE_FACTOR

def smart_resize (height, width, factor=IMAGE_FACTOR,
                 min_pixels=MIN_PIXELS, max_pixels=MAX_PIXELS):
  """Align image dimensions to factor multiples and scale to [min_pixels, max_pixels] range."""
  h_bar = max (factor, round (height / factor) * factor)
  w_bar = max (factor, round (width / factor) * factor)

  if h_bar * w_bar > max_pixels:
    beta = math.sqrt((height * width) / max_pixels)
    h_bar = math.floor(height / beta / factor) * factor
    w_bar = math.floor(width / beta / factor) * factor
  elif h_bar * w_bar < min_pixels:
    beta = math.sqrt(min_pixels / (height * width))
    h_bar = math.ceil(height * beta / factor) * factor
    w_bar = math.ceil(width * beta / factor) * factor

  return h_bar, w_bar

def token_calculate (image_path=''):
  if len (image_path) > 0:
    image = Image.open(image_path)
    height = image.height
    width = image.width
    print (f"Original size: {width}x{height}")
    resized_h, resized_w = smart_resize(height, width)
    token = int (resized_h * resized_w / (IMAGE_FACTOR * IMAGE_FACTOR)) + 2
    print (f"Resized: {resized_w}x{resized_h}, Tokens: {token}")
    return token
  else:
    raise ValueError ("Image path cannot be empty. Provide a valid image file path")

if __name__ == "__main__":
  token = token_calculate(image_path="xxx/test.jpg")

Vídeo, arquivos de vídeo geram dois tipos de fichas: video_tokens (visual) e audio_tokens (áudio).

# Before use, install: pip install opencv-python
import math
import os
import logging
import cv2

# Fixed parameters
FRAME_FACTOR = 2

# For Qwen3.5-Omni and Qwen3-Omni-Flash, IMAGE_FACTOR is 32
IMAGE_FACTOR = 32

# Aspect ratio of video frames
MAX_RATIO = 200

# Lower limit for video frame pixels. For Qwen3.5-Omni and Qwen3-Omni-Flash: 128 * 32 * 32
VIDEO_MIN_PIXELS = 128 * 32 * 32

# Upper limit for video frame pixels. For Qwen3.5-Omni and Qwen3-Omni-Flash: 768 * 32 * 32
VIDEO_MAX_PIXELS = 768 * 32 * 32

FPS = 2
# Minimum number of extracted frames
FPS_MIN_FRAMES = 4

# Maximum number of extracted frames
# Maximum number of extracted frames for Qwen3.5-Omni and Qwen3-Omni-Flash: 128
FPS_MAX_FRAMES = 128

# Maximum pixel value for video input. For Qwen3.5-Omni and Qwen3-Omni-Flash: 16384 * 32 * 32
VIDEO_TOTAL_PIXELS = 16384 * 32 * 32

def round_by_factor (number, factor):
  return round (number / factor) * factor

def ceil_by_factor (number, factor):
  return math.ceil(number / factor) * factor

def floor_by_factor (number, factor):
  return math.floor(number / factor) * factor

def get_video (video_path):
  cap = cv2.VideoCapture(video_path)
  frame_width = int (cap.get(cv2.CAP_PROP_FRAME_WIDTH))
  frame_height = int (cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
  total_frames = int (cap.get(cv2.CAP_PROP_FRAME_COUNT))
  video_fps = cap.get(cv2.CAP_PROP_FPS)
  cap.release()
  return frame_height, frame_width, total_frames, video_fps

def smart_nframes (total_frames, video_fps):
  min_frames = ceil_by_factor(FPS_MIN_FRAMES, FRAME_FACTOR)
  max_frames = floor_by_factor(min (FPS_MAX_FRAMES, total_frames), FRAME_FACTOR)
  duration = total_frames / video_fps if video_fps != 0 else 0
  if duration - int (duration) > (1 / FPS):
    total_frames = math.ceil(duration * video_fps)
  else:
    total_frames = math.ceil(int (duration) * video_fps)
  nframes = total_frames / video_fps * FPS
  nframes = int (min (min (max (nframes, min_frames), max_frames), total_frames))
  if not (FRAME_FACTOR <= nframes <= total_frames):
    raise ValueError (f"nframes should in interval [{FRAME_FACTOR}, {total_frames}], but got {nframes}.")
  return nframes

def smart_resize (height, width, nframes, factor=IMAGE_FACTOR):
  min_pixels = VIDEO_MIN_PIXELS
  total_pixels = VIDEO_TOTAL_PIXELS
  max_pixels = max (min (VIDEO_MAX_PIXELS, total_pixels / nframes * FRAME_FACTOR), int (min_pixels * 1.05))
  if max (height, width) / min (height, width) > MAX_RATIO:
    raise ValueError (f"absolute aspect ratio must be smaller than {MAX_RATIO}, got {max (height, width) / min (height, width)}")
  h_bar = max (factor, round_by_factor(height, factor))
  w_bar = max (factor, round_by_factor(width, factor))
  if h_bar * w_bar > max_pixels:
    beta = math.sqrt((height * width) / max_pixels)
    h_bar = floor_by_factor(height / beta, factor)
    w_bar = floor_by_factor(width / beta, factor)
  elif h_bar * w_bar < min_pixels:
    beta = math.sqrt(min_pixels / (height * width))
    h_bar = ceil_by_factor(height * beta, factor)
    w_bar = ceil_by_factor(width * beta, factor)
  return h_bar, w_bar

def video_token_calculate (video_path):
  height, width, total_frames, video_fps = get_video(video_path)
  nframes = smart_nframes(total_frames, video_fps)
  resized_height, resized_width = smart_resize(height, width, nframes)
  video_token = int (math.ceil(nframes / FPS) * resized_height / 32 * resized_width / 32)
  video_token += 2  # Visual marks
  return video_token

if __name__ == "__main__":
  video_path = "spring_mountain.mp4"  # Your video path
  video_token = video_token_calculate(video_path)
  print ("video_tokens:", video_token)

​ Códigos de erro.

Se uma ligação falhar, veja [Mensagens de erro] (https://www.qwencloud.com/api-reference/preparation/error-messages).

​ Lista de voz

Para as vozes, cada modelo omnimodal suporta, incluindo amostras e voice valores dos parâmetros, veja [lista de voz Omni-modal] (https://www.qwencloud.com/developer-guides/speech/omni-voice-list). [Anterior] (https://www.qwencloud.com/developer-guides/speech/realtime-multimodal-speech[Lista de voz Vozes, pré-visualizações de áudio, valores de parâmetros de voz e idiomas suportados para Qwen-Omni e Qwen-LiveTranslate Modelos. Próximo.https://www.qwencloud.com/developer-guides/speech/omni-voice-list)

Quadros de referência

Tabelas restauradas a partir da referência original; a ordem das colunas e os valores técnicos foram preservados.

Quadro 1

Entrada Saída
VídeoGere uma descrição detalhada e cronometrada deste vídeo. 00:00.000 - 00:02.500 Uma rua da cidade encharcada de chuva enche a tela larga. A fotografia de longa exposição cria faixas de luzes vermelhas e azuis no pavimento molhado. Um homem solitário com um casaco escuro e camisa clara com uma gravata caminha em direção à câmera ao longo da calçada direita.00:12.300 - 00:14.533A câmera se inclina para baixo. Dentro de um raio de lanterna, um lençol amarelo e envelhecido está numa poça rasa, bordas carbonizadas e cantos enrolados. O homem ajoelha-se, estendendo os dedos com luvas para o documento, ondulando ao se aproximar da mão. Só são mostradas as datas parciais.

Quadro 2

Entrada Saída
VídeoDescreva as características vocais, estado emocional e letras completas para cantores em diferentes dialetos chineses. Este áudio apresenta apresentações de rap de cantores de várias regiões da China.Dialeto de Hangzhou (Wu)A voz do cantor masculino é suave e gentil, com a suavidade e calor característicos do dialeto Wu. Sua entonação sobe e cai suavemente, e seu ritmo é leve e rápido, criando um efeito de contar histórias relaxado.Estado emocional: Relaxado, confortável e cheio de vida cotidiana. Ele descreve o estilo de vida lento de Hangzhou e mudanças urbanas, expressando afeto por sua cidade natal. Só resultados parciais são mostrados.

Quadro 3

Entrada Saída
VídeoTranscreva as letras da música e forneça datas para cada linha. O fio de gato passa pela luz do luar nas árvores. Só resultados parciais são mostrados.

Quadro 4

Demo 1 Demo 2
Vídeo Vídeo

Quadro 5

Use o caso. Comprimento de vídeo recomendado Prompt recomendado Valor recomendado de max_pixels
Revisão rápida, baixo custo. ≤ 60 minutos Simples prompt em 50 palavras 230,400
Extração de conteúdo (segmentação de vídeo longa) ≤ 60 minutos Simples prompt em 50 palavras 921,600 to 2,073,600
Análise padrão (gravação de vídeo curto) ≤4 minutos Use o prompt estruturado abaixo. 921,600 to 2,073,600
Análise de fino tamanho (multiple speakers/cenas complexas) ≤ 2 minutos Use o prompt estruturado abaixo. 2,073,600