BRASILTradução de apoio para consulta. Código e identificadores técnicos preservados. Consultar referência original ↗
Áudio em tempo real e compreensão de vídeo
Conversa de áudio e vídeo ao vivo
Como usar
1. Estabelecer uma conexão
Os exemplos de conexão WebSocket e WebRTC usam qwen3.8-omni-flash-realtime. Exemplos de configuração específicos de modelos usam seus modelos indicados.
Qwen-Omni-Realtime Suporta dois protocolos: WebSocket e WebRTC. WebSocket é adequado para integração do lado do servidor e configuração rápida. WebRTC é projetado para cenários de voz baseados em navegadores e de baixa latência, com áudio transmitido diretamente sobre UDP e cancelamento de eco integrado e redução de ruído.
Para duração da sessão e limites do histórico de conversa, veja [Limitações] (#limitations).
WebSocket
Native WebSocket connection
Os seguintes itens de configuração são necessários:
| Item de configuração | Descrição |
|---|---|
| Ponto final | wss://maas.qwencloudapi.com/api-ws/v1/realtime |
| Parâmetro de consulta | O parâmetro de consulta émodel. Coloque no nome do modelo que você want - Para o acesso. Veja. Seleção de modelos. Exemplo:?model=qwen3.8-omni-flash-realtime |
| Requisito cabeçalho | Use o Token de Bearer para autenticação:Authorization: Bearer DASHSCOPE_API_KEY. DASHSCOPE APIKY é o Chave da APIque você pediu em QwenCloud. |
```python
# pip install websocket-client
import json
import websocket
import os
API_KEY=os.getenv("DASHSCOPE_API_KEY")
API_URL = "wss://maas.qwencloudapi.com/api-ws/v1/realtime?model=qwen3.8-omni-flash-realtime"
headers = [
"Authorization: Bearer " + API_KEY
]
def on_open(ws):
print(f"Connected to server: {API_URL}")
def on_message(ws, message):
data = json.loads(message)
print("Received event:", json.dumps(data, indent=2))
def on_error(ws, error):
print("Error:", error)
ws = websocket.WebSocketApp(
API_URL,
header=headers,
on_open=on_open,
on_message=on_message,
on_error=on_error
)
ws.run_forever()
```
DashScope SDK
```python Python
# SDK version 1.26.5 or later
import os
import json
from dashscope.audio.qwen_omni import OmniRealtimeConversation,OmniRealtimeCallback
import dashscope
# If you have not configured an API key, change the following line to dashscope.api_key = "sk-xxx"
dashscope.api_key = os.getenv("DASHSCOPE_API_KEY")
class PrintCallback(OmniRealtimeCallback):
def on_open(self) -> None:
print("Connected Successfully")
def on_event(self, response: dict) -> None:
print("Received event:")
print(json.dumps(response, indent=2, ensure_ascii=False))
def on_close(self, close_status_code: int, close_msg: str) -> None:
print(f"Connection closed (code={close_status_code}, msg={close_msg}).")
callback = PrintCallback()
conversation = OmniRealtimeConversation(
model="qwen3.8-omni-flash-realtime",
callback=callback,
url="wss://maas.qwencloudapi.com/api-ws/v1/realtime"
)
try:
conversation.connect()
print("Conversation started. Press Ctrl+C to exit.")
conversation.thread.join()
except KeyboardInterrupt:
conversation.close()
```
```java Java
// SDK version 2.22.15 or later
import com.alibaba.dashscope.audio.omni.*;
import com.alibaba.dashscope.exception.NoApiKeyException;
import com.google.gson.JsonObject;
import java.util.concurrent.CountDownLatch;
public class Main {
public static void main(String[] args) throws InterruptedException, NoApiKeyException {
CountDownLatch latch = new CountDownLatch(1);
OmniRealtimeParam param = OmniRealtimeParam.builder()
.model("qwen3.8-omni-flash-realtime")
.apikey(System.getenv("DASHSCOPE_API_KEY"))
.url("wss://maas.qwencloudapi.com/api-ws/v1/realtime")
.build();
OmniRealtimeConversation conversation = new OmniRealtimeConversation(param, new OmniRealtimeCallback() {
@Override
public void onOpen() {
System.out.println("Connected Successfully");
}
@Override
public void onEvent(JsonObject message) {
System.out.println(message);
}
@Override
public void onClose(int code, String reason) {
System.out.println("connection closed code: " + code + ", reason: " + reason);
latch.countDown();
}
});
conversation.connect();
latch.await();
conversation.close(1000, "bye");
System.exit(0);
}
}
```
WebRTC
Estabelecer uma conexão WebRTC envolve duas etapas:
1. O cliente envia suas capacidades de mídia e endereços de rede (Offer SDP) para o servidor via HTTP POST. O servidor devolve sua informação (Resposta SDP) para completar a negociação de capacidade.
2. Após a negociação, a camada WebRTC estabelece automaticamente o canal de transporte de áudio.
Configuração de troca SDP:
| Parâmetro | Descrição |
|---|---|
| Pedido de URL | PÓS https://{endpoint}/api/v1/webrtc/realtime |
| Parâmetro de consulta | Use omodel parâmetro de consulta para especificar o modelo. Exemplo:?model=qwen3.8-omni-flash-realtime; Exemplo de modelo anterior:?model=qwen3.5-omni-plus-realtime |
| Content-Type | aplicação/pdp |
| Requisito cabeçalho | Autorização: |
| Requisito corpo. | A cadeia de oferta gerada pelo cliente SDP. |
| Resposta | Sucesso: HTTP 200 com a string SDP Resposta do servidor. Falha: HTTP 4xx com uma mensagem de erro JSON. |
O recurso WebRTC está disponível apenas pela lista de permissão. Contate seu gerente de vendas para obter o ponto final.
|
? Parametrizar a consulta ? model parâmetro de consulta para especificar o modelo. Exemplo: ?model=qwen3.8-omni-flash-realtime; Exemplo de modelo anterior: ?model=qwen3.5-omni-plus-realtime |
Tipo de conteúdo, aplicação/passeio
Solicitar cabeçalho Autorização:
Solicitar corpo A oferta gerada pelo cliente SDP
Resposta: Sucesso: HTTP 200 com o servidor Resposta SDP string. Falha: HTTP 4xx com uma mensagem de erro JSON.
Exemplos de código de conexão:
```python Python
# pip install aiortc aiohttp certifi
import asyncio, aiohttp, ssl, certifi
from aiortc import RTCPeerConnection, RTCConfiguration, RTCSessionDescription
from aiortc.mediastreams import AudioStreamTrack
API_KEY = "your-api-key"
MODEL = "qwen3.8-omni-flash-realtime"
SIGNALING_URL = f"https://{{endpoint}}/api/v1/webrtc/realtime?model={MODEL}"
async def connect():
pc = RTCPeerConnection(RTCConfiguration(iceServers=[]))
# Add an audio track to ensure the Offer SDP contains m=audio (required by the server)
pc.addTrack(AudioStreamTrack())
# Create a DataChannel to trigger SDP negotiation (name is customizable; the server pushes events through a channel named "txt")
pc.createDataChannel("oai-events")
# SDP exchange: create an Offer and send it to the server
offer = await pc.createOffer()
await pc.setLocalDescription(offer)
async with aiohttp.ClientSession() as session:
async with session.post(
SIGNALING_URL,
ssl=ssl.create_default_context(cafile=certifi.where()),
data=offer.sdp.encode("utf-8"),
headers={
"Content-Type": "application/sdp",
"Authorization": f"Bearer {API_KEY}",
},
) as resp:
if not resp.ok:
raise Exception(f"SDP exchange failed: {resp.status} {await resp.text()}")
answer_sdp = await resp.text()
print("=== Offer SDP ===")
print(offer.sdp)
print("=== Answer SDP ===")
print(answer_sdp)
# ICE connection is established automatically
await pc.setRemoteDescription(RTCSessionDescription(sdp=answer_sdp, type="answer"))
print("WebRTC connection established")
return pc
```
```javascript JavaScript
const API_KEY = 'your-api-key';
const API_URL = 'https://{endpoint}/api/v1/webrtc/realtime?model=qwen3.8-omni-flash-realtime';
async function connect() {
const pc = new RTCPeerConnection({ iceServers: [] });
// Add an audio track to ensure the Offer SDP contains m=audio (required by the server)
const stream = await navigator.mediaDevices.getUserMedia({ audio: true });
stream.getAudioTracks().forEach(t => pc.addTrack(t, stream));
// Create a DataChannel to trigger SDP negotiation (name is customizable; the server pushes events through a channel named "txt")
pc.createDataChannel('oai-events');
// Wait for ICE gathering to complete before sending the Offer to get the Answer
pc.onicegatheringstatechange = async () => {
if (pc.iceGatheringState !== 'complete') return;
const resp = await fetch(API_URL, {
method: 'POST',
headers: {
'Content-Type': 'application/sdp',
'Authorization': `Bearer ${API_KEY}`,
},
body: pc.localDescription.sdp,
});
if (!resp.ok) throw new Error('SDP exchange failed: ' + resp.status);
const answerSdp = await resp.text();
// ICE connection is established automatically
await pc.setRemoteDescription({ type: 'answer', sdp: answerSdp });
console.log('WebRTC connection established');
};
// Create the Offer
const offer = await pc.createOffer();
await pc.setLocalDescription(offer);
return pc;
}
```
AOQ
O servidor de aplicativos envia uma solicitação HTTP para obter credenciais de conexão, e o cliente SDK estabelece a conexão de mídia. Veja [Autenticação do Token] (/api-reference/realtime-api/token-auth). Para completar `qwen3.8-omni-flash-realtime` Exemplos, veja [chamas em tempo real] (/developer-guides/tutorials/realtime/aoq-omni-realtime Ou [Conversas de Empurra-para-falar] (/developer-guides/tutorials/realtime/aoq-omni-ptt).
- Dois. Configura a sessão
Envie a [sessão.atualização] (/api-reference/real-time-multimodal/client-events) evento cliente:
Use session.update Para configurar modalidades de saída, voz, formato de áudio e DAV. O seguinte: Qwen3.8-Omni-Flash-Realtime Configuração do modo manual do WebSocket corresponde ao [exemplo mínimo de áudio] (#qwen38-realtime):
{
"type": "session.update",
"session": {
"modalities": [
"text",
"audio"
],
"turn_detection": null,
"audio": {
"input": {
"format": {
"type": "pcm",
"sample_rate": 16000,
"sample_format": "s16le",
"channels": 1,
"packing": "interleaved",
"channel_layout": "mono"
}
},
"output": {
"voice": "longanlingxin",
"format": {
"type": "pcm",
"sample_rate": 24000
}
}
}
}
}
Qwen3.5-Omni-Realtime configuration example
O exemplo a seguir usa a voz de Tina e semantic_vad configuração de Qwen3.5-Omni-Realtime.
{
// The ID of this event, generated by the client.
"event_id": "event_ToPZqeobitzUJnt3QqtWg",
// The event type. This is fixed to session.update.
"type": "session.update",
// Session configuration.
"session": {
// The output modalities. Supported values are ["text"] (text only) or ["text", "audio"] (text and audio).
"modalities": [
"text",
"audio"
],
// The voice for the output audio.
"voice": "Tina",
// The input audio format. Only "pcm" is supported. The input audio must be a PCM audio stream at a 16 kHz sample rate.
"input_audio_format": "pcm",
// The output audio format. Only "pcm" is supported. The output audio is a PCM audio stream at a 24 kHz sample rate.
"output_audio_format": "pcm",
// The system message. It is used to set the model's goal or role.
"instructions": "You are an AI customer service agent for a five-star hotel. Answer customer inquiries about room types, facilities, prices, and booking policies accurately and friendly. Always respond with a professional and helpful attitude. Do not provide unconfirmed information or information beyond the scope of the hotel's services.",
// Specifies whether to enable voice activity detection. To enable it, pass a configuration object. The server will automatically detect the start and end of speech based on this object.
// Set to null to let the client decide when to initiate a model response.
"turn_detection": {
// The VAD type. Valid values: server_vad and semantic_vad. We recommend that you set this parameter to semantic_vad for models like qwen3.5-omni-realtime.
"type": "semantic_vad",
// The VAD detection threshold. Increase this value in noisy environments and decrease it in quiet environments.
"threshold": 0.5,
// The duration of silence to detect the end of speech. If this value is exceeded, a model response is triggered.
"silence_duration_ms": 800
}
}
}
Qwen3.5-Omni-Realtime Modelos suportam vozes clonadas além de vozes predefinidas. Veja [clonografia de voz] (/developer-guides/speech/voice-cloning) para criar uma voz personalizada de uma amostra de áudio.
Para os exemplos de áudio multicanal, agregação de vídeo, configuração MCP e SDK, veja Configuração de sessão avançada.
3. Áudio de entrada e imagens
WebSocket
O cliente envia dados de áudio e imagem codificados para o servidor usando o [input\_audio\_buffer.append](/api-reference/real-time-multimodal/client-eventsE.../api-reference/real-time-multimodal/client-events) eventos. Entrada de áudio é necessária. A entrada da imagem é opcional e pode vir de arquivos locais ou de um fluxo de vídeo em tempo real.
Quando a detecção de atividade de voz do lado do servidor (VAD) é ativada, o servidor automaticamente envia os dados e desencadeia uma resposta quando detecta o fim da fala. Quando o VAD está desativado (modo manual), o cliente deve chamar o [input\_audio\_buffer.commit](/api-reference/real-time-multimodal/client-events) evento para enviar os dados.
WebRTC
As faixas de áudio e vídeo (meios de mídia RTP) adicionadas durante o estabelecimento de conexão automaticamente transmitem dados para o servidor.
- Transmitido diretamente pela faixa de áudio (RTP). Não precisa enviar. `input_audio_buffer.append` eventos.
- Enviado como molduras de vídeo através da faixa de vídeo (RTP). O `input_image_buffer.append` O evento não é suportado.
WebRTC só suporta o modo VAD do lado do servidor (server_vad Ou semantic_vad). Modo manual não é suportado.
- Quatro. Receba respostas do modelo.
O formato da resposta do modelo depende das modalidades de saída configuradas.
WebSocket
- Apenas texto
Receba mensagens através do [resposta.texto.delta] (/api-reference/real-time-multimodal/server-events) evento. Recupere o texto completo com o [resposta.texto.feito] (/api-reference/real-time-multimodal/server-events) evento.
- Texto e áudio
- Tradução:/api-reference/real-time-multimodal/server-events) evento. Recupere o texto completo com o [resposta.audio\_transcript.done] (/api-reference/real-time-multimodal/server-events) evento.
- Recupere Base 64 codificada transmitindo dados de saída de áudio através da [resposta.audio.delta]/api-reference/real-time-multimodal/server-events) evento. [resposta.audio.feito] (/api-reference/real-time-multimodal/server-eventsO evento indica que a geração de áudio está completa.
WebRTC
- Apenas texto
O mesmo que WebSocket. Receba transmissões de texto através do DataChannel.
- Texto e áudio
- **Texto**: Recebido através do DataChannel como transmissão de eventos de texto, assim como WebSocket.
- Recebido e jogado em tempo real através de faixas RTP. Não há necessidade de recuperar dados de áudio através `response.audio.delta` eventos.
Seleção de modelos
Use Qwen3.8-Omni-Flash-Realtime para conversas em tempo real de áudio/vídeo. Aceita streaming de áudio/vídeo, retorna texto e áudio, e suporta chamadas personalizadas de função, MCP, áudio multicanal e clonagem de voz. Veja [Estabelecer uma conexão] (#1-establish-a-connection).
Suporta reconhecimento de fala para [113 línguas e dialetos] (/developer-guides/speech/omni-models) e geração de fala para [36 línguas e dialetos] (/developer-guides/speech/omni-models). Veja a lista de voz (/developer-guides/speech/omni-voice-list#qwen38-voices) para vozes e pré-visualizações de áudio.
Qwen3.5-Omni-Realtime capabilities
Qwen3.5-Omni-Realtime fornece as seguintes capacidades:
- Nível de inteligência
Em par com Qwen3.5-Plus.
-
** Busca na internet **
Pesquisa na web: o modelo busca de forma autônoma para responder perguntas em tempo real. Para detalhes, veja [Busca na Web] (#web-search).
-
"Tool chamando"
Chamada de função: o modelo invoca ferramentas externas. Para detalhes, veja [Qwen-Omni-Realtime Série] (/developer-guides/tool-calling/function-calling#rt02realtime01).
-
Interrupção semântica
Identifica intenção de conversação para evitar interrupções de backcanaling e ruído de fundo.
-
Controle de voz
Controle volume, taxa de fala, e emoção através de comandos de voz (por exemplo, "falar mais rápido", "mais alto", "em um tom feliz").
-
Linguagens suportadas
Suporta reconhecimento de fala para [113 línguas e dialetos] (/developer-guides/speech/omni-models) e geração de fala para [36 línguas e dialetos] (/developer-guides/speech/omni-models).
-
Vozes apoiadas
Suporta 55 vozes, incluindo 47 vozes multilingues e 8 vozes dialéticas. Para uma lista completa, veja [lista de voz] (#voice-list).
-
Clonagem de voz
Use uma voz clonada personalizada para conversas em tempo real.Qwen3.5-omni-plus-realtime e Qwen3.5-omni-flash-realtime). Para detalhes, veja [clonografia de voz] (/developer-guides/speech/voice-cloning).
Limitações
- Mutualmente exclusivas características: busca na web e chamada de ferramentas não podem ser ativadas ao mesmo tempo.
- Uma sessão de WebSocket pode durar até 120 minutos. A conexão fecha automaticamente neste limite.
qwen3.8-omni-flash-realtime suporta até 196608 tokens de entrada. O comprimento de entrada é calculado usando as mesmas regras que Qwen3.5-Omni-Realtime.
A história da conversa limita-se.
O modelo mantém o histórico de conversa até os seguintes limites de tempo e turno. Quando ultrapassado, a história mais antiga é descartada. Não é a duração da sessão inteira.
| Modelo | Audio max voltas | Vídeo max voltas | Duração máxima de áudio | Duração máxima do vídeo |
|---|---|---|---|---|
| qwen3.8-omni-flash-realtime | 100 voltas. | 50 voltas. | 600 segundos. | 240 segundos. |
| qwen3.5-omni-plus-realtime | 100 voltas. | 50 voltas. | 600 segundos. | 240 segundos. |
| qwen3.5-omni-flash-realtime | 80 voltas. | 50 voltas. | 480 segundos. | 120 segundos. |
- O vídeo é introduzido como quadros extraídos (recomendado: 1 fps). A duração máxima do vídeo é a duração acumulada do quadro retido - por exemplo, 240 s significa que apenas quadros dos últimos 240 segundos são mantidos.
Começando.
[Pegue uma chave API] (/api-reference/preparation/api-keyE o define como uma variável de ambiente./api-reference/preparation/export-api-key-env).
DashScope Python SDK
Prepare o ambiente de corrida
Sua versão Python deve ser 3,10 ou mais tarde.
Primeiro, instale PyAudio baseado no seu sistema operacional.
macOS
```bash
brew install portaudio && pip install pyaudio
```
Debian/Ubuntu
- Se você não está usando um ambiente virtual, instale-o diretamente usando o gerenciador de pacotes do sistema:
```bash
sudo apt-get install python3-pyaudio
```
- Se você está em um ambiente virtual, instale a compilação de dependências.
```bash
sudo apt update
sudo apt install -y python3-dev portaudio19-dev
```
Então, você pode instalar o pacote usando pip no ambiente virtual ativado.
```bash
pip install pyaudio
```
CentOS
```bash
sudo yum install -y portaudio portaudio-devel && pip install pyaudio
```
Windows
```powershell
pip install pyaudio
```
Após a instalação completa, instale as dependências restantes usando pip:
```bash
pip install websocket-client dashscope
```
Escolha um modo de interação
- Modo VAD (automaticamente detecta o início e o fim da fala)
O servidor determina automaticamente quando o usuário começa e pára de falar e responde de acordo.
- Modo manual (pressione para falar, solte para enviar)
O cliente controla o início e o fim do discurso. Depois que o usuário terminar de falar, o cliente deve enviar uma mensagem para o servidor.
VAD mode
Crie um novo arquivo Python chamado vad\_dash.py e copie o seguinte código no arquivo:
vad_dash.py
```python
# Dependencies: dashscope >= 1.26.5, pyaudio
import os
import base64
import time
import pyaudio
from dashscope.audio.qwen_omni import MultiModality, AudioFormat,OmniRealtimeCallback,OmniRealtimeConversation
import dashscope
# Configuration parameters: URL, API key, voice, model, model role
url = 'wss://maas.qwencloudapi.com/api-ws/v1/realtime'
# Configure the API key. If you have not set an environment variable, replace the following line with dashscope.api_key = "sk-xxx"
dashscope.api_key = os.getenv('DASHSCOPE_API_KEY')
# Specify the voice
voice = 'Tina'
# Specify the model
model = 'qwen3.8-omni-flash-realtime'
# Specify the model role
instructions = "You are Xiaoyun, a personal assistant. Please answer the user's questions in a humorous and witty way."
class SimpleCallback(OmniRealtimeCallback):
def __init__(self, pya):
self.pya = pya
self.out = None
def on_open(self):
# Initialize the audio output stream
self.out = self.pya.open(
format=pyaudio.paInt16,
channels=1,
rate=24000,
output=True
)
def on_event(self, response):
if response['type'] == 'response.audio.delta':
# Play the audio
self.out.write(base64.b64decode(response['delta']))
elif response['type'] == 'conversation.item.input_audio_transcription.delta':
# Streaming preview: text is the confirmed prefix, stash is the unconfirmed suffix.
preview = response.get('text', '') + response.get('stash', '')
print(f"\r[User] {preview}", end='', flush=True)
elif response['type'] == 'conversation.item.input_audio_transcription.completed':
# Print the transcribed text
print(f"\r[User] {response['transcript']}")
elif response['type'] == 'response.audio_transcript.done':
# Print the assistant's reply text
print(f"[LLM] {response['transcript']}")
# 1. Initialize the audio device
pya = pyaudio.PyAudio()
# 2. Create the callback function and session
callback = SimpleCallback(pya)
conv = OmniRealtimeConversation(model=model, callback=callback, url=url)
# 3. Establish the connection and configure the session
conv.connect()
conv.update_session(output_modalities=[MultiModality.AUDIO, MultiModality.TEXT], voice=voice, instructions=instructions)
# 4. Initialize the audio input stream
mic = pya.open(format=pyaudio.paInt16, channels=1, rate=16000, input=True)
# 5. Main loop to process audio input
print("Conversation started. Speak into the microphone (Ctrl+C to exit)...")
try:
while True:
audio_data = mic.read(3200, exception_on_overflow=False)
conv.append_audio(base64.b64encode(audio_data).decode())
time.sleep(0.01)
except KeyboardInterrupt:
# Clean up resources
conv.close()
mic.close()
callback.out.close()
pya.terminate()
print("\nConversation ended")
```
Corra. `vad_dash.py` Ter uma conversa em tempo real com Qwen-Omni-Realtime através do microfone. O sistema detecta o início e o fim de sua fala e automaticamente envia para o servidor sem intervenção manual.
Manual mode
Criar um novo arquivo Python chamado `manual_dash.py` e copie o seguinte código no arquivo:
manual_dash.py
```python
# This example uses Qwen3.5-Omni-Plus-Realtime for input transcription and session control.
# Dependencies: dashscope >= 1.26.5, pyaudio.
import os
import base64
import sys
import threading
import pyaudio
from dashscope.audio.qwen_omni import *
import dashscope
# If you have not set an environment variable, replace the following line with your API key: dashscope.api_key = "sk-xxx"
dashscope.api_key = os.getenv('DASHSCOPE_API_KEY')
voice = 'Tina'
class MyCallback(OmniRealtimeCallback):
"""Minimal callback: Initializes the speaker upon connection and plays the returned audio directly in the event."""
def __init__(self, ctx):
super().__init__()
self.ctx = ctx
def on_open(self) -> None:
# Initialize PyAudio and the speaker (24k/mono/16bit) after connection is established.
print('connection opened')
try:
self.ctx['pya'] = pyaudio.PyAudio()
self.ctx['out'] = self.ctx['pya'].open(
format=pyaudio.paInt16,
channels=1,
rate=24000,
output=True
)
print('audio output initialized')
except Exception as e:
print('[Error] audio init failed: {}'.format(e))
def on_close(self, close_status_code, close_msg) -> None:
print('connection closed with code: {}, msg: {}'.format(close_status_code, close_msg))
sys.exit(0)
def on_event(self, response: str) -> None:
try:
t = response['type']
handlers = {
'session.created': lambda r: print('start session: {}'.format(r['session']['id'])),
'conversation.item.input_audio_transcription.delta': self._transcription_delta,
'conversation.item.input_audio_transcription.completed': self._transcription_completed,
'response.audio_transcript.delta': lambda r: print('llm text: {}'.format(r['delta'])),
'response.audio.delta': self._play_audio,
'response.done': self._response_done,
}
h = handlers.get(t)
if h:
h(response)
except Exception as e:
print('[Error] {}'.format(e))
def _transcription_delta(self, response):
# Streaming preview: text is the confirmed prefix, stash is the unconfirmed suffix.
preview = response.get('text', '') + response.get('stash', '')
print(f"\r[User] {preview}", end='', flush=True)
def _transcription_completed(self, response):
print()
self.ctx['transcription_done'].set()
def _play_audio(self, response):
# Directly decode base64 and write to the output stream for playback.
if self.ctx['out'] is None:
return
try:
data = base64.b64decode(response['delta'])
self.ctx['out'].write(data)
except Exception as e:
print('[Error] audio playback failed: {}'.format(e))
def _response_done(self, response):
# Mark the current conversation turn as complete for the main loop to wait.
if self.ctx['conv'] is not None:
print('[Metric] response: {}, first text delay: {}, first audio delay: {}'.format(
self.ctx['conv'].get_last_response_id(),
self.ctx['conv'].get_last_first_text_delay(),
self.ctx['conv'].get_last_first_audio_delay(),
))
if self.ctx['resp_done'] is not None:
self.ctx['resp_done'].set()
def shutdown_ctx(ctx):
"""Safely release audio and PyAudio resources."""
try:
if ctx['out'] is not None:
ctx['out'].close()
ctx['out'] = None
except Exception:
pass
try:
if ctx['pya'] is not None:
ctx['pya'].terminate()
ctx['pya'] = None
except Exception:
pass
def stream_record_and_send(pya_inst: pyaudio.PyAudio, conversation, sample_rate=16000, chunk_size=3200):
"""Press Enter to stop recording; each chunk is streamed to the server immediately."""
stop_evt = threading.Event()
stream = pya_inst.open(
format=pyaudio.paInt16,
channels=1,
rate=sample_rate,
input=True,
frames_per_buffer=chunk_size
)
def _reader():
while not stop_evt.is_set():
try:
data = stream.read(chunk_size, exception_on_overflow=False)
conversation.append_audio(base64.b64encode(data).decode())
except Exception:
break
t = threading.Thread(target=_reader, daemon=True)
t.start()
input() # User presses Enter again to stop recording.
stop_evt.set()
t.join(timeout=1.0)
stream.close()
if __name__ == '__main__':
print('Initializing ...')
# Runtime context: Stores audio and session handles.
ctx = {'pya': None, 'out': None, 'conv': None, 'resp_done': threading.Event(), 'transcription_done': threading.Event()}
callback = MyCallback(ctx)
conversation = OmniRealtimeConversation(
model='qwen3.5-omni-plus-realtime',
callback=callback,
url="wss://maas.qwencloudapi.com/api-ws/v1/realtime",
)
try:
conversation.connect()
except Exception as e:
print('[Error] connect failed: {}'.format(e))
sys.exit(1)
ctx['conv'] = conversation
# Session configuration: Enable text and audio output (disable server-side VAD, switch to manual recording).
conversation.update_session(
output_modalities=[MultiModality.AUDIO, MultiModality.TEXT],
voice=voice,
enable_input_audio_transcription=True,
# The model for transcribing input audio.
input_audio_transcription_model='qwen3-asr-flash-realtime',
enable_turn_detection=False,
instructions="You are Xiaoyun, a personal assistant. Please answer the user's questions accurately and friendly, always responding with a helpful attitude."
)
try:
turn = 1
while True:
print(f"\n--- Turn {turn} ---")
print("Press Enter to start recording (enter q to exit)...")
user_input = input()
if user_input.strip().lower() in ['q', 'quit']:
print("User requested to exit...")
break
print("Recording... Press Enter again to stop.")
if ctx['pya'] is None:
ctx['pya'] = pyaudio.PyAudio()
stream_record_and_send(ctx['pya'], conversation)
ctx['resp_done'].clear()
ctx['transcription_done'].clear()
conversation.commit()
ctx['transcription_done'].wait(timeout=10)
print("Waiting for model response...")
conversation.create_response()
ctx['resp_done'].wait()
turn += 1
except KeyboardInterrupt:
print("\nProgram interrupted by user.")
finally:
shutdown_ctx(ctx)
print("Program exited.")
```
Corra. `manual_dash.py`. Pressione Enter para começar a falar. Pressione Enter novamente para receber a resposta de áudio do modelo.
Código de amostra no GitHub
Baixe [código de amostra completo] (https://github.com/aliyun/alibabacloud-bailian-speech-demo/tree/master/samples/conversation/omni/python) de GitHub, que inclui:
1. Captura áudio em tempo real de um microfone com modo VAD.`enable_turn_detection` - Verdade, e suporta interrupção de voz.
2. Captura áudio e vídeo em tempo real de um microfone e câmera com modo VAD e suporta interrupção de voz.
3. ** Chamada local**: Usa áudio local e imagens como entrada com o modo manual (`enable_turn_detection` - Falso.
Use fones de ouvido para reproduzir áudio para evitar que ecos acionem a interrupção da voz.
DashScope Java SDK
DashScope Java SDK versão 2.22.15 ou posterior é necessária. Veja [Install DashScope SDK] (/api-reference/preparation/install-sdk) para Maven e Gradle armação.
Escolha um modo de interação
- Modo VAD (automaticamente detecta o início e o fim da fala)
A API em tempo real determina automaticamente quando o usuário começa e pára de falar e responde de acordo.
- Modo manual (pressione para falar, solte para enviar)
O cliente controla o início e o fim do discurso. Depois que o usuário terminar de falar, o cliente deve enviar uma mensagem para o servidor.
VAD mode
OmniServerVad.java
```java
// This example uses Qwen3.5-Omni-Plus-Realtime for input transcription and session control.
import com.alibaba.dashscope.audio.omni.*;
import com.alibaba.dashscope.exception.NoApiKeyException;
import com.google.gson.JsonObject;
import javax.sound.sampled.*;
import java.nio.ByteBuffer;
import java.util.Arrays;
import java.util.Base64;
import java.util.Map;
import java.util.Queue;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.atomic.AtomicBoolean;
public class OmniServerVad {
static class SequentialAudioPlayer {
private final SourceDataLine line;
private final Queue<byte[]> audioQueue = new ConcurrentLinkedQueue<>();
private final Thread playerThread;
private final AtomicBoolean shouldStop = new AtomicBoolean(false);
public SequentialAudioPlayer() throws LineUnavailableException {
AudioFormat format = new AudioFormat(24000, 16, 1, true, false);
line = AudioSystem.getSourceDataLine(format);
line.open(format);
line.start();
playerThread = new Thread(() -> {
while (!shouldStop.get()) {
byte[] audio = audioQueue.poll();
if (audio != null) {
line.write(audio, 0, audio.length);
} else {
try { Thread.sleep(10); } catch (InterruptedException ignored) {}
}
}
}, "AudioPlayer");
playerThread.start();
}
public void play(String base64Audio) {
try {
byte[] audio = Base64.getDecoder().decode(base64Audio);
audioQueue.add(audio);
} catch (Exception e) {
System.err.println("Audio decoding failed: " + e.getMessage());
}
}
public void cancel() {
audioQueue.clear();
line.flush();
}
public void close() {
shouldStop.set(true);
try { playerThread.join(1000); } catch (InterruptedException ignored) {}
line.drain();
line.close();
}
}
public static void main(String[] args) {
try {
SequentialAudioPlayer player = new SequentialAudioPlayer();
AtomicBoolean userIsSpeaking = new AtomicBoolean(false);
AtomicBoolean shouldStop = new AtomicBoolean(false);
OmniRealtimeParam param = OmniRealtimeParam.builder()
.model("qwen3.5-omni-plus-realtime")
.apikey(System.getenv("DASHSCOPE_API_KEY"))
.url("wss://maas.qwencloudapi.com/api-ws/v1/realtime")
.build();
OmniRealtimeConversation conversation = new OmniRealtimeConversation(param, new OmniRealtimeCallback() {
@Override public void onOpen() {
System.out.println("Connection established");
}
@Override public void onClose(int code, String reason) {
System.out.println("Connection closed (" + code + "): " + reason);
shouldStop.set(true);
}
@Override public void onEvent(JsonObject event) {
handleEvent(event, player, userIsSpeaking);
}
});
conversation.connect();
conversation.updateSession(OmniRealtimeConfig.builder()
.modalities(Arrays.asList(OmniRealtimeModality.AUDIO, OmniRealtimeModality.TEXT))
.voice("Tina")
.enableTurnDetection(true)
.enableInputAudioTranscription(true)
.parameters(Map.of("instructions",
"You are an AI customer service agent for a five-star hotel. Answer customer inquiries about room types, facilities, prices, and booking policies accurately and friendly. Always respond with a professional and helpful attitude. Do not provide unconfirmed information or information beyond the scope of the hotel's services."))
.build()
);
System.out.println("Please start speaking (automatic detection of speech start/end, press Ctrl+C to exit)...");
AudioFormat format = new AudioFormat(16000, 16, 1, true, false);
TargetDataLine mic = AudioSystem.getTargetDataLine(format);
mic.open(format);
mic.start();
ByteBuffer buffer = ByteBuffer.allocate(3200);
while (!shouldStop.get()) {
int bytesRead = mic.read(buffer.array(), 0, buffer.capacity());
if (bytesRead > 0) {
try {
conversation.appendAudio(Base64.getEncoder().encodeToString(buffer.array()));
} catch (Exception e) {
if (e.getMessage() != null && e.getMessage().contains("closed")) {
System.out.println("Conversation closed. Stopping recording.");
break;
}
}
}
Thread.sleep(20);
}
conversation.close(1000, "Normal exit");
player.close();
mic.close();
System.out.println("\nProgram exited.");
} catch (NoApiKeyException e) {
System.err.println("API KEY not found: Please set the DASHSCOPE_API_KEY environment variable.");
System.exit(1);
} catch (Exception e) {
e.printStackTrace();
}
}
private static void handleEvent(JsonObject event, SequentialAudioPlayer player, AtomicBoolean userIsSpeaking) {
String type = event.get("type").getAsString();
switch (type) {
case "input_audio_buffer.speech_started":
System.out.println("\n[User started speaking]");
player.cancel();
userIsSpeaking.set(true);
break;
case "input_audio_buffer.speech_stopped":
System.out.println("[User stopped speaking]");
userIsSpeaking.set(false);
break;
case "response.audio.delta":
if (!userIsSpeaking.get()) {
player.play(event.get("delta").getAsString());
}
break;
case "conversation.item.input_audio_transcription.delta":
// Streaming preview: text is confirmed prefix, stash is unconfirmed suffix
String preview = event.get("text").getAsString() + event.get("stash").getAsString();
System.out.print("\rUser: " + preview);
break;
case "conversation.item.input_audio_transcription.completed":
System.out.println("\rUser: " + event.get("transcript").getAsString());
break;
case "response.audio_transcript.delta":
System.out.print(event.get("delta").getAsString());
break;
case "response.done":
System.out.println("Response complete");
break;
}
}
}
```
Você pode correr o `OmniServerVad.main()` método para ter uma conversa em tempo real com o modelo em tempo real usando seu microfone. O sistema detecta o início do seu áudio e automaticamente envia para o servidor, então você não precisa enviá-lo manualmente.
Manual mode
OmniWithoutServerVad.java
```java
// This example uses Qwen3.5-Omni-Plus-Realtime for input transcription and session control.
// DashScope Java SDK version 2.22.15 or later
import com.alibaba.dashscope.audio.omni.*;
import com.alibaba.dashscope.exception.NoApiKeyException;
import com.google.gson.JsonObject;
import javax.sound.sampled.*;
import java.io.IOException;
import java.util.Arrays;
import java.util.Base64;
import java.util.HashMap;
import java.util.Queue;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicBoolean;
import java.util.concurrent.atomic.AtomicReference;
public class Main {
// RealtimePcmPlayer class definition starts
public static class RealtimePcmPlayer {
private int sampleRate;
private SourceDataLine line;
private AudioFormat audioFormat;
private Thread decoderThread;
private Thread playerThread;
private AtomicBoolean stopped = new AtomicBoolean(false);
private Queue<String> b64AudioBuffer = new ConcurrentLinkedQueue<>();
private Queue<byte[]> RawAudioBuffer = new ConcurrentLinkedQueue<>();
// The constructor initializes the audio format and audio line.
public RealtimePcmPlayer(int sampleRate) throws LineUnavailableException {
this.sampleRate = sampleRate;
this.audioFormat = new AudioFormat(this.sampleRate, 16, 1, true, false);
DataLine.Info info = new DataLine.Info(SourceDataLine.class, audioFormat);
line = (SourceDataLine) AudioSystem.getLine(info);
line.open(audioFormat);
line.start();
decoderThread = new Thread(new Runnable() {
@Override
public void run() {
while (!stopped.get()) {
String b64Audio = b64AudioBuffer.poll();
if (b64Audio != null) {
byte[] rawAudio = Base64.getDecoder().decode(b64Audio);
RawAudioBuffer.add(rawAudio);
} else {
try {
Thread.sleep(100);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
}
}
});
playerThread = new Thread(new Runnable() {
@Override
public void run() {
while (!stopped.get()) {
byte[] rawAudio = RawAudioBuffer.poll();
if (rawAudio != null) {
try {
playChunk(rawAudio);
} catch (IOException e) {
throw new RuntimeException(e);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
} else {
try {
Thread.sleep(100);
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
}
}
}
});
decoderThread.start();
playerThread.start();
}
// Play an audio chunk and block until playback is complete.
private void playChunk(byte[] chunk) throws IOException, InterruptedException {
if (chunk == null || chunk.length == 0) return;
int bytesWritten = 0;
while (bytesWritten < chunk.length) {
bytesWritten += line.write(chunk, bytesWritten, chunk.length - bytesWritten);
}
int audioLength = chunk.length / (this.sampleRate*2/1000);
// Wait for the audio in the buffer to finish playing.
Thread.sleep(audioLength - 10);
}
public void write(String b64Audio) {
b64AudioBuffer.add(b64Audio);
}
public void cancel() {
b64AudioBuffer.clear();
RawAudioBuffer.clear();
}
public void waitForComplete() throws InterruptedException {
while (!b64AudioBuffer.isEmpty() || !RawAudioBuffer.isEmpty()) {
Thread.sleep(100);
}
line.drain();
}
public void shutdown() throws InterruptedException {
stopped.set(true);
decoderThread.join();
playerThread.join();
if (line != null && line.isRunning()) {
line.drain();
line.close();
}
}
} // RealtimePcmPlayer class definition ends
// Add a recording method
private static void recordAndSend(TargetDataLine line, OmniRealtimeConversation conversation) {
byte[] buffer = new byte[3200];
AtomicBoolean stopRecording = new AtomicBoolean(false);
// Start a thread to listen for the Enter key.
Thread enterKeyListener = new Thread(() -> {
try {
System.in.read();
stopRecording.set(true);
} catch (IOException e) {
e.printStackTrace();
}
});
enterKeyListener.start();
// Recording loop
while (!stopRecording.get()) {
int count = line.read(buffer, 0, buffer.length);
if (count > 0) {
byte[] chunk = Arrays.copyOf(buffer, count);
conversation.appendAudio(Base64.getEncoder().encodeToString(chunk));
}
}
}
public static void main(String[] args) throws InterruptedException, LineUnavailableException {
OmniRealtimeParam param = OmniRealtimeParam.builder()
.model("qwen3.5-omni-plus-realtime")
// If you have not configured an environment variable, replace the following line with your API key: .apikey("sk-xxx")
.apikey(System.getenv("DASHSCOPE_API_KEY"))
.url("wss://maas.qwencloudapi.com/api-ws/v1/realtime")
.build();
AtomicReference<CountDownLatch> responseDoneLatch = new AtomicReference<>(null);
responseDoneLatch.set(new CountDownLatch(1));
AtomicReference<CountDownLatch> transcriptionDoneLatch = new AtomicReference<>(null);
transcriptionDoneLatch.set(new CountDownLatch(1));
RealtimePcmPlayer audioPlayer = new RealtimePcmPlayer(24000);
final AtomicReference<OmniRealtimeConversation> conversationRef = new AtomicReference<>(null);
OmniRealtimeConversation conversation = new OmniRealtimeConversation(param, new OmniRealtimeCallback() {
@Override
public void onOpen() {
System.out.println("connection opened");
}
@Override
public void onEvent(JsonObject message) {
String type = message.get("type").getAsString();
switch(type) {
case "session.created":
System.out.println("start session: " + message.get("session").getAsJsonObject().get("id").getAsString());
break;
case "conversation.item.input_audio_transcription.delta":
String preview = message.get("text").getAsString() + message.get("stash").getAsString();
System.out.print("\rquestion: " + preview);
break;
case "conversation.item.input_audio_transcription.completed":
System.out.println("\rquestion: " + message.get("transcript").getAsString());
transcriptionDoneLatch.get().countDown();
break;
case "response.audio_transcript.delta":
System.out.println("got llm response delta: " + message.get("delta").getAsString());
break;
case "response.audio.delta":
String recvAudioB64 = message.get("delta").getAsString();
audioPlayer.write(recvAudioB64);
break;
case "response.done":
System.out.println("======RESPONSE DONE======");
if (conversationRef.get() != null) {
System.out.println("[Metric] response: " + conversationRef.get().getResponseId() +
", first text delay: " + conversationRef.get().getFirstTextDelay() +
" ms, first audio delay: " + conversationRef.get().getFirstAudioDelay() + " ms");
}
responseDoneLatch.get().countDown();
break;
default:
break;
}
}
@Override
public void onClose(int code, String reason) {
System.out.println("connection closed code: " + code + ", reason: " + reason);
}
});
conversationRef.set(conversation);
try {
conversation.connect();
} catch (NoApiKeyException e) {
throw new RuntimeException(e);
}
OmniRealtimeConfig config = OmniRealtimeConfig.builder()
.modalities(Arrays.asList(OmniRealtimeModality.AUDIO, OmniRealtimeModality.TEXT))
.voice("Tina")
.enableTurnDetection(false)
// Set the model role.
.parameters(new HashMap<String, Object>() {{
put("instructions","You are Xiaoyun, a personal assistant. Please answer the user's questions accurately and friendly, always responding with a helpful attitude.");
}})
.build();
conversation.updateSession(config);
// Add microphone recording functionality.
AudioFormat format = new AudioFormat(16000, 16, 1, true, false);
DataLine.Info info = new DataLine.Info(TargetDataLine.class, format);
if (!AudioSystem.isLineSupported(info)) {
System.out.println("Line not supported");
return;
}
TargetDataLine line = null;
try {
line = (TargetDataLine) AudioSystem.getLine(info);
line.open(format);
line.start();
while (true) {
System.out.println("Press Enter to start recording...");
System.in.read();
System.out.println("Recording started. Please speak... Press Enter again to stop recording and send.");
recordAndSend(line, conversation);
conversation.commit();
transcriptionDoneLatch.get().await(10, TimeUnit.SECONDS);
System.out.println("Waiting for model response...");
conversation.createResponse(null, null);
// Reset the latch for the next wait.
responseDoneLatch.set(new CountDownLatch(1));
transcriptionDoneLatch.set(new CountDownLatch(1));
}
} catch (LineUnavailableException | IOException | InterruptedException e) {
e.printStackTrace();
} finally {
if (line != null) {
line.stop();
line.close();
}
}
}
}
```
Corra o `OmniWithoutServerVad.main()` Método. Pressione Enter para começar a gravar. Durante a gravação, pressione Enter novamente para parar de gravar e enviar o áudio. Então, receba e jogue a resposta do modelo.
Código de amostra no GitHub
Baixe [código de amostra completo] (https://github.com/aliyun/alibabacloud-bailian-speech-demo/tree/master/samples/conversation/omni/java) de GitHub, que inclui:
1. Captura áudio em tempo real de um microfone com modo VAD.`enableTurnDetection` E suporta interrupção de voz.
2. Captura áudio e vídeo em tempo real de um microfone e câmera com modo VAD e suporta interrupção de voz.
3. ** Chamada local**: Usa áudio local e imagens como entrada com o modo manual (`enableTurnDetection` - Falso.
Defina o enableTurnDetection parâmetro para true para modo VAD ou false para o modo manual. Use fones de ouvido para reproduzir áudio para evitar que ecos acionem a interrupção da voz.
WebSocket (Python)
<a id="qwen38-realtime" />
### Pergunta e resposta de áudio mínima.
Configure uma [chave API] (/api-reference/preparation/api-key) e instalar `websocket-client`. Preparar. `REALTIME_WS_URL` O exemplo adiciona o parâmetro de consulta do modelo. Preparar. `PCM_FILE` Para um curto 16 kHz, 16 bits, arquivo de fala mono PCM sem cabeçalho de arquivo.
Este exemplo usa o modo manual: aguarde a configuração da sessão, envie áudio, commita o buffer e peça uma resposta. O texto está impresso no terminal, o áudio está salvo. `reply.pcm` 24 kHz PCM, não um arquivo WAV).
Execute os seguintes comandos em um terminal MacOS ou Linux. Defina o URL do terminal WebSocket sem o parâmetro de consulta do modelo; o exemplo adiciona-o automaticamente.
```bash
python3 -m pip install websocket-client
export REALTIME_WS_URL="<workspace-websocket-url>"
export PCM_FILE="./input.pcm"
```
Se você tem um arquivo mono, 16 kHz, 16 bits PCM WAV chamado `input.wav`Use Python para extrair suas molduras PCM. Este passo verifica o formato, não reamostra o áudio.
```python
import wave
from pathlib import Path
with wave.open("input.wav", "rb") as source:
if (source.getnchannels(), source.getframerate(), source.getsampwidth(),
source.getcomptype()) != (1, 16000, 2, "NONE"):
raise ValueError("Expected mono 16 kHz, 16-bit PCM WAV audio.")
Path("input.pcm").write_bytes(source.readframes(source.getnframes()))
```
Execute o seguinte cliente depois de preparar a entrada:
```python
import base64
import json
import os
from pathlib import Path
from urllib.parse import parse_qsl, urlencode, urlsplit, urlunsplit
import websocket
parts = urlsplit(os.environ["REALTIME_WS_URL"])
query = dict(parse_qsl(parts.query))
query["model"] = "qwen3.8-omni-flash-realtime"
url = urlunsplit((parts.scheme, parts.netloc, parts.path, urlencode(query), ""))
audio = Path(os.environ["PCM_FILE"]).read_bytes()
if not audio or len(audio) % 2:
raise ValueError("Provide non-empty 16-bit mono PCM audio.")
ws = websocket.create_connection(
url,
header=["Authorization: Bearer " + os.environ["DASHSCOPE_API_KEY"]],
timeout=30,
)
def receive():
event = json.loads(ws.recv())
if event["type"] == "error":
raise RuntimeError(event.get("error"))
return event
try:
while receive()["type"] != "session.created":
pass
ws.send(json.dumps({
"type": "session.update",
"session": {
"modalities": ["text", "audio"],
"turn_detection": None,
"audio": {
"input": {"format": {
"type": "pcm", "sample_rate": 16000,
"sample_format": "s16le", "channels": 1,
"packing": "interleaved", "channel_layout": "mono",
}},
"output": {
"voice": "longanlingxin",
"format": {"type": "pcm", "sample_rate": 24000},
},
},
},
}))
while receive()["type"] != "session.updated":
pass
for offset in range(0, len(audio), 3200):
ws.send(json.dumps({
"type": "input_audio_buffer.append",
"audio": base64.b64encode(audio[offset:offset + 3200]).decode("ascii"),
}))
ws.send(json.dumps({"type": "input_audio_buffer.commit"}))
while receive()["type"] != "input_audio_buffer.committed":
pass
ws.send(json.dumps({"type": "response.create"}))
with open("reply.pcm", "wb") as output:
while True:
event = receive()
if event["type"] == "response.audio.delta":
output.write(base64.b64decode(event["delta"]))
elif event["type"] in ("response.audio_transcript.delta", "response.text.delta"):
print(event["delta"], end="", flush=True)
elif event["type"] == "response.done":
print("\nResponse status:", event["response"]["status"])
break
finally:
ws.close()
```
Depois de uma resposta bem sucedida, converta `reply.pcm` para um arquivo WAV para reprodução com um reprodutor de áudio:
```python
import wave
from pathlib import Path
with wave.open("reply.wav", "wb") as target:
target.setnchannels(1)
target.setsampwidth(2)
target.setframerate(24000)
target.writeframes(Path("reply.pcm").read_bytes())
```
Prepare the runtime environment
Sua versão Python deve ser 3,10 ou mais tarde.
Primeiro, instale PyAudio baseado no seu sistema operacional.
macOS
```bash
brew install portaudio && pip install pyaudio
```
Debian/Ubuntu
```bash
sudo apt-get install python3-pyaudio
or
pip install pyaudio
```
Nós recomendamos correr. `pip install pyaudio`. Se a instalação falhar, primeiro instale o `portaudio` dependência para o seu sistema operacional.
CentOS
```bash
sudo yum install -y portaudio portaudio-devel && pip install pyaudio
```
Windows
```powershell
pip install pyaudio
```
Após a instalação completa, instale as dependências relacionadas ao WebSocket usando pip:
```bash
pip install websockets==15.0.1
```
Crie o cliente.
Criar um novo arquivo Python chamado `omni_realtime_client.py` Em seu diretório local e copie o seguinte código no arquivo:
omni_realtime_client.py
```python
import asyncio
import websockets
import json
import base64
import time
from typing import Optional, Callable, List, Dict, Any
from enum import Enum
class TurnDetectionMode(Enum):
SERVER_VAD = "server_vad"
SEMANTIC_VAD = "semantic_vad" # Recommended for Qwen3.8-Omni-Flash-Realtime and the Qwen3.5-Omni-Realtime series
MANUAL = "manual"
class OmniRealtimeClient:
def __init__(
self,
base_url,
api_key: str,
model: str = "",
voice: str = "Tina",
instructions: str = "You are a helpful assistant.",
turn_detection_mode: TurnDetectionMode = TurnDetectionMode.SERVER_VAD,
on_text_delta: Optional[Callable[[str], None]] = None,
on_audio_delta: Optional[Callable[[bytes], None]] = None,
on_input_transcript: Optional[Callable[[str], None]] = None,
on_output_transcript: Optional[Callable[[str], None]] = None,
extra_event_handlers: Optional[Dict[str, Callable[[Dict[str, Any]], None]]] = None
):
self.base_url = base_url
self.api_key = api_key
self.model = model
self.voice = voice
self.instructions = instructions
self.ws = None
self.on_text_delta = on_text_delta
self.on_audio_delta = on_audio_delta
self.on_input_transcript = on_input_transcript
self.on_output_transcript = on_output_transcript
self.turn_detection_mode = turn_detection_mode
self.extra_event_handlers = extra_event_handlers or {}
# Current response status
self._current_response_id = None
self._current_item_id = None
self._is_responding = False
# Input/output transcript printing status
self._print_input_transcript = True
self._output_transcript_buffer = ""
async def connect(self) -> None:
"""Establish a WebSocket connection with the Realtime API."""
url = f"{self.base_url}?model={self.model}"
headers = {
"Authorization": f"Bearer {self.api_key}"
}
self.ws = await websockets.connect(url, additional_headers=headers)
# Session configuration
session_config = {
"modalities": ["text", "audio"],
"voice": self.voice,
"instructions": self.instructions,
"input_audio_format": "pcm",
"output_audio_format": "pcm",
"input_audio_transcription": {
"model": "qwen3-asr-flash-realtime"
}
}
if self.turn_detection_mode == TurnDetectionMode.MANUAL:
session_config['turn_detection'] = None
await self.update_session(session_config)
elif self.turn_detection_mode == TurnDetectionMode.SERVER_VAD:
session_config['turn_detection'] = {
"type": "server_vad",
"threshold": 0.1,
"prefix_padding_ms": 500,
"silence_duration_ms": 900
}
await self.update_session(session_config)
elif self.turn_detection_mode == TurnDetectionMode.SEMANTIC_VAD:
session_config['turn_detection'] = {
"type": "semantic_vad",
"threshold": 0.1,
"prefix_padding_ms": 500,
"silence_duration_ms": 900
}
await self.update_session(session_config)
else:
raise ValueError(f"Invalid turn detection mode: {self.turn_detection_mode}")
async def send_event(self, event) -> None:
event['event_id'] = "event_" + str(int(time.time() * 1000))
await self.ws.send(json.dumps(event))
async def update_session(self, config: Dict[str, Any]) -> None:
"""Update the session configuration."""
event = {
"type": "session.update",
"session": config
}
await self.send_event(event)
async def stream_audio(self, audio_chunk: bytes) -> None:
"""Stream raw audio data to the API."""
# Only 16-bit, 16 kHz, mono PCM is supported.
audio_b64 = base64.b64encode(audio_chunk).decode()
append_event = {
"type": "input_audio_buffer.append",
"audio": audio_b64
}
await self.send_event(append_event)
async def commit_audio_buffer(self) -> None:
"""Commit the audio buffer to trigger processing."""
event = {
"type": "input_audio_buffer.commit"
}
await self.send_event(event)
async def append_image(self, image_chunk: bytes) -> None:
"""Append image data to the image buffer.
Image data can come from local files or a real-time video stream.
Note:
- The image format must be JPG or JPEG. A resolution of 480p or 720p is recommended. The maximum supported resolution is 1080p.
- A single image after Base64 encoding must not exceed 256 KB. We recommend keeping the raw image size below 190 KB before encoding.
- Encode the image data to Base64 before sending.
- We recommend sending images to the server at a rate of no more than 1 frame per second.
- You must send audio data at least once before sending image data.
"""
image_b64 = base64.b64encode(image_chunk).decode()
event = {
"type": "input_image_buffer.append",
"image": image_b64
}
await self.send_event(event)
async def create_response(self) -> None:
"""Request the API to generate a response (only needs to be called in manual mode)."""
event = {
"type": "response.create"
}
await self.send_event(event)
async def cancel_response(self) -> None:
"""Cancel the current response."""
event = {
"type": "response.cancel"
}
await self.send_event(event)
async def handle_interruption(self):
"""Handle user interruption of the current response."""
if not self._is_responding:
return
# 1. Cancel the current response.
if self._current_response_id:
await self.cancel_response()
self._is_responding = False
self._current_response_id = None
self._current_item_id = None
async def handle_messages(self) -> None:
try:
async for message in self.ws:
event = json.loads(message)
event_type = event.get("type")
if event_type == "error":
print(" Error: ", event['error'])
continue
elif event_type == "response.created":
self._current_response_id = event.get("response", {}).get("id")
self._is_responding = True
elif event_type == "response.output_item.added":
self._current_item_id = event.get("item", {}).get("id")
elif event_type == "response.done":
self._is_responding = False
self._current_response_id = None
self._current_item_id = None
if event_type in self.extra_event_handlers:
self.extra_event_handlers[event_type](event)
elif event_type == "input_audio_buffer.speech_started":
print("Speech start detected")
if self._is_responding:
print("Handling interruption")
await self.handle_interruption()
elif event_type == "input_audio_buffer.speech_stopped":
print("Speech end detected")
elif event_type == "response.text.delta":
if self.on_text_delta:
self.on_text_delta(event["delta"])
elif event_type == "response.audio.delta":
if self.on_audio_delta:
audio_bytes = base64.b64decode(event["delta"])
self.on_audio_delta(audio_bytes)
elif event_type == "conversation.item.input_audio_transcription.delta":
# Streaming preview: text is confirmed prefix, stash is unconfirmed suffix
preview = event.get("text", "") + event.get("stash", "")
print(f"\rUser: {preview}", end="", flush=True)
elif event_type == "conversation.item.input_audio_transcription.completed":
transcript = event.get("transcript", "")
print(f"\rUser: {transcript}")
if self.on_input_transcript:
await asyncio.to_thread(self.on_input_transcript, transcript)
self._print_input_transcript = True
elif event_type == "response.audio_transcript.delta":
if self.on_output_transcript:
delta = event.get("delta", "")
if not self._print_input_transcript:
self._output_transcript_buffer += delta
else:
if self._output_transcript_buffer:
await asyncio.to_thread(self.on_output_transcript, self._output_transcript_buffer)
self._output_transcript_buffer = ""
await asyncio.to_thread(self.on_output_transcript, delta)
elif event_type == "response.audio_transcript.done":
print(f"LLM: {event.get('transcript', '')}")
self._print_input_transcript = False
elif event_type in self.extra_event_handlers:
self.extra_event_handlers[event_type](event)
except websockets.exceptions.ConnectionClosed:
print(" Connection closed")
except Exception as e:
print(" Error in message handling: ", str(e))
async def close(self) -> None:
"""Close the WebSocket connection."""
if self.ws:
await self.ws.close()
```
Escolha um modo de interação.
- Modo VAD (automaticamente detecta o início e o fim da fala)
A API em tempo real determina automaticamente quando o usuário começa e pára de falar e responde de acordo.
- Modo manual (pressione para falar, solte para enviar)
O cliente controla o início e o fim do discurso. Depois que o usuário terminar de falar, o cliente deve enviar uma mensagem para o servidor.
VAD mode
No mesmo diretório que `omni_realtime_client.py`, criar outro arquivo Python chamado `vad_mode.py` e copie o seguinte código no arquivo:
vad_mode.py
```python
# -- coding: utf-8 --
import os, asyncio, pyaudio, queue, threading
from omni_realtime_client import OmniRealtimeClient, TurnDetectionMode
# Audio player class (handles interruptions)
class AudioPlayer:
def __init__(self, pyaudio_instance, rate=24000):
self.stream = pyaudio_instance.open(format=pyaudio.paInt16, channels=1, rate=rate, output=True)
self.queue = queue.Queue()
self.stop_evt = threading.Event()
self.interrupt_evt = threading.Event()
threading.Thread(target=self._run, daemon=True).start()
def _run(self):
while not self.stop_evt.is_set():
try:
data = self.queue.get(timeout=0.5)
if data is None: break
if not self.interrupt_evt.is_set(): self.stream.write(data)
self.queue.task_done()
except queue.Empty: continue
def add_audio(self, data): self.queue.put(data)
def handle_interrupt(self): self.interrupt_evt.set(); self.queue.queue.clear()
def stop(self): self.stop_evt.set(); self.queue.put(None); self.stream.stop_stream(); self.stream.close()
# Record from microphone and send
async def record_and_send(client):
p = pyaudio.PyAudio()
stream = p.open(format=pyaudio.paInt16, channels=1, rate=16000, input=True, frames_per_buffer=3200)
print("Recording started. Please speak...")
try:
while True:
audio_data = stream.read(3200)
await client.stream_audio(audio_data)
await asyncio.sleep(0.02)
finally:
stream.stop_stream(); stream.close(); p.terminate()
async def main():
p = pyaudio.PyAudio()
player = AudioPlayer(pyaudio_instance=p)
client = OmniRealtimeClient(
base_url="wss://maas.qwencloudapi.com/api-ws/v1/realtime",
api_key=os.environ.get("DASHSCOPE_API_KEY"),
model="qwen3.8-omni-flash-realtime",
voice="Tina",
instructions="You are Xiaoyun, a witty and humorous assistant.",
# We recommend that you set this parameter to SEMANTIC_VAD when you use the qwen3.5-omni-realtime model.
turn_detection_mode=TurnDetectionMode.SEMANTIC_VAD,
on_text_delta=lambda t: print(f"\nAssistant: {t}", end="", flush=True),
on_audio_delta=player.add_audio,
)
await client.connect()
print("Connection successful. Starting real-time conversation...")
# Run concurrently
await asyncio.gather(client.handle_messages(), record_and_send(client))
if __name__ == "__main__":
try:
asyncio.run(main())
except KeyboardInterrupt:
print("\nProgram exited.")
```
Corra. `vad_mode.py` Ter uma conversa em tempo real com Qwen-Omni-Realtime através do microfone. O sistema detecta o início e o fim de sua fala e automaticamente envia para o servidor sem intervenção manual.
Manual mode
No mesmo diretório que `omni_realtime_client.py`, criar outro arquivo Python chamado `manual_mode.py` e copie o seguinte código no arquivo:
manual_mode.py
```python
# -- coding: utf-8 --
import os
import asyncio
import time
import threading
import queue
import pyaudio
from omni_realtime_client import OmniRealtimeClient, TurnDetectionMode
class AudioPlayer:
"""Real-time audio player class"""
def __init__(self, sample_rate=24000, channels=1, sample_width=2):
self.sample_rate = sample_rate
self.channels = channels
self.sample_width = sample_width # 2 bytes for 16-bit
self.audio_queue = queue.Queue()
self.is_playing = False
self.play_thread = None
self.pyaudio_instance = None
self.stream = None
self._lock = threading.Lock() # Add a lock for synchronized access
self._last_data_time = time.time() # Record the time the last data was received
self._response_done = False # Add a flag to indicate response completion
self._waiting_for_response = False # Flag to indicate if waiting for a server response
# Record the time the last data was written to the audio stream and the duration of the most recent audio chunk for more accurate playback end detection
self._last_play_time = time.time()
self._last_chunk_duration = 0.0
def start(self):
"""Start the audio player"""
with self._lock:
if self.is_playing:
return
self.is_playing = True
try:
self.pyaudio_instance = pyaudio.PyAudio()
# Create an audio output stream
self.stream = self.pyaudio_instance.open(
format=pyaudio.paInt16, # 16-bit
channels=self.channels,
rate=self.sample_rate,
output=True,
frames_per_buffer=1024
)
# Start the playback thread
self.play_thread = threading.Thread(target=self._play_audio)
self.play_thread.daemon = True
self.play_thread.start()
print("Audio player started")
except Exception as e:
print(f"Failed to start audio player: {e}")
self._cleanup_resources()
raise
def stop(self):
"""Stop the audio player"""
with self._lock:
if not self.is_playing:
return
self.is_playing = False
# Clear the queue
while not self.audio_queue.empty():
try:
self.audio_queue.get_nowait()
except queue.Empty:
break
# Wait for the playback thread to finish (wait outside the lock to avoid deadlock)
if self.play_thread and self.play_thread.is_alive():
self.play_thread.join(timeout=2.0)
# Acquire the lock again to clean up resources
with self._lock:
self._cleanup_resources()
print("Audio player stopped")
def _cleanup_resources(self):
"""Clean up audio resources (must be called within the lock)"""
try:
# Close the audio stream
if self.stream:
if not self.stream.is_stopped():
self.stream.stop_stream()
self.stream.close()
self.stream = None
except Exception as e:
print(f"Error closing audio stream: {e}")
try:
if self.pyaudio_instance:
self.pyaudio_instance.terminate()
self.pyaudio_instance = None
except Exception as e:
print(f"Error terminating PyAudio: {e}")
def add_audio_data(self, audio_data):
"""Add audio data to the playback queue"""
if self.is_playing and audio_data:
self.audio_queue.put(audio_data)
with self._lock:
self._last_data_time = time.time() # Update the time the last data was received
self._waiting_for_response = False # Data received, no longer waiting
def stop_receiving_data(self):
"""Mark that no more new audio data will be received"""
with self._lock:
self._response_done = True
self._waiting_for_response = False # Response ended, no longer waiting
def prepare_for_next_turn(self):
"""Reset the player state for the next conversation turn."""
with self._lock:
self._response_done = False
self._last_data_time = time.time()
self._last_play_time = time.time()
self._last_chunk_duration = 0.0
self._waiting_for_response = True # Start waiting for the next response
# Clear any remaining audio data from the previous turn
while not self.audio_queue.empty():
try:
self.audio_queue.get_nowait()
except queue.Empty:
break
def is_finished_playing(self):
"""Check if all audio data has been played"""
with self._lock:
queue_size = self.audio_queue.qsize()
time_since_last_data = time.time() - self._last_data_time
time_since_last_play = time.time() - self._last_play_time
# ---------------------- Smart end detection ----------------------
# 1. Preferred: If the server has marked completion and the playback queue is empty.
# Wait for the most recent audio chunk to finish playing (chunk duration + 0.1s tolerance).
if self._response_done and queue_size == 0:
min_wait = max(self._last_chunk_duration + 0.1, 0.5) # Wait at least 0.5s
if time_since_last_play >= min_wait:
return True
# 2. Fallback: If no new data has been received for a long time and the playback queue is empty.
# This logic serves as a safeguard if the server does not explicitly send `response.done`.
if not self._waiting_for_response and queue_size == 0 and time_since_last_data > 1.0:
print("\n(No new audio received for a while, assuming playback is finished)")
return True
return False
def _play_audio(self):
"""Worker thread for playing audio data"""
while True:
# Check if it should stop
with self._lock:
if not self.is_playing:
break
stream_ref = self.stream # Get a reference to the stream
try:
# Get audio data from the queue, with a timeout of 0.1 seconds
audio_data = self.audio_queue.get(timeout=0.1)
# Check the status and stream validity again
with self._lock:
if self.is_playing and stream_ref and not stream_ref.is_stopped():
try:
# Play the audio data
stream_ref.write(audio_data)
# Update the latest playback information
self._last_play_time = time.time()
self._last_chunk_duration = len(audio_data) / (
self.channels * self.sample_width) / self.sample_rate
except Exception as e:
print(f"Error writing to audio stream: {e}")
break
# Mark this data block as processed
self.audio_queue.task_done()
except queue.Empty:
# Continue waiting if the queue is empty
continue
except Exception as e:
print(f"Error playing audio: {e}")
break
class MicrophoneRecorder:
"""Real-time microphone recorder"""
def __init__(self, sample_rate=16000, channels=1, chunk_size=3200):
self.sample_rate = sample_rate
self.channels = channels
self.chunk_size = chunk_size
self.pyaudio_instance = None
self.stream = None
self.frames = []
self._is_recording = False
self._record_thread = None
def _recording_thread(self):
"""Recording worker thread"""
# Continuously read data from the audio stream while _is_recording is True
while self._is_recording:
try:
# Use exception_on_overflow=False to avoid crashing due to buffer overflow
data = self.stream.read(self.chunk_size, exception_on_overflow=False)
self.frames.append(data)
except (IOError, OSError) as e:
# Reading from the stream might raise an error when it's closed
print(f"Error reading from recording stream, it might be closed: {e}")
break
def start(self):
"""Start recording"""
if self._is_recording:
print("Recording is already in progress.")
return
self.frames = []
self._is_recording = True
try:
self.pyaudio_instance = pyaudio.PyAudio()
self.stream = self.pyaudio_instance.open(
format=pyaudio.paInt16,
channels=self.channels,
rate=self.sample_rate,
input=True,
frames_per_buffer=self.chunk_size
)
self._record_thread = threading.Thread(target=self._recording_thread)
self._record_thread.daemon = True
self._record_thread.start()
print("Microphone recording started...")
except Exception as e:
print(f"Failed to start microphone: {e}")
self._is_recording = False
self._cleanup()
raise
def stop(self):
"""Stop recording and return the audio data"""
if not self._is_recording:
return None
self._is_recording = False
# Wait for the recording thread to exit safely
if self._record_thread:
self._record_thread.join(timeout=1.0)
self._cleanup()
print("Microphone recording stopped.")
return b''.join(self.frames)
def _cleanup(self):
"""Safely clean up PyAudio resources"""
if self.stream:
try:
if self.stream.is_active():
self.stream.stop_stream()
self.stream.close()
except Exception as e:
print(f"Error closing audio stream: {e}")
if self.pyaudio_instance:
try:
self.pyaudio_instance.terminate()
except Exception as e:
print(f"Error terminating PyAudio instance: {e}")
self.stream = None
self.pyaudio_instance = None
async def interactive_test():
"""
Interactive test script: Allows for multi-turn conversations, with audio and images sent in each turn.
"""
# ------------------- 1. Initialization and connection (one-time) -------------------
api_key = os.environ.get("DASHSCOPE_API_KEY")
if not api_key:
print("Please set the DASHSCOPE_API_KEY environment variable.")
return
print("--- Real-time Multimodal Audio/Video Chat Client ---")
print("Initializing audio player and client...")
audio_player = AudioPlayer()
audio_player.start()
def on_audio_received(audio_data):
audio_player.add_audio_data(audio_data)
def on_response_done(event):
print("\n(Received response end marker)")
audio_player.stop_receiving_data()
realtime_client = OmniRealtimeClient(
base_url="wss://maas.qwencloudapi.com/api-ws/v1/realtime",
api_key=api_key,
model="qwen3.8-omni-flash-realtime",
voice="Tina",
instructions="You are Xiaoyun, a personal assistant. Please answer the user's questions accurately and friendly, always responding with a helpful attitude.", # Set the model role
on_text_delta=lambda text: print(f"Assistant reply: {text}", end="", flush=True),
on_audio_delta=on_audio_received,
turn_detection_mode=TurnDetectionMode.MANUAL,
extra_event_handlers={"response.done": on_response_done}
)
message_handler_task = None
try:
await realtime_client.connect()
print("Connected to the server. Enter 'q' or 'quit' to exit at any time.")
message_handler_task = asyncio.create_task(realtime_client.handle_messages())
await asyncio.sleep(0.5)
turn_counter = 1
# ------------------- 2. Multi-turn conversation loop -------------------
while True:
print(f"\n--- Turn {turn_counter} ---")
audio_player.prepare_for_next_turn()
recorded_audio = None
image_paths = []
# --- Get user input: Record from microphone ---
loop = asyncio.get_event_loop()
recorder = MicrophoneRecorder(sample_rate=16000) # 16k sample rate is recommended for speech recognition
print("Ready to record. Press Enter to start recording (or enter 'q' to exit)...")
user_input = await loop.run_in_executor(None, input)
if user_input.strip().lower() in ['q', 'quit']:
print("User requested to exit...")
return
try:
recorder.start()
except Exception:
print("Could not start recording. Please check your microphone permissions and device. Skipping this turn.")
continue
print("Recording... Press Enter again to stop.")
await loop.run_in_executor(None, input)
recorded_audio = recorder.stop()
if not recorded_audio or len(recorded_audio) == 0:
print("No valid audio was recorded. Please start this turn again.")
continue
# --- 3. Send data and get response ---
print("\n--- Input Confirmation ---")
print(f"Audio to process: 1 (from microphone), Images: {len(image_paths)}")
print("------------------")
# 3.1 Send the recorded audio
try:
print(f"Sending microphone recording ({len(recorded_audio)} bytes)")
await realtime_client.stream_audio(recorded_audio)
await asyncio.sleep(0.1)
except Exception as e:
print(f"Failed to send microphone recording: {e}")
continue
# 3.3 Submit and wait for response
print("Submitting all inputs, requesting server response...")
await realtime_client.commit_audio_buffer()
await realtime_client.create_response()
print("Waiting for and playing server response audio...")
start_time = time.time()
max_wait_time = 60
while not audio_player.is_finished_playing():
if time.time() - start_time > max_wait_time:
print(f"\nWait timed out ({max_wait_time} seconds). Moving to the next turn.")
break
await asyncio.sleep(0.2)
print("\nAudio playback for this turn is complete!")
turn_counter += 1
except (asyncio.CancelledError, KeyboardInterrupt):
print("\nProgram was interrupted.")
except Exception as e:
print(f"An unhandled error occurred: {e}")
finally:
# ------------------- 4. Clean up resources -------------------
print("\nClosing connection and cleaning up resources...")
if message_handler_task and not message_handler_task.done():
message_handler_task.cancel()
if 'realtime_client' in locals() and realtime_client.ws and not realtime_client.ws.close:
await realtime_client.close()
print("Connection closed.")
audio_player.stop()
print("Program exited.")
if __name__ == "__main__":
try:
asyncio.run(interactive_test())
except KeyboardInterrupt:
print("\nProgram was forcibly exited by the user.")
```
Corra. `manual_mode.py`. Pressione Enter para começar a falar. Pressione Enter novamente para receber a resposta de áudio do modelo.
Para o fluxo de interação VAD/Manual modo e o Qwen3.8-Omni-Flash-Realtime Fluxo de interação MCP (descoberta de ferramentas, invocação, aprovação e continuação), veja [Processo de interação] (/api-reference/real-time-multimodal/interaction-process).
Áudio multicanal, agregação de vídeo e MCP
A seguinte configuração se aplica a Qwen3.8-Omni-Flash-Realtime. Configure essas opções como necessário antes de enviar o primeiro segmento de áudio.
- ** Áudio multicanal (Acesso WebSocket)**: configure
session.audio.input.formatAntes do primeiro segmento de áudio. Ele suporta 1, 2 ou 4 canais, entrada multicanal requer PCM, 16000 Hz, s16le, e amostras intercaladas. Useraw_mic_arraypara 2 canais efoa_ambixPor 4. Veja [acontecimentos de clientes] (/api-reference/real-time-multimodal/client-events#session-audio-multichannel) por restrições e fragmentos de JSON. - ** Agregação de vídeo**:
session.video.input.representation_compactpredefinições paranone. Usenormalpara agregar representações de vídeo e reduzir a computação quando detalhes visuais são desnecessários. Coloque antes do primeiro segmento de áudio, não pode mudar depois que a entrada de áudio começar. - A voz padrão é
Tina. Usesession.audio.output.voicePara novas integrações, tem precedência sobresession.voice. As vozes disponíveis incluemlonganlingxin(Longa Ling Xin), uma voz calorosa e empática. Veja./developer-guides/speech/omni-voice-list) para o novolonganlingxinclonagem de voz e voz. - MCP: configurar um serviço HTTPS MCP público em HTTP
session.tools. A aprovação é exigida por defeito. Espere a descoberta para completar antes de criar uma resposta que precise da ferramenta. Veja o [fluxo de interação MPC] (/api-reference/real-time-multimodal/interaction-process#qwen38-mcp-flow) para resultados, rejeições, falhas e continuação.
Configure a agregação de vídeo com o SDK
Qwen3.8-Omni-Flash-Realtime requer DashScope Python SDK 1.26.5 ou posterior, ou Java SDK 2.22.15 ou posterior. Passe o parâmetro de agregação de vídeo através do SDK: fornecimento video Para Python's. update_session, ou usar OmniRealtimeConfig.builder().parameters(...) em Java.
Este exemplo configura uma sessão de modo manual com texto e saída de áudio, o Tina voz, transcrição de entrada desabilitada, e agregação de vídeo habilitada. Estabeleça uma conexão usando os exemplos SDK abaixo com o modelo definido para qwen3.8-omni-flash-realtime. Substitua a configuração do exemplo por esta, e chame-a uma vez antes de enviar o primeiro segmento de áudio. conversation é a sessão SDK conectada. O SDK também envia configurações de voz e DAV, configura qualquer voz diferente, DAV, ou configurações de transcrição na mesma chamada. Em Java, importa java.util.Arrays, java.util.Map, java.util.HashMap, com.alibaba.dashscope.audio.omni.OmniRealtimeConfig, e com.alibaba.dashscope.audio.omni.OmniRealtimeModality.
from dashscope.audio.qwen_omni import MultiModality
conversation.update_session(
output_modalities=[MultiModality.AUDIO, MultiModality.TEXT],
voice="Tina",
enable_turn_detection=False,
enable_input_audio_transcription=False,
video={
"input": {
"representation_compact": "normal",
},
},
)
Map<String, Object> videoInput = new HashMap<>();
videoInput.put("representation_compact", "normal");
Map<String, Object> video = new HashMap<>();
video.put("input", videoInput);
Map<String, Object> parameters = new HashMap<>();
parameters.put("video", video);
OmniRealtimeConfig config =
OmniRealtimeConfig.builder()
.modalities(Arrays.asList(OmniRealtimeModality.AUDIO, OmniRealtimeModality.TEXT))
.voice("Tina")
.enableTurnDetection(false)
.enableInputAudioTranscription(false)
.parameters(parameters)
.build();
conversation.updateSession(config);
Pesquisa na Web
A busca na web permite que o modelo responda usando dados recuperados em tempo real para cenários que precisam de informações atualizadas, como preços de ações ou previsões meteorológicas. O modelo decide se deve procurar.
Qwen3.8-Omni-Flash-Realtime e o Qwen3.5-Omni-Realtime Série de busca na web. Está desativado por padrão. Active-o usando o session.update evento.
Para os detalhes de faturamento, veja o agent política no [Billing detalhes] (/developer-guides/getting-started/pricing).
Como permitir
Na session.update Evento, adicione estes parâmetros:
enable_search- Pronto paratruepara ativar a busca na web.search_options.enable_source- Pronto paratruepara devolver uma lista de fontes de resultados de busca.
Para detalhes completos do parâmetro, veja sessão.atualização.
Formato de resposta
Depois de ativar a busca na web, response.done O evento inclui um novo plugins - Não, não, não. usage Protesto. Este campo registra métricas de uso de busca:
{
"usage": {
"total_tokens": 2937,
"input_tokens": 2554,
"output_tokens": 383,
"input_tokens_details": {
"text_tokens": 2512,
"audio_tokens": 42
},
"output_tokens_details": {
"text_tokens": 90,
"audio_tokens": 293
},
"plugins": {
"search": {
"count": 1,
"strategy": "agent"
}
}
}
}
Exemplos de código
Os exemplos a seguir mostram como habilitar a busca na web.
DashScope Python SDK
Na `update_session` - Não, não, não. `enable_search` e `search_options` parâmetros:
```python
import os
import base64
import time
import json
import pyaudio
from dashscope.audio.qwen_omni import MultiModality, AudioFormat, OmniRealtimeCallback, OmniRealtimeConversation
import dashscope
dashscope.api_key = os.getenv('DASHSCOPE_API_KEY')
url = 'wss://maas.qwencloudapi.com/api-ws/v1/realtime'
model = 'qwen3.8-omni-flash-realtime'
voice = 'Tina'
class SearchCallback(OmniRealtimeCallback):
def __init__(self, pya):
self.pya = pya
self.out = None
def on_open(self):
self.out = self.pya.open(format=pyaudio.paInt16, channels=1, rate=24000, output=True)
def on_event(self, response):
if response['type'] == 'response.audio.delta':
self.out.write(base64.b64decode(response['delta']))
elif response['type'] == 'conversation.item.input_audio_transcription.delta':
# Streaming preview: text is the confirmed prefix, stash is the unconfirmed suffix.
preview = response.get('text', '') + response.get('stash', '')
print(f"\r[User] {preview}", end='', flush=True)
elif response['type'] == 'conversation.item.input_audio_transcription.completed':
print(f"\r[User] {response['transcript']}")
elif response['type'] == 'response.audio_transcript.done':
print(f"[LLM] {response['transcript']}")
elif response['type'] == 'response.done':
usage = response.get('response', {}).get('usage', {})
plugins = usage.get('plugins', {})
if plugins.get('search'):
print(f"[Search] count={plugins['search']['count']}, strategy={plugins['search']['strategy']}")
pya = pyaudio.PyAudio()
callback = SearchCallback(pya)
conv = OmniRealtimeConversation(model=model, callback=callback, url=url)
conv.connect()
conv.update_session(
output_modalities=[MultiModality.AUDIO, MultiModality.TEXT],
voice=voice,
instructions="You are Xiao Yun, a personal assistant",
enable_search=True,
search_options={'enable_source': True}
)
mic = pya.open(format=pyaudio.paInt16, channels=1, rate=16000, input=True)
print("Web search is enabled. Speak into the microphone (press Ctrl+C to exit)...")
try:
while True:
audio_data = mic.read(3200, exception_on_overflow=False)
conv.append_audio(base64.b64encode(audio_data).decode())
time.sleep(0.01)
except KeyboardInterrupt:
conv.close()
mic.close()
callback.out.close()
pya.terminate()
print("\nConversation ended")
```
DashScope Java SDK
Em `updateSession`, passe as configurações de busca na web através do `parameters` - Sim.
```java
import com.alibaba.dashscope.audio.omni.*;
import com.alibaba.dashscope.exception.NoApiKeyException;
import com.google.gson.JsonObject;
import javax.sound.sampled.*;
import java.nio.ByteBuffer;
import java.util.*;
import java.util.concurrent.ConcurrentLinkedQueue;
import java.util.concurrent.atomic.AtomicBoolean;
public class OmniSearch {
static class SequentialAudioPlayer {
private final SourceDataLine line;
private final Queue<byte[]> audioQueue = new ConcurrentLinkedQueue<>();
private final Thread playerThread;
private final AtomicBoolean shouldStop = new AtomicBoolean(false);
public SequentialAudioPlayer() throws LineUnavailableException {
AudioFormat format = new AudioFormat(24000, 16, 1, true, false);
line = AudioSystem.getSourceDataLine(format);
line.open(format);
line.start();
playerThread = new Thread(() -> {
while (!shouldStop.get()) {
byte[] audio = audioQueue.poll();
if (audio != null) {
line.write(audio, 0, audio.length);
} else {
try { Thread.sleep(10); } catch (InterruptedException ignored) {}
}
}
}, "AudioPlayer");
playerThread.start();
}
public void play(String base64Audio) {
audioQueue.add(Base64.getDecoder().decode(base64Audio));
}
public void close() {
shouldStop.set(true);
try { playerThread.join(1000); } catch (InterruptedException ignored) {}
line.drain();
line.close();
}
}
public static void main(String[] args) {
try {
SequentialAudioPlayer player = new SequentialAudioPlayer();
AtomicBoolean shouldStop = new AtomicBoolean(false);
OmniRealtimeParam param = OmniRealtimeParam.builder()
.model("qwen3.8-omni-flash-realtime")
.apikey(System.getenv("DASHSCOPE_API_KEY"))
.url("wss://maas.qwencloudapi.com/api-ws/v1/realtime")
.build();
OmniRealtimeConversation conversation = new OmniRealtimeConversation(param, new OmniRealtimeCallback() {
@Override public void onOpen() {
System.out.println("Connection established");
}
@Override public void onClose(int code, String reason) {
System.out.println("Connection closed");
shouldStop.set(true);
}
@Override public void onEvent(JsonObject event) {
String type = event.get("type").getAsString();
if ("response.audio.delta".equals(type)) {
player.play(event.get("delta").getAsString());
} else if ("response.audio_transcript.done".equals(type)) {
System.out.println("[LLM] " + event.get("transcript").getAsString());
} else if ("response.done".equals(type)) {
JsonObject response = event.getAsJsonObject("response");
if (response != null && response.has("usage")) {
JsonObject usage = response.getAsJsonObject("usage");
if (usage.has("plugins")) {
JsonObject plugins = usage.getAsJsonObject("plugins");
if (plugins.has("search")) {
JsonObject search = plugins.getAsJsonObject("search");
System.out.println("[Search] count=" + search.get("count").getAsInt()
+ ", strategy=" + search.get("strategy").getAsString());
}
}
}
}
}
});
conversation.connect();
conversation.updateSession(OmniRealtimeConfig.builder()
.modalities(Arrays.asList(OmniRealtimeModality.AUDIO, OmniRealtimeModality.TEXT))
.voice("Tina")
.enableTurnDetection(true)
.enableInputAudioTranscription(true)
.parameters(Map.of(
"instructions", "You are Xiao Yun, a personal assistant",
"enable_search", true,
"search_options", Map.of("enable_source", true)
))
.build()
);
System.out.println("Web search is enabled. Start speaking (press Ctrl+C to exit)...");
AudioFormat format = new AudioFormat(16000, 16, 1, true, false);
TargetDataLine mic = AudioSystem.getTargetDataLine(format);
mic.open(format);
mic.start();
ByteBuffer buffer = ByteBuffer.allocate(3200);
while (!shouldStop.get()) {
int bytesRead = mic.read(buffer.array(), 0, buffer.capacity());
if (bytesRead > 0) {
conversation.appendAudio(Base64.getEncoder().encodeToString(buffer.array()));
}
Thread.sleep(20);
}
conversation.close(1000, "Normal end");
player.close();
mic.close();
} catch (NoApiKeyException e) {
System.err.println("API key not found: Set the DASHSCOPE_API_KEY environment variable");
} catch (Exception e) {
e.printStackTrace();
}
}
}
```
WebSocket (Python)
Na carga JSON para `session.update`, adicione o `enable_search` e `search_options` Campos:
```python
import json
import os
import websocket
import base64
import pyaudio
import threading
API_KEY = os.getenv("DASHSCOPE_API_KEY")
API_URL = "wss://maas.qwencloudapi.com/api-ws/v1/realtime?model=qwen3.8-omni-flash-realtime"
pya = pyaudio.PyAudio()
out_stream = pya.open(format=pyaudio.paInt16, channels=1, rate=24000, output=True)
def on_open(ws):
ws.send(json.dumps({
"type": "session.update",
"session": {
"modalities": ["text", "audio"],
"voice": "Tina",
"instructions": "You are Xiao Yun, a personal assistant",
"input_audio_format": "pcm",
"output_audio_format": "pcm",
"enable_search": True,
"search_options": {
"enable_source": True
}
}
}))
print("Web search is enabled. Speak into the microphone...")
def send_audio():
mic = pya.open(format=pyaudio.paInt16, channels=1, rate=16000, input=True)
try:
while True:
audio = mic.read(3200, exception_on_overflow=False)
ws.send(json.dumps({
"type": "input_audio_buffer.append",
"audio": base64.b64encode(audio).decode()
}))
except Exception:
mic.close()
threading.Thread(target=send_audio, daemon=True).start()
def on_message(ws, message):
event = json.loads(message)
if event["type"] == "response.audio.delta":
out_stream.write(base64.b64decode(event["delta"]))
elif event["type"] == "response.audio_transcript.done":
print(f"[LLM] {event['transcript']}")
elif event["type"] == "response.done":
usage = event.get("response", {}).get("usage", {})
plugins = usage.get("plugins", {})
if plugins.get("search"):
print(f"[Search] count={plugins['search']['count']}, strategy={plugins['search']['strategy']}")
def on_error(ws, error):
print(f"Error: {error}")
headers = ["Authorization: Bearer " + API_KEY]
ws = websocket.WebSocketApp(API_URL, header=headers, on_open=on_open, on_message=on_message, on_error=on_error)
ws.run_forever()
```
Referência da API
- [Acontecimentos do cliente] (/api-reference/real-time-multimodal/client-events)
- [Eventos de servidor] (/api-reference/real-time-multimodal/server-events)
Limites de faturamento e taxa
Regras de faturamento
Qwen-Omni-Realtime é faturado com base no número de fichas usadas para diferentes modalidades de entrada, como áudio e imagens. Para mais informações sobre faturamento, veja [Pregando] (/developer-guides/getting-started/pricing).
Para a saída da fala, qwen3.8-omni-flash-realtime Conta tanto o áudio quanto o texto correspondente em suas respectivas taxas de saída. Qwen3.5-Omni-Realtime Modelos cobram apenas o áudio, o texto correspondente não é carregado.
As invocações de ferramentas MCP não implicam em taxas adicionais, a inferência de modelos permanece sujeita a preços de modelos. Veja o [fluxo de interação MPC] (/api-reference/real-time-multimodal/interaction-process#qwen38-mcp-flow) para o procedimento de chamada e limites.
Em uma conversa em tempo real, cada vez que o modelo gera uma resposta, ele processa todo o conteúdo histórico de conversa dentro da janela de contexto, incluindo áudio, imagens e texto de turnos anteriores, junto com a nova entrada da virada atual como tokens de entrada. Como resultado, os tokens de entrada se acumulam a cada volta, em vez de serem contados apenas para a nova entrada na volta atual.
Por exemplo, se uma entrada de áudio de 10 segundos converter para 70 tokens (Qwen3.5-Omni-Realtime), e esse áudio ainda está dentro da janela de contexto na curva 3, ele ainda será contado para as fichas de entrada para a volta 3. Os símbolos de entrada faturados = símbolos de todas as voltas históricas dentro da janela de contexto + símbolos da nova entrada na volta atual.
Regras para converter áudio, imagens e vídeo em fichas
Audio
- Qwen3.8-Omni-Flash-Realtime e Qwen3.5-Omni-RealtimeInput audio tokens totais = duração do áudio (em segundos) x 7; Saída de áudio tokens totais = Duração de áudio (em segundos) x 12,5
Se a duração do áudio é menor que 1 segundo, é calculado como 1 segundo.
Por `qwen3.8-omni-flash-realtime`, entrada de áudio espacial usa o dobro de tokens de áudio de entrada como áudio regular. O multiplicador é o mesmo para entrada de 2 canais e 4 canais.
Image
<a id="realtime-image-tokens" />
- `qwen3.8-omni-flash-realtime` e Qwen3.5-Omni-Realtime Modelos usam 1 token por `32x32` pixels
Uma imagem requer um mínimo de 4 fichas e suporta um máximo de 1.280 fichas. Use o seguinte código para estimar o número total de fichas consumidas por uma imagem:
```python
# Install the Pillow library using the following command: pip install Pillow
from PIL import Image
import math
# For Qwen3.8-Omni-Flash-Realtime and Qwen3.5-Omni-Realtime models, the zoom factor is 32.
factor = 32
def token_calculate(image_path='', duration=10):
"""
:param image_path: The path of the image.
:param duration: The duration of the session connection.
:return: The number of tokens for the image.
"""
if len(image_path) > 0:
# Open the specified PNG image file.
image = Image.open(image_path)
# Get the original dimensions of the image.
height = image.height
width = image.width
print(f"Image dimensions before scaling: height={height}, width={width}")
# Adjust the height to be an integer multiple of the factor.
h_bar = round(height / factor) * factor
# Adjust the width to be an integer multiple of the factor.
w_bar = round(width / factor) * factor
# Lower limit for image tokens: 4 tokens.
min_pixels = factor * factor * 4
# Upper limit for image tokens: 1,280 tokens.
max_pixels = 1280 * factor * factor
# Scale the image to ensure the total number of pixels is within the range [min_pixels, max_pixels].
if h_bar * w_bar > max_pixels:
# Calculate the scaling factor beta so that the total number of pixels of the scaled image does not exceed max_pixels.
beta = math.sqrt((height * width) / max_pixels)
# Recalculate the adjusted height to ensure it is an integer multiple of the factor.
h_bar = math.floor(height / beta / factor) * factor
# Recalculate the adjusted width to ensure it is an integer multiple of the factor.
w_bar = math.floor(width / beta / factor) * factor
elif h_bar * w_bar < min_pixels:
# Calculate the scaling factor beta so that the total number of pixels of the scaled image is not less than min_pixels.
beta = math.sqrt(min_pixels / (height * width))
# Recalculate the adjusted height to ensure it is an integer multiple of the factor.
h_bar = math.ceil(height * beta / factor) * factor
# Recalculate the adjusted width to ensure it is an integer multiple of the factor.
w_bar = math.ceil(width * beta / factor) * factor
print(f"Image dimensions after scaling: height={h_bar}, width={w_bar}")
# Calculate the number of tokens for the image: total pixels divided by (factor x factor).
token = int((h_bar * w_bar) / (factor * factor))
print(f"Number of tokens after scaling: {token}")
total_token = token * math.ceil(duration / 2)
print(f"Total number of tokens: {total_token}")
return total_token
else:
print("Error: image_path is empty. Cannot calculate tokens")
return 0
if __name__ == "__main__":
total_token = token_calculate(image_path="xxx/test.jpg", duration=10)
```
Video
`qwen3.8-omni-flash-realtime` usa as mesmas regras de vídeo-token base como Qwen3.5-Omni-Realtime. Para a conversão pixel-to-token de quadros de vídeo, veja [Regras de símbolo de imagem] (#realtime-image-tokens).
Por `qwen3.8-omni-flash-realtime`, set `representation_compact="normal"` reduz as fichas de vídeo para um quarto da contagem sem agregação.
Limites de taxa
Para mais informações sobre as regras de limite de taxa de modelo, veja [limites de taxa] (/developer-guides/administration/rate-limits).
Códigos de erro.
Se uma ligação falhar, veja [Códigos de erro] (/api-reference/preparation/error-messages).
Lista de voz
Para as vozes, cada modelo omnimodal suporta, incluindo amostras e voice valores dos parâmetros, veja [lista de voz Omni-modal] (/developer-guides/speech/omni-voice-list).