Lip-Sync vs Amplitude for a Talking Avatar

Do I need real lip-sync or is amplitude enough?

It depends on the face. Lip-sync means matching the mouth to the sounds being spoken. That requires two things: timing data that says which sound happens when, and a character drawn with a distinct mouth pose for each group of sounds (a viseme). Amplitude-driven talking needs neither. It only asks whether the voice is currently audible and plays a talking loop while it is.

Sources: Microsoft Learn, Amazon Polly docs.

Small, stylized characters, such as a 64 px badge beside a chat or a mascot with a simple mouth, usually read as "speaking" from rhythm alone. Visemes start to pay off on larger, more human faces where a mismatched mouth is noticeable.

Be honest about what your character supports. Charmotion's sets, for example, include a talking loop (mood 7) but no mouth-shape or viseme inputs, so they pair with amplitude gating, not phoneme lip-sync. That is a deliberate fit for chat and voice agents where the words also appear as text.

How do I measure speech level with an AnalyserNode?

The Web Audio API's AnalyserNode exposes real-time time-domain and frequency data without changing the audio passing through it (MDN). For a talking gate, the time-domain waveform is the simplest input. getByteTimeDomainData() fills a Uint8Array with samples from 0 to 255, where 128 is silence (MDN). Convert to −1…1 and take the root mean square:

const ctx = new AudioContext();
const analyser = ctx.createAnalyser();
analyser.fftSize = 1024;

// Pick the source that carries the assistant's voice:
const source = ctx.createMediaElementSource(audioElement);        // <audio> playback
// const source = ctx.createMediaStreamSource(remoteStream);      // WebRTC track
source.connect(analyser);
analyser.connect(ctx.destination); // only for media element sources, so it stays audible

const samples = new Uint8Array(analyser.fftSize);

function rms(): number {
  analyser.getByteTimeDomainData(samples);
  let sum = 0;
  for (let i = 0; i < samples.length; i++) {
    const v = (samples[i] - 128) / 128;
    sum += v * v;
  }
  return Math.sqrt(sum / samples.length);
}

A media element source reroutes the element's audio through the graph, so connect the analyser onward to the destination or the voice goes silent. A stream source from a WebRTC track only taps the audio; let the <audio> element that plays the track handle output.

Create or resume the AudioContext from a user gesture, such as the button that starts the conversation, or browsers will keep it suspended (MDN autoplay guide).

getByteFrequencyData() works too if you want to weight speech frequencies: it returns frequencyBinCount values from 0 to 255, spread linearly from 0 Hz to half the sample rate (MDN). For an on/off gate, RMS is simpler and plenty.

How do I turn the level into talking vs idle without flicker?

Speech is full of short gaps between syllables and words. A single threshold flips the avatar between talking and idle several times a second, which looks broken. Use hysteresis: a higher threshold to start talking, a lower one to stop, and a hold time before you actually stop.

const ON = 0.04;      // start talking above this RMS
const OFF = 0.02;     // consider silence below this
const HOLD_MS = 300;  // silence must last this long before stopping

let talking = false;
let quietSince = 0;
let level = 0;

function tick(now: number) {
  level = level * 0.8 + rms() * 0.2; // light smoothing

  if (!talking && level > ON) {
    talking = true;
    setMood(MOOD.talking);
  } else if (talking) {
    if (level > OFF) quietSince = 0;
    else if (!quietSince) quietSince = now;
    else if (now - quietSince > HOLD_MS) {
      talking = false;
      quietSince = 0;
      setMood(MOOD.listening); // or MOOD.idle
    }
  }
  requestAnimationFrame(tick);
}
requestAnimationFrame(tick);

The thresholds and hold time are starting values to tune against your actual voice and volume, not standards. Log level for a few sentences and set ON just above the noise floor.

Only call setMood when the state changes, as above. Writing the same value every frame is wasted work. With a Rive view model, setMood can be as small as vmi.number("mood").value = m.

If your character does expose a continuous mouth-open number, you can feed it level scaled to its range on every frame instead. That is a property of the specific file; many mascot sets use a talking loop instead.

What are visemes and which TTS providers send them?

A viseme is the visual counterpart of a phoneme: the mouth position for a group of sounds that look alike. "P", "b" and "m" all close the lips, for example, so they share a viseme. Text-to-speech engines that know which phonemes they are producing can report visemes with timestamps.

Azure AI Speech offers 22 visemes, numbered from 0 (silence) to 21 (p, b, m). The Speech SDK raises a VisemeReceived event for each one, with an audio offset in ticks of 100 nanoseconds. Azure can also return blend-shape frames at 60 fps, each with 55 facial positions, for 3D faces, and SVG mouth images, which are supported only for the en-US locale (Microsoft Learn).

Amazon Polly returns speech marks: metadata describing the speech, requested instead of audio. They come as a line-delimited JSON stream where each object has a time in milliseconds from the start of the audio, a type (sentence, word, viseme or ssml) and a value, which for visemes is the viseme name (Amazon Polly). Polly documents speech marks for its neural, long-form and standard engines (Amazon Polly).

Realtime speech-to-speech models, such as the OpenAI Realtime API, stream audio without per-phoneme timing in the events covered in our Realtime guide, which is one reason amplitude gating is the usual choice for voice agents.

How do Azure viseme IDs and Polly speech marks differ?

The main differences are delivery and units. Azure pushes viseme events during synthesis; Polly gives you a separate metadata response you fetch alongside the audio.

With the Azure Speech SDK for JavaScript, subscribe to visemeReceived before you synthesize and convert the offset to milliseconds:

import * as sdk from "microsoft-cognitiveservices-speech-sdk";

const synthesizer = new sdk.SpeechSynthesizer(speechConfig, audioConfig);
const visemes: { id: number; ms: number }[] = [];

synthesizer.visemeReceived = (_sender, e) => {
  visemes.push({ id: e.visemeId, ms: e.audioOffset / 10000 }); // 100-ns ticks -> ms
};

synthesizer.speakTextAsync(text, () => synthesizer.close(), (err) => {
  console.error(err);
  synthesizer.close();
});

With Polly, make one request with OutputFormat: "json" and SpeechMarkTypes: ["viseme"], and a second request for the audio itself with the same text and voice. Parse the marks by splitting on newlines and calling JSON.parse on each line.

The ID sets differ too. Azure uses numbers 0–21; Polly uses short viseme names. Each needs its own lookup table to your character's mouth poses, and both assume those poses exist. A character with only a talking loop gains nothing from either.

How do I sync viseme timestamps to audio playback?

Viseme times are relative to the start of the audio, so schedule them against the playback clock, not against when the data arrived. Reading audio.currentTime on every animation frame and applying every mark that is due keeps the mouth in step even if playback stalls or the tab drops frames:

type Mark = { time: number; value: string }; // Polly: ms from audio start

function playWithVisemes(audio: HTMLAudioElement, marks: Mark[], show: (v: string) => void) {
  let i = 0;
  const tick = () => {
    const now = audio.currentTime * 1000;
    while (i < marks.length && marks[i].time <= now) show(marks[i++].value);
    if (!audio.paused && !audio.ended) requestAnimationFrame(tick);
    else show("sil");
  };
  audio.addEventListener("play", () => requestAnimationFrame(tick), { once: true });
  audio.play();
}

The same loop works for Azure if you store ms instead of time. If the user seeks or you restart playback, reset i by searching for the first mark after the new position.

Smooth the result. Jumping straight between mouth poses on every mark looks mechanical; a short blend between poses in the animation file, or skipping marks shorter than a frame, reads far better. The same idea applies to amplitude: the smoothing factor and hold time in the gate above exist to keep motion calm.

Frequently asked questions

Is amplitude enough to animate a talking avatar?

For stylized 2D mascots, usually yes. Gating a talking loop on the audio level from an AnalyserNode reads as speech without phoneme data and works with any voice source, including realtime agents. True lip-sync needs viseme timing from the TTS provider plus a mouth shape for each viseme.

Should I use getByteTimeDomainData or getByteFrequencyData?

For an on/off talking gate, getByteTimeDomainData is simplest: convert samples around 128 to −1…1 and compute RMS. getByteFrequencyData is useful when you want to weight speech frequencies or draw a visualizer. Either way, smooth the value and use hysteresis so the avatar doesn't flicker.

How many visemes does Azure Speech provide?

Azure Speech provides 22 viseme IDs, from 0 for silence to 21 for p, b and m. The JavaScript SDK delivers them through visemeReceived with audioOffset in 100-nanosecond ticks, so divide by 10,000 for milliseconds. Azure can also return 55-value blend-shape frames at 60 fps.

Can I lip-sync a character that only has a talking loop?

Not meaningfully. Viseme data tells you which mouth shape to show, so it needs a character drawn with those shapes. With only a talking loop, the best result comes from amplitude gating: play the loop while the voice is audible and return to listening or idle in pauses.

Quick answers from this guide

• Is amplitude enough to animate a talking avatar?
• How many visemes does Azure Speech provide?
• How do Amazon Polly viseme speech marks work?

Sources

• AnalyserNode — MDN Web Docs
• AnalyserNode: getByteTimeDomainData() method — MDN Web Docs
• AnalyserNode: getByteFrequencyData() method — MDN Web Docs
• Autoplay guide for media and Web Audio APIs — MDN Web Docs
• Get facial position with viseme — Microsoft Learn
• Speech mark output — Amazon Polly Developer Guide
• Using speech marks — Amazon Polly Developer Guide