Skip to content

Visemes, coarticulation & curves

The viseme model (the Oculus/Meta 15-viseme set and the phoneme→viseme map), the Cohen–Massaro dominance-blending coarticulation solver, the FaceTrack data model and keyframe reduction, data-driven weighted mapping, and IPA support.

openfacefx.visemes

Viseme inventory and phoneme -> viseme mapping.

A viseme is the visual mouth shape corresponding to one or more phonemes. Many phonemes are visually indistinguishable (e.g. /p/, /b/, /m/ are all a lip closure), so the mapping is many-to-one.

We ship the 15-target set popularised by the Oculus/Meta LipSync SDK because it is a widely adopted, well-documented, IP-free convention that most character rigs already provide blendshapes for. Each viseme name below is a blendshape your rig is expected to expose.

VISEMES = ['sil', 'PP', 'FF', 'TH', 'DD', 'kk', 'CH', 'SS', 'nn', 'RR', 'aa', 'E', 'I', 'O', 'U'] module-attribute

VISEME_INDEX = {name: i for i, name in (enumerate(VISEMES))} module-attribute

PHONEME_TO_VISEME = {'P': 'PP', 'B': 'PP', 'M': 'PP', 'F': 'FF', 'V': 'FF', 'TH': 'TH', 'DH': 'TH', 'T': 'DD', 'D': 'DD', 'L': 'DD', 'N': 'nn', 'NG': 'nn', 'K': 'kk', 'G': 'kk', 'HH': 'kk', 'CH': 'CH', 'JH': 'CH', 'SH': 'CH', 'ZH': 'CH', 'S': 'SS', 'Z': 'SS', 'R': 'RR', 'ER': 'RR', 'W': 'U', 'Y': 'I', 'AA': 'aa', 'AE': 'aa', 'AH': 'aa', 'AY': 'aa', 'EH': 'E', 'EY': 'E', 'IH': 'E', 'IY': 'I', 'AO': 'O', 'OW': 'O', 'OY': 'O', 'AW': 'O', 'UW': 'U', 'UH': 'U', SILENCE: 'sil'} module-attribute

phoneme_to_viseme(phoneme: str) -> str

Map a (possibly stressed) ARPAbet phoneme to a viseme name.

Source code in src/openfacefx/visemes.py 
70
71
72
73
def phoneme_to_viseme(phoneme: str) -> str:
    """Map a (possibly stressed) ARPAbet phoneme to a viseme name."""
    p = strip_stress(phoneme).upper() if phoneme != SILENCE else SILENCE
    return PHONEME_TO_VISEME.get(p, "sil")

viseme_index(name: str) -> int

Source code in src/openfacefx/visemes.py 
76
77
def viseme_index(name: str) -> int:
    return VISEME_INDEX[name]

openfacefx.coarticulation

Coarticulation via dominance functions (Cohen & Massaro, 1993).

Real speech is not a sequence of discrete mouth poses -- each phoneme's shape is pulled toward its neighbours. A common, well-cited way to model this is to give every phoneme segment a dominance function: a bump in time, peaked at the segment centre, that decays outward. The activation of a viseme channel at any instant is the dominance-weighted average of the targets of all nearby segments.

F_v(t) = sum_i D_i(t) * target(i, v)  /  sum_i D_i(t)

where D_i(t) = alpha_i * exp( -theta_i * |t - c_i| ) (a Laplacian bump), c_i is the segment centre, and target(i, v) is 1 if segment i maps to viseme v.

The result is smooth, overlapping viseme curves rather than hard switches.

CoartParams(lead: Dict[str, Tuple[float, float]] = (lambda: {'basic': (0.4, 0.45), 'jaw': (0.4, 0.45), 'lips': (0.3, 0.3), 'tongue': (0.15, 0.15)})(), short_silence: float = 0.27, closure_floor: float = 0.9, split_diphthongs: bool = True, preroll: float = 0.0, allow_negative_time: bool = False, intensity: float = 1.0, gains: Dict[str, float] = (lambda: {'basic': 1.0, 'jaw': 1.0, 'lips': 1.0, 'tongue': 1.0})()) dataclass

Component-model tunables (FaceFX-style ca_* knobs).

lead gives per-articulator-class (lead_in, lead_out) extents in seconds — how far a segment's influence reaches before/after its centre. The "basic"/"jaw" defaults reproduce the classic symmetric model; lips and especially tongue targets are tighter, so a quick stop does not smear across neighbouring vowels.

intensity (master) and gains (per-articulator-class) are JALI-style artistic dials: after the curves are normalized, every channel's opening is scaled by intensity * gains[class] and the freed weight flows into sil (see _apply_intensity). All 1.0 is a byte-identical no-op; <1 mumbles / softens a class, >1 hyper-articulates, 0 mutes it. Enforced lip closures still win afterwards, so a whispered bilabial seals.

lead: Dict[str, Tuple[float, float]] = field(default_factory=(lambda: {'basic': (0.4, 0.45), 'jaw': (0.4, 0.45), 'lips': (0.3, 0.3), 'tongue': (0.15, 0.15)})) class-attribute instance-attribute

short_silence: float = 0.27 class-attribute instance-attribute

closure_floor: float = 0.9 class-attribute instance-attribute

split_diphthongs: bool = True class-attribute instance-attribute

preroll: float = 0.0 class-attribute instance-attribute

allow_negative_time: bool = False class-attribute instance-attribute

intensity: float = 1.0 class-attribute instance-attribute

gains: Dict[str, float] = field(default_factory=(lambda: {'basic': 1.0, 'jaw': 1.0, 'lips': 1.0, 'tongue': 1.0})) class-attribute instance-attribute

build_viseme_curves(segments: List[PhonemeSegment], fps: float = 60.0, mapping: Optional[Mapping] = None, params: Optional[CoartParams] = None) -> tuple

Return (times, matrix) where matrix[frame, target] in [0,1].

times is a 1-D array of sample times. Without mapping, columns follow visemes.VISEMES; with a Mapping they follow mapping.target_names and any phoneme may drive several targets with fractional weights. params tunes the component coarticulation model (per-articulator lead in/out, silence absorption, closure enforcement, diphthong splitting, onset pre-roll).

Source code in src/openfacefx/coarticulation.py 
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
def build_viseme_curves(
    segments: List[PhonemeSegment],
    fps: float = 60.0,
    mapping: Optional[Mapping] = None,
    params: Optional[CoartParams] = None,
) -> tuple:
    """Return (times, matrix) where matrix[frame, target] in [0,1].

    ``times`` is a 1-D array of sample times. Without ``mapping``, columns
    follow ``visemes.VISEMES``; with a ``Mapping`` they follow
    ``mapping.target_names`` and any phoneme may drive several targets with
    fractional weights. ``params`` tunes the component coarticulation model
    (per-articulator lead in/out, silence absorption, closure enforcement,
    diphthong splitting, onset pre-roll).
    """
    params = params or CoartParams()
    n_targets = len(mapping.targets) if mapping is not None else len(VISEMES)
    if not segments:
        return np.zeros(0), np.zeros((0, n_targets))
    segments = _preprocess(segments, params)

    t0 = segments[0].start - params.preroll
    if not params.allow_negative_time:
        t0 = max(t0, 0.0) if segments[0].start >= 0.0 else segments[0].start
    t1 = segments[-1].end
    n = max(int(round((t1 - t0) * fps)) + 1, 1)
    times = t0 + np.arange(n) / fps

    centres = np.array([(s.start + s.end) / 2 for s in segments])
    alphas = np.array([_alpha(s) for s in segments])
    thetas = np.array([_theta(s) for s in segments])

    # Per-class asymmetric influence: scale the decay rate on each side by
    # (basic_lead / class_lead), so "basic"/"jaw" reproduce the classic
    # symmetric model and tighter articulators rise/decay faster.
    base_in, base_out = 0.40, 0.45
    lead = [params.lead.get(_segment_class(s, mapping), (base_in, base_out))
            for s in segments]
    scale_in = np.array([base_in / max(li, 1e-3) for li, _ in lead])
    scale_out = np.array([base_out / max(lo, 1e-3) for _, lo in lead])

    # Per-segment target weights: shape (n_seg, n_targets). The built-in
    # table is one-hot, so the weighted path reproduces it bit-for-bit.
    weights = np.zeros((len(segments), n_targets))
    if mapping is not None:
        for i, s in enumerate(segments):
            for idx, w in mapping.row(s.phoneme).items():
                weights[i, idx] = w
    else:
        idx = [VISEME_INDEX[phoneme_to_viseme(s.phoneme)] for s in segments]
        weights[np.arange(len(segments)), idx] = 1.0

    # Dominance of every segment at every sample time: shape (n, n_seg)
    dt = np.abs(times[:, None] - centres[None, :])
    before = times[:, None] < centres[None, :]
    theta_eff = np.where(before,
                         (thetas * scale_in)[None, :],
                         (thetas * scale_out)[None, :])
    dom = alphas[None, :] * np.exp(-theta_eff * dt)

    denom = dom.sum(axis=1, keepdims=True)
    denom[denom == 0] = 1.0

    matrix = np.zeros((n, n_targets))
    for v in range(n_targets):
        matrix[:, v] = (dom * weights[None, :, v]).sum(axis=1) / denom[:, 0]

    # Clean numerical dust and clamp, apply the artistic intensity/gain dials
    # (a no-op at defaults), then enforce closures so enforced frames end up
    # summing to exactly 1 (closures win over the dials).
    matrix[matrix < 1e-4] = 0.0
    np.clip(matrix, 0.0, 1.0, out=matrix)
    _apply_intensity(matrix, mapping, params)
    _enforce_closures(times, matrix, segments, mapping, weights, params)
    return times, matrix

openfacefx.curves

Animation curves: keyframe reduction and track containers.

The dominance model produces one dense sample per frame. Rigs and engines prefer sparse keyframes, so we thin each channel with the Ramer-Douglas-Peucker algorithm: drop samples that lie within epsilon of the straight line between their neighbours. This is lossy but perceptually safe and shrinks output a lot.

Keyframe(time: float, value: float) dataclass

time: float instance-attribute

value: float instance-attribute

Channel(name: str, keys: List[Keyframe] = list()) dataclass

name: str instance-attribute

keys: List[Keyframe] = field(default_factory=list) class-attribute instance-attribute

FaceTrack(fps: float, channels: List[Channel], target_set: List[str] = None, events: 'List[Event]' = list(), variants: 'Optional[Variants]' = None) dataclass

fps: float instance-attribute

channels: List[Channel] instance-attribute

target_set: List[str] = None class-attribute instance-attribute

events: 'List[Event]' = field(default_factory=list) class-attribute instance-attribute

variants: 'Optional[Variants]' = None class-attribute instance-attribute

duration: float property

reduce_to_track(times: np.ndarray, matrix: np.ndarray, fps: float, epsilon: float = 0.015, targets=None) -> FaceTrack

targets: optional list of mapping.Target — supplies channel names and per-target min/max clamps. Defaults to the Oculus viseme set with no clamping (identical to previous releases).

Source code in src/openfacefx/curves.py 
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
def reduce_to_track(times: np.ndarray, matrix: np.ndarray, fps: float,
                    epsilon: float = 0.015, targets=None) -> FaceTrack:
    """``targets``: optional list of ``mapping.Target`` — supplies channel
    names and per-target min/max clamps. Defaults to the Oculus viseme set
    with no clamping (identical to previous releases)."""
    if targets is None:
        names, clamps = VISEMES, [None] * len(VISEMES)
    else:
        names = [t.name for t in targets]
        clamps = [(t.lo, t.hi) if (t.lo, t.hi) != (0.0, 1.0) else None
                  for t in targets]
    channels: List[Channel] = []
    for v, name in enumerate(names):
        col = matrix[:, v]
        if clamps[v] is not None:
            col = np.clip(col, clamps[v][0], clamps[v][1])
        if not np.any(col > 1e-3):
            continue  # channel never fires; skip entirely
        idx = _rdp(times, col, epsilon)
        keys = [Keyframe(float(times[i]), round(float(col[i]), 4)) for i in idx]
        channels.append(Channel(name, keys))
    return FaceTrack(fps=fps, channels=channels,
                     target_set=None if targets is None else list(names))

openfacefx.mapping

Data-driven phoneme -> target mapping (FaceFX-style "mapping spreadsheet").

The built-in behavior maps each phoneme to exactly one Oculus-15 viseme at weight 1.0 (visemes.PHONEME_TO_VISEME). A Mapping generalizes that: any phoneme may drive any set of named targets with fractional weights, each target may declare an articulator class (used by the coarticulation model) and min/max clamps applied before keyframe reduction.

JSON file format (validated on load)::

{
  "format": "openfacefx.mapping",
  "version": 1,
  "targets": [
    {"name": "PP", "class": "lips", "min": 0.0, "max": 1.0},
    ...
  ],
  "phonemes": { "P": {"PP": 1.0}, "AY": {"aa": 0.7, "E": 0.3}, ... }
}

Mapping.default() reproduces the built-in table exactly — running without --mapping is bit-for-bit identical to previous releases.

ARTICULATOR_CLASSES = ('basic', 'jaw', 'lips', 'tongue') module-attribute

Target(name: str, articulator: str = 'basic', lo: float = 0.0, hi: float = 1.0) dataclass

name: str instance-attribute

articulator: str = 'basic' class-attribute instance-attribute

lo: float = 0.0 class-attribute instance-attribute

hi: float = 1.0 class-attribute instance-attribute

Mapping(targets: List[Target], rows: Dict[str, Dict[str, float]] = dict(), allow_custom_symbols: bool = False, normalize: Optional[Callable[[str], str]] = None) dataclass

targets: List[Target] instance-attribute

rows: Dict[str, Dict[str, float]] = field(default_factory=dict) class-attribute instance-attribute

allow_custom_symbols: bool = False class-attribute instance-attribute

normalize: Optional[Callable[[str], str]] = None class-attribute instance-attribute

target_names: List[str] property

row(phoneme: str) -> Dict[int, float]

Target-index -> weight for a (possibly stressed) phoneme. Unknown phonemes fall back to the silence row, like the built-in map. With allow_custom_symbols the key is matched verbatim (vendor symbols carry no stress digit and are case-significant), unless a normalize hook is set, which is applied to the key first.

Source code in src/openfacefx/mapping.py 
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
def row(self, phoneme: str) -> Dict[int, float]:
    """Target-index -> weight for a (possibly stressed) phoneme.
    Unknown phonemes fall back to the silence row, like the built-in map.
    With ``allow_custom_symbols`` the key is matched verbatim (vendor
    symbols carry no stress digit and are case-significant), unless a
    ``normalize`` hook is set, which is applied to the key first."""
    if self.allow_custom_symbols:
        key = self.normalize(phoneme) if self.normalize is not None else phoneme
        row = self.rows.get(key)
        if row is None:
            row = self.rows.get(SILENCE) or {}
    else:
        key = strip_stress(phoneme).upper() if phoneme != SILENCE else SILENCE
        row = self.rows.get(key) or self.rows.get(SILENCE) or {}
    return {self.index[n]: w for n, w in row.items()}

default() -> 'Mapping' classmethod

Source code in src/openfacefx/mapping.py 
122
123
124
125
126
@classmethod
def default(cls) -> "Mapping":
    targets = [Target(v, _DEFAULT_CLASSES.get(v, "basic")) for v in VISEMES]
    rows = {ph: {vis: 1.0} for ph, vis in PHONEME_TO_VISEME.items()}
    return cls(targets, rows)

from_json(path: str) -> 'Mapping' classmethod

Source code in src/openfacefx/mapping.py 
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
@classmethod
def from_json(cls, path: str) -> "Mapping":
    with open(path, encoding="utf-8") as fh:
        try:
            d = json.load(fh)
        except json.JSONDecodeError as e:
            raise ValueError(f"{path}: not valid JSON ({e})") from None
    if d.get("format") != "openfacefx.mapping" or d.get("version") != 1:
        raise ValueError(
            f"{path}: expected format 'openfacefx.mapping' version 1")
    try:
        targets = [Target(t["name"], t.get("class", "basic"),
                          float(t.get("min", 0.0)), float(t.get("max", 1.0)))
                   for t in d["targets"]]
    except (KeyError, TypeError) as e:
        raise ValueError(f"{path}: malformed targets entry ({e})") from None
    phonemes = d.get("phonemes")
    if not isinstance(phonemes, dict) or not phonemes:
        raise ValueError(f"{path}: 'phonemes' must be a non-empty object")
    # "custom_symbols": true lets a mapping file key rows by a non-ARPABET
    # alphabet (SAMPA/IPA from TTS timing sources) — matched verbatim.
    return cls(targets, phonemes,
               allow_custom_symbols=bool(d.get("custom_symbols", False)))

to_json(path: str) -> None

Source code in src/openfacefx/mapping.py 
152
153
154
155
156
157
158
159
160
161
162
163
164
165
def to_json(self, path: str) -> None:
    d = {
        "format": "openfacefx.mapping",
        "version": 1,
        "targets": [
            {"name": t.name, "class": t.articulator,
             "min": t.lo, "max": t.hi}
            for t in self.targets
        ],
        "phonemes": self.rows,
    }
    with open(path, "w", encoding="utf-8") as fh:
        json.dump(d, fh, indent=2)
        fh.write("\n")

openfacefx.ipa

Built-in IPA -> Oculus-15 mapping preset (issue #32).

Piper and Cartesia timestamp their phonemes in IPA, and espeak-ng's MBROLA .pho dumps a SAMPA variant -- neither matches the ARPABET the default mapping expects, so from-timing used to relax those sources to silence unless the user hand-wrote a custom_symbols mapping. This module ships that mapping as data: IPA_MAPPING keys the Oculus-15 targets by the IPA inventory those engines actually emit, matched through _normalize_ipa so the diacritics real dumps carry collapse onto the base symbol instead of duplicating a table row per variant.

Normalization rules (_normalize_ipa), applied to the lookup key: * primary/secondary stress marks ˈ ˌ are dropped; * length marks ː ˑ are dropped (so ɑː matches ɑ); * the MFA-style secondary-articulation modifier letters ʰ ʲ ʷ are dropped (so pʰ tʲ kʷ match p t k); * every combining mark is dropped, which folds the affricate tie bar (t͡ʃ -> tʃ, matching the plain digraph too), the dental t̪ -> t, the syllabic n̩ -> n and nasalization ẽ -> e; * ASCII ' (stress) and : (X-SAMPA length) are dropped. It is idempotent and a no-op on ARPABET (which carries none of these), so the default pipeline is untouched.

Symbol inventory is grounded in verifiable sources
  • espeak-ng's phoneme guide -- affricates written as tie bars (t͡ʃ d͡ʒ), stress ˈ ˌ, length ː ˑ (espeak-ng/espeak-ng docs/phonemes.md).
  • the Montreal Forced Aligner US-English phone set, which Cartesia's sonic models use verbatim -- the aj aw ej ow ɔj diphthong spellings and the pʰ pʲ pʷ tʰ tʲ tʷ kʰ kʷ secondary articulations (docs.cartesia.ai, "Specify Custom Pronunciations").
  • the English IPA key most G2P/TTS front-ends follow -- the aɪ aʊ eɪ oʊ ɔɪ diphthongs and ɜ ɝ ɚ r-coloured vowels (Wikipedia Help:IPA/English).

The IPA-symbol -> viseme assignment itself is our articulatory synthesis -- the same many-to-one judgement calls visemes.PHONEME_TO_VISEME documents for ARPABET, not a figure lifted from any single source.

IPA_VOWELS = frozenset(_VOWELS) | frozenset(_SAMPA_VOWELS) module-attribute

IPA_MAPPING = Mapping([(Target(v, _DEFAULT_CLASSES.get(v, 'basic'))) for v in VISEMES], _IPA_ROWS, allow_custom_symbols=True, normalize=_normalize_ipa) module-attribute

is_ipa_vowel(symbol: str) -> bool

True if a raw IPA/SAMPA token is a vowel (monophthong, diphthong or r-coloured), consulted by the coarticulation dominance model so vendor vowels get the broad vowel bump. Normalizes first (ˈaɪ, ɑː -> aɪ, ɑ). Returns False for every ARPABET symbol, so the ARPABET path is byte-for-byte unchanged.

Source code in src/openfacefx/ipa.py 
183
184
185
186
187
188
189
def is_ipa_vowel(symbol: str) -> bool:
    """True if a raw IPA/SAMPA token is a vowel (monophthong, diphthong or
    r-coloured), consulted by the coarticulation dominance model so vendor
    vowels get the broad vowel bump. Normalizes first (``ˈaɪ``, ``ɑː`` ->
    ``aɪ``, ``ɑ``). Returns False for every ARPABET symbol, so the ARPABET path
    is byte-for-byte unchanged."""
    return _normalize_ipa(symbol) in IPA_VOWELS

ipa_unknown_symbols(symbols: Iterable[str]) -> List[str]

QA warnings for phoneme symbols the preset can't place -- they route to silence -- one line per distinct symbol (sorted), mirroring the vendor viseme path. A lone suprasegmental (a bare stress/length mark that normalizes to empty) is a structural token, not an unknown, so it never warns.

Source code in src/openfacefx/ipa.py 
192
193
194
195
196
197
198
199
200
201
202
203
204
def ipa_unknown_symbols(symbols: Iterable[str]) -> List[str]:
    """QA warnings for phoneme symbols the preset can't place -- they route to
    silence -- one line per distinct symbol (sorted), mirroring the vendor
    viseme path. A lone suprasegmental (a bare stress/length mark that
    normalizes to empty) is a structural token, not an unknown, so it never
    warns."""
    counts: Dict[str, int] = {}
    for s in symbols:
        norm = _normalize_ipa(s)
        if norm and norm not in _IPA_ROWS:
            counts[s] = counts.get(s, 0) + 1
    return [f"unknown IPA symbol {s!r} ({n}x) routed to silence"
            for s, n in sorted(counts.items())]