# Baked (Pre-Baked) Lip Sync

Pre-bake your audio clips' lip sync data offline in the editor, then play it back at runtime with **zero FFT cost**. Ideal for WebGL, mobile, deterministic playback, or performance-critical scenes.

<figure><img src="https://1935854846-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FuciaAIoc6XDGWTpXdIP7%2Fuploads%2FvEqMlLXREhfigeXCvn3C%2Fimage.png?alt=media&#x26;token=ab654aad-3981-4fcb-aea9-4bff29a81139" alt=""><figcaption></figcaption></figure>

***

### Overview

The baked lip sync workflow has three parts:

1. **Bake** - An editor window analyzes your AudioClip offline using the same spectral analysis as real-time lip sync, and saves the per-frame viseme weights into a ScriptableObject asset.
2. **Store** - The `CrystalLipSyncClipData` ScriptableObject holds all baked data: viseme weights, dominant visemes, and RMS volumes at a configurable frame rate.
3. **Play** - The `CrystalBakedLipSync` runtime component reads the baked data synchronized with an AudioSource and writes viseme weights to the controller ... no FFT at runtime.

***

### When To Use Baked Lip Sync

| Scenario                       |             Recommendation             |
| ------------------------------ | :------------------------------------: |
| WebGL build                    |   ✅ Best option (no WebGL FFT quirks)  |
| Mobile (battery/thermal)       |      ✅ Zero CPU cost per character     |
| Many characters speaking       | ✅ Scales to dozens without frame drops |
| Cutscenes / pre-recorded VO    |   ✅ Deterministic results every time   |
| Dynamic / user-generated audio |    ❌ Use real-time lip sync instead    |
| Microphone input               |        ❌ Use real-time lip sync        |
| Text lip sync (no audio)       |        ❌ Use CrystalTextLipSync        |

***

### Baking Audio Clips

#### Open the Bake Window

**Tools → Crystal LipSync → Bake Lip Sync**

<figure><img src="https://1935854846-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FuciaAIoc6XDGWTpXdIP7%2Fuploads%2FVzWjUB1WqZ6MqXrLZMIW%2Fimage.png?alt=media&#x26;token=723d0f7f-139d-45c6-948c-79fa9f96910e" alt=""><figcaption></figcaption></figure>

#### Single Clip

1. Drag an **AudioClip** into the **Clip** field
2. (Optional) Assign a **Profile** to auto-fill sensitivity/threshold/smoothing
3. Adjust analysis settings if needed:
   * **FFT Size** ... Higher = better frequency resolution (1024 recommended)
   * **Sensitivity** ... Match your controller's sensitivity setting
   * **Volume Threshold** ... Match your controller's threshold
   * **Smoothing Attack / Release** ... Match your controller's smoothing
4. Set the **Frame Rate** (60 fps default ... 30 fps saves memory, 60 fps is smoother)
5. Click **Bake Single Clip**
6. Choose a save location ... a `.asset` file is created

#### Batch Bake

1. Expand **Batch Bake** in the bake window
2. Add multiple clips via **+ Add Clip** or drag-and-drop
3. Click **Bake All (Batch)**
4. Choose a destination folder ... one `.asset` per clip

#### Matching Settings to Your Controller

For best results, use the **same settings** in the bake window as on your `CrystalLipSyncController`:

| Bake Window       | Controller Inspector |
| ----------------- | -------------------- |
| FFT Size          | FFT Size             |
| Sensitivity       | Sensitivity          |
| Volume Threshold  | Volume Threshold     |
| Smoothing Attack  | Smoothing Attack     |
| Smoothing Release | Smoothing Release    |

Or assign the same **Profile** asset to both.

***

### Runtime Setup

#### Components Required

| Component                                                         | GameObject            | Purpose                                   |
| ----------------------------------------------------------------- | --------------------- | ----------------------------------------- |
| `CrystalLipSyncController`                                        | Character             | Holds `VisemeWeights[]`, fires events     |
| `CrystalBakedLipSync`                                             | Character             | Reads baked data, writes to controller    |
| `CrystalLipSyncBlendshapeTarget` or `CrystalLipSyncJawBoneTarget` | Character             | Reads `VisemeWeights[]`, drives mesh/bone |
| `AudioSource`                                                     | Character (or shared) | Plays the voice audio                     |

#### Basic Setup

1. Add `CrystalBakedLipSync` to the same GameObject as your `CrystalLipSyncController`
2. Assign the baked **Clip Data** asset
3. (Optional) Assign the **Audio Source** ... if left empty, uses the controller's AudioSource
4. Play the AudioClip on the AudioSource ... lip sync starts automatically

<figure><img src="https://1935854846-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FuciaAIoc6XDGWTpXdIP7%2Fuploads%2Fwi7rrkYAxub3sdRxNRnG%2Fimage.png?alt=media&#x26;token=78fefcb2-5419-41ea-a5a0-4b8aad586e7a" alt=""><figcaption></figcaption></figure>

#### Inspector Settings

| Setting                  | Default             | Description                                                  |
| ------------------------ | ------------------- | ------------------------------------------------------------ |
| **Controller**           | Auto-detect         | The controller to write weights into                         |
| **Audio Source**         | Controller's source | AudioSource to sync timing with                              |
| **Clip Data**            | (required)          | The baked `CrystalLipSyncClipData` asset                     |
| **Auto Play**            | ✅                   | Start/stop with the AudioSource automatically                |
| **Time Offset**          | 0                   | Shift lip sync timing (positive = later, negative = earlier) |
| **Intensity**            | 1.0                 | Weight multiplier for all baked visemes                      |
| **Additional Smoothing** | 0                   | Extra smoothing on top of baked data (0 = raw)               |

***

### Scripting API

#### Swap Clips at Runtime

```csharp
var bakedLipSync = GetComponent<CrystalBakedLipSync>();
bakedLipSync.SetClipData(newClipData);    // assign new baked data
audioSource.clip = newAudioClip;           // assign matching audio
audioSource.Play();                        // auto-play triggers lip sync
```

#### Manual Play/Stop

```csharp
bakedLipSync.SetClipData(clipData);
bakedLipSync.Play();    // start manually (disable Auto Play in inspector)
// ... later
bakedLipSync.Stop();    // smooth decay to rest
```

#### Events

```csharp
bakedLipSync.OnPlaybackStarted += () => Debug.Log("Lip sync started");
bakedLipSync.OnPlaybackStopped += () => Debug.Log("Lip sync stopped");
```

#### Querying Baked Data Directly

```csharp
CrystalLipSyncClipData data = bakedLipSync.ClipData;

// Sample at arbitrary time
float[] weights = new float[VisemeConstants.VisemeCount];
data.SampleWeights(1.5f, weights);          // weights at t=1.5s
int dominant = data.SampleDominantViseme(1.5f);
float volume = data.SampleVolume(1.5f);

// Metadata
Debug.Log($"Duration: {data.Duration}s, Frames: {data.FrameCount}, Rate: {data.FrameRate}");
```

***

### Priority & Coexistence

`CrystalBakedLipSync` **takes priority** over real-time FFT when actively playing:

* **Baked overrides real-time.** When baked playback is active, it writes to `VisemeWeights[]` **after** the controller's real-time analysis (via `[DefaultExecutionOrder(100)]`), effectively overriding whatever the controller wrote. This is intentional ... baked data represents the same spectral analysis, just pre-computed.
* **Real-time fills gaps.** When baked playback is NOT active, the controller's real-time FFT drives the weights as usual.
* **Text lip sync yields.** `CrystalTextLipSync` only writes when `IsActive` is false (no audio playing). This is unchanged.

> **Tip:** You can freely have both `CrystalBakedLipSync` and real-time analysis on the same controller. When baked is playing, it wins. When baked stops, real-time takes over seamlessly.

#### Recommended WebGL Setup

For WebGL, you have three options:

| Approach          | Pros                                       | Cons                       |
| ----------------- | ------------------------------------------ | -------------------------- |
| **Real-time**     | No baking needed, works with dynamic audio | Managed FFT cost per frame |
| **Baked only**    | Zero CPU cost, deterministic               | Requires baking step       |
| **Hybrid** (both) | Best of both worlds                        | Slightly more setup        |

> **Note:** Real-time lip sync works on WebGL out of the box ... the controller automatically uses `OnAudioFilterRead` + managed FFT. Baking is an optional optimization, not a requirement.

For a baked-only setup, do NOT assign the AudioSource to the Controller ... assign it only to `CrystalBakedLipSync`:

```
CrystalLipSyncController  → AudioSource: None
CrystalBakedLipSync        → AudioSource: (your AudioSource)
                           → Clip Data: (baked asset)
```

***

### Dialogue System Integration

Baked lip sync works automatically with **Game Creator 2 Dialogue** and **PixelCrushers Dialogue System** ... no per-line scripting needed. The key is the **Baked Clip Lookup** table.

#### How It Works

When you bake an AudioClip, you get a `CrystalLipSyncClipData` asset. The **Baked Clip Lookup** (`CrystalBakedClipLookup`) maps each AudioClip to its baked data:

```
AudioClip "greeting_01.wav"  →  CrystalLipSyncClipData "greeting_01_baked.asset"
AudioClip "goodbye_03.wav"   →  CrystalLipSyncClipData "goodbye_03_baked.asset"
AudioClip "quest_info.wav"   →  CrystalLipSyncClipData "quest_info_baked.asset"
```

At runtime, when a dialogue line plays an AudioClip, the bridge component looks it up in the table. If a match exists, it automatically assigns the baked data and starts playback ... the character's mouth moves using the pre-baked viseme weights for that specific clip.

**Any baked clip works automatically.** As long as the AudioClip is registered in the lookup table, switching between different dialogue lines, different speakers, or different conversations all works without any extra setup per line.

#### Setup - Game Creator 2

1. **Bake** all your voice-over AudioClips (use Batch Bake with **Auto-Populate Lookup** enabled)
2. **Create** a Baked Clip Lookup asset: **Assets → Create → Crystal LipSync → Baked Clip Lookup**
3. **Register** each AudioClip ↔ ClipData pair (automatic if Auto-Populate is configured in the Bake Window)
4. **Assign** the lookup to `CrystalDialogueLipSync` → **Baked Clip Lookup** field
5. **Ensure** each speaking character has a `CrystalBakedLipSync` component

The flow at runtime:

```
GC2 Dialogue node starts
    → CrystalDialogueLipSync receives EventStartNext
    → node.GetAudio() returns the AudioClip
    → bakedClipLookup.Find(audioClip) → matching baked data
    → bakedLipSync.SetClipData(clipData) + Play()
    → character's mouth animates from baked data
```

If a clip has no baked data in the lookup, the bridge skips silently and the controller's real-time FFT handles it instead.

#### Setup - PixelCrushers Dialogue System

1. **Bake** all your voice-over AudioClips
2. **Create** and populate a Baked Clip Lookup asset
3. **Assign** the lookup to `CrystalDialogueSystemLipSync` → **Baked Clip Lookup** field
4. **Ensure** each speaking character has a `CrystalBakedLipSync` component

The flow at runtime:

```
Dialogue entry plays (with AudioWait/Audio/Voice in Sequence)
    → CrystalDialogueSystemLipSync receives OnConversationLine
    → waits for the Sequencer to load and play the AudioClip
    → reads audioSource.clip once it's playing
    → bakedClipLookup.Find(audioSource.clip) → matching baked data
    → bakedLipSync.SetClipData(clipData) + Play()
    → character's mouth animates from baked data
```

#### Mixing Baked and Non-Baked Lines

You don't need to bake every clip. Within the same conversation:

* Lines **with** baked data → baked lip sync drives the mouth
* Lines **without** baked data → real-time FFT analysis takes over
* Lines **without audio** → text lip sync kicks in (if configured)

All three modes can coexist on the same character seamlessly.

***

### Memory & Performance

#### Baked Asset Size

| Frame Rate | Clip Length | Frames | Weight Data | Total Asset |
| ---------- | ----------- | ------ | ----------- | ----------- |
| 30 fps     | 5s          | 150    | 9 KB        | \~10 KB     |
| 60 fps     | 5s          | 300    | 18 KB       | \~19 KB     |
| 60 fps     | 30s         | 1800   | 108 KB      | \~112 KB    |
| 60 fps     | 60s         | 3600   | 216 KB      | \~224 KB    |

Each frame stores 15 floats (weights) + 1 int (dominant) + 1 float (volume) = 68 bytes.

#### Runtime Cost

| Operation             | Cost                      |
| --------------------- | ------------------------- |
| Sample weights (lerp) | \~15 multiplies + 15 adds |
| Apply to controller   | \~15 copies               |
| Total per frame       | **< 0.01ms**              |

***

### Bake Window Preview

After baking, the editor window shows a visual timeline of the baked data:

* **Height** = RMS volume
* **Color** = dominant viseme (hue-mapped)
* **Width** = clip duration

This gives you a quick visual sanity check before testing in-game.

***

### Troubleshooting

#### Lips don't move

* Verify the **Clip Data** asset has data (`HasData` checkbox in inspector)
* Check that `Auto Play` is enabled, or call `Play()` manually
* Ensure the controller's `IsActive` is `false` (no real-time FFT overriding baked data)

#### Timing is off

* Adjust **Time Offset** (positive = later, negative = earlier)
* Ensure the AudioSource and baked data reference the same AudioClip

#### Baked data looks wrong

* Re-bake with settings matching your controller's (especially Sensitivity and Smoothing)
* Try a higher **Frame Rate** (60+) for faster speech

#### "No data" in baked asset

* The AudioClip may be compressed with **Load Type: Streaming**. Change to **Decompress On Load** or **Compressed In Memory** and re-bake.

***

### FAQ

**Q: Can I bake clips at edit time and play them on WebGL?** A: Yes ... that's the primary use case. Bake in the editor, ship the `.asset` files with your build.

**Q: Do I still need CrystalLipSyncController?** A: Yes. The controller holds `VisemeWeights[]` and fires `OnVisemeWeightsUpdated`. Targets read from the controller.

**Q: Can I use baked lip sync and text lip sync together?** A: They both write to `controller.VisemeWeights[]` when `IsActive` is false. The last writer wins. Avoid running both simultaneously on the same controller.

**Q: Can I re-bake a clip without losing the reference?** A: Yes. If you save to the same path, the existing asset is updated in-place. All references remain valid.

**Q: What frame rate should I use?** A: 60 fps is recommended for smooth results. 30 fps is fine for stylized characters. Below 30 fps may look choppy.