INTEGRATION.md

LiteRT integration contract: nemotron3-diarization-low-latency-v2

This contract describes the INT8-weight, 1.04 s input-buffer profile. Treat int8/nemotron3-diarization-manifest.json as authoritative. Verify its SHA-256 against publication.json, then verify every inventoried file before constructing interpreters.

Graph inventory and ownership

Load both graphs from one int8 directory and keep them resident:

1. feature_stacker.tflite
2. encoder_head.tflite

Every signature is serving_default. Bind inputs by numeric args_N order, never lexical tensor-name order. Copy an output before invoking the same interpreter again if the Android runtime may reuse its backing buffer.

The application owns PCM framing, log-mel extraction, fixed-shape padding, RoPE inputs, AOSC/FIFO state, output slicing, and segment stabilization. LiteRT owns neural inference only. The Python reference is executable pseudocode for these responsibilities.

Audio and feature extraction

Input is mono Float32 PCM at 16,000 Hz. Convert PCM16 and resample outside the graphs. The feature extractor uses:

• pre-emphasis coefficient 0.97;
• 512-point STFT, 400-sample non-periodic Hann window, 160-sample hop;
• constant zero padding and centered windows only for the first chunk;
• 128 Slaney-normalized mel filters from 0 to 8 kHz; and
• log(mel_power + 2^-24) without mean/variance normalization.

For the first step, provide samples [0, 16680) and use centered STFT. It yields 104 valid mel frames. Thereafter the frame cursor advances by 72 mel frames. A regular chunk beginning at mel frame f reads 17,040 samples beginning at f*160 - 256 and uses uncentered STFT, again yielding 104 frames. The 32-mel-frame overlap is the four-frame right context. A final short chunk is allowed and scores all of its valid frames.

Zero-pad valid features to [1,104,128] and invoke feature_stacker. Its output is [1,13,512]; retain only ceil(validMelFrames/8) embeddings for a short final chunk.

Fixed encoder sequence

Maintain:

• AOSC: up to 264 encoder frames;
• FIFO: up to 264 encoder frames; and
• current chunk: up to 13 encoder frames, of which the last four are look-ahead except on finalization.

Concatenate populated AOSC, populated FIFO, and current embeddings. Right-align them into [1,541,512], leaving zero padding on the left. Construct additive attention mask [1,1,1,541]: 0 for valid keys and Float.MIN_VALUE/negative finite max magnitude for left padding.

Precompute RoPE cosine and sine tensors [1,541,64] for positions 0 through 540 with base 10,000. Because valid content is right-aligned consistently, these fixed relative positions preserve the source attention behavior.

Invoke encoder_head.tflite with hidden state, additive mask, cosine, sine, and Boolean valid-frame mask [1,541]. The graph contains encoder layers 00–30 followed by the classification head and returns logits [1,4328,8] (541 × 8 subframes).

Slice current output beginning at (leftPadding + cachedFrames)*8. Emit nine encoder frames (72 logits) for a regular chunk. On finalization, emit every valid current frame. Each logit frame represents 10 ms and each of eight channels is an arrival-ordered speaker; multiple positive channels represent overlap.

AOSC/FIFO update

After output slicing, pool sigmoid logits in non-overlapping groups of eight to score encoder frames. Add only the nine current frames to FIFO; never add four look-ahead frames because they are processed again as current input on the next step.

FIFO capacity is 264. On overflow, move at least 222 oldest frames into AOSC. When AOSC exceeds 264, apply the source score policy in reference_runtime.py: per-speaker positive scoring, latest-frame boost 0.05, strong/weak top-frame boosts, one learned silence frame per speaker, global top-264 selection, then speaker-major/original-order sorting. silence_embedding.npy is required state, not optional metadata.

Reset AOSC/FIFO only for a new independent recording. Microphone packet boundaries do not reset model state. Missing packets must be surfaced; silently closing time gaps corrupts the speaker timeline.

Output stabilization

The source processor applies a plain probability threshold of 0.5. INT8 perturbations can create short boundary flickers around that threshold. The published reference closes inactive gaps of at most 100 ms per speaker, then removes active events shorter than 100 ms. Keep overlap channels independent. Applications may expose raw logits, but the default segment API should apply this stabilizer and document any changes to it.

Coordination with speech recognition

This model is independent from ASR. Run diarization over the same continuous PCM timeline and intersect its speaker activity with the ASR engine's time-aligned words or segments:

• Nemotron/Parakeet: use native token timings.
• Whisper: use word or segment alignment tokens.
• Built-in/platform STT: use platform timestamps when present; otherwise only coarse chunk-level attribution is defensible.
• Gemma text-only output: retain the existing Pyannote/TitaNet timeline unless Gemma gains reliable time alignment; do not invent timestamps from text position.

Do not serialize ASR and diarization through one inference worker unless memory measurements require it. Keep separate workers and bound concurrent LiteRT execution after profiling delegate contention.

Unmet production gates

The bundle is evaluation_only until it passes corpus-level DER/JER and overlap evaluation, a broader Android golden-output/device matrix, sustained thermal tests, concurrent-ASR resource testing, process-death cleanup, and delegate fallback coverage. File size is not peak memory. Desktop and single-device RTF measurements do not predict a particular phone.