RTF evaluation issues
Thank you very much for your great work. I have some questions about canary-1b-flash. The RTFx speed you mentioned in your project is over 1000+ on a100 device, which is about RTF<0.001. However, in our actual test, the speed is far from enough. Can you help me find out where the problem is?
The test code is
`
test code
from NeMo.nemo.collections.asr.models import EncDecMultiTaskModel
from NeMo.nemo.collections.asr.parts.utils.streaming_utils import FrameBatchMultiTaskAED
from NeMo.nemo.collections.asr.parts.utils.transcribe_utils import get_buffered_pred_feat_multitaskAED
import time
import torch
device = 'cuda' if torch.cuda.is_available() else 'cpu'
canary_model = EncDecMultiTaskModel.restore_from('canary-1b-flash/canary-1b-flash.nemo', map_location=device)
canary_model = canary_model.to(device)
decode_cfg = canary_model.cfg.decoding
decode_cfg.beam.beam_size = 1
canary_model.change_decoding_strategy(decode_cfg)
feature_stride = canary_model.cfg.preprocessor['window_stride']
model_stride_in_secs = feature_stride * 8
start = time.time()
output = canary_model.transcribe(
"mix.wav",
batch_size=128, # batch size to run the inference with
# pnc='yes', # generate output with Punctuation and Capitalization
# timestamps='yes', # generate output with timestamps
)
predicted_text_1 = output[0].text
print(time.time()-start)
print(predicted_text_1)
start = time.time()
output = canary_model.transcribe(
"mix.wav",
batch_size=128, # batch size to run the inference with
# pnc='yes', # generate output with Punctuation and Capitalization
# timestamps='yes', # generate output with timestamps
# source_lang="en", # language of the audio input, set source_lang==target_lang for ASR, choices=['en','de','es','fr']
# target_lang="de", # language of the text output, choices=['en','de','es','fr']
)
predicted_text_1 = output[0].text
print(time.time()-start)
print(predicted_text_1)`
The processed audio time is about 10s, and the processed time is about 2s. Is this correct?
Our device is a single a100-40GB.
Encountered the same issue. The speed doesn't match the description.
Hi @cX1y ,
Thank you for trying out the model! From the code snippet you shared, it looks like you're passing a single audio file ("mix.wav") while setting batch_size=128. In this case, batching isn't actually being utilized, as you're only processing one file.
To better leverage your GPU, we recommend processing multiple audio files in parallel. You can do this either by:
- Creating a manifest file containing the paths to your audio files and passing it to canary_model.transcribe(), or
- Directly passing a list of audio files to the transcribe() function.
Both methods are described in more detail in the model card. .
Let me know if you have any follow-up questions!
I see, thanks for the reply, I'll try it