I want to stream audio from the web and convert that to text using Python Google-cloud-speech API. I have integrated that in my Django channels code.
For frontend, I have directly copied this code and the backend has this code (please see below). Now, coming to the problem, I am not getting any exceptions or errors but I was not getting any results from google API.
What I tried:
I put debug points inside for loop of
processfunction, the control never reaches inside the loop.I have gone through the java code here and tried to understand that. I have a setup that java code in my local and debugged it. One thing I understood is in java code, the method
onWebSocketBinaryis receiving an integer array, from frontend we are sending that like this.socket.send(Int16Array.from(floatSamples.map(function (n) {return n * MAX_INT;})));In java, they are converting into bytestring then sending it to Google. Whereas in Django, I put debug points and noticed that I am getting data in a binary string. So, I felt I don't need to do anything with that. but, I tried few several ways by converting that to integer array, but that didn't work because google is expecting in bytes itself (you can see the commented code below).
I went through this example code and this from Google and I am doing the same thing, I didn't understand what I am doing it wrong here.
Django Code:
import json
from channels.generic.websocket import WebsocketConsumer
# Imports the Google Cloud client library
from google.cloud import speech
from google.cloud.speech import enums
from google.cloud.speech import types
# Instantiates a client
client = speech.SpeechClient()
language_code = "en-US"
streaming_config = None
class SpeechToTextConsumer(WebsocketConsumer):
def connect(self):
self.accept()
def disconnect(self, close_code):
pass
def process(self, streaming_recognize_response: types.StreamingRecognitionResult):
for response in streaming_recognize_response:
if not response.results:
continue
result = response.results[0]
self.send(text_data=json.dumps(result))
def receive(self, text_data=None, bytes_data=None):
global streaming_config
if text_data:
data = json.loads(text_data)
rate = data["sampleRate"]
config = types.RecognitionConfig(
encoding=enums.RecognitionConfig.AudioEncoding.LINEAR16,
sample_rate_hertz=rate,
language_code=language_code,
)
streaming_config = types.StreamingRecognitionConfig(
config=config, interim_results=True, single_utterance=False
)
types.StreamingRecognizeRequest(streaming_config=streaming_config)
self.send(text_data=json.dumps({"message": "processing..."}))
if bytes_data:
# bytes_data = bytes_data[math.floor(len(bytes_data) / 2) :]
# bytes_data = bytes_data.lstrip(b"\x00")
# bytes_data = int.from_bytes(bytes_data, "little")
stream = [bytes_data]
requests = (
types.StreamingRecognizeRequest(audio_content=chunk) for chunk in stream
)
responses = client.streaming_recognize(streaming_config, requests)
self.process(responses)
