transcript of conversation

[arunpaul93]

Description of the feature request:

No response

What problem are you trying to solve with this feature?

No response

Any other information you'd like to share?

No response

[bhaskarblur]

Hi. Yes, it is possible to receive both input and output transcription while receiving audio. There are 2 params you've to send in setup options at root of the config for Gemini:

outputTranscription: {}
inputTranscription: {}
Paste as it is.

Cheers

[arunpaul93]

Hi. Yes, it is possible to receive both input and output transcription while receiving audio. There are 2 params you've to send in setup options at root of the config for Gemini:

outputTranscription: {} inputTranscription: {} Paste as it is.

Cheers

is that in the [ResponseModalitySelector.tsx]

I tried , but didnt work for me

[bhaskarblur]

Read:
https://ai.google.dev/api/live#BidiGenerateContentTranscription

https://ai.google.dev/api/live#AudioTranscriptionConfig

            "outputAudioTranscription": {}, # Enable audio transcription for output
            "inputAudioTranscription": {} # Enable audio transcription for input

[bhaskarblur]

@arunpaul93 Tried this? Did it work for you?

[arunpaul93]

i had tried but more than one modality wasn't working
in the issues, there was previously one such comment by someone

[bhaskarblur]

This works like charm for me, i get both input(user) and output(ai) transcriptions.
Also you do not need to change modality, keep it AUDIO only. Just add these 2 configs i shared and it will start giving you transcriptions in a separate msg from the audio buffers.

[arunpaul93]

which file did you had to edit from this repo?

[bhaskarblur]

Here.
setConfig({
responseModalities: [Modality.AUDIO],
speechConfig: {
voiceConfig: { prebuiltVoiceConfig: { voiceName: "Aoede" } },
},
// Put those 2 transcription configs here...

  systemInstruction: {
    parts: [
      {
        text: 'You are my helpful assistant. Any time I ask you for a graph call the "render_altair" function I have provided you. Dont ask for additional information just make your best judgement.',
      },
    ],
  },
  tools: [
    // there is a free-tier quota for search
    { googleSearch: {} },
    { functionDeclarations: [declaration] },
  ],
});

In the src/components/altair/Altair.tsx

[secret104278]

@bhaskarblur thanks tons!!! that exactly im looking for.

[bhaskarblur]

Welcome! Cheers 🥂

[arunpaul93]

i tried this , however could'nt see the transcripts anywhere
where will it be

[bhaskarblur]

They come in a separate message from Audio buffers,
Here's an examplecode:

Output: response["serverContent"]["outputTranscription"]["text"]
Input: response["serverContent"]["inputTranscription"]["text"]

So, find them gemini ws response in serverContent -> inputTranscription/outputTranscription -> text

[glued]

They come in a separate message from Audio buffers, Here's an examplecode:

Output: response["serverContent"]["outputTranscription"]["text"] Input: response["serverContent"]["inputTranscription"]["text"]

So, find them gemini ws response in serverContent -> inputTranscription/outputTranscription -> text

that does not work

[bhaskarblur]

Not sure why you guys are facing failure. Works good for me.