Open a different process to capture dictation in MacOs

[eduardoarandah]

In MacOs, dictation doesn't work because it's very specific to the os.

See: #3732 (comment)

However, dictation is a central part for some people vibing with LLMs or simply being forced to type less due to health issues.

What if instead of you could open another process, a simple modal triggering dictation and then capture the text?

The modal being a process that implements "cocoa view infrastructure", read: #3732 (comment)

dictation.mp4

[kovidgoyal]

Feel free to try.

[eduardoarandah]

There's this tool "hear" that triggers voice recognition and works inside kitty!

https://github.com/sveinbjornt/hear?tab=readme-ov-file

using-hear.mp4

Saw it mentioned in wezterm

wezterm/wezterm#4592 (comment)

I'd love to integrate it or somehow make a plugin for kitty but I don't have the knowledge to do it, any ideas?

[kovidgoyal]

Should be pretty trivial map f1 remote_control_script my_hear_script Then in my_hear_script do something like: 1) Run hear and get text output 2) Use kitten @ send-text to send the text to the active kitty window If you want to run it in its own window, something like map f1 launch --allow-remote-control my_hear_script ```sh #!/bin/sh text=$(hear) kitten @ send-text -m state:overlay_parent -- "$text" ``` make it more robust as you like.

[eduardoarandah]

ok this works!

however, the output of hear is somehow rewritten and I get repetition. I guess there's a way to fix it.. idk

kitty.conf:

map f2 launch --type=overlay --allow-remote-control /Users/lalo/repos/dotfiles/hear.sh

the script:

#!/bin/sh
tmpfile=$(mktemp)
/opt/homebrew/bin/hear --locale=en-US -t 2 -m -p 2>&1 | tee $tmpfile
text=$(cat $tmpfile)
kitten @ send-text -m state:overlay_parent -- "ok text: $text"
rm $tmpfile

dictation-attempt.mov_4fps.mp4

[kovidgoyal]

You can make various improvements. Assuming whatever program you are runningin the terminal supports bracketed paste, you can use send-text with --bracketed-paste and also use either --from-file or --stdin so that the text is not interpreted for escapes. Read kitten @ send-text --help

[eduardoarandah]

:-D It's really close to working!

somehow, the dictation changes constantly what was heard.

I tried also with no luck:

• saving to a file and then read from it
• saving to a file, wait 2 seconds and read from it

dictation.mov_4fps.mp4

#!/bin/sh
# --mode Enable single-line output mode (mic only)
/opt/homebrew/bin/hear --locale=en-US --timeout 2 --mode --punctuation | \
tee /dev/tty | \
kitten @ send-text --match state:overlay_parent --stdin

[eduardoarandah]

This is strange, when executing this script, output goes correctly to out.txt

but when used with kitten @ send-text then it goes wrong

#!/usr/bin/env zsh
rm -f out.txt
/opt/homebrew/bin/hear --locale=en-US --timeout 2 --mode --punctuation | \
tee /dev/tty | \
> out.txt;

echo ""
echo "here's the output:"
cat out.txt

strange.mov_4fps.mp4

[kovidgoyal]

You have not added --bracketed-paste=auto

[kovidgoyal]

And I tested the kitty parts of this, with:

kitty --config=NONE -o 'map f1 launch --type=overlay --allow-remote-control sh -c "echo -n some text | kitten @ send-text --bracketed-paste=auto --match state:overlay_parent --stdin"'

Pressing F1 causes "some text" to appear at the shell prompt as expected.

[eduardoarandah]

yeah, kitten sends the text, however for some reason it does it wrong for apple dictation

Here's one liners if you have a mac near and want to give it a try.

map f2 launch --type=overlay --allow-remote-control sh -c "/opt/homebrew/bin/hear --locale=en-US --timeout 2 --mode --punctuation | tee /dev/tty | kitten @ send-text --bracketed-paste=auto --match state:overlay_parent --stdin"

or

kitty --config=NONE -o 'map f2 launch --type=overlay --allow-remote-control sh -c "/opt/homebrew/bin/hear --locale=en-US --timeout 2 --mode --punctuation | tee /dev/tty | kitten @ send-text --bracketed-paste=auto --match state:overlay_parent --stdin"'

bracketed.mov_4fps.mp4

Installing hear

brew tap sveinbjornt/hear https://github.com/sveinbjornt/hear
brew install sveinbjornt/hear/hear

# uninstall with
brew uninstall hear

https://github.com/sveinbjornt/hear?tab=readme-ov-file#install-via-homebrew

[eduardoarandah]

Dictation is super useful when working with LLMs because they need a lot of explaining in natural language and apps like claude code run inside kitty.

I just tried an app called "superwhisper" and it works fine for dictation in kitty in its free version, however it would be great using what's already built-in in the os 😄

[kovidgoyal]

You cannot tee to /dev/tty and use kitten @ send-text which is performing I/O on /dev/tty at the same time. Instead capture your output from hear into a file or a shell variable or similar and *once it is complete* use send-text on the result.

[eduardoarandah]

this is schrodingers cat or something, tmp.txt file ends up fine, but kitten @ send-text sends another thing.

I made sure to wait enough just in case. (I changed sleep to 10 with same result)

kitty.conf

map f2 launch --type=overlay --allow-remote-control /Users/lalo/repos/dotfiles/hear.sh

hear.sh

#!/usr/bin/env zsh

# generate text from hear
text=$(/opt/homebrew/bin/hear -d --locale=en-US --timeout 2 --mode --punctuation) 

# write text to file
echo "$text" > /Users/lalo/repos/dotfiles/tmp.txt

# let's wait a bit, maybe hear takes a while to generate the text
sleep 10 

# send text to kitty
kitten @ send-text --bracketed-paste=auto --match state:overlay_parent --from-file /Users/lalo/repos/dotfiles/tmp.txt

why-no-work.mov_4fps.mp4

[kovidgoyal]

That's because the single line mode in hear is implemented as outputting word then the escape code to clear the line the the next two words and so on. Dont use single line mode. Use normal mode and simply take the last line of output. Proof of concept:

kitty --config=NONE -o 'map f1 launch --type=overlay --allow-remote-control zsh -c "echo Waiting for speech...; text=$(/opt/homebrew/bin/hear -d --locale=en-US --timeout 2 --punctuation | tee /dev/tty | tail -n1); echo Spoke: $text; echo Press enter  to send to terminal; read; echo -n \"$text\" | kitten @ send-text --stdin --bracketed-paste=auto -m state:overlay_parent;"'

You can make it a little nicer/more robust