Using Python to Code by Voice

> The second thing we need to figure out is how we can compress voice input to make it faster to transmit. What’s the voice equivalent of a thumbs-up or a keyboard shortcut? Can I prompt Claude faster with simple sounds and whistles? This reminds me of the amazing 2013 video of Travis Rudd coding python by voice: ▶ this video The number of times in the last few years I've wanted that level of "verbal hotkeys"... The latencies of many coding llms are still a little bit too low to allow for my ideal level of flow (though admittedly I haven't tried one's hosted on services like groq), but I can clearly envision a time when I'm issuing tight commands to a coder model that's chatting with me and watching my program evolve on screen in real time. On a somewhat related note to conversational interfaces, the other day I wanted to study some first aid stuff - used Gemini to read the whole textbook and generate Anki flash cards, then copied and pasted the flashcards directly into chat GPT voice mode and had it quiz me. That was probably the most miraculous experience of voice interface I've had in a long time - I could do chores while being constantly quizzed on what I wanted to learn, and anytime I had a question or comment I could just ask it to explain or expound on a term or tangent.

Are you thinking perhaps of this [1] PyCon talk by Tavis Rudd, who had RSI and a rotator cuff injury? He demos a voice command system that's all short nonsense syllables, including with a few humorous video clips interspersed, and then explains it after the demo. [1] ▶ this video (demo starts 9 minutes in)

One of my back-burner projects is putting some esp32s or similar around the house with microphones, streaming whatever audio they hear to Dragon NaturallySpeaking like in this video ▶ this video , but hooked up to home assistant scenes / automations instead of emacs commands.

Xah first wrote this over a decade ago. He’s had some RSI issues, and the real problem is simply typing in any form. I remember reading several of his articles thinking that surely within a decade, we’d have better voice assist for programming. Tavis Rudd’s demo looked promising in 2013: ▶ this video I’m not sure what’s the best solution. Here are a few that I known of: link link link

I'm surprised that there seem to be no examples anywhere of anyone seriously using chorded typing for programming. After cheap chorded keyboards entered the market and the associated software became free, I assumed it would only be a matter of time before something like Travis Rudd's incredible demo of voice coding [1] came out. Perhaps that it hasn't speaks to how difficult it is to get real speed improvements with a chorded setup? [1] ▶ this video

Maybe lisp could shine here? While writing lisp code in classic editors, you already shouldn't be editing or navigating code with classic commands, instead you will be using structured editing. Of course it isn't limited to lisp, but languages with C-like syntax are harder to use for structured editing. I highly recommend watching this video, where the speaker uses voice commands to write both in emacs-lisp and in python. ▶ this video

I wonder if sub-vocal will happen anytime soon, and if it does, I am still not sure how it will solve voice being sub-optimal for some types of text. One example is plain old programming, and needs a fairly elaborate setup. The best example I've been able to find was Tavis Rudd's presentation [1], which seems to be a decent solution. I have tried to more or less replicate this setup, but found it less satisfactory. Aside from it demanding near-perfect audio quality to work remotely well in general (as well as a heavy time investment, initial and ongoing), you frequently run into edge cases where it just doesn't want to understand something you're saying. After the 5th or 6th mis-transcription of some specific phrase, you tend to just give up and reach for the keyboard, RSI be damned. (And this as a native English-speaker, with a relatively neutral accent.) I can definitely imagine better fleshed out solutions to either voice recognition for programming, or programming languages whose syntax lends themselves to easier and more productive voice recognition. I could see some definite productivity enhancements if it were achieved. [1] ▶ this video

Tavis Rudd's Using Python to Code by Voice [0] is also interesting. Looking up that URL, I came across the more recent Coding by Voice with Dragonfly [1]. Haven't seen that one yet, but just quickly skimmed through. Unfortunately the live demos are silent (you can't hear speaker Boudewijn Aasman talk to the computer), but it seems worth checking out if you're interested in setting up some recent voice recognition software with your own custom voice command syntax. [0] ▶ this video [1] link

Your comment reminded me of an interesting video I saw a while back. The video is a talk [0] by a programmer explaining and demonstrating how he developed and used a voice recognition system to program in emacs, after a significant RSI injury. The commands and syntax he uses to interact with the system might be a decent approximation of the custom English syntax you suggest. [0]: ▶ this video

Tavis Rudd uses voice recognition to do his editing (in Emacs, no less): ▶ this video The key is that he basically made up his own language to handle some things like special characters.

First, it doesn't feel unpleasant as I do it. There are a few reasons why it might look that way. I have very little practice at this point, especially considering I'm still changing my grammars regularly, so the editing video there is more a technical demo of the software's performance in spite of my own awkwardness while learning. That was my first and only take. I'm basically learning to type again, with cache misses as I try to remember commands and keep track of my position (like trying to remember where keys are, or the next note in a song as you pluck it out). It's also possible the most accurate form of voice input does not sound like natural language. Finally, it's a different mental exercise to copy text that is already written versus writing and editing my own code, so I'm triply out of my element when recording these early videos. Compare to Tavis Rudd's video from 2013, where he pauses between every command: ▶ this video I've recorded my own version of his demo today for comparison: link

IMHO I don't think it would fly. The most difficult things of programming are too abstract for that. You may be interested in the guy that did all his coding with voice for a while because he couldn't use his hands, and still uses it to this day about half of the time. ▶ this video

See this: ▶ this video > In a fast-paced live demo, I will create a small system using Python, plus a few other languages for good measure, and deploy it without touching the keyboard. The demo gods will make a scheduled appearance. I hope to convince you that voice recognition is no longer a crutch for the disabled or limited to plain prose. It's now a highly effective tool that could benefit all programmers. Turns out it's doable, you just need to invent your own language.

I have been suffering from RSI in both arms for a decade now. It started after coding 10+ hour days working at a game company. The pain ebbs and flows with usage, but the inflammation and tightness has never disappeared completely. Over the years, I've met a number of people who also also suffer from RSI in various forms. Fortunately, the majority of people I talk to suffer for some period of time, but do eventually recover, and are able to continue on with their lives having to be carful to avoid future flareups. A handful though, including myself, seem to fall into the "for life" category. I've tried everything you can think of to varying results: * 15+ doctors in various specialties * Physical therapy * Massage therapists * Chiropractors * Acupuncture * Most every ergonomic keyboard and pointing device on the market * Various desk/chair/keyboard tray ergonomic adjustments * Addressing TMS (See books by Dr. Sarno) * Taking a year off of work (multiple times) * NSAIDs This is resulting in me spending a lot of time looking into other career potabilities, but I keep ending up back writing code to some extent. Voice coding is literally the first thing that comes to everyone's mind when you tell them you have arm pain, but it's a dead end in my mind. I tried to follow in the footsteps of Tavis Rudd, but I just ended up with voice strain from talking too much. You might have better luck though: ▶ this video At one point in my life, I had an intern who knew how to code type for me. No

There was an interesting talk at PyCon 2013 about programming using only your voice: ▶ this video Might be worth looking into for your case. It's worth mentioning that the guy did this for himself due to severe RSI in the hands, but there was another talk on using a stenographic keyboard to program and I remember this guy stood up and said he wished he'd known about it earlier as it's another approach for people who suffer from RSI since there's less typing.

[This talk by Tavis Rudd]( ▶ this video ) about a system he used when his hands were afflicted with RSI is pretty interesting. Not entirely a no-hands approach, but I can't seem to remember the developer who made his first app in the hospital by typing with one/two fingers. He wrote a blogpost about his process that was really inspiring.

I also have hand problems. I use speech recognition to type everything, including code, and a game controller to move around the mouse. The controller I'm currently using, a PS4 controller, has a trackpad in the middle, so I was able to mash that against my face and give her technique a shot. It works better than you'd think. The PS4 trackpad isn't exactly brilliant, but I can move around the mouse and click on what I want to with some accuracy. Of course, the trackpad on the controller is very small and not very accurate, so it's not really practical for artwork or anything. But, with a better, larger trackpad, I can imagine this technique actually working. I might give it a shot at some point. I am a bit worried about the inevitable neck and nose pain, though. I wish she had gone into a little more detail about how she avoids that. Maybe she just has a neck of steel? For the curious, these are some other resources I've found about people working around RSI. Most of these are about using Dragon NaturallySpeaking to code by voice, since that's what I'm most interested in, but I think it's still interesting. There really needs to be a list somewhere for open-source workarounds to disabilities. To the best of my knowledge, there really isn't one. Natlink + Dragon NaturallySpeaking: (NatLink, which lets you make custom speech commands for Dragon in Python, is currently being developed at link , but that site's pretty incomprehensible. The original author's site is at link .

You might be interested in watching this talk: ▶ this video Although, if I recall correctly, he uses Dragon Naturally Speaking, which is a commercial product.

> I'd bet that, unless you had an optimized "dialect" for coding or a language designed to be efficiently spoken, a keyboard will beat a human speaking code every time. More or less what this guy did: ▶ this video It seems to me that languages with a more regular syntax are going to have a dramatic advantage here.

A similar project using Emacs and really clever use of Python and Dragon Naturally Speaking SDK was my favorite presentation at PyCon this past year. ▶ this video