Teaching Our Trip-Ticket Bot to Listen: Six Walls We Hit Along the Way
I have a Telegram bot that creates trip tickets for our field vehicles. It already works: tap a destination, a purpose, a date, a vehicle, a time, a driver, the passengers, then Confirm. That is about ten small prompts for something I would rather say in one breath: "Trip to Antipolo tomorrow at nine, Hiace, driver is Jose." So, with Claude Code as my pair programmer, I set out to let the bot listen. It took a surprising number of small walls. Here is the honest version, in the order we hit them.
Wall one: a bot cannot call you
My first idea was a bot that phones me and interviews me until the ticket is complete. Telegram does not allow it. Bots cannot place or answer calls, because call setup belongs to the client side of the protocol. The only route is to automate a real user account, which risks a ban and means feeding live call audio through speech-to-text and text-to-speech on my own. We settled for voice notes in and voice notes out. It is turn-by-turn instead of live, but it works inside the app everyone already uses.
The plan, and the parts we already had
A voice note arrives. A Whisper speech-to-text service I already run at home turns it into text in about three to four seconds. An n8n workflow asks Claude to pull the trip fields out of that text, which takes four to twenty-five seconds. The bot then checks everything and asks for what is missing, one question at a time, until it can show the usual Confirm and Cancel summary. Nothing is submitted until I tap Confirm.
Wall two: Whisper hears sounds, not names
We tested with synthetic speech first, and the recognizer behaved exactly as I should have expected. A town came back misspelled. Hiace became "the highest", then "Hias". Mang Jose became "Meng Jose". A mountain site turned into something that looked like a license plate.
The fix was not a better recognizer. The workflow hands the model the real lists of sites, vehicles and people and tells it to match by sound. Then the bot refuses to trust the answer: any id that is not in those lists is thrown away, and if two people could both be "Jose" it does not pick one, it shows two buttons.
Wall three: the pipeline understood me perfectly and was still wrong
My first real recording was eight seconds long: "Create a trip ticket to Antipolo today at 9 am." I made it at about ten-forty at night. Every word came through correctly, and the result was a trip for a time that had already passed. The model cannot know what time it is unless you tell it, and even then you should not rely on it. So the bot now checks dates against Manila time in plain code, drops a past date or time with a message, and also drops a vehicle or driver that is already booked on that day, instead of letting the API reject the ticket at the very end.
The same test showed a smaller oddity: Whisper labelled my English clip as Tagalog with under fifty percent confidence. The words were right, so the bot ignores the label. Changing the settings of a service that other bots depend on was not worth the risk, so I left it alone.
Wall four: giving the bot a voice
I wanted the bot to answer by voice too. The first candidate was a self-hosted Tagalog model, which sounded plausible until we read its license: non-commercial use only, a poor fit for a company tool. We dropped it and listened to six English voices from Microsoft Edge instead, and I picked one by ear. It is an unofficial endpoint and the reply text goes to Microsoft, so every spoken reply also carries the text and the buttons, and if the voice fails the bot just sends text.
Getting there had its own small detours. On my Mac, Python could not verify certificates until it was pointed at a certificate bundle. A different open-source voice engine failed on missing data files when we tried it. Neither mattered once the final package was tested where it would actually run: the homelab.
Wall five: the tools fought back, in small ways
- The Claude step. n8n reaches Claude through a locked-down SSH key whose script reads the prompt from the SSH command itself, so calling that script by hand fails with an unbound variable. To test prompts directly we ran the Claude command line with the same flags the script uses. Its output also arrives with a warning line and code fences around the JSON, so the parser strips both.
- A hostile voice note. The transcript ends up inside a prompt, so we tried one that said to ignore all previous instructions. The model ignored it and even reported the attempt, but I do not rely on that. The real defence is the code that validates every field afterwards.
- CI failed on the first push. We had added a package to the bot but not to the lock file at the root of the workspace, and the install step is strict about it. Regenerating the lock file fixed it, and we checked with the same dry run CI uses.
Wall six: the safety rails
The bot's settings live in a file the tooling will not let Claude touch, because it is a secrets file. That is the right rule, so we did not look for a way around it. Claude prepared the new settings in a private file and the real file stayed in human hands. The restart needs a sudo password that Claude does not have either. When it came time to run those two commands, it turned out they had effectively already been done: the timestamps showed the settings applied and the bot restarted before my paste, which is why my last attempt reported a missing file.
While we were deploying, my browser also declared that my n8n site was unreachable. It was not down. From the Mac and from the homelab it answered normally, and the cause was IPv6 on my own machine, which failed for every site. It cleared up by itself. I mention it because it looks exactly like you broke everything.
Did it work?
I tried it on Telegram and it seems to be working: I sent a voice note, and the bot picked up the conversation and asked for the rest. I know where it is still weak. The speech service is reached through an internal address that could change, the voice comes from an endpoint nobody promised me would stay, and the Telegram conversation itself has no automated test. The pure logic has sixteen unit tests, and everything else was tested by talking to it.
Twenty years in broadcast taught me never to trust a signal path just because it passed audio once. You monitor every hop and plan for the one that drops out. A voice interface is the same chain with more links, and what made this one work was not the clever part. It was the plain checks after each hop.