And I mention this in the blog, but I'm really not sure how bad this actually is. I have no concept for how much it costs (per token) for each of these services (or if they even charge per-token). I imagine it's significantly more than not hooking it into an LLM.

It seems unnecessary to me that Substack would ever need their customer support bot to process 4 paragraphs of text, and yet it does. Which makes it incredibly easy to exploit.

AT&T seemed to have solved most of the issues by turning it into a slightly better search but then for some reason they still wanted to keep generating an answer instead of tying the answer to one of their pre-selected questions. Which I cannot understand whatsoever.

And for some reason there's an entire industry (at least 3 different companies that I stumbled upon but likely many more?) who's main purpose seems to be creating a widget that is a wrapper for their API that is a wrapper for OpenAI or Gemini's API? Surely, that is either not profitable or will not be profitable long term right?

Anyways, all the services mentioned in this thread, and many more, have been put together in a basic python library that lets you interface with any of them anywhere. Probably, to be safe, I recommend only using this behind a VPN:

https://github.com/TomCasavant/openllms

And also the Maubot plugin for matrix:

https://github.com/TomCasavant/openllms-maubot