So, today we’ve been working on the interface. As we wanted, we’ve now got a lovely little WhatApp style messenger interface working on a Colab instance.
It works very nicely, and we’re very happy with it. Even has the little bouncing dots as the AI is “typing” the message. The backend that deals with the API has been repurposed from code I wrote before, and if you want to be salty about that, go ahead. I really don’t get the argument of “What happens at a Hackathon must only happen at a Hackathon!” Get over yourselves. And if you’re going to get salty over code reuse then I got two words for ya – effing libraries.
Anyway, there is a slight problem with our Koboldcpp as the backend server, and our messenger as the frontend interface. That is, when we run Koboldcpp, it doesn’t release the cell. And we’re doing all of this using free resources. Which means we’re having to use a second account to run the interface in another instance of Colab. Hardly ideal. Especially given as we’re using Python, and Python doesn’t love multi-threaded processes (yes, we know, we’re getting to that bit!)
But we have a plan! You can nudge Python into using multi-threaded processes via libraries, and we’re going to try and figure out a way to run Koboldcpp as a a multi-threaded process, freeing up the cell, then letting the user run a second cell containing our lovely little interface. Or, we might try some fancy trickery to see if we can make the whole thing one easy to use cell.