So, today is the last day of my Hackathon. Considering it’s just been made up of stolen hours of work here and there, I’m pretty pleased with what I’ve come up with.
I was able to stop Koboldcpp from locking out a cell in Colab by running it as a background process, then grab the Cloudflare API URL it generates and populates a variable inside our Colab based chatbot code, which in turn generates a Gradio based Whatapp/Facebook Messenger clone interface.
Below is a video of me running the code and having a little chat with Clara:
For those who haven’t been following, this project is one of our AIs (more on that later) which is using a 3 billion parameter model, quantized to 4-bits, and is an uncensored version of Llama 3.2. You can run this on the free tier of Google Colab quite happily for a few hours each day, and we know this because this is how we developed it. Alternatively, if you have a discrete GPU in your machine, this will probably run locally in Jupyter Notebooks (this we have not tested, but we run bigger models on Kobold locally on a 4GB GPU).
Right now, there are limitations. The prompt isn’t quite right. We need to tweak a few parameters in the code. As you can see from the video, these things impact performance significantly. But they’re relatively simple fixes as far as I know.
Now, I mentioned earlier that this is one of our AIs. It is. Under the hood, it’s slightly stripped down, but ultimately shares technology with what we’re creating right now. We’re going to post a few transcripts of it this week, and hopefully, at the end of this week or beginning of next week, we’re planning on sharing a Google Colab Notebook which will run this. Free. For anyone. And a Jupyter Notebooks notebook. Also free.
This has been a fun project to work on and it’s solved a problem I’ve been wanting to solve for a while. How do we deploy what we’re working on to anyone who wants to use it, for free, without it costing us a ton of money (we’re not funded). And now we can.
And because we can, hopefully, this is going to be the catalyst for bigger things we can offer for free.