I gave a small LLM access to the BAR forum's knowledge: BARtender

[tribbitt]

It's the summer before my freshman year of college, and I've been learning about neural networks and machine learning. It's mostly pretty dry stuff once you start reading papers, but learning it lets you do some interesting things. At some point, I have to entertain myself. What to do?

I don't have the compute to train much of any model - the tiny, few-hundred-thousand parameter ones I've written already take ages to train on my laptop. Instead, I decided to try a different kind of software.

I scraped BAR for post-2020 threads, broke them into snippets, and put them in a local vector database. I then wrapped an already-trained small LLM (AI chatbot) from Google in a bit of software: when it's asked a question, I locate and feed it relevant snippets from the vector database in a process called retrieval-augmented generation (RAG).

The result is that the very small, bad-at-reefing LLM stops "making things up" for the most part, because it's fed the information relevant to the question before it produces a response. After each response, it provides links to the exact thread the information came from.

This is a personal project, pretty rudimentary stuff and hosted locally (on my computer only, basically). I won't be putting this on the web because 1) I can't pay for the hosting and compute, and 2) this is a toy project, not polished enough to put online. It was originally was entirely CLI-based, and I only threw together a UI with Claude Code at the last minute. I know similar professional services exist, like NotebookLM. But I did think it was pretty fun to build anyway and wanted to share!

 

[grizfyrfyter]

Get yourself an nvidia jetson nano super (~$250). It's an edge ai devce with 8gb of unified memory. It doesn't make a reliable llm but with some 4bit compression you can use some decent open source models out the box.

If you can train a custom model on something bigger, then the jetson is great. I've used them for a handful of things.
The 64gb AGX Orin is sweet. Still waiting to get my hands on a Thor or Spark

 

[grizfyrfyter]

As far as a UI, I like openwebui but my job prefers librechat.

 

[tribbitt]

Get yourself an nvidia jetson nano super (~$250). It's an edge ai devce with 8gb of unified memory. It doesn't make a reliable llm but with some 4bit compression you can use some decent open source models out the box.

If you can train a custom model on something bigger, then the jetson is great. I've used them for a handful of things.
The 64gb AGX Orin is sweet. Still waiting to get my hands on a Thor or Spark

I was considering buying an old V100, 3d printing a cooling shroud and sticking it in my old desktop so I can ssh into it. Even then it's still not enough for training much.

Some of the new GPUs look sweet. But I've never used one and wonder how much the memory bandwidth limits their performance

 

[Coral reefer]

Bar tender, lol

 

[grizfyrfyter]

I was considering buying an old V100, 3d printing a cooling shroud and sticking it in my old desktop so I can ssh into it. Even then it's still not enough for training much.

Some of the new GPUs look sweet. But I've never used one and wonder how much the memory bandwidth limits their performance

The biggest gpu I have a 5090 mobile (24gb vram) and while I can fit an 8B model into a custom vllm built from source container but my text window left over is only about 4k. It's not really a good training tool unless I am going to train something 4B or smaller.

 

[tribbitt]

Bar tender, lol

I thought it a fitting name! We have BARcode too.

The biggest gpu I have a 5090 mobile (24gb vram) and while I can fit an 8B model into a custom vllm built from source container but my text window left over is only about 4k. It's not really a good training tool unless I am going to train something 4B or smaller.

That's some pretty decent specs! I uh, have a Macbook. And a 1070 in a desktop. Not a lot of hardware lol, a 4-8b transformer barely fits in memory too. I started playing with state-space models because they squish context into a fixed-size recurrent state so it doesn't balloon, and inference is pretty fast o(n) instead of o(n^2)

 

[MichaelB]

I understand absolutely zero here. My cousin use to work at intel now Nvidia. He's talked to me for hours about similar tech things. Still remains over my head.

Like walking into a room of brain surgeons. Yes I know what a brain is but umm thats about it.

@tribbitt in any case happy to see you taking initiative here. Wishing you the best in college. Learn something useful, study hard, do your best and you'll succeed. In your down time make something cool for the club

 

[tankguy]

Break this down so the anti tech crowd can understand ( pretty much me ) I have zero tech skills and very little understanding of it

 

[Turkeysammich]

Yeah. Tell the old guys what your talking about. We want to know too.

 

[Thales]

Yeah. Tell the old guys what your talking about. We want to know too.

You feed an Ai a bunch of sources and that AI only pulls from that source. So it is less likely to make stuff up or report bs - given you trust the sources. We have a notebooklm that we fed all the transcripts of reef beef podcasts, so you can as it questions and it will tell you only what was on the podcast, so you can find out what we said about something in particular, and it will give you a link to the episode and where in the episode whatever you asked about was discussed. That one has like 170 sources. I have one for my website, like 177 sources, so it is easy to ask a specific question about coral spawning or cephalopod care or whatever instead of hunting through the entrie site.

A notbooklm of the BAR site will get you quick answers from just the BAR site instead of all reefing on the web. Pretty useful IMO

The rest they are talking about seems to be specificis on modeling and where to keep the models. LOL, maybe you were asking about that and I answered a different version of your queston!

 

[tribbitt]

You feed an Ai a bunch of sources and that AI only pulls from that source. So it is less likely to make stuff up or report bs - given you trust the sources. We have a notebooklm that we fed all the transcripts of reef beef podcasts, so you can as it questions and it will tell you only what was on the podcast, so you can find out what we said about something in particular, and it will give you a link to the episode and where in the episode whatever you asked about was discussed. That one has like 170 sources. I have one for my website, like 177 sources, so it is easy to ask a specific question about coral spawning or cephalopod care or whatever instead of hunting through the entrie site.

A notbooklm of the BAR site will get you quick answers from just the BAR site instead of all reefing on the web. Pretty useful IMO

The rest they are talking about seems to be specificis on modeling and where to keep the models. LOL, maybe you were asking about that and I answered a different version of your queston!

This is basically it! I just did this DIY instead of using a service like NotebookLM.

Small AI chatbot understands English but not reefing specifics. I build a “system” that gives it relevant reefing information whenever needed, vastly improving answers

 

[grizfyrfyter]

Rich gave a good summary of the initial post.
Everything else is under the hood stuff about how complex the models are and how efficiently and accurately they access the information on local devices (instead of cloud based computing to run the models).

 

[grizfyrfyter]

I have only custom trained one model and it was while building a Helldivers 2 (video game) tactical companion app. It worked far better than I ever expected but I am a code moron and vibe code everything so by the time I had something worth having a professional look at, there were already better apps available.

 

[grizfyrfyter]

You don't have to custom train a model to get good results though. You can use a RAG agent pulling from a BD. It's not as efficient or as accurate as a custom model but you don't have to have a big gpu to train them, you can load an easy to build, local, ollama model into openwebui and it will let you upload documents and build a rag agent. This does not pipe into a website or something very well but it's great to have running on my Home Assistant operating system (I also have llm agents that run a daily and weekly tank telemetry data review, hasn't been worth the effort yet but fun).

I am not a fan of using cloud service AI from data centers. I am a fan of using my gaming computers to run 14B & 32B models (at 4bit quantization) and only paying for the electricity.

 

[grizfyrfyter]

IMB has some granite models that will run on your phone. They aren't great but they work.

 

[grizfyrfyter]

I need to train a YOLO model to count clownfish eggs. Realistically, I probably find some code that does it just fine...

 

[PjFish]

I clicked this thread hoping for a cocktails!

 

[BAYMAC]

Rich gave a good summary of the initial post.

He used AI