TechieDamien@lemmy.mltoProgrammer Humor@lemmy.ml•A Containerized Night Out: Docker, Podman, and LXC Walk into a Bar
1·
11 months agoYou can run llms on text-generation-ui such as open llama and gpt2. It is very similar to the stable diffusion web ui.
If I’m being honest, it is fairly slow. It takes a good few seconds to respond on a 6800XT using the medium vram option. But that is the price to pay to running ai locally. Of course, a cluster should drastically improve the speed of the model.