Oh no, I missed it this year 😨

GM oranges

local llms with ollama + open-webui + litellm configured with some apis the free Claude 3.5 sonnet is my daily driver now. When I run out of free messages, I switch over to open-webui and have access to various flagship models for a few cents. The local models usually suffice when I'm asking a dumb question Here's my `shell.nix`: ``` { pkgs ? import <nixpkgs> {} }: (pkgs.buildFHSEnv { name = "simple-fhs-env"; targetPkgs = pkgs: with pkgs; [ tmux bash python311 ]; runScript = '' #!/usr/bin/env bash set -x set -e source .venv/bin/activate tmux new-session -d -s textgen tmux send-keys -t textgen "open-webui serve" C-m tmux split-window -v -t textgen tmux send-keys -t textgen "LITELLM_MASTER_KEY=hunter2 litellm --config litellm.yaml --port 8031" C-m tmux attach -t textgen ''; }).env ``` And here's `litellm.yaml`: ``` model_list: - model_name: codestral litellm_params: model: mistral/codestral-latest api_key: hunter2 - model_name: claude-3.5 litellm_params: model: anthropic/claude-3-5-sonnet-20240620 api_key: sk-hunter2 - model_name: gemini-pro litellm_params: model: gemini/gemini-1.5-pro-latest api_key: hunter2 safety_settings: - category: HARM_CATEGORY_HARASSMENT threshold: BLOCK_NONE - category: HARM_CATEGORY_HATE_SPEECH threshold: BLOCK_NONE - category: HARM_CATEGORY_SEXUALLY_EXPLICIT threshold: BLOCK_NONE - category: HARM_CATEGORY_DANGEROUS_CONTENT threshold: BLOCK_NONE ```

I had way too much with LivePortrait this morning. Couldn't quite get it to run on Nix (I still suck at nix derivations), but they're still running a free demo space on HF. Also out of China in the last few weeks was ChatTTS and Qwen2, state of the art local text to speech and llm. I've been running it with ollama and open-webui, and it's not half bad compared to 4o or claude, in one prompt it actually gave me a better answer. There's a ComfyUI node for LivePortrait. There's a lot you could do with this stuff. Deepfakes and destroy democracy, sure. But maybe more relevant to my interests is we're nearly upon the toolset needed to build a fully artificial (and local) talking head. if there's a model that can generate facial expressions to match a voice, I think that's basically everything we need is right there. then use ollama to RAG your own documents and legit talk to your home PC face to face. what a time to be alive