local llms with ollama + open-webui + litellm configured with some apis
the free Claude 3.5 sonnet is my daily driver now. When I run out of free messages, I switch over to open-webui and have access to various flagship models for a few cents. The local models usually suffice when I'm asking a dumb question
Here's my `shell.nix`:
```
{ pkgs ? import <nixpkgs> {} }:
(pkgs.buildFHSEnv {
name = "simple-fhs-env";
targetPkgs = pkgs: with pkgs; [
tmux
bash
python311
];
runScript = ''
#!/usr/bin/env bash
set -x
set -e
source .venv/bin/activate
tmux new-session -d -s textgen
tmux send-keys -t textgen "open-webui serve" C-m
tmux split-window -v -t textgen
tmux send-keys -t textgen "LITELLM_MASTER_KEY=hunter2 litellm --config litellm.yaml --port 8031" C-m
tmux attach -t textgen
'';
}).env
```
And here's `litellm.yaml`:
```
model_list:
- model_name: codestral
litellm_params:
model: mistral/codestral-latest
api_key: hunter2
- model_name: claude-3.5
litellm_params:
model: anthropic/claude-3-5-sonnet-20240620
api_key: sk-hunter2
- model_name: gemini-pro
litellm_params:
model: gemini/gemini-1.5-pro-latest
api_key: hunter2
safety_settings:
- category: HARM_CATEGORY_HARASSMENT
threshold: BLOCK_NONE
- category: HARM_CATEGORY_HATE_SPEECH
threshold: BLOCK_NONE
- category: HARM_CATEGORY_SEXUALLY_EXPLICIT
threshold: BLOCK_NONE
- category: HARM_CATEGORY_DANGEROUS_CONTENT
threshold: BLOCK_NONE
```
