Add support for other LLM provider.

[buildr-daemon]

In whistleblower we are currently only using OPENAI models , extend on it and give the user options to run it with multiple llm provider like anthropic , gemini, full integration is expected here.

[staru09]

Hey is this open to work?

[buildr-daemon]

@staru09 yes!, create a sub-issue and propose a plan of action

[staru09]

Hi @repello-daemon I was thinking of using a unified gateway as litellm to support different LLM providers. It supports openrouter as well so instead of we having to add support for different LLMs, we can let openrouter and litellm handle it. The best part is it supports Ollama as well, so we can use opensource models as well.

The current implementation

os.environ['OPENAI_API_KEY'] = api_key
client = OpenAI()
response = client.chat.completions.create(
    model=model,
    messages=message,
    max_tokens=512,
)
output = response.choices[0].message.content

can be changed with

from core.llm_client import run_llm

output = run_llm(model, message, max_tokens=1024, api_key=api_key)

Litellm also gives us automatic cost tracking, caching and model routing as well.

[buildr-daemon]

@staru09 that's an interesting approach , let's continue with that .

[LifeHashed]

@Aaditya-G we can implement other llm providers like gemini, grok, anthropic, to increase versatility of the application