CoLLM

CoLLM is a Reflective, Iterative Approach to Collaborative LLM Work Quality and Accuracy Estimation

Prerequisites

• Make sure YOU HAVE miniconda or anaconda ALREADY installed and initialized; e.g.,

# install miniconda (instructions from website)
# You can also use homebrew to install miniconda
mkdir -p ~/miniconda3
curl https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-arm64.sh -o ~/miniconda3/miniconda.sh
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
rm ~/miniconda3/miniconda.sh

# activate conda
source ~/miniconda3/bin/activate

# initialize all available shells
conda init --all

🚀  How To Get Started

In your local .env file, set your OpenAI and Anthropic API keys:

OPENAI_API_KEY="<openai-api-key>"
ANTHROPIC_API_KEY="<anthropic-api-key>"

Install local LLM models (assumes Ollama is installed):

# install local LLM models (assumes ollama is installed)
› ollama pull qwen2.5:32b
› ollama pull llama3.1:70b
› ollama pull mixtral:8x22b
› ollama pull firefunction-v2
› ollama pull codestral # mostly for code-related

Git clone the project and create a conda environment:

# clone project
› git clone https://github.com/semantiQs/collm
› cd collm

# create conda environment
› ./scripts/conda_create_env.sh
› conda activate collm

🏃   Run a few basic commands to get started:

# 1. run CoLLM's help
› python -m collm.main help
Welcome to CoLLM!
usage: collm [options]... <command> [options]... [arg]...

Type 'help' for a list of commands.

The most commonly used commands are:
 deal with Prompt procedures:

 python -m collm.main procedure ls
 or python -m collm.main procedure cat <proc>
 or python -m collm.main procedure edit <proc>

 deal with CoLLM requests:
 
 python -m collm.main request submit [options]... <query>
 or python -m collm.main request execute [options]... <proc>
 or python -m collm.main request load [options]... <workload>

 curate data with CoLLM:
 
 python -m collm.main data curation [options]... <filepath>

CoLLM is a reflective, iterative approach for 
performing Multi-LLM work, along with 
estimating the quality and accuracy of 
this Multi-LLM work.

# 2. List available Coral procedures
› python -m collm.main procedure ls
Available Prompt procedures:
1. transform.coral
2. revise.coral
3. reviewer.coral
4. focus.coral
5. generator.coral
6. sec.coral
7. reflect.coral
8. fix-json.coral

Let's submit a basic request to Collm:

› python -m collm.main request submit "What is the meaning of life?"
Welcome to CoLLM!

Submitted user request: what is the meaning ... (trial: Trial_622)
Generating answers 100% 3/3 0:00:31 0:00:31.494034
Reviewing answers  100% 3/3 0:00:08 0:00:08.772418
Output saved to trial_622.txt
Goodbye!

And the results are saved in trial_622.txt:

Table 1: LLM-generated Answers.
                                      
| # | ID     | Model       | Quality |
|---|--------|-------------|---------|
| 1 | Item_2 | gpt-4       | 8.596   |
| 2 | Item_1 | qwen2.5:32b | 8.388   |
| 3 | Item_3 | gpt-4o-mini | 7.016   |
                                      
Note: gpt-4o-mini failed to generate an answer.

Answers sorted by overall quality     
(descending). The overall quality     
score of an answer is the 'Empirical  
Bayes Estimator' (EBE) of its grades. 
                                      
                              
Table 2: LLMs Grading Variance.
                              
| # | Model       | Variance |
|---|-------------|----------|
| 1 | qwen2.5:32b | 0.011    |
| 2 | gpt-4o-mini | 0.149    |
| 3 | gpt-4       | 0.534    |
                              
LLMs sorted by their variance 
(ascending). This variance    
indicates LLMs' task          
precision. The lower the      
variance, the better          
(improving LLM's task         
accuracy).                    
                              
Table 3: LLMs Grading Accuracy.
                              
| # | Model       | Accuracy |
|---|-------------|----------|
| 1 | qwen2.5:32b | 0.851    |
| 2 | gpt-4o-mini | 0.448    |
| 3 | gpt-4       | 0.000    |
                              
This score measures each LLM's
task accuracy, by comparing   
the generated answers and the 
judgments from an LLM with the
answers and judgments from    
other LLMs, giving more more  
weight to the input of LLMs   
with higher measured accuracy.
                              

# Show only top 1 answer
'Trial_622' Results:
1. Item_2:
The philosophical question regarding the purpose or significance of life or existence in general does not have one definitive answer, as it can vary greatly depending on different cultural, religious, philosophical, or personal beliefs. Some propose that the meaning of life is individually self-determined, while others assert that it is determined by a higher power.

How To Install It (as a library) 👇

We assume you have python and pip installed and you have already cloned the collm repository.

#in collm/ 
# Install an editable version of CoLLM
› python -m pip install --user -e .

And that is it! You can now use the collm command line tool, directly from your terminal.

NOTE: By default, CoLLM's a default LLM configuration for the models listed above. As long as it is an Anthropic model, or an OpenAI model, or a local model supported by Ollama, you can add your own models to a local .collm.toml file in same directory as python script.

The syntax of .collm.toml files supported by CoLLM is similar to this one:

title = "My local  LLMs"

[llms]
[llms.claude-3-5-sonnet]
model = "claude-3-5-sonnet-latest"
temperature = 0.5
mode = "remote"
enabled = true

By default, CoLLM doesn't override the settings of existing models. This means that if you have a model with the same name in your local .collm.toml file, CoLLM will skip any details you specify in your .collm.toml. Future versions of CoLLM will support overriding existing models.

Additionally, if you installed the collm library using --user (installed on a user-specific Python directory, not the system-wide one), you may need to add the following to your .bash_profile or .zshrc file to make sure the collm command is available in your terminal:

export PATH="$HOME/.local/bin:$PATH"

Note: --user will install collm into a user-specific location (typically ~/.local on Linux/macOS, or %APPDATA%\Python on Windows). In this case, the actual binaries/commands go into:

# Linux/macOS
~/.local/bin/
# Windows
%APPDATA%\Python\Scripts\

🤝 Contributions

We have a set of guidelines for contributing to collm. These are guidelines, not rules. Use your best judgment, and feel free to propose changes to this document in a pull request.