Issue with Token Calculation for Open Source Models & Container Creation Error

[Hassn11q]

I am encountering two distinct issues while using the Pentagi framework:

1. Warning for Token Calculation with Open Source Models:

Error Message:

[WARN] Failed to calculate number of tokens for model, falling back to approximate count.

Details:
This warning appears when using open-source models such as QWQ and DeepSeek. The Pentagi framework is defaulting to an approximate token count instead of calculating the exact number of tokens.

---

2. Container Creation Failure:

Error Message:

createFlow failed to create flow worker: failed to prepare flow resources: failed to spawn container 'pentagi-terminal-26': failed to create container: Error response from daemon: invalid reference format: repository name.

Details:
This happens when I try to create a flow worker, and the error is related to an invalid reference format for the container name.

---

Thank you for your attention to these issues. I look forward to your assistance in resolving them.

[mazamizo21]

Can you please elaborate on how do you use deepseek API, also I am looking to integrate pentagon with an open source penetration testing AI called whiterabbitneo and a company called kinda provide a free api : curl https://llm.kindo.ai/v1/chat/completions
-H "api-key: <KINDO_API_KEY>"
-H "content-type: application/json"
-d '{
"model": "<MODEL_NAME>",
"messages": [{"role": "user", "content": "Hello, world!"}]
}'

[Hassn11q]

Hi @mazamizo21, I’m running the DeepSeek model locally using vLLM. Here’s how I’ve configured my .env file:

Custom LLM provider

LLM_SERVER_URL=http://host.docker.internal:8000/v1
LLM_SERVER_KEY="your api key"
LLM_SERVER_MODEL=deepseek-ai/DeepSeek-R1-Distill-Qwen-32B

Using WhiteRabbitNeo:

If you’re looking to integrate WhiteRabbitNeo, you can configure it similarly with the following setup:

LLM_SERVER_URL=https://llm.kindo.ai/v1/chat/completions
LLM_SERVER_KEY="api key"
LLM_SERVER_MODEL=model_name

[asdek]

Hello @Hassn11q

Thank you for your report. Let me address each issue:

Warning for Token Calculation with Open Source Models

It seems this warning originates from the file langchaingo/llms/count_tokens.go. The issue arises because the function in tiktoken/encoding.go does not recognize your model from list for encoding, leading to a fallback to an approximate count.

I will think about a more reliable solution to this problem and get back to you. A possible interim solution could be adding a new parameter LLM_SERVER_MODEL_ENCODING=, which allows users to specify the correct model for token calculation or an approximate ratio of the number of characters to one token.

Another option could be to look for libraries with broader support for various models, including open-source ones.

Container Creation Failure

Based on the error message you've encountered, it appears that there was an incorrect Docker image name selection. I can enhance the logging by including the image name and ensure that this error is reported directly in the PentAGI interface, eliminating the need to check the log file. A similar issue and explanation can be found here.

The selection of the Docker image is governed by this prompt. If you want PentAGI to select a specific image for your task, you can specify it explicitly in the job description. This way, it won't guess, but instead will use the image from your instructions. It seems that the model might have generated an incorrect path for the Docker image, which led to this error.

I appreciate your patience and will strive to address these issues promptly.

[asdek]

Hello @mazamizo21

Regarding your question about using deepseek models via API, currently, our custom LLM provider supports only OpenAI-compatible APIs. For local execution, the vLLM backend, as suggested by Hassn11q, might be suitable. However, when utilizing such APIs, please ensure, based on the provider's documentation or that of the local framework and specific model, that it supports tool calling and correctly responds to utility invocation requests. Unfortunately, based on my testing experience, not all open-source models consistently adhere to the specified output restrictions, such as JSON Schema format.

In the early development stages, I had code that allowed calling a single function within a completion, generating a JSON object. However, I quickly abandoned this approach as managing the structure of the JSON became too cumbersome, and predicting what the model intended to convey was challenging. This was initially done to support the ollama backend, which lacks explicit tool invocation functionality. In the future, I plan to reintegrate this code to better accommodate open-source models that don't support tool calling.

Additionally, here are some observed issues with using open-source models:

1. The model occasionally gets caught in a loop, repetitively attempting the same action, leading to a hard limit being triggered in the code, which results in the subtask's termination.
2. When reaching a certain context size, the model stops calling functions and switches to completion mode, and even the reflector functionality doesn't solve this. Eventually, the hard limit on reflector invocations is reached, causing task termination.
3. When forming a JSON object as a utility call argument, open-source models often make type errors, requiring backend-side corrections such as implicit conversion of strings to numbers or lists of strings, or parsing strings back into their original data structure from the JSON object. If this conversion fails, the function is not called with partially filled parameters, resulting in an error message which returning back to the model. The issue is that this doesn't correct the problem, and the subsequent response from model is identical, with the code again reaching a hard limit under these conditions.

In summary, while open-source models are progressing, they currently fall short of replacing OpenAI and Anthropic models with the same level of quality, particularly for tasks involving frequent utility calls during interactions.

[Hassn11q]

Hello @asdek,

Thank you for your response. I tried specifying the image explicitly, but I am still encountering the following error:

Docker error: invalid reference format: repository name must be lowercase , failed to spawn container 'pentagi-terminal-4': failed to create container: Error response from daemon: invalid reference format

To ensure correctness, I updated the prompt to enforce the exact format:

# SECURITY IMAGE SELECTION PROTOCOL  
**You MUST respond with EXACTLY this format:**  
booyaabes/kali-linux-full:latest  

**Rules:**  
1. **Never** explain your reasoning  
2. **Never** use markdown/XML tags  
3. **Always** use lowercase  
4. **Only** output this exact string: booyaabes/kali-linux-full:latest  

**Examples of VALID responses:**  
booyaabes/kali-linux-full:latest  

**Examples of INVALID responses:**  
<think>...booyaabes/kali...  
Kali-Linux-Full  
security-tools:latest  

**Task:**  
{{.Input}}  

**Required Image:** 

However, despite this change, the error persists. Do you have any insights into why this might still be happening? Could there be an issue with how the response is being parsed or passed to Docker?

Looking forward to your thoughts.

[asdek]

Hello @Hassn11q

Thanks for feedback.

From the error message you're seeing, it seems like there might be a problem with how the Docker image name is being parsed or passed to Docker. Even though you've updated the prompt to enforce the exact format, sometimes hidden characters or formatting quirks can slip in, especially when dealing with AI-generated text.

To get a clearer picture, could you check the logs for the line that contains "spawning container"? This line should show the exact image name that PentAGI is attempting to use. Specifically, it includes the "image" variable, which reflects the Docker image name. This might help pinpoint whether the image name is being altered or if there are any unexpected characters causing the invalid reference format error.

You can see where this logging happens in the code here:
PentAGI docker client

Additionally, I've made some updates that introduce a new environment variable: DOCKER_DEFAULT_IMAGE. By setting this in your .env file, you can specify a default Docker image that PentAGI will use whenever it encounters an error running or pulling the image specified for a task. This might help as a workaround while we troubleshoot the issue. Just add something like this to your .env:

DOCKER_DEFAULT_IMAGE=booyaabes/kali-linux-full:latest

Regarding local models, it's possible they might not always generate the output in the exact format we need, which could be contributing to the problem. To rule out any issues with the backend or parsing, you might try using a cloud LLM provider like DeepInfra or OpenRouter. I've had reasonable success with the deepseek/deepseek-chat model on OpenRouter—it's not perfect, but it tends to produce more consistent results.

Please let me know if checking the logs or setting the DOCKER_DEFAULT_IMAGE helps. I'm happy to continue digging into this with you to find a solution. 😉

[Hassn11q]

Hi @asdek, thanks for your feedback!

I was able to successfully create the container for booyaabes/kali-linux-full:latest after setting DOCKER_DEFAULT_IMAGE=booyaabes/kali-linux-full:latest. However, I'm unable to complete the task due to the following error:

2025-02-02 16:04:01 pentagi     | time="2025-02-02T13:04:01Z" level=error msg="failed to call agent chain: max retries reached, 3" agent=custom error="API returned unexpected status code: 400: " flow_id=11 msg_chain_id=113 subtask_id="<nil>" task_id=0xc0002d6f68
2025-02-02 16:04:01 pentagi     | time="2025-02-02T13:04:01Z" level=error msg="failed to perform subtasks generator" error="failed to get subtasks generator result: failed to call agent chain: max retries reached, 3: API returned unexpected status code: 400: "
2025-02-02 16:04:01 pentagi     | time="2025-02-02T13:04:01Z" level=error msg="failed to process input" component=worker error="failed to create task for flow 11: failed to create task worker: failed to generate subtasks: failed to generate subtasks for task 8: failed to perform subtasks generator: failed to get subtasks generator result: failed to call agent chain: max retries reached, 3: API returned unexpected status code: 400: " flow_id=11 input="execute sql injection attack on this target \"https://flask-sql.onrender.com\"\n\n" trace_id=89edebaa-9923-407c-a5c7-6343063279bc user_id=1

I've also tried using DeepInfra and OpenRouter, but the same error persists across both.

Would you happen to have any insights on what might be causing the "API returned unexpected status code: 400" error? Could this be related to how the task input is being processed or an issue with the agent chain itself?

Any guidance would be greatly appreciated! Thanks again

[asdek]

Hello @Hassn11q

It sounds like the issue may be tied to the tool calling functionality in your chosen model. When interacting with the LLM backend, it's crucial to ensure that this backend fully complies with the OpenAI specification, especially regarding tokens for retrieving available functions. It should be capable of generating JSON-formatted responses accordingly.

A pivotal aspect is the handling of options used in the completion API request. You can find more details here: Custom Provider Code. Given that the response code 400 is returned, it's likely that one or more specified parameters aren't supported by the LLM backend. Though I haven't encountered this specific error before, removing these extended options might resolve the request successfully.

To simplify experimenting with model parameters, I've externalized the settings into a configuration file: .custom.provider.yml. You can map this configuration file into the container as a volume from your host file system and reference it via the LLM_SERVER_CONFIG_PATH environment variable. The full list of supported configuration options is available here: Config Options. I have tested this configuration with the OpenRouter provider.

I hope this guidance is helpful! Let me know if you have any further questions or need additional assistance.