Building Custom AI assistant - how to get started?

[rsoika]

Question

Hi,
I want to develop my first AI Agent. The agent should provide a default system prompt and than passes the answer to a small Java Programm that opens a resource file (an XML document) and modifies this resource file.
I found this general documentation about the topic: https://theia-ide.org/docs/theia_ai/

But I did not understand how to get started. Is there a kind of hello-world skeleton available that I can use to start? I do not understand how my project structure should look like, how to build the thing (I guess it is an extension .vsix?) and how to integrate this.
Thanks for any hints.
Ralph

[github-actions[bot]]

Hi @rsoika, thanks for starting this discussion! 👋

The Theia community will take a look soon. In the meantime, you might find helpful information in:

• 📖 Theia Documentation
• ❓ FAQ
• 💬 Previous Discussions

---

💙 Eclipse Theia is built and maintained by a community of contributors and sponsors. If Theia is valuable to your work, consider sponsoring the project. For professional support, training, or consulting services, learn more about available options.

[planger]

Hi Ralph,

If you want to build upon Theia AI (including its agent framework, chat UI, etc.), you need to implement it as a Theia extension.
To create a Theia application with a custom Theia extension. See the Theia Extension docs for more details.

Creating a custom agent in this extension is then simply binding a dedicated chat agent implementation, as described in the Theia AI docs. There are several more examples in the Theia code base to look at, see e.g. the ai-ide module as well as the API examples about certain AI chat features.

I hope this helps!

[rsoika]

Hi Philip,
all of this stuff puts me off a little :-( I don’t want to build a completely new IDE right away... ;-)

Now I wonder if the following is possible:

Can I define a new agent template and somehow tell it that

1. the agent should produce a specific output format
2. save the result as a text file in my test folder
3. call my existing JUnit test 'ValidateAndGenerateModel.java'

My test class is generating a BPMN file form the text input. So my agent would automate what I am doing manually now.

Is something like this possible?

I do not really understand how to describe the prompttemplate in a way to get such a result

1. User asks a question
2. Agent calls LLM to generates a structured file
3. Agent copies the result into my workspace
4. Agent starts an existing junit test

[planger]

Hi Ralph,

alternatively, you can also create a custom agent in your workspace (see Creating a Custom Agent), if you don't want to provide a Theia Extension yet.

In a custom agent, you can define the agent name, the LLM to be used, give it a system message. In the system message you can specify the tool functions you want to make available for the agent using ~{<tool-id>} (see also doc on tool functions). With Theia's workspace functions and the file change set functions, you should be able to describe the workflow you want, with the only prerequisite that you define a task in the tasks.json for executing the ValidateAndGenerateModel.java first. Otherwise the agent cannot invoke such commands.

Relevant tool functions to be included and explained in your custom agent's system message:

• getFileContent
• writeFileContent
• getWorkspaceFileList and getWorkspaceDirectoryStructure if you need workspace exploration
• listTasks (from tasks.json)
• runTask (from tasks.json)

See Theia Coder prompt for an example of a more complex prompt that involves tool functions.

[chongzixuan-ai]

Hello @rsoika
How are you doing?
I am a senior full stack ai engineer focused on full stack ai development.
I saw your post message and I am very interested about your project.
So I provide method follow:
For your AI agent, you can start with the Theia IDE extension development template. A good starting point would be creating an extension using Theia's generator, which will give you a basic structure for building extensions.
Let me know if you need further details!