Skip to content

Commit 9fe79cc

Browse files
jira import and confluence metadata (#13)
* remove the duplicate * prepend the confluence base url if we are iterating over multiple pages of a space in the fetchSpacePages method * added metadata for page as frontmatter * add additional metadata to confluence pages * added Jira Import functionality * fix wrong parameter * added JIRA_ADAPTER.md * moved JIRA_ADAPTER.md --------- Co-authored-by: Phil <91220442+CastAIPhil@users.noreply.github.com>
1 parent eaff46c commit 9fe79cc

10 files changed

Lines changed: 1177 additions & 35 deletions

File tree

README.md

Lines changed: 54 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -306,6 +306,46 @@ local_folders:
306306
knowledge_id: "local-knowledge-base"
307307
```
308308

309+
## Jira Adapter
310+
311+
The Jira adapter syncs Jira issues from specified projects to OpenWebUI knowledge bases.
312+
313+
### Jira Configuration
314+
315+
Map different Jira projects to different knowledge bases:
316+
317+
```yaml
318+
jira:
319+
enabled: true
320+
base_url: "https://your-domain.atlassian.net"
321+
username: "your-email@example.com"
322+
api_key: "" # Set via JIRA_API_KEY environment variable
323+
project_mappings:
324+
- project_key: "PROJ"
325+
knowledge_id: "project-knowledge-base"
326+
- project_key: "ANOTHER"
327+
knowledge_id: "another-knowledge-base"
328+
```
329+
330+
### Jira Features
331+
332+
- **Project-based Sync**: Sync all issues from specified Jira projects
333+
- **Multiple Knowledge Bases**: Map different projects to different knowledge bases
334+
- **JSON Export**: Each issue is returned as a JSON file
335+
- **Content Hashing**: Only syncs changed issues based on SHA256 hashes
336+
- **File Naming**: Issues are saved as `{issue-key}.json`
337+
338+
### Jira Example Output
339+
340+
```
341+
INFO[0000] Syncing files from adapter: jira
342+
DEBU[0000] Fetching files from Jira project: PROJ
343+
DEBU[0000] Found 25 issues in Jira project PROJ
344+
INFO[0001] Successfully synced file: PROJ-123.json
345+
INFO[0001] Successfully synced file: PROJ-124.json
346+
INFO[0001] Successfully synced file: PROJ-125.json
347+
```
348+
309349
## Configuration
310350
311351
### Environment Variables
@@ -318,6 +358,7 @@ local_folders:
318358
- `CONFLUENCE_BASE_URL`: Confluence instance URL (optional, can be set in config)
319359
- `CONFLUENCE_USERNAME`: Confluence username (optional, can be set in config)
320360
- `CONFLUENCE_KNOWLEDGE_ID`: OpenWebUI knowledge ID for Confluence files
361+
- `JIRA_API_KEY`: Jira API key
321362
- `STORAGE_PATH`: Local storage path (default: /data)
322363
- `LOG_LEVEL`: Log level (debug, info, warn, error)
323364
@@ -358,6 +399,19 @@ confluence:
358399
knowledge_id: "" # Set via CONFLUENCE_KNOWLEDGE_ID environment variable
359400
page_limit: 100 # Maximum pages to fetch per space (0 = no limit)
360401
include_attachments: true # Whether to download and sync page attachments
402+
403+
# Jira adapter configuration
404+
jira:
405+
enabled: false
406+
base_url: "https://your-domain.atlassian.net"
407+
username: "your-email@example.com"
408+
page_limit: 100 # Maximum pages to fetch per space (default = 100)
409+
api_key: "" # Set via JIRA_API_KEY environment variable
410+
project_mappings:
411+
- project_key: "PROJ"
412+
knowledge_id: "your-knowledge-base-id"
413+
- project_key: "ANOTHER"
414+
knowledge_id: "another-knowledge-base-id"
361415
```
362416

363417
## Adapter Architecture

adapter_readme/CONFLUENCE_ADAPTER.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,7 @@ The Confluence adapter uses Basic Authentication with your Confluence username a
110110
| `include_attachments` | boolean | No | `true` | Whether to download and sync page attachments |
111111
| `include_blog_posts` | boolean | No | `false` | Whether to download and sync blog posts |
112112
| `use_markdown_parser` | boolean | No | `false` | Whether to use markdown parser for HTML content conversion (true = markdown, false = plain text) |
113+
| `add_additional_data` | boolean | No | `false` | Whether to fetch additional user data (display names) for pages and blog posts |
113114

114115
## File Processing
115116

adapter_readme/JIRA_ADAPTER.md

Lines changed: 192 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,192 @@
1+
# Jira Adapter
2+
3+
The Jira adapter allows you to sync content from Atlassian Jira projects into OpenWebUI knowledge bases. This adapter uses the Jira REST API to fetch issues and comments from specified Jira projects and uploads them to OpenWebUI.
4+
5+
## API Compatibility
6+
7+
This adapter uses Jira REST API v3, which provides:
8+
- Modern cursor-based pagination
9+
- Improved performance and reliability
10+
- Better support for large projects
11+
- Enhanced metadata and content structure
12+
13+
## Features
14+
15+
- **Issue Content Sync**: Fetches all issues from specified Jira projects using Jira API v3
16+
- **HTML to Markdown Conversion**: Converts Jira's HTML content to markdown format
17+
- **Comment Support**: downloads and syncs issue comments
18+
- **Multi-Project Support**: Can sync from multiple Jira projects
19+
- **Cursor-based Pagination**: Uses modern cursor-based pagination for efficient data retrieval
20+
21+
## Configuration
22+
23+
### YAML Configuration
24+
25+
Add the following to your `config.yaml`:
26+
27+
```yaml
28+
jira:
29+
enabled: true
30+
base_url: "https://your-domain.atlassian.net"
31+
username: "your-email@example.com"
32+
api_key: "your-jira-api-key"
33+
project_mappings:
34+
- project_key: "PROJ"
35+
knowledge_id: "your-knowledge-base-id"
36+
- project_key: "ANOTHER"
37+
knowledge_id: "another-knowledge-base-id"
38+
page_limit: 100
39+
```
40+
41+
### Environment Variables
42+
43+
Only the API key can be configured via environment variable (for security):
44+
45+
```bash
46+
JIRA_API_KEY="your-jira-api-key"
47+
```
48+
49+
All other configuration should be done in the `config.yaml` file.
50+
51+
### Kubernetes Configuration
52+
53+
#### ConfigMap
54+
55+
```yaml
56+
apiVersion: v1
57+
kind: ConfigMap
58+
metadata:
59+
name: connector-config
60+
data:
61+
config.yaml: |
62+
jira:
63+
enabled: true
64+
base_url: "https://your-domain.atlassian.net"
65+
username: "your-email@example.com"
66+
project_mappings:
67+
- project_key: "PROJ"
68+
knowledge_id: "your-knowledge-base-id"
69+
- project_key: "ANOTHER"
70+
knowledge_id: "another-knowledge-base-id"
71+
page_limit: 100
72+
```
73+
74+
## Authentication
75+
76+
The Jira adapter uses Basic Authentication with your Jira username and API key. To get an API key:
77+
78+
1. Go to [Atlassian Account Settings](https://id.atlassian.com/manage-profile/security/api-tokens)
79+
2. Click "Create API token"
80+
3. Give it a label and copy the generated token
81+
4. Use your email address as the username and the token as the API key
82+
83+
## Configuration Parameters
84+
85+
| Parameter | Type | Required | Default | Description |
86+
|-----------|------|----------|---------|-------------|
87+
| `enabled` | boolean | No | `false` | Enable the Jira adapter |
88+
| `base_url` | string | Yes | - | Your Jira instance URL (e.g., `https://your-domain.atlassian.net`) |
89+
| `username` | string | Yes | - | Your Jira username (usually your email) |
90+
| `api_key` | string | Yes | - | Your Jira API key |
91+
| `project_mappings` | array | Yes | - | List of Jira project keys and their corresponding OpenWebUI knowledge base IDs |
92+
| `page_limit` | integer | No | `100` | Maximum number of issues to fetch per project |
93+
94+
## File Processing
95+
96+
### Issue Content
97+
98+
- Jira issues are converted from HTML to markdown format
99+
- Issues are saved as `.md` files with sanitized filenames
100+
- File paths follow the pattern: `{issue-id}.md`
101+
102+
### Issue Metadata
103+
104+
Each issue file includes:
105+
- Issue key
106+
- Reporter name
107+
- Issue type
108+
- Status
109+
- Resolution status
110+
111+
### Comments
112+
113+
- Comments are fetched and included in the markdown file
114+
- Each comment includes the author's display name and timestamp
115+
- Comments are formatted in markdown
116+
117+
## Error Handling
118+
119+
- **Authentication Errors**: Invalid credentials will cause the adapter to fail initialization
120+
- **API Errors**: HTTP errors from Jira API are logged and may cause individual issue processing to fail
121+
- **File Processing Errors**: Individual file processing errors are logged but don't stop the overall sync
122+
- **Network Errors**: Connection timeouts and network issues are handled gracefully
123+
124+
## Limitations
125+
126+
1. **API Rate Limits**: Jira has API rate limits that may affect sync performance
127+
2. **Large Projects**: Very large projects with many issues may take significant time to sync
128+
3. **HTML Conversion**: The HTML to markdown conversion is basic and may not preserve all formatting
129+
4. **Comment Limitations**: Comments are limited to the Jira API's available fields
130+
131+
## Troubleshooting
132+
133+
### Common Issues
134+
135+
1. **Authentication Failed**
136+
- Verify your username and API key are correct
137+
- Ensure your API key has the necessary permissions
138+
139+
2. **Project Not Found**
140+
- Check that the project key is correct
141+
- Verify you have access to the project
142+
143+
3. **No Content Synced**
144+
- Check that the project contains issues
145+
- Verify the `page_limit` setting is appropriate
146+
- Check logs for API errors
147+
148+
### Debug Mode
149+
150+
Enable debug logging to see detailed information about the sync process:
151+
152+
```yaml
153+
log_level: debug
154+
```
155+
156+
## Example Usage
157+
158+
### Basic Configuration
159+
160+
```yaml
161+
jira:
162+
enabled: true
163+
base_url: "https://mycompany.atlassian.net"
164+
username: "john.doe@mycompany.com"
165+
api_key: "ATATT3xFfGF0..."
166+
project_mappings:
167+
- project_key: "DOCS"
168+
knowledge_id: "fbc18bc4-72c1-40f0-84b1-52055368c583"
169+
- project_key: "PROJ"
170+
knowledge_id: "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
171+
page_limit: 100
172+
```
173+
174+
### Advanced Configuration
175+
176+
```yaml
177+
jira:
178+
enabled: true
179+
base_url: "https://mycompany.atlassian.net"
180+
username: "john.doe@mycompany.com"
181+
api_key: "ATATT3xFfGF0..."
182+
project_mappings:
183+
- project_key: "DOCS"
184+
knowledge_id: "fbc18bc4-72c1-40f0-84b1-52055368c583"
185+
- project_key: "PROJ"
186+
knowledge_id: "a1b2c3d4-e5f6-7890-abcd-ef1234567890"
187+
- project_key: "OPS"
188+
knowledge_id: "98765432-10fe-dcba-0987-6543210fedcb"
189+
page_limit: 500
190+
```
191+
192+
This configuration will sync up to 500 issues from each of the three specified projects.

config.example.yaml

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -98,6 +98,19 @@ slack:
9898
message_limit: 1000 # Max messages per channel per run (default: 1000)
9999
include_threads: true # Whether to include thread messages (default: true)
100100
include_reactions: false # Whether to include reaction data (default: false)
101+
# Jira adapter configuration
102+
jira:
103+
enabled: false
104+
base_url: "https://your-domain.atlassian.net" # Your Jira instance URL
105+
username: "your-email@example.com" # Your Jira username (usually email)
106+
api_key: "" # Set via JIRA_API_KEY environment variable
107+
page_limit: 100 # Maximum pages to fetch per space (0 = no limit)
108+
109+
project_mappings:
110+
- project_key: "PROJ"
111+
knowledge_id: "your-knowledge-base-id"
112+
- project_key: "ANOTHER"
113+
knowledge_id: "another-knowledge-base-id"
101114

102115
# Example configurations for different environments:
103116

go.sum

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,10 @@ github.com/robfig/cron/v3 v3.0.1/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzG
3535
github.com/rogpeppe/go-internal v1.9.0/go.mod h1:WtVeX8xhTBvf0smdhujwtBcq4Qrzq/fJaraNFVN+nFs=
3636
github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ=
3737
github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog=
38+
github.com/sebdah/goldie/v2 v2.7.1 h1:PkBHymaYdtvEkZV7TmyqKxdmn5/Vcj+8TpATWZjnG5E=
39+
github.com/sebdah/goldie/v2 v2.7.1/go.mod h1:oZ9fp0+se1eapSRjfYbsV/0Hqhbuu3bJVvKI/NNtssI=
40+
github.com/sergi/go-diff v1.4.0 h1:n/SP9D5ad1fORl+llWyN+D6qoUETXNZARKjyY2/KVCw=
41+
github.com/sergi/go-diff v1.4.0/go.mod h1:A0bzQcvG0E7Rwjx0REVgAGH58e96+X0MeOfepqsbeW4=
3842
github.com/sirupsen/logrus v1.9.3 h1:dueUQJ1C2q9oE3F7wvmSGAaVtTmUizReu6fjN8uqzbQ=
3943
github.com/sirupsen/logrus v1.9.3/go.mod h1:naHLuLoDiP4jHNo9R0sCBMtWGeIprob74mVsIT4qYEQ=
4044
github.com/slack-go/slack v0.17.3 h1:zV5qO3Q+WJAQ/XwbGfNFrRMaJ5T/naqaonyPV/1TP4g=
@@ -44,18 +48,16 @@ github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/
4448
github.com/stretchr/testify v1.8.4 h1:CcVxjf3Q8PM0mHUKJCdn+eZZtm5yQwehR5yeSVQQcUk=
4549
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
4650
github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
51+
github.com/yuin/goldmark v1.7.13 h1:GPddIs617DnBLFFVJFgpo1aBfe/4xcvMc3SB5t/D0pA=
52+
github.com/yuin/goldmark v1.7.13/go.mod h1:ip/1k0VRfGynBgxOz0yCqHrbZXhcjxyuS66Brc7iBKg=
4753
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
4854
golang.org/x/net v0.0.0-20190603091049-60506f45cf65/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
49-
golang.org/x/net v0.19.0 h1:zTwKpTd2XuCqf8huc7Fo2iSy+4RHPd10s4KzeTnVr1c=
50-
golang.org/x/net v0.19.0/go.mod h1:CfAk/cbD4CthTvqiEl8NpboMuiuOYsAr/7NOjZJtv1U=
5155
golang.org/x/net v0.43.0 h1:lat02VYK2j4aLzMzecihNvTlJNQUq316m2Mr9rnM6YE=
5256
golang.org/x/net v0.43.0/go.mod h1:vhO1fvI4dGsIjh73sWfUVjj3N7CA9WkKJNQm2svM6Jg=
5357
golang.org/x/oauth2 v0.15.0 h1:s8pnnxNVzjWyrvYdFUQq5llS1PX2zhPXmccZv99h7uQ=
5458
golang.org/x/oauth2 v0.15.0/go.mod h1:q48ptWNTY5XWf+JNten23lcvHpLJ0ZSxF5ttTHKVCAM=
5559
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
5660
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
57-
golang.org/x/sys v0.15.0 h1:h48lPFYpsTvQJZF4EKyI4aLHaev3CxivZmv7yZig9pc=
58-
golang.org/x/sys v0.15.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
5961
golang.org/x/sys v0.35.0 h1:vz1N37gP5bs89s7He8XuIYXpyY0+QlsKmzipCbUtyxI=
6062
golang.org/x/sys v0.35.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
6163
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=

0 commit comments

Comments
 (0)