Skip to content

Commit a6682e6

Browse files
oshezafoshezaf
committed
ASIM updater: update empty tables tool
1 parent 75fe9f9 commit a6682e6

File tree

15 files changed

+933
-64
lines changed

15 files changed

+933
-64
lines changed

‎.github/workflows/asim-schema-updater.yml‎

Lines changed: 95 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,95 @@
1+
name: ASIM Schema Updater
2+
3+
on:
4+
# Run weekly on Sundays at midnight UTC
5+
schedule:
6+
- cron: '0 0 * * 0'
7+
8+
# Run on push when relevant files change
9+
push:
10+
branches:
11+
- master
12+
- main
13+
paths:
14+
- 'ASIM/dev/ASimTester/ASimTester.csv'
15+
- 'Parsers/ASim*/Parsers/vim*Empty.yaml'
16+
- 'ASIM/tools/ASIM Updater/**'
17+
18+
# Run on pull requests for validation
19+
pull_request:
20+
branches:
21+
- master
22+
- main
23+
paths:
24+
- 'ASIM/dev/ASimTester/ASimTester.csv'
25+
- 'Parsers/ASim*/Parsers/vim*Empty.yaml'
26+
- 'ASIM/tools/ASIM Updater/**'
27+
28+
# Allow manual trigger
29+
workflow_dispatch:
30+
31+
jobs:
32+
update-empty-parsers:
33+
runs-on: ubuntu-latest
34+
35+
steps:
36+
- name: Checkout repository
37+
uses: actions/checkout@v4
38+
with:
39+
fetch-depth: 0
40+
41+
- name: Set up Python
42+
uses: actions/setup-python@v5
43+
with:
44+
python-version: '3.11'
45+
46+
- name: Run empty parser updater (dry-run for PRs)
47+
if: github.event_name == 'pull_request'
48+
run: |
49+
python "ASIM/tools/ASIM Updater/update_empty_parsers.py" --dry-run --verbose
50+
continue-on-error: false
51+
52+
- name: Run empty parser updater
53+
if: github.event_name != 'pull_request'
54+
run: |
55+
python "ASIM/tools/ASIM Updater/update_empty_parsers.py" --verbose
56+
57+
- name: Check for changes
58+
if: github.event_name != 'pull_request'
59+
id: check-changes
60+
run: |
61+
if git diff --quiet; then
62+
echo "has_changes=false" >> $GITHUB_OUTPUT
63+
else
64+
echo "has_changes=true" >> $GITHUB_OUTPUT
65+
git diff --stat
66+
fi
67+
68+
- name: Create Pull Request
69+
if: github.event_name != 'pull_request' && steps.check-changes.outputs.has_changes == 'true'
70+
uses: peter-evans/create-pull-request@v6
71+
with:
72+
token: ${{ secrets.GITHUB_TOKEN }}
73+
commit-message: 'chore: Update ASIM empty parsers to match schema'
74+
title: '[Automated] Update ASIM empty parsers'
75+
body: |
76+
This PR was automatically generated by the ASIM Schema Updater workflow.
77+
78+
## Changes
79+
80+
Updated empty parser files to match the field definitions in `ASIM/dev/ASimTester/ASimTester.csv`.
81+
82+
## Review Checklist
83+
84+
- [ ] Verify added fields are correct for each schema
85+
- [ ] Verify removed fields should no longer be in the schema
86+
- [ ] Verify type changes are correct
87+
88+
---
89+
*This PR was automatically created by the [ASIM Schema Updater](.github/workflows/asim-schema-updater.yml) workflow.*
90+
branch: automated/asim-empty-parser-update
91+
delete-branch: true
92+
labels: |
93+
automated
94+
asim
95+
schema-update

‎ASIM/dev/Forked Repos PRs/README.MD‎

Lines changed: 0 additions & 1 deletion
This file was deleted.

‎ASIM/tools/ASIM Updater/README.md‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# ASIM Schema Updater Tools
2+
3+
This directory contains tools to maintain ASIM (Advanced Security Information Model) schema consistency across the Azure-Sentinel repository.
4+
5+
## Tools
6+
7+
### update_empty_parsers.py
8+
9+
Updates empty parser YAML files (`vim*Empty.yaml`) to match the field definitions in `ASIM/dev/ASimTester/ASimTester.csv`.
10+
11+
**Key Features:**
12+
- **Surgical edits**: Instead of regenerating entire datatables, makes minimal changes (add, remove, modify) for easier code review
13+
- **Preserves formatting**: Maintains the existing code style and comments
14+
- **Schema-aware**: Handles schema inheritance (Common fields apply to all schemas)
15+
- **Dry-run support**: Preview changes before applying them
16+
17+
**Usage:**
18+
19+
```bash
20+
# Preview changes (dry-run)
21+
python update_empty_parsers.py --dry-run
22+
23+
# Apply changes to all empty parsers
24+
python update_empty_parsers.py
25+
26+
# Update a specific parser
27+
python update_empty_parsers.py --parser "/path/to/vimAuditEventEmpty.yaml"
28+
29+
# Verbose output
30+
python update_empty_parsers.py --verbose
31+
```
32+
33+
**Options:**
34+
- `--repo-root`: Root directory of the Azure-Sentinel repository (auto-detected if not specified)
35+
- `--csv-path`: Path to ASimTester.csv (auto-detected if not specified)
36+
- `--dry-run`: Show what would be changed without making changes
37+
- `--parser`: Update only the specified parser file
38+
- `--verbose`, `-v`: Show detailed output including files with no changes
39+
40+
## GitHub Workflow
41+
42+
The tools are automatically run via GitHub Actions:
43+
- **Scheduled**: Weekly on Sundays at midnight UTC
44+
- **On push**: When relevant files are changed (CSV schema or empty parsers)
45+
46+
See `.github/workflows/asim-schema-updater.yml` for the workflow configuration.
47+
48+
## Schema Source
49+
50+
The canonical schema definitions are in `ASIM/dev/ASimTester/ASimTester.csv`, which contains:
51+
- `ColumnName`: Field name
52+
- `ColumnType`: Data type (string, int, datetime, etc.)
53+
- `Class`: Field class (Mandatory, Recommended, Optional, Conditional, Alias)
54+
- `Schema`: Schema name (AuditEvent, NetworkSession, Dns, etc.)
55+
- `LogicalType`: Logical type hint
56+
- `ListOfValues`: Allowed values for enumerated fields
57+
- `Aliased`: Field this is an alias for (if Class is Alias)
58+
59+
## Empty Parsers
60+
61+
Empty parsers are located at:
62+
```
63+
Parsers/ASim*/Parsers/vim*Empty.yaml
64+
```
65+
66+
They define the schema structure using an empty KQL datatable and are used for:
67+
- Schema validation
68+
- Union operations in parsers
69+
- Documentation generation

0 commit comments

Comments
 (0)