Skip to content

Commit ee5d683

Browse files
rshipprshipp
authored
Merge pull request #60 from InQuest/issue/17-docs
Documentation and queue workers
2 parents 2b82b40 + b16723c commit ee5d683

78 files changed

Lines changed: 2741 additions & 1296 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@
66
*.log
77
*.egg-info/
88
*.sqlite3
9+
*.db
910
config.yml
1011
test*.yml
1112

‎.travis.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ language: python
22
python:
33
- "3.6"
44
install:
5+
- "pip install -r requirements.txt"
56
- "pip install -r requirements-testing.txt"
67
- "pip install coveralls"
78
- "pip install codacy-coverage"

‎MANIFEST.in‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
include LICENSE
2+
include README.rst
3+
include requirements.txt
4+
include requirements-testing.txt

‎README.rst‎

Lines changed: 67 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,5 @@
1-
.. raw:: html
2-
3-
<p align="center">
4-
<img height="128" src="https://inquest.readthedocs.io/projects/threatingestor/en/latest/_static/threatingestor.png" alt="threatingestor" title="threatingestor">
5-
</p>
6-
7-
<h1 align="center">ThreatIngestor</h1>
1+
ThreatIngestor
2+
==============
83

94
.. image:: https://inquest.net/images/inquest-badge.svg
105
:target: https://inquest.net/
@@ -25,47 +20,93 @@
2520
:target: https://pypi.python.org/pypi/ThreatIngestor
2621
:alt: PyPi Version
2722

28-
An extendable tool to extract and aggregate IOCs from threat feeds.
23+
An extendable tool to extract and aggregate IOCs_ from threat feeds.
2924

30-
Designed for use with `InQuest ThreatKB`_, but can be used without it.
25+
Integrates out-of-the-box with ThreatKB_ and MISP_, and can fit seamlessly into any existing worflow with SQS_, Beanstalk_, and `custom plugins`_.
3126

3227
Overview
3328
--------
3429

35-
ThreatIngestor can be configured to watch Twitter, RSS feeds, or other
36-
sources, extract meaningful information such as C2 IPs/domains and YARA
37-
signatures, and send that information to another system for analysis.
30+
ThreatIngestor can be configured to watch Twitter, RSS feeds, or other sources, extract meaningful information such as malicious IPs/domains and YARA signatures, and send that information to another system for analysis.
31+
32+
`Try it out!`_
3833

3934
Installation
4035
------------
4136

42-
ThreatIngestor requires Python 3.6+.
37+
ThreatIngestor requires Python 3.6+, with development headers.
38+
39+
Install ThreatIngestor from PyPI::
4340

44-
Install ThreatIngestor and its dependencies::
41+
pip install threatingestor
4542

46-
pip3 install -r requirements.txt
47-
python3 setup.py install
43+
Install optional dependencies for using some plugins, as needed::
44+
45+
pip install threatingestor[all]
46+
47+
View the `full installation instructions`_ for more information.
4848

4949
Usage
5050
-----
5151

52-
Create a new ``config.yml`` file, and configure each source and operator module
53-
you want to use. (See ``config.example.yml`` for layout.) Then run the script::
52+
Create a new ``config.yml`` file, and configure each source and operator module you want to use. (See ``config.example.yml`` for layout.) Then run the script::
5453

5554
threatingestor config.yml
5655

57-
By default, it will run forever, polling each configured source every 15
58-
minutes.
56+
By default, it will run forever, polling each configured source every 15 minutes.
57+
58+
View the `full ThreatIngestor documentation`_ for more information.
59+
60+
Plugins
61+
-------
62+
63+
ThreatIngestor uses a plugin architecture with "source" (input) and "operator" (output) plugins. The currently supported integrations are:
64+
65+
Sources
66+
~~~~~~~
67+
68+
* Beanstalk work queues
69+
* Git repositories
70+
* GitHub repository search
71+
* RSS feeds
72+
* Amazon SQS queues
73+
* Twitter
74+
* Generic web pages
75+
76+
Operators
77+
~~~~~~~~~
78+
79+
* Beanstalk work queues
80+
* CSV files
81+
* MISP
82+
* SQLite database
83+
* Amazon SQS queues
84+
* ThreatKB
85+
* Twitter
86+
87+
View the `full ThreatIngestor documentation`_ for more information on included plugins, and how to create your own.
88+
89+
Support
90+
-------
91+
92+
If you need help getting set up, or run into any issues, feel free to open an Issue_. You can also reach out to `@InQuest`_ on Twitter.
5993

60-
For full documentation, see the `ThreatIngestor ReadTheDocs site`_.
94+
We'd love to hear any feedback you have on ThreatIngestor, its documentation, or how you're putting it to work for you!
6195

6296
Contributing
6397
------------
6498

65-
Issues and pull requests are welcomed. Please keep Python code PEP8 compliant.
66-
By submitting a pull request you agree to release your submissions under the
67-
terms of the LICENSE_.
99+
Issues and pull requests are welcomed. Please keep Python code PEP8 compliant. By submitting a pull request you agree to release your submissions under the terms of the LICENSE_.
68100

69-
.. _InQuest ThreatKB: https://github.com/InQuest/ThreatKB
101+
.. _ThreatKB: https://github.com/InQuest/ThreatKB
70102
.. _LICENSE: https://github.com/InQuest/threat-ingestors/blob/master/LICENSE
71-
.. _ThreatIngestor ReadTheDocs site: https://threatingestor.readthedocs.io/
103+
.. _full ThreatIngestor Documentation: https://threatingestor.readthedocs.io/
104+
.. _SQS: https://aws.amazon.com/sqs/
105+
.. _Beanstalk: https://beanstalkd.github.io/
106+
.. _MISP: https://www.misp-project.org/
107+
.. _custom plugins: https://threatingestor.readthedocs.io/en/latest/developing.html
108+
.. _IOCs: https://en.wikipedia.org/wiki/Indicator_of_compromise
109+
.. _full installation instructions: https://threatingestor.readthedocs.io/en/latest/installation.html
110+
.. _Issue: https://github.com/InQuest/ThreatIngestor/issues
111+
.. _@InQuest: https://twitter.com/InQuest
112+
.. _Try it out!: https://inquest.readthedocs.io/projects/threatingestor/en/latest/welcome.html#try-it-out

‎config.example.yml‎

Lines changed: 27 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
general:
22
daemon: true
33
sleep: 900
4-
state_path: state.sqlite3
4+
state_path: state.db
55

66
credentials:
77
- name: twitter-auth
@@ -27,6 +27,11 @@ credentials:
2727
aws_secret_access_key: MY_SECRET
2828
aws_region: MY_REGION
2929

30+
- name: misp-auth
31+
url: https://mymisp
32+
key: MY_API_KEY
33+
ssl: False
34+
3035
sources:
3136
- name: twitter-inquest-c2-list
3237
module: twitter
@@ -50,19 +55,23 @@ sources:
5055
- name: github-query
5156
module: github
5257
credentials: github-auth
53-
saved_state:
5458
search: CVE-2018-
5559

5660
- name: rss-myiocfeed
5761
module: rss
5862
url: https://example.com/rss.xml
5963
feed_type: messy
6064

65+
- name: beanstalk-input
66+
module: beanstalk
67+
host: 127.0.0.1
68+
port: 11300
69+
queue_name: threatingestor-input
70+
6171
operators:
6272
- name: csv
6373
# Write artifacts to a CSV file
6474
module: csv
65-
artifact_types: [Task]
6675
filename: output.csv
6776

6877
- name: mythreatkb
@@ -72,6 +81,14 @@ operators:
7281
allowed_sources: [twitter-inquest-c2-list, rss-.*]
7382
state: Inbox
7483

84+
- name: beanstalk-output
85+
module: beanstalk
86+
host: 127.0.0.1
87+
port: 11300
88+
queue_name: my-queue-worker-input
89+
artifact_types: [URL]
90+
url: {url}
91+
7592
- name: inquest-threatcrawler
7693
module: sqs
7794
credentials: aws-auth
@@ -84,3 +101,10 @@ operators:
84101
source_type: url
85102
download_path: /var/crawler/ingestor
86103
source_type_tags: dir
104+
105+
- name: sqlite-db
106+
module: sqlite
107+
filename: artifacts.db
108+
109+
- name: misp-instance
110+
module: misp

‎docs/_static/custom.css‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -185,11 +185,16 @@ div.body {
185185
overflow-x: auto;
186186
}
187187

188+
pre {
189+
max-height: 25em;
190+
overflow-y: auto;
191+
}
192+
188193
/* -------------------------------------------------------------------------- */
189194
/* SOCIAL ICONS ------------------------------------------------------------- */
190195
/* -------------------------------------------------------------------------- */
191196
svg.icon {
192197
height: 15px;
193198
width: 15px;
194199
vertical-align: middle;
195-
}
200+

‎docs/_static/inquest-blog-sqlite.yml‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
general:
2+
daemon: false
3+
sleep: 900
4+
state_path: state.db
5+
6+
sources:
7+
- name: inquest-blog
8+
# Read from the InQuest blog.
9+
module: rss
10+
url: http://feeds.feedburner.com/Inquestblog?format=xml
11+
feed_type: messy
12+
13+
operators:
14+
- name: sqlite-artifacts
15+
# Write artifacts to a SQLite database.
16+
module: sqlite
17+
file_path: artifacts.db

‎docs/_static/malwareconfig_twitter.png‎

50.5 KB
Loading

‎docs/_static/mermaid-everything.mermaid‎

Lines changed: 83 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
graph LR
2+
3+
A[Twitter C2 Feed]
4+
B[Twitter Search: OpenDir]
5+
C[Twitter Search: Pastebin]
6+
D[Twitter Search: Masq]
7+
E[Twitter Search: VT]
8+
F[GitHub CVEs]
9+
G[Git YARA Rule Repo]
10+
H[RSS IOC Feed]
11+
I[RSS Vendor X Blog]
12+
J[SQS Input]
13+
K[SQS FSWatcher]
14+
L[Web Masq Feed]
15+
M[Web Attack Feed]
16+
17+
18+
Q(Known-Good Check)
19+
R(ThreatKB)
20+
S(MISP)
21+
T(Pastebin Processor)-->|Paste Contents|J
22+
U(Crawler)
23+
V(OSINT Enrich Domain)
24+
W(OSINT Enrich IP)
25+
X(RepDB Check)
26+
Y(YARA Scan)
27+
Z(VirusTotal Downloader)
28+
29+
1(Queue Worker: Sandbox Analysis)-->|Sandbox Report IOCs|J
30+
2(Queue Worker: VT Analysis)-->|VirusTotal Report IOCs|J
31+
3(Queue Worker: Static Analysis)-->|Extracted IOCs|J
32+
33+
A-->R
34+
A-->S
35+
B-->S
36+
B-->U
37+
C-->T
38+
D-->S
39+
D-->U
40+
E-->Z
41+
F-->|Tasks|R
42+
G-->R
43+
G-->S
44+
G-->Y
45+
H-->R
46+
H-->S
47+
I-->R
48+
I-->S
49+
J-->R
50+
J-->S
51+
K-->R
52+
K-->S
53+
K-->Y
54+
L-->S
55+
L-->U
56+
M-->S
57+
58+
R-->Q
59+
R-->X
60+
R-->V
61+
R-->W
62+
U-->|Files|1
63+
U-->|Files|2
64+
U-->|Files|3
65+
V-->|Extra Context|S
66+
W-->|Extra Context|S
67+
V-->|Extra Context|R
68+
W-->|Extra Context|R
69+
70+
Q-->|Delete False Positives|R
71+
X-->|Delete False Positives|R
72+
73+
Y-->|Files|1
74+
Y-->|Files|2
75+
Y-->|Files|3
76+
77+
Z-->|Files|1
78+
Z-->|Files|2
79+
Z-->|Files|3
80+
81+
1-->|Full Report|S
82+
2-->|Full Report|S
83+
3-->|Full Report|S

‎docs/_static/mermaid-everything.png‎

256 KB
Loading

0 commit comments

Comments
 (0)