Caddy Scraper

This is the modular scraper designed for caddy-chatbot.

Installation

With a virtual environment activated, run the following command to install the dependencies:

pip install -r requirements.txt

To run the scraper, execute the following command:

python main.py

Name		Name	Last commit message	Last commit date
Latest commit History 41 Commits
.devcontainer		.devcontainer
.github/workflows		.github/workflows
caddy_scraper		caddy_scraper
infra		infra
scraper_methods		scraper_methods
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
Dockerfile		Dockerfile
LICENSE		LICENSE
__init__.py		__init__.py
core_utils.py		core_utils.py
docker-compose.yaml		docker-compose.yaml
example.env		example.env
example_nb.ipynb		example_nb.ipynb
excluded_domains.json		excluded_domains.json
generate_vectorstore.py		generate_vectorstore.py
main.py		main.py
pyproject.toml		pyproject.toml
readme.md		readme.md
requirements.txt		requirements.txt
scraper_registry.py		scraper_registry.py
usage_demo.ipynb		usage_demo.ipynb