Skip to content
Merged

v0.4 #131

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
138 commits
Select commit Hold shift + click to select a range
6a73f9d
refactor: internal API changes to be easily used as indicators for sp…
D4Vinci Jan 7, 2026
9eb5f68
test: update tests accordingly
D4Vinci Jan 7, 2026
6553d90
ops: update workflows to run for this branch
D4Vinci Jan 7, 2026
08f3e6e
fix: add missing line
D4Vinci Jan 7, 2026
3a360ac
build: pump version up temporary
D4Vinci Jan 7, 2026
f80e882
refactor: internal API changes
D4Vinci Jan 7, 2026
0634f57
feat(spiders): Add follow function to the response
D4Vinci Jan 10, 2026
8e36962
ops: add local Claude files to gitignore
D4Vinci Jan 10, 2026
c5bc82d
fix(browsers): solve a bug with setting referer on the request level
D4Vinci Jan 11, 2026
aa4817f
feat: Change logger to be flexible enough for the spiders classes
D4Vinci Jan 11, 2026
1443a46
fix: handle a bug with response follow referer flow argument
D4Vinci Jan 11, 2026
dd5c30c
feat(response): Option to set the referer as the previous response url.
D4Vinci Jan 11, 2026
059a708
feat(spiders system): a prototype of the new spiders system
D4Vinci Jan 11, 2026
32f4f2a
build: Add AnyIO to the dependencies
D4Vinci Jan 11, 2026
d5b9ed1
fix(browsers): Close pages that give error
D4Vinci Jan 11, 2026
0bfdab7
feat: spiders use uvloop/winloop when available
D4Vinci Jan 11, 2026
774364b
style(spiders): make the usage of uvloop optional
D4Vinci Jan 11, 2026
736873a
feat(spiders): Add logic to detect blocked requests and retry them
D4Vinci Jan 11, 2026
5a11c24
fix: make `retry_blocked_request` take the response as well
D4Vinci Jan 11, 2026
a17bb1d
feat(browsers): Add option to retry tabs that gives errors
D4Vinci Jan 11, 2026
72c3c27
feat(spiders): add simple export system for the results
D4Vinci Jan 11, 2026
f2c6216
style: remove unwanted import from spiders main init file
D4Vinci Jan 11, 2026
927cfd8
feat(spiders): The option to start sessions lazily
D4Vinci Jan 12, 2026
e0f94cf
feat(spiders): Add a stream mode to collect items in realtime
D4Vinci Jan 12, 2026
3466bca
feat: add hook to run after collecting an item
D4Vinci Jan 12, 2026
35d5701
feat(spiders): Add option to logging format and date format
D4Vinci Jan 12, 2026
dac854a
fix(parser): handle responses with empty body
D4Vinci Jan 13, 2026
41c7a7e
fix(spiders engine): Improve logging
D4Vinci Jan 17, 2026
afe6b2e
feat(spiders): Change items hook to make it for processing items + ad…
D4Vinci Jan 17, 2026
02c9dff
style(spiders): make parse function abstract
D4Vinci Jan 17, 2026
bf5aa02
fix(browsers): solving an issues with leaving playwright loop open wh…
D4Vinci Jan 18, 2026
acdb399
ops: update bandit checklist
D4Vinci Jan 19, 2026
1f0f86e
feat(spiders): Add pause/resume system for crawls
D4Vinci Jan 19, 2026
0d003f6
ops: update bandit checklist
D4Vinci Jan 19, 2026
d247b5b
test: add tests for the spiders system
D4Vinci Jan 19, 2026
cf0de7e
Merge branch 'main' into v4
D4Vinci Jan 19, 2026
4ecf8a6
fix(spiders): Fix for when the pausing system has large queue
D4Vinci Jan 19, 2026
fbc8be5
fix(spiders): fix a bug with session stats counter
D4Vinci Jan 19, 2026
25850db
fix(spiders): handle errors with manual pause during stream mode
D4Vinci Jan 19, 2026
94bc759
test: updates tests to include new logic
D4Vinci Jan 19, 2026
6947066
feat(spiders): Make graceful shutdown always enabled
D4Vinci Jan 19, 2026
0ec78e2
test: update tests accordingly
D4Vinci Jan 19, 2026
95955af
build: Add w3lib to deps
D4Vinci Jan 20, 2026
d0f52c3
test: remove the tests for the removed html_utils module
D4Vinci Jan 20, 2026
5aa178a
style: update post requests type hints to be more accurate
D4Vinci Jan 20, 2026
122923c
feat(spiders/requests): Use a reliable method for fingerprint generation
D4Vinci Jan 20, 2026
d90189e
test: update tests accordingly
D4Vinci Jan 20, 2026
5dbbd84
fix(requests): return the cached fingerprint on second call
D4Vinci Jan 20, 2026
90c52c4
feat(parser): replacing `tldextract` with `tld` library
D4Vinci Jan 22, 2026
0c3d359
fix(browsers): disable autoplay for all browsers
D4Vinci Jan 28, 2026
65f1a23
fix(browsers): Improve stealth and speed by adjusting flags
D4Vinci Jan 28, 2026
ef8c5bc
feat(spiders/fetchers): Adding proxy rotation logic and change retry …
D4Vinci Feb 1, 2026
6e0277f
test: add new tests for the proxy rotation logic
D4Vinci Feb 1, 2026
f885027
feat(proxy control): Force a proxy at request level at any given point
D4Vinci Feb 2, 2026
aa7a95f
feat(parser)!: Make all selection return selector objects by default
D4Vinci Feb 6, 2026
9acc819
test: update the tests accordingly
D4Vinci Feb 6, 2026
05f62fb
fix(response): Force the body of the response to always be bytes
D4Vinci Feb 6, 2026
d00c34e
style(parser): Improve the type hint for `find_by_text` and `find_by_…
D4Vinci Feb 6, 2026
66476d4
feat(browsers): Add option to block requests to specific domains
D4Vinci Feb 6, 2026
5f557a7
test: add tests for the new feature
D4Vinci Feb 6, 2026
5ec9294
style: Fix all mypy errors and add type hints to untyped function bodies
D4Vinci Feb 7, 2026
e777342
build: pump up version and deps
D4Vinci Feb 7, 2026
ec7c7f8
Merge branch 'main' into v4
D4Vinci Feb 7, 2026
16852cb
style: Use shorter and more accurate naming for constants
D4Vinci Feb 7, 2026
b1ce0b4
ops: Add mypy and pyright to the code quality workflow
D4Vinci Feb 7, 2026
3eaa7ff
ops: exclude site directory from git
D4Vinci Feb 8, 2026
219db79
docs: update logo and media
D4Vinci Feb 8, 2026
0e2e9fb
docs: moving readme files to be outside of the website build
D4Vinci Feb 8, 2026
db41a6c
docs: update pages with the new media for testing zensical
D4Vinci Feb 8, 2026
2ae11d2
docs: full switch from mkdocs to zensical
D4Vinci Feb 8, 2026
09cd5e6
docs(readme): update translation links
D4Vinci Feb 8, 2026
12b731b
docs: move readme files to the root directory
D4Vinci Feb 8, 2026
9a10816
docs: move translated README files to docs folder again
D4Vinci Feb 8, 2026
8d82d3c
docs: update readme file
D4Vinci Feb 8, 2026
ca46070
docs: Improve README.md description formatting
D4Vinci Feb 8, 2026
0c86d70
docs: update the website main page
D4Vinci Feb 8, 2026
1f9eb17
ops: update benchmarks script
D4Vinci Feb 8, 2026
2a4d008
docs: full update for main readme
D4Vinci Feb 8, 2026
3dc4bdc
docs: small adjustments to the readme file
D4Vinci Feb 8, 2026
e688093
docs: update the translated readme files with the new content
D4Vinci Feb 8, 2026
8ed71e4
V4 (#130)
D4Vinci Feb 8, 2026
dc071ee
docs: possible fix for RTD zensical build
D4Vinci Feb 8, 2026
9b46cb3
docs: possible fix for RTD zensical build
D4Vinci Feb 8, 2026
33eb2ff
docs: possible fix for RTD zensical build
D4Vinci Feb 8, 2026
8505c4a
docs: possible fix for RTD zensical build
D4Vinci Feb 8, 2026
a5abf1f
docs: updating overview page and some corrections
D4Vinci Feb 10, 2026
989fdca
docs: updating benchmarks page
D4Vinci Feb 10, 2026
44e2c42
docs: updating selection page and some corrections
D4Vinci Feb 10, 2026
11bd4e4
docs: updating `main classes` page and some corrections
D4Vinci Feb 10, 2026
87b99a4
docs: updating `adaptive` page and some corrections
D4Vinci Feb 10, 2026
f25fe64
docs: updating `fetcher choosing` page and some corrections
D4Vinci Feb 10, 2026
8df74dc
docs: updating `http requests` page and some corrections
D4Vinci Feb 10, 2026
4e3829f
style: expose `ProxyRotator` class to the public API
D4Vinci Feb 10, 2026
8b1704a
docs: use better import for `ProxyRotator` class
D4Vinci Feb 10, 2026
dba3e9d
docs: updating `dynamic websites` page and some corrections
D4Vinci Feb 10, 2026
f350800
docs: updating `Dynamic websites with protection` page and some corre…
D4Vinci Feb 10, 2026
2dbd402
docs: update links to use the current branch
D4Vinci Feb 10, 2026
d3ee9a6
ops: remove branch v4 from workflows
D4Vinci Feb 10, 2026
0489923
build: pump up the version status
D4Vinci Feb 10, 2026
d4766a0
build: update the website description
D4Vinci Feb 10, 2026
a72886f
docs: update camoufox instructions for StealthyFetcher
D4Vinci Feb 10, 2026
7fb57d6
docs: updating `Interactive shell` page and some corrections
D4Vinci Feb 10, 2026
8e4b4a2
docs: updating `extract commans` page
D4Vinci Feb 10, 2026
b52d35c
docs: update beautifulsoup migration guide
D4Vinci Feb 10, 2026
a38f5af
docs: fix image in the `interactive shell` page
D4Vinci Feb 10, 2026
6ac40ac
build: adjustments to build file for PyPi
D4Vinci Feb 10, 2026
8902034
docs: improve wording, style, and use more visual ways
D4Vinci Feb 11, 2026
10cd46e
docs: add the first page of the spiders section
D4Vinci Feb 11, 2026
5a59944
docs: update libraries
D4Vinci Feb 12, 2026
d0b2cec
docs: simplify the architecture mermaid
D4Vinci Feb 12, 2026
bbb8c26
docs: Improve spider architecture page
D4Vinci Feb 12, 2026
14143dd
docs: add `getting started` page to the spiders section
D4Vinci Feb 12, 2026
1c2c7c5
docs: use a better looking architecture image instead of mermaid
D4Vinci Feb 12, 2026
a62330d
fix: add typed overloads to Selectors.get() for proper default type i…
D4Vinci Feb 13, 2026
2eea66d
style(spiders): Lower the default number of concurrent requests
D4Vinci Feb 13, 2026
c6a662e
docs(dynamic pages): add an important warning
D4Vinci Feb 13, 2026
6a0c323
docs: adding a new `Pick Your Path` section to the overview page
D4Vinci Feb 13, 2026
d0a5a01
test: update spider check
D4Vinci Feb 14, 2026
e785cf2
fix(parser)!: Optimize parser for repeated operations
D4Vinci Feb 14, 2026
2997d56
test: remove irrelevent test
D4Vinci Feb 14, 2026
a8f701f
docs(spiders): add a new page for `requests & responses`
D4Vinci Feb 14, 2026
dddb179
docs: add new api references
D4Vinci Feb 14, 2026
108e11f
docs: add a new issue template for documentation issues
D4Vinci Feb 14, 2026
c94ebd2
docs(spiders): add a page to explain sessions logic
D4Vinci Feb 14, 2026
0ab5485
fix: use more accurate naming for proxy rotation strategy
D4Vinci Feb 14, 2026
dee0fd0
docs: use more accurate naming for proxy rotation strategy
D4Vinci Feb 14, 2026
77e825e
docs(spider): add a new page for proxy management & blocking
D4Vinci Feb 14, 2026
ba4d333
fix: make browser fetchers type hints always present
D4Vinci Feb 14, 2026
c3374bc
docs: add a note about response body
D4Vinci Feb 14, 2026
7b4523a
docs: update the titles for all files
D4Vinci Feb 15, 2026
7b8fc91
docs: add a page for spiders advanced usage
D4Vinci Feb 15, 2026
1bf6bb8
docs: update the images urls to use main branch
D4Vinci Feb 15, 2026
93bcb16
style: docstrings corrections for accuracy
D4Vinci Feb 15, 2026
b8b9a7f
docs: general correction across all the pages
D4Vinci Feb 15, 2026
02b11c5
fix(spiders): corrections and removal of dead code
D4Vinci Feb 15, 2026
e35ba34
test: add tests for spiders engine
D4Vinci Feb 15, 2026
b19fa5e
fix(checkpoint): correct error message
D4Vinci Feb 15, 2026
95ad284
docs: add an entry for the 0.4 version
D4Vinci Feb 15, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion .bandit.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,4 +6,6 @@ skips:
- B404 # Using subprocess library
- B602 # subprocess call with shell=True identified
- B110 # Try, Except, Pass detected.
- B104 # Possible binding to all interfaces.
- B104 # Possible binding to all interfaces.
- B301 # Pickle and modules that wrap it can be unsafe when used to deserialize untrusted data, possible security issue.
- B108 # Probable insecure usage of temp file/directory.
40 changes: 40 additions & 0 deletions .github/ISSUE_TEMPLATE/04-docs_issue.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
name: Documentation issue
description: Report incorrect, unclear, or missing documentation.
labels: [documentation]
body:
- type: checkboxes
attributes:
label: Have you searched if there an existing issue for this?
description: Please search [existing issues](https://github.com/D4Vinci/Scrapling/labels/documentation).
options:
- label: I have searched the existing issues
required: true

- type: input
attributes:
label: "Page URL"
description: "Link to the documentation page with the issue."
placeholder: "https://scrapling.readthedocs.io/en/latest/..."
validations:
required: true

- type: dropdown
attributes:
label: "Type of issue"
options:
- Incorrect information
- Unclear or confusing
- Missing information
- Typo or formatting
- Broken link
- Other
default: 0
validations:
required: true

- type: textarea
attributes:
label: "Description"
description: "Describe what's wrong and what you expected to find."
validations:
required: true
36 changes: 35 additions & 1 deletion .github/workflows/code-quality.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,9 @@ jobs:
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install bandit[toml] ruff vermin
pip install bandit[toml] ruff vermin mypy pyright
pip install -e ".[all]"
pip install lxml-stubs

- name: Run Bandit (Security Linter)
id: bandit
Expand Down Expand Up @@ -85,6 +87,22 @@ jobs:
vermin -t=3.10- --violations --eval-annotations --no-tips scrapling/
echo "::endgroup::"

- name: Run Mypy (Static Type Checker)
id: mypy
continue-on-error: true
run: |
echo "::group::Mypy - Static Type Checker"
mypy scrapling/
echo "::endgroup::"

- name: Run Pyright (Static Type Checker)
id: pyright
continue-on-error: true
run: |
echo "::group::Pyright - Static Type Checker"
pyright scrapling/
echo "::endgroup::"

- name: Check results and create summary
if: always()
run: |
Expand Down Expand Up @@ -126,6 +144,22 @@ jobs:
all_passed=false
fi

# Check Mypy
if [ "${{ steps.mypy.outcome }}" == "success" ]; then
echo "βœ… **Mypy (Type Checker)**: Passed" >> $GITHUB_STEP_SUMMARY
else
echo "❌ **Mypy (Type Checker)**: Failed" >> $GITHUB_STEP_SUMMARY
all_passed=false
fi

# Check Pyright
if [ "${{ steps.pyright.outcome }}" == "success" ]; then
echo "βœ… **Pyright (Type Checker)**: Passed" >> $GITHUB_STEP_SUMMARY
else
echo "❌ **Pyright (Type Checker)**: Failed" >> $GITHUB_STEP_SUMMARY
all_passed=false
fi

echo "" >> $GITHUB_STEP_SUMMARY

if [ "$all_passed" == "true" ]; then
Expand Down
6 changes: 6 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
site/*

# AI related files
.claude/*
CLAUDE.md

# cached files
__pycache__/
*.py[cod]
Expand Down
26 changes: 11 additions & 15 deletions .readthedocs.yaml
Original file line number Diff line number Diff line change
@@ -1,25 +1,21 @@
# Read the Docs configuration file
# See https://docs.readthedocs.io/en/stable/config-file/v2.html for details
# See https://docs.readthedocs.com/platform/stable/intro/zensical.html for details
# Example: https://github.com/readthedocs/test-builds/tree/zensical

# Required
version: 2

# Set the OS, Python version, and other tools you might need
build:
os: ubuntu-24.04
apt_packages:
- pngquant
tools:
python: "3.13"

# Build documentation with Mkdocs
mkdocs:
configuration: mkdocs.yml

# Optionally, but recommended,
# declare the Python requirements required to build your documentation
# See https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
python:
jobs:
install:
- requirements: docs/requirements.txt

- pip install -r docs/requirements.txt
- pip install ".[all]"
build:
html:
- zensical build
post_build:
- mkdir -p $READTHEDOCS_OUTPUT/html/
- cp --recursive site/* $READTHEDOCS_OUTPUT/html/
Loading
Loading