Skip to content

Commit 177421d

Browse files
Enhance documentation for scrape_test and sbr_test scripts
Clarified descriptions and added details for the scrape_test and sbr_test scripts, including usage, functions, workflows, and examples.
1 parent 70e4bb2 commit 177421d

1 file changed

Lines changed: 68 additions & 4 deletions

File tree

docs/scripts.md

Lines changed: 68 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ nav_order: 5
77

88
# scrape_test / sbr_test Scripts
99

10-
`scrape_test` and `sbr_test` are two internal scripts used to stress-test and improve the CAPTCHA solver by mass-collecting image data when SR is low.
10+
`scrape_test` and `sbr_test` are two internal scripts, which are used to stress-test and improve the CAPTCHA solver by mass-collecting image data when SR is low.
1111

1212
---
1313

@@ -17,10 +17,9 @@ This is a straightforward tool designed to "hammer" a site with requests to trig
1717

1818
**When to Use**: CAPTCHA appears immediately on URL.
1919

20-
### Function
21-
It allows for high-volume automated requests without complex user logic.
20+
1. **Function:** It allows for high-volume automated requests without complex user logic.
2221

23-
### Workflow
22+
2. **Workflow:**
2423

2524
**Step 1**: Download a list of target URLs (e.g., product pages).
2625

@@ -33,3 +32,68 @@ scrape_test urls -l 10000 -d homedepot.com -g pdp --days 1 > homedepot_pdp.txt
3332
```bash
3433
scrape_test run -u './homedepot_pdp.txt' -g scrape_r.js -t -B 'output' --clear -n 10
3534
```
35+
36+
3. **Best Practice:** It’s recommended to use randomized URLs rather than hitting the same page repeatedly to avoid being blocked by the target site’s server.
37+
38+
4. **CAPTCHA Integration:** If the inhouse_solver rules are configured with the correct Selectors for that site, `scrape_test` will automatically trigger a CAPTCHA solve and save the images to the dashboard.
39+
40+
---
41+
42+
## 2. sbr_test (The Scenario Solver):
43+
44+
This is a more functional tool that executes a complex, step-by-step user scenario to reach a CAPTCHA.
45+
46+
**When to Use:** CAPTCHA appears after form/click.
47+
48+
1. **Function:** Unlike the request-only `scrape_test`, this script can fill out forms, click buttons, and navigate deep into a site.
49+
50+
```bash
51+
sbr_test "@chatgpt_view-source.js" --parallel 30 --samples 30
52+
```
53+
54+
2. **Technology:** It supports various automation frameworks like Puppeteer, Playwright, and Selenium.
55+
56+
3. **Workflow:** It follows a predefined "scenario" file provided by the support or development teams. For example, a script might navigate to a checkout page and only then attempt to solve the CAPTCHA that appears.
57+
58+
Example for sbr_test scenario:
59+
```javascript
60+
// sbr_test --scenario scenario.js
61+
module.exports = async (page, {client, sleep, idx}) => {
62+
await client.send('Unlocker.setRules', {rules: {
63+
// captcha_solvers: ['deathbycaptcha'],
64+
captcha_solvers: ['anti_captcha'],
65+
// captcha_solvers: ['capmonster'],
66+
hcaptcha_conf: {
67+
pass_proxy: false,
68+
submit_form: true,
69+
},
70+
}});
71+
72+
console.log('Navigating...');
73+
await page.goto('https://iapps.courts.state.ny.us/webcivil/FCASCalendarSearch');
74+
console.log('Navigated, filling form...');
75+
await page.waitForSelector('select#cboCourt');
76+
await page.select('select#cboCourt', '80');
77+
await page.waitForSelector('select#cboCourtPart');
78+
await page.select('select#cboCourtPart', '38968');
79+
console.log('clicking "find calendar" and waiting results');
80+
await page.locator("input#btnFindCalendar").click();
81+
await page.waitForNavigation();
82+
console.log('Waiting captcha to solve...');
83+
const {status} = await client.send('Captcha.waitForSolve');
84+
console.log('Captcha solve status:', status);
85+
try {
86+
await page.waitForSelector(`a[href*="FCASCalendarDetail"]`, { timeout: 60000 });
87+
console.log("Search results appeared");
88+
} finally {
89+
await page.screenshot({ path: `iapps.courts.state.ny.us2_${idx}_res.png` });
90+
}
91+
};
92+
93+
```
94+
95+
4. **Debug Mode:** It features a visual Debug Mode that opens a browser window so you can watch the script execute in real-time.
96+
97+
```bash
98+
sbr_test "@chatgpt_view-source.js" --debug
99+
```

0 commit comments

Comments
 (0)