feat: add MrScraper components for Langflow - #12542
Conversation
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
WalkthroughThis PR integrates MrScraper SDK support by adding backend dependencies, frontend icon assets and UI registration, and eight new LFX components that expose MrScraper scraping, result retrieval, and batch operation capabilities. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error, 2 warnings, 1 inconclusive)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
ai-mrscraper
changed the title
github-actions
Bot
added
enhancement
github-actions
Bot
added
enhancement
github-actions
Bot
added
enhancement
github-actions
Bot
added
enhancement
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🧹 Nitpick comments (1)
src/frontend/src/icons/Mrscraper/MrscraperIcon.jsx (1)
1-30: Missing dark mode support for the icon.As per coding guidelines, SVG icon components should use the
isDarkprop to switch between light and dark color schemes. Currently, the fill color#1BD0EFis hardcoded without dark mode handling.♻️ Proposed fix to add dark mode support
-const SvgMrscraperLogo = (props) => ( +const SvgMrscraperLogo = ({ isDark, ...props }) => ( <svg viewBox="0 0 180 180" xmlns="http://www.w3.org/2000/svg" preserveAspectRatio="xMidYMid meet" {...props} > <path d="M0 0 C1.53741858 1.14472235 ..." - fill="#1BD0EF" + fill={isDark ? "#1BD0EF" : "#1BD0EF"} transform="translate(141.4375,20.5)" /> {/* Apply same pattern to other path elements */} </svg> );Note: If the MrScraper brand color should remain consistent across light/dark modes, you can keep the same color for both, but the pattern should still be in place for consistency with other icons in the codebase.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/frontend/src/icons/Mrscraper/MrscraperIcon.jsx` around lines 1 - 30, The SvgMrscraperLogo component hardcodes fill="#1BD0EF" on each <path>, missing dark-mode support; update SvgMrscraperLogo to accept an isDark prop (or read it from props), derive a fillColor variable (use the existing brand hex for light and an appropriate dark variant or the same if brand requires), and replace the hardcoded fill="#1BD0EF" on every path with the computed fillColor so the SVG switches colors based on isDark while keeping the component signature (SvgMrscraperLogo(props)) intact.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@src/lfx/src/lfx/components/mrscraper/mrscraper_ai_scraper.py`:
- Line 73: The code currently passes the literal string "None" for missing
proxy_country; change the parameter passed to the mrscraper call to use a Python
None (null) instead of the string. Locate the call that sets proxy_country
(e.g., proxy_country=self.proxy_country or "None") in mrscraper_ai_scraper.py
and replace the string fallback with None (or omit the field when
self.proxy_country is falsy) so the SDK receives a true null/absent value rather
than the invalid country string.
In `@src/lfx/src/lfx/components/mrscraper/mrscraper_batch_scrape.py`:
- Around line 69-78: The code silently treats any non-"AI" mode as manual;
update the check in mrscraper_batch_scrape.py to validate self.mode explicitly
(e.g., if self.mode == "AI": call client.bulk_rerun_ai_scraper(...); elif
self.mode == "Manual": call client.bulk_rerun_manual_scraper(...); else: raise a
clear exception such as ValueError or RuntimeError that includes the invalid
self.mode and scraper_id) so unsupported modes fail fast instead of defaulting
to manual; reference the methods bulk_rerun_ai_scraper and
bulk_rerun_manual_scraper and the attribute self.scraper_id when constructing
the error message.
In `@src/lfx/src/lfx/components/mrscraper/mrscraper_crawl_website.py`:
- Line 79: The call currently uses "max_depth=self.max_depth or 2", which treats
0 as falsy and overrides a valid 0; change the fallback to an explicit None
check so a user-specified 0 is preserved (e.g., replace the "or 2" pattern with
a conditional that uses self.max_depth if self.max_depth is not None, otherwise
2). Update the parameter passed to max_depth in the method call that references
self.max_depth (in mrscraper_crawl_website.py) so it uses the explicit
None-check expression instead of the boolean "or" fallback.
In `@src/lfx/src/lfx/components/mrscraper/mrscraper_get_result.py`:
- Around line 34-43: The get_result method currently wraps the whole response
from MrScraper.get_result_by_id into Data, but that SDK returns a dict
{"status_code": ..., "data": ..., "headers": ...}; update get_result to extract
only the "data" field from the response before creating Data. Specifically, in
async def get_result(self) use the MrScraper client (constructed with
self.api_token) to await client.get_result_by_id(result_id=self.result_id), then
pass result["data"] into Data(data=...) instead of the full result dict.
In `@src/lfx/src/lfx/components/mrscraper/mrscraper_run_ai_scraper.py`:
- Line 88: The call that sets max_depth is overwriting an explicit 0 because it
uses a truthy fallback (max_depth=self.max_depth or 2); change the fallback to
only apply when self.max_depth is None so that an explicit 0 is honored. Locate
where max_depth is passed (the argument name max_depth in the call, referencing
self.max_depth) and replace the truthy-coalescing logic with a None-check (use
self.max_depth if not None, otherwise 2) so explicit 0 values are preserved.
---
Nitpick comments:
In `@src/frontend/src/icons/Mrscraper/MrscraperIcon.jsx`:
- Around line 1-30: The SvgMrscraperLogo component hardcodes fill="#1BD0EF" on
each <path>, missing dark-mode support; update SvgMrscraperLogo to accept an
isDark prop (or read it from props), derive a fillColor variable (use the
existing brand hex for light and an appropriate dark variant or the same if
brand requires), and replace the hardcoded fill="#1BD0EF" on every path with the
computed fillColor so the SVG switches colors based on isDark while keeping the
component signature (SvgMrscraperLogo(props)) intact.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 5f18e504-7bc3-4e7e-af50-7b0c9d30d093
⛔ Files ignored due to path filters (2)
src/frontend/src/icons/Mrscraper/mrscraper.svgis excluded by!**/*.svguv.lockis excluded by!**/*.lock
📒 Files selected for processing (16)
src/backend/base/pyproject.tomlsrc/frontend/src/icons/Mrscraper/MrscraperIcon.jsxsrc/frontend/src/icons/Mrscraper/index.tsxsrc/frontend/src/icons/eagerIconImports.tssrc/frontend/src/icons/lazyIconImports.tssrc/frontend/src/utils/styleUtils.tssrc/lfx/src/lfx/components/__init__.pysrc/lfx/src/lfx/components/mrscraper/__init__.pysrc/lfx/src/lfx/components/mrscraper/mrscraper_ai_scraper.pysrc/lfx/src/lfx/components/mrscraper/mrscraper_batch_scrape.pysrc/lfx/src/lfx/components/mrscraper/mrscraper_crawl_website.pysrc/lfx/src/lfx/components/mrscraper/mrscraper_fetch_html.pysrc/lfx/src/lfx/components/mrscraper/mrscraper_get_result.pysrc/lfx/src/lfx/components/mrscraper/mrscraper_get_results.pysrc/lfx/src/lfx/components/mrscraper/mrscraper_run_ai_scraper.pysrc/lfx/src/lfx/components/mrscraper/mrscraper_run_manual_scraper.py
| url=self.url, | ||
| message=self.message, | ||
| agent=self.agent, | ||
| proxy_country=self.proxy_country or "None", |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
In the mrscraper Python SDK, what values are valid for create_scraper(..., proxy_country=...) when no proxy country is desired? Is Python None supported, and is the string "None" considered invalid?
💡 Result:
When you don’t want a proxy country, the MrScraper API treats proxyCountry as optional—so the “no proxy country” value is to omit the field entirely. (docs.mrscraper.com)
In the mrscraper-sdk Python client, create_scraper(..., proxy_country=...) is designed with that same intent: you should either not pass proxy_country at all, or pass Python None (so the client can omit it / send it as JSON null, depending on implementation). (pypi.org)
The literal string "None" is not a special value; it would be sent as a regular country code string, and since proxyCountry is documented as a proxy country code, "None" should be treated as invalid. (docs.mrscraper.com)
Citations:
- 1: https://docs.mrscraper.com/docs/api/v3/scraper/ai-init?utm_source=openai
- 2: https://pypi.org/pypi/mrscraper-sdk/json
- 3: https://docs.mrscraper.com/docs/api/v3/scraper/ai-init?utm_source=openai
Use None (null) for missing proxy_country, not the string "None".
The mrscraper SDK documentation confirms that proxyCountry is optional and expects either Python None or the field to be omitted entirely. The literal string "None" is not a valid country code and will be rejected by the API.
🔧 Proposed fix
- proxy_country=self.proxy_country or "None",
+ proxy_country=self.proxy_country or None,📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| proxy_country=self.proxy_country or "None", | |
| proxy_country=self.proxy_country or None, |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/lfx/src/lfx/components/mrscraper/mrscraper_ai_scraper.py` at line 73, The
code currently passes the literal string "None" for missing proxy_country;
change the parameter passed to the mrscraper call to use a Python None (null)
instead of the string. Locate the call that sets proxy_country (e.g.,
proxy_country=self.proxy_country or "None") in mrscraper_ai_scraper.py and
replace the string fallback with None (or omit the field when self.proxy_country
is falsy) so the SDK receives a true null/absent value rather than the invalid
country string.
| if self.mode == "AI": | ||
| result = await client.bulk_rerun_ai_scraper( | ||
| scraper_id=self.scraper_id, | ||
| urls=url_list, | ||
| ) | ||
| else: | ||
| result = await client.bulk_rerun_manual_scraper( | ||
| scraper_id=self.scraper_id, | ||
| urls=url_list, | ||
| ) |
There was a problem hiding this comment.
Unexpected mode values currently default to Manual silently.
Please fail fast for unsupported values instead of routing to manual implicitly.
🔧 Proposed fix
- if self.mode == "AI":
+ if self.mode == "AI":
result = await client.bulk_rerun_ai_scraper(
scraper_id=self.scraper_id,
urls=url_list,
)
- else:
+ elif self.mode == "Manual":
result = await client.bulk_rerun_manual_scraper(
scraper_id=self.scraper_id,
urls=url_list,
)
+ else:
+ msg = f"Unsupported mode: {self.mode!r}. Expected 'AI' or 'Manual'."
+ raise ValueError(msg)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| if self.mode == "AI": | |
| result = await client.bulk_rerun_ai_scraper( | |
| scraper_id=self.scraper_id, | |
| urls=url_list, | |
| ) | |
| else: | |
| result = await client.bulk_rerun_manual_scraper( | |
| scraper_id=self.scraper_id, | |
| urls=url_list, | |
| ) | |
| if self.mode == "AI": | |
| result = await client.bulk_rerun_ai_scraper( | |
| scraper_id=self.scraper_id, | |
| urls=url_list, | |
| ) | |
| elif self.mode == "Manual": | |
| result = await client.bulk_rerun_manual_scraper( | |
| scraper_id=self.scraper_id, | |
| urls=url_list, | |
| ) | |
| else: | |
| msg = f"Unsupported mode: {self.mode!r}. Expected 'AI' or 'Manual'." | |
| raise ValueError(msg) |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/lfx/src/lfx/components/mrscraper/mrscraper_batch_scrape.py` around lines
69 - 78, The code silently treats any non-"AI" mode as manual; update the check
in mrscraper_batch_scrape.py to validate self.mode explicitly (e.g., if
self.mode == "AI": call client.bulk_rerun_ai_scraper(...); elif self.mode ==
"Manual": call client.bulk_rerun_manual_scraper(...); else: raise a clear
exception such as ValueError or RuntimeError that includes the invalid self.mode
and scraper_id) so unsupported modes fail fast instead of defaulting to manual;
reference the methods bulk_rerun_ai_scraper and bulk_rerun_manual_scraper and
the attribute self.scraper_id when constructing the error message.
| url=self.url, | ||
| message="", | ||
| agent="map", | ||
| max_depth=self.max_depth or 2, |
There was a problem hiding this comment.
max_depth=0 is currently ignored due to falsy fallback.
Line 79 overrides a valid 0 value (0 = start URL only per Line 35) back to 2, changing user intent.
🔧 Proposed fix
- max_depth=self.max_depth or 2,
+ max_depth=2 if self.max_depth is None else self.max_depth,📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| max_depth=self.max_depth or 2, | |
| max_depth=2 if self.max_depth is None else self.max_depth, |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/lfx/src/lfx/components/mrscraper/mrscraper_crawl_website.py` at line 79,
The call currently uses "max_depth=self.max_depth or 2", which treats 0 as falsy
and overrides a valid 0; change the fallback to an explicit None check so a
user-specified 0 is preserved (e.g., replace the "or 2" pattern with a
conditional that uses self.max_depth if self.max_depth is not None, otherwise
2). Update the parameter passed to max_depth in the method call that references
self.max_depth (in mrscraper_crawl_website.py) so it uses the explicit
None-check expression instead of the boolean "or" fallback.
| async def get_result(self) -> Data: | ||
| try: | ||
| from mrscraper import MrScraper | ||
| except ImportError as e: | ||
| msg = "Could not import mrscraper SDK. Please install it with `pip install mrscraper-sdk`." | ||
| raise ImportError(msg) from e | ||
|
|
||
| client = MrScraper(token=self.api_token) | ||
| result = await client.get_result_by_id(result_id=self.result_id) | ||
| return Data(data=result) |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
What is the API for the mrscraper-sdk Python package? How do you use MrScraper class and get_result_by_id method?
💡 Result:
mrscraper-sdk Python API (core)
Install:
pip install mrscraper-sdkThe package exposes a single main async client class: MrScraper. You initialize it with your API token, and all methods are coroutines (must be awaited). (pypi.org)
MrScraper class (methods)
fetch_html(url, *, timeout, geo_code, block_resources)create_scraper(url, message, *, agent, proxy_country, ...)rerun_scraper(scraper_id, url, *, max_depth, max_pages, limit, ...)bulk_rerun_ai_scraper(scraper_id, urls)rerun_manual_scraper(scraper_id, url)bulk_rerun_manual_scraper(scraper_id, urls)get_all_results(*, sort_field, sort_order, page_size, page, search, ...)get_result_by_id(result_id)(pypi.org)
All methods return a dict shaped like:
{"status_code": int, "data": Any, "headers": dict} (pypi.org)
How to use MrScraper and get_result_by_id
import asyncio
from mrscraper import MrScraper
async def main():
client = MrScraper(token="MRSCRAPER_API_TOKEN")
# Fetch one result by its ID
resp = await client.get_result_by_id("result_12345")
print(resp["status_code"])
print(resp["data"]) # parsed JSON body (result detail)
# print(resp["headers"])
asyncio.run(main())get_result_by_id(result_id) “fetch[es] a single result by its ID”. (pypi.org)
Citations:
- 1: https://pypi.org/project/mrscraper-sdk/
- 2: https://pypi.org/project/mrscraper-sdk/
- 3: https://pypi.org/project/mrscraper-sdk/
- 4: https://pypi.org/project/mrscraper-sdk/
Fix response data extraction from MrScraper SDK.
The mrscraper SDK's get_result_by_id() method returns a dict with shape {"status_code": int, "data": Any, "headers": dict}. The code currently passes the entire response dict to Data(), but should extract only the data field:
Suggested fix
result = await client.get_result_by_id(result_id=self.result_id)
return Data(data=result["data"])🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/lfx/src/lfx/components/mrscraper/mrscraper_get_result.py` around lines 34
- 43, The get_result method currently wraps the whole response from
MrScraper.get_result_by_id into Data, but that SDK returns a dict
{"status_code": ..., "data": ..., "headers": ...}; update get_result to extract
only the "data" field from the response before creating Data. Specifically, in
async def get_result(self) use the MrScraper client (constructed with
self.api_token) to await client.get_result_by_id(result_id=self.result_id), then
pass result["data"] into Data(data=...) instead of the full result dict.
| result = await client.rerun_scraper( | ||
| scraper_id=self.scraper_id, | ||
| url=self.url, | ||
| max_depth=self.max_depth or 2, |
There was a problem hiding this comment.
max_depth=0 cannot be honored with current fallback logic.
Line 88 converts an explicit 0 to 2, which conflicts with the documented behavior on Line 42.
🔧 Proposed fix
- max_depth=self.max_depth or 2,
+ max_depth=2 if self.max_depth is None else self.max_depth,📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| max_depth=self.max_depth or 2, | |
| max_depth=2 if self.max_depth is None else self.max_depth, |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/lfx/src/lfx/components/mrscraper/mrscraper_run_ai_scraper.py` at line 88,
The call that sets max_depth is overwriting an explicit 0 because it uses a
truthy fallback (max_depth=self.max_depth or 2); change the fallback to only
apply when self.max_depth is None so that an explicit 0 is honored. Locate where
max_depth is passed (the argument name max_depth in the call, referencing
self.max_depth) and replace the truthy-coalescing logic with a None-check (use
self.max_depth if not None, otherwise 2) so explicit 0 values are preserved.
d5a6017 to
7345dcf
Compare
github-actions
Bot
added
enhancement
|
Hi team, just following up on this PR. I noticed the integration is already in place, but it looks like the PR is still blocked by pending review and a few failing checks. From the thread, there were also earlier concerns around test coverage and some component issues that may have affected the review flow. Would appreciate it if someone could take a look and let me know what is still needed from my side to move this forward. Happy to address any remaining feedback. Thanks! |
|
Hi @vjgit96 and @Adam-Aghili Would you mind taking a look at this PR when you have a chance We added the MrScraper integration and would really appreciate your review and any feedback Thank you |
2 participants
Summary
Add MrScraper as a new tool integration — 8 components covering AI scraping, website crawling, rendered HTML fetch, batch reruns, and result listing/detail.
MrScraper provides AI-powered and manual web scraping: natural-language extraction, map-based crawling, stealth browser HTML, and dashboard-style scraper reruns with paginated results.
Components added (8)
All components live under
src/lfx/src/lfx/components/mrscraper/and are registered insrc/lfx/src/lfx/components/__init__.py.Implementation notes
Component.SecretStrInputfor the MrScraper API token.tool_mode=Truewhere appropriate for agent workflows.mrscraperPython SDK (pip install mrscraper-sdk); components raise a clear import error if the extra is not installed.Data(consistent with other LFX integrations).Frontend
MrScraperentry insrc/frontend/src/utils/styleUtils.ts.MrScraperwired ineagerIconImports.tsandlazyIconImports.ts(SVG undersrc/frontend/src/icons/Mrscraper/).Dependencies
mrscraper-sdk— optional extra onlangflow-base(e.g.langflow-base[mrscraper]), version range as defined insrc/backend/base/pyproject.toml.Links
Release notes (optional)
New features
Component / UI
Summary by CodeRabbit
Release Notes