FetchNode passes the page HTML as convert_to_md's baseurl, inflating model input ~74x

[julianriegler]

Version: scrapegraphai 2.2.2 (call site unchanged on master), Python 3.14, macOS

What happens

FetchNode.handle_web_source calls (scrapegraphai/nodes/fetch_node.py:398 on master):

parsed_content = convert_to_md(document[0].page_content, parsed_content)

At that point parsed_content is document[0].page_content, i.e. the full HTML of the page.
convert_to_md(html, url) assigns its second argument to html2text.HTML2Text().baseurl, so
html2text prepends the entire HTML document to every relative link it finds.

Reproduction (no LLM needed)

import urllib.request
from scrapegraphai.utils.convert_to_md import convert_to_md

raw = urllib.request.urlopen("https://books.toscrape.com/").read().decode()
print(len(raw))                                                # 51274
print(len(convert_to_md(raw, "https://books.toscrape.com/")))  # 13843    <- correct
print(len(convert_to_md(raw, raw)))                            # 3816199  <- what FetchNode does

End to end with SmartScraperGraph on the same page, ParseNode reports 1,948,618 chars of
parsed content instead of ~13.8k. All of it is sent as model input, and the markdown links are
unusable because each href contains the whole document.

With google/gemini-2.5-flash that is roughly 0.15 USD per run instead of 0.001 USD, for an
identical answer. On a page with many relative links the factor is ~74x.

Expected

baseurl should be the source URL, so relative links become absolute.

Suggested fix

parsed_content = convert_to_md(document[0].page_content, source)

The use_soup branch (fetch_node.py:302) looks affected too:

parsed_content = convert_to_md(source, parsed_content)

Here source is the URL and parsed_content the HTML, so the two arguments appear swapped.
In that same branch parsed_content is unbound when cut is True, because it is only assigned
inside if not self.cut:.

[dosubot]

Hi @julianriegler! I'm Dosu and I'm helping the ScrapeGraphAI, Inc team.

Your diagnosis is spot-on. On master, handle_web_source calls convert_to_md(document[0].page_content, parsed_content) where parsed_content is document[0].page_content [1], so the full HTML gets assigned to h.baseurl [2]. The use_soup branch has the swapped-arguments problem you flagged too — convert_to_md(source, parsed_content) passes the URL as HTML and vice versa [3] — and parsed_content is indeed unbound when cut=True since it's only assigned inside if not self.cut: [4].

All of these bugs are addressed in PR #1112, which makes exactly the changes you suggested — convert_to_md(doc.page_content, source) and convert_to_md(parsed_content, source) — plus fixes the cleanup_html call (was receiving a Response object instead of response.text, and its tuple return wasn't being unwrapped) and the unbound parsed_content in the cut=True path [5]. That PR is still open and not yet merged [6].

Separately, PR #1131 (merged) improved convert_to_md so that when a real URL is passed, relative links resolve against the full document path instead of just the site root [7]. That improves link resolution once #1112 lands, but doesn't fix the caller-side bug you're reporting since convert_to_md still blindly assigns whatever it receives to h.baseurl [2].

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[MonsterDylan]

Confirming this on the released 2.2.2 package (not just master), and flagging that the impact is worse than input inflation — on <table>-based pages it causes complete data loss / empty output, so it's a correctness bug, not just a cost/token one.

Repro (Hacker News)

from scrapegraphai.graphs import SmartScraperGraph

SmartScraperGraph(
    prompt="Extract the top stories with rank, title, url, points, comments.",
    source="https://news.ycombinator.com/",
    config={"llm": {"api_key": "...", "model": "openai/gpt-4o-mini"}},
).run()
# -> {'stories': []}

What happens at each stage

Stage	Chars	Note
Playwright fetch	34,133	full HN HTML, no block
convert_to_md(html, url=<the same HTML>)	44,827	output starts | [![](<html lang="en"... — the whole doc becomes a broken markdown-image "URL" because it's assigned to h.baseurl
ParseNode Html2TextTransformer	9	collapses to | [![](\n\n
GenerateAnswer	—	model sees ~nothing → NA / {'stories': []}

So with a ChatOpenAI model (which triggers the convert_to_md branch in FetchNode), any page whose markdown begins with a linked/image element ahead of a table layout can silently return empty. Modern div-based pages (e.g. books.toscrape.com) happen to survive.

PR #1112 fixes exactly this (convert_to_md(doc.page_content, source)). Given it turns some sites into silent empty results rather than merely bloated input, it'd be great to get that merged. Happy to help test.

Workaround until then, for anyone hitting empty results: patch the caller so the bogus base-URL is dropped when it's HTML —

import scrapegraphai.nodes.fetch_node as fn
_orig = fn.convert_to_md
fn.convert_to_md = lambda html, url=None: _orig(
    html, None if (url and (len(url) > 2000 or url.lstrip()[:1] == "<")) else url
)

[chenhz01]

To recap the call site: it passes parsed_content, which is the raw HTML, where convert_to_md expects a URL string, so html2text's baseurl becomes the entire document and every relative link gets the full HTML prepended. An open PR already fixes the call site to pass the real source URL; my addition below is the mechanical guard that would have caught this. Beyond that, I'd add a cheap input-hygiene guard that would have caught this mechanically: assert baseurl parses as a URL and is under a sane length, plus a regression test asserting converted output stays within a small factor of the raw HTML length - a 74x blow-up should trip a test, not a user's token bill. Want me to take a crack at it? Happy to open a PR with working code.