[sail3r]

web_search: more ways to tune each service, typed settings, Firecrawl research papers, Perplexity exclusions

Follow-up to #4, which added the catalogue of search services. Branch feat/search-dials, 5 commits on edc3186. agentty has a matching PR that moves its submodule to this branch, so this one needs to be merged first.

Why

When the catalogue went in, each service got at most three settings, chosen because they change the quality of the results. Settings that mainly change cost or how much text comes back were left out on purpose. This PR goes back through every vendor's current documentation and adds the remaining settings that change which results come back, with the same rule: nothing whose main effect is cost, tokens or extra page text.

The same review turned up lists that had drifted from the docs, an API change Firecrawl has announced for 2026-11-16 that would break its Research category, and a way to exclude sites on Perplexity that was thought not to exist. All three are fixed here.

1. Lists that were missing values or out of date

• Firecrawl Category is now Developer, Research, PDF and Government. GitHub is gone: Firecrawl no longer lists it, and GitHub content now sits in its Developer index. If someone already has GitHub saved, that setting is ignored and the search runs with no category, as if Any were chosen. It does not make the search fail.
• Past hour is added to Freshness for Firecrawl, Serper and Exa. Brave, Tavily and DuckDuckGo don't offer anything shorter than a day, so they keep their lists. Perplexity already had it.
• Exa Mode lists all six modes: Auto, Fast, Instant, Deep lite, Deep and Deep reasoning. The deep modes research and summarise before they answer. Exa quotes about 4 seconds for Deep lite, 4-15 for Deep, and 12-40 for Deep reasoning. A host that gives up on a request after 30 seconds (agentty does) will sometimes see Deep reasoning time out. The next service in the chain then answers instead. The setting's help text says this.
• Brave Country gains All regions. Without it, Brave answers as if the search came from the US.
• DuckDuckGo Region lists all 68 regions on DuckDuckGo's URL-parameters help page, up from 13, in the page's order.

2. New settings

Service	Setting	What it does
Tavily (Free and API key)	Exact phrases	Words in "quotes" must appear exactly as written, with no synonyms. Useful for error messages and function names.
	Language match	Boost (the default) ranks results in the chosen language first. Only this language removes the rest. It uses the host's language setting and does nothing when none is set.
	Country	Ranks results from that country higher. Tavily only accepts it with the General topic, so it is not sent with News or Finance.
	Safe search	Filters adult content. Tavily doesn't support it with the Fast and Ultra fast depths, so it is not sent with those.
Tavily Free	Depth	Tavily API key already had this. On Tavily Free, Advanced uses 2 credits of the key-less allowance instead of 1. Tavily doesn't publish the key-less limit.
Brave API key	Spell correction	When on (Brave's default), Brave searches its own corrected version of the query. That can "fix" a function name or an error string into something else. Off searches exactly what was asked.
Firecrawl (Free and API key)	Safe search	Filters explicit content.
	Sort	Newest first sorts by date, within the Freshness window if one is set.
Exa	Safe search	Filters unsafe content.
	Country	Where the searcher is. Exa uses it to localise results.
Perplexity	Mode	Web (the default) or Fast, a lighter and quicker search. Perplexity's third mode, People, searches for people rather than pages, so it is left out.

A service can now have up to seven settings (kMaxWebSearchDials). Tavily uses all seven.

When one setting only works together with another, as with Tavily's Country, Safe search and Language match, the library applies the rule itself when it builds the request. A host can show every setting as a plain choice and never sends a combination the vendor would reject. Tavily also wants a country name ("new zealand") rather than a code ("NZ"), so the shared country codes are translated for it.

3. Typed settings

Until now every setting was a choice from a list. Two useful ones aren't lists, so a setting can now be free text:

• Brave Goggle. A Goggle is a small rules file that re-ranks Brave's results: it raises sites you trust and lowers or removes others. The setting takes the web address of one hosted Goggle. That is a file on GitHub, GitLab or a Gist, registered with Brave at search.brave.com/goggles/create. An example is https://raw.githubusercontent.com/brave/goggles-quickstart/main/goggles/tech_blogs.goggle. Brave also accepts rules typed straight into the request, but it recommends hosted files and limits typed rules by URL length, so only an address is accepted.
• Firecrawl Location. A place to search from, such as San Francisco,California,United States. Firecrawl recommends setting Country as well.

Accepted text is one line of valid UTF-8 with no control characters, within a size limit: 2048 bytes for a Goggle address, 200 for a Location. A Goggle address must also start with https:// and contain no spaces. The library runs this check before every request and drops a value that fails. The check is also public (web_search_dial_value_ok), so a host can refuse a bad value as soon as it is typed. Otherwise the host would store a value that every later search ignores.

For host authors:

• A typed setting has no list of values: values is empty and the new text field says which kind it is. A host that draws every setting as a dropdown needs to handle this, or it will show an empty dropdown.
• Code that builds a WebSearchDial as {id, label, help, values} still compiles unchanged. The new fields come after those and have defaults.

4. Firecrawl research papers

On 2026-11-16 Firecrawl changes what its Research category returns:

• Before: ordinary web pages from about 14 academic websites.
• After: results from its Research Index, about 43 million paper abstracts from PubMed, bioRxiv, medRxiv and arXiv. They come back as paper records, in a different part of the reply.

Until now the library only looked for web results and treated a reply without them as broken. From that date every Research search would have failed and fallen through to the next service.

Both parts of the reply are read now: web results first, then papers, together counting toward the number of results asked for. Until the date nothing changes, because the papers part is simply absent.

A paper record has no web address, and the library drops any result without one: the model needs it to cite the result and to open it. So an address is built from the paper's id. Firecrawl documents four kinds of id:

Id	Links to
arxiv:2105.05233	https://arxiv.org/abs/2105.05233
pmid:31452104	https://pubmed.ncbi.nlm.nih.gov/31452104/
pmcid:PMC6824467	https://pmc.ncbi.nlm.nih.gov/articles/PMC6824467/
doi:10.1038/s41586-021-03819-2	https://doi.org/10.1038/s41586-021-03819-2

A paper with any other kind of id, or none, links to Firecrawl's own page for it. That page uses the Firecrawl id that every record carries. So if Firecrawl adds a new kind of id later, those papers still appear with a working link instead of disappearing. That link is Firecrawl's data page for the paper, not the publisher's. The paper's abstract is used as the result's text.

Firecrawl's docs don't say whether PubMed Central ids keep their usual PMC prefix, so both spellings give the same link.

5. Perplexity uses its own filter for excluded sites

A host can ask for sites to be left out of every search (agentty calls this Never return). For Tavily, Firecrawl and Exa the library sends them through each service's own exclusion filter. For Perplexity it couldn't: when the catalogue went in, we believed Perplexity's domain filter could only limit a search to certain sites, not exclude them. So excluded sites were written into the search text as -site:reddit.com. Perplexity doesn't document that it understands this form at all.

Its documentation does describe excluding: put a - in front of a site in its domain filter. The filter takes at most 20 sites, and must either all include or all exclude. Perplexity now gets excluded sites the same way the other three services do, through that filter.

The library already lifted at most 20 excluded sites out of the search text, so the filter never goes over Perplexity's limit. A 21st and later site stays in the search text as before. Only the exclude form is ever sent, so the two forms are never mixed.

Checked against

Every request field and value comes from the vendor's own documentation, read during this work:

• Brave's Web Search reference and Goggles guide
• Tavily's search reference and its key-less access page
• Firecrawl's search reference, search guide and Research Index page
• Exa's OpenAPI spec and search quickstart
• Perplexity's Search API reference
• DuckDuckGo's URL-parameters help page

Left out, and why

• Serper location. Serper's own reference needs a login. The only public schema is third-party, and it doesn't list a location field either, so it couldn't be verified.
• Anything mainly about cost or the amount of text. That covers Brave's extra snippets and summary; Tavily's answer, raw content, chunks and auto parameters; Firecrawl's page scraping; Exa's page contents; and Perplexity's token limits.
• Options whose results land somewhere the library doesn't read. These are Brave's result filter, Firecrawl's news and image sources, and Serper's other endpoints.
• Custom date ranges, Exa's objective and system prompt. These fit better as something the model sets on each search than as a fixed setting.

Testing

tests/web_search_services_test.cpp covers:

• the values each setting offers, and how many settings each service has;

• each new setting reaching the request, and staying out of it when left at its default or when a rule says it can't be sent;

• what typed settings accept and refuse;

• links for each kind of paper id and the fallback, web results and papers in one reply, and which replies count as readable.

• excluded sites reaching Perplexity's filter (and leaving the search text), the 20-site limit, and no filter being sent when nothing is excluded.

• The changed files compile with g++ 16.2 (-std=c++23 -Wall -Wextra) and give no warnings.

• Full build and ctest on Linux.

Commits

• 2ec79df web_search: dials offer every value their vendor documents
• 3e6d8e6 web_search: more quality dials for Tavily, Brave, Firecrawl, Exa, Perplexity
• f0eba67 web_search: free-text dials, for Brave Goggles and Firecrawl location
• 1e69b25 web_search: Firecrawl paper records, each with a link
• 791957d web_search: Perplexity takes excluded sites as its domain filter