Repository navigation
[Proposal] Support Python asyncio by migrating to httpx from requests #27
Description
Activity
I agree with the motivation. I have already created a branch with 0.2.0 based on aiohttp which is based on asyncio.
But I stopped in my tracks for 2 reasons related to customer concerns.
- python 2.x is no longer supported.
- dependencies: new dependencies which breaks enterprise stacks.
- synchronization with https://serpapi.com/integrations/python
Historically we have done a bad job synchronizing the web documentation update, and the plugin release.
The plugin doc is pull automatically in the website, but the "Code to integrate" need manual update.
solutions:
- drop python2 compatibility. (need to check market share left of python 2)
- add the new library under python3 entry, and keep python2
I'd like 2 because it does break existing enterprise stack, and allow them to migrate slowly. but maybe this argument doesn't stand anymore in the age of AI.
what do you think ?
Hi @jvmvik,
Thanks for sharing the details. Please see my comments below:
- Breaking changes/updates needed for integrations page: My idea is to keep the existing sync API as-is. We will not break any existing code. Existing users will not require any code changes.
Even after the
httpxmigration, the below sync usage will work:import os import serpapi def send(): client = serpapi.Client(api_key=os.getenv("SERPAPI_KEY")) results = client.search({ "engine": "google", "q": "coffee" }) print(results)
For Async, we can introduce a new client (
serpapi.AsyncClient)- this was not mentioned correctly in my previous comment:import os import serpapi async def send(): client = serpapi.AsyncClient(api_key=os.getenv("SERPAPI_KEY")) results = await client.search({ "engine": "google", "q": "coffee" }) print(results)
This is the advantage I see with
httpx: it supports bothsyncandasyncnetwork requests and has a design that is almost compatible with our currentrequestslibrary. By migrating tohttpx, we can maintain backward compatibility with our user-facing methods and support async as well.Because of the backward compatibility, we can keep our integrations page as-is and can mention async support in our library docs.
- Regarding the performance benefits, I agree that
aiohttpwill be more performant. But as you rightly noted, a full migration will be costly - we need to add a new package so that users of our current sync client will not see any breaking changes. Or we need to have sync calls from our package usingrequestsunder the hood (keeping existing behavior) and implement new async methods that useaiohttp, which will increase the maintenance burden.
Before starting work on this, I will run a benchmark using our real API calls to measure actual performance.
If we are moving forward with
httpx, we can implement support for bothsyncandasynccalls with a single dependency with minimal changes to existing code. This is the unique benefit of httpx.# sync r = httpx.get(''https://www.example.com/')
# async async with httpx.AsyncClient() as client: r = await client.get('https://www.example.com/')
Also, the community is very active around
httpx- Pydantic is maintaining their fork of httpx, and I expect to see performance improvements going forward.-
Python 2 compatibility: I don't think this should be a concern for us. Python 2 reached end of life in 2020, and most of our dependencies do not support it. The current version of
requestssupports only Python 3.9+. Ourpyproject.tomlcurrently lists the required Python version as>=3.6, so Python2 support is safe to ignore. I also verified this using a Python 2.7 Docker image:pip install serpapifails on Python 2. -
Dependency breakages: As a part of this change, we will have our dependency changed from
requeststohttpx. Due to backward-compatible design, if a customer integration gets our updated client, their existing code will work without any major updates. Also, since we will be bumping the version number for the release, pinned package install will have the previous version usingrequestsinstalled.
I guess internal dependency migrations like this are common (saw that OpenAI Python SDK also made a similar migration tohttpxfromrequests, ref)
Convince me httpx is the right choice. Benchmarks between httpx and aiohttp would likely be too close to show
meaningful differences. Most developers avoid async programming because it adds substantial complexity for minimal
returns in real-world applications—the only real benefit is high performance, and Python isn't a high-performance
language anyway.about the package manager, what are you planning to use ? for marketing reason we should document both uv and pip3
Reacted by Adarsh Divakaran@jvmvik Please see https://serpapi.slack.com/archives/C09AS452KMZ/p1781258667885529 - added a few benchmark results and analysis on our internal discussion thread.
- marked Add async support (e.g. AsyncClient) for use in async frameworks #35 as a duplicate of this issue
on Jul 10, 2026
Motivation
asyncpattern works well with I/O blocked apps.Plan:
and will also add support for async calls:
Users can choose the pattern that best fits their application.