Skip to content

Commit bb0b40a

Browse files
committed
feat: add asyncio client with HTTPX
1 parent e2c817c commit bb0b40a

26 files changed

Lines changed: 1460 additions & 351 deletions

‎HISTORY.md‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,18 @@
11
Release History
22
===============
33

4+
Unreleased
5+
----------
6+
7+
- Added `serpapi.AsyncClient` with async search, archive, account, locations,
8+
image upload, and pagination support.
9+
- Migrated the synchronous HTTP transport from Requests to HTTPX while keeping
10+
the existing `serpapi.Client` API and request-option compatibility.
11+
- Added explicit sync and async client lifecycle management, deterministic
12+
concurrency tests, asyncio documentation, and a local HTTP benchmark.
13+
- Updated the minimum supported Python version to 3.8 to match the SDK's CI
14+
matrix and HTTPX requirements.
15+
416
1.1.2 (2026-09-15)
517
------------------
618

‎MANIFEST.in‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ include README.md CONTRIBUTING.md HISTORY.md LICENSE
22
include .readthedocs.yaml
33
include scripts/check_docs_revision.py scripts/publish_docs.py
44
recursive-include tests *.py
5+
recursive-include benchmarks *.py *.md
56
recursive-include docs *.py *.md *.txt *.css Makefile
67
include assets/serpapi-logo.svg assets/serpapi-icon.png
78
prune docs/_build

‎README.md‎

Lines changed: 39 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,12 +9,18 @@ Query a vast range of data at scale, including web search results, flight schedu
99

1010
## Installation
1111

12-
To install the `serpapi` package, simply run the following command:
12+
Install the `serpapi` package with pip or add it to a uv project:
1313

1414
```bash
15-
$ pip install serpapi
15+
pip3 install serpapi
1616
```
1717

18+
```bash
19+
uv add serpapi
20+
```
21+
22+
Python 3.8 or newer is required.
23+
1824
Please note that this package is separate from the legacy `serpapi` module, which is available on PyPi as `google-search-results`. This package is maintained by SerpApi, and is the recommended way to access the SerpApi service from Python.
1925

2026
## Simple Usage
@@ -36,6 +42,37 @@ print(results)
3642

3743
The `results` variable now contains a `SerpResults` object, which acts just like a standard dictionary, with some convenient functions added on top.
3844

45+
## Async Usage
46+
47+
Use `AsyncClient` in applications built on `asyncio`. Reuse one client so its
48+
connection pool can serve all concurrent requests:
49+
50+
```python
51+
import asyncio
52+
import os
53+
54+
import serpapi
55+
56+
57+
async def main():
58+
async with serpapi.AsyncClient(
59+
api_key=os.environ["SERPAPI_KEY"]
60+
) as client:
61+
results = await asyncio.gather(
62+
client.search(engine="google", q="coffee"),
63+
client.search(engine="google", q="tea"),
64+
client.search(engine="google", q="pizza"),
65+
)
66+
print([result["search_metadata"]["id"] for result in results])
67+
68+
69+
asyncio.run(main())
70+
```
71+
72+
The synchronous `Client` and module-level helpers remain available. See the
73+
[Asyncio Client guide](https://serpapi-python.readthedocs.io/en/latest/user_guide/asyncio.html)
74+
for lifecycle, pagination, error-handling, and upload examples.
75+
3976
This example runs a search for "coffee" on Google. It then returns the results as a regular Python Hash.
4077
See the [playground](https://serpapi.com/playground) to generate your own code.
4178

‎benchmarks/README.md‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
# HTTP client benchmark
2+
3+
This benchmark compares the previous `requests.Session` transport, the
4+
synchronous HTTPX-backed `serpapi.Client`, and concurrent
5+
`serpapi.AsyncClient` calls.
6+
7+
It runs against a local threaded server with a configurable response delay, so
8+
it does not need an API key, consume searches, or mix client behavior with
9+
internet and SerpApi server variance.
10+
11+
From the repository root, run:
12+
13+
```bash
14+
uv run --with requests python benchmarks/http_clients.py
15+
```
16+
17+
Or install the development package and benchmark-only dependency with pip:
18+
19+
```bash
20+
pip3 install -e . requests
21+
python benchmarks/http_clients.py
22+
```
23+
24+
Each result is the median of three runs and includes the observed range. Client
25+
construction and shutdown are excluded consistently. Change the workload with
26+
`--count`, `--delay`, and `--repeats`. For example:
27+
28+
```bash
29+
uv run --with requests python benchmarks/http_clients.py --count 50 --delay 0.1 --repeats 5
30+
```
31+
32+
The sequential `requests` and HTTPX results show transport overhead under the
33+
same workload. The async result demonstrates throughput when independent
34+
I/O-bound calls overlap; it does not claim that one SerpApi search becomes
35+
faster.

‎benchmarks/http_clients.py‎

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
"""Compare sequential requests with concurrent SerpApi AsyncClient calls.
2+
3+
The benchmark uses a local delayed HTTP server. It measures client-side
4+
concurrency without consuming SerpApi searches or introducing internet and API
5+
server variance.
6+
"""
7+
8+
import argparse
9+
import asyncio
10+
import json
11+
import statistics
12+
import threading
13+
import time
14+
from contextlib import contextmanager
15+
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
16+
17+
import requests
18+
19+
import serpapi
20+
21+
22+
class DelayedJSONHandler(BaseHTTPRequestHandler):
23+
delay = 0.05
24+
25+
def do_GET(self):
26+
time.sleep(self.delay)
27+
payload = json.dumps({"search_metadata": {"status": "Success"}}).encode()
28+
self.send_response(200)
29+
self.send_header("Content-Type", "application/json")
30+
self.send_header("Content-Length", str(len(payload)))
31+
self.end_headers()
32+
self.wfile.write(payload)
33+
34+
def log_message(self, format, *args):
35+
pass
36+
37+
38+
@contextmanager
39+
def delayed_server(delay):
40+
DelayedJSONHandler.delay = delay
41+
server = ThreadingHTTPServer(("127.0.0.1", 0), DelayedJSONHandler)
42+
thread = threading.Thread(target=server.serve_forever, daemon=True)
43+
thread.start()
44+
try:
45+
host, port = server.server_address
46+
yield f"http://{host}:{port}"
47+
finally:
48+
server.shutdown()
49+
server.server_close()
50+
thread.join()
51+
52+
53+
def measure_requests(base_url, count):
54+
session = requests.Session()
55+
session.trust_env = False
56+
try:
57+
started = time.perf_counter()
58+
for _ in range(count):
59+
response = session.get(f"{base_url}/search", timeout=10)
60+
response.raise_for_status()
61+
return time.perf_counter() - started
62+
finally:
63+
session.close()
64+
65+
66+
def measure_sync_client(base_url, count):
67+
client = serpapi.Client(trust_env=False, timeout=10)
68+
client.BASE_DOMAIN = base_url
69+
try:
70+
started = time.perf_counter()
71+
for index in range(count):
72+
client.search(q=f"query-{index}")
73+
return time.perf_counter() - started
74+
finally:
75+
client.close()
76+
77+
78+
async def measure_async_client(base_url, count):
79+
client = serpapi.AsyncClient(trust_env=False, timeout=10)
80+
async with client:
81+
client.BASE_DOMAIN = base_url
82+
started = time.perf_counter()
83+
await asyncio.gather(
84+
*(client.search(q=f"query-{index}") for index in range(count))
85+
)
86+
return time.perf_counter() - started
87+
88+
89+
def summarize(label, timings):
90+
median = statistics.median(timings)
91+
spread = f"{min(timings):.3f}-{max(timings):.3f}s"
92+
print(f"{label:<29} {median:.3f}s median ({spread})")
93+
return median
94+
95+
96+
def main():
97+
parser = argparse.ArgumentParser()
98+
parser.add_argument("--count", type=int, default=20)
99+
parser.add_argument("--delay", type=float, default=0.05)
100+
parser.add_argument("--repeats", type=int, default=3)
101+
args = parser.parse_args()
102+
if args.count < 1:
103+
parser.error("--count must be at least 1")
104+
if args.delay < 0:
105+
parser.error("--delay must not be negative")
106+
if args.repeats < 1:
107+
parser.error("--repeats must be at least 1")
108+
109+
with delayed_server(args.delay) as base_url:
110+
requests_timings = [
111+
measure_requests(base_url, args.count) for _ in range(args.repeats)
112+
]
113+
sync_timings = [
114+
measure_sync_client(base_url, args.count) for _ in range(args.repeats)
115+
]
116+
async_timings = [
117+
asyncio.run(measure_async_client(base_url, args.count))
118+
for _ in range(args.repeats)
119+
]
120+
121+
summarize("requests.Session sequential:", requests_timings)
122+
sync_median = summarize("serpapi.Client sequential:", sync_timings)
123+
async_median = summarize("AsyncClient concurrent:", async_timings)
124+
print(f"async vs sync speedup: {sync_median / async_median:.2f}x")
125+
126+
127+
if __name__ == "__main__":
128+
main()

‎docs/index.md‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ Run one of these commands in your terminal. Use `pip` to install into your Pytho
2020
:::{tab-item} pip
2121

2222
```bash
23-
pip install serpapi
23+
pip3 install serpapi
2424
```
2525

2626
:::
@@ -35,7 +35,7 @@ uv add serpapi
3535

3636
::::
3737

38-
The package requires Python 3.6 or newer.
38+
The package requires Python 3.8 or newer.
3939

4040
## First Request
4141

@@ -92,7 +92,7 @@ Use the same search parameter names as the [SerpApi API documentation](https://s
9292
- [Output Formats](user_guide/output-formats.md) explains when to use JSON, Markdown, or HTML.
9393
- [Migration Guide](user_guide/migrating-from-google-search-results.md) shows how to replace `google-search-results` with this package.
9494
- [Parameters and Engines](user_guide/parameters-and-engines.md) explains search parameters and how to test them in the [SerpApi Playground](https://serpapi.com/playground).
95-
- For scripts that collect results, see [pagination](user_guide/pagination.md), [timeouts and errors](user_guide/errors-and-timeouts.md), [request options](user_guide/request-options.md), and [account and locations](user_guide/account-and-locations.md).
95+
- For concurrent applications and scripts that collect results, see the [Asyncio Client](user_guide/asyncio.md), [pagination](user_guide/pagination.md), [timeouts and errors](user_guide/errors-and-timeouts.md), [request options](user_guide/request-options.md), and [account and locations](user_guide/account-and-locations.md).
9696
- For searches you retrieve later, selected response fields, and data retention settings, see [Async Search Archive](user_guide/async-search-archive.md), [JSON Restrictor](user_guide/json-restrictor.md), and [Zero Trace](user_guide/zero-trace.md).
9797
- [Examples](examples/index.md) includes searches for web pages, AI answers, local businesses, products, travel, finance, trends, jobs, media, apps, and research.
9898

@@ -132,6 +132,7 @@ user_guide/migrating-from-google-search-results
132132
:caption: Advanced Usage
133133
134134
user_guide/async-search-archive
135+
user_guide/asyncio
135136
user_guide/threading
136137
user_guide/multiprocessing
137138
user_guide/zero-trace

‎docs/user_guide/asyncio.md‎

Lines changed: 108 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,108 @@
1+
---
2+
title: "Asyncio Client"
3+
description: "Run concurrent SerpApi requests with AsyncClient and asyncio."
4+
---
5+
6+
# Asyncio Client
7+
8+
`serpapi.AsyncClient` provides non-blocking versions of the SDK methods for
9+
applications that use Python's `asyncio` event loop. This is separate from the
10+
[Async Search Archive](async-search-archive.md), which is a SerpApi API feature
11+
for retrieving a search after the server finishes processing it.
12+
13+
## Create and Close a Client
14+
15+
Use an async context manager so the connection pool is always closed:
16+
17+
```python
18+
import asyncio
19+
import os
20+
21+
import serpapi
22+
23+
24+
async def main():
25+
async with serpapi.AsyncClient(
26+
api_key=os.environ["SERPAPI_KEY"],
27+
timeout=20,
28+
) as client:
29+
results = await client.search(engine="google", q="coffee")
30+
print(results["search_metadata"]["id"])
31+
32+
33+
asyncio.run(main())
34+
```
35+
36+
When a context manager does not fit the application lifecycle, call
37+
`await client.aclose()` during shutdown. Create one client per event loop and
38+
reuse it instead of constructing a client for every request.
39+
40+
## Run Independent Searches Concurrently
41+
42+
`asyncio.gather()` lets other requests make progress while one request waits
43+
for network I/O:
44+
45+
```python
46+
async def search_many(client):
47+
return await asyncio.gather(
48+
client.search(engine="google", q="coffee"),
49+
client.search(engine="google", q="tea"),
50+
client.search(engine="google", q="pizza"),
51+
)
52+
```
53+
54+
Concurrency improves throughput for independent I/O-bound requests. It does
55+
not make one search finish faster, and each call still consumes a search from
56+
the account.
57+
58+
## Other Async Methods
59+
60+
The async client supports the same endpoint parameters and response formats as
61+
the synchronous client:
62+
63+
```python
64+
async def inspect_account(client):
65+
account = await client.account()
66+
locations = await client.locations(q="Austin", limit=3)
67+
archived = await client.search_archive(search_id="search-id")
68+
upload = await client.upload_image("image.png")
69+
return account, locations, archived, upload
70+
```
71+
72+
The example above is an illustrative fragment and assumes it runs inside the
73+
same async function and client context as the first example.
74+
75+
## Async Pagination
76+
77+
JSON search responses are `AsyncSerpResults` objects. Fetch one more page with
78+
`await`, or iterate through several pages with `async for`:
79+
80+
```python
81+
async def read_pages(results):
82+
next_page = await results.next_page()
83+
84+
async for page in results.yield_pages(max_pages=5):
85+
for item in page.get("organic_results", []):
86+
print(item.get("title"))
87+
88+
return next_page
89+
```
90+
91+
## Error Handling
92+
93+
Async methods raise the same SerpApi exceptions as synchronous methods:
94+
95+
```python
96+
async def safe_search(client):
97+
try:
98+
return await client.search(engine="google", q="coffee")
99+
except serpapi.TimeoutError:
100+
print("The request timed out.")
101+
except serpapi.HTTPConnectionError:
102+
print("Could not connect to SerpApi.")
103+
except serpapi.HTTPError as exc:
104+
print(exc.status_code, exc.error)
105+
```
106+
107+
See [Errors and Timeouts](errors-and-timeouts.md) and
108+
[Request Options](request-options.md) for shared configuration details.

0 commit comments

Comments
 (0)