Skip to content

Commit 69087a7

Browse files
masklinnmasklinn
committed
Typed API & parsers API
New API with full typing ======================== Seems pretty self-explanatory, rather than returning somewhat ad-hoc dicts this API works off of dataclasses, it should be compatible with the legacy version through the magic of ~~buying two of them~~ `dataclasses.asdict`. Parser API ========== The legacy version had "parsers" which really represent individual parsing rules. In the new API the job of a parser is what the top-level functions did, they wrap around the entire job of parsing a user-agent string. The core API is just `__call__`, with a selection flag for the domains (seems like the least bad term for what "user agent", "os", and "device" are, other alternatives I considered are "component" and "category", but I'm still ambivalent). Overridable helpers are provided which match the old API's methods (with PEP8 conventions), as well as the same style of helpers at the package toplevel. This resolves a number of limitations: Concurrency ----------- While the library should be thread-safe (and I need to find a way to test that) the ability to instantiate parsers should provide the opportunity for things like thread-local parsers, or actual parallelism if we start using native extensions (regex, re2). It also allows running multiple *parser configurations* concurrently, including e.g. multiple independent custom yaml sets. Not sure there's a use for it, but why not? At the very least it should make using custom YAML datasets much easier than having to set envvars. The caching parser being stateful, it's protected by an optional lock seems like the best way to make caching thread-safe. When only using a single thread, or using thread-local parsers, caching can be disabled by using a `contextlib.nullcontext` as lock. Customization ------------- Public APIs are provided both to instantiate and tune parsers, and to set the global parser. Hopefully this makes evaluating proposed parsers as well as evaluating & tuning caches (algorithm & size) easier. Even more so as we should provide some sort of evaluation CLI in #163. Caches ------ In the old API, package-provided API could only be global and with a single implementation as it had to integrate with the toplevel parsing functions. By reifying the parsing job, a cache is just a parser which delegates the parse if it doesn't have a hit. This allows more easily providing, testing, and evolving alternative cache strategies. Bulk APIs --------- The current parser checks rules (regexes) one at a time on the input, but there are advanced regex APIs which can check a regex *set* and return which one(s) matched, allowing much more efficicent bulk matching e.g. google's re2, rust's regex. With the old scheme, this would be a pretty significant change in use / behaviour, obviating the use of the "parsers" with no recourse. Under the new parsing scheme, these can just be different "base" parsers, they can be the default, they can be cached, and users can instantiate their own parser instead. Misc ---- The new API's UA extractor pipeline supports `patch_minor`, though that requires excluding that bit from the tests as there are apparently broken test cases around that item (ua-parser/uap-core#562). Init Helpers ============ Having proper parsers is the opportunity to allow setting parsers at runtime more easily (instead of load-time envvars), however optional constructors (classmethods) turns out to be iffy from an API and typing perspective both. Instead have the "base" parsers (the ones doing the actual parsing of the UAs) just take a uniform parsed data set, and have utility loaders provide that from various data sources (precompiled, preformatted, or data files). This avoids redundancy and the need for mixins / inheritance, and mypy is *much* happier. Legacy Parsers -> New Matchers ============================== The bridging of the legacy parsers and the new results turned out to be pretty mid. Instead, the new API relies on similar but better typed matcher classes, with a slightly different API: they return `None` on a match failure instead of a triplet, which make them compose better in iteration (e.g. can just `filter` them out). Add a `Matchers` alias to carry them around (a tuple of lists of matchers) for convenience, as well as as base parser parameter. Also clarify the replacer rules, and hopefully implement the thing more clearly. Fixes #93, fixes #142, closes #116
1 parent 6ad6f04 commit 69087a7

14 files changed

Lines changed: 1357 additions & 97 deletions

‎README.rst‎

Lines changed: 83 additions & 75 deletions
Original file line numberDiff line numberDiff line change
@@ -1,119 +1,127 @@
11
uap-python
22
==========
33

4-
A python implementation of the UA Parser (https://github.com/ua-parser,
5-
formerly https://github.com/tobie/ua-parser)
4+
Official python implementation of the `User Agent String
5+
Parser <https://github.com/ua-parser>`_ project.
66

77
Build Status
88
------------
99

1010
.. image:: https://github.com/ua-parser/uap-python/actions/workflows/ci.yml/badge.svg
1111
:alt: CI on the master branch
1212

13-
1413
Installing
1514
----------
1615

17-
Install via pip
18-
~~~~~~~~~~~~~~~
19-
20-
Just run:
16+
Just add ``ua-parser`` to your project's dependencies, or run
2117

2218
.. code-block:: sh
2319
2420
$ pip install ua-parser
2521
26-
Manual install
27-
~~~~~~~~~~~~~~
28-
29-
In the top-level directory run:
30-
31-
.. code-block:: sh
32-
33-
$ python setup.py install
34-
35-
Change Log
36-
---------------
37-
Because this repo is mostly a python wrapper for the User Agent String Parser repo (https://github.com/ua-parser/uap-core), the changes made to this repo are best described by the update diffs in that project. Please see the diffs for this submodule (https://github.com/ua-parser/uap-core/releases) for a list of what has changed between versions of this package.
22+
to install in the current environment.
3823

3924
Getting Started
4025
---------------
4126

42-
Retrieve data on a user-agent string
43-
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
27+
Retrieve all data on a user-agent string
28+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4429

4530
.. code-block:: python
4631
47-
>>> from ua_parser import user_agent_parser
48-
>>> import pprint
49-
>>> pp = pprint.PrettyPrinter(indent=4)
32+
>>> from ua_parser import parse
5033
>>> ua_string = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.104 Safari/537.36'
51-
>>> parsed_string = user_agent_parser.Parse(ua_string)
52-
>>> pp.pprint(parsed_string)
53-
{ 'device': {'brand': 'Apple', 'family': 'Mac', 'model': 'Mac'},
54-
'os': { 'family': 'Mac OS X',
55-
'major': '10',
56-
'minor': '9',
57-
'patch': '4',
58-
'patch_minor': None},
59-
'string': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) '
60-
'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.104 '
61-
'Safari/537.36',
62-
'user_agent': { 'family': 'Chrome',
63-
'major': '41',
64-
'minor': '0',
65-
'patch': '2272'}}
66-
67-
Extract browser data from user-agent string
68-
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
34+
>>> parse(ua_string) # doctest: +NORMALIZE_WHITESPACE, +ELLIPSIS
35+
ParseResult(user_agent=UserAgent(family='Chrome',
36+
major='41',
37+
minor='0',
38+
patch='2272',
39+
patch_minor='104'),
40+
os=OS(family='Mac OS X',
41+
major='10',
42+
minor='9',
43+
patch='4',
44+
patch_minor=None),
45+
device=Device(family='Mac',
46+
brand='Apple',
47+
model='Mac'),
48+
string='Mozilla/5.0 (Macintosh; Intel Mac OS...
49+
50+
Any datum not found in the user agent string is set to ``None``::
51+
52+
>>> parse("")
53+
ParseResult(user_agent=None, os=None, device=None, string='')
54+
55+
Extract only browser data from user-agent string
56+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6957
7058
.. code-block:: python
7159
72-
>>> from ua_parser import user_agent_parser
73-
>>> import pprint
74-
>>> pp = pprint.PrettyPrinter(indent=4)
60+
>>> from ua_parser import parse_user_agent
7561
>>> ua_string = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.104 Safari/537.36'
76-
>>> parsed_string = user_agent_parser.ParseUserAgent(ua_string)
77-
>>> pp.pprint(parsed_string)
78-
{'family': 'Chrome', 'major': '41', 'minor': '0', 'patch': '2272'}
62+
>>> parse_user_agent(ua_string)
63+
UserAgent(family='Chrome', major='41', minor='0', patch='2272', patch_minor='104')
7964
80-
..
65+
For specific domains, a match failure just returns ``None``::
8166
82-
⚠️Before 0.15, the convenience parsers (``ParseUserAgent``,
83-
``ParseOs``, and ``ParseDevice``) were not cached, which could
84-
result in degraded performances when parsing large amounts of
85-
identical user-agents (which might occur for real-world datasets).
86-
87-
For these versions (up to 0.10 included), prefer using ``Parse``
88-
and extracting the sub-component you need from the resulting
89-
dictionary.
67+
>>> parse_user_agent("")
9068
9169
Extract OS information from user-agent string
9270
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
9371
9472
.. code-block:: python
9573
96-
>>> from ua_parser import user_agent_parser
97-
>>> import pprint
98-
>>> pp = pprint.PrettyPrinter(indent=4)
74+
>>> from ua_parser import parse_os
9975
>>> ua_string = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.104 Safari/537.36'
100-
>>> parsed_string = user_agent_parser.ParseOS(ua_string)
101-
>>> pp.pprint(parsed_string)
102-
{ 'family': 'Mac OS X',
103-
'major': '10',
104-
'minor': '9',
105-
'patch': '4',
106-
'patch_minor': None}
107-
108-
Extract Device information from user-agent string
76+
>>> parse_os(ua_string)
77+
OS(family='Mac OS X', major='10', minor='9', patch='4', patch_minor=None)
78+
79+
Extract device information from user-agent string
10980
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11081
11182
.. code-block:: python
11283
113-
>>> from ua_parser import user_agent_parser
114-
>>> import pprint
115-
>>> pp = pprint.PrettyPrinter(indent=4)
84+
>>> from ua_parser import parse_device
11685
>>> ua_string = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.104 Safari/537.36'
117-
>>> parsed_string = user_agent_parser.ParseDevice(ua_string)
118-
>>> pp.pprint(parsed_string)
119-
{'brand': 'Apple', 'family': 'Mac', 'model': 'Mac'}
86+
>>> parse_device(ua_string)
87+
Device(family='Mac', brand='Apple', model='Mac')
88+
89+
Parser
90+
~~~~~~
91+
92+
Parsers expose the same functions (``parse``, ``parse_user_agent``,
93+
``parse_os``, and ``parse_device``) as the top-level of the package,
94+
however these are all *utility* methods.
95+
96+
The actual protocol of parsers, and the one method which must be
97+
implemented / overridden is::
98+
99+
def __call__(self, str, Components, /) -> ParseResult:
100+
101+
It's similar to but more flexible than ``parse``:
102+
103+
- The ``str`` is the user agent string.
104+
- The ``Components`` is a hint, through which the caller requests the
105+
domain (component) they are looking for, any combination of
106+
``Components.USER_AGENT``, ``Components.OS``, and
107+
``Components.DEVICE``. ``Domains.ALL`` exists as a convenience alias
108+
for the combination of all three.
109+
110+
The parser *must* return at least the requested information, but if
111+
that's more convenient or no more expensive it *can* return more.
112+
- The ``ParseResult`` is similar to ``CompleteParseResult``, except
113+
all the attributes are ``Optional`` and it has a ``components:
114+
Components`` attribute which specifies whether a component was never
115+
requested (its value for the user agent string is unknown) or it has
116+
been requested but could not be resolved (no match was found for the
117+
user agent).
118+
119+
``ParseResult.complete()`` convert to a ``CompleteParseResult`` if
120+
all the components are set, and raise an exception otherwise. If
121+
some of the components are set to ``None``, they'll be swapped for a
122+
default value.
123+
124+
Calling the parser directly is part of the public API. One of the
125+
advantage is that it does not return default values, as such it allows
126+
more easily differentiating between a non-match (= ``None``) and a
127+
default fallback (``family = "Other"``).

‎pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
55
[project]
66
name = "ua-parser"
77
description = "Python port of Browserscope's user agent parser"
8-
version = "1.0.0a"
8+
version = "1.0.0a1"
99
readme = "README.rst"
1010
requires-python = ">=3.8"
1111
dependencies = []

‎setup.py‎

Lines changed: 61 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
11
#!/usr/bin/env python
22
# flake8: noqa
3+
import io
34
from contextlib import suppress
45
from os import fspath
56
from pathlib import Path
@@ -51,6 +52,13 @@ def run(self) -> None:
5152
f"Unable to find regexes.yaml, should be at {yaml_src!r}"
5253
)
5354

55+
def write_matcher(f, typ: str, fields: List[Optional[object]]):
56+
f.write(f" {typ}(".encode())
57+
while len(fields) > 1 and fields[-1] is None:
58+
fields = fields[:-1]
59+
f.write(", ".join(map(repr, fields)).encode())
60+
f.write(b"),\n")
61+
5462
def write_params(fields):
5563
# strip trailing None values
5664
while len(fields) > 1 and fields[-1] is None:
@@ -70,10 +78,20 @@ def write_params(fields):
7078
outdir = dist_dir / self.pkg_name
7179
outdir.mkdir(parents=True, exist_ok=True)
7280

73-
dest = outdir / "_regexes.py"
81+
dest = outdir / "_matchers.py"
82+
dest_legacy = outdir / "_regexes.py"
7483

75-
with dest.open("wb") as fp:
84+
with dest.open("wb") as f, dest_legacy.open("wb") as fp:
7685
# fmt: off
86+
f.write(b"""\
87+
########################################################
88+
# NOTICE: this file is autogenerated from regexes.yaml #
89+
########################################################
90+
91+
from .core import Matchers, UserAgentMatcher, OSMatcher, DeviceMatcher
92+
93+
MATCHERS: Matchers = ([
94+
""")
7795
fp.write(b"# -*- coding: utf-8 -*-\n")
7896
fp.write(b"########################################################\n")
7997
fp.write(b"# NOTICE: This file is autogenerated from regexes.yaml #\n")
@@ -87,31 +105,35 @@ def write_params(fields):
87105
fp.write(b"\n")
88106
fp.write(b"USER_AGENT_PARSERS = [\n")
89107
for device_parser in regexes["user_agent_parsers"]:
90-
fp.write(b" UserAgentParser(\n")
91-
write_params([
108+
write_matcher(f, "UserAgentMatcher", [
92109
device_parser["regex"],
93110
device_parser.get("family_replacement"),
94111
device_parser.get("v1_replacement"),
95112
device_parser.get("v2_replacement"),
96113
])
97-
fp.write(b" ),\n")
98-
fp.write(b"]\n")
99-
fp.write(b"\n")
100-
fp.write(b"DEVICE_PARSERS = [\n")
101-
for device_parser in regexes["device_parsers"]:
102-
fp.write(b" DeviceParser(\n")
114+
115+
fp.write(b" UserAgentParser(\n")
103116
write_params([
104117
device_parser["regex"],
105-
device_parser.get("regex_flag"),
106-
device_parser.get("device_replacement"),
107-
device_parser.get("brand_replacement"),
108-
device_parser.get("model_replacement"),
118+
device_parser.get("family_replacement"),
119+
device_parser.get("v1_replacement"),
120+
device_parser.get("v2_replacement"),
109121
])
110122
fp.write(b" ),\n")
111-
fp.write(b"]\n")
112-
fp.write(b"\n")
123+
f.write(b" ], [\n")
124+
fp.write(b"]\n\n")
125+
113126
fp.write(b"OS_PARSERS = [\n")
114127
for device_parser in regexes["os_parsers"]:
128+
write_matcher(f, "OSMatcher", [
129+
device_parser["regex"],
130+
device_parser.get("os_replacement"),
131+
device_parser.get("os_v1_replacement"),
132+
device_parser.get("os_v2_replacement"),
133+
device_parser.get("os_v3_replacement"),
134+
device_parser.get("os_v4_replacement"),
135+
])
136+
115137
fp.write(b" OSParser(\n")
116138
write_params([
117139
device_parser["regex"],
@@ -122,6 +144,29 @@ def write_params(fields):
122144
device_parser.get("os_v4_replacement"),
123145
])
124146
fp.write(b" ),\n")
147+
f.write(b" ], [\n")
148+
fp.write(b"]\n\n")
149+
150+
fp.write(b"DEVICE_PARSERS = [\n")
151+
for device_parser in regexes["device_parsers"]:
152+
write_matcher(f, "DeviceMatcher", [
153+
device_parser["regex"],
154+
device_parser.get("regex_flag"),
155+
device_parser.get("device_replacement"),
156+
device_parser.get("brand_replacement"),
157+
device_parser.get("model_replacement"),
158+
])
159+
160+
fp.write(b" DeviceParser(\n")
161+
write_params([
162+
device_parser["regex"],
163+
device_parser.get("regex_flag"),
164+
device_parser.get("device_replacement"),
165+
device_parser.get("brand_replacement"),
166+
device_parser.get("model_replacement"),
167+
])
168+
fp.write(b" ),\n")
169+
f.write(b"])\n")
125170
fp.write(b"]\n")
126171
# fmt: on
127172

0 commit comments

Comments
 (0)