Merge branch 'main' into pydoc-repl

Author: StanFromIreland

Doc/builtins/exceptions.rst

@@ -396,10 +396,9 @@ The following exceptions are the exceptions that are usually raised.
 
    .. attribute:: strerror
 
-      The corresponding error message, as provided by
-      the operating system.  It is formatted by the C
-      functions :c:func:`!perror` under POSIX, and :c:func:`!FormatMessage`
-      under Windows.
+      The corresponding error message, as provided by the operating system.
+      Under POSIX, it is obtained using the C functions :c:func:`!strerror`
+      or :c:func:`!strerror_r`; under Windows, :c:func:`!FormatMessage` is used.
 
    .. attribute:: filename
                   filename2
@@ -567,11 +566,6 @@ The following exceptions are the exceptions that are usually raised.
       The column in the end line where the error occurred finishes. This is
       1-indexed: the first character in the line has an ``offset`` of 1.
 
-   For errors in f-string fields, the message is prefixed by "f-string: "
-   and the offsets are offsets in a text constructed from the replacement
-   expression.  For example, compiling f'Bad {a b} field' results in this
-   args attribute: ('f-string: ...', ('', 1, 2, '(a b)\n', 1, 5)).
-
    .. versionchanged:: 3.10
       Added the :attr:`end_lineno` and :attr:`end_offset` attributes.
 

Doc/builtins/functions.rst

@@ -772,7 +772,7 @@ are always available.  They are listed here in alphabetical order.
    still access all builtins.
 
    The *closure* argument specifies a closure--a tuple of cellvars.
-   It's only valid when the *object* is a code object containing
+   It's only valid when *source* is a code object containing
    :term:`free (closure) variables <closure variable>`.
    The length of the tuple must exactly match the length of the code object's
    :attr:`~codeobject.co_freevars` attribute.
@@ -2166,10 +2166,10 @@ are always available.  They are listed here in alphabetical order.
 
    When called directly within an ordinary method of a class, both arguments may
    be omitted ("zero-argument :func:`!super`"). In this case, *type* will be the
-   enclosing class, and *obj* will be the first argument of the immediately
-   enclosing function (typically ``self``). (This means that zero-argument
+   enclosing class, and *object_or_type* will be the first argument of the
+   immediately enclosing function (typically ``self``). Zero-argument
    :func:`!super` will not work as expected within nested functions, including
-   generator expressions, which implicitly create nested functions.)
+   generator expressions, which implicitly create nested functions.
 
    There are two typical use cases for *super*.  In a class hierarchy with
    single inheritance, *super* can be used to refer to parent classes without
@@ -2428,10 +2428,9 @@ are always available.  They are listed here in alphabetical order.
    determine the package context of the :keyword:`import` statement.
 
    *level* specifies whether to use absolute or relative imports. ``0`` (the
-   default) means only perform absolute imports.  Positive values for
-   *level* indicate the number of parent directories to search relative to the
-   directory of the module calling :func:`__import__` (see :pep:`328` for the
-   details).
+   default) means only perform absolute imports.  For relative imports,
+   ``1`` refers to the current package, ``2`` to its parent package, and so on
+   (see :pep:`328` for details).
 
    When the *name* variable is of the form ``package.module``, normally, the
    top-level package (the name up till the first dot) is returned, *not* the

Doc/builtins/stdtypes.rst

@@ -5667,7 +5667,7 @@ can be used interchangeably to index the same dictionary entry.
 
    * Use a comma-separated list of ``key: value`` pairs within braces:
      ``{'jack': 4098, 'sjoerd': 4127}`` or ``{4098: 'jack', 4127: 'sjoerd'}``
-   * Use a dict comprehension: ``{}``, ``{x: x ** 2 for x in range(10)}``
+   * Use a dict comprehension: ``{x: x ** 2 for x in range(10)}``
    * Use the type constructor: ``dict()``,
      ``dict([('foo', 100), ('bar', 200)])``, ``dict(foo=100, bar=200)``
 
@@ -5878,9 +5878,8 @@ can be used interchangeably to index the same dictionary entry.
       Return a new view of the dictionary's values.  See the
       :ref:`documentation of view objects <dict-views>`.
 
-      An equality comparison between one ``dict.values()`` view and another
-      will always return ``False``. This also applies when comparing
-      ``dict.values()`` to itself::
+      An equality comparison between two distinct ``dict.values()`` views
+      will always return ``False``, even for views of the same dictionary::
 
          >>> d = {'a': 1}
          >>> d.values() == d.values()

Doc/library/inspect.rst

@@ -284,6 +284,12 @@ attributes (see :ref:`import-mod-attrs` for module attributes):
 |                 |                   | ``AGEN_SUSPENDED``, or    |
 |                 |                   | ``AGEN_CLOSED``           |
 +-----------------+-------------------+---------------------------+
+| async generator | aga_gen           | async generator driven by |
+| awaitable       |                   | the awaitable returned    |
+|                 |                   | from ``asend()``,         |
+|                 |                   | ``athrow()``, or          |
+|                 |                   | ``aclose()``              |
++-----------------+-------------------+---------------------------+
 | coroutine       | __name__          | name                      |
 +-----------------+-------------------+---------------------------+
 |                 | __qualname__      | qualified name            |
@@ -360,6 +366,11 @@ attributes (see :ref:`import-mod-attrs` for module attributes):
    Add ``gi_state`` attribute to generators, ``cr_state`` attribute to
    coroutines, and ``ag_state`` attribute to async generators.
 
+.. versionchanged:: 3.16
+
+   Add ``aga_gen`` attribute to the awaitables returned by the ``asend()``,
+   ``athrow()`` and ``aclose()`` methods of async generators.
+
 .. function:: getmembers(object[, predicate])
 
    Return all the members of an object in a list of ``(name, value)``

Lib/asyncio/graph.py

@@ -70,6 +70,9 @@ def _build_graph_for_future(
             if coro.gi_frame is not None:
                 st.append(FrameCallGraphEntry(coro.gi_frame))
             coro = coro.gi_yieldfrom
+        elif hasattr(coro, 'aga_gen'):
+            # An asend()/athrow() awaitable: step over it to its generator
+            coro = coro.aga_gen
         else:
             break
 

Lib/asyncio/streams.py

@@ -553,6 +553,16 @@ async def _wait_for_data(self, func_name):
         finally:
             self._waiter = None
 
+    def _readuntil_fast(self, separator):
+        # Fast linear search inside already buffered data.
+        if type(separator) is bytes and separator and self._exception is None:
+            isep = self._buffer.find(separator)
+            if 0 <= isep <= self._limit:
+                chunk = self._buffer.take_bytes(isep + len(separator))
+                self._maybe_resume_transport()
+                return chunk
+        return None
+
     async def readline(self):
         r"""Read chunk of data from the stream until newline (b'\n') is found.
 
@@ -570,19 +580,18 @@ async def readline(self):
         needed.
         """
         sep = b'\n'
-        seplen = len(sep)
+        if (line := self._readuntil_fast(sep)) is not None:
+            return line
         try:
-            line = await self.readuntil(sep)
+            return await self.readuntil(sep)
         except exceptions.IncompleteReadError as e:
             return e.partial
         except exceptions.LimitOverrunError as e:
-            if self._buffer.startswith(sep, e.consumed):
-                del self._buffer[:e.consumed + seplen]
-            else:
-                self._buffer.clear()
+            # e.consumed is the newline's index, or len(self._buffer) if
+            # there is none: drop the line, or everything.
+            del self._buffer[:e.consumed + 1]
             self._maybe_resume_transport()
             raise ValueError(e.args[0])
-        return line
 
     async def readuntil(self, separator=b'\n'):
         """Read data from the stream until ``separator`` is found.
@@ -610,17 +619,18 @@ async def readuntil(self, separator=b'\n'):
         the shortest possible separator is considered to be the one that
         matched.
         """
+        if (chunk := self._readuntil_fast(separator)) is not None:
+            return chunk
         if isinstance(separator, tuple):
             # Makes sure shortest matches wins
             separator = sorted(separator, key=len)
         else:
             separator = [separator]
         if not separator:
             raise ValueError('Separator should contain at least one element')
-        min_seplen = len(separator[0])
-        max_seplen = len(separator[-1])
-        if min_seplen == 0:
+        if len(separator[0]) == 0:
             raise ValueError('Separator should be at least one-byte string')
+        max_seplen = len(separator[-1])
 
         if self._exception is not None:
             raise self._exception
@@ -650,32 +660,23 @@ async def readuntil(self, separator=b'\n'):
         # or an EOF has happened.
         while True:
             buflen = len(self._buffer)
-
-            # Check if we now have enough data in the buffer for shortest
-            # separator to fit.
-            if buflen - offset >= min_seplen:
-                match_start = None
-                match_end = None
-                for sep in separator:
-                    isep = self._buffer.find(sep, offset)
-
-                    if isep != -1:
-                        # `separator` is in the buffer. `match_start` and
-                        # `match_end` will be used later to retrieve the
-                        # data.
-                        end = isep + len(sep)
-                        if match_end is None or end < match_end:
-                            match_end = end
-                            match_start = isep
-                if match_end is not None:
-                    break
-
-                # see upper comment for explanation.
-                offset = max(0, buflen + 1 - max_seplen)
-                if offset > self._limit:
-                    raise exceptions.LimitOverrunError(
-                        'Separator is not found, and chunk exceed the limit',
-                        offset)
+            match_end = buflen + 1
+            for sep in separator:
+                # Only a match that ends before the best one so far wins,
+                # so don't search past it.
+                isep = self._buffer.find(sep, offset, match_end - 1)
+                if isep != -1:
+                    match_start = isep
+                    match_end = isep + len(sep)
+            if match_end <= buflen:
+                break
+
+            # see upper comment for explanation.
+            offset = max(0, buflen + 1 - max_seplen)
+            if offset > self._limit:
+                raise exceptions.LimitOverrunError(
+                    'Separator is not found, and chunk exceed the limit',
+                    offset)
 
             # Complete message (with full separator) may be present in buffer
             # even when EOF flag is set. This may happen when the last chunk

Lib/test/exception_hierarchy.txt

@@ -14,7 +14,7 @@ BaseException
       ├── EOFError
       ├── ExceptionGroup [BaseExceptionGroup]
       ├── ImportError
-      │    └── ImportCycleError
+      │    ├── ImportCycleError
       │    └── ModuleNotFoundError
       ├── LookupError
       │    ├── IndexError

Lib/test/test_asyncgen.py

@@ -399,6 +399,23 @@ async def gen():
                 r"cannot reuse already awaited __anext__\(\)/asend\(\)"):
             an.send(None)
 
+    def test_async_gen_asend_athrow_aga_gen(self):
+        # gh-160837: asend()/athrow() awaitables expose the generator they drive
+        async def gen():
+            yield 1
+
+        g = gen()
+        asend = g.asend(None)
+        athrow = g.athrow(ValueError)
+        try:
+            self.assertIs(asend.aga_gen, g)
+            self.assertIs(athrow.aga_gen, g)
+            with self.assertRaises(AttributeError):
+                asend.aga_gen = None
+        finally:
+            asend.close()
+            athrow.close()
+
     def test_async_gen_asend_throw_concurrent_with_send(self):
         import types
 

Lib/test/test_asyncio/test_graph.py

@@ -430,6 +430,77 @@ async def probe(t):
             ['T<probe>', ['a _cancel_and_wait', 'a wait_for', 'a probe'], []],
         ])
 
+    async def test_stack_async_gen_asend(self):
+        # gh-160837: an async_generator_asend must not truncate the stack
+        fut = asyncio.Future()
+        stack_for_consumer = None
+
+        async def deep():
+            await fut
+
+        async def gen():
+            await deep()
+            yield 1
+
+        async def consumer():
+            async for _ in gen():
+                pass
+
+        async def main():
+            nonlocal stack_for_consumer
+
+            async with asyncio.TaskGroup() as g:
+                t = g.create_task(consumer(), name='consumer')
+                for _ in range(5):
+                    await asyncio.sleep(0)
+
+                stack_for_consumer = capture_test_stack(fut=t)
+                fut.set_result(None)
+
+        await main()
+
+        self.assertEqual(stack_for_consumer[0][:2], [
+            'T<consumer>',
+            ['a deep', 'ag gen', 'a consumer'],
+        ])
+
+    async def test_stack_async_gen_aclose(self):
+        # gh-160837: an async_generator_athrow must not truncate the stack
+        fut = asyncio.Future()
+        stack_for_consumer = None
+
+        async def deep():
+            await fut
+
+        async def gen():
+            try:
+                yield 1
+            finally:
+                await deep()
+
+        async def consumer():
+            agen = gen()
+            await anext(agen)
+            await agen.aclose()
+
+        async def main():
+            nonlocal stack_for_consumer
+
+            async with asyncio.TaskGroup() as g:
+                t = g.create_task(consumer(), name='consumer')
+                for _ in range(5):
+                    await asyncio.sleep(0)
+
+                stack_for_consumer = capture_test_stack(fut=t)
+                fut.set_result(None)
+
+        await main()
+
+        self.assertEqual(stack_for_consumer[0][:2], [
+            'T<consumer>',
+            ['a deep', 'ag gen', 'a consumer'],
+        ])
+
     async def test_stack_gather(self):
 
         stack_for_deep = None

Lib/test/test_asyncio/test_streams.py

@@ -592,6 +592,14 @@ def test_exception(self):
         stream.set_exception(exc)
         self.assertIs(stream.exception(), exc)
 
+        # Raised even when the data is already buffered.
+        stream.feed_data(b'line\n')
+        self.assertRaises(
+            ValueError, self.loop.run_until_complete, stream.readline())
+        self.assertRaises(
+            ValueError, self.loop.run_until_complete, stream.readuntil())
+        self.assertEqual(b'line\n', stream._buffer)
+
     def test_exception_waiter(self):
         stream = asyncio.StreamReader(loop=self.loop)