[Spanner] Retries of UNAVAILABLE errors when resuming a result stream not working properly

[taka-oyama]

Result::rows() can retry UNAVAILABLE errors forever, when the server keeps returning UNAVAILABLE after the first stream is interrupted. The retry limit from RetrySettings (default 3) is never applied.

This issue is a bit hard to explain but here goes...

The bit of code that needs fixing is here.

google-cloud-php/Spanner/src/Result.php

Line 182 in c891a48

$this->generator = $backoff->execute($call, [$this->resumeToken, $this->transaction()]);

This block is called when a ServiceException is thrown with status UNAVAILABLE while the Generator returned from $call is iterating. I think the expected behavior here is to re-execute the $call, reconnect and resume the iterating process. At first glance it seemed like it's doing exactly that.
But when I actually run it, I noticed that when $call is actually invoked in ExponentialBackoff::execute($call, ...), it's just returning the Generator that is not started (see below).

google-cloud-php/Core/src/ExponentialBackoff.php

Line 97 in c891a48

return call_user_func_array($function, $arguments);

Since the code inside the $call is not executed, the unstarted generator gets incorrectly assigned and the loop is continued. When $this->generator->current() is called again (Result.php:L138), and if another ServiceException (with status UNAVAILABLE) is thrown again, the whole process is repeated.

I think the expected behavior here is to have the generator start when $backoff->execute(...) is run so that backoff can actually be applied.

Result::createGenerator(...) run with a similar logic, but there the generator is started inside the callback so it runs correctly like so...

google-cloud-php/Spanner/src/Result.php

Lines 480 to 483 in c891a48

	$valid = $backoff->execute(function () use ($call, &$generator) {
	$generator = $call();
	return $generator->valid();
	});

So I think this can be fixed by applying by changing...

$this->generator = $backoff->execute($call, [$this->resumeToken, $this->transaction()]);

to

$this->generator = $backoff->execute(function () use ($call) {
    $generator = $call();
    $generator->valid();
    return $generator;
});

Environment details

• OS: Alpine Linux
• PHP version: 8.5.10
• Package name and version: google/cloud-spanner v2.11.0

Steps to reproduce

1. Run a large query using Database::execute(...)
2. Call rows()
3. Start iterating.
4. Network disconnects and throw UNAVAILABLE midway.