Streaming data to BQ using Storage Write API not possible

[andriussev]

Environment details

• OS: Windows/Mac
• PHP version: 8.2/8.3
• Package name and version: google/cloud-bigquery-storage v2.1.3

Steps to reproduce

It seems to be impossible write data because of descriptor_fix.php (though it might be solving other issues).

I couldn't find any other way to use AppendRows other than using Proto-stuffs functionality.

The provided sample does not relieve the pain of explaining how data is sent.

There are mentions of fixes and AppendsRows, mainly by @bshaffer , but I could not find any information on how to move forward without using Proto-stuffs.

Code example

// prerequisites
$bigQueryWriteClient = new BigQueryWriteClient();
$parent = 'pathhere';

// setup the structure
$row = new Struct();
$row->setFields([
	'impressions' => (new Value())->setNumberValue(213),
]);
$protoRows = new ProtoRows();
$protoRows->setSerializedRows([$row->serializeToString()]);
// add the data
$protoData = new ProtoData([
	'rows' => $protoRows,
]);
// set the schema, which requires the descriptor
$protoData->setWriterSchema(new ProtoSchema([
	'proto_descriptor' => new \Google\Protobuf\DescriptorProto(),
]));

// create the append rows request
$appendRowsRequest = new AppendRowsRequest([
	'write_stream' => $parent . '/streams/_default',
	'proto_rows' => $protoData
]);

// do the writing
$stream = $bigQueryWriteClient->appendRows();
$stream->write($appendRowsRequest);

// finalize
foreach ($stream->closeWriteAndReadAll() as $element) {
	printf('Element data: %s' . PHP_EOL, $element->serializeToJsonString());
}

[jlindenbaum]

Did you ever get this working?

[aidar87]

+1
Also facing this issue

[inphekd]

Any progress on this one?

[bshaffer]

After looking into this, I do not believe there's any way for PHP to support this call until we can support Protobuf Editions.

The descriptor_fix.php is there because the ProtoSchema contains a "proto 2" field, and Protobuf for PHP does not support proto 2.

I commented out the reference to ProtoSchema, and I get the error from the API "Expect a user schema to be passed in", which seems fairly conclusive. So we can't call the API without the schema, and we can't create the schema without proto 2 support.

We hope to have support for this feature in PHP protobuf by EOY

/cc @haberman

[arturaslaz]

@bshaffer it's been end of year now. Do you have any updates on this?

[bshaffer]

@arturaslaz unfortunately no, PHP still does not support protobuf editions and it is not prioritized