Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions docs/cn/urma.md
Original file line number Diff line number Diff line change
Expand Up @@ -256,3 +256,46 @@ URMA 恢复及 64 KiB 附件回显。正常短连接关闭也在同一测试中
`--urma_copy_send=true` 将每个数据 WR 合并到 64 字节对齐的注册缓冲区,并持有至完成,
增加一次 CPU 拷贝。两端保持设备、队列和消息大小一致,对比原发送路径与复制路径。
初始化日志记录 `copy_send`,失败日志记录完成方向、槽位及注册段范围。

## 业务 SL 与优先级

`ChannelOptions.urma_qos` 控制请求发送方向,`ServerOptions.urma_qos` 控制响应发送方向:

```cpp
brpc::ChannelOptions client_options;
client_options.socket_mode = brpc::SOCKET_MODE_URMA;
client_options.urma_qos.service_level = 2;

brpc::ServerOptions server_options;
server_options.socket_mode = brpc::SOCKET_MODE_URMA;
server_options.urma_qos.service_level = 2;
```

`--urma_tp_type=auto` 按设备能力优先选择 CTP,其次 RTP;`ctp`、`rtp` 显式选择路径。
两端使用相同路径,进程初始化后保持固定。设备未提供能力表时,auto/ctp 使用 priority 6,
rtp 使用 priority 0。

| 配置 | 默认 | 显式范围 | 含义 |
| --- | --- | --- | --- |
| `service_level` | -1 | 0–255 | 选择当前路径下首个映射到该 SL 的 priority |
| `priority` | -1 | 0–15 | 指定当前路径支持的 priority |

两项均为 -1 时使用设备默认 priority;同时指定时须匹配同一设备映射。
不支持或冲突时,Channel 初始化、Server 启动报错,并列出当前路径支持的 priority:SL。
例如 RTP priority 0→SL 2 时配置 `service_level=2`,Jetty 使用 priority 0。
平台配置设备映射与调度策略,brpc 使用进程初始化时的映射快照。

不同配置使用独立连接,相同配置共享连接。配置随服务发现、连接池、短连接和
服务端 accept 传递,Jetty 重建时重新应用。预建池服务于默认 priority,
其他 priority 按需创建。v2/v3 握手使用所选路径。

### 功能验证

用 `urma_admin show --dev "$URMA_DEVICE" --whole` 核对当前路径的 priority:SL。
`example/urma_performance` 的两端均支持 `--urma_tp_type=rtp --urma_sl=2`,
也可通过 `--urma_priority` 指定匹配项。先以默认配置完成 RPC,再分别改变请求和响应 SL,
验证合法映射、不支持的 SL 和 SL/priority 冲突。对 v2/v3、single/pooled 及 polling 模式重复验证。

连接调试信息提供 `state`、`urma_effective_priority` 和 `urma_effective_sl`。
核对 URMA 连接为 `ESTABLISHED`、映射符合设备配置、RPC 无错误。
调度效果测试结合竞争流量、设备队列计数器、吞吐与 P99。
51 changes: 51 additions & 0 deletions docs/en/urma.md
Original file line number Diff line number Diff line change
Expand Up @@ -279,3 +279,54 @@ including normal short-connection closes before the injection.
buffer held until completion, adding one CPU copy. Keep device, queue and payload
settings consistent when comparing send paths. Startup logs report `copy_send`;
failure logs report direction, send slot and registered segment range.

## Business SL and priority

Set `ChannelOptions.urma_qos` for outgoing requests and
`ServerOptions.urma_qos` for outgoing responses:

```cpp
brpc::ChannelOptions client_options;
client_options.socket_mode = brpc::SOCKET_MODE_URMA;
client_options.urma_qos.service_level = 2;

brpc::ServerOptions server_options;
server_options.socket_mode = brpc::SOCKET_MODE_URMA;
server_options.urma_qos.service_level = 2;
```

`--urma_tp_type=auto` prefers CTP, then RTP, using device capabilities.
Use `ctp` or `rtp` to select a path explicitly. Both peers use the same path,
fixed at process initialization. Providers without a capability table use
priority 6 for auto/CTP and priority 0 for RTP.

| Option | Default | Explicit range | Meaning |
| --- | --- | --- | --- |
| `service_level` | -1 | 0–255 | Select the first priority mapping to this SL on the selected path |
| `priority` | -1 | 0–15 | Select a priority supported by the selected path |

Both fields at -1 select the device default. When both are explicit, they must
match the same device mapping. Channel initialization and Server startup reject
unsupported or conflicting settings and report supported priority:SL pairs.
For example, RTP priority 0 mapping to SL 2 resolves `service_level=2` to priority 0.
The platform configures mappings and scheduling; brpc uses the initialization snapshot.

Different configurations use separate connections; equal configurations share
connections. Naming services, pooled and short connections, accepted sockets and
rebuilt Jettys inherit the configuration. The prepared pool serves the default
priority; other priorities allocate Jettys on demand. Both v2/v3 handshakes carry
the selected path.

### Functional validation

Inspect the selected path's priority:SL map with
`urma_admin show --dev "$URMA_DEVICE" --whole`.
Both examples in `example/urma_performance` accept `--urma_tp_type=rtp --urma_sl=2`
and an optional matching `--urma_priority`. Establish RPCs with defaults, then
change request and response SLs separately. Check supported mappings, unsupported
SLs and conflicting pairs across v2/v3, single/pooled and polling modes.

Connection diagnostics expose `state`, `urma_effective_priority` and
`urma_effective_sl`. Check that URMA is `ESTABLISHED`, the mapping matches the
device and RPCs succeed. Evaluate scheduling with competing traffic, device queue
counters, throughput and P99.
12 changes: 10 additions & 2 deletions example/urma_performance/client.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,9 @@ DEFINE_int32(attachment_size, -1, "Attachment size is used (in Bytes)");
DEFINE_int32(rpc_timeout_ms, 5000, "Timeout for each RPC in milliseconds");
DEFINE_bool(echo_attachment, false, "Select whether attachment should be echo");
DEFINE_bool(use_urma, true, "Use URMA transport (true) or TCP (false)");
DEFINE_int32(urma_priority, -1,
"Outbound URMA priority: -1 automatic, or [0,15]");
DEFINE_int32(urma_sl, -1, "Outbound URMA SL: -1 unspecified, or [0,255]");

bvar::LatencyRecorder g_latency("client");
bvar::Adder<int64_t> g_error_count("client_error_count");
Expand Down Expand Up @@ -88,6 +91,8 @@ static void* worker(void* arg) {
int main(int argc, char* argv[]) {
gflags::ParseCommandLineFlags(&argc, &argv, true);
brpc::ChannelOptions options;
options.urma_qos.priority = FLAGS_urma_priority;
options.urma_qos.service_level = FLAGS_urma_sl;
options.socket_mode = FLAGS_use_urma ? brpc::SOCKET_MODE_URMA
: brpc::SOCKET_MODE_TCP;
options.connect_timeout_ms = FLAGS_rpc_timeout_ms;
Expand All @@ -110,7 +115,9 @@ int main(int argc, char* argv[]) {
stub.Test(&warmup_cntl, &warmup_req, &warmup_resp, nullptr);
if (warmup_cntl.Failed()) {
LOG(ERROR) << "Warm-up RPC failed after timeout_ms="
<< FLAGS_rpc_timeout_ms << ": "
<< FLAGS_rpc_timeout_ms << " use_urma=" << FLAGS_use_urma
<< " urma_priority=" << FLAGS_urma_priority
<< " urma_sl=" << FLAGS_urma_sl << ": "
<< warmup_cntl.ErrorText();
return -1;
}
Expand All @@ -131,7 +138,8 @@ int main(int argc, char* argv[]) {
}
LOG(INFO) << "URMA performance client started (server=" << FLAGS_server
<< ", use_urma=" << FLAGS_use_urma
<< ", threads=" << thread_num
<< ", urma_priority=" << FLAGS_urma_priority
<< ", urma_sl=" << FLAGS_urma_sl << ", threads=" << thread_num
<< ", rpc_timeout_ms=" << FLAGS_rpc_timeout_ms << ")";
while (!brpc::IsAskedToQuit()) {
sleep(1);
Expand Down
9 changes: 8 additions & 1 deletion example/urma_performance/server.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,9 @@

DEFINE_int32(port, 8003, "TCP Port of this server");
DEFINE_bool(use_urma, true, "Use URMA transport (true) or TCP (false)");
DEFINE_int32(urma_priority, -1,
"Outbound URMA priority: -1 automatic, or [0,15]");
DEFINE_int32(urma_sl, -1, "Outbound URMA SL: -1 unspecified, or [0,255]");

butil::atomic<uint64_t> g_last_time(0);

Expand Down Expand Up @@ -73,14 +76,18 @@ int main(int argc, char* argv[]) {
}

brpc::ServerOptions options;
options.urma_qos.priority = FLAGS_urma_priority;
options.urma_qos.service_level = FLAGS_urma_sl;
options.socket_mode = FLAGS_use_urma ? brpc::SOCKET_MODE_URMA
: brpc::SOCKET_MODE_TCP;
if (server.Start(FLAGS_port, &options) != 0) {
LOG(ERROR) << "Fail to start server";
return -1;
}
LOG(INFO) << "URMA performance server started on port " << FLAGS_port
<< " (use_urma=" << FLAGS_use_urma << ")";
<< " (use_urma=" << FLAGS_use_urma
<< ", urma_priority=" << FLAGS_urma_priority
<< ", urma_sl=" << FLAGS_urma_sl << ")";
server.RunUntilAskedToQuit();
return 0;
}
Expand Down
1 change: 1 addition & 0 deletions src/brpc/acceptor.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -338,6 +338,7 @@ void Acceptor::OnNewConnectionsUntilEAGAIN(Socket* acception) {
options.force_ssl = am->_force_ssl;
options.initial_ssl_ctx = am->_ssl_ctx;
options.socket_mode = am->_socket_mode;
options.urma_qos = am->_urma_qos;
options.bthread_tag = am->_bthread_tag;
if (Socket::Create(options, &socket_id) != 0) {
am->ReleaseConnectionSlot();
Expand Down
3 changes: 2 additions & 1 deletion src/brpc/acceptor.h
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@
#include "butil/containers/flat_map.h"
#include "brpc/input_messenger.h"
#include "brpc/socket_mode.h"

#include "brpc/urma/urma_qos.h"

namespace brpc {

Expand Down Expand Up @@ -137,6 +137,7 @@ friend class Server;

// Choose to use a certain socket: 0 TCP, 1 RDMA
SocketMode _socket_mode;
UrmaQosOptions _urma_qos;

// Acceptor belongs to this tag
bthread_tag_t _bthread_tag;
Expand Down
20 changes: 15 additions & 5 deletions src/brpc/channel.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -78,11 +78,9 @@ ChannelSSLOptions* ChannelOptions::mutable_ssl_options() {
}

static ChannelSignature ComputeChannelSignature(const ChannelOptions& opt) {
if (opt.auth == nullptr &&
!opt.has_ssl_options() &&
opt.client_host.empty() &&
opt.device_name.empty() &&
opt.connection_group.empty() &&
if (opt.socket_mode != SOCKET_MODE_URMA && opt.auth == nullptr &&
!opt.has_ssl_options() && opt.client_host.empty() &&
opt.device_name.empty() && opt.connection_group.empty() &&
opt.hc_option.health_check_path.empty()) {
// Returning zeroized result by default is more intuitive for users.
return ChannelSignature();
Expand Down Expand Up @@ -141,6 +139,10 @@ static ChannelSignature ComputeChannelSignature(const ChannelOptions& opt) {
buf.append("|rdma");
} else if (opt.socket_mode == SOCKET_MODE_URMA) {
buf.append("|urma");
buf.append("|priority=")
.append(std::to_string(opt.urma_qos.priority));
buf.append("|sl=").append(
std::to_string(opt.urma_qos.service_level));
}
butil::MurmurHash3_x64_128_Update(&mm_ctx, buf.data(), buf.size());
buf.clear();
Expand Down Expand Up @@ -187,6 +189,12 @@ int Channel::InitChannelOptions(const ChannelOptions* options) {
if (options) { // Override default options if user provided one.
_options = *options;
}
if (_options.socket_mode == SOCKET_MODE_URMA &&
!_options.urma_qos.IsValid()) {
LOG(ERROR) << "URMA priority must be -1 or in [0,15], "
"and SL must be -1 or in [0,255]";
return -1;
}
const Protocol* protocol = FindProtocol(_options.protocol);
if (nullptr == protocol || !protocol->support_client()) {
LOG(ERROR) << "Channel does not support the protocol";
Expand Down Expand Up @@ -407,6 +415,7 @@ int Channel::InitSingle(const butil::EndPoint& server_addr_and_port,
opt.local_side = client_endpoint;
opt.initial_ssl_ctx = ssl_ctx;
opt.socket_mode = _options.socket_mode;
opt.urma_qos = _options.urma_qos;
opt.hc_option = _options.hc_option;
opt.device_name = _options.device_name;
if (SocketMapInsert(SocketMapKey(server_addr_and_port, sig),
Expand Down Expand Up @@ -453,6 +462,7 @@ int Channel::Init(const char* ns_url,
ns_opt.succeed_without_server = _options.succeed_without_server;
ns_opt.log_succeed_without_server = _options.log_succeed_without_server;
ns_opt.socket_option.socket_mode = _options.socket_mode;
ns_opt.socket_option.urma_qos = _options.urma_qos;
ns_opt.channel_signature = ComputeChannelSignature(_options);
ns_opt.socket_option.hc_option = _options.hc_option;
ns_opt.socket_option.local_side = client_endpoint;
Expand Down
4 changes: 4 additions & 0 deletions src/brpc/channel.h
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@
#include "brpc/naming_service_filter.h"
#include "brpc/health_check_option.h"
#include "brpc/socket_mode.h"
#include "brpc/urma/urma_qos.h"

namespace brpc {

Expand Down Expand Up @@ -111,6 +112,9 @@ struct ChannelOptions {
// Default: SOCKET_MODE_TCP
SocketMode socket_mode;

// Outbound URMA QoS for this Channel. Used when socket_mode is URMA.
UrmaQosOptions urma_qos;

// Turn on authentication for this channel if `auth' is not nullptr.
// Note `auth' will not be deleted by channel and must remain valid when
// the channel is being used.
Expand Down
8 changes: 8 additions & 0 deletions src/brpc/server.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -876,6 +876,13 @@ int Server::StartInternal(const butil::EndPoint& endpoint,
const ServerOptions default_opt;
const ServerOptions& real_opt = opt ? *opt : default_opt;

if (real_opt.socket_mode == SOCKET_MODE_URMA &&
!real_opt.urma_qos.IsValid()) {
LOG(ERROR) << "URMA priority must be -1 or in [0,15], "
"and SL must be -1 or in [0,255]";
return -1;
}

if (!real_opt.h2_settings.IsValid(true/*log_error*/)) {
LOG(ERROR) << "Invalid h2_settings";
return -1;
Expand Down Expand Up @@ -1163,6 +1170,7 @@ int Server::StartInternal(const butil::EndPoint& endpoint,
_am->_socket_mode = _options.socket_mode;
_am->_bthread_tag = _options.bthread_tag;
}
_am->_urma_qos = _options.urma_qos;
// Set `_status' to RUNNING before accepting connections
// to prevent requests being rejected as ELOGOFF
_status = RUNNING;
Expand Down
4 changes: 4 additions & 0 deletions src/brpc/server.h
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@
#include "brpc/baidu_master_service.h"
#include "brpc/rpc_pb_message_factory.h"
#include "brpc/socket_mode.h"
#include "brpc/urma/urma_qos.h"

namespace brpc {

Expand Down Expand Up @@ -242,6 +243,9 @@ struct ServerOptions {
// Default: SOCKET_MODE_TCP
SocketMode socket_mode;

// Outbound URMA QoS for responses on accepted connections.
UrmaQosOptions urma_qos;

// [CAUTION] This option is for implementing specialized baidu-std proxies,
// most users don't need it. Don't change this option unless you fully
// understand the description below.
Expand Down
6 changes: 6 additions & 0 deletions src/brpc/socket.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -711,6 +711,10 @@ void SetSocketBufferOptions(int fd) {
// version: from version part of _versioned_nref, must be an EVEN number.
// slot: designated by ResourcePool.
int Socket::Create(const SocketOptions& options, SocketId* id) {
if (options.socket_mode == SOCKET_MODE_URMA && !options.urma_qos.IsValid()) {
errno = EINVAL;
return -1;
}
return VersionedRefWithId<Socket>::Create(id, options);
}

Expand All @@ -726,6 +730,7 @@ int Socket::OnCreated(const SocketOptions& options) {
});
// start build the transport
_socket_mode = options.socket_mode;
_urma_qos = options.urma_qos;
_transport = TransportFactory::CreateTransport(options.socket_mode);
CHECK(nullptr != _transport);
_transport->Init(this, options);
Expand Down Expand Up @@ -2814,6 +2819,7 @@ SocketOptions Socket::GetSubSocketOptions() const {
opt.keytable_pool = _keytable_pool;
opt.app_connect = _app_connect;
opt.socket_mode = _socket_mode;
opt.urma_qos = _urma_qos;
opt.keepalive_options = _keepalive_options;
opt.tcp_user_timeout_ms = _tcp_user_timeout_ms;
opt.bthread_tag = _io_event.bthread_tag();
Expand Down
4 changes: 4 additions & 0 deletions src/brpc/socket.h
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@
#include "brpc/versioned_ref_with_id.h"
#include "brpc/health_check_option.h"
#include "brpc/socket_mode.h"
#include "brpc/urma/urma_qos.h"
#include "brpc/input_messenger_processor.h" // InputMessengerProcessor

namespace brpc {
Expand Down Expand Up @@ -310,6 +311,7 @@ struct SocketOptions {
bool force_ssl{false};
std::shared_ptr<SocketSSLContext> initial_ssl_ctx;
SocketMode socket_mode{SOCKET_MODE_TCP};
UrmaQosOptions urma_qos;
bthread_keytable_pool_t* keytable_pool{nullptr};
SocketConnection* conn{nullptr};
std::shared_ptr<AppConnect> app_connect;
Expand Down Expand Up @@ -640,6 +642,7 @@ friend class TransportFactory;

// The socket mode decided at creation.
SocketMode socket_mode() const { return _socket_mode; }
const UrmaQosOptions& urma_qos() const { return _urma_qos; }

// Get an UNUSED socket connecting to the same place as this socket
// from the SocketPool of this socket.
Expand Down Expand Up @@ -1017,6 +1020,7 @@ friend class TransportFactory;

// Should use SOCKET_MODE_RDMA or SOCKET_MODE_TCP or Other, default is SOCKET_MODE_TCP Transport
SocketMode _socket_mode;
UrmaQosOptions _urma_qos;
std::unique_ptr<Transport> _transport;

// Pass from controller, for progressive reading.
Expand Down
10 changes: 8 additions & 2 deletions src/brpc/urma/mock_urma.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -240,14 +240,20 @@ urma_status_t urma_query_device(urma_device_t *device,
if (!device || !attr) {
return URMA_EINVAL;
}
mock_device_attr.dev_cap.priority_info[3].tp_type.bs.rtp = 1;
mock_device_attr.dev_cap.priority_info[3].SL = 3;
mock_device_attr.dev_cap.priority_info[6].tp_type.bs.ctp = 1;
mock_device_attr.dev_cap.priority_info[6].tp_type.bs.rtp = 1;
mock_device_attr.dev_cap.priority_info[6].SL = 2;
mock_device_attr.dev_cap.priority_info[10].tp_type.bs.ctp = 1;
mock_device_attr.dev_cap.priority_info[10].tp_type.bs.rtp = 1;
mock_device_attr.dev_cap.priority_info[10].SL = 4;
mock_device_attr.dev_cap.max_jfc = 1024;
mock_device_attr.dev_cap.max_jetty = 1024;
mock_device_attr.dev_cap.max_jfs_sge = 8;
mock_device_attr.dev_cap.max_jfr_sge = 8;
mock_device_attr.dev_cap.priority_info[0].tp_type.bs.rtp = 1;
mock_device_attr.dev_cap.priority_info[0].SL = 2;
mock_device_attr.dev_cap.priority_info[6].tp_type.bs.ctp = 1;
mock_device_attr.dev_cap.priority_info[6].SL = 6;
memcpy(attr, &mock_device_attr, sizeof(urma_device_attr_t));
return URMA_SUCCESS;
}
Expand Down
Loading
Loading