F-Stack behavior at high CPS

[Sai-Raveendra-Kandregula]

Hello maintainers. I have an application written with f-stack that works as a transparent proxy, with some modifications to the tcp payload if dictated by application rules.

I was able to get a very decent amount of connection rate (around 13k cps) with all the traffic being modified, on a single CPU core. At this point CPU utilization is at around 100%. On even slightly increasing the cps post that is resulting in delayed connection deletions and active connections keep piling up, eventually making my application un-responsive due to mbuf exhaustion (speculation, since we also experienced a case where ff_ipc tools like netstat stop responding too).

If I did hit the maximum performance of my application, I would like to drop packets instead of my app getting unresponsive.

Please let me know if my configuration is something that is causing this. We slightly modified F-Stack to support mem_if configuration (and had to disable the mandatory rss) and programming ipv6 ipfw rules.

F-Stack Version: 1.24
Configuration:

[dpdk]
# Hexadecimal bitmask of cores to run on.
lcore_mask=$lcore_mask

# Huge Page Memory to use
memory=$huge_page_memory

# Size of F-Stack internal packet dispatch ring (default is 2048)
dispatch_ring_size=16384

# Number of memory channels.
channel=4

# Promiscuous mode of nic, defualt: enabled.
promiscuous=1
numa_on=1

# TX checksum offload skip, default: disabled.
# We need this switch enabled in the following cases:
# -> The application want to enforce wrong checksum for testing purposes
# -> Some cards advertize the offload capability. However, doesn't calculate checksum.
tx_csum_offoad_skip=0

# TCP segment offload, default: disabled.
tso=0

# HW vlan strip, default: enabled.
vlan_strip=1

# Set [vlanN]'s addrs like [portN] later
# the format is same as port_list
# Set vlan filter id, to enable L3/L4 RSS below vlan hdr is not enable after f-stack-1.22.
vlan_filter=1,2,4-6

# sleep when no pkts incomming
# unit: microseconds
idle_sleep=1

# sent packet delay time(0-100) while send less than 32 pkts.
# default 100 us.
# if set 0, means send pkts immediately.
# if set >100, will delay 100 us.
# unit: microseconds
pkt_tx_delay=100

# use symmetric Receive-side Scaling(RSS) key, default: disabled.
symmetric_rss=0

port_list=$port_list

# Number of vdev.
nb_vdev=0

# Number of bond.
nb_bond=0

# Number of memif.
nb_memif=$nb_memif

[memif0]
id=13
socket_abstract=no
role=client
rsize=14

[port$index]
addr=$ip
netmask=$netmask
broadcast=$broadcast_ip
gateway=$gw_ip
promiscuous=1
rx_queue_size=4096
tx_queue_size=4096

# This expected to be static during the application lifecycle. Modifying with any internal commands might break user commands.
if_name=$if_name

# FreeBSD network performance tuning configurations.
# Most native FreeBSD configurations are supported.
[freebsd.boot]
# If use rack/bbr which depend HPTS, you should set a greater value of hz, such as 1000000 means a tick is 1us.
hz=1000000

# Block out a range of descriptors to avoid overlap
# with the kernel's descriptor space.
# You can increase this value according to your app.
fd_reserve=1024

kern.ipc.maxsockets=1048576

# With a 100K session burst, your SYN cache buckets will overflow immediately, causing F-Stack 
# to drop incoming SYNs. Increasing hashsize to 32,768 (always use powers of 2) drastically 
# reduces hash collisions during the burst.
net.inet.tcp.syncache.hashsize=32768
net.inet.tcp.syncache.bucketlimit=100

net.inet.tcp.tcbhashsize=262144

kern.ncallout=2097152

kern.features.inet6=1

# H-TCP Congestion Control for a more aggressive increase in speed on higher
# latency, high bandwidth networks with some packet loss.
cc_htcp_load=YES

# hostcache cachelimit is the number of ip addresses in the hostcache list.
# Setting the value to zero(0) stops any ip address connection information from
# being cached and negates the need for "net.inet.tcp.hostcache.expire". We
# find disabling the hostcache increases burst data rates by 2x if a subnet was
# incorrectly graded as slow on a previous connection. A host cache entry is
# the client's cached tcp connection details and metrics (TTL, SSTRESH and
# VARTTL) the server can use to improve future performance of connections
# between the same two hosts. When a tcp connection is completed, our server
# will cache information about the connection until an expire timeout. If a new
# connection between the same client is initiated before the cache has expired,
# the connection will use the cached connection details to setup the
# connection's internal variables. This pre-cached setup allows the client and
# server to reach optimal performance significantly faster because the server
# will not need to go through the usual steps of re-learning the optimal
# parameters for the connection. To view the current host cache stats use
# "sysctl net.inet.tcp.hostcache.list"
net.inet.tcp.hostcache.cachelimit=0

# 30 second host cache
# net.inet.tcp.hostcache.expire=30000

# Enable the optimized version of soreceive() for stream (TCP) sockets.
# soreceive_stream() only does one sockbuf unlock/lock per receive independent
# of the length of data to be moved into the uio compared to soreceive() which
# unlocks/locks per *mbuf*. soreceive_stream() can significantly reduced CPU
# usage and lock contention when receiving fast TCP streams. Additional gains
# are obtained when the receiving application is using SO_RCVLOWAT to batch up
# some data before a read (and wakeup) is done.
# (default 0)
net.inet.tcp.soreceive_stream=1

[freebsd.sysctl]
kern.ipc.somaxconn=1048576
kern.ipc.maxsockbuf=16777216

net.add_addr_allfibs=1

net.link.ether.inet.maxhold=5

net.inet.tcp.sendspace=65536
net.inet.tcp.recvspace=65536
net.inet.tcp.nolocaltimewait=1
net.inet.tcp.cc.algorithm=cubic
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.sendbuf_auto=1
net.inet.tcp.recvbuf_auto=1
net.inet.tcp.sendbuf_inc=16384
#net.inet.tcp.recvbuf_inc=524288
net.inet.tcp.sack.enable=1
net.inet.tcp.blackhole=2
net.inet.tcp.msl=5000
net.inet.tcp.delayed_ack=1
net.inet.tcp.rfc1323=1
# Force close idle TCP connections
net.inet.tcp.always_keepalive=1
# Start keepalive probes after : 5 seconds
net.inet.tcp.keepidle=5000
# Keepalive probes interval : 5 seconds
net.inet.tcp.keepintvl=5000
# Number of probes before dropping conn
net.inet.tcp.keepcnt=3
# Handling Zombie connections FIN_WAIT_2
net.inet.tcp.fast_finwait2_recycle=1
# Duration (in milliseconds) before the socket is forcibly closed.
net.inet.tcp.finwait2_timeout=5000
# Prevent SYN-FIN flood attacks
net.inet.tcp.drop_synfin=1

net.inet.tcp.syncookies=1
net.inet.tcp.keepinit=5000
net.inet.tcp.syncache.rexmtlimit=1

net.inet.udp.blackhole=1
net.inet.ip.redirect=0
net.inet.ip.forwarding=1

net.inet6.ip6.auto_linklocal=0
net.inet6.ip6.accept_rtadv=0
net.inet6.icmp6.rediraccept=1
net.inet6.ip6.forwarding=1

# set default stacks:freebsd, rack or bbr, may be you need increase the value of parameter 'freebsd.boot.hz' while use rack or bbr.
net.inet.tcp.functions_default=freebsd
# need by bbr, should enable it.
net.inet.tcp.hpts.skip_swi=0
# Interval between calls to hpts_timeout_dir. default min 250us, max 256-512ms, default 512ms.
net.inet.tcp.hpts.minsleep=10
# [25600-51200]
net.inet.tcp.hpts.maxsleep=25600

[jfb8856606]

Root cause

When your single-core CPU hits 100% at ~13k CPS, the TCP stack cannot process connection teardowns as fast as new connections arrive. This causes active connections to pile up, and each connection holds mbufs for its send/receive buffers. Once the mbuf pool is fully consumed, the entire stack becomes unresponsive — including ff_ipc tools like netstat, since they also need mbufs to communicate.

F-Stack does not currently have a built-in backpressure mechanism that automatically drops new connections when approaching mbuf exhaustion.

Recommendations

1. Scale horizontally (primary solution)
Add more lcores to distribute the CPS load. This is the most effective approach — F-Stack scales linearly with cores.

2. Increase mbuf pool size
Allocate more hugepage memory in your config:

memory=4096   # or higher, in MB

This gives you a larger mbuf pool, providing more headroom before exhaustion.

3. Reduce timer overhead
Your hz=1000000 (1µs tick) is designed for RACK/BBR, but you're using the freebsd congestion control stack. This creates excessive timer interrupt overhead at high CPS. If you don't need RACK/BBR, reduce it:

hz=1000

Also consider reducing kern.ncallout accordingly.

4. Application-level connection limiting
Implement a maximum active connection check in your application. When the threshold is reached, reject new connections with RST instead of accepting them. This is the most direct way to achieve "drop instead of becoming unresponsive."

5. Use ipfw for SYN rate limiting
Configure ipfw rules to limit the incoming SYN rate, preventing the stack from accepting more connections than it can handle:

ipfw add allow tcp from any to any setup limit src-addr 100

6. Aggressive timeout tuning
Your timeouts are already fairly aggressive, but you could go further:

net.inet.tcp.msl=2000
net.inet.tcp.finwait2_timeout=2000

This helps reclaim mbuf resources from closing connections faster.

Note on memif

The F-Stack community release does not officially support or test memif — this appears to be your own implementation. Beyond the configuration tuning suggestions above, you may need to perform additional code review on your memif integration to ensure there are no mbuf leak paths specific to that code.

Also make sure the memif ring size is not a bottleneck. Your rsize=14 means 2^14 = 16384 ring slots — this should be adequate, but verify that the memif peer can consume at the same rate.

Next steps

We will also consider adding a convenient way to enforce a concurrent connection (CC) limit at the stack level in future releases.

Feel free to follow up with further findings as you test — we are happy to help troubleshoot together.

[Sai-Raveendra-Kandregula]

Thank you for your detailed and insightful response @jfb8856606

Just to clarify, scaling across multiple lcores needs some kind of RSS support on the interfaces, yes? If yes, memif interfaces do not support RSS, but we may ensure some kind of software RSS on the sender side so that a flow always lands on the same tx queue.

As for the rest of the recommendations, I will get back to you after running some tests.