Network Traffic Hashing

Benchmarking network-traffic hashing across DPDK, CPU, and DPU offload.

🔗 You can also read this post on our website: https://amododesign.com/ai-verification/

Hashing network traffic creates a record of what data has transited a link. Combined with memory wipes, recomputation checks, and side-channel protections, it could play an important part of AI workload verification systems.

We present this exploration in the style of a technical lab notebook - but with an introduction to the key terms and definitions used throughout. We hope this maximises the pace of iteration and quality of feedback.

Background

Scenario: A cluster is operated by the prover and checked by the verifier. The verifier’s goal is to verify “completeness” and “correctness” of the workloads running on the cluster.

Network hashing is one part of achieving completeness: the verifier can create a compact record of all of the traffic on the network link, by hashing it. This can be especially useful in recomputation schemes, especially with mutually trusted hashing hardware.

We will not attempt to motivate this here, but rather explore the design of such a device. One example scheme where generating a hash ledger of traffic is useful is presented here. There is other work that relies on this type of network hashing (example).

A hash function takes an input of arbitrary size and produces a fixed-size output - a “digest”, in a way that is:

• Deterministic: the same input always produces the same output but

• Effectively one-way: in practice knowing the output tells you nothing useful about the input.

• Collision-resistant: it is very unlikely to hash two different inputs that produce the same digest. This means a digest is a reliable fingerprint: if two pieces of data have the same digest, they are almost certainly the same data. This is largely a function of the digest size.

• (Possibly) Cryptographically secure: it is computationally infeasible to find two different inputs that produce the same output.

Our approach is to place a hashing device behind a TAP (Test Access Point) on a network link, which captures all traffic passing across it. We have done some analysis of TAP tech in the past here. We aim to discuss more about this in the future, but for now we explore the hashing side of the problem.

What to Hash

Before choosing a hash algorithm or hardware platform, we must define what “hashing all network traffic” means.

Figure 1. Ethernet frame/packet architecture

Hashing network traffic could mean:

• Hashing each entire Layer 1 Ethernet packet

• Hashing each Layer 2 Ethernet frame (including Ethernet header)

• Hashing just the payload

• Parsing the payload (e.g stripping IP headers, reconstructing TCP flows, etc.)

The level of reconstruction that needs to happen depends on other aspects of the verification mechanism, which we will not explore here.

The choice of “reconstruction depth” of traffic is pivotal to the overall scheme - we will present our thoughts on this in an upcoming post.

For now, we assume that we are hashing each Layer 2 Ethernet frame. Much of the following focuses on data throughput, which will not change significantly if stripping more headers - the payload data is most of the throughput.

We did some tests with stripping the Layer 2 headers to confirm that the speed differences are small (see later).

Speeds

Current frontier AI datacentre rack-to-rack links operate at 100G, 200G, 400G, and 800GbE, with 1.6TbE rolling out through 2026.

Solving line-rate hashing for state-of-the-art 1.6T is difficult right now due to limited hardware availability.

Our MVP target was 400GbE, where suitable off-the-shelf components are available.

At 400GbE, a link is carrying roughly 50 GB of data per second. How many packets is this?

Minimum size Ethernet L2 frames are 64B (see Figure 1). Each is an 84B Layer 1 Ethernet packet (see Figure 1).

400 GbE = 400 × 10⁹ bits per second

84 B = 84 × 8 bits = 672 bits

(400 × 10⁹) ÷ (84 × 8) ≈ 595,000,000 packets per second

Jumbo frames can typically extend the payload size to 9000B (frame size 9018B). They’re heavily used in modern AI datacentres to speed up massive data transfers over high-bandwidth links.

The following visualises what it would require to store these volumes of hashed packets:

Table 1. Packet rates and hash-log storage requirements at 400GbE.

Choosing a Hash Function

Do we need a cryptographic hash?

Here, a “cryptographic hash” means it is computationally infeasible to find two different inputs that produce the same output.

Non-cryptographic hashes are faster, but they are not designed to be secure against an adversary who knows the hash function and can choose or modify their inputs.

For example, CRC32 is non-crytographic, and an attacker who wants to replace content X with content Y can calculate a small correction (a few extra bytes of padding) that restores the original CRC value after the substitution. The verifier sees the same checksum over different content. As a concrete illustration:

TRANSFER=1000;TO=ALICE;PAD=WYN6LL  →  CRC32 = 0xCD63D39A
TRANSFER=9000;TO=ALICE;PAD=Z0MGWP  →  CRC32 = 0xCD63D39A

We expect to need a proper cryptographic hash where collisions are computationally infeasible to find.

Definitions: Message Authentication Codes (MACs) & Keyed Hashes

Keyed hashes, unlike regular hashes, take both the input and a secret key, and produce a tag that can only be computed or verified by someone with that key.

In our setting, where the verifier controls the hashing device, this could be an advantage: a valid tag can only have been produced by the verifier’s hardware, which limits the prover’s ability to forge or manipulate records.

Message Authentication Codes (MACs) are short, fixed-size blocks of data attached to a message. They act as a digital fingerprint to verify both the message’s integrity (that it hasn’t been altered) and its authenticity (that it came from the stated sender).

Even though the primary purpose of a MAC is to authenticate a message, they can still be used in isolation on arbitrary data, and essentially act like a hash function, producing an irreversible, fixed-size representation of input data. MACs are one use of keyed hashes.

Additional Authenticated Data (AAD) is data that is included in the MAC computation, but not encrypted as part of the payload. This means that it is verified as unmodified by the tag, but its contents remain visible in plaintext. AES-GCM, an encryption algorithm which uses MACs to authenticate messages, typically uses AAD for metadata such as packet headers.

Candidate algorithms

The main candidates are BLAKE3, SHA-256, SHA-512, SHA3-256, AES-GMAC, and SipHash. Before comparing them we need to pin down how large a digest has to be.

Collision Resistance

Hashing such a high volume of data, we need to evaluate the likelihood of cryptographic collisions for different output digest sizes.

We can use the Birthday Bound approximation to calculate the expected number of outputs needed to produce the first collision for each output size, with the equation:

E[k] = √(πN ⁄ 2)

where N is the number of possible outputs (2⁶⁴ for 64-bit etc.).

This number can then be divided by the packet rate to determine how long this would take to reach:

Table 2. Expected time to first digest collision at 400GbE packet rates (birthday bound).

It’s clear that the collision rate for 64-bit digests is too high for a high bandwidth hash log, so we need to consider algorithms that have a minimum of a 128-bit digest size.

It’s also worth noting that this is the average time to first collision. The per-hash probability of colliding with an existing hash increases as more hashes accumulate, so the time between first and second collision would be shorter, and even shorter for each subsequent collision.

Candidates

Table 3. Candidate hash algorithms.

Hardware Architecture

We considered three architectures:

• CPU-based hashing

• DPU (Data Processing Unit) based hashing

• FPGA (Field-Programmable Gate Array) based hashing

These are not mutually exclusive; the best long-term answer may combine elements of more than one. For example, we could use a DPU for packet processing, and then pipe the output to an FPGA or CPU to perform the hashing, or similar.

CPU hashing with DPDK

The simplest path to a working prototype - shown in Figure 2.

1. A 400G Network Interface Card (NIC) feeds a high-core-count server.

2. Packets are captured using DPDK (Data Plane Development Kit), an open-source framework that bypasses the standard OS kernel networking stack and delivers frames directly to user-space processes.

3. Packets are hashed in parallel across CPU cores and written to an append-only log.

The NIC can connect via PCIe 5.0 x16, which provides approximately 64 GB/s (512 Gbit/s) of bandwidth. Moving to 800G would require PCIe 6.0 x16, which provides double this (128 GB/s - 1024 Gbit/s).

It’s also worth noting that this route would require something like a 2U server on every single network link, which would add significant physical space overhead to a datacentre.

DPU hashing

Data Processing Units (DPUs) are programmable network processors that sit between the NIC and the host CPU. They are used to offload tasks that would otherwise occupy the host. They typically consist of:

• High speed networking

• System-on-a-Chip (SoC) for processing

• Onboard memory

• Various hardware accelerators for cryptography, compression etc.

Nvidia’s BlueField-3 supports 400G and BlueField-4 supports 800G, and they provide the DOCA SDK for developing software on these devices, with relevant libraries:

• DOCA Flow for packet processing pipelines

• DOCA AES-GCM for hardware-accelerated AES-GCM operations

DOCA SHA also exists, but support was dropped for this library on the BlueField-3 as they no longer contain a SHA acceleration engine, and BlueField-2 which has a SHA engine only supports 200G networking.

We outlined two different approaches to explore on the DPUs:

1. AES-GMAC hashing via the DOCA libraries

2. Software hashing on the ARM SoC cores

The DOCA hashing pipeline, shown in Figure 3:

1. Steers inflow traffic to queues in memory via DOCA Flow API.

2. Reads frames from memory into DOCA AES-GCM hardware accelerated engine.

3. Hashes each frame via AES-GMAC, by calling the AES-GCM function with empty plaintext and the frame as AAD.

4. Writes AAD copy + tag (hash digest) back to memory

5. Writes hashes to append-only log

Figure 3. DPU DOCA hashing design

The software hashing pipeline, shown in Figure 4, is the same implementation as the CPU hashing prototype, and is executed inside the DPU’s linux environment on the ARM cores.

Figure 4. DPU software hashing design

FPGA hashing

A Field-Programmable Gate Array (FPGA) is a configurable integrated circuit that can be repeatedly programmed after manufacturing to perform custom digital logic. FPGAs can process data in parallel, and are often designed with high-speed networking in mind. One option would be to design a custom FPGA-based device to hash network traffic and log the hashes to host. Open-source VHDL and Verilog implementations of SHA-256, AES, and SHA-3 are available from the OpenCores archive.

1. The FPGA receives packets from the network and runs the hash pipeline directly over physical digital circuits.

2. It sends hash records out over another link, e.g. PCIe, Ethernet, etc.

The FPGA devices could likely be physically much smaller than the alternatives.

We intend to explore this option in future work.

Experiments

Benchmark Dashboard

To make it easier to analyse results across different configurations for each experiment, we added CSV logging to each experiment’s benchmark tool and built a HTML dashboard that visualises the output. Graphs update automatically as new results are logged. This was useful for identifying performance scaling trends.

We include a static version here to present all of the raw data.

Explore the results in the interactive dashboard →

CPU Hashing in Isolation

We built a benchmarking tool to benchmark the performance of different hash algorithms. The tool measures raw hashing throughput in isolation, using static buffers in memory to represent packet data - no network traffic at all.

The tool tests BLAKE3, and variants of AES-GMAC and SipHash across a range of input sizes from 64B (minimum Ethernet) up to 32 MiB.

CPU benchmarks - AMD EPYC 9354P (32 cores / 64 threads), 128GB RAM

Specs:

• Dell PowerEdge R7615

• CPU - AMD EPYC 9354P 32C/64T @ 3.8GHz

• RAM - 128GB DDR5 4800MT/s

• GPU - Integrated graphics

• OS - Ubuntu 24.04.4 LTS

All 64 threads:

Figure 5. CPU hashing results on EPYC 9354P with 64 threads

• SipHash-1-3-128 clears 400 Gbps at every packet size tested

• SipHash outperforms AES-GMAC even with hardware acceleration

• SipHash 128 bit variant is only marginally slower than the 64 bit variant, which we need to use for adequate collision resistance

• BLAKE3 only reaches 400G above roughly 4 KiB inputs

The sharp uptick between 1000 B and 32 MiB for BLAKE3 is expected, as we know it’s optimised for larger inputs. However, it’s interesting that SipHash-1-3 variants’ throughput also increased here yet SipHash-2-4 and AES-GMAC didn’t.

At large inputs, SipHash-1-3 transitions from being memory-latency-bound to compute-bound. SipHash-2-4 and AES-GMAC don’t show this because they were already compute-bound at smaller input sizes:

• SipHash-2-4 due to its extra compression and finalisation rounds

• AES-GMAC because its hardware acceleration pipeline keeps execution units busy even on short inputs

SipHash-1-3-128 arises as the leading candidate for frame-sized hashing. AES-128-GMAC remains a strong alternative on hardware where AES acceleration is available.

CPU Benchmark - BlueField-3 DPU (16-core ARM SoC)

Specs:

• NVIDIA BlueField-3 B3240 P-Series Dual-slot FHHL DPU

• 400GbE / NDR InfiniBand (default mode)

• Dual-port QSFP112

• PCIe Gen5.0 x16 with x16 PCIe extension option

• 16 Arm cores

• 32GB on-board DDR5

• Crypto Enabled

Running the same CPU hashing benchmark tool on the DPU’s 16 ARM cores confirmed that no algorithm reaches 400G.

This result was expected, as we could estimate based on published performance metrics for each hashing algorithm’s per-core software performance that 16 cores would be insufficient for high-bandwidth (32 - 595Mpps) hashing.

A DPU with 32-64 cores may provide the performance necessary for this and would be beneficial to significantly reduce the performance requirements of the host device.

Figure 6. CPU hashing results on BlueField-3 DPU’s ARM cores

DOCA AES-GCM on BlueField-3 - hardware-accelerated path

To test the hardware acceleration path, we built a second benchmark using the DOCA AES-GCM API. In order to use only the GMAC part of the AES-GCM algorithm, we supplied our input data as the AAD and left the plaintext empty. This produces a digest over the input data without encrypting it.

The benchmark was run at the most efficient configuration available: 8 worker queues with 256 in-flight tasks per queue.

Figure 7. DPU hashing results on BlueField-3 via DOCA AES-GCM

The throughput plateaus at roughly 162 Gbit/s regardless of packet size.

This plateau is set by the DRAM bandwidth available to the hardware-accelerated path, not by the AES engine itself. The bottleneck is the double DRAM crossing, illustrated earlier in Figure 3.

Each hashing operation requires:

1. The frame to be read from DRAM into the hardware accelerator

2. The accelerator performs the hash function

3. The plaintext and tag are written back to DRAM as the algorithm is a MAC - which is designed to process the whole message, so the API passes it all back.

DRAM traffic is approximately 2× the source data volume, so we are heavily memory-bound.

Nvidia’s claimed 400G line-rate AES-GCM performance applies to inline offload, where packets are authenticated without any intermediate DRAM transactions. However, there are two issues with using this for our purposes:

• There is no mechanism in the hardware to extract the tag without DMA’ing the full packet to DRAM, which reintroduces exactly the DRAM bandwidth cost we were trying to avoid. Software only receives a pass/fail status flag in the completion descriptor.

• The AAD in IPSec includes the packet sequence number, which means that frames with identical headers and payloads do not have the same digest (breaks determinism).

Summary

Table 4. CPU hashing in isolation — peak throughput summary.

Conclusions:

• CPU is fast enough: All algorithms except BLAKE3 could achieve 400Gbps hashing down to 64b packets (see Figure 5)

• SipHash-1-3-128 was the fastest algorithm on the CPU

• DPU is not viable via either method

DPDK and Hashing In-Flight Data

The hashing benchmarks showed that the raw compute exists to hash at 400G.

We wanted to see how those throughput figures translate when packets are actually arriving from a real NIC, being processed through DPDK, and hashed in a full receive loop. To test this we built a second benchmark tool, written in C, that integrates DPDK packet receive with the hashing algorithms directly, pulling packets from the NIC into user space and hashing them as they arrive.

Loopback Testing

We used a physical loopback: a 400G DAC cable running out of port 0 back into port 1 on the same card (a Bluefield-3 DPU in NIC mode). The host both generates/transmits and receives/hashes traffic.

Traffic Generation

Pktgen

We used Pktgen, which was low friction to get running. Pktgen transmitted out of port 1 while the benchmark pulled packets from port 0.

Figure 8. Loopback traffic generation and hashing setup

The Pktgen configuration used a range of source IPs, destination IPs, and ports to ensure that the Receive Side Scaling (RSS) spread incoming packets across all of the benchmark’s receive queues. Without this, all packets would arrive on a single queue and pin a single core, which would make the benchmark look bottlenecked regardless of how many cores were allocated. We are interested in further investigation to see if this is realistic for frontier data center traffic.

However, generating traffic to saturate a 400G link requires substantial compute. Sharing the server between traffic generation and benchmarking gave two issues:

• Allocating 16 cores to Pktgen left fewer cores for the benchmark itself.

• Pktgen could only generate around 35 Gbit/s of traffic at small packet sizes (64B).

The results from this phase weren’t a true measure of the system’s hashing capacity, but they were useful to confirm the benchmark pipeline was functioning correctly.

Moving Traffic Generation to a dedicated machine

We moved traffic generation off the benchmark server onto a separate server.

We initially tried DPDK Pktgen again on the new machine but observed unexpectedly poor performance, possibly a configuration issue with that server. We switched to DPDK’s testpmd in txonly mode instead, which proved more reliable and reached closer to true line rate.

The critical configuration detail when using testpmd is enabling --txonly-multi-flow, which varies the source IP per packet - see RSS discussion above.

Figure 9. Dedicated traffic-generation server feeding the hashing benchmark over a 400G DAC link

Results

Each graph includes a grey reference line labelled “none (RX baseline)” which shows the actual throughput of traffic received by the NIC in that run. This is the ceiling that any hashing algorithm can reach, and should match this if the algorithm can keep up with the line-rate. If it falls below the baseline, the hash computation cannot keep up and packets are being lost. The baseline was taken in its own run, som

All results shown use header stripping, where only the packet payload is hashed, not the Ethernet, IP, or UDP headers. We also tested configurations with no stripping and L2 header stripping only, but hashing the payload alone is the most realistic result for our use case.

Throughput vs Worker Cores

64B frame

Figure 10. DPDK + hashing results on 64B frames (without 400G baseline)

Figure 11. DPDK + hashing results on 64B frame (with 400G baseline)

• Generating 64B frames at 400G was too demanding for the generator, so RX baseline is low and algorithm performance is capped there

• Per-packet overhead of the DPDK receive path and hash initialisation dominates at 64B

• SipHash and AES-128-GMAC are the best performing algorithms

• None of the algorithms able to process frames as fast as they arrive

Interestingly, performance visibly degrades past 32 allocated cores. The EPYC 9354P CPU uses hyper-threading, giving 64 logical threads from 32 physical cores. Once more than 32 worker threads are allocated, threads start sharing physical cores. The NIC distributes packets evenly across all receive queues, so each queue still receives its full 1/N share of traffic, but the threads servicing those queues now get roughly half the compute time they had before, causing queues to overflow and packets to be dropped. The result is that allocating logical threads beyond the physical core count actively reduces throughput.

128B frames

Figure 12. DPDK + hashing results on 128B frames (without 400G baseline)

Figure 13. DPDK + hashing results on 128B frames (with 400G baseline)

• Similar to 64B, SipHash and AES-128-GMAC lead

• Again, performance degrades after 32 threads due to hyper-threading

• Slightly better throughput than 64B frames

512B frames

Figure 14. DPDK + hashing results on 512B frames

• Generator is able to push the transmit/receive baseline much closer to 400G

• None of the algorithms reach the receive rate within the available threads, though the gap is smaller

• AES variants overtake SipHash variants past 32 threads

• BLAKE3 scales very linearly with physical cores

• The results suggest that a server with 64 physical cores would potentially be sufficient for 512B frames

1500B frames

Figure 15. DPDK + hashing results on 1500B frames

• Receive baseline is close to the full 400G now

• Hyper-threading degradation largely disappears as fewer packets per second mean less per-frame dispatch pressure

• All algorithms except BLAKE3 reach the receive baseline at 63 threads

• SipHash reaches baseline first at 48 cores

4096B frames (jumbo frames)

Figure 16. DPDK + hashing results on 4096B frames

• Performance separation between algorithms more pronounced

• AES variants are leaders at this size, reaching the baseline at only 16 threads

• SipHash variants follow at 32 threads

• BLAKE3 scales quite linearly again but does not reach receive rate

Performance does degrade again slightly past 32 cores even though it had stopped doing so at 1500B.

There is evidently still some hyper-threading overhead at this packet size (see dip at the top right of the plot). However, the receive baseline receive speed is not affected, only the hashing algorithm speed is reduced slightly. This rules out the queue overflow issue we saw with smaller packets (see above).

Our theory: 4096B packet buffers are exactly the size of a memory page on our system, which causes every buffer to land in the same narrow slice of CPU cache instead of spreading out evenly. This becomes a real cost once two threads are sharing a physical core, because they’re now both competing for that same narrow slice.

Frame sizes that aren’t an exact page size (1500B, 9000B) don’t have this alignment, so they spread across the cache normally and avoid the issue.

Since AES-GMAC and SipHash both sustain the receive rate at 32 physical cores, this post-32 degradation has no practical impact.

9000B frame (jumbo frames)

Figure 17. DPDK + hashing results on 9000B frames

• Receive baseline is now essentially at full 400G line rate

• All algorithms reach the receive baseline at different thread thresholds

• First frame size where BLAKE3 reaches line rate within the available threads

Throughput vs Packet Size

Figure 18. DPDK + hashing throughput vs packet size results for 32 threads

Looking across all packet sizes at a fixed 32-core allocation:

• BLAKE3 performs significantly below the other algorithms at every size up to 9000B

• AES-128-GMAC and SipHash-1-3 variants perform closely together for smaller inputs, with AES-GMAC pulling ahead for larger packets.

BLAKE3 is optimised for hashing large amounts of data in parallel, so on frame-sized inputs it pays the setup cost for that parallelism without having enough data to benefit from it.

This confirms the picture from the isolated CPU benchmarks: BLAKE3 is competitive only at large inputs, while AES-GMAC and SipHash are both strong across the full packet size range.

Header Stripping Overhead

Figure 19. DPDK + hashing header stripping overhead

We benchmarked the performance of the algorithms with different stripping modes: no stripping, L2 stripping, and full payload-only stripping, to understand whether the cost of identifying and skipping the headers was significant. Even at 64B, where the overhead would be most visible relative to total packet size, the difference is minimal. In some cases stripping actually improves throughput marginally because the algorithm ends up hashing slightly fewer bytes per packet. At 9000B the overhead is negligible across all modes.

Summary

Table 5. Whether each algorithm sustains the receive rate through the DPDK path, by frame size (and the core count where it first does).

For worst-case 64B minimum frames, no algorithm on this hardware reaches full line rate without exceeding the physical core count, a problem that a higher core count CPU or an FPGA-based design may address.

For realistic traffic profiles at 1500B and above, AES-128-GMAC and SipHash-1-3-128 are both capable of sustaining line-rate hashing at practical core counts, with AES-GMAC having a modest advantage at larger packet sizes.

Conclusions

1. Hashing network traffic at line rate is possible, with relatively heavyweight CPU hashing across many cores. This seems unlikely to be an acceptable solution for some verification regimes, where 10k+ links’ traffic need to be hashed.

2. We were not able to leverage “cheap” DPU-accelerated hashing enough to make it a viable option.

3. FPGA/ASIC is likely needed for some verification plans

Future Work

Consistent traffic representation. The most important open question is exactly what payloads are being hashed and how to define them reproducibly. We need reproducible workloads, traffic flows, headers, packetisation, etc.

FPGA prototype. Testing a basic frame-receive and hash pipeline on the FPGA will validate the architecture and give a realistic estimate of the development effort required for a production-grade design.

800G and 1.6T readiness. Map the hardware availability timeline for passive taps, SmartNICs, and active tap appliances at 800GbE and 1.6TbE speeds. At 1.6T the available hardware is still emerging; understanding when suitable components will be commercially available determines when a higher-bandwidth design becomes feasible without custom hardware.