networking

10Gbps Networking at Home

Fouad Chamoun

Fouad Chamoun

— 4 min read
10Gbps Networking at Home
Photo by Thomas Jensen / Unsplash

I'm fortunate enough to live in a place where 10Gbps fiber (FTTH) is not only available but also cheap. Here's how I'm taking advantage of this.

Switching

My ISP provides a home router that has a 10G SFP+ port:

Freebox Ultra - Back

But a single port wouldn't be enough to play with in a homelab would it ? 🙂 So I went searching for a good "prosumer" SFP+ switch that wouldn't cost me an arm and a leg. I found offerings from Ubiquity, MikroTik, and TP-Link.

Ubiquity's USW-Aggregation switch is quite nice – I've used their Unifi Access Points – but above my budget, at 300€.

MikroTik's CRS309-1G-8S+IN is also a solid option. Also above budget, between 250€ and 300€ depending on the vendor.

As for TP-Link, I had sworn in the past to never use their "home-user" offerings again as their equipment was consistently unreliable. But I discovered their Omada "Enterprise" lineup and it was compelling for my use case. The TL-SX3008F goes for 219€ and has all the features I was looking for:

  • 8 SFP+ ports
  • managed
  • L2+ (L2 switching + some L3 functions)
  • supports LAG
  • supports VLANs
TP-Link TL-SX3008F

Cables

Next up on the list was cables. I initially chose DAC cables, which support 10G speeds and are more durable than fiber. However, I found them to be quite stiff and only practical for short-distance connections.

So I also picked up AOC OM2 cables. While they’re more flexible and easier to route, they’re also more fragile by nature. In my experience, they’re a better choice for longer runs. Like Ethernet, they come in various specifications—OM2 being sufficient for the speeds and distances I need.

DAC Cable - AOC OM2 Cable

Servers

My existing server is an old Dell T110-ii with an integrated 1Gbps Ethernet port. I'm planning on adding an SFP+ PCIe extension card in the future, but in the meantime, let me show you this beauty I got recently: the Minisforum MS-01. It deserves a post on its own so I'll just mention its networking capabilities here.

Minisforum MS-01

This PC has 2 SFP+ 10G ports, 2 Ethernet 2.5G ports and 2 USB4 ports that will later be used for 20G Thunderbolt networking.

Connections

Here's how the connections look like between the home router, the TP-Link switch, and the MS-01.

The MS-01 uses the 10G links as its main connection. They are bonded together in a LAG that allows for parallel traffic to the server over the 2 cables. The traffic is load-balanced between the two according to the source/destination MAC addresses (config screenshots in the next section). If one of the fiber cables is disconnected, the other one handles all the traffic. If both fiber cables are removed, traffic goes through the 2.5G failover Ethernet connection.

Configuration

All of this is configured in the switch and in Proxmox, the Hypervisor OS running on the MS-01.

Switch

I simply enabled Passive LACP mode on ports 2 & 3 and put them in LAG Group 1.

Proxmox

  • I created bond0 with the 2 SFP+ ports in LACP mode.
  • I created bond1 with bond0 and the 2.5G Ethernet port in active-backup mode.
  • The vmbr0 bridge was already present. I just "pointed" it to bond1.

Verifications

cat /proc/net/bonding/bond0 outputs the following info (redacted):

Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up

802.3ad info
LACP active: on
LACP rate: slow
Active Aggregator Info:
        Aggregator ID: 1
        Number of ports: 2

Slave Interface: enp2s0f0np0
MII Status: up
Speed: 10000 Mbps
Duplex: full
Aggregator ID: 1

Slave Interface: enp2s0f1np1
MII Status: up
Speed: 10000 Mbps
Duplex: full
Aggregator ID: 1

cat /proc/net/bonding/bond1 outputs the following info (redacted):

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: bond0 (primary_reselect always)
Currently Active Slave: bond0
MII Status: up

Slave Interface: bond0
MII Status: up
Speed: Unknown
Duplex: Unknown

Slave Interface: enp87s0
MII Status: up
Speed: 2500 Mbps
Duplex: full

I'm not sure why the speed and duplex mode for bond0 show up as Unknown in bond1 . It doesn't seem to be problematic though.

I ran some tests with iperf and tried disconnecting the cables. The connection gracefully degraded to 2.5G and went back to 10G when I reconnected everything.

Conclusion

It's so satisfying to see the failover in action! I guess this server is now ready for some workload. I'm also planning on adding two more MS-01 for a 3-node cluster. Each node would have a 10G LAG and all 3 would be linked together with Thunderbolt 20G ring network. I'll be able to test a High Availability setup and maybe even Kubernetes. Stay tuned!

Read more