A lot of small to medium businesses haven’t made the transition over to 10Gb speeds from the typical 1Gb (at least around where I live), let alone us homelabbers, because of it’s cost. Even with “cheaper” 10GBASE-T switches coming out, like the D-Link DXS-1210-12TC, $1,470 is still a lot more than most of us can convince our wives to let us spend on one. Also, do you have 10Gb NICs on your hosts and NAS/SANs? What about the right cables? These things aren’t cheap.
Inspired by a Reddit post I found over on the /r/homelab subreddit, I decided to try a 10Gb point-to-point connection between my ESXi host (a Dell PowerEdge R520) and my HP ProLiant DL320e that I’m using as my mini-SAN (running Windows Server 2016 TP4). For those that don’t know, a point-to-point connection is a small network between two endpoints or clients, so no switch is needed. My switch (an HP ProCurve 2920) doesn’t have the 10Gb modules needed but I wanted the 10Gb connection between my ESXi host and SAN anyway, so a P2P connection between those two would be perfect.
The NICs talked about in the /r/homelab thread were used HP Mellanox ConnectX-2 10GbE NICs you can find on eBay for SUPER cheap (currently $18.78 each on eBay) connected via this 10GBASE SFP+ cable for $22.94 shipped. I found the NICs to be widely supported but I couldn’t find a lot of information on just how supported they are on the newest operating systems. My host is running ESXi 6 while my ProLiant is running Windows Server 2016 TP4, but for a total of about $65, it was worth the risk of not being supported.
UPDATE: Thanks to Reddit user /u/negabiggz for mentioning that these Mellanox ConnectX-2 NICs do not work under FreeNAS. If you’d still like to create a cheap 10Gb P2P connection in FreeNAS, you can pick up these Chelsio S310E-CR 10Gb NICs on eBayor (shown in figure below) wait till the drivers are natively supported in version 10.1.
About a week after placing the orders for the NICs and cable, these two beautiful pieces of used hardware came in along with the SFP+ cable. I immediately took the R520 and ProLiant DL320e out of my rack and got them both easily installed. I fired both machines up and got what every sysadmin and/or homelabber loves to see: both NICs working properly out of the box. Turns out ESXi 6 AND Server 2016 TP4 really do support these cheap, beautiful NICs without the need to install/uninstall/reinstall a ton of different drivers.
This is a screenshot of the 10Gb NIC on my ProLiant right after booting it up and configuring the static IP.
This is a screenshot of the NIC on my ESXi host after booting it up and getting the other NIC configured on my 2016 TP4 box.
I haven’t done a lot of speed tests to get the official read/write speeds, but I have added the 10Gb NIC to my media downloading VM and transferred a few 720p TV episode files between them and my ProLiant. I tried to take a screenshot of the transfer rate when I copied over a 1GB 720p episode of a TV show I downloaded (legitimately, I swear!), but the screen never came up. I didn’t even get a chance to screenshot it. It was like I just moved the video between folders on the same drive. I also configured an iSCSI connection using the 10Gb link and performance so far has been great.
So there you have it. An awesome 10Gb speed on your homelab all for under $70. Thanks to the /r/homelab community for the idea and for suggesting the hardware, especially for those of us that didn’t want to convince our wives why we need to spend hundreds of dollars so we can transfer files using a puny 1Gb connection.
Source: https://thatservernerd.com/2016/02/23/10gb-in-your-homelab-for-under-70