Roll Your Own Hyperconverged Solution

Over the past few months, we’ve been putting a homegrown hyper-converged solution through its paces.  The premise behind this effort started around the time that Spectre and Meltdown vulnerabilities were announced.  The goals for this solution were fairly simple:

• Reduce overall power consumption where possible:  Traditional disks are a known quantity of consumption and don’t really have an available way to reduce power draw outside of picking units with the appropriate characteristics (lower spindle speeds, sleep mechanisms).
• Simplify networking and reduce total port count:  Running standalone NAS and dedicated host with a corresponding hypervisor can quickly increase the total number of network ports required for the entire solution.  We had to procure additional switches to support the aforementioned configuration as each component had, at a minimum, (4) 1 Gb ethernet ports available.
• Provide robust and high performance storage services:  The feature sets of commercial, off the shelf NAS solutions are diverse with respect to how resilient and vigilant the underlying solution functions with respect to addressing the integrity of data.  QNAP only offers ZFS in very expensive solutions, Netgear and Synology support btrfs to enhance data integrity and mitigate digital bit rot.
• Achieve a cooling and noise profile that may be greater than a consumer NAS, yet will be quieter than a 2U SuperMicro rack mount server:  Attempts to re-establish a FreeNAS solution using pre-existing hardware resulted in an unbearable amount of fan noise that could be heard from a far distance away.  While the understanding related to cooling drives involves sufficient airflow to keep the units operating at recommended temperatures for longevity exists, the reality of how quickly multiple 80mm fans must spin to achieve this goal runs counter to keeping such a solution anywhere near living quarters.  The potential to move comparable airflow using larger fans running at a lower speed was compelling, and would align with cooling solutions used by commercial vendors for SMB/home storage solutions.
• Retain hot swap drive capabilities:  Mechanical disks will fail.  This is a fact of life.  With a smaller sample set than Backblaze, our luck with WD Reds and various Seagate products has been less than spectacular.  The visual indicators and ease of replacement that are inherent with hot swap solutions makes the associated maintenance activities far easier than having to remember which serial number is installed in a given bay within a mid tower or full tower case.

With these design goals in play, our solution was based around an AMD Ryzen 7 1700 processor.  The 65W TDP for an eight core/sixteen thread CPU was the perfect balance of price and performance.  Additional supporting data that set our direction include the following tidbits:

• VMware ESXi 6.5 U1 has resolved the initial teething problems that existed during the initial introduction of the Ryzen architecture.  The hacks related to disabling SMT are no longer applicable.
• FreeNAS 11.1 has support for the solution if it were to be installed on bare metal.  We posted our concerns related to the about-face that happened with FreeNAS Corral.  However, the capability to run FreeNAS virtually with no additional costs versus the licenses required for vSAN or other software-defined storage offerings resulted in our selection of this solution to provide the virtual storage backend.
• Unofficial ECC Memory Support provides an additional layer of stability and integrity for the solution.  The Kingston DIMMs we had available were compatible with the solution, and a BIOS update provided the necessary switches to enable support.

The parts bin for this system beyond the aforementioned processors included the following components:

(1) Chenbro SR107 Tower Server:  We selected this unit due to the available expansion options that add hot swap drive capabilities to the enclosure.  An excessive number of negative reviews of the Silverstone SST-CS380’s drive cooling capabilities and precarious backplane design ruled out what may have been a better fit with some creative fabrication of airflow channels.  The Chenbro case was paired with: (2) of the 4-bay, 3.5″ Hot Swap bays that reside behind the front panel door.  The three 5.25″ bays were converted to provide (5) additional 3.5″ Hot Swap bays using the SK33502 solution.  This provided a grand total of 13 3.5″ hot swap bays.

(1) Biostar B350GT5 motherboard:  With a 16x and 4x slot available, our design goals were almost met with this board.  It’s been incredibly stable, offers a dual BIOS setup that is controlled by a dip switch, and includes (2) PCI slots.  We’ll delve into our findings and challenges on PCI slot operation in another post.

(4) 16GB Kingston DDR4-2133 ECC UDIMMs

(1) Intel i340-T4 Quad Port Gigabit NIC:  Effectively a gold standard of compatibility for both ESXi and FreeNAS.

(1) LSI Logic 9305-16i 12Gb SAS HBA:  The gold standard for FreeNAS.  With the goal of virtual FreeNAS, the controller will be passed through to the guest.  In addition to this compatibility, we’ll exceed the available hot swap drive bay count with a better power and cooling profile versus the 9300 controller.

(6) 5 TB hard drives and (4) 4 TB hard drives:  We’ll have a total of two pools within this solution that consist of a 20 TB RAID-Z2 pool for general purpose file storage, and an 8 TB RAID-10 pool for virtualization-related tasks.

(1) 750W FSP Hydro G Gold-rated Modular PSU:  We had it on hand and it works exceptionally well.  Provisioning it for this solution prevents it from becoming eWaste.  This model received a Gold award at HardOCP.

(1) 480 GB Crucial M5 SATA SSD:  Another carryover from prior systems.  The SSD is attached to one of the four onboard SATA ports provided by the B350 chipset and simply holds the hypervisor + the boot image for FreeNAS.

Although performance exceeded expectations with this setup, the noise factor remained an outstanding issue.  While the chassis and associated drive bays were slightly more quiet than a 2U SuperMicro server chassis, continuous operation close to living quarters will produce enough fan noise to make this type of setup a non-starter for home lab environments.  This noise, when combined with the SMB bug that was introduced in FreeNAS 11.1 U3, resulted in revisiting the prior deployment model for operations.