Circumference Datacentre-in-a-Box Features up to 32 Raspberry Pi 3 B+ Managed by a UDOO x86 Ultra Board (Crowdfunding)

14 Replies to “Circumference Datacentre-in-a-Box Features up to 32 Raspberry Pi 3 B+ Managed by a UDOO x86 Ultra Board (Crowdfunding)”

1. The design is so freaking cool, the 32x Pi model looks like a 1965 DEC PDP-8.
   Really nice project.

   Reply

2. What’s the purpose of the fans?

   Reply

3. Just as a reference: for anyone interested in SBC clusters a design that takes care of sufficient heat dissipation (unlike the above designs) is mandatory since why would anyone want throttling?

   Here is a nice example also making use of custom powering backplanes and allowing for efficiently cooling the individual boards due to proper orientation and position of the fans: http://forum.khadas.com/t/vim2-based-crypto-currency-miner-build/

   And of course for a low count cluster using something like Pine’s Clusterboard with a couple of SoPine modules is a way better approach than cramping normal SBCs with a bunch of Ethernet switches and cables in a box. Though suffers from the same problem as RPi 3/3+: Cortex-A53 in 40nm is too slow and generates way too much heat at the same time.

   Reply

   1. Good point on the thermal.
      I was also wondering how reliable such a setup would be.

      Reply

      1. > I was also wondering how reliable such a setup would be

         The primary OS on those Raspberries (a RTOS called ThreadX) implements throttling on its own starting at 80°C. Of course the secondary OS (Linux for example) won’t know and while the boards (especially those in the middle of such a stack) in reality are throttled down to e.g. 800 MHz due to overheating the kernel will still happily report 1200 MHz or even 1400 MHz on the Pi 3+ (RPi cheating as usual).

         The first line only returns bogus values while only the 2nd call provides the real clockspeed (talking directly to ThreadX that controls the hardware):

         cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq vcgencmd measure_clock arm

         1 2	cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq vcgencmd measure_clock arm

         (reference: https://github.com/ThomasKaiser/OMV_for_Raspberries/blob/master/usr/sbin/raspimon)

         So this is not that much a problem of reliability but lower performance after some time of sustained load when the boards in the middle start to overheat. It’s interesting to see a ‘cluster design’ in 2018 simply not caring about appropriate heat dissipation at all. But most probably the target audience doesn’t care anyway…

         Reply

         1. My concern was also wrt reliability of the nodes.
            I’m not sure if the setup allows for netboot or local storage.
            My experience with SD cards over time is not that good.

            Reply

            1. > I’m not sure if the setup allows for netboot or local storage.

               Both. The RPi SoC allows for some sort of proprietary netbooting in the meantime. But even with local storage on SD card there’s nothing to worry as long as you keep the ‘flash memory’ challenges in mind:

               1) Always be aware that there’s fake/counterfeit crap around faking a higher capacity than the real one so cards need to be tested prior to usage (since otherwise it’s game over once the amount of data written exceeds the real capacity — such counterfeit cards die over thousand times faster than cards with real capacity)

               2) ‘Write Amplification’. It’s really all about this and distros *not* taking care of this (e.g. Raspbian) will lead to the SD card wearing out magnitudes faster compared to distros that address this problem (increasing the commit interval so stuff gets written to flash storage only every 10 minutes in a batch which already greatly reduces Write Amplification, same with keeping logs in RAM and writing to card hourly/daily).

               I tried to explain the importance of lowering the write amplification in length here: https://forum.armbian.com/topic/6444-varlog-file-fills-up-to-100-using-pihole/?do=findComment&comment=50833

               Anyway… if it’s just about a ‘cluster to go’ for educational purposes I would still better look at Pine’s Clusterboard: https://www.cnx-software.com/2018/02/01/pine64-clusterboard-is-now-available-for-100-with-one-sopine-a64-system-on-module/

               For less than 400 bucks you get everything you need: the Clusterboard containing a GbE switch, 7 sockets for the SoPine modules (29 bucks each), the appropriate 75W PSU and any Mini ITX enclosure. The SoPine modules have some SPI NOR flash you can burn a bootloader into that makes all devices PXE boot compliant. So simply throw another SBC with an SSD as NFS server into the enclosure and you’re done getting an ‘SD cards free’ cluster booting in a standards compliant way (skipping the proprietary RPi VideoCore stuff)

4. While I find that it *looks* nice, I’m really shocked anyone can think about using an inefficient RPi to build a cluster. You build clusters because you need performance, not for community support. In this design, the x86 board controlling the cluster might have almost as much power as all the RPis together (OK I’m exagerating a bit though not that much when they start to throttle, but you get it). There are a lot of alternative boards around for various purposes. The NanoPI-Fire3 provides 8*1.4 GHz A53. Stronger per-core performance can be obtained using RK3288 as found in MiQi or Tinkerboard, as well as various of the Exynos 5422-based Odroid boards. A single MiQi, Fire3 or Odroid-MC1 board would roughly replace 3 RPi, reducing the number of switch ports, fans and the enclosure size. It’s sad that people don’t study what’s available before rushing on a well-known generic but inefficient device. Sometimes I’m wondering if one day we’d not even see clusters of Arduinos…

   Reply

   1. They don’t sell the boards with the kit, so it should be easy enough to use Tinker boards instead of Raspberry Pi 3 boards.

      Reply

      1. Those RK3288 boards like Tinkerboard or MiQi suffer even more from the broken thermal design of these cluster boxes here. They need efficient heat dissipation to not throttle down to a minimum which is impossible with this box (only the board at the stack’s top receives appropriate cooling, the 7 below not)

         Which is fascinating since the guys behind the project (Ground Electronics) know how to do it correctly (maximum airflow over the boards/heatsinks with a fan blowing laterally): https://groundelectronics.com/products/parallella-cluster-kit/

         But maybe it’s really all about the target audience? If people want to build a cluster out of RPi they must be clueless like hell (or not giving a sh*t about anything that’s important) so better offer them something looking nice over functioning efficiently?

         Reply

         1. > only the board at the stack’s top receives appropriate cooling, the 7 below not

            I didn’t notice when I first saw the photos, you’re absolutely right, what an horror! So it’s just a noise generator.

            Reply

   2. > You build clusters because you need performance, not for community support.

      They claim such a physical cluster setup would be great for educational purposes. I do not agree that much since Datacentre-in-a-laptop is IMO the better idea if you want to learn clustering and be mobile at the same time. My latest laptop has 16 GB DRAM which allows for 20 virtual servers easily.

      Reply

   3. https://www.crowdsupply.com/ground-electronics/circumference/updates/a-gnu-parallel-clustering-demo

      Unbelievable. They really only care about the most inefficient SBC possible for cluster use cases (Raspberries). A cluster job when running on 8 nodes being just 3.72 faster compared to a single node is really ‘impressive’. They even explain why using Raspberries for this is such a crappy idea:

      ‘This overhead eats into the potential performance gain – as does the 100 Mb/s network port of the Raspberry Pi 3 Model B, which can be improved by a rough factor of three by switching the nodes to use the newer and faster Raspberry Pi 3 Model B+ with its gigabit-compatible Ethernet port’

      As if powerful boards with real Gigabit Ethernet being 10 times faster wouldn’t exist. But unfortunately due to the broken thermal design of the box also more performant boards would run throttled to the minimum.

      Reply

5. This a ridiculous for the price considering you still have to purchase the boards and other components. Good luck.

   Reply

Leave a Reply

Connect with:

Your email address will not be published. Required fields are marked *

Comment *

Name *

Email *

Website