.DX.
10-06-2008, 13:25
This is the story of Helmer. A linux cluster in a IKEA Helmer cabinet.
http://helmer.sfe.se/1-helmer-1.jpg
3D computer rendering are very CPU intensive and the best way so speed up slow render problems, are usually to distribute them on to more computers. Render farms are usually very large, expensive and run using ALLOT of energy. I wanted to build something that could be put in my home, not make too much noise and run using very little energy... and be dirt cheep, big problem? :) no computer stuff cost almost nothing these days, it just a matter of finding fun stuff to play with.
I wanted to use Intel Quad core 65 nm or better. I surfed the web and found 6 of then at a good price. Then I looked after the cheapest motherboard that could run these. The result of my investigation was the Gigabyte S-series GA-G33M-DS2R/S2 card.
http://helmer.sfe.se/2-delar-helmer.JPG
..and my cat approve.
http://helmer.sfe.se/3-6st-intel-quad-core.JPG
http://helmer.sfe.se/4-48GB-ram.JPG
Next, finding memory. The motherboard could hold up to 8GB of memory each, and I wanted to maximize all I could. Here are the 12x4GB =48GBmemory modules.
http://helmer.sfe.se/5-moderkort.JPG
http://helmer.sfe.se/6-kort-pa-spisen.JPG
Mounting CPU coolers.
http://helmer.sfe.se/7-helmer-baksida.JPG
Next problem was to find a good computer case that could hold these motherboards. But this was a big problem. 6x cases cost almost as much as the motherboards and CPU's! So here was some room for improvement. I found the IKEA helmer cabinet. Perfect! Who knew IKEA made there stuff ATX compatible. :)
http://helmer.sfe.se/8-insida.JPG
Some extra support for backside of box.
http://helmer.sfe.se/9-natag-i-lada.JPG
Mounting power supplies and fans in zig zag pattern to maximize airflow.
http://helmer.sfe.se/10-natag-i-laga2.JPG
http://helmer.sfe.se/11-halvbyggd.JPG
Motherboard H1, H2 and H3 mounted. Fans on wrong side. I mounted 3 mm plexiglas under the motherboard so it would not short circuit when I installed them.
http://helmer.sfe.se/12-halvbyggd2.JPG
http://helmer.sfe.se/13-ihoppkoppling.JPG
Big mess on floor.
http://helmer.sfe.se/14-sladdar.JPG
http://helmer.sfe.se/15-disk-cloning.JPG
Disk cloning in progress. I used Fedora 8, and the command "dd" to clone the discs from the bootable System rescue CD disc (http://www.sysresccd.org/).
> dd if=/dev/sda of=/dev/sdb
http://helmer.sfe.se/19-helmer-klar.JPG
Done! I use the software Dr Queue (http://www.drqueue.org/) as batch job dispatcher. All connected to a 8 port 3 com gigabit switch. Common storage using nfs to a FreeNAS (http://www.freenas.org/) server running on a Via C7 machine (that just need 20 W power .. another story :) Instead of power switch I used a simple cable to the "pwr" jumper on motherboard. (Wires tejped on right side)
Instructions of how I did the DrQueue setup is here:
DrQueueHOW-TO.pdf (http://helmer.sfe.se/DrQueueHOW-TO.pdf)
http://helmer.sfe.se/18-garderoben.JPG
The most amazing is that this machine just cost as a better standard PC, but has 24 cores that run each at 2.4 Ghz, a total of 48GB ram, and just need 400W of power!! This means that it hardly gets warm, and make less noise then my desktop pc.
Render jobs that took all night, now gets done in 10-12 min.
..next one will be in red..
/Janne
www.sfe.se (http://www.sfe.se/)
Update 1:
It has been crazy last days. I have got allot of questions regarding details, construction and software setup. I'll do a better page as soon... or I build the Red Helmer II :) Would be fun to use the AMD 16 core Bulldozer cpu x 6 = 96 core in a box :)))
If a hardware manufacturer/pr-guy happened to read this. I'm very happy to receive hardware play around with ;)
Update 2:
It has happened! I filled the ram, and all boards begun to swap. During this the power consumption was up to 800W. Running now a while, I noticed that H1 board (bottom one) gets hotter than the other boards. I think this is because the other boards are cooled from the fan below also.
Update 3:
Well, genral planning and webpage have begun on Red Helmer II - thank you Google :)
Update 4:
First alternative hardware for Red Helmer is selected, and will do about 12Tflops. (64x times faster then Helmer 1) and cost about the same :) Stay tuned for more!
Update 5:
A hardware god may help me, and then Helmer II will do about 50 TFLOPS!!!! please please, may the force be with me :)
Helmer Data Sheet (http://helmer.sfe.se/HelmerDataSheet.pdf)
More Helmer pics here (http://helmore.sfe.se/)
http://helmer.sfe.se/helmer1-1024x768.jpg (http://helmer.sfe.se/helmer1-1024x768.png)
Get the Helmer desktop image! (http://helmer.sfe.se/helmer1-1024x768.png)
Some render tests result here:
The storming sea scene (http://helmer.sfe.se/HelmerRendertest1.jpg) file 24 frames in resolution 4000x4000 pixels.
- On a MacPro 2 x DualCore Xenon 2.66 Ghz with 4 GB ram
took 552 minutes (9.2 h)
- Helmer did same 24 frames in 4k format in 64 min.
Some approximate numbers give Helmer a floating point capacity of 186 Gflops
My other life-hacks are here:
Building a wind sensor from a PS2-mouse (http://nslu2windsensor.sfe.se/)
Building a small linux robot (http://robot.sfe.se/)
Building a underwater camera house (http://underwaterhouse.sfe.se/)
Building a solar powered UAV (http://dementor.sfe.se/)
A better spelled version of ths page is here (by Dan - Thanks:) (http://helmer.sfe.se/thedanpage.html)
http://helmer.sfe.se/20-snapshot1.jpg
http://helmer.sfe.se/
http://helmer.sfe.se/1-helmer-1.jpg
3D computer rendering are very CPU intensive and the best way so speed up slow render problems, are usually to distribute them on to more computers. Render farms are usually very large, expensive and run using ALLOT of energy. I wanted to build something that could be put in my home, not make too much noise and run using very little energy... and be dirt cheep, big problem? :) no computer stuff cost almost nothing these days, it just a matter of finding fun stuff to play with.
I wanted to use Intel Quad core 65 nm or better. I surfed the web and found 6 of then at a good price. Then I looked after the cheapest motherboard that could run these. The result of my investigation was the Gigabyte S-series GA-G33M-DS2R/S2 card.
http://helmer.sfe.se/2-delar-helmer.JPG
..and my cat approve.
http://helmer.sfe.se/3-6st-intel-quad-core.JPG
http://helmer.sfe.se/4-48GB-ram.JPG
Next, finding memory. The motherboard could hold up to 8GB of memory each, and I wanted to maximize all I could. Here are the 12x4GB =48GBmemory modules.
http://helmer.sfe.se/5-moderkort.JPG
http://helmer.sfe.se/6-kort-pa-spisen.JPG
Mounting CPU coolers.
http://helmer.sfe.se/7-helmer-baksida.JPG
Next problem was to find a good computer case that could hold these motherboards. But this was a big problem. 6x cases cost almost as much as the motherboards and CPU's! So here was some room for improvement. I found the IKEA helmer cabinet. Perfect! Who knew IKEA made there stuff ATX compatible. :)
http://helmer.sfe.se/8-insida.JPG
Some extra support for backside of box.
http://helmer.sfe.se/9-natag-i-lada.JPG
Mounting power supplies and fans in zig zag pattern to maximize airflow.
http://helmer.sfe.se/10-natag-i-laga2.JPG
http://helmer.sfe.se/11-halvbyggd.JPG
Motherboard H1, H2 and H3 mounted. Fans on wrong side. I mounted 3 mm plexiglas under the motherboard so it would not short circuit when I installed them.
http://helmer.sfe.se/12-halvbyggd2.JPG
http://helmer.sfe.se/13-ihoppkoppling.JPG
Big mess on floor.
http://helmer.sfe.se/14-sladdar.JPG
http://helmer.sfe.se/15-disk-cloning.JPG
Disk cloning in progress. I used Fedora 8, and the command "dd" to clone the discs from the bootable System rescue CD disc (http://www.sysresccd.org/).
> dd if=/dev/sda of=/dev/sdb
http://helmer.sfe.se/19-helmer-klar.JPG
Done! I use the software Dr Queue (http://www.drqueue.org/) as batch job dispatcher. All connected to a 8 port 3 com gigabit switch. Common storage using nfs to a FreeNAS (http://www.freenas.org/) server running on a Via C7 machine (that just need 20 W power .. another story :) Instead of power switch I used a simple cable to the "pwr" jumper on motherboard. (Wires tejped on right side)
Instructions of how I did the DrQueue setup is here:
DrQueueHOW-TO.pdf (http://helmer.sfe.se/DrQueueHOW-TO.pdf)
http://helmer.sfe.se/18-garderoben.JPG
The most amazing is that this machine just cost as a better standard PC, but has 24 cores that run each at 2.4 Ghz, a total of 48GB ram, and just need 400W of power!! This means that it hardly gets warm, and make less noise then my desktop pc.
Render jobs that took all night, now gets done in 10-12 min.
..next one will be in red..
/Janne
www.sfe.se (http://www.sfe.se/)
Update 1:
It has been crazy last days. I have got allot of questions regarding details, construction and software setup. I'll do a better page as soon... or I build the Red Helmer II :) Would be fun to use the AMD 16 core Bulldozer cpu x 6 = 96 core in a box :)))
If a hardware manufacturer/pr-guy happened to read this. I'm very happy to receive hardware play around with ;)
Update 2:
It has happened! I filled the ram, and all boards begun to swap. During this the power consumption was up to 800W. Running now a while, I noticed that H1 board (bottom one) gets hotter than the other boards. I think this is because the other boards are cooled from the fan below also.
Update 3:
Well, genral planning and webpage have begun on Red Helmer II - thank you Google :)
Update 4:
First alternative hardware for Red Helmer is selected, and will do about 12Tflops. (64x times faster then Helmer 1) and cost about the same :) Stay tuned for more!
Update 5:
A hardware god may help me, and then Helmer II will do about 50 TFLOPS!!!! please please, may the force be with me :)
Helmer Data Sheet (http://helmer.sfe.se/HelmerDataSheet.pdf)
More Helmer pics here (http://helmore.sfe.se/)
http://helmer.sfe.se/helmer1-1024x768.jpg (http://helmer.sfe.se/helmer1-1024x768.png)
Get the Helmer desktop image! (http://helmer.sfe.se/helmer1-1024x768.png)
Some render tests result here:
The storming sea scene (http://helmer.sfe.se/HelmerRendertest1.jpg) file 24 frames in resolution 4000x4000 pixels.
- On a MacPro 2 x DualCore Xenon 2.66 Ghz with 4 GB ram
took 552 minutes (9.2 h)
- Helmer did same 24 frames in 4k format in 64 min.
Some approximate numbers give Helmer a floating point capacity of 186 Gflops
My other life-hacks are here:
Building a wind sensor from a PS2-mouse (http://nslu2windsensor.sfe.se/)
Building a small linux robot (http://robot.sfe.se/)
Building a underwater camera house (http://underwaterhouse.sfe.se/)
Building a solar powered UAV (http://dementor.sfe.se/)
A better spelled version of ths page is here (by Dan - Thanks:) (http://helmer.sfe.se/thedanpage.html)
http://helmer.sfe.se/20-snapshot1.jpg
http://helmer.sfe.se/