Update (2023-01): This failed because the motherboard I got was faulty and doesn’t boot with the 2nd CPU. Since DDR-4 is cheaper now, I decided to abandon this project.
I wanted a cheap powerful system with plenty of RAM and PCIe sockets.
Since DDR4 is still a bit expensive, and DDR3 is cheap now, it means Ivy Bridge or earlier.
Eventually, I came a across a very cheap Z9PG-D16 online, which even included adapter cables for powering it with a standard ATX PSU, and I couldn’t resist, so I bought it.
I’ll document the build here.
The pros are:
- Remote management with IPMI (AST2300-based)
- 8 RAM slots per CPU for a maximum of 16x32G = 512GB RAM. I’ll use cheaper 16G sticks, so I’ll have 256GB usable RAM.
- 1 16x and 4 8x PCIe slots from CPU-1
- 4 8x PCIe slots from CPU-2 (most motherboards don’t use many lanes from CPU-2 at all!)
The cons are:
- Very non-standard form factor. This is meant to be in a 2U chassis and I don’t have that one, so I have to make the enclosure myself. Consumer GPUs with waterblocks won’t fit in the 2U chassis anyways.
- It doesn’t come with PCIe risers. I’ll have to source 16x to 2 8x bifurcating risers and hope they work.
- Very few USB ports for a workstation. I’ll have to buy additional USB Cards. This might be an advantage, since I get to choose the chipset.
- It’s old.
So it finally arrived, and I attached the power adapter. The adapter takes the ATX 24-pin, a 6-pin GPU power, and a 8-pin CPU power and adapts it to the 20-pin PWR1 proprietary connector. PWR2 is just a standard 8-pin CPU power connector, so no adapter needed there.
As expected, BMC turns on and boots. It was configured to DHCP on the LAN port 3 (the one above the USB ports) and had the default username and password (admin:admin).
Unfortunately, the remove KVM seems to require java 1.6 enabled in the browser, and even then something doesn’t work right.
It seems like the jnlp gets cut off for some reason, and is unusable, and for some reason, when I ssh into it, i get dropped straight into a busybox shell, and I’m able to change the root password, so now I can ssh in as root.
It seems like the BMC firmware is somehow damaged, and I suspect an upgrade will fix it, but for now I’ll leave it as it is, it might turn out to be useful.
I’ve ordered a test clip so I can dump the SPI ROM before I upgrade, and maybe even make my own firmware for it. It looks like a very standard AST2300, and people say it’s very similar to the AST2400 which has a openbmc port. It has less memory, so while openbmc probably won’t fit, I’m sure something useful can be cobbled together.
ipmitool works, and I’m able to start a SOL session, but nothing is returned on it. I probably have to configure BIOS to enable serial redirection.
And when connecting a monitor, I’m not able to get it to boot.
I tried running with only 1 CPU, and after pulling the BIOS chip and reprogramming it with the latest BIOS, it boots. I’m still not sure if it’s the second socket, or the BIOS which was the problem, I’ll test that when I have time to work on it again.
Really nice of Asus to make the BIOS rom chip easy to remove and replace. I should definitively make a similar adapter for the BMC ROM so it’s easier to roll out new firmware for it, something that fits my 3.3V-modified ch341a programmer.
Next, I’ll have to get/make some kind of chassis for it. The idea is to use PCIe Riser cables to Bifurcation cards so the GPU(s) and USB cards can be placed more freely.
Since I already have water blocks and radiators, CPUs and GPUs will be water cooled.
I’m thinking to make a tower design, with the motherboard mounted vertically with LAN/USB connectors facing down and the power connectors at the top. Then, the tubes for the water cooling are also routed up from the CPUs, and down behind the motherboard. The GPUs can be mounted on the sides, with flexible riser cables for easy access to the connectors. Hoping to find some good USB cards to add as well, since 5 USB 2.0 ports are probably not enough.