I reported a security issue in Supermicro's IPMI implementation to them. They dismissed the issue and never fixed it.
These companies don't care if there's no way to stop a remote control service (iLo, iDRAC, IPMI) from binding to a motherboard ethernet port. In the case of Supermicro, you can't configure via the built-in BIOS, there are no jumpers you can use, and you must have a full network setup and installed tools to configure a new machine. This makes deploying in the field much, much more problematic.
If the BIOS battery dies and settings set are lost, the motherboard defaults to joining whichever ethernet is active, with default credentials. It's incredibly stupid and insecure.
I will never buy Supermicro again, but for the hardware that was already bought, I mandated loopback plugs for all IPMI ports so the IPMI wouldn't switch to the system ethernet.
Supermicro switched to unique default passwords (same as Dell/HPE) a few years back. Silly to not have day one but really when you look at any of these they are all a security nightmare.
For most probably as you have to know to order it specifically with secure password unlike HPE/Supermicro which just do it every time now. Likely they got hit with a government/large customer requirement and half assed it, as most things BMC are.
I like to quote the Internet Census of 2012 [1] whenever that question comes up:
"A lot of devices and services we have seen during our research should never be connected to the public Internet at all. As a rule of thumb, if you believe that "nobody would connect that to the Internet, really nobody", there are at least 1000 people who did. Whenever you think "that shouldn't be on the Internet but will probably be found a few times" it's there a few hundred thousand times. Like half a million printers, or a Million Webcams, or devices that have root as a root password."
They are often inline with eth0. Sometimes they have a dedicated ilo/ipmi/etc, but if that's not connected it sticks ilo on eth0.
I once connected a server to a network and it wouldn't take the room's VGA KVM, so I started a dhcp server on my laptop to get onto the ipmi
63 leases were given out in a few seconds -- the lights-out (ilo, ipmi) that were on that network had never been configured, were just sat sending out dhcp requests and getting no response. The servers themselves were all statics, but nobody had bothered setting up the ilos. Most were of the age of a default password (ADMIN/ADMIN for supermicro for example)
Now this is on a private network with internet via a proxy, so fairly bad, but not catastrophic. It shows how easy it is to connect something to the internet accidently though.
Personally I put ilos on a dedicated vlan with an ACL blocking access to anywhere other than the management servers, it's not just incoming connections from the internet or from compromised machines behind your firewall, it's outgoing connections from the ilo too.
Which means everything is fine, then the ipmi switch port is shut down, and suddenly you're exposed in an unexpected fashion. It's not good. At least the password is no longer ADMIN/ADMIN on new kit, although I had one supermicro machine connected for me in India last week which was old enough that it was ADMIN/ADMIN
If you have a single server in colocation you may not have a choice. The only thing that allows me to sleep at night is that SM's IPMI has a firewall and you can whitelist a limited number of IPs.
BMCs are slightly different I think (more powerful!) but at my previous workplace we once had a server provided to us by our hosting provider that had its BMC exposed on the internet with default credentials. They never told us the machine even had one (most of our servers from them didn’t). We only figured it out once we found a Monero miner on the machine.
People on here love promoting dedicated servers over cloud VMs, but it’s so much easier for this sort of thing to go wrong with dedicated hosting.
Integrated Lights-Out (iLO) is the HP flavor of baseboard management controller (BMC) platform. Dell calls theirs Dell Remote Access Controller (DRAC), other vendors have their own branding. They all do more or less the same thing.
BMC is the hardware, iLO is the interface. you can also have other types of interfaces but in the end it’s still a BMC.
a chip that sits on the southbridge of the server and has management capabilities (power operations and access to the underlying os being the 2 big ones).
The latest generations of servers ship with randomized BMC passwords. This was indeed a problem in the past when they shipped with credientials such as ADMIN / ADMIN or no password at all.
"Latest generation" = last 2 years. Three years ago I was buying SMC servers which were the first round configured with randomized passwords to comply with CA law (some batches had ADMIN/ADMIN) and gigabyte servers still were out of compliance.
I bought my last batch of Dell machines in 2017 (so, 5 years? wow), and they had randomised passwords.
But yes, some providers put the BMC on the internet because it's easier, a provider I used once did this and I was quite displeased as iDRAC's are quite weak and suffer under the weight of bot-spam. -- even if there were no security issues.
It is possible not every gets this option from Dell but one can get a text file that has all the MAC addresses and programmed passwords for the dracs from them. Ask your rep how to get this. One can potentially use this information in their automation. The list has serial numbers, eth macs, ilo mac, password, model, etc...
As far as I can remember, HP has always randomized the default iLO password. ProLiants used to come with a "toe tag" (little card attached to rack mount server by string) that had the iLO MAC, iLO hostname, default username and default password. Newer ProLiants put it on a sticker on the server itself or on a pull-out tab.
This is actually really much lower than I expected..
Though what helps is that most servers have a dedicated iLO interface and you really have to choose to configure it on the regular ones along with normal traffic. So out of band is default.
So in this case it's only people who have deliberately configured this. I think this is why it's not hundreds of thousands.
Not sure I agree with you 100% on your police work there, Lou.
I worked with a good number of SuperMicro servers that - to my surprise - the IPMI interface was by default set to "fail over to bridge to primary interface". That is to say if the IPMI port was disconnected, it would bridge onto your primary NIC with a second MAC address. The way to disable it was to download an obscure utility from SuperMicro's website, which only ran under Windows, and to pass an equally obscure hexadecimal command line flag.
Fortunately in my case, these ran on a network with no DHCP server, but I can't assume that's true in every case.
Yeah those supermicros scared the bejeezus out of me when I saw them in the catalog. I mean, yeah, you can configure the switch to listen onto different vlan ids, but what if something weird happens and that stateful information is lost or corrupted?
Yeah, that's why I never put WAN on a shared interface.
HP DOES THIS TOO. At least, they did. For example, on some g6 servers iLO/LO100 physical interface was optional (a small but costly add-on card) and, when not installed, iLO was shared with normal eth.
If someone manages to hack a server enough to reset the shared iLO, at least it shouldn't immediately pop up into internet.
We once (~2 years ago) had an Asus server with their BMC, which is much easier to expose: while it has a dedicated physical interface by default it is also exposed on the first LAN port and configured to use DHCP. I can easily see how those boards leave you accidentally exposed in a colocation setup. But I'm still struggling to come up with a way to accidentally expose that on the internet. Public IPv4 isn't usually handed out via DHCP.
My best guess is that the vast majority of these 20k servers have their management interface deliberately exposed. Only takes 2000 people with 10 servers each to think this is a good idea.
I've only dabbled with iLO but even when operating on the same interface I'm not sure how this gets onto the internet? From what I remember you configure it with a static IP, so these people surely are configuring their iLO to listen on a publicly routable IP?
The amount of access that BMCs have to the system seems nuts to me. How about a BMC-esque thing whose only function is to provide remote access to the host machine's essential interfaces?
On one side, an ethernet port, that you plug into your management plane. On the other side, a serial port, to access the host's console, a device-side SATA port, so it can present a virtual boot disk to the host, and a wire to the power supply, to turn the host on and off. Maybe also some read-only interface to motherboard-level monitoring like SMART and so on; maybe you get that by cooperation with the host's firmware, over the serial or SATA interface. That would let you bring a machine up and configure it, reboot it remotely, force it to boot from a recovery disk, etc. But a hacked BMC couldn't be used to subtly interfere with the host.
Also, does anyone know what Oxide are doing about BMC?
The older iLO2 and 3 were from the Gen7/8 HP rack mount servers, which were released around 2013 or something like that. I'd have thought that the majority of commercial uses have long passed, so I was wondering if these generation of machines are really being used for home labs, that sort of thing, as they are passed their use by date (I think the gen7 machines were EOSL'd in 2018 so they'll have been chucked out of datacentres).
Actually, I can imagine a fair few are still in use as small enterprise servers, with iLO being visible for remote admin, which is a shame, as that suggests they are not behind a tunnel, so those addresses probably indicate larger problems than a visible iLO.
I've had the bad fortune of dealing with iLO in the past. There's absolutely nothing surprising to me that it would be remotely exploitable, as well as default remote accessible.
Seems to me that we'd get some pretty good infosec bang-for-buck to hire some new FBI agents to just wear badges and knock on doors of businesses with such abjectly shittastic security.
Either their IT people have been begging to fix it and management won't give them the resources, or they don't have IT people who even see it as a problem. Either way, a couple special agents having a sit-down with the CEO might change some course real fast.
Sometimes people don't have choice - it's colocation, dc doesn't offer management vpn/firewalling, you can't install separate vpn router unless you pay for another unit in a rack. And iLO/LOM/WTF at least offers https... what can you do? Well, some people call dc to have their LOs (un-)plugged for maintenance time.
Is there a consensus on whether it's ok to port forward from the firewall to port 22 (SSH) on the BMC, with a strong admin password? I guess this should be safer than exposing the web interface.
You never want these interfaces on public internet. All my iDRAC ports are on an isolated management vlan that is only routed to from certain internal networks.
1. You don't want the device even considering requests from anyone but the sole person(s) responsible for accessing the mgmt interface. Someone might not be able to get in, but they could enumerate your hardware characteristics, or perform a denial of service.
2. Don't confuse SSH with secure. You don't know what version/brand/make/model of SSH is running on the embedded device. It might be ancient. It might not even be OpenSSH.
3. If an exploit becomes known for your management interfaces on your machines, you are screwed. First because someone might have already exploited you. Second because now you need to patch your boxes, and that might require a hard reboot of the entire machine. If a patch even exists.
Really, it isn't worth the trouble. There are too many risks.
I love how iDrac shipped with a password everyone knows. I would have thought it was trivial to make it a 1-time passthrough to change it, to stop this being the cartoon character we know and love.
These companies don't care if there's no way to stop a remote control service (iLo, iDRAC, IPMI) from binding to a motherboard ethernet port. In the case of Supermicro, you can't configure via the built-in BIOS, there are no jumpers you can use, and you must have a full network setup and installed tools to configure a new machine. This makes deploying in the field much, much more problematic.
If the BIOS battery dies and settings set are lost, the motherboard defaults to joining whichever ethernet is active, with default credentials. It's incredibly stupid and insecure.
I will never buy Supermicro again, but for the hardware that was already bought, I mandated loopback plugs for all IPMI ports so the IPMI wouldn't switch to the system ethernet.