Discussion:
[IP] re Setting the Record Straight on Bloomberg BusinessWeek’s Erroneous Article | AWS Security Blog
Dave Farber
2018-10-06 00:56:46 UTC
Permalink
Worth reading carefully djf
Date: October 6, 2018 at 3:06:55 AM GMT+9
Subject: Re: [IP] Setting the Record Straight on Bloomberg BusinessWeek’s Erroneous Article | AWS Security Blog
Also worth reading is The Register's coverage, which (among other things) points out some holes that weaken Apple and Amazon's apparently-strong statements: https://www.theregister.co.uk/2018/10/04/supermicro_bloomberg/
Security
Insider threat
Decoding the Chinese Super Micro super spy-chip super-scandal: What do we know – and who is telling the truth?
Who's your money on? Bloomberg's sources? Apple? Amazon? Super Micro?
By Kieren McCarthy in San Francisco 4 Oct 2018 at 23:01
136 SHARE ▌
Analysis Chinese government agents sneaked spy chips into Super Micro servers used by Amazon, Apple, the US government, and about 30 other organizations, giving Beijing's snoops access to highly sensitive data, according to a bombshell Bloomberg report today.
The story, which has been a year in the making and covers events it says happened three years ago, had a huge impact on the markets: the company at the center of the story, San Jose-based Super Micro, saw its share price drop by nearly 50 per cent; likewise Apple's share price dropped by just under two per cent, and Amazon's dropped by more than two per cent.
But the article has been strongly denied by the three main companies involved: Apple, Amazon, and Super Micro. Each has issued strong and seemingly unambiguous statements denying the existence and discovery of such chips or any investigation by the US intelligence services into the surveillance implants.
These statements will have gone through layers of lawyers to make sure they do not open these publicly traded corporations to lawsuits and securities fraud claims down the line. Similarly, Bloomberg employs veteran reporters and layers of editors, who check and refine stories, and has a zero tolerance for inaccuracies.
So which is true: did the Chinese government succeed in infiltrating the hardware supply chain and install spy chips in highly sensitive US systems; or did Bloomberg's journalists go too far in their assertions? We'll dig in.
The report
First up, the key details of the exclusive. According to the report, tiny microchips that were made to look like signal conditioning couplers were added to Super Micro data center server motherboards manufactured by sub-contractors based in China.
Those spy chips were not on the original board designs, and were secretly added after factory bosses were pressured or bribed into altering the blueprints, it is claimed. The surveillance chips, we're told, contained enough memory and processing power to effectively backdoor the host systems so that outside agents could, say, meddle with the servers and exfiltrate information.
The Bloomberg article is not particularly technical, so a lot of us are having to guesstimate how the hack worked. From what we can tell, the spy chip was designed to look like an innocuous component on the motherboard with a few connector pins – just enough for power and a serial interface, perhaps. One version was sandwiched between the fiberglass layers of the PCB, it is claimed.
The spy chip could have been placed electrically between the baseboard management controller (BMC) and its SPI flash or serial EEPROM storage containing the BMC's firmware. Thus, when the BMC fetched and executed its code from this memory, the spy chip would intercept the signals and modify the bitstream to inject malicious code into the BMC processor, allowing its masters to control the BMC.
The BMC is a crucial component on a server motherboard. It allows administrators to remotely monitor and repair machines, typically over a network, without having to find the box in a data center, physically pull it out of the rack, fix it, and re-rack it. The BMC and its firmware can be told to power-cycle the server, reinstall or modify the host operating system, mount additional storage containing malicious code and data, access a virtual keyboard and terminal connected to the computer, and so on. If you can reach the BMC and its software, you have total control over the box.
With the BMC compromised, it is possible the alleged spies modified the controller's firmware and/or the host operating system and software to allow attackers to connect in or allow data to flow out. We've been covering BMC security issues for a while.
Here is Bloomberg's layman explanation for how that snoop-chip worked: the component "manipulated the core operating instructions that tell the server what to do as data move across a motherboard
 this happened at a crucial moment, as small bits of the operating system were being stored in the board’s temporary memory en route to the server’s central processor, the CPU. The implant was placed on the board in a way that allowed it to effectively edit this information queue, injecting its own code or altering the order of the instructions the CPU was meant to follow."
There are a few things to bear in mind: one is that it should be possible to detect weird network traffic coming from the compromised machine, and another is that modifying BMC firmware on the fly to compromise the host system is non-trivial but also not impossible. Various methods are described, here.
"It is technically plausible," said infosec expert and US military veteran Jake Williams in a hastily organized web conference on Thursday morning. "If I wanted to do this, this is how I'd do it."
The BMC would be a "great place to put it," said Williams, because the controller has access to the server's main memory, allowing it to inject backdoor code into the host operating system kernel. From there, it could pull down second-stage spyware and execute it, assuming this doesn't set off any firewall rules.
A third thing to consider is this: if true, a lot of effort went into this surveillance operation. It's not the sort of thing that would be added to any Super Micro server shipping to any old company – it would be highly targeted to minimize its discovery. If you've bought Super Micro kit, it's very unlikely it has a spy chip in it, we reckon, if the report is correct. Other than Apple and Amazon, the other 30 or so organizations that used allegedly compromised Super Micro boxes included a major bank and government contractors.
A fourth thing is this: why go to the bother of smuggling another chip on the board, when a chip already due to be placed in the circuitry could be tampered with during manufacture, using bribes and pressure? Why not switch the SPI flash chip with a backdoored one – one that looks identical to a legit one? Perhaps the disguised signal coupler was the best way to go.
And a fifth thing: the chip allegedly fits on a pencil tip. That it can intercept and rewrite data on the fly from SPI flash or a serial EEPROM is not impossible. However, it has to contain enough data to replace the fetched BMC firmware code, that then alters the running operating system or otherwise implements a viable backdoor. Either the chip pictured in Bloomberg's article is incorrect and just an illustration, and the actual device is larger, or there is state-of-the-art custom semiconductor fabrication involved here.
One final point: you would expect corporations like Apple and Amazon to have in place systems that detect not only unexpected network traffic, but also unexpected operating system states. It should be possible that alterations to the kernel and the stack of software above it should set off alarms during or after boot.
Bloomberg claims the chip was first noticed in 2015 in a third-party security audit of Super Micro servers that was carried out when it was doing due diligence into a company called Elemental Technologies that it was thinking of acquiring. Elemental used Super Micro's servers to do super-fast video processing.
Big problem
Amazon reported what it found to the authorities and, according to Bloomberg, that "sent a shudder" through the intelligence community because similar motherboards were in use "in Department of Defense data centers, the CIA’s drone operations, and the onboard networks of Navy warships."
Around the same time, Apple also found the tiny chips, according to the report, "after detecting odd network activity and firmware problems." Apple contacted the FBI and gave the agency access to the actual hardware. US intelligence agencies then tracked the hardware components backwards through the supply chain, and used their various spying programs to sift through intercepted communications, eventually ending up with a focus on four sub-contracting factories in China.
According to Bloomberg, the US intelligence agencies were then able to uncover how the seeding process worked: "Plant managers were approached by people who claimed to represent Super Micro or who held positions suggesting a connection to the government. The middlemen would request changes to the motherboards’ original designs, initially offering bribes in conjunction with their unusual requests. If that didn’t work, they threatened factory managers with inspections that could shut down their plants. Once arrangements were in place, the middlemen would organize delivery of the chips to the factories."
This explanation seemingly passes the sniff test: it fits what we know of US intelligence agencies investigative approaches, their spy programs, and how the Chinese government works when interacting with private businesses.
The report then provides various forms of circumstantial evidence that adds weight to the idea that this all happened by pointing to subsequent actions of both Apple and Amazon. Apple ditched Super Micro entirely as a supplier, over the course of just a few weeks, despite planning to put in a massive order for thousands of motherboards. And Amazon sold off its Beijing data center to its local partner, Beijing Sinnet, for $300m.
Supply Chain Security Speculation
Everything thrown at the wall that seemed to stick
The illicit chips 
 were connected to the baseboard management controller
Before the wild speculation though, it must be mentioned that the story is short on evidence and high on flat out denials.
Update: more evidence from an earlier Ars Technica article seems to support the Bloomberg report.
Update: Amazon is pretty emphatic that everything Bloomberg said about them and Supermicro is wrong.
Update: In 2016 Apple did have security issues with Supermicro, but the circumstances are far from clear. It looks like maybe Apple is bluffing Supermicro about a bad firmware, then ghosts. If they actually did find a problem, engage in a coverup, then dump the whole problem on the .gov, it explains the weird messaging going on.
Update: Apple comes out swinging with another “nope!”
Update: ServeTheHome has a good write up on BMCs, but I think they may be attributing too much technical coherence to the Bloomberg article. The hypothetical attack – altering the password verification routine – is not particularly practical for an attacker. A backdoor with direct memory access, and just a few operations (read, write, jump) would be simpler, more robust, and much more useful.
Something is rotten in the state of supply chain attack reports
All of the named companies in the report flatly deny pretty much every statement in Bloomberg’s article. These denials are not “non-denial” denials, but directly refute specific statements of fact in Bloomberg’s report, as well as explicitly denying the core premise of the supply chain attack.
Bloomberg claims that the circa 2015 modchip, about “the size of a grain of rice,” was discovered by a third party security auditor. I can think of people who are capable of detecting this sort of modchip hack. I cannot think of a reason why a due diligence audit of a server would go down to that level.
On the other hand, Baseboard Management Controllers (BMC) and the Intelligent Platform Management Interface (IPMI) protocol are a horrendous tire fire for cyber security. That’s why Amazon’s statement about the audit rings true to me.
The pre-acquisition audit described four issues with a web application (not hardware or chips) that SuperMicro provides for management of their motherboards. All these findings were fully addressed before we acquired Elemental. The first two issues, which the auditor deemed as critical, related to a vulnerability in versions prior to 3.15 of this web application (our audit covered prior versions of Elemental appliances as well),
Two critical issues in the BMC web server (accessible over IPMI)
Two non critical ones (probably about encryption or lack thereof) that were mitigated by Amazon’s planned deployment
These findings ring true to me, this is what a typical infosec due diligence analysis is going to do — look at the interfaces and ports, see what functionality there is, what bugs there are, and what needs to be hardened/fixed.
firmware errors when reflashing the modchipped unit (checksums?)
unusual network traffic (e.g. beaconing) generated by the modchip
anything else weird and unusual that raises redflags
Supply chain attacks exist. Is this article accurate? It feels a little off, but I don’t know.
What do we know?
There’s not much we can speculate about the modchip because the Bloomberg description of whatever it does is gibberish. It is safer to simply examine what is known about Supermicro’s server boards.
Supermicro boards have third party BMC hardware to handle IPMI
There are at least three hardware providers: ASPEED, ATEN, and Nuvoton
ASPEED and Nuvoton use AMI software. ATEN has their own software stack
All Supermicro IPMI controllers appear to provide an extensive range of functionality that would be useful for an attacker
Keyboard Video Mou
se (KVM) over IP
SSH
Serial over LAN (SOL), and SSH over SOL
Web server (default login: ADMIN:ADMIN)
Remote power management

TCP 80, 443: web interface
TCP 3520, 5900: KVM access
TCP 623: menu access, allowing full control of the hardware
Good supply chain attack?
To compromise a server with a tiny modchip, a backdoor into the BMC would be pretty good. For example, a simple ICMP shell that beaconed out and provided basic commands to interact with the system would work in many places. The modchip’s backdoor would have to be more complex if the idea was to breach a hard target, but the BMC is certainly a good place to start.
So, what’s the deal?
For me, Bloomberg’s article could go either way. The logic of backdooring the BMC makes a lot of sense. Whether it happened in this case — given all the categorical denials — I have no idea.
The real takeaway from this is that IPMI is a raging tire fire, BMCs are Satan spawn, and never ever expose IPMI interfaces to the Internet. Unless you want hackers, because that’s how you get hackers.
Security
Infosec
Operational Security
China
Supply Chain
the grugq
Information Security Researcher :: keybase.io/grugq :: https://www.patreon.com/grugq
https://aws.amazon.com/blogs/security/setting-the-record-straight-on-bloomberg-businessweeks-erroneous-article/
Setting the Record Straight on Bloomberg BusinessWeek’s Erroneous Article
04 OCT 2018
Today, Bloomberg BusinessWeek published a story claiming that AWS was aware of modified hardware or malicious chips in SuperMicro motherboards in Elemental Media’s hardware at the time Amazon acquired Elemental in 2015, and that Amazon was aware of modified hardware or chips in AWS’s China Region.
As we shared with Bloomberg BusinessWeek multiple times over the last couple months, this is untrue. At no time, past or present, have we ever found any issues relating to modified hardware or malicious chips in SuperMicro motherboards in any Elemental or Amazon systems. Nor have we engaged in an investigation with the government.
There are so many inaccuracies in ‎this article as it relates to Amazon that they’re hard to count. We will name only a few of them here. First, when Amazon was considering acquiring Elemental, we did a lot of due diligence with our own security team, and also commissioned a single external security company to do a security assessment for us as well. That report did not identify any issues with modified chips or hardware. As is typical with most of these audits, it offered some recommended areas to remediate, and we fixed all critical issues before the acquisition closed. This was the sole external security report commissioned. Bloomberg has admittedly never seen our commissioned security report nor any other (and refused to share any details of any purported other report with us).
The article also claims that after learning of hardware modifications and malicious chips in Elemental servers, we conducted a network-wide audit of SuperMicro motherboards and discovered the malicious chips in a Beijing data center. This claim is similarly untrue. The first and most obvious reason is that we never found modified hardware or malicious chips in Elemental servers. Aside from that, we never found modified hardware or malicious chips in servers in any of our data centers. And, this notion that we sold off the hardware and datacenter in China to our partner Sinnet because we wanted to rid ourselves of SuperMicro servers is absurd. Sinnet had been running these data centers since we ‎launched in China, they owned these data centers from the start, and the hardware we “sold” to them was a transfer-of-assets agreement mandated by new China regulations for non-Chinese cloud providers to continue to operate in China.
Amazon employs stringent security standards across our supply chain – investigating all hardware and software prior to going into production and performing regular security audits internally and with our supply chain partners. We further strengthen our security posture by implementing our own hardware designs for critical components such as processors, servers, storage systems, and networking equipment.
Security will always be our top priority. AWS is trusted by many of the world’s most risk-sensitive organizations precisely because we have demonstrated this unwavering commitment to putting their security above all else. We are constantly vigilant about potential threats to our customers, and we take swift and decisive action to address them whenever they are identified.
– Steve Schmidt, Chief Information Security Officer
This message was sent to the list address and trashed, but can be found online.
-------------------------------------------
Archives: https://www.listbox.com/member/archive/247/=now
Modify Your Subscription: https://www.listbox.com/member/?member_id=26461375
Unsubscribe Now: https://www.listbox.com/unsubscribe/?member_id=26461375&id_secret=26461375-c2b8a462&post_id=20181005205659:B8593CC4-C902-11E8-9500-835B961E2E9A
Powered by Listbox: https://www.listbox.com
Dave Farber
2018-10-05 18:42:39 UTC
Permalink
Date: October 6, 2018 at 3:03:57 AM GMT+9
Subject: Re: [IP] Setting the Record Straight on Bloomberg BusinessWeek’s Erroneous Article | AWS Security Blog
Supply Chain Security Speculation
Everything thrown at the wall that seemed to stick
The illicit chips 
 were connected to the baseboard management controller
Before the wild speculation though, it must be mentioned that the story is short on evidence and high on flat out denials.
Update: more evidence from an earlier Ars Technica article seems to support the Bloomberg report.
Update: Amazon is pretty emphatic that everything Bloomberg said about them and Supermicro is wrong.
Update: In 2016 Apple did have security issues with Supermicro, but the circumstances are far from clear. It looks like maybe Apple is bluffing Supermicro about a bad firmware, then ghosts. If they actually did find a problem, engage in a coverup, then dump the whole problem on the .gov, it explains the weird messaging going on.
Update: Apple comes out swinging with another “nope!”
Update: ServeTheHome has a good write up on BMCs, but I think they may be attributing too much technical coherence to the Bloomberg article. The hypothetical attack – altering the password verification routine – is not particularly practical for an attacker. A backdoor with direct memory access, and just a few operations (read, write, jump) would be simpler, more robust, and much more useful.
Something is rotten in the state of supply chain attack reports
All of the named companies in the report flatly deny pretty much every statement in Bloomberg’s article. These denials are not “non-denial” denials, but directly refute specific statements of fact in Bloomberg’s report, as well as explicitly denying the core premise of the supply chain attack.
Bloomberg claims that the circa 2015 modchip, about “the size of a grain of rice,” was discovered by a third party security auditor. I can think of people who are capable of detecting this sort of modchip hack. I cannot think of a reason why a due diligence audit of a server would go down to that level.
On the other hand, Baseboard Management Controllers (BMC) and the Intelligent Platform Management Interface (IPMI) protocol are a horrendous tire fire for cyber security. That’s why Amazon’s statement about the audit rings true to me.
The pre-acquisition audit described four issues with a web application (not hardware or chips) that SuperMicro provides for management of their motherboards. All these findings were fully addressed before we acquired Elemental. The first two issues, which the auditor deemed as critical, related to a vulnerability in versions prior to 3.15 of this web application (our audit covered prior versions of Elemental appliances as well),
Two critical issues in the BMC web server (accessible over IPMI)
Two non critical ones (probably about encryption or lack thereof) that were mitigated by Amazon’s planned deployment
These findings ring true to me, this is what a typical infosec due diligence analysis is going to do — look at the interfaces and ports, see what functionality there is, what bugs there are, and what needs to be hardened/fixed.
firmware errors when reflashing the modchipped unit (checksums?)
unusual network traffic (e.g. beaconing) generated by the modchip
anything else weird and unusual that raises redflags
Supply chain attacks exist. Is this article accurate? It feels a little off, but I don’t know.
What do we know?
There’s not much we can speculate about the modchip because the Bloomberg description of whatever it does is gibberish. It is safer to simply examine what is known about Supermicro’s server boards.
Supermicro boards have third party BMC hardware to handle IPMI
There are at least three hardware providers: ASPEED, ATEN, and Nuvoton
ASPEED and Nuvoton use AMI software. ATEN has their own software stack
All Supermicro IPMI controllers appear to provide an extensive range of functionality that would be useful for an attacker
Keyboard Video Mou
se (KVM) over IP
SSH
Serial over LAN (SOL), and SSH over SOL
Web server (default login: ADMIN:ADMIN)
Remote power management

TCP 80, 443: web interface
TCP 3520, 5900: KVM access
TCP 623: menu access, allowing full control of the hardware
Good supply chain attack?
To compromise a server with a tiny modchip, a backdoor into the BMC would be pretty good. For example, a simple ICMP shell that beaconed out and provided basic commands to interact with the system would work in many places. The modchip’s backdoor would have to be more complex if the idea was to breach a hard target, but the BMC is certainly a good place to start.
So, what’s the deal?
For me, Bloomberg’s article could go either way. The logic of backdooring the BMC makes a lot of sense. Whether it happened in this case — given all the categorical denials — I have no idea.
The real takeaway from this is that IPMI is a raging tire fire, BMCs are Satan spawn, and never ever expose IPMI interfaces to the Internet. Unless you want hackers, because that’s how you get hackers.
Security
Infosec
Operational Security
China
Supply Chain
the grugq
Information Security Researcher :: keybase.io/grugq :: https://www.patreon.com/grugq
https://aws.amazon.com/blogs/security/setting-the-record-straight-on-bloomberg-businessweeks-erroneous-article/
Setting the Record Straight on Bloomberg BusinessWeek’s Erroneous Article
04 OCT 2018
Today, Bloomberg BusinessWeek published a story claiming that AWS was aware of modified hardware or malicious chips in SuperMicro motherboards in Elemental Media’s hardware at the time Amazon acquired Elemental in 2015, and that Amazon was aware of modified hardware or chips in AWS’s China Region.
As we shared with Bloomberg BusinessWeek multiple times over the last couple months, this is untrue. At no time, past or present, have we ever found any issues relating to modified hardware or malicious chips in SuperMicro motherboards in any Elemental or Amazon systems. Nor have we engaged in an investigation with the government.
There are so many inaccuracies in ‎this article as it relates to Amazon that they’re hard to count. We will name only a few of them here. First, when Amazon was considering acquiring Elemental, we did a lot of due diligence with our own security team, and also commissioned a single external security company to do a security assessment for us as well. That report did not identify any issues with modified chips or hardware. As is typical with most of these audits, it offered some recommended areas to remediate, and we fixed all critical issues before the acquisition closed. This was the sole external security report commissioned. Bloomberg has admittedly never seen our commissioned security report nor any other (and refused to share any details of any purported other report with us).
The article also claims that after learning of hardware modifications and malicious chips in Elemental servers, we conducted a network-wide audit of SuperMicro motherboards and discovered the malicious chips in a Beijing data center. This claim is similarly untrue. The first and most obvious reason is that we never found modified hardware or malicious chips in Elemental servers. Aside from that, we never found modified hardware or malicious chips in servers in any of our data centers. And, this notion that we sold off the hardware and datacenter in China to our partner Sinnet because we wanted to rid ourselves of SuperMicro servers is absurd. Sinnet had been running these data centers since we ‎launched in China, they owned these data centers from the start, and the hardware we “sold” to them was a transfer-of-assets agreement mandated by new China regulations for non-Chinese cloud providers to continue to operate in China.
Amazon employs stringent security standards across our supply chain – investigating all hardware and software prior to going into production and performing regular security audits internally and with our supply chain partners. We further strengthen our security posture by implementing our own hardware designs for critical components such as processors, servers, storage systems, and networking equipment.
Security will always be our top priority. AWS is trusted by many of the world’s most risk-sensitive organizations precisely because we have demonstrated this unwavering commitment to putting their security above all else. We are constantly vigilant about potential threats to our customers, and we take swift and decisive action to address them whenever they are identified.
– Steve Schmidt, Chief Information Security Officer
This message was sent to the list address and trashed, but can be found online.
-------------------------------------------
Archives: https://www.listbox.com/member/archive/247/=now
Modify Your Subscription: https://www.listbox.com/member/?member_id=26461375
Unsubscribe Now: https://www.listbox.com/unsubscribe/?member_id=26461375&id_secret=26461375-c2b8a462&post_id=20181005144249:736BACF2-C8CE-11E8-B388-FFE600B5174E
Powered by Listbox: https://www.listbox.com

Loading...