Several months back I was updating our vulnerability scanner checks for various Solarwinds products. As I added a detection script for a product called Log and Event Manager (LEM), I realized that there were really no significant publicly disclosed vulnerabilities for it. This spurred me to download their trial, which comes as a virtual appliance, and look for some flaws.
After initially setting up and configuring the Log and Event Manager appliance, I began exploring it’s functionality to determine where there might be potential issues. I immediately added some agents to various virtual machines in our lab and watched as they connected back to Log and Event Manager. Curious about what the agents were sending to the manager, I ran a packet capture of the traffic. As expected, the traffic was encrypted. While I won’t go into specific details on this blog for the sake of brevity, it *is* possible to decrypt and MITM the traffic between the agents and manager. It took a bit of work, but eventually I was successful and able to observe XML messages being passed back and forth.
The absolute first thing I think about when I see XML being used for data serialization is external entity injection(XXE). Platform implementations that use Java almost always have this vulnerability right out of the box because external entities are not disabled by default. Many developers who do not have a background in security generally forget to disable them, and as such, it makes for an easy target. Fortunately, this kind of vulnerability patching is relatively simple, and many companies can quickly find and eliminate it in their products during security testing. I know that Solarwinds certainly does test for this internally after a cursory look at some of their other products, however it’s possible it was simply overlooked in this case or might have had something to do with the fact that Log and Event Manager was acquired from TriGeo and re-branded as LEM. Regardless, the manager is vulnerable to an XXE. As I was attempting to understand where and how this was occurring, I had to look a little closer at the source.
When handling agent messages, the Log and Event Manager sets up a jaxb unmarshaller that it uses to deserialize the XML messages. This occurs in the LEMProtocolHandler.class within lem_communication.jar. When creating the unmarshaller, external entity resolution is never disabled. As a result, it is possible to abuse the XXE resolution to read arbitrary files on the local system and return them to the attacker. Unfortunately for an attacker, while errors are output to the manager.log file on the virtual appliance, no errors are ever returned in the response to the XXE request. This makes obtaining sensitive information slightly more difficult. The use of http/https protocol handlers was explored for data exfiltration, however the content of most files causes a MalformedURLException. It was, however, possible to abuse the ftp protocol in the XXE injection to exfiltrate data from the appliance.
XXE injections by themselves are somewhat uninteresting. You need a file that has some good information in it that can help you gain access to the affected target. From the attackers point of view, thankfully the LEM appliance also utilizes an HSQL server for storing alert data. HSQL listens publicly on port 9001 of the appliance. Apparently, at one point this server was configured with a static easily guessable password (CVE-2014-5504). To resolve this issue, it seems that the passwords for the ‘sa’ and ‘trigeo’ users are now dynamically generated on the fly when the virtual appliance is set up for the first time. Unfortunately, these credentials are left in plaintext in the passwords.data file, instead of being hashed. That makes the passwords.data file a great target for exfiltration via the XXE injection.
The nice thing about HSQL is that it allows the creation of functions and procedures that can call public static methods of any class that is included in the classpath of the java application. In order to achieve remote command execution, it was necessary to find a jar file in the classpath that had a public static method that would call Runtime.getRuntime().exec. After examining a number of jar files, I found a suitable method called openBrowser within the h2-1.3.172.jar file. This method was originally designed to use a browser to test other features, however it is possible to change its intended functionality and force it to run any supplied command. To abuse this, I used the exfiltrated ‘sa’ user’s password to login to the HSQL server and generate a function to set a property called h2.browser. This function was passed a parameter, which was set to the command I wanted to run. Next, I created an HSQL procedure to call the openBrowser method while passing in a string that will get supplied to the command stored in h2.browser as an argument. Finally, utilizing another feature of HSQL known as text tables, I added a bash script for a reverse shell to a new table, and then copied that table to an HSQL text table with it's source specified as a local file. This causes the copied data to be written to a file in /var/alertdata/hsql. After all of this was set up, I passed the command to execute as ‘bash’ and supplied the path to this newly created file in order to obtain a remote shell on the virtual appliance.
Once a remote shell is obtained, the user that the attacker will be running under will be 'trigeo'. This is a limited account, but there is enough access to take full control of the appliance. The 'trigeo' user can write to most files in /usr/local/contego where the core of the application exists. Using a LEM-specific hashing algorithm, a new admin user can be added to /usr/local/contego/run/manager/UserContextLibrary.xml and it will now be possible to log in to the web console and control any connected endpoint agent. If root access is required, it is possible to modify the /usr/local/contego/scripts/mountsolr.pl file to execute arbitrary system commands instead of it's intended functionality. This script is called every 15 minutes through a cron job that runs as the root user.
So there you have it. Digital Defense, Inc. customers were notified in advanced and had early access to vulnerability checks for this flaw prior to public disclosure. Furthermore, all details in this blog post have been reported to Solarwinds and they have already patched the vulnerabilities. Credit goes to them for working quickly to resolve the issues and helping to keep their product secure for their customers. Since a fix has already been developed and deployed, for further technical details please see the proof of concept code posted to exploit-db.