A general manager at Salesforce subsidiary Heroku is still apologizing to a community of developers for a major data breach in early April where threat actors stole OAuth user tokens which led to more credential compromises and ultimately stealing source code.
Gerardo A Dada is CMO, Keeper Security.
The incident has far ranging implications as it looks like the hackers were able to pivot into private repositories. While it is unlikely to know the extent of the damage for a while, the attack is the latest in a series of high-profile incidents related to malicious actors’ theft of infrastructure secrets – the machine-to-machine credentials that give one system access to another one: such as a password for a database, SSH credentials, or an API certificate.
The incident is indicative of the increased trend in companies trying to target privileged accounts in various build and deployment systems, which makes sense, as these are the accounts with access to a company's crown jewels. This train wreck could have been prevented if the credentials had been handled as infrastructure secrets early on and stored safely in a secure vault instead of being laid open on the system.
Threat actors
The incident, which occurred starting on April 7, 2022, was described by Salesforce as “A threat actor obtained access to a Heroku database and downloaded stored customer GitHub integration OAuth tokens. Access to the environment was gained by leveraging a compromised token for a Heroku machine account. ..The attacker downloaded a subset of the Heroku private GitHub repositories from GitHub, containing some Heroku source code…the same compromised token was leveraged to gain access to a database and exfiltrate the hashed and salted passwords for customers’ user accounts“.
As a result, salesforce revoked all GitHub integration OAuth tokens, preventing customers from deploying apps from GitHub through the Heroku Dashboard or via automation. Then they had to reset all Heroku user passwords and (manually) rotated internal Heroku credentials.
This is a major incident with severe security, functionality and trust implications. And it all started with gaining access to one database via a compromised token.
Not isolated
The Heroku incident is not isolated. Quite the contrary, it is increasingly common for malicious actors to target infrastructure secrets: machine-to-machine credentials such as API keys, database passwords, and digital certificates. These often give malicious actors direct access to vast amounts of data and critical systems.
Unfortunately, most companies have not adopted a discipline of managing and protecting infrastructure secrets. In addition to being vulnerable to cybercriminals, this can be a big issue as there are no controls for contractors or employees leaving the company. If a DBA exits an organization, most organizations will not be certain about the credentials they have in their power, or even about making sure someone in the organization has all the credentials to systems and databases this person had.
Password managers
Just like password managers are designed to manage and secure human credentials, secrets management solutions protect and manage infrastructure secrets. First, secrets are stored in a vault, permissions are assigned using a console or API to specific users of applications, often locked to a specific IP address. Software applications no longer hard-code credentials, they get a token which is ten used to request the credential, which is then provisioned dynamically from the vault. The system can confirm the requester is authenticated and is in a specific IP address. By managing the credentials as secrets, they can be automatically rotated and changed periodically i.e. every month or maybe every hour.
These solutions often facilitate credential management and integrate with Ci/CD systems as well as programming languages. There are cloud-based as well as on-premises based secret management solutions. Developers and DBAs no longer need to see the credentials, they only need a token that is used to make the connection between the application and the secrets vault. Similarly, infrastructure secrets become invisible for malicious actors and almost impossible to steal.
First steps
A good first step is to make an inventory of all the workloads in the organization where infrastructure secrets may be used. Development teams are a good place to start, but there are also keys and credentials between a web server and a database, and between a marketing automation and a CRM system. The organization must decide if a cloud solution is more appropriate given its flexibility and time to value. Once a vendor is selected, implementation should be completed in priority given the complexity and risk of the inventoried systems.
The Heroku incident is indicative of the increase in cybercriminals targeting privileged infrastructure credentials in various build and deployment systems. Organizations should make it a priority to implement a secrets management solution to protect them from such attacks. It’s time to look at credentials as infrastructure secrets and manage them off the system.
