ERP integration–protecting API keys & cloud credentials

Securely manage your API keys

With a greater number of API’s in the enterprise that are dealing with increasingly sensitive business processes or data, it is now more important than ever to actively manage API keys and credentials in a secure way. This is of interest to consultants, developers and system administrators.

Service based API’s are ubiquitous in enterprise ERP environments, some examples include:

• Postal code lookup
• Carrier integrations
• Product review services
• Customer surveying
• Credit card processing
• Credit checking
• 3PL fulfilment services
• Product feeds
• Paypal/BitCoin banking
• Mailing list managers

Why worry?

These “keys” for on line web services are usually in the form of certificate files or plain text strings. They secret keys are then used by the software to access the service, so they are included in code or associated configuration files for that integration or add-in solution. The issue is securing those keys, ensuring they don’t get into the wrong hands as often misuse can incur metered charges or loss of corporate data. Worryingly those keys may get checked into source control databases by developers where they are accessible to a wide audience of testers, other developers and perhaps many more people than intended as the source control repository is not a tightly controlled environment. On the other hand it may be that the secret data is openly accessible by system administrators or contractors logged in to your system with enough permission to browse the application folders. It is difficult to control this sort of access for console or service applications. IIS does better, it has always protected files ending in .config, preventing them being served, this does not help with direct access to file systems from inside though. Even with the config file protection, it is so easy to make the mistake of creating a custom configuration file, without thinking of the consequences that uses another file suffix that does not get that protection. How often have you done something like backing up a configuration file to a .bak or .configbak, thus stripping the protection that IIS would otherwise have provided. Not only that such a name easy to guess by the bad guys!

Some organisations have poor/lazy management of security roles. It may be many more people than expected could potentially access program files directories on servers (think terminal servers where a service might be innocently deployed) or on developer machine shares put there to give easy access for other developers to each other’s machines. Consider this, if the user or admin can view the information, spyware or malware, executing under that same users’ context will also be able to access and harvest that information, thus potentially punching it out the organisation for misuse. You may consider it far fetched, but if money is to be made, someone will exploit you sooner or later. Make a risk assessment on the impact and decide how much effort to put into doing things better. Perhaps the keys are the customers that you are using for development, these are keys that you don’t want to be responsible for leaking to the internet, or things could get very uncomfortable! .Another issue may be that the keys are your own for a service you pay for that you are using as part of a solution that you have deployed for a customer. In this case you don’t want the customer to see your private keys in the configuration, but yet need a way to easily update them, with the customer applying the updates. If web based updates are not possible, easy to update configuration files have the least friction for the enterprise IT people to work with. It is also no use to compile secrets into the application code. If you think that hides the information, you’d generally be wrong, unless they are obfuscated (which is still not secure), by simply opening the exe in a text editor, it may reveal the text resources containing those secrets. Obviously there different levels of effort to exert in protecting information, perhaps it is only casual dissemination of keys rather than determined hackers that is causing concern, in which case simple scrambling of the secret info is enough to prevent casual theft or misuse. Decisions have to be made in light of your risk assessments.

Be aware also if using source control repository for source control, especially if it is outside the organisation, then a breach of that repository security could expose your secrets to the world. Even public repositories can be hacked and compromised. Certainly public repositories are targeted by key harvesting bots looking for valuable keys accidentally checked into the repository. Reportedly an engineer checked Amazon AWS keys into GitHub for duration of only five mins before removing them. He faced a $2375 bill from the subsequent abuse of his Amazon AWS account due to a bot getting hold of those keys and using them for bit coin mining[Why Exposed API Keys and Sensitive Data are Growing Cause for Concern] . Imagine if that was keys to something more sensitive like credit card APIs!

Discovery and maintenance issue

The problem with configuration files is that they are too easy to work with compared to alternatives making it difficult to wean the industry off them. For years IT people have been conditioned to all know where to look for them and how to edit them. They are often intuitive and self describing, hence system admins don’t need to know much about the application to maintain its settings. The names of these files are familiar to us all, plain text configuration files with names such as “.config” “.ini” “.settings” or similar. This familiarity is great for a software vendor to take advantage of the user being able to self discover and maintain the configuration themselves with a simple text editor and with little or no input from anyone that in turn keeps support calls short or non-existent that leads to lower running costs for everyone.

Using certificates for external API authentication was in fashion once, however most people hate working with certificates and maintaining certificate stores, it ends up as a support cost to the software vendor. For example there needs to be a website or application to generate the certificates and applying them to the system can be arduous so extra support material needs writing and inevitably extra support cases are raised.

More recently it is common to encounter the the OAuth style authentication or shared secret type techniques for securing API’s. These use simple text strings that must be stored by the application but they are too easily exposed to carelessness due to their simple text file nature. For example, tempting as it may be, should you leave the email with the keys in your in box, should malware get access to your mail, or your mail provider be hacked, stealing all mails from well known providers is all to easy.

So how to secure the secrets?

The answer is to apply common sense, but it needs effort to implement and maintain good practice. API keys should ideally be kept out of the source controlled folder hierarchy and NEVER checked in to the repository. The secrets should be held in separate configuration files, ideally out side the source tree and source code repository. Ideally the values in the configuration files should be encrypted. I prefer that the values are encrypted rather than the entire file, as this provides the person maintaining the file contents with an idea about the architecture of the solution by them seeing the various settings that are available/exist which allows them to ask the right questions when things go wrong.

Encrypting the values means providing some form of user interface for the user to edit and generate those values. It is hard to write good encryption, so do instead use a known encryption library or product, beware hackers are very clever people with good toolkits. Encryption means swapping one problem for another as you then have to mange the encryption keys for the encryption (unless you use a generic key store/API)! All this is extra work, especially if the application is a windows service or console application, as adding user interfaces to manage keys in a pain. If you choose to avoid a key management GUI then asking users to use command line switches to update configuration settings, although possible, is very Unix like and a very difficult experience than editing a text file in Notepad++. Although there are solutions like key vault in Azure, this is not helpful to console applications on local machines but well worth embracing if you use Azure as a platform.

Encrypting the settings can also present new issues, such as the encryption output being tied to one machine, making the application less portable, although not entirely a bad thing as prevents the whole application being lifted and used on another unauthorised machine. However it leads to user education that it can’t be simply copy deployed. Have look at Windows Data Protection API (DPAPI) to encrypt and decrypt data. Once of the ASP.NET config providers uses this.

Keeping separate configuration files allows them to be substituted using using whatever ALM build/deploy system is used, thus Dev, Test and Production can all have different API keys that only exist in the respective environment, especially useful where the service provider offers testing environments as well.

Other suggestions for handling keys are:

Source control ignore

Having the configuration folder or files marked as excluded or .gitignore so they don’t go into source control. This is is prone to error with a new developer not realising what they are doing and letting them get un-ignored, or simply making mistakes and allowing the file to get into source control (remember my renamed file example earlier). It shouldn’t happen, but we all know in our hearts it will. Small teams grow into big development teams, what once worked falls down at that point.

Store in environment

Common technique is to store the sensitive data in environment variables. This is flawed according to this post. Environment Variables Considered Harmful for Your Secrets to quote from the post:

1. The environment is implicitly available to the process. It's hard to track access and how its content gets exposed. It's easy to grab the whole environment and print it out (can be useful for debugging) or send it as part of an error report for instance.
2. The whole environment is passed down to child processes (if not explicitly filtered) which violates the principle of least privilege. So your secret keys are implicitly made available to any 3rd-party tools you're using (like ImageMagick to resize an image). It's hard to say what those 3rd-party tools will do with the environment, especially in rare occasions like crashes.
3. External developers are not necessarily aware that your environment contains secret keys. By doing so your requirements (tightly controlled access) no longer match with most developers implicit expectation (nothing special in the environment, just generic system configuration). This is a dangerous mismatch.

It does point out that DevOps tools like chef can step in to help manage the files on the operations side of things instead.

Summary

One thing is for certain, there is not one single universal approach to take, there are many different techniques and tools available to developers and DevOps teams that may help, but take action and consider the points I have raised next time you type those keys into a customer or your own server configuration file! Do a risk assessment, at the very least inside your head, fit the level of solution to the circumstances of a particular requirement. I would say it is an important to research the limitations and pit falls of any chosen solution so everyone is aware of any risks it may bring.

Secrets with Azure and ASP.NET 5

Scott Hanselman blogs about ASP.NET 5 and Azure that introduce facilities for dealing with the problem, making it easier to manage secrets in an application. This does not help us today with windows services or console applications, but gives hope for the future. Best practices for private config data and connection strings in configuration in ASP.NET and Azure

Credits

Thanks to Steve Endow & Chad Bruels for reigniting this discussion a couple of times prompting me to post this.