Security is increasingly becoming the responsibility of DevOps teams. Movements like DevSecOps are contributing to this change. This security checklist aims to give DevOps professionals a list of security best practices they can follow according to their company stage. Doing the basics goes a long way in keeping your company and product secure. This third1 edition of the DevOps Security Checklist provides actionable security best practices devops teams can use to harden their security. This list is far from exhaustive, incomplete by nature since the security you need depends on your company, product, and assets.
🚀 Your Culture
Have an incident response plan and mechanism to communicate post-mortems
It is not a question of “if” but “when”. Evaluate your risks, prepare a proper action plan in case of a breach and communicate properly after the fact.
https://blog.serverdensity.com/how-to-write-a-postmortem/
https://codeascraft.com/2012/05/22/blameless-postmortems/
https://blog.sqreen.io/cybersecurity-risk-assessment-for-startup-cto/
Stay on top of best practices
DevOps is an ever-changing landscape. Ensure that you stay up to date in terms of new technologies, vulnerabilities or best practices.
https://aws.amazon.com/whitepapers/architecting-for-the-aws-cloud-best-practices/
https://docs.microsoft.com/en-us/azure/security/azure-security-network-security-best-practices
Understand the risk
The cost of breaches is drastically increasing and security should be taken seriously inside an organization. DevOps engineers should play an important role in advocating for better security practices
http://fortune.com/2016/06/15/data-breach-cost-study-ibm/
https://www.troyhunt.com/the-emergence-of-historical-mega-breaches/
Gamify security and train employees on a regular basis
Humans are the weakest links in the security chain. DevOps contribute to the security awareness of all the employees in a company. By explaining how an attacker could infiltrate your company, you will increase the awareness and thus minimize the chance of a hack. Don’t forget fishing and spear-fishing attacks.
https://www.schneier.com/blog/archives/2008/03/the_security_mi_1.html
Follow an onboarding / offboarding checklist
This checklist should contain a list of all the steps you need to enforce when an employee, contractor, intern, etc… joins your company. A similar list can also be used when the someone is leaving your team.
💻 Your Code
Don't implement your own crypto
The problem with cryptography is, that you don’t know you are wrong until you are hacked. So don’t do your own crypto. Use standards instead.
http://crypto.stackexchange.com/questions/43272/why-is-writing-your-own-encryptiondiscouraged
Protect your CI/CD tools like your product
Your continuous deployment pipeline is the backbone of your IT. Security should be checked at each step. Your CI builds should fail if you detect a security vulnerability. Store your CI configuration for traceability and audit.
https://wiki.jenkins-ci.org/display/JENKINS/JobConfigHistory+Plugin
https://www.slideshare.net/kponiatowski/if-cicd-teams-have-time-for-security-so-do-you
Ensure you are using security headers
Modern browsers support a set of headers dedicated to block certain types of attacks. Make sure you properly implemented all security headers. Don’t forget about the CSP.
Keep your dependencies up to date
Third-party libraries can put your application at risk. Make sure you track your vulnerable packages and update them regularly.
Run Security tests on your code
Static Application Security Testing (SAST) is an easy and fast way to find security vulnerabilities in your code. You can enforce SAST security checks in your CI, but be aware of the high number of false positives that can frustrate developers.
https://www.owasp.org/index.php/Source_Code_Analysis_Tools
Integrate security scanners in your CI pipeline
Integrate a Dynamic Application Security Testing (DAST) tool in your CI, but just like SAST be aware of the high number of false positives.
http://www.arachni-scanner.com/
https://www.owasp.org/index.php/OWASP_Zed_Attack_Proxy_Project
Go hack yourself
If your company doesn’t have yet a structured security team, help create a multidisciplinary Red Team to stress your application and infrastructure. Providing an easy environment for the Red Team to attack the application should be part of the scope of DevOps.
http://www.devsecops.org/blog/2015/12/10/red-team-pwning-the-hearts-and-minds-one-ticket-at-a-time
🏗 Your Infrastructure
Backup, test your backups, then backup again
Backup all your critical assets. Ensure that you attempt to restore your backups frequently so you can guarantee that they’re working as intended. S3 is a very cheap and effective way to backup your assets. Instrument monitoring to ensure backups and restoration verification are working as intended. Research whether offsite and encrypted backups make sense for your product, infrastructure, and regulatory concerns (PII and user data retention).
MongoDB Backup: https://docs.mongodb.com/manual/core/backups/
Postgresql: https://www.postgresql.org/docs/current/static/backup.html
Linux: http://www.tecmint.com/linux-system-backup-tools/
https://www.dataone.org/best-practices/ensure-integrity-and-accessibility-whenmaking-backups-data
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithAutomatedBackups.html
https://aws.amazon.com/getting-started/backup-files-to-amazon-s3/
Check your SSL / TLS configurations
Use free tools to scan your infrastructure regularly and make sure the SSL configurations are correct.
https://observatory.mozilla.org/
https://diogomonica.com/2015/12/29/from-double-f-to-double-a/
Control access on your cloud providers
The best way to protect your services (database, file storage) is to not use passwords at all. Use the built-in Identity and Access Management (IAM) functions to securely control access to your ressources.
http://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html
https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances
Log all the things
Infrastructure logs and application logs are your most precious allies for investigating a data breach. Make sure your logs are stored somewhere safe and central. Also make sure you whitelist- or blacklist-specific incoming data to avoid storing personally identifiable information (PII) data.
https://qbox.io/blog/welcome-to-the-elk-stack-elasticsearch-logstash-kibana
Upgrade your servers regularly
Server packages and libraries are often updated when security vulnerabilities are found. You should update them as soon as a security vulnerability is found.
Automatically configure & update your servers
An automated configuration management tool helps you ensure that your servers are updated and secured.
Chef: https://learn.chef.io/tutorials/
Ansible: http://docs.ansible.com/ansible/intro_getting_started.html
Salt: https://docs.saltstack.com/en/latest/topics/tutorials/walkthrough.html
Keep your containers protected
Use Docker (or Kubernetes), and ensure that they are patched and secure. Use tools to automatically update and scan your containers for security vulnerabilities.
https://www.docker.com/docker-security
https://docs.docker.com/docker-cloud/builds/image-scan/
https://jpetazzo.github.io/2015/05/27/docker-images-vulnerabilities/
https://www.slideshare.net/MichaelCherny/security-best-practices-for-kubernetes-deployment
Store encrypted passwords in your configuration management
Storing passwords (like databases ones) can be done on a dedicated database with restricted access. An other solution is to store them encrypted in your Source Code Management (SCM) system. That way, you just need the master key to decrypt them.
Chef: https://github.com/chef/chef-vault
Puppet: https://puppet.com/blog/encrypt-your-data-using-hiera-eyaml
Salt: https://docs.saltstack.com/en/latest/ref/renderers/all/salt.renderers.gpg.html
Ansible: http://docs.ansible.com/ansible/playbooks_vault.html
Encrypt all the things
SSL performance problems are a myth and you don’t have any good reasons not to use SSL on all your public services.
Harden SSH configurations
SSH is the defacto remote login mechanism on Linux environments. It’s also the de facto penetration vector for hackers. Make sure you have proper SSH configurations.
https://devops.profitbricks.com/tutorials/secure-the-ssh-server-on-ubuntu/
https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys–2
Manage secrets with dedicated tools and vaults
When you need to store cryptographic secrets (other than database password, TLS certificate, …) and perform encryption with them, you should use dedicated tools. This way the cryptographic secret never leaves the tool and you get auditing features.
https://github.com/square/keywhiz
Use an immutable infrastructure
Use immutable infrastructures to avoid having to manage and update your servers.
https://martinfowler.com/bliki/ImmutableServer.html
https://hackernoon.com/configuration-management-is-an-antipattern-e677e34be64c#.n68b1i3eo
🛡️ Protection
Enforce Two-factor authentication (2FA)
Enforce 2FA on all the services used (whenever possible).
https://landing.google.com/advancedprotection/
https://support.google.com/a/answer/184711
https://get.slack.help/hc/en-us/articles/212221668-Require-two-factor-authentication-for-your-team
Don’t store credit card information (if you don’t need to)
Use third-party services to store credit card information to avoid having to manage and protect them.
https://www.braintreepayments.com
https://www.pcisecuritystandards.org/pdfs/pciscc_ten_common_myths.pdf
https://medium.com/@folsen/accepting-payments-is-getting-harder-1b2f342e4ea#.897akko4q
Have a public security policy
This is a page on your corporate website describing how you plan to respond to external security reports. You should advise you support responsible disclosure. Keep in mind that most of the reports that you receive probably won’t be relevant.
https://www.intercom.com/security
Protect your applications against breaches
Detect and block attacks in real-time using a protection solution. All the OWASP top-10 vulnerabilities (SQL injections, NoSQL injections, cross-site scripting attacks, code/command injections, etc.) are covered.
Protect your servers and infrastructure
Your servers will be scanned in order to fingerprint your application and locate open services, misconfiguration, etc. You can setup tools to keep these scanners away from your servers.
https://www.digitalocean.com/community/tutorials/how-to-protect-ssh-with-fail2ban-on-ubuntu-14-04
Protect your users against account takeovers
Account takeovers or brute force attacks are easy to setup. You should make sure your users are protected against account takeovers.
https://www.owasp.org/index.php/Blocking_Brute_Force_Attacks
Have a public bug bounty program
A bug bounty program will allow external hackers to report vulnerabilities. Most of the bug bounties program set rewards in place. You need security aware people inside your development teams to evaluate any reports you receive.
Protect against Denial Of Service (DoS)
DoS attacks are meant to break your application and make it unavailable to your customers. Use a specific service to protect your app against Distributed Denial Of Service attacks.
https://www.cloudflare.com/ddos/
https://www.ovh.com/us/news/articles/a1171.protection-anti-ddos-service-standard
Ensure Compliance with Relevant Industry Standards
Comply to standards to ensure you follow industry best practices and answer your customer needs. But simple compliance will never protect your apps.
https://cloudsecurityalliance.org/
https://en.wikipedia.org/wiki/ISO/IEC_27001:2013
https://en.wikipedia.org/wiki/Payment_Card_Industry_Data_Security_Standard
🩺 Monitoring
Audit your infrastructure on a regular basis
With cloud providers, it’s easy to start instances and forget about them. You will need to create and maintain a list of your assets (servers, network devices, services exposed etc…), and review it regularly to determine if you still need them, keep them up to date, and ensure that they benefit from your latest deployments.
http://docs.aws.amazon.com/general/latest/gr/aws-security-audit-guide.html
http://searchenterpriselinux.techtarget.com/tip/Creating-an-inventory-with-nmapnetwork-scanning
Renew your certificates on time
You should be using TLS certificates. It can be a hassle to configure and monitor but don’t forget to renew them!
https://serverlesscode.com/post/ssl-expiration-alerts-with-lambda/
Detect insider threats
The most important attacks will come from insider threats. Those can be users or attackers gaining access to privileged user accounts. Make sure you monitor your users to detect attackers before an attack happens.
Monitor third party vendors
You’re likely to use third party products to manage your servers / payrolls / logs or even just social media. Just like you’re likely to be hacked, they can be too. Make sure you follow the news and react immediately after a breach.
Monitor your domain expiration date
Just like TLS certificates, domains can expire. Make sure you monitor your domain expiration automatically.
Get notified when your app is under attack
You will be attacked. Make sure you have a monitoring system in place that will detect security events targeting your application before it’s too late. Knowing when your application is starting to get massively scanned is key to stop more advanced attacks.
https://www.linode.com/docs/security/using-fail2ban-for-security#email-alerts
Monitor your authorizations
Be proactive and be alerted when authorizations or keys binary are changed in your production.
http://techblog.netflix.com/2017/03/netflix-security-monkey-on-google-cloud.html
👉 Others in the checklist series
Further Reading
- When should my startup prioritize infosec? - How do I know when I should start paying down security tech debt?
- Building your DevSecOps pipeline: 5 essential activities
1 CCA ShareAlike 4.0 International. This guide is based on, adapts, and picks up the baton where the excellent Sqreen checklist left off (which is no longer available since it redirects to their acquisition announcement).
