Infrastructure as Code and Secrets Management – Lesson 4

1. Start the ssh-agent to store our keys and store the created for this lab

• * The ssh-agent is a daemon that runs your provisioning server. It allows you to register
• a private ssh key, and it will manage providing those keys to additional servers as you connect
• to them. In this case, we are about to connect to our bastion host, and we don’t want to mess
• around with running ssh with the “-i” flag all the time.

• * An interesting fact is that when ssh-agent starts, it outputs environmental variables so that
• your shell can find the agent. running the command inside of eval $() sets the variable for us.
• Cool huh?

• eval $(ssh-agent)
• ssh-add ~/iac_lab/terraform/ssh/id_rsa
•  

1. Build an file with environmental variables set for the bastion host to make life easier when we get there

• * This is a script that pulls all of the outputs defined in terraform outputs.tf file and
• drops them into a file called bastion.profile. It then copies that file up to the bastion
• host so that it will be there for us when we log in.

• * We’re doing this because setting up vault requires a bunch of commands that each need to know
• something about our infrastructure. This is the easiest way to make the lab work for everyone.
• In reality, you wouldn’t store things like your database password in a file on the provisioning
• host or the bastion host

• cd ~/iac_lab/vault/
• ./create_bastion.profile.sh
• . ./bastion.profile

1. Connect to the bastion host

• * See, look ma, no “-i”. Thanks to the ssh agent. We are using the “-A” flag that forwards our
• private key to the next host. This makes it super easy to connect to other hosts in the app
• environment from the bastion host

• ssh -A ec2-user@$BASTION_IP

1. Set the shell prompt

• * Set the shell prompt to identify this host so that we can stay oriented

• echo ‘export PS1=”[\u@\h (BASTION) \W]\$ “‘ >> ~/.bashrc
• . ~/.bashrc

1. Set the env variables

• * Ok, now we are working on the bastion host. It is highly advised that you write down the
• last octet of the host name so that you can remember where you are working. It is really
• easy to get lost when you are ssh hopping from host to host in the cloud. 🙂

• * We copied the bastion.profile file up to the bastion host, now lets source it so that we
• have all the env VARs set. This is critical for the next 10-15 steps

• . ~/bastion.profile

• * Check that it worked. Don’t proceed unless you see some env variables.

• env | grep MYSQL

• * EXAMPLE OUTPUT (to confirm that the environment variables are set)
• > MYSQL_HOST=myprojectdev.amazon.bla.blah.com<br>

1. Install the vault client on the bastion host

• * We are using the stock Amazon linux image for our bastion host. Need to install vault binary

• wget https://releases.hashicorp.com/vault/0.11.3/vault_0.11.3_linux_amd64.zip
• sudo unzip -j vault_*_linux_amd64.zip -d /usr/local/bin

1. Set the VAULT_ADDR env variable and check connectivity

• * Sure we could have done this in the bastion.profile file. But doing this by steps
• lets you see that the `vault` command requires the VAULT_HOST env variable to be set

• export VAULT_ADDR=http://${VAULT_IP}:8200
• vault status

• * EXAMPLE OUTPUT (for uninitialized Vault)

• * Specifically it **should say**
• “server is not yet initialized”. This because we have a brand new vault server.

1. Initialize the vault

• * The keys generated by the vault unseal command are only created once, upon the init command,
• and are **very sensitive**. In a real environment, each Unseal Key should be stored
• by separate people in the organization, and NEVER stored anywhere near the vault
• server, or those that access it. For this lab, we’ll just leave it in ./vaultkeys.txt.
• In real life, get that file off the bastion host as soon as the vault is initialized.

• vault operator init 2>&1 | egrep ‘^Unseal Key|Initial Root Token’ | tee ./vaultkeys.txt
• chmod 600 ./vaultkeys.txt
•  
• * EXAMPLE OUTPUT (for vault initialization)

1. Unseal the vault

• * At this stage, the vault is initialized, but not unsealed. We use the “Unseal Keys” provide

• * The following commands send 3 of the Unseal Keys to the `vault unseal` command.

• egrep -m3 ‘^Unseal Key’ vaultkeys.txt | cut -f2- -d: | tr -d ‘ ‘ | while read key; do vault operator unseal ${key}; echo; done

• * EXAMPLE OUTPUT (for vault unsealing)

• * Verify that the vault is unsealed

• vault status

• * EXAMPLE OUTPUT (for vault status)

1. Set the vault root user token (aka password) as an environmental variable

• * Vault has several ways that users can authenticate to it. But the root user always
• authenticates with the token that was provided at init time. To allow our bastion host
• to access the vault server, set the VAULT_TOKEN env variable from the vaultkeys.txt file we saved
• at init

• export VAULT_TOKEN=`egrep ‘^Initial Root Token:’ ./vaultkeys.txt | awk -F’:’ ‘{print$2}’ | sed s/’ ‘//g`

1. Write out first secret to the vault database

• * In this step, we are going to store information for the application servers about
• where to find the database. The secret/ storage path is simple key/value pairs and can
• contain almost anything. Here, we’ll store some data in a path called secure/mysql. (mysql could be any name)

• vault write secret/mysql host=${MYSQL_HOST} port=${MYSQL_PORT} database=${MYSQL_DB}

• * EXAMPLE OUTPUT

• * Check that we can read the secret back. This is similar to the query that our applications will
• run when they want to know where to find the database

• vault read secret/mysql
•  
• * EXAMPLE OUTPUT

• * Awesome, we just stored simple key/value pairs in the path secret/mysql. Now lets set up MySQL passwords

1. Lets configure MySQL secrets back end that will issue dynamic passwords to our application

• * First, install the mysql client so that we can connect to the database and create a user for vault

• sudo yum install -y mysql

• * EXAMPLE OUTPUT

• * Now we can create a mysql user & pass for vault service that has ALL privs (including GRANT) on our db.
• For this lab, we are using the root password configured when we ran `terraform apply` as the vault password.
• In the real world, these should be different passwords.

• MYSQL=”mysql -h $MYSQL_HOST -u $MYSQL_USER -p$MYSQL_PASS -Bsse”
• $MYSQL “CREATE USER ${MYSQL_VAULT_USER}@’%’ IDENTIFIED BY ‘${MYSQL_VAULT_PASS}’;”
• $MYSQL “GRANT ALL ON ${MYSQL_DB}.* TO ${MYSQL_VAULT_USER}@’%’ WITH GRANT OPTION;”
• $MYSQL “GRANT CREATE USER ON *.* TO ${MYSQL_VAULT_USER}@’%’;”
• $MYSQL “FLUSH PRIVILEGES;”

• * *Note*: if you happen to screw up the copy/paste above and want to delete the user and start over, run

1. Configure the vault mysql secrets database

• * Vault is configurable with a variety of storage, authentication and secrets databases
• Each has to be “mounted” in order to use them. Let’s mount the “mysql” secrets database below
• so that we can have it create usernames & passwords on the aurora database for us.

• vault secrets enable mysql
•  
• * EXAMPLE OUTPUT

• * We have to tell the vault mysql module how to find and log into our mysql database, so that it can create
• the users and passwords.

• vault write mysql/config/connection \
• connection_url=”vault:${MYSQL_VAULT_PASS}@tcp(${MYSQL_HOST}:${MYSQL_PORT})/” \
• allowed_roles=”readwrite”

• * EXAMPLE OUTPUT

• * And when vault’s mysql module creates a user for our web servers, we want the connection to have
• a short lease (1 hour) in case the user/password is ever comprimised.

• vault write mysql/config/lease lease=1h lease_max=12h

• * EXAMPLE OUTPUT

• * Finally, we want to create role called “readwrite” that grants our users ALL privs on our database

• vault write mysql/roles/readwrite \
• sql=”CREATE USER ‘{{name}}’@’%’ IDENTIFIED BY ‘{{password}}’;GRANT ALL ON ${MYSQL_DB}.* TO ‘{{name}}’@’%’;”

• * EXAMPLE OUTPUT

1. Enough already, lets test it out as root

• * use the vault CLI and root user to read some mysql credentials
•  
• vault read mysql/creds/readwrite

• * EXAMPLE OUTPUT

• * Bingo! Vault created a temporary user for us with a 1 hour lease. If you feel froggy, prove that
• you can log into mysql with that user. It would look something like
• `mysql -u read-root-a9f162 -p’1cd3fa9f-0cd8-cb68-e415-b0a8e80a5533 -h myproject-dev-rds-cluster-20180108021323654300000002.cluster-ctw6wkbqhjcx.us-east-2.rds.amazonaws.com myprojectdevdb`

1. Create a policy for web access to secrets

• * Now we will create & write a policy call web-policy that allows a user to access the secrets we just configured

• cat <<EOF > ./web-policy.hcl
• path “sys/*” {
• policy = “deny”
• }
• path “secret/mysql*” {
• capabilities = [“read”]
• }
• path “mysql/creds/readwrite” {
• capabilities = [“read”]
• }
• EOF
• vault policy-write web-policy ./web-policy.hcl

1. Enable the AWS EC2 Auth module

• * Enable Vault auth module so that we can configure vault to authenticate and ec2 instance

• vault auth enable aws-ec2

• * Configure aws-ec2 auth module allow the web-role to authenticate if they coming from a legit
• EC2 instance, and have our IAM policy (myproject-dev-webinst-role) attached. Assign them to the
• policy web-policy (that allowed access to our secrets)

• vault write auth/aws-ec2/role/web-role bound_iam_instance_profile_arn=${WEB_PROFILE_ARN} policies=web-policy

1. Test a web server

• * Go to ALB URL and check the vault and database tabs

• cd ~/iac_lab/terraform
• terraform output LOADBALANCER_DNS

• * Or, from the AWS console naviget to the following: AWS Console > EC2 > Load Balancers > select your ALB > Public DNS

• * Wait, a minute, they still cant authenticate… The reason is that the ultra simple way we

• * To fix this, just terminate the *-webinst EC2 severs and let the autoscaling group launch

•  
• aws ec2 describe-instances –region=us-east-2 –filters “Name=tag:Name,Values=’iaclab-dev-webinst'” –query Reservations[*].Instances[*].[InstanceId] –output text | xargs -i aws ec2 terminate-instances –region=us-east-2 –instance-ids {}

1. Take a screen shot!!!

• * **WAIT!** If you are completing this lab as part of a Cloud Shift

1. Started the ssh-agent on the provisioning server and added the ssh keys to connect to bastion host
2. Pulled output variables from terraform and put into a profile
3. Connected to the bastion host
4. Set up temporary environmental variables on bastion host
5. Installed the vault client on the bastion host
6. Configured the $VAULT_ADDR enviornmental variable on the bastion host and tested access to vault
7. Initialized the vault
8. Unsealed the vault and checked the status of vault unseal
9. Set the VAULT_TOKEN variable so that we can gain root access to the vault
10. Wrote a simple secret to secret/mysql with database host name & port number
11. Installed mysql client & created a vault user that can create more users in Aurora / MySQL
12. Configured the vault mysql secrets database to create users for us
13. Tested the temporary credentials from root user
14. Created a policy for the web servers to access these secretes
15. Enabled the aws-ec2 auth modules that allow our web servers to authenticate w/ vault
16. Tested that the web servers can access the secrets and the database
17. Captured a screen shot for a chance to win $50