Self-hosted Grafana, EC2 and a secure initial password

Introduction

After a few years of running a single AWS account with all of my different projects in it (including this website), I recently completed the process of splitting out my projects into different AWS accounts under an AWS Organisation. I wanted a clear degree of separation between my different projects and a multi-account structure is a really good way to achieve this.

Whilst moving to a multi-account structure did allow a clear separation between my projects one thing that I did lose was the ability to monitor all of my projects easily; whereas previously I could view all of my projects within the AWS CloudWatch console I now found myself having to log into each AWS account to monitor the project(s) within that account.

I’d had some experience with Grafana at my job and quite liked it so that seemed like a good choice to investigate.

Research

I began researching how to set up Grafana on EC2 and came across several blog posts, including one from AWS themselves. Whilst this (and other blog posts) were great posts there was one common problem, they all used the default Grafana credentials (admin/admin) and relied on the person setting up the instance to quickly change the password. This made me uncomfortable, whilst there was only a small window where a bad actor could use those default credentials I wasn’t happy with their presence at all given there have been numerous (1 , 2 , 3 ) successful attacks against Grafana instances that would not have been possible if not for the use of default credentials.

Secrets Manager

To avoid the risk of such an attack I wanted the initial password to be unknown outside of the boundary of the AWS account that would host my Grafana instance. One way of achieving this would be to have something inside my AWS account generate a secure default password, luckily AWS Secrets Manager allows you to do just that.

Implementing secure default credentials

To start with I created a Secrets Manager Secret in CloudFormation which would contain the initial username and password for my Grafana instance:

InitialAdminUser:
  Type: AWS::SecretsManager::Secret
  UpdateReplacePolicy: Delete
  DeletionPolicy: Delete
  Properties:
    Name: grafana-initial-admin-user
    GenerateSecretString:
      SecretStringTemplate: '{"UserName": "admin"}'
      GenerateStringKey: "Password"
      PasswordLength: 32
      ExcludePunctuation: true
      IncludeSpace: false

The above resource will create a key/value secret with two keys; the UserName key has the value admin and the value of Password key is automatically generated by Secrets Manager according to the constraints in the CloudFormation (32 characters, punctuation and spaces excluded).

I then made use of Packer to fetch the secret at build time and include it in an Amazon Machine Image (AMI), ready to be launched on an EC2 instance.

Given a custom Grafana configuration file as below:

[security]
admin_user = ADMIN_USER_NAME
admin_password = ADMIN_PASSWORD

I replaced these template values at build time with a shell script that Packer runs as part of the build process:

apt-get install -y apt-transport-https
apt-get install -y software-properties-common wget
wget -q -O /usr/share/keyrings/grafana.key https://apt.grafana.com/gpg.key

echo "deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main" | tee -a /etc/apt/sources.list.d/grafana.list

apt-get update -y
apt-get install grafana -y
apt-get install jq unzip -y

curl "https://awscli.amazonaws.com/awscli-exe-linux-aarch64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
./aws/install

ADMIN_USER_NAME=$(aws secretsmanager get-secret-value \
  --secret-id "arn:aws:secretsmanager:eu-west-1:111111111111:secret:grafana-initial-admin-user" --region eu-west-1 \
  | jq -r '.SecretString | fromjson | .UserName')
ADMIN_PASSWORD=$(aws secretsmanager get-secret-value \
  --secret-id "arn:aws:secretsmanager:eu-west-1:111111111111:secret:grafana-initial-admin-user" --region eu-west-1 \
  | jq -r '.SecretString | fromjson | .Password')

sed -i "s/ADMIN_USER_NAME/$ADMIN_USER_NAME/g" /tmp/grafana/src/etc/grafana/grafana.ini
sed -i "s/ADMIN_PASSWORD/$ADMIN_PASSWORD/g" /tmp/grafana/src/etc/grafana/grafana.ini

mv /tmp/grafana/src/etc/grafana/grafana.ini /etc/grafana/grafana.ini

systemctl daemon-reload
systemctl enable grafana-server

As a result of this, I had an AMI that can be used to launch Grafana with default credentials that aren’t known to anyone outside of my AWS account.

Conclusion

I hope that this post serves as a useful thinking point for people setting up Grafana (or indeed any other off-the-shelf software that makes use of well-known default credentials) on EC2. Default credentials are often a recipe for disaster and if there is any way you can avoid them, you should.