Run a HOP application on AWS
In this tutorial you will bootstrap a basic HOP project with the following features:
Duct Framework based backend.
Reactive ClojureScript front-end.
Infrastructure provisioning in the Amazon Web Services deployment target.
CI/CD integration with GitHub Actions.
The HOP Command Line Interface tool (HOP CLI, from here on) provides more bootstrapping features, but this tutorial will focus in a bare minimum installation.
Having the HOP CLI installed, as explained in the Installation tutorial, is enough for bootstrapping a new HOP project. But in order to run the generated project locally on your machine, you will need some additional tools:
Apart from the software requirements, you will also need an Amazon Web Services account with administrator credentials, in order to create the infrastructure needed.
As of this writing, HOP projects require Docker version 18.09 or later. You can check if you have Docker already installed, and that it is a supported version, by running the following command in a terminal:
$ docker --version Docker version 18.09.1, build 4c52b90
If you do not have Docker installed, or if it is an older version than the ones supported, you can install a supported version on your operating system of choice, by following the instructions in the Docker installation documentation.
Once you have installed (or upgraded) Docker, you can check that it is correctly installed by running the following command:
$ docker run --rm hello-world Hello from Docker! This message shows that your installation appears to be working correctly. [... more text appears here ...] For more examples and ideas, visit: https://docs.docker.com/get-started/
If you get an error about not having permission to connect to the Docker daemon socket, check the Docker post-installation steps documentation:
You can also check that you have a supported version by re-running the command you used before:
$ docker --version Docker version 18.09.1, build 4c52b90
Install Docker Compose
As of this writing, HOP projects require Docker Compose version 1.27.0 or later.
You can check if you have Docker Compose already installed, with a supported version, by running the following command:
$ docker-compose --version docker-compose version 1.27.0, build 980ec85b
If you do not have Docker Compose installed, or if it is an older version than the ones supported, you can install a supported version on your operating system of choice, by following the instructions in the Docker Compose installation documentation.
Once you have installed (or upgraded) Docker Compose, you can check that it is correctly installed, and using a supported version, by running the following command in a terminal:
$ docker-compose --version docker-compose version 1.27.0, build 980ec85b
Install AWS Vault
As of this writing, no specific version of AWS Vault is required, but the latest available stable version is recommended.
To install it in your operating system of choice, please refer to the AWS Vault documentation.
If you install AWS Vault by downloading an executable binary from
GitHub Releases, do not forget to rename the downloaded file to
Also, if your operating system uses permissions to decide whether a
file can be executed or not (e.g., Linux or macOS), make sure you set
the executable permission to the
Once you have installed AWS Vault, you can check that it is installed correctly by running the following command:
$ aws-vault --version v6.6.1
Prepare AWS credentials
The HOP CLI will automatically provision the infrastructure needed to run the HOP project using AWS Cloudformation. In order to do that, you will need an existing AWS account (an IAM User) with administrator access.
Do not use the AWS root account user! Create a new IAM User and give it administrator access instead.
That IAM User (the administrator user, from here on) will need security credentials in the form of an Access Key. If the administrator user does not have an Access Key yet, you can create an Access Key from the AWS Console.
Although it is not mandatory, we strongly recommend storing the administrator user credentials using AWS Vault 1. Once you have the credentials, you can store them in AWS Vault using the following command:
$ aws-vault add --add-config hop/hop-tutorial-admin
The tool will ask you for the administrator user AWS Access Key Id, and its corresponding AWS Secret Access Key. Once you enter them, the setup will be done.
If you get an error saying:
aws-vault: error: Specified keyring backend not available, try --help
you are probably running AWS Vault in a system with no graphical
environment. In that case, for this tutorial, you can use the
AWS_VAULT_BACKEND environment variable and specify the
backend type. That backend will ask you for a passphrase and store
the credentials in an encrypted file on disk.
See aws-vault Backends documentation for additional details and, as suggested, run:
$ aws-vault add --help
for additional available options.
Configure the project settings
The HOP CLI allows the user to configure certain characteristics of the project to be generated and provisioned in AWS. That configuration is specified using a settings file. This tutorial uses the web-based Settings Editor for creating and editing that file. If you would like to edit the settings manually please refer to Configure the project settings using a text editor.
First launch the HOP CLI Settings Editor by running the following command:
$ hop bootstrap open-settings-editor Settings Editor running at http://localhost:8090
Now open the URL in a web browser and you will see the Settings Editor’s home-page. The first step is to select the HOP profiles. For this tutorial we will select the following: Core, Amazon Web Services, CI/CD, and Frontend.
Now hit next to configure the rest of the project settings.
In order to make this tutorial as simple as possible edit the following configuration options:
name: We will set the project name to
region: The AWS region where you want to create the project resources. Change to your desired region. So far the HOP CLI has been mainly tested on the
eu-west-1region. So we recommend you to use that region in order to ensure that all the services required by HOP application will be available 3.
Once you are done export the settings by clicking the “Export
settings” button. The browser will download a
that you will use in the next steps.
Make sure that the AWS region you configure is enabled in your AWS account. Not all the regions are enabled by default.
Also, make sure that the AWS region you configure has the AWS
Elastic Beanstalk service available. At the time of this writing,
some of them (e.g.,
eu-south-2) do not have it available. You
can check the list of available regions at AWS Elastic Beanstalk
endpoints and quotas.
If you already have an AWS account with existing resources, please refer to Settings file configuration for existing AWS account document for further considerations.
If this is the second time you are following this tutorial, some of the AWS resources created the first time you run the tutorial will still exist. The HOP CLI does not delete any AWS resources, to avoid deleting resources that may be in use. The HOP CLI does not overwrite any existing resource either, for the same reason.
This means you will need to delete those AWS resources manually yourself. Refer to How to delete AWS resources created by HOP for additional details.
Run the bootstrap command
Make sure that the clock of the machine where you run the following commands is correctly synchronized. The AWS services APIs used by the HOP CLI perform signature validation. And those signatures include the local time of the machine. If that local time is off by more than a certain threshold, the requests are denied.
Once you are happy with the selected configuration, you can bootstrap
the project by running the following command, where you need to
your-aws-region by the AWS region you configured in
the previous step.
$ aws-vault exec --no-session --region your-aws-region hop/hop-tutorial-admin -- hop bootstrap new-project --settings-file-path hop-tutorial-settings.edn --target-project-dir hop-tutorial-project
You are telling
aws-vault to run the
command, with the
hop/hop-tutorial-admin credentials that you
stored in the vault in a previous step.
Bootstrapping the project will take several minutes (mostly because of the AWS infrastructure provisioning). The tool will keep you informed about each step that it performs. Those steps are:
AWS infrastructure provisioning:
Account resources: Creates AWS resources that will be shared between multiple HOP projects.
Project resources: Creates AWS resources that will be shared between the different environments (test, production, etc.) inside the
Dev environment resources: Creates the AWS resources needed for local development on your machine.
Test environment resources: Creates the AWS resources for deploying and running the test environment.
Project files creation: Creates the local project files in the
Post-installation steps: If required, the tool will print additional steps that have to be performed manually after the bootstrap process has been completed.
Certain operations of the bootstrap process cannot be easily and securely automated. So the tool will print the steps that have to be performed manually. It is important not to lose the output of these post-installation steps until you have performed them all. Or to save that output in a secure place, because the output includes security credentials.
The post-installation steps output includes all the details about what to do, but we will describe them here too. Nevertheless, you will still need to pay attention to your post-installation steps output, to get the actual credentials generated by the bootstrap process.
Configure the local project credentials
First you will have to add the credentials for the AWS user used for local development. This user will be shared among all the HOP projects you run on your system. So this step will only have to be performed the first time you bootstrap a HOP project.
$ aws-vault add --add-config hop/hop-local-dev
and provide the Access Key ID and Secret Access Key values provided in the post-installation steps output.
Then you will have to configure the AWS IAM role used for running the
hop-tutorial project. That role contains the specific permissions
for interacting with the resources in the dev environment for the
hop-tutorial project. You will have to edit the AWS client
configuration file (usually in
~/.aws/config, but see AWS CLI
Configuration and credential file settings), and add the lines
provided in the post-installation steps output. It should look like
# Example configuration [profile hop/hop-tutorial-dev-env] source_profile = hop/hop-local-dev role_arn = arn:aws:iam::your-aws-account-id:role/hop-tutorial-dev-role region = your-aws-region
The tool will also print the Access Key ID and Secret Access Key for the CI/CD user. Take note of them, as you will need them in a later step to configure GitHub Actions.
Initialize the Git repository
Before you make any changes to the code, we suggest to initialize the git repository with the project files. That way you will easily track any changes that you make to the files generated by the HOP CLI.
$ cd hop-tutorial-project $ git init --initial-branch=main
If you have an older version of git that does not support the
--initial-branch option, you can run the following commands
$ git init $ echo 'ref: refs/heads/main' > .git/HEAD
As you can see, we are setting the git branch to
main, which is
the default value in the settings file. This setting is configurable
deployment-branch.name) and you can set it to any
other desired value in the settings file.
But it is important that the branch used in the above command aligns with the one configured in the settings file. Because the branch name configured in the settings file will be the only one used as the deployment branch by the CI/CD pipeline.
Now you can make the initial commit:
$ git add . $ git commit -m "Initial commit"
Run the project in the local development environment
At this point you are ready to run the project in the local
development environment. For that, simply run the
script. In short, the script will take care of three things:
Selecting the relevant docker-compose files used in the local development environment, and starting the Docker containers.
Making sure that the environment is started in a clean state.
Running the project with the
hop/hop-tutorial-dev-envcredentials, as they are needed to access the development environment AWS resources.
When you run the
start-dev.sh script, you will get all the log
entries from the Docker containers:
$ ./start-dev.sh ... ... lots of additional output ... ... app_1 | 2022-12-29T17:48:33.808145053Z nREPL server started on port 4001 on host 0.0.0.0 - nrepl://0.0.0.0:4001
You can see that the script started two Docker containers: the HOP web
application and a Nginx reverse proxy. The final log line, when
both containers have started and the application container is up and
running, should say something like
nREPL server started on port 4001
on host 0.0.0.0.
You can keep the script running and displaying new container log entries as they are produced, or you can stop the script execution. Stopping the script at this point does not have any unintended side effects like stopping the Docker containers, as they are started in the background. It simply stops displaying new container log entries.
The application container is up and running, but that does not mean
that the application web server is up and running too. You need to
start it manually from the REPL. In the application logs you should
see that the REPL is running on the port
Using your favorite Clojure(Script) IDE connect to the REPL. Some IDE’s call this type of REPL external or remote.
Once connected, you can load up the Duct development environment:
user=> (dev) "WARNING: abs already refers to: #'clojure.core/abs in namespace: medley.core, being replaced by: #'medley.core/abs" "WARNING: update-keys already refers to: #'clojure.core/update-keys in namespace: io.aviso.exception, being replaced by: #'io.aviso.exception/update-keys" :loaded dev=>
And then start the application itself:
At this point the application’s web server will be up and running, and
you should be able to connect to it from a web browser. The
command outputted that the web server is listening on port
3000. But you should connect through the Nginx reverse proxy that
is listening on port
Open a web browser and go to
http://localhost. You should see
HOP’s welcome page there.
We will not make any change to the code now, so once you check that the application is running, you can stop the whole development environment by executing the following script:
Create and configure the external GitHub repository
In this tutorial we will use GitHub for hosting the code, and GitHub Actions for the CI/CD pipeline. Both services are free of charge for public repositories. You can follow GitHub’s official documentation to create the repository.
Once you create the repository, GitHub will display a set of instructions to do your first commit and push. You need to ignore them however. You just need to configure the secrets used to deploy the HOP application to AWS, by following these steps:
Open the repository settings.
In the sidebar, go to Security → Secrets → Actions.
You will need to configure three secrets here, with the values of the CI user provided in the post installation steps output.
Then configure the local git repository to point to GitHub’s remote repository by execute the following command:
$ git remote add origin <github-repository-url>
And you can finally push your initial commit:
$ git push -u origin main
Deploy application to test environment
The first push to the GitHub repository will not trigger the CI/CD pipeline. So you will have to make a second commit to proceed.
app/src/hop_tutorial/client/landing.cljs file using your
favorite IDE and change the
"Your application is up and running!"
"Your application is up and running on AWS!!!!!!".
Now commit and push the change to GitHub:
$ git add app/src/hop_tutorial/client/landing.cljs $ git commit -m "Change landing message" $ git push
This second commit will start the CI/CE pipeline, which will perform the following steps:
Run linting and formatting checks against the
Run linting and formatting checks against the Clojure(Script) project files, using
Execute the project unit and integration tests.
Deploy the application to AWS Elastic Beanstalk test environment. In order to do that, the Docker production image is built and uploaded to AWS ECR first.
Once the pipeline has run successfully, the application should be already deployed to AWS. You can check that the deployment was successfully done from the the AWS Console. Log in into your AWS Account and go to the AWS Elastic Beanstalk service.
Now choose the
hop-tutorial-test environment under the
If the deployment was success, you will see a green check in the Health section. It might happen that when you open the page the application is still being deployed to the actual EC2 instance where it will run. So do not worry if you see a red or yellow health check for some time. It takes from one to three minutes for the whole deployment process to be completed, and the application being up and running and ready to serve requests.
If something goes wrong the errors will be displayed in the “Recent events” section, just below the health check section.
Accessing the web application in the test environment
Under the AWS Elastic Beanstalk environment name you will see the application’s publicly available URL.
When accessing it for the first time you will get a warning about the website SSL certificate being invalid. That is completely expected, as the web application is using a self-signed SSL certificate created by the HOP CLI.
You can tell the browser to ignore the warning until you setup a real SSL certificate 4.
There you have your new HOP-based application!
The tool provides secure storage for AWS credentials for the local development environment. The tool is also used in later steps for running the project locally,
The file uses the EDN format, and it needs to conform to a HOP-specific Malli Schema. Its structure is loosely based on GNU Emacs customization settings.
If you use any other AWS region and find any problem, please open an issue in the HOP CLI issue tracker.
You will need to provision the SSL certificate in AWS Certificate Manager. And then configure the AWS Load Balancer used by the test environment, to use that new certificate.