Run a HOP application on AWS

Introduction

In this tutorial you will bootstrap a basic HOP project with the following features:

Note

The HOP Command Line Interface tool (HOP CLI, from here on) provides more bootstrapping features, but this tutorial will focus in a bare minimum installation.

Prerequisites

Having the HOP CLI installed, as explained in the Installation tutorial, is enough for bootstrapping a new HOP project. But in order to run the generated project locally on your machine, you will need some additional tools:

Apart from the software requirements, you will also need an Amazon Web Services account with administrator credentials, in order to create the infrastructure needed.

Install Docker

As of this writing, HOP projects require Docker version 18.09 or later. You can check if you have Docker already installed, and that it is a supported version, by running the following command in a terminal:

$ docker --version
Docker version 18.09.1, build 4c52b90

If you do not have Docker installed, or if it is an older version than the ones supported, you can install a supported version on your operating system of choice, by following the instructions in the Docker installation documentation.

Once you have installed (or upgraded) Docker, you can check that it is correctly installed by running the following command:

$ docker run --rm hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

[... more text appears here ...]

For more examples and ideas, visit:
https://docs.docker.com/get-started/

Note

If you get an error about not having permission to connect to the Docker daemon socket, check the Docker post-installation steps documentation:

You can also check that you have a supported version by re-running the command you used before:

$ docker --version
Docker version 18.09.1, build 4c52b90

Install Docker Compose

As of this writing, HOP projects require Docker Compose version 1.27.0 or later.

You can check if you have Docker Compose already installed, with a supported version, by running the following command:

$ docker-compose --version
docker-compose version 1.27.0, build 980ec85b

If you do not have Docker Compose installed, or if it is an older version than the ones supported, you can install a supported version on your operating system of choice, by following the instructions in the Docker Compose installation documentation.

Once you have installed (or upgraded) Docker Compose, you can check that it is correctly installed, and using a supported version, by running the following command in a terminal:

$ docker-compose --version
docker-compose version 1.27.0, build 980ec85b

Install AWS Vault

As of this writing, no specific version of AWS Vault is required, but the latest available stable version is recommended.

To install it in your operating system of choice, please refer to the AWS Vault documentation.

Note

If you install AWS Vault by downloading an executable binary from GitHub Releases, do not forget to rename the downloaded file to aws-vault.

Also, if your operating system uses permissions to decide whether a file can be executed or not (e.g., Linux or macOS), make sure you set the executable permission to the aws-vault file.

Once you have installed AWS Vault, you can check that it is installed correctly by running the following command:

$ aws-vault --version
v6.6.1

Prepare AWS credentials

The HOP CLI will automatically provision the infrastructure needed to run the HOP project using AWS Cloudformation. In order to do that, you will need an existing AWS account (an IAM User) with administrator access.

Warning

Do not use the AWS root account user! Create a new IAM User and give it administrator access instead.

That IAM User (the administrator user, from here on) will need security credentials in the form of an Access Key. If the administrator user does not have an Access Key yet, you can create an Access Key from the AWS Console.

Although it is not mandatory, we strongly recommend storing the administrator user credentials using AWS Vault [1]. Once you have the credentials, you can store them in AWS Vault using the following command:

$ aws-vault add --add-config hop/hop-tutorial-admin

The tool will ask you for the administrator user AWS Access Key Id, and its corresponding AWS Secret Access Key. Once you enter them, the setup will be done.

Note

If you get an error saying:

aws-vault: error: Specified keyring backend not available, try --help

you are probably running AWS Vault in a system with no graphical environment. In that case, for this tutorial, you can use the AWS_VAULT_BACKEND environment variable and specify the file backend type. That backend will ask you for a passphrase and store the credentials in an encrypted file on disk.

See aws-vault Backends documentation for additional details and, as suggested, run:

$ aws-vault add --help

for additional available options.

Configure the project settings

The HOP CLI allows the user to configure certain characteristics of the project to be generated and provisioned in AWS. That configuration is specified using a settings file. This tutorial uses the web-based Settings Editor for creating and editing that file. If you would like to edit the settings manually please refer to Configure the project settings using a text editor.

First launch the HOP CLI Settings Editor by running the following command:

$ hop bootstrap open-settings-editor
Settings Editor running at http://localhost:8090

Now open the URL in a web browser and you will see the Settings Editor’s home-page. The first step is to select the HOP profiles. For this tutorial we will select the following: Core, Amazon Web Services, CI/CD, and Frontend.

../../_images/settings-editor-profile-picker.png

Now hit next to configure the rest of the project settings.

../../_images/settings-editor-editor.png

In order to make this tutorial as simple as possible edit the following configuration options:

  • projectname: We will set the project name to "hop-tutorial".

  • deployment-targetawsaccountregion: The AWS region where you want to create the project resources. Change to your desired region. So far the HOP CLI has been mainly tested on the eu-west-1 region. So we recommend you to use that region in order to ensure that all the services required by HOP application will be available [3].

Once you are done export the settings by clicking the “Export settings” button. The browser will download a settings.edn file that you will use in the next steps.

Note

Make sure that the AWS region you configure is enabled in your AWS account. Not all the regions are enabled by default.

Also, make sure that the AWS region you configure has the AWS Elastic Beanstalk service available. At the time of this writing, some of them (e.g., eu-south-2) do not have it available. You can check the list of available regions at AWS Elastic Beanstalk endpoints and quotas.

Warning

If you already have an AWS account with existing resources, please refer to Settings file configuration for existing AWS account document for further considerations.

Note

If this is the second time you are following this tutorial, some of the AWS resources created the first time you run the tutorial will still exist. The HOP CLI does not delete any AWS resources, to avoid deleting resources that may be in use. The HOP CLI does not overwrite any existing resource either, for the same reason.

This means you will need to delete those AWS resources manually yourself. Refer to How to delete AWS resources created by HOP for additional details.

Run the bootstrap command

Note

Make sure that the clock of the machine where you run the following commands is correctly synchronized. The AWS services APIs used by the HOP CLI perform signature validation. And those signatures include the local time of the machine. If that local time is off by more than a certain threshold, the requests are denied.

Once you are happy with the selected configuration, you can bootstrap the project by running the following command, where you need to substitute your-aws-region by the AWS region you configured in the previous step.

$ aws-vault exec --no-session --region your-aws-region hop/hop-tutorial-admin -- hop bootstrap new-project --settings-file-path hop-tutorial-settings.edn --target-project-dir hop-tutorial-project

You are telling aws-vault to run the hop bootstrap command, with the hop/hop-tutorial-admin credentials that you stored in the vault in a previous step.

Bootstrapping the project will take several minutes (mostly because of the AWS infrastructure provisioning). The tool will keep you informed about each step that it performs. Those steps are:

  • AWS infrastructure provisioning:

    • Account resources: Creates AWS resources that will be shared between multiple HOP projects.

    • Project resources: Creates AWS resources that will be shared between the different environments (test, production, etc.) inside the hop-tutorial project.

    • Dev environment resources: Creates the AWS resources needed for local development on your machine.

    • Test environment resources: Creates the AWS resources for deploying and running the test environment.

  • Project files creation: Creates the local project files in the hop-tutorial-project folder.

  • Post-installation steps: If required, the tool will print additional steps that have to be performed manually after the bootstrap process has been completed.

    Certain operations of the bootstrap process cannot be easily and securely automated. So the tool will print the steps that have to be performed manually. It is important not to lose the output of these post-installation steps until you have performed them all. Or to save that output in a secure place, because the output includes security credentials.

    The post-installation steps output includes all the details about what to do, but we will describe them here too. Nevertheless, you will still need to pay attention to your post-installation steps output, to get the actual credentials generated by the bootstrap process.

Configure the local project credentials

First you will have to add the credentials for the AWS user used for local development. This user will be shared among all the HOP projects you run on your system. So this step will only have to be performed the first time you bootstrap a HOP project.

$ aws-vault add --add-config hop/hop-local-dev

and provide the Access Key ID and Secret Access Key values provided in the post-installation steps output.

Then you will have to configure the AWS IAM role used for running the hop-tutorial project. That role contains the specific permissions for interacting with the resources in the dev environment for the hop-tutorial project. You will have to edit the AWS client configuration file (usually in ~/.aws/config, but see AWS CLI Configuration and credential file settings), and add the lines provided in the post-installation steps output. It should look like the following:

# Example configuration
[profile hop/hop-tutorial-dev-env]
source_profile = hop/hop-local-dev
role_arn = arn:aws:iam::your-aws-account-id:role/hop-tutorial-dev-role
region = your-aws-region

The tool will also print the Access Key ID and Secret Access Key for the CI/CD user. Take note of them, as you will need them in a later step to configure GitHub Actions.

Initialize the Git repository

Before you make any changes to the code, we suggest to initialize the git repository with the project files. That way you will easily track any changes that you make to the files generated by the HOP CLI.

$ cd hop-tutorial-project
$ git init --initial-branch=main

Note

If you have an older version of git that does not support the --initial-branch option, you can run the following commands instead:

$ git init
$ echo 'ref: refs/heads/main' > .git/HEAD

As you can see, we are setting the git branch to main, which is the default value in the settings file. This setting is configurable (under projectprofilescicontinuous-deploymentgitdeployment-branch.name) and you can set it to any other desired value in the settings file.

But it is important that the branch used in the above command aligns with the one configured in the settings file. Because the branch name configured in the settings file will be the only one used as the deployment branch by the CI/CD pipeline.

Now you can make the initial commit:

$ git add .
$ git commit -m "Initial commit"

Run the project in the local development environment

At this point you are ready to run the project in the local development environment. For that, simply run the start-dev.sh script. In short, the script will take care of three things:

  • Selecting the relevant docker-compose files used in the local development environment, and starting the Docker containers.

  • Making sure that the environment is started in a clean state.

  • Running the project with the hop/hop-tutorial-dev-env credentials, as they are needed to access the development environment AWS resources.

When you run the start-dev.sh script, you will get all the log entries from the Docker containers:

$ ./start-dev.sh
...
... lots of additional output  ...
...
app_1    | 2022-12-29T17:48:33.808145053Z nREPL server started on port 4001 on host 0.0.0.0 - nrepl://0.0.0.0:4001

You can see that the script started two Docker containers: the HOP web application and a Nginx reverse proxy. The final log line, when both containers have started and the application container is up and running, should say something like nREPL server started on port 4001 on host 0.0.0.0.

You can keep the script running and displaying new container log entries as they are produced, or you can stop the script execution. Stopping the script at this point does not have any unintended side effects like stopping the Docker containers, as they are started in the background. It simply stops displaying new container log entries.

The application container is up and running, but that does not mean that the application web server is up and running too. You need to start it manually from the REPL. In the application logs you should see that the REPL is running on the port 4001.

Using your favorite Clojure(Script) IDE connect to the REPL. Some IDE’s call this type of REPL external or remote.

Once connected, you can load up the Duct development environment:

user=> (dev)
"WARNING: abs already refers to: #'clojure.core/abs in namespace: medley.core, being replaced by: #'medley.core/abs"
"WARNING: update-keys already refers to: #'clojure.core/update-keys in namespace: io.aviso.exception, being replaced by: #'io.aviso.exception/update-keys"
:loaded
dev=>

And then start the application itself:

dev=> (go)
:duct.server.http.jetty/starting-server {:port 3000}
"WARNING: abs already refers to: #'clojure.core/abs in namespace: day8.re-frame-10x.inlined-deps.garden.v1v3v10.garden.color, being replaced by: #'day8.re-frame-10x.inlined-deps.garden.v1v3v10.garden.color/abs"
"JavaScript environment will not launch automatically when :open-url is false"
:initiated
dev=>

At this point the application’s web server will be up and running, and you should be able to connect to it from a web browser. The (go) command outputted that the web server is listening on port 3000. But you should connect through the Nginx reverse proxy that is listening on port 80 instead.

Open a web browser and go to http://localhost. You should see HOP’s welcome page there.

../../_images/local-running-hop-app.png

We will not make any change to the code now, so once you check that the application is running, you can stop the whole development environment by executing the following script:

$ ./stop-dev.sh

Create and configure the external GitHub repository

In this tutorial we will use GitHub for hosting the code, and GitHub Actions for the CI/CD pipeline. Both services are free of charge for public repositories. You can follow GitHub’s official documentation to create the repository.

Once you create the repository, GitHub will display a set of instructions to do your first commit and push. You need to ignore them however. You just need to configure the secrets used to deploy the HOP application to AWS, by following these steps:

  1. Open the repository settings.

  2. In the sidebar, go to Security → Secrets → Actions.

  3. You will need to configure three secrets here, with the values of the CI user provided in the post installation steps output.

    • AWS_ACCESS_KEY_ID

    • AWS_SECRET_ACCESS_KEY

    • AWS_DEFAULT_REGION

../../_images/github-secrets.png

Then configure the local git repository to point to GitHub’s remote repository by execute the following command:

$ git remote add origin <github-repository-url>

And you can finally push your initial commit:

$ git push -u origin main

Deploy application to test environment

The first push to the GitHub repository will not trigger the CI/CD pipeline. So you will have to make a second commit to proceed.

Open the app/src/hop_tutorial/client/landing.cljs file using your favorite IDE and change the "Your application is up and running!" message to "Your application is up and running on AWS!!!!!!".

Now commit and push the change to GitHub:

$ git add app/src/hop_tutorial/client/landing.cljs
$ git commit -m "Change landing message"
$ git push

This second commit will start the CI/CE pipeline, which will perform the following steps:

  1. Run linting and formatting checks against the sh, yaml, json and Dockerfile files.

  2. Run linting and formatting checks against the Clojure(Script) project files, using clj-kondo, eastwood and cljfmt.

  3. Execute the project unit and integration tests.

  4. Deploy the application to AWS Elastic Beanstalk test environment. In order to do that, the Docker production image is built and uploaded to AWS ECR first.

../../_images/github-actions-pipeline-run.png

Once the pipeline has run successfully, the application should be already deployed to AWS. You can check that the deployment was successfully done from the the AWS Console. Log in into your AWS Account and go to the AWS Elastic Beanstalk service.

../../_images/elasticbeanstalk-environment-list.png

Now choose the hop-tutorial-test environment under the hop-tutorial application.

../../_images/elasticbeanstalk-environment.png

If the deployment was success, you will see a green check in the Health section. It might happen that when you open the page the application is still being deployed to the actual EC2 instance where it will run. So do not worry if you see a red or yellow health check for some time. It takes from one to three minutes for the whole deployment process to be completed, and the application being up and running and ready to serve requests.

If something goes wrong the errors will be displayed in the “Recent events” section, just below the health check section.

Accessing the web application in the test environment

Under the AWS Elastic Beanstalk environment name you will see the application’s publicly available URL.

../../_images/elasticbeanstalk-environment-url.png

When accessing it for the first time you will get a warning about the website SSL certificate being invalid. That is completely expected, as the web application is using a self-signed SSL certificate created by the HOP CLI.

../../_images/self-signed-certificate-warning.png

You can tell the browser to ignore the warning until you setup a real SSL certificate [4].

../../_images/aws-running-hop-app.png

There you have your new HOP-based application!

Footnotes