CI for this Blog with Azure DevOps

After I had set up my hugo driven blog on my local machine I wanted a solution that would help me publish any changes in an easy manner and also a backup solution. Therefore I decided to use some continuous integration/deployment approach.

Initial situation

In order to set up the CI for the blog, the files comprising it are in a folder on my computer and the folder is a local git repository. This is exactly the situation after following my previous post.

Goals

1. The blog should reside in a git repository in Azure Repos.
2. When pushing to the master branch of that repository, the blog should be built (i.e. hugo.exe should be executed) and published to blog.adwise.ch.
3. While publishing the site should be as little site downtime as possible. (For example the partially published site looking corrupted.)

Why Azure DevOps

• It’s free for public projects and, up to certain limits, even for private projects.
• It offers source control (Azure Repos) and build functionality (Azure Pipelines).
• It’s a platform that has just been rebranded from Visual Studio Team Services (VSTS) and therefore setting up this blog’s CI is a good oportunity to tinker with it.
• I already have a VSTS account (which is now an Azure DevOps account).

Plan

1. Initially push the local git repository to Azure Repos.
2. Set up an Azure Pipeline to have the blog generated whenever there is a push to the master branch of the Azure repository.
3. Upload the generated blog files to a working folder on my web server.
4. Swap the current public blog folder and the working folder.

Steps I perfomed to achieve above goals

Create a project in Azure DevOps

1. Logged in to Azure DevOps at https://dev.azure.com/<myUserName>
2. Created a new public project “adw1blog”.
   (Why public? Because I get unlimited build time that way. The drawback is that everybody can see the build logs. So no secrets must appear in build output ever.)

Push the local git repository to Azure Repos

1. Opened CMD on my computer and cd‘ed to my local adw1blog directory.
2. Executed the command
   git remote add origin https://teggno@dev.azure.com/teggno/adw1blog/_git/adw1blog
3. In the popup window, clicked my user account.

Create the build pipeline

1. In the browser, in the left navigation of Azure DevOps, clicked Pipelines.
2. On the right hand side under No build pipelines were found, clicked New Pipeline
   This prompted me to select a repository that should be the source for the build pipeline.
3. Under Select a source, selected Azure Repos Git.
4. Selected my adw1blog repository and the master branch in the respective dropdowns and clicked Continue.
5. At the top, under Select a template, clicked Empty job.
6. On the right hand side under Agent pool, select Hosted Windows Container.
   This will change the build computer (the “Agent”) to a computer that basically just has a Windows server operating system installed.
   For more information about the Hosted Windows Container agent see https://github.com/Microsoft/azure-pipelines-image-generation/blob/master/images/win/WindowsContainer1803-Readme.md
7. On the Save & queue dropdown button, clicked Save.

These steps left me with a build pipeline that would just get the source files of my blog. At this stage, the build pipeline also contained an empty Agent Job.

Add a task that installs hugo

As hugo is required to build the blog, it needs to be installed on the build agent.

1. Clicked the plus icon + near Agent Job 1 to add a task to install hugo on the build agent.
2. In the search box, entered command and clicked Command Line in the list of tasks below.
   This is a task that will execute arbitrary Windows CMD commands.
3. Clicked Add.
4. Clicked the Command Line Script item that had just been inserted under Agent job 1.
5. In the Display name box, entered “Install hugo”.
6. In the Script box, entered
   choco install hugo -confirm
   Using the Chocolatey package manager that is pre-installed on the build agent, this will install hugo on the build agent.

Add a task that runs hugo

After hugo has been installed, it can now be used to build the blog.

1. Clicked the plus icon + near Agent Job 1 to add a task to build the blog with hugo.
2. In the search box, entered command and clicked Command Line in the list of tasks below.
3. Clicked Add.
4. Clicked the Command Line Script item that had just been inserted under the previously created Install hugo task.
5. In the Display name box, entered “Run hugo”.
6. In the Script box, entered
   hugo
   (Notice that, by installing hugo using Chocolatey, hugo’s location was added to the PATH environment variable of the build agent which enables executing hugo by entering just the name of the exe instead of its full path.)

Add a task that creates the working folder

In case the working folder doesn’t exist yet on my web server, it needs to be created. This working folder will be the folder where the blog files will be uploaded to.

1. Clicked the plus icon + near Agent Job 1 to add a task to build the blog with hugo.
2. In the search box, entered command and clicked Command Line in the list of tasks below.
3. Clicked Add.
4. Clicked the Command Line Script item that had just been inserted under the previously created Run hugo task.
5. In the Display name box, entered “Create blogtemp folder”.
6. In the Script box, entered
   curl -u $(ftpUser):$(ftpPassword) ftp://$(ftpHost)/blogtemp/ --ftp-create-dirs

Variables

The above script curl -u $(ftpUser)... contains the variables $(ftpUser), $(ftpPassword) and $(ftpHost) which must be created in order to make the skript work. While doing that, variables that are needed for the steps below are created as well.
Variables are a good way to configure this kind of values because they enable storing values only in one place instead of repeating them in multiple tasks. More importantly variables can be of type “secret” which prevents their values from being viewed. This is especially helpful for passwords.

Creating the variables with the GUI is pretty straight forward. Just click the Variables tab and enter the variables with their respective values, one by one.

The following variables need to be created:

• ftpHost
• ftpUser
• ftpPassword
• sftpHost
• sftpUser
• sftpPassword

ftpPassword and sftpPassword need to be of type “secret”.

Note: As some of these variables are used in command line scripts, their values should conform to being used there. For example, if ftpUser contains %1 (which is interpreted as a variable by the command line), that might cause problems.

Add a task that uploads the blog files

This task will upload the output of the hugo executable to the working folder on the web server using SFTP.
The Azure DevOps Marketplace contains the SFTP Upload task that does just that. So first this task has to be installed from the marketplace.

Install the SFTP Upload task

1. Clicked the plus icon + near Agent Job 1 to add a new task.
2. In the search box, entered sftp and clicked SFTP Upload in the list of tasks below.
3. Clicked Get it free
   This opens the Azure DevOps Marketplace page in a new tab.
4. On the Azure DevOps Marketplace page, clicked Get it free.
5. Selected the Organization where I created the current project.
6. Clicked Install.

Use the SFTP Upload task to upload the blog

1. Went back to the Azure DevOps browser tab and, on the right hand side, next to Add tasks, clicked Refresh.
2. Clicked the SFTP Upload item again. It now had an Add button.
3. Clicked Add.
4. Clicked the SFTP item that had just been inserted under the previously created Create blogtemp folder task.
5. Filled the boxes as follows:
   Display name: “SFTP Upload to blogtemp.adwise.ch”
   Source folder: public
   This is the folder where hugo puts the generated blog files in the “Run hugo” task.
   Contents: **/*
   This will upload all the files in all subfolders of the public folder.
   Target folder: blogtemp.
   This is the working folder on the web server.
   Host name: $(sftpHost)
   User name: $(sftpUser)
   Password: $(sftpPassword)
6. In the Advanced section, checked
   Clean target folder
   This is required because the working folder might still contain the files from last time.
   Fail on clean errors
   Fail if no files found to copy

Add a task that swaps the working folder and the current public folder

As a last step, the working folder where the blog has just been uploaded to and the current public folder on the web server need to be swapped.

1. Clicked the plus icon + near Agent Job 1 to add a task to build the blog with hugo.

2. In the search box, entered command and clicked Command Line in the list of tasks below.

3. Clicked Add.

4. Clicked the Command Line Script item that had just been inserted under the previously created SFTP Upload to blogtemp.adwise.ch task.

5. In the Display name box, entered “Swap folders”.

6. In the Script box, entered

   curl -u $(ftpUser):$(ftpPassword) ftp://$(ftpHost) -Q "RNFR blog" -Q "RNTO blogold"
   curl -u $(ftpUser):$(ftpPassword) ftp://$(ftpHost) -Q "RNFR blogtemp" -Q "RNTO blog"
   curl -u $(ftpUser):$(ftpPassword) ftp://$(ftpHost) -Q "RNFR blogold" -Q "RNTO blogtemp"
   

Run the build pipeline when pushing to the master branch

As noted in the goals above, whenever changes are pushed to the Azure Repos repository, the blog should be published. The whole pipeline enabling that is now created but there is not yet a trigger that runs the pipeline automatically.

To run the pipeline automatically, I performed the following steps:

1. With the build pipeline still open in the browser, clicked Triggers.
2. Checked Enable continuous integration.
3. In Branch filters selected Type = Include and Branch specification = master.