Category: Go (Golang)

Deploy Golang App to Azure Web Apps with CI/CD on DevOps

by Chun Lin, posted in Cloud Computing: Microsoft Azure, DevOps, Experience, Git, Go (Golang)

Continue from the previous topic…

After we have our code on Github repository, now it’s time to automate our builds and deployments so that our Golang application will always be updated whenever there is a new change to our code on Github.

Sample Golang Web App DevOps Pipelines

To do that, we will use Azure DevOps and its Pipelines module. We can easily create a DevOps project in Azure Portal for our Golang application because there is a template available.

Golang is one of the supported languages in Azure DevOps.

As a start, we will focus on “Windows Web App” instead of containers. After that, we just need to configure basic information of the web app, such as its name, location, resource group, pricing tier, and application insights.

We can configure Application Insights while creating the DevOps project.

After that, we shall be able to see a new DevOps project created with the following two folders, Application and ArmTemplates, in Repos. Application folder contains a sample Golang application.

However, why is there an ArmTemplates folder? This is because by default when we create a new Azure DevOps project for Golang application using the steps above, it will also automatically create a web app for us. Hence, this is the ARM (Azure Resource Manager) template Azure uses to do that.

Content of ArtTemplate which is used to create/update the Azure web app.

With this pipeline setup, we can simply update the default Golang code in the Repos to launch our Golang application on Azure. However, what if we want to link Azure DevOps with the codes we already have on our Github repo?

Connecting DevOps with Github

To do that, let’s start again by creating a new project on Azure DevOps, instead of Azure portal. Here, I will make the DevOps project to be Public so that you can access it while reading this article.

Creating a new public DevOps project.

Once the project is created, we can proceed to the Project Settings page of the project to disable some modules that we don’t need, i.e. Boards and Repos.

We need to hide both Boards and Repos because Github provides us similar features.

Setting up Build Pipeline

After this, we then can proceed to create our Build pipeline by first a connecting to our Github repo.

If our code is neither on DevOps or Github, we can click “Use the visual designer” to proceed.

Before continuing to choose the corresponding Github repo, we need to have a azure-pipelines.yml. To understand the guidelines to write proper Azure DevOps Pipelines YAML, we can refer to the official guide. For Golang, there is another specific documentation on how to build and test Golang projects with Azure DevOps Pipelines.

For our case, we will have the following pipeline YAML file.

# Go 
# Build your Go project. 

resources: 
- repo: self 

pool:
   vmImage: 'vs2017-win2016'
 
steps: 
- task: GoTool@0
   inputs:
     version: 1.11.5
   displayName: 'Use Go 1.11.5'
- task: Go@0
   displayName: 'go get'
   inputs:
     arguments: '-d'
     workingDirectory: '$(System.DefaultWorkingDirectory)'
- task: Go@0
   displayName: 'go build'
   inputs:
     command: build
     arguments: '-o "$(System.TeamProject).exe"'
     workingDirectory: '$(System.DefaultWorkingDirectory)'
- task: ArchiveFiles@2
   displayName: 'Archive Files'
   inputs:
     rootFolderOrFile: '$(Build.Repository.LocalPath)'     
     includeRootFolder: False 
- task: PublishBuildArtifacts@1
   displayName: 'Publish Artifact'
   inputs:
     artifactName: drop

There are a few virtual machine images from Microsoft-hosted agent pool. We choose the “Visual Studio 2017 on Windows Server 2016 (vs2017-win2016)” image because I normally use Visual Studio 2017 for development.

The first task is the Go Tool Installer task. It will find and download a specific version of the Go tool into the tool cache and add it to the PATH. Here we will use the latest version of Golang which is 1.11.5 at the point of writing this article.

The subsequent step will be running go get. This command will download the packages along with their dependencies. Since the -d argument is present, it will only download them but not install them.

After that, it will run go build. This step compiles the packages along with their dependencies, but it does not install the results. By default, the build command will write the resulting executable to an output file named after the first source file (or the source code directory). However, with the -o flag here, it forces build to write the resulting executable to the output file named $(System.TeamProject).exe, i.e. GoLab.exe.

Next we use the Archive Files task to create an archive file from a source folder. Finally, we use the Publish Build Artifacts task to publish build artifact to DevOps pipelines. With Archive Files task, it will generate a zip file called as such D:\a\1\a\54.zip where 54 is the build id. Publish Build Artifacts task will then upload the zip file to file container called drop.

Details of the Archive Files task.

To find out what is inside the file container drop, we can download it from the Summary page of the build. It is actually a folder containing all the files of our Golang application.

We can download the drop from the Summary page of the build.

Setting up Release Pipeline

Now we can proceed to create our Release pipeline. Luckily, there is already a template available to help us kick starting the Release pipeline.

The “Deploy a Go app to Azure App Service” pipeline is what we need here.

After selecting the template, we will need to specify the artifact, as shown below. There is version that we can choose, for example, the latest version from a specific branch with tags. Here we choose Latest so that our latest code change will always get deployed to Azure Web Apps.

Adding artifact.

Next, we need to enable the CD trigger as shown in the following screenshot so that a new release will be created every time a new build is available.

Enabling CD trigger.

Now we are at Pipeline tab. What we need to next is to move on to the Tasks tab, which is now having a red exclamation mark. We just need to authorize the Release pipeline to our Azure subscription and then connect it to the Azure Web App in the subscription.

Completing tasks.

Now, as you can see, the agent basically does three steps:

  • Stop the Azure Web App;
  • Deploy our code to Web App;
  • Start the Web App.

What interests us here is the second step. The reason why we need to generate a zip file in Build pipeline is also because in the second step, we need to specify the file path to the zip files to deploy.

Default configuration of second step.

Finally, we can just Save the pipeline and rename the “New release pipeline” to another friendlier name.

Now we can manually create a Release to test it out.

Create a new release manually.

Since we trigger this release manually, we also need to click in to deploy it manually.

Deploying to Azure App Service in progress.

After the deployment is done, we can view its summary as shown below.

The deployment process of the agent.

Conclusion

That’s all for setting up simple build and release pipelines on Azure DevOps to deploy our Golang web app to Azure Web Apps.

Getting Started: Making It a Golang Web Application

by Chun Lin, posted in Experience, Go (Golang), PostgreSQL, SQL

Continue from the previous topic…

Playing YouTube video via console is cool but it’s not user friendly. Hence, I decided to move on to make it a web app. Design of a web app that I chose to follow is the one shared in Sau Sheong Chang’s book, Go Web Programming. The following diagram covers the overview of the architecture of the web app.

Overview of the web application. (Source: Go Web Programming)

Hence, I started of with two folders and two Go files in the project folder. The two folders are public and templates folders. The public folder stores all CSS files and Javascript files. Then we have one index.go file which is basically the handler function for the homepage of our web app and finally server.go file.

The server.go

In the server.go file, we have our multiplexer which in charge of inspecting the URL being requested and redirecting the request to the correct handler. The main function also sits in server.go file. There is where we also connect to the database which has its connection made to be global. The following code shows the beginning of our main function.

package main

...

var db *sql.DB

func main() { 

    var err error 

    // Initialize connection string.
    var connectionString = fmt.Sprintf(os.Getenv("CONNECTION_STRING"))

    // Initialize connection object.
    db, err = sql.Open("postgres", connectionString)
    checkError(err)

    // Set up multiplexer
    mux := http.NewServeMux()

    // Handle static files in the public folder
    staticFiles := http.FileServer(http.Dir("public"))
    mux.Handle("/static/", http.StripPrefix("/static/", staticFiles))

    mux.HandleFunc("/index", index)

    server := &http.Server{
        Addr:    "127.0.0.1:8081",
        Handler: mux,
    }

    server.ListenAndServe()
}

There is one thing that needs to take note is that, as described on the GoDoc, “The returned DB (of sql.Open) is safe for concurrent use by multiple goroutines and maintains its own pool of idle connections. Thus, the Open function should be called just once. It is rarely necessary to close a DB.” So we don’t close the DB here.

The index.go

For the index.go file where we have our homepage handler function, we will first connect to the database to retrieve the list of videos from the table. In addition, the index handler function will be in charge of generating the HTML responses with templates that we define in templates folder, which is index.html in this case.

The following code shows the index handler function where it retrieves all the video records from the table then store them into a map, the built-in dictionary data type in Go.

package main

...

func index(writer http.ResponseWriter, request *http.Request) {
    template, _ := template.ParseFiles("templates/index.html")

    err := db.Ping()
    checkError(err)

    if err != nil {
        template.Execute(writer, "Cannot connect to the database")
    } else {
        // Read data from table.
        ...

        sqlStatement := "SELECT * FROM videos;"

        rows, err := db.Query(sqlStatement)
        checkError(err)

        defer rows.Close()

        videos := make(map[string]string)

        ...

        template.Execute(writer, videos)
    }
}

The templates/index.html

Now, let’s see what’s inside index.html template file. It basically uses the Bootstrap 4.2.1 template that I downloaded from Bootswatch.

<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="utf-8">
    <title>YouTube RePlayer</title>
    <meta name="viewport" content="width=device-width, initial-scale=1">
    <meta http-equiv="X-UA-Compatible" content="IE=edge" />
    <link rel="stylesheet" href="static/bootstrap.min.css" media="screen">
</head>

<body> 
    ...
    {{ range $videoId, $videoName := . }}
        ...
        <span>{{ $videoName }}</span>
        ...
    {{ end }}
    ...
</body>
</html>

There are two highlights in the code above.

Firstly, even though we put the CSS and JS files in the public folder, we still have their relative path as “static/…”. The reason is because of the lines in server.go as shown below.

// Handle static files in the public folder
staticFiles := http.FileServer(http.Dir("public"))
mux.Handle("/static/", http.StripPrefix("/static/", staticFiles))

Secondly, we iterate through the map using range. That will help to list down the videos we retrieved earlier on the HTML side.

The public Folder

Since we are using Bootstrap, we need to have the CSS and JS files of the framework locally. Hence, they are all put under the public folder. Then files with my customized CSS and JS codes are also put in this folder.

This is the web app so far with the “Add To List” function that will be covered in another article later.

Form Submission

Now I move on to add a function that allows users to add new YouTube video the the video list. On HTML, we have the following hidden fields. They are hidden fields because the values, i.e. YouTube video id and video title, are retrieved from the URL and the YouTube API, respectively.

<form action="/addVideo" method="POST">
    <input id="hidVideoID" name="hidVideoID" type="hidden" />
    <input id="hidVideoName" name="hidVideoName" type="hidden" />

    <input id="btnAddToList" type="submit" class="btn btn-primary btn-lg" value="Add to List"></input>
</form>

After that, due to the fact that I want the form to be posted to the relative request URL /addVideo, so there is a new handler needed. Hence, a new line as follows is added to the server.go file.

mux.HandleFunc("/addVideo", addVideo)

Then we have our new handler function in a new file, addVideo.go, as shown below.

package main

...

func addVideo(writer http.ResponseWriter, request *http.Request) {
    request.ParseForm()

    err := db.Ping()
    checkError(err)

    if err != nil {

        http.Redirect(writer, request, "/index", http.StatusSeeOther)

    } else {

        // Insert data into the table.
        sqlStatement := "INSERT INTO videos (name, url) VALUES ($1, $2);"

        _, err = db.Exec(sqlStatement, request.PostForm["hidVideoName"][0], "https://www.youtube.com/watch?v="+(request.PostForm["hidVideoID"][0]))
        checkError(err)

        http.Redirect(writer, request, "/index", http.StatusSeeOther)
    }
}

As the code above shows, after the data is successfully saved to the database table, we need to redirect the user back to the homepage.

References

1. Golang: http.Redirect;
2. Golang SQL Database Open and Close;
3. Golang: How to Redirect to an URL.

Getting Started: Connecting Golang Console App with Azure PostgreSQL Database

by Chun Lin, posted in Cloud Computing: Microsoft Azure, Experience, Go (Golang), PostgreSQL, SQL

Setting up Go in VS Code

Firstly, I need to equip Visual Studio Code with the Go language support by installing Go extension in the IDE. Installing the extension helps us to do day-to-day job, such as, code navigating, code editing, code testing and debugging in an easy and efficient manner.

Installed and enabled the Go extension from Microsoft.

There are many cool features I like in the extension.

Firstly, it’s the convenience of F12 Code Navigation. With Go extension, in Go code I can easily view the source code of the type definition with just F12 or Alt+F12 on it. This is similar to my C# coding experience on Visual Studio too. To show all references of the type, I can simply use Shift+F12.

Secondly, for every file save, the extension will build, vet, and lint. Build (go build) builds the command or the package. Vet (go vet) examines Go source code and reports suspicious constructs, such as Printf calls whose arguments do not align with the format string. A linter is a tool giving coding style feedback and suggestions. By default, this extension uses the official golint as a linter.

We can update go.lintTool to other linter, for example the more advanced Go Meta Linter. Take note of the warnings (and errors) shown under the Problems tab below.

After I have installed the Go extension, I proceed to start coding. The app that I am going to build in this learning journey is a YouTube video player app. So, let’s get started!

Storing Connection String in Environment

I will start with building a console application connecting to Azure PostgreSQL. To do that, I will connect to the database as follows.

package main;

const (
     // Initialize connection constants.
     HOST     = "...postgres.database.azure.com"
     DATABASE = "..."
     USER     = "..."
     PASSWORD = "..."
)

...

function main() {
    var connectionString = fmt.Sprintf("host=%s port=5432 user=%s password=%s dbname=%s sslmode=require", HOST, USER, PASSWORD, DATABASE) 
    ...
}

The code above is quite straight-forward. However, it does have a problem. The configuration to connect to database is hard-coded in the code. This will thus reduce the code maintainability. Also, it is extremely bad to have password appearing in the code in plain text. To store such sensitive data, we need to store them as Environment variables, as recommended by the article Best Practice for Configuration File in Your Code.

In Powershell, there are two ways of setting Environment variable. The first one is a temporarily variable that lasts only as long as the Powershell session. The command is as follows.

> $env:CONNECTION_STRING = '...'

If what you are looking for is a permanent one, you can do as follows.

> [environment]::SetEnvironmentVariable("CONNECTION_STRING", "...", "User")

After setting the connection string in environment variable, we then can edit the earlier code to be something shorter as follows.

package main;

…

function main() {
    var connectionString = fmt.Sprintf(os.Getenv("CONNECTION_STRING")) 

     …
}

Connecting to Database

After that, we can initialize the connection object.

package main

...
 
func checkError(err error) {
    if err != nil {
        log.Fatal(err)
        panic(err)
    }
} 

func main() {
    var connectionString = ...

    // Initialize connection string.
    db, err := sql.Open("postgres", connectionString)
    checkError(err)

    err = db.Ping()
    checkError(err)
    fmt.Println("Successfully created connection to database") 

    ...
}

The Ping() function verifies a connection to the database is alive and it will establish a connection if necessary. After that, we can use the database handler db to do CRUD operations, as demonstrated below.

Remember to allow the PostgreSQL access to Azure service and also your local IP if you need to access the database from your local machine.

Insert Data into Table

sqlStatement := "INSERT INTO table (column1) VALUES ($1);"
_, err = db.Exec(sqlStatement, "New Value")
checkError(err)

Read Data from Table

var id int
var column1 string

sqlStatement := "SELECT * from table;"
rows, err := db.Query(sqlStatement)
checkError(err)

defer rows.Close()

for rows.Next() {
    switch err := rows.Scan(&id, &column1); err {
    case sql.ErrNoRows:
       fmt.Println("No rows were returned")
    case nil:
       fmt.Printf("Data row = (%d, %s)\n", id, column1)
    default:
       checkError(err)
    }
}

Update Data in Database

sqlStatement := "UPDATE table SET column1 = $1 WHERE id = $2;"
_, err = db.Exec(sqlStatement, "New Value 2", 1)
checkError(err)

Delete Data from Database

sqlStatement := "DELETE FROM table WHERE id = $1;"
_, err = db.Exec(sqlStatement, 1)
checkError(err)

A YouTube Playlist Console App

With the codes above, we then can create a simple table to store our YouTube video URLs as a playlist, as shown in the following screenshot.

The console app allows user to choose video from the list and plays it on default browser.

To make the program to play the video on a browser, we then can make use of the code written by hyg (黄勇刚) which is to open a web page on the default browser of the machine, as shown below.

func openbrowser(url string) {
    var err error

    switch runtime.GOOS {
    case "linux":
        err = exec.Command("xdg-open", url).Start()
    case "windows":
        err = exec.Command("rundll32", "url.dll,FileProtocolHandler", url).Start()
    case "darwin":
        err = exec.Command("open", url).Start()
    default:
        err = fmt.Errorf("unsupported platform")
    }

    if err != nil {
        checkError(err)
    }

}

Playing YouTube video via console is cool, isn’t it? Next, we will see how we can make it to be web application so that we can have a better UI/UX.

To be continued in next article…

References

1. Quickstart: Create an Azure Database for PostgreSQL server in the Azure portal;
2. Azure Database for PostgreSQL: Use Go language to connect and query data;
3. [StackOverflow] List all environment variables from command line;
4. Azure Database for PostgreSQL Server firewall rules;
5. PostgreSQL – CREATE Database;
6. [StackOverflow] How to switch databases in psql?;
7. [StackOverflow] How to read input from console line?;
8. [StackOverflow] Convert string to integer type in Go?;
9. [StackOverflow] Reading an integer from standard input;
10. [StackOverflow] How to exit from PostgreSQL command line utility: psql.

Spring 2019 Golang Self-Learning

by Chun Lin, posted in Cloud Computing: Microsoft Azure, Experience, Go (Golang)

As a web developer in C# and .NET ecosystem for seven years, I decided to learn Golang after I was introduced the Chang Sausheong’s book “Go Web Programming”.

Objectives

Learning a new language other than C#, especially a language which is not in .NET, gives me a new perspective on understanding web development. With powerful framework like ASP .NET Core, it’s easy for developers who build their web applications do not understand why the frameworks do things in certain ways and trade-offs in the frameworks.

ASP .NET Core makes developers’ life easier with Convention over Configuration concept. As long as we know the conventions of where everything is located and you place each component into its correct location, we can easily build a web application. However, this also hide too much details from us and eventually makes us hard to master web development. Hence, learning Golang helps me to gain a new perspective in understanding web development.

When I am working with ASP .NET Core, I normally deal with MS SQL Server or Azure SQL (which is cloud-based MS SQL Server). In the Golang web development, I switch to use PostgreSQL, which is also available on Microsoft Azure.

Contents

During the long holiday in Chinese New Year, I spent my time in reading the book and online resources regarding Golang web programming. With the new opening of library beside my workplace, I also get to spend time there after work to do some Golang coding.

After one to two months of self-learning, I managed to compile some notes about what I’ve learnt in this Spring. The following is a list of nine topics that I learned in the early-stage of my Golang journey.

  1. Getting Started: Connecting Golang Console App with Azure PostgresSQL Database;
  2. Getting Started: Making It a Golang Web Application;
  3. Deploy Golang App to Azure Web Apps with CI/CD on DevOps;
  4. Monitoring Golang Web App with Application Insights;
  5. RESTful Web Service in Golang and Front-end Served with VueJS;
  6. Authenticating Users in Golang Web App;
  7. Unit Testing with Golang;
  8. Containerize Golang Code and Deploy to Azure Web App;
  9. Leveraging Golang Concurrency in Web App Development.

Learning After Work

I’m working in Haulio, a startup incubated by PSA. Since it’s a startup and I am the CTO, I have no choice but to stay late in office most of the time, especially the period when we rushed for the launch of our mobile application in end of February.

Hence, whenever I have time, I will spend it on doing some Golang research and coding. Sometimes, however, I am just too tired and I would not write anything even though it’s weekend. Hence, I end up finishing all the nine topics only in mid of March.

I’m also very fortunate to have to share what I have learned with engineers in Azure Community Singapore. It turns out that many of them are using Golang too. Hence, that also gives me a great opportunity to learn from those experienced Golang developers. =)

Sharing how to deploy Golang web application with Azure DevOps Pipelines.

Again, I am not that hardworking to work on personal projects every day. Sometimes, I will accompany my Mom at home. Sometimes I will have dinner with friends. Sometimes, I will travel to overseas (I am writing this in Japan). Sometimes, I will also play computer games or simply just sleep at home. So ya, this self-learning project takes a longer time to complete.

Working on personal projects after work is stressful also. Yup, so the project involved in this self-learning is about creating a YouTube Re-Player to loop my favourite YouTube music to calm myself down. =P