RESTful Web Service in Golang and Front-end Served with VueJS

Continue from the previous topic

There is one behaviour in our Golang web application that will frustrate our users. Everytime we add, update, or delete a video record, the web page gets refreshed. So if at that time there is a video being played, then poof, it’s gone after you add, update, or delete a record. That’s a bad UX.

To overcome there, I decide to implement RESTful web services in the Golang project and the frontend will use VueJS library to update the web page.

Firstly, we need to wrap a web service interface over the CRUD functions we have in our web application. JSON will be used as the data transport format. To do that, we will introduce a new handler function to multiplex request to the correct function in our RESTful web service.

func handleVideoAPIRequests(video models.IVideo) http.HandlerFunc {
return func(writer http.ResponseWriter, request *http.Request) {

var err error
switch request.Method {
case "GET":
err = handleVideoAPIGet(writer, request, video)

case "POST":
err = handleVideoAPIPost(writer, request, video)

case "PUT":
err = handleVideoAPIPut(writer, request, video)

case "DELETE":
err = handleVideoAPIDelete(writer, request, video)
}

if err != nil {
util.CheckError(err)

return
}

}
}

HTTP GET

Then for each of the HTTP methods, we will process the request independently. For example, to retrieve the list of videos or one of the videos, we will use GET method, i.e. handleVideoAPIGet().

The list of video records at the right are retrieved using HTTP GET.
func handleVideoAPIGet(writer http.ResponseWriter, request *http.Request, video models.IVideo) (err error) {

videoIDURL := path.Base(request.URL.Path)

var output []byte

if videoIDURL == "video" {
videos, errIf := video.GetAllVideos()
err = errIf
util.CheckError(errIf)

output, errIf = json.MarshalIndent(&videos, "", "\t")
err = errIf
util.CheckError(errIf)

writer.Header().Set("Content-Type", "application/json")
writer.Write(output)

return
}

videoID, err := strconv.Atoi(videoIDURL)

if err != nil {
util.CheckError(err)
return
}

err = video.GetVideo(videoID)
util.CheckError(err)

output, err = json.MarshalIndent(&video, "", "\t")
util.CheckError(err)

writer.Header().Set("Content-Type", "application/json")
writer.Write(output)

return
}

This method seems long but in the first half of the function, it is checking if the URL ends with /video or it ends with a number. If it ends with /video, that means it is not asking for a specific video but it’s asking for all valid videos. Thus video.GetAllVideos() is called and the videos are returned as JSON array.

So what if it’s only requesting for particular video with video ID? There is where the second half of the function comes in. It will first convert the part of the URL into an integer using strconv.Atoi(). Then we will retrieve the video based on that integer as video ID and then return it as a JSON object.

HTTP POST

To create a new video record in database, the system will call the handleVideoAPIPost().

func handleVideoAPIPost(writer http.ResponseWriter, request *http.Request, video models.IVideo) (err error) {

length := request.ContentLength
body := make([]byte, length)
request.Body.Read(body)
json.Unmarshal(body, &video)

err = video.CreateVideo()
if err != nil {
util.CheckError(err)
apiStatus := models.APIStatus{
Status: false,
Message: err.Error(),
}

output, err := json.MarshalIndent(&apiStatus, "", "\t")
util.CheckError(err)

writer.WriteHeader(400)
writer.Header().Set("Content-Type", "application/json")

writer.Write(output)
} else {
apiStatus := models.APIStatus{
Status: true,
Message: "A video is successfully added to the database.",
}

output, err := json.MarshalIndent(&apiStatus, "", "\t")
util.CheckError(err)

writer.WriteHeader(200)
writer.Header().Set("Content-Type", "application/json")
writer.Write(output)
}

return
}

The beginning of the function for POST method is reading from body is because we will post a JSON object containing the new video record information to it. So it needs to retrieve the JSON object from the body.

Another interesting thing in this function is that it will return JSON object indicating whether the action of adding new record is successful or not with a corresponding HTTP status code of 400 or 200. Doing so is to let the frontend feedback to the user so that the user knows whether the record is successfully inserted to the database or not.

HTTP PUT

How about if we want to update existing video record? Well, we can rely on the PUT method. The PUT method requires us to tell it which resource it will update. Hence the function handleVideoAPIPut() is as follows.

func handleVideoAPIPut(writer http.ResponseWriter, request *http.Request, video models.IVideo) (err error) {

videoIDURL := path.Base(request.URL.Path)
videoID, err := strconv.Atoi(videoIDURL)
if err != nil {
util.CheckError(err)
return
}

err = video.GetVideo(videoID)
if err != nil {
util.CheckError(err)
apiStatus := models.APIStatus{
Status: false,
Message: err.Error(),
}

output, err := json.MarshalIndent(&apiStatus, "", "\t")
util.CheckError(err)

writer.WriteHeader(400)
writer.Header().Set("Content-Type", "application/json")
writer.Write(output)
}

length := request.ContentLength
body := make([]byte, length)
request.Body.Read(body)
json.Unmarshal(body, &video)

err = video.UpdateVideo()
if err != nil {
util.CheckError(err)
apiStatus := models.APIStatus{
Status: false,
Message: err.Error(),
}

output, err := json.MarshalIndent(&apiStatus, "", "\t")
util.CheckError(err)

writer.WriteHeader(400)
writer.Header().Set("Content-Type", "application/json")
writer.Write(output)
} else {
apiStatus := models.APIStatus{
Status: true,
Message: "A video record is successfully updated.",
}

output, err := json.MarshalIndent(&apiStatus, "", "\t")
util.CheckError(err)

writer.WriteHeader(200)
writer.Header().Set("Content-Type", "application/json")
writer.Write(output)
}

return
}

Similar to how we have done in handleVideoAPIGet(), we first need to get the video ID from the URL with the help of strconv.Atoi(). Then we will check whether there is an existing video in the database with the video ID. If there is none, then we simply return JSON object updating frontend with an error message. If there is video found with the video ID, we will then proceed to update it with the info from the JSON object passed via the request body.

There is one thing to take note here is that we are not replacing the existing video record entirely. We are only updating part of it. So the JSON object should contain only the fields needed to be updated.

Updating the video record.

HTTP DELETE

The function to handle DELETE method will be similar to the one handling PUT method.

func handleVideoAPIDelete(writer http.ResponseWriter, request *http.Request, video models.IVideo) (err error) {

   videoIDURL := path.Base(request.URL.Path)
    videoID, err := strconv.Atoi(videoIDURL)

    if err != nil {
        util.CheckError(err)
        return
    }

   err = video.GetVideo(videoID)
   if err != nil {
        util.CheckError(err)

        apiStatus := models.APIStatus{
            Status: false,
            Message: err.Error(),
        }
        output, err := json.MarshalIndent(&apiStatus, "", "\t")
        util.CheckError(err)

        writer.WriteHeader(400)
        writer.Header().Set("Content-Type", "application/json")
        writer.Write(output)

    }

    err = video.DeleteVideo()
    if err != nil {

        util.CheckError(err)

        apiStatus := models.APIStatus{
            Status: false,
            Message: err.Error(),
        }
        output, err := json.MarshalIndent(&apiStatus, "", "\t")
        util.CheckError(err)

        writer.WriteHeader(400)
        writer.Header().Set("Content-Type", "application/json")
        writer.Write(output)

    } else {

        apiStatus := models.APIStatus{
            Status: true,
            Message: "A video record is deleted.",
        }
        output, err := json.MarshalIndent(&apiStatus, "", "\t")
        util.CheckError(err)

        writer.WriteHeader(200)
        writer.Header().Set("Content-Type", "application/json")
        writer.Write(output)

    }

    return
}

Similar to how we have done in handleVideoAPIPut(), we first need to get the video ID from the URL with the help of strconv.Atoi(). Then we will check whether there is an existing video in the database with the video ID. If there is none, then we simply return JSON object updating frontend with an error message. If there is video found with the video ID, we will then proceed to delete it.

The popup to check if user really wants the video record to be removed from the list.

Frontend with VueJS

Now, let’s see how we use VueJS library to display the video list.

It is done with just a for loop to list down all the relevant videos and having values stored in data attributes for update and delete the video record.

References

Monitoring Golang Web App with Application Insights

Continue from the previous topic

Application Insights is available on Azure Portal as a way for developers to monitor their live web applications and to detect performance anomalies. It has a dashboard with charts to help developers diagnose issues and understand user behaviors on the applications. It works for apps written on multiple programming languages other than .NET too.

Setup of Application Insights on Azure

It is straightforward to setup Application Insights on Azure Portal. If we have already setup a default CI/CD for simple Golang web app, an Application Insights account will be created automatically.

Creating new Application Insights account. The golab002 is automatically created when we setup a new Golang DevOps project on Azure Portal.

Once the Application Insights account is created, we need to get its Instrument Key which is required before any telemetry can be sent via the SDK.

Simplicity in ASP .NET Core

In ASP .NET Core projects, we can easily include Application Insights by including the Nuget package Microsoft.ApplicationInsights.AspNetCore and adding the following highlighted code in Program.cs.

public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
WebHost.CreateDefaultBuilder(args)
.UseStartup()
.UseApplicationInsights();

Setup of Application Insights Telemetry in Golang

So, what if we want to monitor our Golang applications which are hosted on Azure App Service? Luckily, Microsoft officially published an Application Insights SDK for Golang which is also open sourced on GitHub.

Since June 2015, Luke Kim, the Principal Group Software Engineering Manger at Microsoft, and other Microsoft engineers have been working on this open source project.

Introducing Application Insights to Golang application is not as straightforward as doing the same in ASP .NET Core project described above. Here, I will cover only how to use Telemetry.

First of all, we need to download and install the relevant package with the following go get command.

go get github.com/Microsoft/ApplicationInsights-Go/appinsights

Tracing Errors

Previously, we already have a centralized checkError function to handle errors returned from different sources in our code. So, we will have the following code added in the function to send traces back to Application Insights when an error occurs.

func checkError(err error) {
    if err != nil {
        client := appinsights.NewTelemetryClient(os.Getenv("APPINSIGHTS_INSTRUMENTATIONKEY"))

        trace := appinsights.NewTraceTelemetry(err.Error(), appinsights.Error)
        trace.Timestamp = time.Now()

        client.Track(trace)

        panic(err)    
}
}

So, when there is an error on our application, we will receive a trace record as such on the Metrics of Application Insights as shown below.

An error is captured. In this case, it’s because wrong DB host is stated.

However, doing this way doesn’t give us details such as call stack. Hence, if we want to log an exception in our application, we need to use TrackPanic in the SDK as follows.

func checkError(err error) {
    if err != nil {
        client := appinsights.NewTelemetryClient(os.Getenv("APPINSIGHTS_INSTRUMENTATIONKEY"))

        trace := appinsights.NewTraceTelemetry(err.Error(), appinsights.Error)
        trace.Timestamp = time.Now()

        client.Track(trace)

        // false indicates that we should have this handle the panic, and
        // not re-throw it.
        defer appinsights.TrackPanic(client, false)

        panic(err)    
}
}

This will capture and report call stack of the panic and display it on Azure Portal. With this, we can easily see which exceptions are occurring and how often.

Traces and exceptions. The details of exception includes call stack.

Tracing Page Views

Besides errors, let’s capture the page views as well so that we can easily tell which handler function is called and how much time is spent in it. To do so, we introduce a new function called handleRequestWithLog.

func handleRequestWithLog(h func(http.ResponseWriter, *http.Request)) http.HandlerFunc {

    return http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {

        startTime := time.Now()
        h(writer, request)
        duration := time.Now().Sub(startTime)

        client := appinsights.NewTelemetryClient(
os.Getenv("APPINSIGHTS_INSTRUMENTATIONKEY"))

        trace := appinsights.NewRequestTelemetry(
request.Method, request.URL.Path, duration, "200")
        trace.Timestamp = time.Now()

        client.Track(trace)
    })
}

Then we can modify our server.go to be as follows.

mux.HandleFunc("/", handleRequestWithLog(index))
mux.HandleFunc("/addVideo", handleRequestWithLog(addVideo))
mux.HandleFunc("/updateVideo", handleRequestWithLog(updateVideo))
mux.HandleFunc("/deleteVideo", handleRequestWithLog(deleteVideo))

Now whenever we visit a page or perform an action, the behaviour will be logged on Application Insights, as shown in the following screenshot. As you can see, the server response time is logged too.

Adding new video, deleting video, and viewing homepage actions.

With these records, the Performance chart in Application Insights will be plotted too.

Monitoring the performance of our Golang web application.

Tracing Static File Downloads

Besides the web pages, we are also interested at static files, such as understanding how fast the server responses when the static file is retrieved.

To do so, we first need to introduce a new handler function called staticFile.go.

package main

import (
    "mime"
    "net/http"
    "strings"
)

func staticFile(writer http.ResponseWriter, request *http.Request) {
    urlComponents := strings.Split(request.URL.Path, "/")

    http.ServeFile(
writer, request, "public/"+urlComponents[len(urlComponents)-1])

    fileComponents := strings.Split(
urlComponents[len(urlComponents)-1], ".")
    fileExtension := fileComponents[len(fileComponents)-1]

    writer.Header().Set(
"Content-Type", mime.TypeByExtension(fileExtension))
}

The reason why we need do as such is because we want to apply the handleRequestWithLog function for static files in server.go too.

mux.HandleFunc("/static/", handleRequestWithLog(staticFile))

By doing so, we will start to see the following on Search of Application Insights.

A list of downloaded CSS and JS static files and their corresponding server response time.

Conclusion

In ASP .NET Core applications, we normally need add the UseApplicationInsights as shown below in Program.cs then all the server actions will be automatically traced. However, this is not the case for Golang applications where there is no such convenience.

References

  1. What is Application Insights;
  2. Exploring Metrics in Application Insights;
  3. In Golang, how to convert an error to a string?
  4. [Stack Overflow] How to get URL in http.Request?
  5. [Stack Overflow] How to get request string and method?
  6. [Stack Overflow] Golang http handler – time taken for request;
  7. [golang.org] func Split;
  8. Find the Length of an Array/Slice;
  9. [GitHub] Microsoft Application Insights SDK for Go;
  10. Golang 1.6: 使用mime.TypeByExtension来设置Content-Type;
  11. [Stack Overflow] What to use? http.ServeFile(..) or http.FileServer(..)?
  12. [Stack Overflow] How do you serve a static html file using a go web server?

Deploy Golang App to Azure Web Apps with CI/CD on DevOps

Continue from the previous topic

After we have our code on Github repository, now it’s time to automate our builds and deployments so that our Golang application will always be updated whenever there is a new change to our code on Github.

Sample Golang Web App DevOps Pipelines

To do that, we will use Azure DevOps and its Pipelines module. We can easily create a DevOps project in Azure Portal for our Golang application because there is a template available.

Golang is one of the supported languages in Azure DevOps.

As a start, we will focus on “Windows Web App” instead of containers. After that, we just need to configure basic information of the web app, such as its name, location, resource group, pricing tier, and application insights.

We can configure Application Insights while creating the DevOps project.

After that, we shall be able to see a new DevOps project created with the following two folders, Application and ArmTemplates, in Repos. Application folder contains a sample Golang application.

However, why is there an ArmTemplates folder? This is because by default when we create a new Azure DevOps project for Golang application using the steps above, it will also automatically create a web app for us. Hence, this is the ARM (Azure Resource Manager) template Azure uses to do that.

Content of ArtTemplate which is used to create/update the Azure web app.

With this pipeline setup, we can simply update the default Golang code in the Repos to launch our Golang application on Azure. However, what if we want to link Azure DevOps with the codes we already have on our Github repo?

Connecting DevOps with Github

To do that, let’s start again by creating a new project on Azure DevOps, instead of Azure portal. Here, I will make the DevOps project to be Public so that you can access it while reading this article.

Creating a new public DevOps project.

Once the project is created, we can proceed to the Project Settings page of the project to disable some modules that we don’t need, i.e. Boards and Repos.

We need to hide both Boards and Repos because Github provides us similar features.

Setting up Build Pipeline

After this, we then can proceed to create our Build pipeline by first a connecting to our Github repo.

If our code is neither on DevOps or Github, we can click “Use the visual designer” to proceed.

Before continuing to choose the corresponding Github repo, we need to have a azure-pipelines.yml. To understand the guidelines to write proper Azure DevOps Pipelines YAML, we can refer to the official guide. For Golang, there is another specific documentation on how to build and test Golang projects with Azure DevOps Pipelines.

For our case, we will have the following pipeline YAML file.

# Go 
# Build your Go project.

resources:
- repo: self

pool:
vmImage: 'vs2017-win2016'

steps:
- task: GoTool@0
inputs:
version: 1.11.5
displayName: 'Use Go 1.11.5'
- task: Go@0
displayName: 'go get'
inputs:
arguments: '-d'
workingDirectory: '$(System.DefaultWorkingDirectory)'
- task: Go@0
displayName: 'go build'
inputs:
command: build
arguments: '-o "$(System.TeamProject).exe"'
workingDirectory: '$(System.DefaultWorkingDirectory)'
- task: ArchiveFiles@2
displayName: 'Archive Files'
inputs:
rootFolderOrFile: '$(Build.Repository.LocalPath)'
includeRootFolder: False
- task: PublishBuildArtifacts@1
displayName: 'Publish Artifact'
inputs:
artifactName: drop

There are a few virtual machine images from Microsoft-hosted agent pool. We choose the “Visual Studio 2017 on Windows Server 2016 (vs2017-win2016)” image because I normally use Visual Studio 2017 for development.

The first task is the Go Tool Installer task. It will find and download a specific version of the Go tool into the tool cache and add it to the PATH. Here we will use the latest version of Golang which is 1.11.5 at the point of writing this article.

The subsequent step will be running go get. This command will download the packages along with their dependencies. Since the -d argument is present, it will only download them but not install them.

After that, it will run go build. This step compiles the packages along with their dependencies, but it does not install the results. By default, the build command will write the resulting executable to an output file named after the first source file (or the source code directory). However, with the -o flag here, it forces build to write the resulting executable to the output file named $(System.TeamProject).exe, i.e. GoLab.exe.

Next we use the Archive Files task to create an archive file from a source folder. Finally, we use the Publish Build Artifacts task to publish build artifact to DevOps pipelines. With Archive Files task, it will generate a zip file called as such D:\a\1\a\54.zip where 54 is the build id. Publish Build Artifacts task will then upload the zip file to file container called drop.

Details of the Archive Files task.

To find out what is inside the file container drop, we can download it from the Summary page of the build. It is actually a folder containing all the files of our Golang application.

We can download the drop from the Summary page of the build.

Setting up Release Pipeline

Now we can proceed to create our Release pipeline. Luckily, there is already a template available to help us kick starting the Release pipeline.

The “Deploy a Go app to Azure App Service” pipeline is what we need here.

After selecting the template, we will need to specify the artifact, as shown below. There is version that we can choose, for example, the latest version from a specific branch with tags. Here we choose Latest so that our latest code change will always get deployed to Azure Web Apps.

Adding artifact.

Next, we need to enable the CD trigger as shown in the following screenshot so that a new release will be created every time a new build is available.

Enabling CD trigger.

Now we are at Pipeline tab. What we need to next is to move on to the Tasks tab, which is now having a red exclamation mark. We just need to authorize the Release pipeline to our Azure subscription and then connect it to the Azure Web App in the subscription.

Completing tasks.

Now, as you can see, the agent basically does three steps:

  • Stop the Azure Web App;
  • Deploy our code to Web App;
  • Start the Web App.

What interests us here is the second step. The reason why we need to generate a zip file in Build pipeline is also because in the second step, we need to specify the file path to the zip files to deploy.

Default configuration of second step.

Finally, we can just Save the pipeline and rename the “New release pipeline” to another friendlier name.

Now we can manually create a Release to test it out.

Create a new release manually.

Since we trigger this release manually, we also need to click in to deploy it manually.

Deploying to Azure App Service in progress.

After the deployment is done, we can view its summary as shown below.

The deployment process of the agent.

Conclusion

That’s all for setting up simple build and release pipelines on Azure DevOps to deploy our Golang web app to Azure Web Apps.