Category: Experience

I like to explore interesting new technologies. I also love to learn more from the materials available on Microsoft Virtual Academy, Google Developers channel, and several other tech/dev events.

Containerize Golang Code and Deploy to Azure Web App

by Chun Lin, posted in Docker, Experience, Go (Golang), Microservices and Containers

Continue from the previous topic…

Learning about containers is essentially a huge topic but for beginners, there needs to be something small to help them get started. Hence, in this article, we will focus only on the key concepts of containers and the steps to containerize the program and deploy it to Azure Web App.

In 2017, Azure introduced Web Apps for Containers. Before that, the Azure Web Apps actually ran on Windows VMs managed by Microsoft. So now with this new feature, we can build a custom Docker image containing all the binaries and files and then run a Docker container based on the image on Azure Web Apps. Hence, we can now bring our own Docker container images supporting Golang to Azure with its PaaS option.

As explained in the book “How to Containerize Your Go Code“, containers isolates an application so that container thinks it’s running on its own private machine. So, a container is similar to a VM but it uses the OS kernel on the host rather than having its own.

Firstly, we need to prepare the Dockerfile, a file having the instructions telling the Docker how to build the images automatically. So, a Dockerfile is simply a text file containing all the commands a user could call on the command line to assemble an image. Traditionally, the Dockerfile is named Dockerfile and located in the root of the context.

The Dockerfile we have for our project is as follows.

FROM scratch

EXPOSE 80

COPY GoLab /
COPY public public
COPY templates templates

ENV APPINSIGHTS_INSTRUMENTATIONKEY '' \
    CONNECTION_STRING '' \
    OAUTH_CLIENT_ID '' \
    OAUTH_CLIENT_SECRET '' \
    COOKIE_STORE_SECRET '' \
    OAUTH2_CALLBACK ''

CMD [ "/GoLab" ]

The Dockerfile starts with a FROM command that specifies the starting point for the image to build. For our project, we don’t have any dependencies, so we can start from scratch. So what is scratch? Scratch is basically a special Docker image that is empty (0B). That means there will be nothing else in our container later aside from what we put in with the rest of the Dockerfile.

The reason why we build from scratch is because not only we can have a smaller image to build later, but also our container will have smaller attack surface. This is because the less code there is within our container, the less likely it is to include a vulnerability.

The EXPOSE 80 command is telling Docker that we need to open the port 80 because the web server is listening on port 80. Hence, in order to access our program from outside the container through HTTP, we need to define it in the Dockerfile that we need the port 80 to be always opened.

The next three COPY commands are basically copying firstly the GoLab executable into the root directory of the container and secondly the two directories, public and templates into the container. Without the HTML, CSS, and JavaScript, our web app will not work.

Now you may wonder why the first COPY command says GoLab instead of GoLab.exe. We shall discuss it later in this article.

After that, we use ENV command to set the environment variables that we will be using in the app.

Finally we have the line CMD [“/GoLab”] to directs the container as to which command to execute when the container is run.

Since the container is not a Windows container, the code that runs inside the container thus needs to be a Linux binary. Fortunately, this is really simple to obtain with the cross-compilation support in Go using the following command.

$ $env:GOOS = "linux"
$ go build -o GoLab .

Thus, in the Dockerfile, we use GoLab file instead of GoLab.exe.

We can now proceed to build the container image with the following command (Take note of the dot in the end of line).

$ docker image build -t chunlindocker/golab:v1 .

The -t flag is for us to specify the name and tag of the container. In this case, I call it chunlindocker/golab:v1 where chunlindocker is the Docker ID of my Docker Hub. Naming in such a way later helps me to push it to a registry, i.e. the Docker Hub.

My Docker Hub profile.

If we want to build the image with another dockerfile, for example Dockerfile.development, we can do it as follows.

$ docker image build -t chunlindocker/golab:v1 -f Dockerfile.development .

Once the docker image is built, we can see it listen when we perform the list command as shown in the screenshot below.

Created docker images.

Now the container image is “deployable”. That means we can run it anywhere with a running docker engine. Since our laptop has Docker installed, so we can proceed to run it locally with the following command.

$ docker container run -P chunlindocker/golab:v1

If you run the command above in the Terminal window inside VS Code, you will see that the command line is “stuck”. This is because the container is already running on local machine. So what we need to do is just open another terminal window and view all the running containers.

The docker ps command by default only shows running containers.

To help humans, Docker auto generates a random name with two words and assigns it to the container. We can see that the container we created is given a random name “nifty_elgama”, lol. So now our container has a “human” name to call. If you want to remove the container later, you not only need to Ctrl+C to stop it, but to totally remove it, you need to use the rm command as follows.

$ docker container rm nifty_elgama

The PORTS column shown in the screenshot is important because it tells us how ports exposed on the container can be accessed from the host. So to test it locally, we shall visit http://localhost:32768.

So our next step is to upload it to a container registry so that later it can be pulled onto any machines, including Azure Web Apps, that will run it. To do so, we do push the image we built above to Docker Hub with the following command.

$ docker push chunlindocker/golab:v1
Successfully push our new container image to Docker Hub.

So, now how do we deploy the container to Azure?

Firstly, we need to create a Web App for Containers on the Azure Portal, as shown in the screenshot below.

Creating Web App for Containers.

The last item in the configuration is the “Configure Container”. Clicking on that, we will be brought to the following screen where we can then specify the container image we want to use and pull it from Docker Hub.

We will be deploying single container called chunlindocker/golab:v4 from Docker Hub.

You can of course deploy a private container from Docker Hub by choosing “Private” as Repository Access. Then Azure Portal will prompt you for Docker Hub login credential for it to pull image from Docker Hub.

Once the App Service is created, we can proceed to read the Logs under “Container Settings”. Then we can see the container initializing process.

Logs about the container in App Service.

After that we can proceed to fill up the Application Settings with the environment variables we have in the web application and then we are good to go.

The website is up and running on Azure Web App for Containers.

References

Unit Testing with Golang

by Chun Lin, posted in Cloud Computing: Microsoft Azure, DevOps, Experience, Go (Golang)

Continue from the previous topic…

Unit Testing is a level of automated software testing that units which can be modular parts of the program are tested. Normally, the “unit” refers to a function, but it doesn’t necessary always be so. A unit typically takes in data and returns an output. Correspondingly, a unit test case passes data into the unit and check the resultant output to see if they meet the expectations.

Unit Testing Files

In Golang, unit test cases are written in <module>_test.go files, grouped according to their functionality. In our case, when we do unit testing for the videos web services, we will have the unit test cases written in video_test.go. Also, the test files need to be in the same package as tested functions.

Necessary Packages

In the beginning, we need to import the “testing” package. In each of our unit test function, we will take in a parameter t which is a pointer to testing.T struct. It is the main struct that we will be using to call out any failure or error.

In our code video_test.go, we use only the function Error in testing.T to log the errors and to mark the test function fails. In fact, Error function is a convenient function in the package that combines calling of Log function and then the Fail function. The Fail function marks the test case has failed but it still allows the execution of the rest of the test case. There is another similar function called FailNow. The FailNow function is stricter and exits the test case once it’s encountered. So, if FailNow function is what you need, you have to call the Fatal function which is another convenient function that combines Log and FailNow instead of the Error function.

Besides the “testing” package, there is another package that we need in order to do unit testing for Golang web applications. It is the “net/http/httptest” package. It allows us to use the client functions of the “net/http” package to send an HTTP request and capturing the HTTP response.

Test Doubles, Mock, and Dependency Injection

Before proceeding to writing unit test functions, we need to get ready with Test Doubles. Test Double is a generic term for any case where we replace a production object for testing purposes. There are several different types of Test Double, of which a Mock is one. Using Test Doubles helps making the unit test cases more independent.

In video_test.go, we apply the Dependency Injection in the design of Test Doubles. Dependency Injection is a design pattern that decouples the layer dependencies in our program. This is done through passing a dependency to the called object, structure, or function. This dependency is used to perform the action instead of the object, structure, or function.

Currently, the handleVideoRequests handler function uses a global sql.DB struct to open a database connection to our PostgreSQL database to perform the CRUD. For unit testing, we should not depend on database connection so much and thus the dependency on sql.DB should be removed. The dependency on sql.DB then should be injected into the process flow from the main program.

To do so, firstly, we need to introduce a new interface called IVideo.

type IVideo interface {

    GetVideo(userID string, id int) (err error)
    GetAllVideos(userID string) (videos []Video, err error)
    CreateVideo(userID string) (err error)
    UpdateVideo(userID string) (err error)
    DeleteVideo() (err error)

}

Secondly, we make our Video struct to implement the new interface and let one of the fields in the Video struct a pointer to sql.DB. Unlike in C#, we have to specify which interface the class is implementing, in Golang, as long as the Video struct implements all the methods that IVideo has (which is already does), then Video struct is implementing the IVideo interface. So now our Video struct looks as following.

type Video struct {
    Db             *sql.DB
    ID             int    `json:"id"`
    Name           string `json:"videoTitle"`
    URL            string `json:"url"`
    YoutubeVideoID string `json:"youtubeVideoId"`
}

As you can see, we added a new field called Db which is a pointer to sql.DB.

Now, we can create a Test Double called FakeVideo which implements IVideo interface to be used in unit testing.

// FakeVideo is a record of favourite video for unit test
type FakeVideo struct {
    ID             int    `json:"id"`
    Name           string `json:"videoTitle"`
    URL            string `json:"url"`
    YoutubeVideoID string `json:"youtubeVideoId"`
    CreatedBy      string `json:"createdBy"`
}


// GetVideo returns one single video record based on id
func (video *FakeVideo) GetVideo(userID string, id int) (err error) {
    jsonFile, err := os.Open("testdata/fake_videos.json")
    if err != nil {
        return
    }

    defer jsonFile.Close()

    jsonData, err := ioutil.ReadAll(jsonFile)
    if err != nil {
        return
    }

    var fakeVideos []FakeVideo
    json.Unmarshal(jsonData, &fakeVideos)

    for _, fakeVideo := range fakeVideos {
        if fakeVideo.ID == id && fakeVideo.CreatedBy == userID {
            video.ID = fakeVideo.ID
            video.Name = fakeVideo.Name
            video.URL = fakeVideo.URL
            video.YoutubeVideoID = fakeVideo.YoutubeVideoID
            
            return
        }
    }
    
    err = errors.New("no corresponding video found")

    return
}
...

So instead of reading the info from the PostgreSQL database, we read mock data from a JSON file which is stored in testdata folder. The testdata folder is a special folder where Golang will ignores when it builds the project. Hence, with this folder, we can easily read our test data from JSON file fake_videos.json through relative path from video_test.go.

Since now the Video struct is updated, we need to update our handleVideoAPIRequests method to be as follows.

func handleVideoAPIRequests(video models.IVideo) http.HandlerFunc {
    return func(writer http.ResponseWriter, request *http.Request) {
        var err error

        ...

        switch request.Method {
        case "GET":
            err = handleVideoAPIGet(writer, request, video, user)
        case "POST":
            err = handleVideoAPIPost(writer, request, video, user)
        case "PUT":
            err = handleVideoAPIPut(writer, request, video, user)
        case "DELETE":
            err = handleVideoAPIDelete(writer, request, video, user)
        }

        if err != nil {
            util.CheckError(err)
            return
        }
    }
}

So now we pass an instance of the Video struct directly into the handleVideoAPIRequests. The various Video methods will use the sql.DB that is a field in the struct instead. At this point of time, handleVideoAPIRequests no longer follows the ServeHTTP method signature and is no longer a handler function.

Thus, in the main function, instead of attaching a handler function to the URL, we call the handleVideoAPIRequests function as follows.

func main() {
    ...
     
    mux.HandleFunc("/api/video/",
        handleRequestWithLog(handleVideoAPIRequests(&models.Video{Db: db}))) 

    ...
}

Writing Unit Test Cases for Web Services

Now we are good to write unit test cases in video_test.go. Instead of passing a Video struct like in server.go, this time we pass in the FakeVideo struct, as highlighted in one of the test cases below.

func TestHandleGetAllVideos(t *testing.T) {
    mux = http.NewServeMux()
    mux.HandleFunc("/api/video/", handleVideoAPIRequests(&models.FakeVideo{}))
    writer = httptest.NewRecorder()

    request, _ := http.NewRequest("GET", "/api/video/", nil)
    mux.ServeHTTP(writer, request)

    if writer.Code != 200 {
        t.Errorf("Response code is %v", writer.Code)
    }

    var videos []models.Video
    json.Unmarshal(writer.Body.Bytes(), &videos)

    if len(videos) != 2 {
        t.Errorf("The list of videos is retrieved wrongly")
    }
}

By doing this, instead of fetching videos from the PostgreSQL database, now it will get from the fake_videos.json in testdata.

Testing with Mock User Info

Now, since we have implemented user authentication, how do we make it works in unit testing also. To do so, in auth.go, we introduce a flag called isTesting which is false as follows.

// This flag is for the use of unit testing to do fake login
var isTesting bool

Then in the TestMain function, which is provided in testing package to do setup or teardown, we will set this to be true.

So how do we use this information? In auth.go, there is this function profileFromSession which retrieves the Google user information stored in the session. For unit testing, we won’t have this kind of user information. Hence, we need to mock this data too as shown below.

if isTesting {
        return &Profile{
            ID:          "154226945598527500122",
            DisplayName: "Chun Lin",
            ImageURL:    "https://avatars1.githubusercontent.com/u/8535306?s=460&v=4",
        }
    }

With this, then we can test whether the functions, for example, are retrieving correct videos of the specified user.

Running Unit Test Locally and on Azure DevOps

Finally, to run the test cases, we simply use the command below.

go test -v

Alternatively, Visual Studio Code allows us to run specified test case by clicking on the “Run Test” link above the test case.

Running test on VS Code.

We can then continue to add the testing as one of the steps in Azure DevOps Build pipeline, as shown below.

Added the go test task in Azure DevOps Build pipeline.

By doing this, if any of the test cases fails, there won’t be a build made and thus our system becomes more stable now.

RESTful Web Service in Golang and Front-end Served with VueJS

by Chun Lin, posted in Experience, Go (Golang), VueJS

Continue from the previous topic…

There is one behaviour in our Golang web application that will frustrate our users. Everytime we add, update, or delete a video record, the web page gets refreshed. So if at that time there is a video being played, then poof, it’s gone after you add, update, or delete a record. That’s a bad UX.

To overcome there, I decide to implement RESTful web services in the Golang project and the frontend will use VueJS library to update the web page.

Firstly, we need to wrap a web service interface over the CRUD functions we have in our web application. JSON will be used as the data transport format. To do that, we will introduce a new handler function to multiplex request to the correct function in our RESTful web service.

func handleVideoAPIRequests(video models.IVideo) http.HandlerFunc {
    return func(writer http.ResponseWriter, request *http.Request) {
    
        var err error
        switch request.Method {
        case "GET":
            err = handleVideoAPIGet(writer, request, video)

        case "POST":
            err = handleVideoAPIPost(writer, request, video)

        case "PUT":
            err = handleVideoAPIPut(writer, request, video)

        case "DELETE":
            err = handleVideoAPIDelete(writer, request, video)
        }

        if err != nil {
            util.CheckError(err)

            return
        }

    }
}

HTTP GET

Then for each of the HTTP methods, we will process the request independently. For example, to retrieve the list of videos or one of the videos, we will use GET method, i.e. handleVideoAPIGet().

The list of video records at the right are retrieved using HTTP GET.
func handleVideoAPIGet(writer http.ResponseWriter, request *http.Request, video models.IVideo) (err error) {

    videoIDURL := path.Base(request.URL.Path)

    var output []byte

    if videoIDURL == "video" {
        videos, errIf := video.GetAllVideos()
        err = errIf
        util.CheckError(errIf)

        output, errIf = json.MarshalIndent(&videos, "", "\t")
        err = errIf
        util.CheckError(errIf)

        writer.Header().Set("Content-Type", "application/json")
        writer.Write(output)

        return
    }

    videoID, err := strconv.Atoi(videoIDURL)

    if err != nil {
        util.CheckError(err)
        return
    }

    err = video.GetVideo(videoID)
    util.CheckError(err)

    output, err = json.MarshalIndent(&video, "", "\t")
    util.CheckError(err)

    writer.Header().Set("Content-Type", "application/json")
    writer.Write(output)
 
    return
}

This method seems long but in the first half of the function, it is checking if the URL ends with /video or it ends with a number. If it ends with /video, that means it is not asking for a specific video but it’s asking for all valid videos. Thus video.GetAllVideos() is called and the videos are returned as JSON array.

So what if it’s only requesting for particular video with video ID? There is where the second half of the function comes in. It will first convert the part of the URL into an integer using strconv.Atoi(). Then we will retrieve the video based on that integer as video ID and then return it as a JSON object.

HTTP POST

To create a new video record in database, the system will call the handleVideoAPIPost().

func handleVideoAPIPost(writer http.ResponseWriter, request *http.Request, video models.IVideo) (err error) {

    length := request.ContentLength
    body := make([]byte, length)
    request.Body.Read(body)
    json.Unmarshal(body, &video)

    err = video.CreateVideo()
    if err != nil {
        util.CheckError(err)
        apiStatus := models.APIStatus{
            Status:  false,
            Message: err.Error(),
        }

        output, err := json.MarshalIndent(&apiStatus, "", "\t")
        util.CheckError(err)

        writer.WriteHeader(400)
        writer.Header().Set("Content-Type", "application/json")

        writer.Write(output)
    } else {
        apiStatus := models.APIStatus{
            Status:  true,
            Message: "A video is successfully added to the database.",
        }

        output, err := json.MarshalIndent(&apiStatus, "", "\t")
        util.CheckError(err)

        writer.WriteHeader(200)
        writer.Header().Set("Content-Type", "application/json")
        writer.Write(output)
    }

    return
}

The beginning of the function for POST method is reading from body is because we will post a JSON object containing the new video record information to it. So it needs to retrieve the JSON object from the body.

Another interesting thing in this function is that it will return JSON object indicating whether the action of adding new record is successful or not with a corresponding HTTP status code of 400 or 200. Doing so is to let the frontend feedback to the user so that the user knows whether the record is successfully inserted to the database or not.

HTTP PUT

How about if we want to update existing video record? Well, we can rely on the PUT method. The PUT method requires us to tell it which resource it will update. Hence the function handleVideoAPIPut() is as follows.

func handleVideoAPIPut(writer http.ResponseWriter, request *http.Request, video models.IVideo) (err error) {

    videoIDURL := path.Base(request.URL.Path)
    videoID, err := strconv.Atoi(videoIDURL)
    if err != nil {
        util.CheckError(err)
        return
    }

    err = video.GetVideo(videoID)
    if err != nil {
        util.CheckError(err)
        apiStatus := models.APIStatus{
            Status:  false,
            Message: err.Error(),
        }

        output, err := json.MarshalIndent(&apiStatus, "", "\t")
        util.CheckError(err)

        writer.WriteHeader(400)
        writer.Header().Set("Content-Type", "application/json")
        writer.Write(output)
    }

    length := request.ContentLength
    body := make([]byte, length)
    request.Body.Read(body)
    json.Unmarshal(body, &video)

    err = video.UpdateVideo()
    if err != nil {
        util.CheckError(err)
        apiStatus := models.APIStatus{
            Status:  false,
            Message: err.Error(),
        }

        output, err := json.MarshalIndent(&apiStatus, "", "\t")
        util.CheckError(err)
        
        writer.WriteHeader(400)
        writer.Header().Set("Content-Type", "application/json")
        writer.Write(output)
    } else {
        apiStatus := models.APIStatus{
            Status:  true,
            Message: "A video record is successfully updated.",
        }

        output, err := json.MarshalIndent(&apiStatus, "", "\t")
        util.CheckError(err)

        writer.WriteHeader(200)
        writer.Header().Set("Content-Type", "application/json")
        writer.Write(output)
    }

    return
}

Similar to how we have done in handleVideoAPIGet(), we first need to get the video ID from the URL with the help of strconv.Atoi(). Then we will check whether there is an existing video in the database with the video ID. If there is none, then we simply return JSON object updating frontend with an error message. If there is video found with the video ID, we will then proceed to update it with the info from the JSON object passed via the request body.

There is one thing to take note here is that we are not replacing the existing video record entirely. We are only updating part of it. So the JSON object should contain only the fields needed to be updated.

Updating the video record.

HTTP DELETE

The function to handle DELETE method will be similar to the one handling PUT method.

func handleVideoAPIDelete(writer http.ResponseWriter, request *http.Request, video models.IVideo) (err error) {

    videoIDURL := path.Base(request.URL.Path)
    videoID, err := strconv.Atoi(videoIDURL)

    if err != nil {
        util.CheckError(err)
        return
    }

    err = video.GetVideo(videoID)
    if err != nil {
        util.CheckError(err)

        apiStatus := models.APIStatus{
            Status:  false,
            Message: err.Error(),
        }
        output, err := json.MarshalIndent(&apiStatus, "", "\t")
        util.CheckError(err)

        writer.WriteHeader(400)
        writer.Header().Set("Content-Type", "application/json")
        writer.Write(output)

    }

    err = video.DeleteVideo()
    if err != nil {

        util.CheckError(err)

        apiStatus := models.APIStatus{
            Status:  false,
            Message: err.Error(),
        }
        output, err := json.MarshalIndent(&apiStatus, "", "\t")
        util.CheckError(err)

        writer.WriteHeader(400)
        writer.Header().Set("Content-Type", "application/json")
        writer.Write(output)

    } else {

        apiStatus := models.APIStatus{
            Status:  true,
            Message: "A video record is deleted.",
        }
        output, err := json.MarshalIndent(&apiStatus, "", "\t")
        util.CheckError(err)

        writer.WriteHeader(200)
        writer.Header().Set("Content-Type", "application/json")
        writer.Write(output)

    }

    return
}

Similar to how we have done in handleVideoAPIPut(), we first need to get the video ID from the URL with the help of strconv.Atoi(). Then we will check whether there is an existing video in the database with the video ID. If there is none, then we simply return JSON object updating frontend with an error message. If there is video found with the video ID, we will then proceed to delete it.

The popup to check if user really wants the video record to be removed from the list.

Frontend with VueJS

Now, let’s see how we use VueJS library to display the video list.

It is done with just a for loop to list down all the relevant videos and having values stored in data attributes for update and delete the video record.

References