Javier E. Fajardo

Random musings from a software developer

Organizing my Lists of Lists with Go

Golang gopher

The Golang Gopher by Renee French (Licensed under CC Attrib 3.0)

 

When I started my undergrad, I remember there was a lot going on and it was just a constant stream of information coming in from all directions. From git basics to the Student Dos and Don'ts of the Google Summer of Code, there was always something to read up on or a new book to go through. I got back into anime about a year in, and there were so many new ones and ones I had never heard of before. I quickly realized that I had to do something about it or risk information overload. Even nowadays, after graduating, professional societies like the ACM and IEEE send you a thousand articles on research and industry findings. It's great to keep up-to-date with the industry, but I don't want to read them ALL the time.

 

So, early on my solution was to use the Pocket web app. But now I had a single long list, growing everyday. At least, I said to myself, I knew where to keep my items and I would read them during my downtime. I could give them some order by using Pocket's tag system. However, by the time I graduated, I had more than 1800 items saved in Pocket and, even within that near endless list of links, other lists had started forming. "The 21 weirdest anime", "five must read books for coders" and even the "top 100 best N64 games" became regular occurrences in my links. And I just kept adding to it. I wanted to go through all these at some point and consolidate a single list, but they were being kept in Pocket among scholarly articles and How-To Youtube videos. I had to go tidy-up my Pocket list or just lose those items forever.

 

Over the summer of 2017, I finally checked my monolithic list and managed to make some progress. There were a lot of dead links and irrelevant items. In three days, I cut the amount of URLs in half and recovered these inner lists of lists and saved them as text files locally. I now had lists of lists made up entirely of URLs, but no order to them. No way of sorting them to see what to consume next, or what might have been saved first or last. Not that it mattered, but keeping them in Pocket meant that I'd get to them in reverse chronological order, starting by the one I saved most recently.

 

Because I had a lot of free time on my hands (and I was, technically, on a long vacation), I decided to go through my anime list first. There was a heuristic that I wanted to try to sort them all by. One that favoured short, highly rated entries and left the lengthier ones for last. At the same time, I was itching to learn a new language. Sure, I could whip up a quick python script to scrape the net and evaluate my list, but I wanted to use the opportunity to pick up golang. So, in six hours, I made a small script using go to test out my concept and learn the basics of the language. Seeing that it worked well enough, I decided to make this a weekend project that would sort out all of my lists and manage them for me, away from Pocket.

The requirements for the project were simple:

  • Have some sort of persistence, most preferably in the form of a single file
  • Use simple commands to manage the lists (i.e. "push", "pop", "scan", "next")
  • Have cross-platform support


After more than a weekend (abut 11 days actually), the golang ordered list executive manager (GOLEM) was ready. I took the extra time in developing golem to try and understand some of the core features in golang as well as best practices and other recommendations when writing go. Most of the development was done in Windows using Visual Studio Code with the Go extension. Some third-party libraries were used to enable web scraping and ORM (more on this later) features, and I'm still surprised by how you can do "go get [insert package/library URL from github]". I've read some people saying that "go get" has a few problems but, at first glance, this seems fine. Perhaps a lot has change since then in the language or I just haven't had to maintain a big enough project for long enough. My only complaint was that the sqlite3 library in golang that my chosen ORM depended on assumed that "gcc" was available in the environment, even on Windows...

 

Other than very short hiccups, things went smoothly. Though the lack of "class" types in golang did catch me off guard at first, after some consideration and re-factoring, I got used to it and enjoyed the simplicity. No more class hierarchies with 26 members and nested inner classes as I've come to see (and have become guilty of doing) in Java.  It had been a while since I did C, so I was used to applying standard OOP for most things. I'm nowhere near an expert in golang, but I can see its place and its growth in the future.

If I had to sum up my first experience with golang, I would say:

Pros:

  • Fast to pick up, fast to build on: I know people who refer to golang as a rather boring language. There's nothing new to it and some refer to it as very standardized. I found this to be a good thing since that means you can get started easily and be productive in the language within hours;
  • Go docs and playground are great: Google has done a good job at supplying go with some good tooling, but having the standard docs and the go playground around are the best way to learn and quickly avoid rookie mistakes in your code;
  • Interpreter and compiler means the best of both worlds for developers: Having an interpreter when you're prototyping and a compiler when you're deploying/releasing is great. You're not bogged down by long compile times, but you also don't have to take the performance hit of an interpreter runtime when you don't want it. By the time I was done with the project, I issued a "go build" command and had a binary that worked without issue.

Cons:

  • Unusually slow interpreter on Linux x86_64: While developing on Linux, I noticed that "go run" would take an unusually long time to actually run my program. I'm not sure what could explain this issue and hopefully it was just a freak of nature. If it helps, I was using go version "go1.6.2 linux/amd64" on Xubuntu 16.04;
  • Throwback to C: programming in golang reminded me of writing C code, but with a garbage collector to watch your back and many security checks to ensure you're not doing something "dangerous". Although you have pointers, the next two points seem to be the language's way of warning programmers not to use them;
  • Pass-by-value default, even when you don't want to: This I found interesting coming from C/C++ where you can have pass-by-reference/pointer functions freely. Let me explain. Pass-by-value is what I would expect from any C-type language by default. However, while using interfaces it seemed that the behaviour is enforced by the language: changing a struct's interface methods to pass-by-pointer meant the struct no longer implements the interface[1]. This came as a big surprise when I realized it, which leads me to my final point;
  • Pointer and interface semantics get messy quickly: In golang, when there is an interface variable to a struct type, there are cases where it is unclear from looking at the code if the interface variable is holding a pointer or a value of said struct type. And this matters a lot of the time when you're doing reflection or using more "hidden" parts of the language. This caused WAY too many problems with the ORM I was using as well, which mostly expected pointer types to be passed in to some of its functions. Not being able to know whether an interface variable in a function was a pointer or not lead to many errors in accessing "unaddressable locations". For me, this is a pretty big issue if you actually like (or need) to use pointers.

 

This was just a first glance at golang through a very small project, but I liked the language and its uniqueness, as boring as some people might say it is. But most importantly, I now have my lists in order!

 

Now, time to get back to the next book on my list: "Game Engine Black Book: Wolfenstein 3D"!

 

 


[1] This is explained by the method sets of a struct type and its pointer equivalent. Method sets are a particularly interesting concept in golang and the name is self-explanatory: they're a set of methods associated to a type. The explanation on why the type and pointer to a type have different method sets can be found in golang's FAQ. Both from that question and this one, it seems the general suggestion is to avoid pointers in the language.