Introducing Pkger — Static File Embedding in Go

One of the reasons I fell in love with Go all those years ago was the idea that I could distribute a single binary that only contained my application, but also the supporting infrastructure needed to run that binary without any tooling installed. Coming from Ruby, a language where you primarily distribute an entire folder structure that requires runtime tooling, this was revolutionary for me, and immediately my mind buzzed with the possibilities.

Several years ago I was building an enterprise web product for a client. This project, as I’ve talked about before, eventually evolved in Buffalo. One of the requirements for Buffalo was to keep the promise of a single binary and this meant embedding static files in Go binaries.

As I searched around I found several tools that did just this. The one that caught my eye was go.Rice. What I liked most about it was that in would fall back to the file system during development meaning you didn’t have to keep re-embedding the files every time you made a change. Early versions of Buffalo shipped with go.Rice as its embedding tooling.

As early adopters started deploying the first Buffalo applications in production, bug reports started coming in with very issues regarding go.Rice. It appeared that, for a variety of reasons, we needed a new solution.

It was that Packr came to be. It adopted the go.Rice API for easy migration of tools. For the past few years Packr has played a large role in Buffalo’s increased popularity by helping to keep that one binary promise.

Unfortunately, Packr’s API and underlying architecture have proven to be both problematic and lossy. Important meta data such as mod time, mode, size, etc… are all lost when using Packr. This makes for a variety of bugs, but also limits what developers can do when using Packr instead of the standard library’s OS package to work with files. It was time to rethink the problem.

The Requirements

Before I thought of writing a line of code I considered the problems I had with Packr and how I could solve them. The solutions to these problems would become the requirements that I would need to work backward from.

The API

As I mentioned early Packr’s API was based on that of go.Rice, but that also meant losing important metadata since the API had no place for it. It was also not idiomatic. User’s have to learn a new API when using Packr. Consider the differences between the standard library filepath.Walk function and packr.Walk.

// standard library
filepath.Walk(".", func(path string, info os.FileInfo, err error) error
// Packr
box.Walk(func(path string, f packd.File) error

I wanted the API to feel like the standard library, and that meant adopting the standard library’s API.

Along with the benefits of an idiomatic API it also provides an immediate reference implementation.

Deterministic Pathing

There are many current, and many more closed, issues regarding Packr’s use of relative pathing and all of the fun ways that pathing can prove unreliable. The solution for this problem needed to provide for a static reference point regardless of where in the project the tools, or libraries, are being used.

The Go tooling has a built in solution to that problem that arrived with Modules.

$ go env GOMOD
$HOME/myapp/go.mod

The GOMOD tells you where the go.mod file is for a project, regardless of how deep you may be in that project. To use this would provide a true North for any tool to be able to find it’s way back to the root of the project.

Using this known root I decided to use an absolute Unix style path with / being the root of the project.

"./public"
"/public"

By removing the relative nature of the . We no longer have to jump through hoops to resolve that dot.

Testing, Dependencies, and More

Finally, I had a short list of other requirements.

• A test harness that could be used to test an implementation of the API against a known set of tests and against the standard library reference implementation. This would ensure that all implementations behave as expected.
• Little to no dependencies.
• Transparent tooling that allows for developers to understand what is to be packed, what declarations were found, etc…

Introducing Pkger

Today I would like to introduce Pkger, a tool for embedding static files into Go binaries that is idiomatic, lossless, and less complex.

The API and Interfaces

Pkger comprises of three core interfaces. The first interface, pkging.Pkger, comprises of functions from the os and path/filepath packages.

type Pkger interface {
    Parse(p string) (here.Path, error)
    Current() (here.Info, error)
    Info(p string) (here.Info, error)
    Create(name string) (File, error)
    MkdirAll(p string, perm os.FileMode) error
    Open(name string) (File, error)
    Stat(name string) (os.FileInfo, error)
    Walk(p string, wf filepath.WalkFunc) error
    Remove(name string) error
    RemoveAll(path string) error
}

The next interface is pkging.File. This interface is designed around the os.File type.

type File interface {
    Close() error
    Info() here.Info
    Name() string
    Open(name string) (http.File, error)
    Path() here.Path
    Read(p []byte) (int, error)
    Readdir(count int) ([]os.FileInfo, error)
    Seek(offset int64, whence int) (int64, error)
    Stat() (os.FileInfo, error)
    Write(b []byte) (int, error)
}

These two interfaces when combined with the os.FileInfo interface provide enough functionality to enable an almost identical experience when using pkger or the standard library.

func run() error {
 info, err := pkger.Stat("/go.mod")
 if err != nil {
  return err
 }
 fmt.Println(info)
 if err := pkger.MkdirAll("/foo/bar/baz", 0755); err != nil {
  return err
 }
 f, err := pkger.Create("/foo/bar/baz/biz.txt")
 if err != nil {
  return err
 }
 f.Write([]byte("BIZ!!"))
 if err := f.Close(); err != nil {
  return err
 }
 f, err = pkger.Open("/foo/bar/baz/biz.txt")
 if err != nil {
  return err
 }
 io.Copy(os.Stdout, f)
 return f.Close()
}

Module Aware Pathing

Pathing in Pkger uses GOMOD and other parts of the Go standard tool chain to determine the root of the Module, it’s import path, other important meta data that allows Pkger to accurately find any file inside of a Module.

Let’s use the following folder structure which we’ll place in $HOME/myapp.

.
├── go.mod
├── go.sum
├── main.go
└── public
    ├── images
    │   ├── img1.png
    │   └── img2.png
    └── index.html

And the following go.mod file.

module myapp
go 1.13
require github.com/markbates/pkger v0.10.0

To stat the index.html file using Pkger we can write a main.go such as the following.

package main
import (
 "fmt"
 "log"
 "github.com/markbates/pkger"
)
func main() {
 if err := run(); err != nil {
  log.Fatal(err)
 }
}
func run() error {
 info, err := pkger.Stat("/public/index.html")
 if err != nil {
  return err
 }
 fmt.Println("Name: ", info.Name())
 fmt.Println("Size: ", info.Size())
 fmt.Println("Mode: ", info.Mode())
 fmt.Println("ModTime: ", info.ModTime())
 return nil
}

The GOMOD for this application is $HOME/myapp Pkger joins that with /public/index.html to find the file on disk.

Name:  index.html
Size:  257
Mode:  -rw-r--r--
ModTime:  2019-11-01 16:12:24.783887046 -0400 EDT

Pkger uses static analysis and walks the AST of your application find certain declarations such as pkger.Open, pkger.Walk, and others. Because of this it can’t determine dynamic runtime values. I would love to see community help to improve, and expand, the parser.

Transparent Tooling

Tooling, especially tooling that provides a transparent view into the tool itself, is crucial to a tool like this.

Usage:
pkger [flags] [args...]
 -h prints help information ("false")
pkger info [flags] [args...]
pkger list [flags] [args...]
 -json prints in JSON format ("false")
pkger parse [flags] [args...]
 -h prints help information ("false")
pkger path [flags] [args...]
 -h prints help information ("false")
pkger serve [flags] [args...]
pkger stat [flags] [args...]

The top level function will generate a pkged.go containing only the files your application uses via Pkger static analysis tools. To find out what files those are the pkger listcommand prints a list of the files that pkger is aware of.

app
 > app:/public/index.html

To see what Pkger’s parser sees, the pkger parse command dumps a JSON representation of the results of parsing the current module.

{
 ".": [
  {
   "file": {
    "Abs": "$HOME/myapp/main.go",
    "Path": {
     "pkg": "myapp",
     "name": "/public/index.html"
    },
    "Here": {
     "Dir": "$HOME/myapp",
     "ImportPath": "myapp",
     "Module": {
      "path": "myapp",
      "main": true,
      "dir": "$HOME/myapp",
      "go_mod": "",
      "go_version": ""
     },
     "Name": "main"
    }
   },
   "pos": {
    "Filename": "$HOME/myapp/main.go",
    "Offset": 183,
    "Line": 17,
    "Column": 26
   },
   "type": "pkger.Stat",
   "value": "/public/index.html"
  }
 ]
}

Wrapping Up and the Future of Packr

I’m asking today for the community’s help in testing, opening PR’s, and feedback on this new tool. The more people who contribute to a project such as this help make it stronger and more stable for everyone else. Please watch the video, play with the examples, and join in helping make this the solution for static file embedding we need.

As for Packr and it’s future, it is unsure. I believe Pkger is the path forward, and barring any large surprises we find in the community, it will most certainly be the path forward for Buffalo. As the project progresses you can expect more posts and videos exploring Pkger.

It is my hope that those who still want to use Packr will help with its maintainence. If you are interested in becoming a maintainer of Packr please reach out.

Are you a company looking for Buffalo or Go training? The Gopher Guides want to help. We offer a variety of options, including In Person Training, Instructor Led Online Workshops, Consulting, and Support Contracts, and our self paced Online Video Learning.

Get your first 3 months of www.gopherguides.tv for 33% off! This offer is only good for a limited time, and a limited number, so act fast!

https://www.gopherguides.tv/?code=buffalo