Today, I discovered an excellent testing package by Roger Peppe: go-internal/testscript.
At work, I’ve been using Bats for testing cli stuff. All of our cli stuff is almost migrated from legacy Perl-scripts to Go (using the kong cmdline parser). Bats allows you to run cli programs and check the results. It does not care about the inner workings, as long as it outputs or generates the right results, the tests pass. But using a shell framework for testing Go stuff just does not feel performant. It’s a bit clumsy (demo later on). Also, its a PITA to setup up a CI pipeline.
What speaks for testscript is that it integrates seamlessly with go test. No setup, just include the package in your go test. Just like Bats, it does not limit you to a certain ecosystem. Go in this case.
Txtar
The test language has just a few commands and supports the txtar format. Txtar as in Text & Tar. For those who associate tar with the tar.gz or tgz file extension, you should know that tar stands for Tape Archive. It’s the file format designed for use with magnetic tapes. It’s not efficient to store more than a few files to it. It basically cats files and directories with their meta data together in one file.
Just like tar, txtar allows you to concatenate several text files together in one file. When running the tests, a test directory is created with the text files in place that you specified in the txtar file. The format is straightforward. All lines that follow the file separator -- filename -- are dumped in the specified file.
The txtar parser is forgiving. What could go wrong? Well nothing! According to the docs:
There are no possible syntax errors in a txtar archive.
Besides the go docs themselves, there’s just a few articles written about testscript. I found this one and this one. Reason for me to write this down. Let’s start with an obvious hello world:
Hello world!
In the testscript repo, there’s quite some testdata, among this hello world test as good starting point to learn something new:
# hello world
exec cat hello.txt
stdout 'hello world\n'
! stderr .
-- hello.text --
hello world
Let’s see what’s going on. This human readable tar file defines a hello.text that we symply miau to stdout. In exec tells to execute the cat command. The stdout and stderr lines do some additional testing. Standard output should match “hello world” followed by a newline. Standard error should not (!) match a single character. Both are regular expression matches.
Running the test!
To run the test completely stand-alone, install testscript first:
go install github.com/rogpeppe/go-internal/cmd/testscript@latest
Invoke testscript:
$ testscript hello.txt
# hello world (0.001s)
PASS
Let’s go!
It should be of no surprise that the test passes ;-) To see what’s going on, run it in -verbose mode, the output should be similar to:
WORK=$WORK
PATH=/home/bram/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
GOTRACEBACK=system
HOME=/no-home
TMPDIR=$WORK/.tmp
devnull=/dev/null
/=/
:=:
$=$
exe=
GOPATH=$WORK/.gopath
CCACHE_DISABLE=1
GOARCH=amd64
GOOS=linux
GOROOT=/usr/lib/go-1.22
GOCACHE=/home/bram/.cache/go-build
GOPROXY=https://proxy.golang.org,direct
goversion=1.22
# hello world (0.001s)
> exec cat hello.text
[stdout]
shello world
> stdout 'hello world\n'
> ! stderr .
# what cat (0.001s)
> exec which cat
[stdout]
/usr/bin/cat
PASS
WORK=$WORK
It shows the exact environment the test ran in.
So how to hook this up in your go tests?
The best resource I found is by abitrolly in a go-internal github issue. The examples in this post really help me to get started.
Basically, import testscript in your test and tell it where the scripts are located:
package main
import (
"testing"
"github.com/rogpeppe/go-internal/testscript"
)
func TestCLI(t *testing.T) {
testscript.Run(t, testscript.Params{
Dir: "testscripts",
})
}
To stick with our hello.txt test. Let’s implement a basic cat in Go.
package main
import (
"fmt"
"io"
"os"
)
func main() {
if len(os.Args) < 2 {
fmt.Fprintln(os.Stderr, "Usage: go run main.go <filename>")
os.Exit(1)
}
filename := os.Args[1]
file, err := os.Open(filename)
if err != nil {
fmt.Fprintf(os.Stderr, "Error opening file: %v\n", err)
os.Exit(1)
}
defer file.Close()
_, err = io.Copy(os.Stdout, file)
if err != nil {
fmt.Fprintf(os.Stderr, "Error reading file: %v\n", err)
os.Exit(1)
}
}
This is what our cat directory looks like:
.
├── go.mod
├── main.go
└── testscripts
└── hello.txt
To run the hello.txt testscript, we add main_test.go:
package main
import (
"testing"
"github.com/rogpeppe/go-internal/testscript"
)
func TestCLI(t *testing.T) {
testscript.Run(t, testscript.Params{
Dir: "testscripts",
})
}
When running go test, it runs all files in the testscripts directory. Neat!
But wait a second, what cat are we actually running?
We can guess, but to find out, let;s add this line to hello.txt:
exec which cat
go test -v reveals:
> exec which cat
[stdout]
/usr/bin/cat
PASS
Should we add our compiled version in the PATH? I tried that, (it works), but it feels to so fragile and ugly!
We can tell the testscript package to compile main() for us and tell it what it should be called:
func TestMain(m *testing.M) {
testscript.Main(m, map[string]func(){
"cat": main,
})
}
All test PASS!
Even more interesting is the verbose output. It reveals how testscript made sure that we executed our own meowing executable:
=== RUN TestCLI
=== RUN TestCLI/hello
=== PAUSE TestCLI/hello
=== CONT TestCLI/hello
testscript.go:584: WORK=$WORK
PATH=/tmp/testscript-main2394752340/bin:/home/bram/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.0.linux-amd64/bin:/home/bram/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
GOTRACEBACK=system
HOME=/no-home
TMPDIR=$WORK/.tmp
devnull=/dev/null
/=/
:=:
$=$
exe=
# hello world (0.002s)
> exec cat hello.text
[stdout]
shello world
> stdout 'hello world\n'
> ! stderr .
# what cat (0.001s)
> exec which cat
[stdout]
/tmp/testscript-main2394752340/bin/cat
PASS
--- PASS: TestCLI (0.00s)
--- PASS: TestCLI/hello (0.00s)
PASS
ok example.com/cat 0.006s
The first directory in the PATH is a temporary directory that the testscript package created for us. Without having to change hello.txt, we got it to use our own executable! Which also matches the output of which cat: /tmp/testscript-main2394752340/bin/cat.
Lots of things are taken care of without feeling too magical, almost simple!
From now on, adding new testscripts can be added by either extenting the hello.txt file. Or just add more txt testfile in the same directory. Make sure the virtual files stay at bottom of the txtar file.
Stdin
To see if my cat also accepts stdin, we start with writing the test. That’s as simple as:
# cat spits out standard input
stdin hello.txt
exec cat
cmp stdout hello.txt
-- hello.txt --
hello world
We tell testscript that the stdin for the next exec-ed command is the contents of the virtual file hello.txt. Then when we cat is ready, cmp compares the stdout with the same file. And to no surprise the testscript hello.txt passes the test.
But running the go test is a different story:
$ go test
--- FAIL: TestCLI (0.00s)
--- FAIL: TestCLI/hello (0.00s)
testscript.go:584: > stdin hello.txt
> exec cat
[stderr]
Usage: go run main.go <filename>
[exit status 1]
FAIL: testscripts/hello.txt:2: unexpected command failure
FAIL
exit status 1
FAIL example.com/cat 0.009s
Indeed! My version of cat can only read 1 file and does not care about standard input. Do you see the test error is quite descriptive? Failed cases are reported with the stdin and exec lines. It even prints what came out of stderr and the exit status. The error occurred on line 2 of the test, the exec line itself. It failed because an os.Exit(1) is issued after jotting the usage message to stderr.
Apart from the failed test (for demonstration purposes), I could have done better btw. It’s confusing that both the test itself and the internal virtual file share the name hello.txt 🤔.
Testing interactivity
One of the troubles we had when using Bats, was testing a script that prompts for a password.
For a cli script called SetPasswd, the bats code looked like:
@test "SetPasswd can set a password interactively" {
# set a known password
_setpw testuser secret123@ >/dev/null
}
Which in turn, called _setpasswd, an expect script that sends the keystrokes to SetPasswd itself:
#!/usr/bin/expect -f
# first argument is username
set loginname [lindex $argv 0]
# second argument is the password
set password [lindex $argv 1]
set timeout -1
spawn SetPasswd $loginname
match_max 100000
expect -exact "New Password: "
send -- "$password\r"
expect -exact "\r\nRe-enter new Password: "
sleep 1
send -- "$password\r"
expect eof
This works, but it’s the clumsy test I told you about.
Now, what would the testscript version look like:
# test SetPasswd
stdin typing
exec SetPasswd alice
-- typing --
blablabla
blablabla
No way! Yes, that’s all. No indirections, you can see what it’s doing in one glance. I can’t imagine how you could design black box testing software that’s easier to use than this. Hats off to Roger!
Golden files
When comparing your software’s output or generated files with an expected file, we call these expectations “🏆 Golden Files ✨”. You guessed it, these can be embedded in the tests themselves:
# test power.go against a golden file
exec go run power.go
cmp stdout golden
!stdout 'hello world'
-- power.go --
package main
import "fmt"
func main() {
for i := 0; i < 10; i++ {
fmt.Printf("2^%d = %d\n", i, 1<<i)
}
}
-- golden --
2^0 = 1
2^1 = 2
2^2 = 4
2^3 = 8
2^4 = 16
2^5 = 32
2^6 = 64
2^7 = 128
2^8 = 256
2^9 = 512
Oh, I got too lazy and embedded the Go source code in the test as well. The exec line just runs the embedded power.go and then compares it to the expected output. And because I was getting tired of “hello world”, I wanted to make sure that was nowhere to be found in stdout (!stdout).
This, very much stand alone, test results in: PASS!
Golden files, part II
Software evolves, golden files change. Do I really need to change all these txtar embedded files? Getting tired already.
There’s a sneaky -u flag to testscript, that simply assumes that the second argument of the cmp command is the desired output and overwrites it!
$ testscript -u power.txt
# test power.go against a golden file (0.156s)
PASS
testscripts/power.txt updated
$ tail -1 power.txt
2^10 = 1024
I realise that I shouldn’t have bothered writing the golden file in the first place! Just the file marker, -- golden --, was enough to have testscript write the content there.
From the testscript help:
The -u flag specifies that if a cmp command within a testscript fails and its second argument refers to a file inside the testscript file, the command will succeed and the testscript file will be updated to reflect the actual content. As such, this is the
cmd/testscriptequivalent oftestscript.Params.UpdateScripts.
So in a go test, we could use this by parsing a flag and call testscript with the UpdateScripts parameter. This should, of course, be done interactively. Our pipelines can continue to compare to the fixed embedded gold.
Conclusion
- Testscript is very well designed, easy to use and brings back (depending where you come from) fun in testing.
- Designed for, but not limited to the Go ecosystem.
- Handles golden files elegantly!
- It’s feature complete, don’t expect new feature MR to be accepted.
- Be aware of the testing context. Are you really testing what you think you’re testing?
catvscat,hello.txtvshello.text testscriptdeserves a better name ;-) It looks like it started as a … well … test script, but got far better than anticipated!