As a developer, you can't envision all of the possible inputs your programs or functions could receive. Even though you can define the major edge cases, you still can't predict how your program will behave in the case of some weird unexpected input. In other words, you can only find bugs you expect to find.
That's where fuzz testing or fuzzing comes to the rescue.
What is Fuzz Testing
Fuzzing is an automated software testing technique that involves inputting a large amount of valid, nearly-valid or invalid random data into a computer program and observing its behavior and output. So the goal of fuzzing is to reveal bugs, crashes and security vulnerabilities in source code you might not find through traditional testing methods.
The Fuzz Testing in Go video which I made a few years ago shows a very simple example of the Go code that may work well unless you provide a certain input:
func Equal(a []byte, b []byte) bool {
for i := range a {
// can panic with runtime error: index out of range.
if a[i] != b[i] {
return false
}
}
return true
}
Fuzzing technique would easily spot this bug by bombarding this function with various inputs.
It's a good practice to integrate fuzzing into your team’s software development lifecycle (SDLC) as well. For example, Microsoft uses fuzzing as one of the stages in its SDLC, to find potential bugs and vulnerabilities.
Fuzz Testing in Go
There are many fuzzing tools that have been available for a while such as oss-fuzz for example, but since Go 1.18 fuzzing was added to Go's standard library and is part of the regular testing package since it is a kind of test. It can also be used together with the other testing primitives which is nice.
The steps to create a fuzz test in Go are the following:
In a _test.go file create a function that starts with Fuzz which accepts *testing.F
Add corpus seeds using f.Add() to allow fuzzer to generate the data based on it.
Call fuzz target using f.Fuzz() by passing fuzzing arguments which our target function accepts.
Start the fuzzer using regular go test command, but with the –fuzz=Fuzz flag
Note, the fuzzing arguments can only be the following types:
string, byte, []byte
int, int8, int16, int32/rune, int64
uint, uint8, uint16, uint32, uint64
float32, float64
bool
A simple fuzz test for the Equal function above may look like this:
// Fuzz test
func FuzzEqual(f *testing.F) {
// Seed corpus addition
f.Add([]byte{'f', 'u', 'z', 'z'}, []byte{'t', 'e', 's', 't'})
// Fuzz target with fuzzing arguments
f.Fuzz(func(t *testing.T, a []byte, b []byte) {
// Call our target function and pass fuzzing arguments
Equal(a, b)
})
}
By default, fuzz tests run forever, so you either need to specify the time limit or wait for fuzz tests to fail. You can specify which tests to run using --fuzz argument.
go test --fuzz=Fuzz -fuzztime=10s
If there are any errors during the execution, the output should look similar to this:
go test --fuzz=Fuzz -fuzztime=30s
--- FAIL: FuzzEqual (0.02s)
--- FAIL: FuzzEqual (0.00s)
testing.go:1591: panic: runtime error: index out of range
Failing input written to testdata/fuzz/FuzzEqual/84ed65595ad05a58
To re-run:
go test -run=FuzzEqual/84ed65595ad05a58
Notice that the input for which fuzz test has failed are written into a file in testdata
folder and can be re-played by using that input identifier.
go test -run=FuzzEqual/84ed65595ad05a58
The testdata folder can be checked into the repository and be used for regular tests, because fuzz tests can also act as regular tests when executed without --fuzz flag.
Fuzzing HTTP Services
It's also possible to fuzz test the http services by writing a test for your HandlerFunc and using the httptest package. Which can be very useful to test the whole HTTP service, not only the underlying functions.
Let's now introduce a more real example such as an HTTP Handler that accepts some user input in the request body and then write a fuzz test for it.
Our handler accepts JSON request with limit
and offset
fields to paginate some static mocked data. Let's define the types first.
type Request struct {
Limit int `json:"limit"`
Offset int `json:"offset"`
}
type Response struct {
Results []int `json:"items"`
PagesCount int `json:"pagesCount"`
}
Our handler function then parses the JSON, paginates the static slice and returns a new JSON in response.
func ProcessRequest(w http.ResponseWriter, r *http.Request) {
var req Request
// Decode JSON request
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
// Apply offset and limit to some static data
all := make([]int, 1000)
start := req.Offset
end := req.Offset + req.Limit
res := Response{
Results: all[start:end],
PagesCount: len(all) / req.Limit,
}
// Send JSON response
if err := json.NewEncoder(w).Encode(res); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.WriteHeader(http.StatusOK)
}
As you may have already noticed, this function doesn't handle slice operations quite well and can easily panic
. Also it can panic if it tries to divide by 0. It's great if we can spot it during the development or using only unit tests, but sometimes not everything is visible to our eye, and our handler may pass the input to other functions and so forth.
Following our FuzzEqual example above, let's implement a fuzz test for the ProcessRequest
handler. The first thing we need to do is to provide the sample inputs for the fuzzer. This is the data that the fuzzer will use and modify into new inputs that are tried. We can craft some sample JSON request and use f.Add() with []byte type.
func FuzzProcessRequest(f *testing.F) {
// Create sample inputs for the fuzzer
testRequests := []Request{
{Limit: -10, Offset: -10},
{Limit: 0, Offset: 0},
{Limit: 100, Offset: 100},
{Limit: 200, Offset: 200},
}
// Add to the seed corpus
for _, r := range testRequests {
if data, err := json.Marshal(r); err == nil {
f.Add(data)
}
}
// ...
}
After that we can use httptest package to create a test HTTP server and make requests to it.
Note: Since our fuzzer can generate invalid non-JSON requests, it's better just to skip them and ignore with t.Skip(). We can also skip BadRequest errors.
func FuzzProcessRequest(f *testing.F) {
// ...
// Create a test server
srv := httptest.NewServer(http.HandlerFunc(ProcessRequest))
defer srv.Close()
// Fuzz target with a single []byte argument
f.Fuzz(func(t *testing.T, data []byte) {
var req Request
if err := json.Unmarshal(data, &req); err != nil {
// Skip invalid JSON requests that may be generated during fuzz
t.Skip("invalid json")
}
// Pass data to the server
resp, err := http.DefaultClient.Post(srv.URL, "application/json", bytes.NewBuffer(data))
if err != nil {
t.Fatalf("unable to call server: %v, data: %s", err, string(data))
}
defer resp.Body.Close()
// Skip BadRequest errors
if resp.StatusCode == http.StatusBadRequest {
t.Skip("invalid json")
}
// Check status code
if resp.StatusCode != http.StatusOK {
t.Fatalf("non-200 status code %d", resp.StatusCode)
}
})
}
Our fuzz target has a single argument with a type []byte that contains the full JSON request, however you can change it to have multiple arguments.
Everything is ready now to run our fuzz tests. When fuzzing http servers, you may need to adjust the amount of parallel workers, otherwise the load may overwhelm the test server. You can do that by setting -parallel=1 flag.
go test --fuzz=Fuzz -fuzztime=10s -parallel=1
And as expected we will see the following errors uncovered.
go test --fuzz=Fuzz -fuzztime=30s
--- FAIL: FuzzProcessRequest (0.02s)
--- FAIL: FuzzProcessRequest (0.00s)
runtime error: integer divide by zero
runtime error: slice bounds out of range
We can also see the fuzz inputs in the testdata
folder to see which JSON contributed to this failure. Here is a sample content of the file:
go test fuzz v1
[]byte("{"limit":0,"offset":0}")
To fix that issue we can introduce input validation and default settings:
if req.Limit <= 0 {
req.Limit = 1
}
if req.Offset < 0 {
req.Offset = 0
}
if req.Offset > len(all) {
start = len(all) - 1
}
if end > len(all) {
end = len(all)
}
With this change the fuzz tests will run for 10 seconds and exit without an error.
Conclusion
Writing fuzz tests for your HTTP services or any other methods is a great way to detect hard-to-find bugs. Fuzzers can detect hard-to-spot bugs that happen for only some weird unexpected input.
It's amazing to see that fuzzing is a part of built-in testing library, making it easy to combine with regular tests. Note: prior to Go 1.18 developers used go-fuzz, which is a great tool for fuzzing as well.
Fuzzy testing is a wonderful topic. Without knowing about it, I wrote an article last year on a debugging technique that me and my competitive programming team used to apply when we had no idea what was wrong with our code. It made me $500 on an online writeathon. So, here's my personal experience with fuzzy testing: https://albexl.substack.com/p/a-tale-of-debugging-the-competitive. Thanks for this beautiful piece, Alex.