Go: Debugging why ParseMultipartForm returns Error "no such file or directory"

For my most recent project u9k, I implemented a simple file upload API endpoint that accepts files as part of a multipart/form-data request. As usual, my implementation worked perfectly fine on my local machine (haha), but after deploying it to my server (wrapped up inside a minimal Docker container), it started to sporadically fail - especially for large files:

2020/09/29 05:57:58 "POST http://u9k.de/file/ HTTP/1.1" from 10.0.0.34:59326 - 400 49B in 8.554993597s
2020/09/29 05:57:58 Failed to parse form: open /tmp/multipart-855377186: no such file or directory

The error message is generated from the following piece of code:

1
2
3
4
5
6
	// parse uploaded data
	err := r.ParseMultipartForm(int64(10 << 20)) // 32 MB in memory, rest on disk
	if err != nil {
		log.Printf("Failed to parse form: %s\n", err)
		return nil
	}

Sooo, what’s happening here? To find out, I started going through the call graph.

Internally, ParseMultipartForm creates a new MultipartReader and calls its ReadForm method:

1
2
3
4
5
6
7
8
9
    mr, err := r.multipartReader(false)
	if err != nil {
		return err
	}

	f, err := mr.ReadForm(maxMemory)
	if err != nil {
		return err
	}

ReadForm does the actual work of reading, parsing and validating the bytes sent by the client. Further down in that method is the following code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
// ReadForm parses an entire multipart message whose parts have
// a Content-Disposition of "form-data".
// It stores up to maxMemory bytes + 10MB (reserved for non-file parts)
// in memory. File parts which can't be stored in memory will be stored on
// disk in temporary files.

...
	if n > maxMemory {
			// too big, write to disk and flush buffer
			file, err := ioutil.TempFile("", "multipart-")
			if err != nil {
				return nil, err
			}
			size, err := io.Copy(file, io.MultiReader(&b, p))
			if cerr := file.Close(); err == nil {
				err = cerr
			}
			if err != nil {
				os.Remove(file.Name())
				return nil, err
			}
            ...
    }

In case the multipart data does not fit in the specified memory size, the data is written to a temporary file on disk instead. This file is created with ioutil.TempFile (line 11).

Looking at the documentation of ioutil.TempFile does not reveal anything interesting or suspicious, but the source code gives the next hint:

1
2
3
if dir == "" {
		dir = os.TempDir()
}

Well, let’s look at the documentation and code of os.TempDir then:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
// TempDir returns the default directory to use for temporary files.
//
// On Unix systems, it returns $TMPDIR if non-empty, else /tmp.
// On Windows, it uses GetTempPath, returning the first non-empty
// value from %TMP%, %TEMP%, %USERPROFILE%, or the Windows directory.
// On Plan 9, it returns /tmp.
//
// The directory is neither guaranteed to exist nor have accessible
// permissions.
func TempDir() string { ... }

Aha!

“The directory is neither guaranteed to exist nor have accessible permissions.”

That’s why the open call fails with “no such file or directory”, but only inside my minimal Docker container. Except for the server binary and a few static assets, the container image does not have any directory structure, so /tmp does not exist yet. Subsequently, the os.OpenFile call in ioutil.TempFile fails and propagates all the way up to the multipart parser and my code!

This was quite a rabbit hole.

Under usual circumstances, /tmp exists in any UNIX environment, but to fix this behavior in minimal containers, I added the following code to my startup routine. It ensures the directory returned by os.TempDir exists and has the correct permissions for a temporary directory.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
	// make sure we have a working tempdir, because:
	// os.TempDir(): The directory is neither guaranteed to exist nor have accessible permissions.
	tempDir := os.TempDir()
	if err := os.MkdirAll(tempDir, 1777); err != nil {
		log.Fatalf("Failed to create temporary directory %s: %s", tempDir, err)
	}
	tempFile, err := ioutil.TempFile("", "genericInit_")
	if err != nil {
		log.Fatalf("Failed to create tempFile: %s", err)
	}
	_, err = fmt.Fprintf(tempFile, "Hello, World!")
	if err != nil {
		log.Fatalf("Failed to write to tempFile: %s", err)
	}
	if err := tempFile.Close(); err != nil {
		log.Fatalf("Failed to close tempFile: %s", err)
	}
	if err := os.Remove(tempFile.Name()); err != nil {
		log.Fatalf("Failed to delete tempFile: %s", err)
	}
	log.Printf("Using temporary directory %s", tempDir)