I was inspired by Niels Thykier’s blog post asking for help with Debian RC bugs to see if I could help. I’m not a Debian Developer, so I was looking for bugs where that wasn’t much of a hurdle.

Searching udd (the “Ultimate Debian Database”) for RC bugs, sorted by most popular projects put a nice tar bug at the top, #901952.

Investigation

Whenever I look into a bug that has previous work, I will review but not trust those efforts.

I like to first understand as much context as possible from scratch. Too much is better than too little. In this case, the key error messages in the bug report meant nothing to me:

gbp:error: Error creating fdroidserver_1.0.6.orig.tar.gz: 
Pristine-tar couldn't checkout "fdroidserver_1.0.6.orig.tar.gz": 
xdelta: expected from file (/tmp/pristine-tar.SljdkfANnj/recreatetarball) of length 7557120 bytes

so I had a lot of investigation to do.

The bug had been filed against pristine-tar and reassigned to tar, though it was not clear that the fix would lie there.

In this case, since I wasn’t familiar with any of the software involved, there was quite a bit of poking around to get context.

First, I downloaded the source to gbp (git-buildpackage) to sort out what it was trying to do and where things were breaking.

The failing step was a pristine-tar checkout, which “regenerates a copy of the specified tarball”, per the man page. Basically, an earlier tar file and xdelta patch have been saved to the git repository for fdroidserver. The call to pristine-tar checkout verifies that it can create an identical tar file.

pristine-tar

pristine-tar helpfully has -v and -d options to show details of what it is doing. Unfortunately, neither showed enough of the command path for me to understand exactly what was going on. All the work is done in temporary directories and the debugging output wasn’t showing the setup of the contents of those directories.

Also making things tricky was that pristine-tar is very good at cleaning up all of those temporary directories, so I couldn’t debug after the fact.

On the positive side, pristine-tar’s main logic is in a single perl script. It is very easy to read, breakpoint, and step through. I simply had to breakpoint once everything was set up for the tar command in question. Then I could run a slightly modified command line and save the tar output somewhere I could review later.

tar has changed

Currently, Debian’s tar policy (on my machine) looks like:

tar:
  Installed: 1.30+dfsg-2
  Candidate: 1.30+dfsg-2
  Version table:
 *** 1.30+dfsg-2 700
        700 https://deb.debian.org/debian testing/main amd64 Packages
        600 https://deb.debian.org/debian unstable/main amd64 Packages
        100 /var/lib/dpkg/status
     1.29b-1.1 500
        500 https://deb.debian.org/debian stable/main amd64 Packages

This is perfect, because 1.30 was breaking things and 1.29 was not, according to the bug details. I could easily swap back and forth between them to test.

After doing so, the tar files were in fact different. The files with tricky Unicode characters in their names were zero-length when using 1.30.

Zero length? What the hell? It’s actually nothing too surprising. This was expected when the file in question is not found. The pristine-tar process makes sure that files in the manifest that are missing are saved as zero-length files.

The bug (or at least the changed behavior) was definitely in tar, no longer managing to find files mentioned in the manifest if they have weird characters. The next step was to figure out why.

Upstream tar

I headed to the upstream repo, specifically to the change log.

The only really likely change between 1.29 and 1.30 was:

2017-11-09	Fix --verbatim-files-from

since this is the option that pristine-tar is using to make sure the manifest is read without unwanted processing.

The change message for that commit includes

* src/names.c (read_next_name): Don't unquote name read from the
file, if --verbatim-files-from option is in effect.

which would definitely break our use case. Using a new rule for unescaping file names with --verbatim-files-from would keep files that needed escaping from matching the manifest. Which is exactly what I was seeing.

All Done?

Okay, now I can update the bug report and move on? No, just to be safe I wanted to verify, even though the answer seemed clear.

Verify the cause

Time to build my own tar binary.

The process, basically:

sudo apt-get build-dep tar
apt-get source tar
wget -O p1  'http://git.savannah.gnu.org/cgit/tar.git/patch/?id=dee7e3f16e74e07504bb8f4d80426005fe4364ae'
cd tar-1.29b
patch -p1 <../p1

dpkg-buildpackage -us -uc2
[fails complaining that there are local changes, needs 'dpkg-source --commit']

dpkg-source --commit
dpkg-buildpackage -us -uc
[wait for very large tar test suite]

After installing the generated .deb file and regenerating the tar file (again, by breakpointing pristine-tar at the right point), the zero-length files are seen. This matches the buggy behavior, so the responsible patch has (probably) been identified.

Update the bug

With a verification that I had the right answer, I updated the bug. I intentionally didn’t include much background, as I definitely don’t have the full context. I just answered the maintainer’s existing question about which change caused the issue.

I subscribed to the bug in the Debian bug tracker, in case more questions arose.

Next Steps

I could leave the Debian and upstream tar maintainers to figure out what is the right outcome. But I wanted to spend a little time trying to figure it out.

The NEWS file for GNU Tar doesn’t explain the change in detail. However, I do see that --verbatim-files-from was new in 1.29. It’s not too surprising the behavior is changing. But is the 1.30 behavior correct? What are the pros and cons? Also, is there a clean pristine-tar workaround?

On the tar side, allowing the file list to have a level of quoting seems like it would be safe to allow. I can’t find an explanation for the change that made --unquote and --verbatim-files-from not work together, so I’ll assume there is a good reason.

On the pristine-tar side, perhaps using this tar option is the way forward:

--null Instruct  subsequent -T options to read null-terminated names
    verbatim (disables special handling of names that start with a dash).

Null-separating the file names may do a better job of removing ambiguity, but just switching to this isn’t simple, as old manifests need to be correctly handled.

Making this change would be a big job for me, since I don’t know pristine-tar well. Since I’m not sure which way Debian wants to go with this, I will leave things here.