The even better news is that I've been able to get completely rid
of the annoying 'view filefield uploads' permission by replacing it
with the more fine-grained field-level permissions from content_permissions
which ships with CCK 6.x.

The new code makes our hook_file_download() a bit more complex,
but on the other hand only cares about files that are actually
contained in at least one file field. Yay for correctness and flexibility!