PDF viewing woes: update

In March 2010 I reported on a couple problems viewing PDFs. The first problem was that my file manager, Dolphin, was unable to generate previews of some PDFs due to an arbitrary limit on file sizes. The second problem was slow rendering of the PDFs in my viewer, Okular. I'm pleased to report that the first of these issues has been fixed, and the second is due to be fixed in the next stable release series of Poppler (the PDF rendering library used by Okular).

LSE PDF cropping

Using the pdfimages tool from Poppler, I've confirmed that all the LSE PDFs consist of full-page RGB DCT images. There are a couple problems with this:

1. This format is lossy and thus the quality of the images could suffer if I transform them.
2. The files are needlessly large, since they are stored in full colour even though the scans are only in shades of grey.

Now, it is possible to extract the images as JPEGs and crop them losslessly using jpegtran, but only when the upper left corner of the cropped region falls on an iMCU boundary. Fortunately, the images are scanned at a high enough resolution that increasing the dimensions of the region by a few extra pixels to ensure boundary alignment shouldn't be a problem. It's also possible to losslessly convert RGB JPEGs to greyscale, though the reduction in file size is negligible (for these images about 3.4%).

So the next order of business is to extract the images from all the LSE PDFs and crop them. It's possible that the exact coordinates for the cropped regions may vary across the images, depending on the paper size (which changed throughout the Standard's run) and the positioning of the paper when the issues were originally photographed for microfiche. I am hoping, though, that large runs of issues will use the same cropping coordinates, allowing me to do most of the work automatically rather than manually.

I used the following bash script extract all the DCT images from the LSE PDFs as JPEGs:

for year in {1904..1972}
do
  pdfimages -f 2 -j LSE_SocialistStandard_$year.pdf $year
done

I'll then have to examine the resulting JPEG files manually to determine the cropping region for the left- and right-hand pages. Once I have these, I can do batch greyscaling and cropping using the following script, where W and H are the pixel width and height of the region, respectively, and X+Y is the offset from the upper left corner of the original image:

for f in *
do
  jpegtran -grayscale -crop W1xH1+X1+Y1 $f >left-$f
  jpegtran -grayscale -crop W2xH2+X2+Y2 $f >right-$f
done

Given the amount of data I have, the above scripts can take about an hour to complete. They also create a large amount of data, and I've found I'm running out of disk space. I've therefore ordered a Samsung Story Station 1.5 TB USB 2.0 external hard drive.

PDF software inventory

Here's an inventory of the PDF manipulation software I have at my disposal for the project. All of these packages are Free Software in the sense that the user has the freedom to run, study, modify, and redistribute the program.

Okular

PDF viewer (GUI)

GNU gv

PDF viewer (GUI)

pdftk

merges and splits PDFs, rotates pages, edits metadata (command-line; GUI via pdftk-qgui)

PDFjam

merges, rotates, and n-ups PDFs (command-line)

PdfMod

adds, reorders, rotates, and removes pages; exports images; edits metadata (GUI)

poppler-tools

lists fonts; extracts images; prints metadata; converts PDF to HTML, PPM, PostScript, or text (command-line)

QPDF

linearizes (web-optimizes) PDFs, various other low-level transformations (command-line)

pspdftool

rearranges, deletes, scales, flips, numbers, crops, rotates, n-ups pages (command-line)

pstoedit

scales, shifts, splits, rotates, and resizes pages; converts PDF to other vector formats (command-line)

Ghostscript

PDF language interpreter; can do pretty much everything, though not necessarily easily (command-line)

PDF viewing woes

I've been running into problems with the programs I use to view the LSE PDFs. For one thing, these PDFs are an average of 261 MB in size, whereas my file manager, Dolphin, won't generate previews for files larger than 100 MB. This is a rather annoying and arbitrary upper limit, especially considering that it takes only a few milliseconds to generate thumbnails for 100 MB PDFs. I've accordingly filed a bug report asking that the limit be removed.

The other problem is that my usual PDF viewer, Okular, is phenomenally slow at rendering the pages of the LSE PDFs—it takes between 10 and 25 seconds per page. (By comparison, the proprietary Adobe Reader renders them almost instantly.) Okular, like many other Free Software document viewers, renders PDFs using the FreeDesktop project's Poppler library, and it is there that the problem lies. Most likely this is due to a known issue, Bug 13518. For now, then, I will be using GNU gv, which isn't based on Poppler and is able to render the LSE PDFs quickly.