Part I

I along with many others have been trying and asking how to index and search PDF files. Once Zend released its Framework, which is a port of Java Lucene to PHP, I decided to jump on board and find a way to index and search PDF files. So… Lets get started.

The Setup…

We will use XPDF to parse the PDF files and Zend_Search_Lucene for indexing and searching. First we need to read the PDF files and get relevant PDF information such as Title, Author, Modification Date, PDF size, and number of pages in the file. To do all this, download XPDF and copy the pdftotext and pdfinfo into a directory on your web server. The pdfinfo call will get us metadata of the PDF and the pdftotext will convert the PDF to a text file. Converting the PDF to a text file will allow us to store the data in the index.

Next we will have to download the Zend Framework and copy it over to the web server. You can just copy over the Zend_Search folder and the necessary functions but for now we will make it easy and copy everything to the server.

The Code…

Lets start by parsing a file and storing the data into the index file. First we need to make sure that our include path is pointed to the working directory and that we can access the Zend Search Lucene functions.

/** Zend_Search_Lucene */
ini_set('include_path','/path/to/working/directory/);
require_once 'Zend/Search/Lucene.php';

Next we create and index or open an existing index. Here we will call the index pdf_index. This is created as a directory with index files inside it. When you call the create index everything will be created. There is no need to create the pdf_index manually.

/** if the index exists then open it.  Otherwise we will create the index */
if(is_dir("/path/to/index/location/pdf_index/") == 1) {
// Open existing index
$index = Zend_Search_Lucene::open('/path/to/index/location/pdf_index/');
}
else {
//Create a new index
$index = Zend_Search_Lucene::create('/path/to/index/location/pdf_index/');
}

After the index we have to instantiate the document object which will be used to add data to the index.

$doc = new Zend_Search_Lucene_Document();

Next is the most important part of the parsing. We will use XPDF‘s pdfinfo to get metadata about the PDF file. This is a critical part of indexing a PDF since it will get the data that we need in order parse through the actual PDF file. We will store this data in an array and parse through it to get the information we need. Not all the metadata will be used but you can certainly use the additional data if you want.

//Name of the pdf document with out the extension
$pdf_filename = "KDS50A3000";
// get pdf information
$output = shell_exec ("pdfinfo ".$pdf_filename.".pdf");
//Gets the metadata
$data = explode("\n", $output); //puts it into an array
//Get the metadata that we need from the PDF.
//Parse through the Array and store in variables. */
for($c=0; $c < count($data); $c++) {
//Number of Pages
if(stristr($data[$c],"pages") == true) {
$pagestr = $data[$c];
}
//Author
if(stristr($data[$c],"author") == true) {
$authorstr = $data[$c];
}
//Title
if(stristr($data[$c],"title") == true) {
$titlestr = $data[$c];
}
//Modification Date
if(stristr($data[$c],"ModDate") == true) {
$moddatestr = $data[$c];
}
//File Size
if(stristr($data[$c],"File size") == true) {
$sizestr = $data[$c];
}
}
//Remove the titles to the metadata
$pages = trim(substr($pagestr,6));
$author = trim(substr($authorstr,7));
$title = trim(substr($titlestr,6));
$date = trim(substr($moddatestr,8));
$size = get_size(trim(substr(trim(substr($sizestr,10)),0,-5)));

With the file size we call another function called get_size to put the size in a human readable form.

function get_size($size) {
$bytes = array('B','KB','MB','GB','TB');
foreach($bytes as $val) {
if($size > 1024) {
$size = $size / 1024;
}
else {
break;
}
}
return round($size, 2)." ".$val;
}

Now we have all the necessary data that we need to parse through the PDF file, here is how we will do it. Since we know how many pages there are in the PDF we will create a for loop and increment through each page. So if we have a 2 page PDF file, we will convert page 1 and store it in the index then convert page two and store it in the index. Each page will be stored as a separate document in the index. This allows us to specify what page of the PDF the relevant search terms are referencing.

Below is the code that does just that.

// parse the pages into the index
for($i=1; $i < = $pages; $i++) {
//Here we will loop through all the page in the PDF

exec ("pdftotext -f ".$i." -l ".$i." ".$pdf_filename.".pdf ".$pdf_filename.$i.".txt");
// Store data in the document object that will be commited to the index at the end of the loop
$doc->addField(Zend_Search_Lucene_Field::Keyword('id', $i)); //Stores the ID
//Stores the File name of the PDF
$doc->addField(Zend_Search_Lucene_Field::Text('url', $pdf_filename.".pdf"));
//Here we will open the txt file (1 page of the pdf) and read it in to a String Variable
//Then we will store that sting in the document object for the index.
$filename = $pdf_filename.$i.".txt";
$handle = fopen ($filename, "rb");
$docContent = fread($handle, filesize($filename));
fclose($handle);
$doc->addField(Zend_Search_Lucene_Field::Text('contents', $docContent));
//Stores the page # so that we can open this page up
$doc->addField(Zend_Search_Lucene_Field::UnIndexed('page', $i));
// Add all the data stored in the document object into the index.
$index->addDocument($doc);
//get rid of the txt file that the page data was stored in.
shell_exec ("rm ".$pdf_filename.$i.".txt");
echo "Page ".$i." Complete";
}
//Optimize index so that things run better.
//This is done after all the pages are inserted into the index
$index->optimize();
//Get some index data
$indexSize = $index->count();
$documents = $index->numDocs();
echo "Index Size: ".$indexSize;
echo "Document Count: ".$documents;

That’s pretty much it for putting the PDF data into the index. Now all that’s left is searching the index. which will be addressed in another post.

Let’s review a little…

Our goal was to index PDF files for searching. Using XPDF, Zend Search Lucene from the Zend Framework, and a PDF file, we were able to get metadata from the PDF file and parse the file page by page while inserting data into the index. What I didn’t get into was how the index works, what Zend_Search_Lucene_Fields work best, and what is index and what is not. That is all documented nicely at the Zend Framework site.

Also, this code is simple. It works with one PDF file, doesn’t check to see if the PDF is already indexed, and so forth. However, it can be very easily modified to do things dynamically. This is just a prototype to show that indexing and searching PDF’s can be done. Part II will be posted soon and will show you how to search the index. I also suggest looking at the following sites that may give you more incite on implementing Zend Search Lucene.

• Content Indexing with Zend_Search_Lucene
• Roll Your Own Search Engine with Zend_Search_Lucene
• Chapter 29. Zend_Search_Lucene
• Creating A Fulltext Search Engine In PHP 5 With The Zend Framework’s Zend Search Lucene

For now, enjoy! Oh yeah…here is the Zip File of the data files.