//open the remote fee from twitter.
$ch = curl_init("http://twitter.com/statuses/user_timeline/2376811.rss");
//overwrite this feed to a local file
$fp = fopen ("twitter_feed.xml", "w");
curl_setopt ($ch, CURLOPT_FILE, $fp);
curl_setopt ($ch, CURLOPT_HEADER, 0);
curl_exec ($ch);
curl_close ($ch);
fclose ($fp);

Then you have to parse the local file that you just created. Twitter’s RSS feed puts the feed data in item nodes but nested in a channel node. Here we will use SimpleXML (PHP5 Only!) and a foreach loop to parse through each item node.

$xml = simplexml_load_file ("twitter_feed.xml");
foreach ($xml->channel->item as $item) {
echo str_replace ("farrelley:", "", $item->description);
}

Basically that is the heart of the code. Next we will put it all together into a function. We will also add in a counter that when you call the function you can specify how many item nodes or tweets you want to see.

// Display Twitter Feed
function getTwitterFeed($disp) {
//use Curl to get the feed and store it locally
$ch = curl_init ("http://twitter.com/statuses/user_timeline/2376811.rss");
//open the local feed.
$fp = fopen ("twitter_feed.xml", "w");
//set the counter = 1 its the first time through the script/
$count = 1;
curl_setopt ($ch, CURLOPT_FILE, $fp);
curl_setopt ($ch, CURLOPT_HEADER, 0);
curl_exec ($ch);
curl_close ($ch);
fclose ($fp);
//parse through locally stored feed
$xml = simplexml_load_file ("twitter_feed.xml");
foreach ($xml->channel->item as $item) {
//strip out my username each description has username: at beginning.
echo str_replace ("farrelley:", "", $item->description);
//if the count = the number to display then break out of foreach
if ($count == $disp) break;
//else increment count and continue.
else $count++;
}
}

Then you just have to add this call to the function where you want the data to be output.

getTwitterFeed (1);

You can see this in action (on the right sidebar) on my other blog at Living in the District (http://www.livinginthedistrict.com/). Like always and comments please post comments.

Many companies large and small have support centers, either be in internal help desks or external help desks. In addition to the help desk, many companies publish PDF documents such as manuals, specs, services guides, and setup/connections guides etc. So instead of a help desk employee (or anyone) remembering what manual does what and what page everything is on, you can simple index these PDF files for easy searching. Just think about it this way, say you have 50 products all having 5 manuals each, that’s 250 manuals that you have to keep track of (not including how many pages each manual has). The easy way would be to index the PDF’s, add the necessary metadata to the manual, build a search form around a web page and wa-la. You have a easy way to search PDF files finding information quickly for a customer or whoever, and saving loads of time searching page by page for the same information.

Many companies do this, and many companies bloat how they are the best at doing it. So next time you are looking for a searchable PDF solution, remember that anyone can do this and it’s easy to do yourself.

I along with many others have been trying and asking how to index and search PDF files. Once Zend released its Framework, which is a port of Java Lucene to PHP, I decided to jump on board and find a way to index and search PDF files. So… Lets get started.

The Setup…

We will use XPDF to parse the PDF files and Zend_Search_Lucene for indexing and searching. First we need to read the PDF files and get relevant PDF information such as Title, Author, Modification Date, PDF size, and number of pages in the file. To do all this, download XPDF and copy the pdftotext and pdfinfo into a directory on your web server. The pdfinfo call will get us metadata of the PDF and the pdftotext will convert the PDF to a text file. Converting the PDF to a text file will allow us to store the data in the index.

Next we will have to download the Zend Framework and copy it over to the web server. You can just copy over the Zend_Search folder and the necessary functions but for now we will make it easy and copy everything to the server.

The Code…

Lets start by parsing a file and storing the data into the index file. First we need to make sure that our include path is pointed to the working directory and that we can access the Zend Search Lucene functions.

/** Zend_Search_Lucene */
ini_set('include_path','/path/to/working/directory/);
require_once 'Zend/Search/Lucene.php';

Next we create and index or open an existing index. Here we will call the index pdf_index. This is created as a directory with index files inside it. When you call the create index everything will be created. There is no need to create the pdf_index manually.

/** if the index exists then open it.  Otherwise we will create the index */
if(is_dir("/path/to/index/location/pdf_index/") == 1) {
// Open existing index
$index = Zend_Search_Lucene::open('/path/to/index/location/pdf_index/');
}
else {
//Create a new index
$index = Zend_Search_Lucene::create('/path/to/index/location/pdf_index/');
}

After the index we have to instantiate the document object which will be used to add data to the index.

$doc = new Zend_Search_Lucene_Document();

Next is the most important part of the parsing. We will use XPDF‘s pdfinfo to get metadata about the PDF file. This is a critical part of indexing a PDF since it will get the data that we need in order parse through the actual PDF file. We will store this data in an array and parse through it to get the information we need. Not all the metadata will be used but you can certainly use the additional data if you want.

//Name of the pdf document with out the extension
$pdf_filename = "KDS50A3000";
// get pdf information
$output = shell_exec ("pdfinfo ".$pdf_filename.".pdf");
//Gets the metadata
$data = explode("\n", $output); //puts it into an array
//Get the metadata that we need from the PDF.
//Parse through the Array and store in variables. */
for($c=0; $c < count($data); $c++) {
//Number of Pages
if(stristr($data[$c],"pages") == true) {
$pagestr = $data[$c];
}
//Author
if(stristr($data[$c],"author") == true) {
$authorstr = $data[$c];
}
//Title
if(stristr($data[$c],"title") == true) {
$titlestr = $data[$c];
}
//Modification Date
if(stristr($data[$c],"ModDate") == true) {
$moddatestr = $data[$c];
}
//File Size
if(stristr($data[$c],"File size") == true) {
$sizestr = $data[$c];
}
}
//Remove the titles to the metadata
$pages = trim(substr($pagestr,6));
$author = trim(substr($authorstr,7));
$title = trim(substr($titlestr,6));
$date = trim(substr($moddatestr,8));
$size = get_size(trim(substr(trim(substr($sizestr,10)),0,-5)));

With the file size we call another function called get_size to put the size in a human readable form.

function get_size($size) {
$bytes = array('B','KB','MB','GB','TB');
foreach($bytes as $val) {
if($size > 1024) {
$size = $size / 1024;
}
else {
break;
}
}
return round($size, 2)." ".$val;
}

Now we have all the necessary data that we need to parse through the PDF file, here is how we will do it. Since we know how many pages there are in the PDF we will create a for loop and increment through each page. So if we have a 2 page PDF file, we will convert page 1 and store it in the index then convert page two and store it in the index. Each page will be stored as a separate document in the index. This allows us to specify what page of the PDF the relevant search terms are referencing.

Below is the code that does just that.

// parse the pages into the index
for($i=1; $i < = $pages; $i++) {
//Here we will loop through all the page in the PDF

exec ("pdftotext -f ".$i." -l ".$i." ".$pdf_filename.".pdf ".$pdf_filename.$i.".txt");
// Store data in the document object that will be commited to the index at the end of the loop
$doc->addField(Zend_Search_Lucene_Field::Keyword('id', $i)); //Stores the ID
//Stores the File name of the PDF
$doc->addField(Zend_Search_Lucene_Field::Text('url', $pdf_filename.".pdf"));
//Here we will open the txt file (1 page of the pdf) and read it in to a String Variable
//Then we will store that sting in the document object for the index.
$filename = $pdf_filename.$i.".txt";
$handle = fopen ($filename, "rb");
$docContent = fread($handle, filesize($filename));
fclose($handle);
$doc->addField(Zend_Search_Lucene_Field::Text('contents', $docContent));
//Stores the page # so that we can open this page up
$doc->addField(Zend_Search_Lucene_Field::UnIndexed('page', $i));
// Add all the data stored in the document object into the index.
$index->addDocument($doc);
//get rid of the txt file that the page data was stored in.
shell_exec ("rm ".$pdf_filename.$i.".txt");
echo "Page ".$i." Complete";
}
//Optimize index so that things run better.
//This is done after all the pages are inserted into the index
$index->optimize();
//Get some index data
$indexSize = $index->count();
$documents = $index->numDocs();
echo "Index Size: ".$indexSize;
echo "Document Count: ".$documents;

That’s pretty much it for putting the PDF data into the index. Now all that’s left is searching the index. which will be addressed in another post.

Let’s review a little…

Our goal was to index PDF files for searching. Using XPDF, Zend Search Lucene from the Zend Framework, and a PDF file, we were able to get metadata from the PDF file and parse the file page by page while inserting data into the index. What I didn’t get into was how the index works, what Zend_Search_Lucene_Fields work best, and what is index and what is not. That is all documented nicely at the Zend Framework site.

Also, this code is simple. It works with one PDF file, doesn’t check to see if the PDF is already indexed, and so forth. However, it can be very easily modified to do things dynamically. This is just a prototype to show that indexing and searching PDF’s can be done. Part II will be posted soon and will show you how to search the index. I also suggest looking at the following sites that may give you more incite on implementing Zend Search Lucene.

• Content Indexing with Zend_Search_Lucene
• Roll Your Own Search Engine with Zend_Search_Lucene
• Chapter 29. Zend_Search_Lucene
• Creating A Fulltext Search Engine In PHP 5 With The Zend Framework’s Zend Search Lucene

For now, enjoy! Oh yeah…here is the Zip File of the data files.

They have a couple of good starts on modules too. Some of these include

• E-commerce
• Forum
• Flickr
• Gallery
• Google Maps
• Blog

By far not as extensive as WordPress or Drupal but it’s a good start. I’m going to keep an eye on this New Zealand Company as they might make a bigger splash into the CMS world here soon. I would say that this CMS Package is a mix between a blogging CMS package and an Enterprise CMS package. Look for more to come on this topic.