Previous Next

Overview

Introduction

Zend_Search_Lucene is a general purpose text search engine written entirely in PHP 5. Since it stores its index on the filesystem and does not require a database server, it can add search capabilities to almost any PHP-driven website. Zend_Search_Lucene supports the following features:

  • Ranked searching - best results returned first

  • Many powerful query types: phrase queries, boolean queries, wildcard queries, proximity queries, range queries and many others.

  • Search by specific field (e.g., title, author, contents)

Zend_Search_Lucene was derived from the Apache Lucene project. The currently (starting from ZF 1.6) supported Lucene index format versions are 1.4 - 2.3. For more information on Lucene, visit http://lucene.apache.org/java/docs/.

Note:

Previous Zend_Search_Lucene implementations support the Lucene 1.4 (1.9) - 2.1 index formats.

Starting from Zend Framework 1.5 any index created using pre-2.1 index format is automatically upgraded to Lucene 2.1 format after the Zend_Search_Lucene update and will not be compatible with Zend_Search_Lucene implementations included into Zend Framework 1.0.x.

Document and Field Objects

Zend_Search_Lucene operates with documents as atomic objects for indexing. A document is divided into named fields, and fields have content that can be searched.

A document is represented by the Zend_Search_Lucene_Document class, and this objects of this class contain instances of Zend_Search_Lucene_Field that represent the fields on the document.

It is important to note that any information can be added to the index. Application-specific information or metadata can be stored in the document fields, and later retrieved with the document during search.

It is the responsibility of your application to control the indexer. This means that data can be indexed from any source that is accessible by your application. For example, this could be the filesystem, a database, an HTML form, etc.

Zend_Search_Lucene_Field class provides several static methods to create fields with different characteristics:

$doc = new Zend_Search_Lucene_Document();

// Field is not tokenized, but is indexed and stored within the index.
// Stored fields can be retrived from the index.
$doc->addField(Zend_Search_Lucene_Field::Keyword('doctype',
                                                 'autogenerated'));

// Field is not tokenized nor indexed, but is stored in the index.
$doc->addField(Zend_Search_Lucene_Field::UnIndexed('created',
                                                   time()));

// Binary String valued Field that is not tokenized nor indexed,
// but is stored in the index.
$doc->addField(Zend_Search_Lucene_Field::Binary('icon',
                                                $iconData));

// Field is tokenized and indexed, and is stored in the index.
$doc->addField(Zend_Search_Lucene_Field::Text('annotation',
                                              'Document annotation text'));

// Field is tokenized and indexed, but is not stored in the index.
$doc->addField(Zend_Search_Lucene_Field::UnStored('contents',
                                                  'My document content'));

Each of these methods (excluding the Zend_Search_Lucene_Field::Binary() method) has an optional $encoding parameter for specifying input data encoding.

Encoding may differ for different documents as well as for different fields within one document:

$doc = new Zend_Search_Lucene_Document();
$doc->addField(Zend_Search_Lucene_Field::Text('title',
                                              $title,
                                              'iso-8859-1'));
$doc->addField(Zend_Search_Lucene_Field::UnStored('contents',
                                                  $contents,
                                                  'utf-8'));

If encoding parameter is omitted, then the current locale is used at processing time. For example:

setlocale(LC_ALL, 'de_DE.iso-8859-1');
...
$doc->addField(Zend_Search_Lucene_Field::UnStored('contents', $contents));

Fields are always stored and returned from the index in UTF-8 encoding. Any required conversion to UTF-8 happens automatically.

Text analyzers (see below) may also convert text to some other encodings. Actually, the default analyzer converts text to 'ASCII//TRANSLIT' encoding. Be careful, however; this translation may depend on current locale.

Fields' names are defined at your discretion in the addField() method.

Java Lucene uses the 'contents' field as a default field to search. Zend_Search_Lucene searches through all fields by default, but the behavior is configurable. See the "Default search field" chapter for details.

Understanding Field Types

  • Keyword fields are stored and indexed, meaning that they can be searched as well as displayed in search results. They are not split up into separate words by tokenization. Enumerated database fields usually translate well to Keyword fields in Zend_Search_Lucene.

  • UnIndexed fields are not searchable, but they are returned with search hits. Database timestamps, primary keys, file system paths, and other external identifiers are good candidates for UnIndexed fields.

  • Binary fields are not tokenized or indexed, but are stored for retrieval with search hits. They can be used to store any data encoded as a binary string, such as an image icon.

  • Text fields are stored, indexed, and tokenized. Text fields are appropriate for storing information like subjects and titles that need to be searchable as well as returned with search results.

  • UnStored fields are tokenized and indexed, but not stored in the index. Large amounts of text are best indexed using this type of field. Storing data creates a larger index on disk, so if you need to search but not redisplay the data, use an UnStored field. UnStored fields are practical when using a Zend_Search_Lucene index in combination with a relational database. You can index large data fields with UnStored fields for searching, and retrieve them from your relational database by using a separate field as an identifier.

    Zend_Search_Lucene_Field Types
    Field Type Stored Indexed Tokenized Binary
    Keyword Yes Yes No No
    UnIndexed Yes No No No
    Binary Yes No No Yes
    Text Yes Yes Yes No
    UnStored No Yes Yes No

HTML documents

Zend_Search_Lucene offers a HTML parsing feature. Documents can be created directly from a HTML file or string:

$doc = Zend_Search_Lucene_Document_Html::loadHTMLFile($filename);
$index->addDocument($doc);
...
$doc = Zend_Search_Lucene_Document_Html::loadHTML($htmlString);
$index->addDocument($doc);

Zend_Search_Lucene_Document_Html class uses the DOMDocument::loadHTML() and DOMDocument::loadHTMLFile() methods to parse the source HTML, so it doesn't need HTML to be well formed or to be XHTML. On the other hand, it's sensitive to the encoding specified by the "meta http-equiv" header tag.

Zend_Search_Lucene_Document_Html class recognizes document title, body and document header meta tags.

The 'title' field is actually the /html/head/title value. It's stored within the index, tokenized and available for search.

The 'body' field is the actual body content of the HTML file or string. It doesn't include scripts, comments or attributes.

The loadHTML() and loadHTMLFile() methods of Zend_Search_Lucene_Document_Html class also have second optional argument. If it's set to true, then body content is also stored within index and can be retrieved from the index. By default, the body is tokenized and indexed, but not stored.

The third parameter of loadHTML() and loadHTMLFile() methods optionally specifies source HTML document encoding. It's used if encoding is not specified using Content-type HTTP-EQUIV meta tag.

Other document header meta tags produce additional document fields. The field 'name' is taken from 'name' attribute, and the 'content' attribute populates the field 'value'. Both are tokenized, indexed and stored, so documents may be searched by their meta tags (for example, by keywords).

Parsed documents may be augmented by the programmer with any other field:

$doc = Zend_Search_Lucene_Document_Html::loadHTML($htmlString);
$doc->addField(Zend_Search_Lucene_Field::UnIndexed('created',
                                                   time()));
$doc->addField(Zend_Search_Lucene_Field::UnIndexed('updated',
                                                   time()));
$doc->addField(Zend_Search_Lucene_Field::Text('annotation',
                                              'Document annotation text'));
$index->addDocument($doc);

Document links are not included in the generated document, but may be retrieved with the Zend_Search_Lucene_Document_Html::getLinks() and Zend_Search_Lucene_Document_Html::getHeaderLinks() methods:

$doc = Zend_Search_Lucene_Document_Html::loadHTML($htmlString);
$linksArray = $doc->getLinks();
$headerLinksArray = $doc->getHeaderLinks();

Starting from Zend Framework 1.6 it's also possible to exclude links with rel attribute set to 'nofollow'. Use Zend_Search_Lucene_Document_Html::setExcludeNoFollowLinks($true) to turn on this option.

Zend_Search_Lucene_Document_Html::getExcludeNoFollowLinks() method returns current state of "Exclude nofollow links" flag.

Word 2007 documents

Zend_Search_Lucene offers a Word 2007 parsing feature. Documents can be created directly from a Word 2007 file:

$doc = Zend_Search_Lucene_Document_Docx::loadDocxFile($filename);
$index->addDocument($doc);

Zend_Search_Lucene_Document_Docx class uses the ZipArchive class and simplexml methods to parse the source document. If the ZipArchive class (from module php_zip) is not available, the Zend_Search_Lucene_Document_Docx will also not be available for use with Zend Framework.

Zend_Search_Lucene_Document_Docx class recognizes document meta data and document text. Meta data consists, depending on document contents, of filename, title, subject, creator, keywords, description, lastModifiedBy, revision, modified, created.

The 'filename' field is the actual Word 2007 file name.

The 'title' field is the actual document title.

The 'subject' field is the actual document subject.

The 'creator' field is the actual document creator.

The 'keywords' field contains the actual document keywords.

The 'description' field is the actual document description.

The 'lastModifiedBy' field is the username who has last modified the actual document.

The 'revision' field is the actual document revision number.

The 'modified' field is the actual document last modified date / time.

The 'created' field is the actual document creation date / time.

The 'body' field is the actual body content of the Word 2007 document. It only includes normal text, comments and revisions are not included.

The loadDocxFile() methods of Zend_Search_Lucene_Document_Docx class also have second optional argument. If it's set to true, then body content is also stored within index and can be retrieved from the index. By default, the body is tokenized and indexed, but not stored.

Parsed documents may be augmented by the programmer with any other field:

$doc = Zend_Search_Lucene_Document_Docx::loadDocxFile($filename);
$doc->addField(Zend_Search_Lucene_Field::UnIndexed(
    'indexTime',
    time())
);
$doc->addField(Zend_Search_Lucene_Field::Text(
    'annotation',
    'Document annotation text')
);
$index->addDocument($doc);

Powerpoint 2007 documents

Zend_Search_Lucene offers a Powerpoint 2007 parsing feature. Documents can be created directly from a Powerpoint 2007 file:

$doc = Zend_Search_Lucene_Document_Pptx::loadPptxFile($filename);
$index->addDocument($doc);

Zend_Search_Lucene_Document_Pptx class uses the ZipArchive class and simplexml methods to parse the source document. If the ZipArchive class (from module php_zip) is not available, the Zend_Search_Lucene_Document_Pptx will also not be available for use with Zend Framework.

Zend_Search_Lucene_Document_Pptx class recognizes document meta data and document text. Meta data consists, depending on document contents, of filename, title, subject, creator, keywords, description, lastModifiedBy, revision, modified, created.

The 'filename' field is the actual Powerpoint 2007 file name.

The 'title' field is the actual document title.

The 'subject' field is the actual document subject.

The 'creator' field is the actual document creator.

The 'keywords' field contains the actual document keywords.

The 'description' field is the actual document description.

The 'lastModifiedBy' field is the username who has last modified the actual document.

The 'revision' field is the actual document revision number.

The 'modified' field is the actual document last modified date / time.

The 'created' field is the actual document creation date / time.

The 'body' field is the actual content of all slides and slide notes in the Powerpoint 2007 document.

The loadPptxFile() methods of Zend_Search_Lucene_Document_Pptx class also have second optional argument. If it's set to true, then body content is also stored within index and can be retrieved from the index. By default, the body is tokenized and indexed, but not stored.

Parsed documents may be augmented by the programmer with any other field:

$doc = Zend_Search_Lucene_Document_Pptx::loadPptxFile($filename);
$doc->addField(Zend_Search_Lucene_Field::UnIndexed(
    'indexTime',
    time()));
$doc->addField(Zend_Search_Lucene_Field::Text(
    'annotation',
    'Document annotation text'));
$index->addDocument($doc);

Excel 2007 documents

Zend_Search_Lucene offers a Excel 2007 parsing feature. Documents can be created directly from a Excel 2007 file:

$doc = Zend_Search_Lucene_Document_Xlsx::loadXlsxFile($filename);
$index->addDocument($doc);

Zend_Search_Lucene_Document_Xlsx class uses the ZipArchive class and simplexml methods to parse the source document. If the ZipArchive class (from module php_zip) is not available, the Zend_Search_Lucene_Document_Xlsx will also not be available for use with Zend Framework.

Zend_Search_Lucene_Document_Xlsx class recognizes document meta data and document text. Meta data consists, depending on document contents, of filename, title, subject, creator, keywords, description, lastModifiedBy, revision, modified, created.

The 'filename' field is the actual Excel 2007 file name.

The 'title' field is the actual document title.

The 'subject' field is the actual document subject.

The 'creator' field is the actual document creator.

The 'keywords' field contains the actual document keywords.

The 'description' field is the actual document description.

The 'lastModifiedBy' field is the username who has last modified the actual document.

The 'revision' field is the actual document revision number.

The 'modified' field is the actual document last modified date / time.

The 'created' field is the actual document creation date / time.

The 'body' field is the actual content of all cells in all worksheets of the Excel 2007 document.

The loadXlsxFile() methods of Zend_Search_Lucene_Document_Xlsx class also have second optional argument. If it's set to true, then body content is also stored within index and can be retrieved from the index. By default, the body is tokenized and indexed, but not stored.

Parsed documents may be augmented by the programmer with any other field:

$doc = Zend_Search_Lucene_Document_Xlsx::loadXlsxFile($filename);
$doc->addField(Zend_Search_Lucene_Field::UnIndexed(
    'indexTime',
    time()));
$doc->addField(Zend_Search_Lucene_Field::Text(
    'annotation',
    'Document annotation text'));
$index->addDocument($doc);

Previous Next
Introduction to Zend Framework
Overview
Installation
Zend_Acl
Introduction
Refining Access Controls
Advanced Usage
Zend_Amf
Introduction
Zend_Amf_Server
Zend_Application
Introduction
Zend_Application Quick Start
Theory of Operation
Examples
Core Functionality
Available Resource Plugins
Zend_Auth
Introduction
Database Table Authentication
Digest Authentication
HTTP Authentication Adapter
LDAP Authentication
Open ID Authentication
Zend_Cache
Introduction
The Theory of Caching
Zend_Cache Frontends
Zend_Cache Backends
Zend_Captcha
Introduction
Captcha Operation
CAPTCHA Adapters
Zend_CodeGenerator
Introduction
Zend_CodeGenerator Examples
Zend_CodeGenerator Reference
Zend_Config
Introduction
Theory of Operation
Zend_Config_Ini
Zend_Config_Xml
Zend_Config_Writer
Zend_Config_Writer
Zend_Console_Getopt
Introduction
Declaring Getopt Rules
Fetching Options and Arguments
Configuring Zend_Console_Getopt
Zend_Controller
Zend_Controller Quick Start
Zend_Controller Basics
The Front Controller
The Request Object
The Standard Router
The Dispatcher
Action Controllers
Action Helpers
The Response Object
Plugins
Using a Conventional Modular Directory Structure
MVC Exceptions
Migrating from Previous Versions
Zend_Currency
Introduction to Zend_Currency
How to Work with Currencies
Migrating from Previous Versions
Zend_Date
Introduction
Theory of Operation
Basic Methods
Zend_Date API Overview
Creation of Dates
Constants for General Date Functions
Working Examples
Zend_Db
Zend_Db_Adapter
Zend_Db_Statement
Zend_Db_Profiler
Zend_Db_Select
Zend_Db_Table
Zend_Db_Table_Row
Zend_Db_Table_Rowset
Zend_Db_Table Relationships
Zend_Db_Table_Definition
Zend_Debug
Dumping Variables
Zend_Dojo
Introduction
Zend_Dojo_Data: dojo.data Envelopes
Dojo View Helpers
Dojo Form Elements and Decorators
Zend_Dojo build layer support
Zend_Dom
Introduction
Zend_Dom_Query
Zend_Exception
Using Exceptions
Zend_Feed
Introduction
Importing Feeds
Retrieving Feeds from Web Pages
Consuming an RSS Feed
Consuming an Atom Feed
Consuming a Single Atom Entry
Modifying Feed and Entry structures
Custom Feed and Entry Classes
Zend_Feed_Reader
Zend_File
Zend_File_Transfer
Validators for Zend_File_Transfer
Filters for Zend_File_Transfer
Migrating from previous versions
Zend_Filter
Introduction
Standard Filter Classes
Filter Chains
Writing Filters
Zend_Filter_Input
Zend_Filter_Inflector
Migrating from Previous Versions
Zend_Form
Zend_Form
Zend_Form Quick Start
Creating Form Elements Using Zend_Form_Element
Creating Forms Using Zend_Form
Creating Custom Form Markup Using Zend_Form_Decorator
Standard Form Elements Shipped With Zend Framework
Standard Form Decorators Shipped With Zend Framework
Internationalization of Zend_Form
Advanced Zend_Form Usage
Zend_Gdata
Introduction
Authenticating with AuthSub
Using the Book Search Data API
Authenticating with ClientLogin
Using Google Calendar
Using Google Documents List Data API
Using Google Health
Using Google Spreadsheets
Using Google Apps Provisioning
Using Google Base
Using Picasa Web Albums
Using the YouTube Data API
Catching Gdata Exceptions
Zend_Http
Introduction
Zend_Http_Client - Advanced Usage
Zend_Http_Client - Connection Adapters
Migrating from previous versions
Zend_Http_Cookie and Zend_Http_CookieJar
Zend_Http_Response
Zend_InfoCard
Introduction
Zend_Json
Introduction
Basic Usage
Advanced Usage of Zend_Json
XML to JSON conversion
Zend_Json_Server - JSON-RPC server
Zend_Layout
Introduction
Zend_Layout Quick Start
Zend_Layout Configuration Options
Zend_Layout Advanced Usage
Zend_Ldap
Introduction
API overview
Usage Scenarios
Tools
Object oriented access to the LDAP tree using Zend_Ldap_Node
Getting information from the LDAP server
Serializing LDAP data to and from LDIF
Zend_Loader
Loading Files and Classes Dynamically
The Autoloader
Resource Autoloaders
Loading Plugins
Zend_Locale
Introduction
Using Zend_Locale
Normalization and Localization
Working with Dates and Times
Supported locales
Migrating from previous versions
Zend_Log
Overview
Writers
Formatters
Filters
Zend_Mail
Introduction
Sending via SMTP
Sending Multiple Mails per SMTP Connection
Using Different Transports
HTML E-Mail
Attachments
Adding Recipients
Controlling the MIME Boundary
Additional Headers
Character Sets
Encoding
SMTP Authentication
Securing SMTP Transport
Reading Mail Messages
Zend_Measure
Introduction
Creation of Measurements
Outputting measurements
Manipulating Measurements
Types of measurements
Zend_Memory
Overview
Memory Manager
Memory Objects
Zend_Mime
Zend_Mime
Zend_Mime_Message
Zend_Mime_Part
Zend_Navigation
Introduction
Pages
Containers
Migrating from Previous Versions
Zend_OpenId
Introduction
Zend_OpenId_Consumer Basics
Zend_OpenId_Provider
Zend_Paginator
Introduction
Usage
Configuration
Advanced usage
Zend_Pdf
Introduction
Creating and Loading PDF Documents
Save Changes to PDF Documents
Working with Pages
Drawing
Interactive Features
Document Info and Metadata
Zend_Pdf module usage example
Zend_ProgressBar
Zend_ProgressBar
Zend_Queue
Introduction
Example usage
Framework
Adapters
Customizing Zend_Queue
Stomp
Zend_Reflection
Introduction
Zend_Reflection Examples
Zend_Reflection Reference
Zend_Registry
Using the Registry
Zend_Rest
Introduction
Zend_Rest_Client
Zend_Rest_Server
Zend_Search_Lucene
Overview
Building Indexes
Searching an Index
Query Language
Query Construction API
Character Set
Extensibility
Interoperating with Java Lucene
Advanced
Best Practices
Zend_Server
Introduction
Zend_Server_Reflection
Zend_Service
Introduction
Zend_Service_Akismet
Zend_Service_Amazon
Zend_Service_Amazon_Ec2
Zend_Service_Amazon_Ec2: Instances
Zend_Service_Amazon_Ec2: Windows Instances
Zend_Service_Amazon_Ec2: Reserved Instances
Zend_Service_Amazon_Ec2: CloudWatch Monitoring
Zend_Service_Amazon_Ec2: Amazon Machine Images (AMI)
Zend_Service_Amazon_Ec2: Elastic Block Stroage (EBS)
Zend_Service_Amazon_Ec2: Elastic IP Addresses
Zend_Service_Amazon_Ec2: Keypairs
Zend_Service_Amazon_Ec2: Regions and Availability Zones
Zend_Service_Amazon_Ec2: Security Groups
Zend_Service_Amazon_S3
Zend_Service_Amazon_Sqs
Zend_Service_Audioscrobbler
Zend_Service_Delicious
Zend_Service_Flickr
Zend_Service_Nirvanix
Zend_Service_ReCaptcha
Zend_Service_Simpy
Introduction
Zend_Service_StrikeIron
Zend_Service_StrikeIron: Bundled Services
Zend_Service_StrikeIron: Advanced Uses
Zend_Service_Technorati
Zend_Service_Twitter
Zend_Service_Yahoo
Zend_Session
Introduction
Basic Usage
Advanced Usage
Global Session Management
Zend_Session_SaveHandler_DbTable
Zend_Soap
Zend_Soap_Server
Zend_Soap_Client
WSDL Accessor
AutoDiscovery
Zend_Tag
Introduction
Zend_Tag_Cloud
Zend_Test
Introduction
Zend_Test_PHPUnit
Zend_Test_PHPUnit_Db
Zend_Text
Zend_Text_Figlet
Zend_Text_Table
Zend_TimeSync
Introduction
Working with Zend_TimeSync
Zend_Tool_Framework
Introduction
Using the CLI Tool
Architecture
Creating Providers to use with Zend_Tool_Framework
Shipped System Providers
Extending and Configuring Zend_Tool_Framework
Zend_Tool_Project
Introduction
Create A Project
Zend Tool Project Providers
Zend_Translate
Introduction
Adapters for Zend_Translate
Using Translation Adapters
Creating source files
Additional features for translation
Plural notations for Translation
Migrating from previous versions
Zend_Uri
Zend_Uri
Zend_Validate
Introduction
Standard Validation Classes
Validator Chains
Writing Validators
Validation Messages
Zend_Version
Getting the Zend Framework Version
Zend_View
Introduction
Controller Scripts
View Scripts
View Helpers
Zend_View_Abstract
Migrating from Previous Versions
Zend_Wildfire
Zend_Wildfire
Zend_XmlRpc
Introduction
Zend_XmlRpc_Client
Zend_XmlRpc_Server
Zend Framework Requirements
Introduction
Zend Framework Coding Standard for PHP
Overview
PHP File Formatting
Naming Conventions
Coding Style
Zend Framework Documentation Standard
Overview
Documentation File Formatting
Recommendations
Recommended Project Structure for Zend Framework MVC Applications
Overview
Recommended Project Directory Structure
Module Structure
Rewrite Configuration Guide
Zend Framework Performance Guide
Introduction
Class Loading
Zend_Db Performance
Internationalization (i18n) and Localization (l10n)
View Rendering
Copyright Information