Previous Next

Building Indexes

Creating a New Index

Index creation and updating capabilities are implemented within the Zend_Search_Lucene component, as well as the Java Lucene project. You can use either of these options to create indexes that Zend_Search_Lucene can search.

The PHP code listing below provides an example of how to index a file using Zend_Search_Lucene indexing API:

// Create index
$index = Zend_Search_Lucene::create('/data/my-index');

$doc = new Zend_Search_Lucene_Document();

// Store document URL to identify it in the search results
$doc->addField(Zend_Search_Lucene_Field::Text('url', $docUrl));

// Index document contents
$doc->addField(Zend_Search_Lucene_Field::UnStored('contents', $docContent));

// Add document to the index
$index->addDocument($doc);

Newly added documents are immediately searchable in the index.

Updating Index

The same procedure is used to update an existing index. The only difference is that the open() method is called instead of the create() method:

// Open existing index
$index = Zend_Search_Lucene::open('/data/my-index');

$doc = new Zend_Search_Lucene_Document();
// Store document URL to identify it in search result.
$doc->addField(Zend_Search_Lucene_Field::Text('url', $docUrl));
// Index document content
$doc->addField(Zend_Search_Lucene_Field::UnStored('contents',
                                                  $docContent));

// Add document to the index.
$index->addDocument($doc);

Updating Documents

The Lucene index file format doesn't support document updating. Documents should be removed and re-added to the index to effectively update them.

Zend_Search_Lucene::delete() method operates with an internal index document id. It can be retrieved from a query hit by 'id' property:

$removePath = ...;
$hits = $index->find('path:' . $removePath);
foreach ($hits as $hit) {
    $index->delete($hit->id);
}

Retrieving Index Size

There are two methods to retrieve the size of an index in Zend_Search_Lucene.

Zend_Search_Lucene::maxDoc() returns one greater than the largest possible document number. It's actually the overall number of the documents in the index including deleted documents, so it has a synonym: Zend_Search_Lucene::count().

Zend_Search_Lucene::numDocs() returns the total number of non-deleted documents.

$indexSize = $index->count();
$documents = $index->numDocs();

Zend_Search_Lucene::isDeleted($id) method may be used to check if a document is deleted.

for ($count = 0; $count < $index->maxDoc(); $count++) {
    if ($index->isDeleted($count)) {
        echo "Document #$id is deleted.\n";
    }
}

Index optimization removes deleted documents and squeezes documents' IDs in to a smaller range. A document's internal id may therefore change during index optimization.

Index optimization

A Lucene index consists of many segments. Each segment is a completely independent set of data.

Lucene index segment files can't be updated by design. A segment update needs full segment reorganization. See Lucene index file formats for details (» http://lucene.apache.org/java/docs/fileformats.html) [1] . New documents are added to the index by creating new segment.

Increasing number of segments reduces quality of the index, but index optimization restores it. Optimization essentially merges several segments into a new one. This process also doesn't update segments. It generates one new large segment and updates segment list ('segments' file).

Full index optimization can be trigger by calling the Zend_Search_Lucene::optimize() method. It merges all index segments into one new segment:

// Open existing index
$index = Zend_Search_Lucene::open('/data/my-index');

// Optimize index.
$index->optimize();

Automatic index optimization is performed to keep indexes in a consistent state.

Automatic optimization is an iterative process managed by several index options. It merges very small segments into larger ones, then merges these larger segments into even larger segments and so on.

MaxBufferedDocs auto-optimization option

MaxBufferedDocs is a minimal number of documents required before the buffered in-memory documents are written into a new segment.

MaxBufferedDocs can be retrieved or set by $index->getMaxBufferedDocs() or $index->setMaxBufferedDocs($maxBufferedDocs) calls.

Default value is 10.

MaxMergeDocs auto-optimization option

MaxMergeDocs is a largest number of documents ever merged by addDocument(). Small values (e.g., less than 10.000) are best for interactive indexing, as this limits the length of pauses while indexing to a few seconds. Larger values are best for batched indexing and speedier searches.

MaxMergeDocs can be retrieved or set by $index->getMaxMergeDocs() or $index->setMaxMergeDocs($maxMergeDocs) calls.

Default value is PHP_INT_MAX.

MergeFactor auto-optimization option

MergeFactor determines how often segment indices are merged by addDocument(). With smaller values, less RAM is used while indexing, and searches on unoptimized indices are faster, but indexing speed is slower. With larger values, more RAM is used during indexing, and while searches on unoptimized indices are slower, indexing is faster. Thus larger values (> 10) are best for batch index creation, and smaller values (< 10) for indices that are interactively maintained.

MergeFactor is a good estimation for average number of segments merged by one auto-optimization pass. Too large values produce large number of segments while they are not merged into new one. It may be a cause of "failed to open stream: Too many open files" error message. This limitation is system dependent.

MergeFactor can be retrieved or set by $index->getMergeFactor() or $index->setMergeFactor($mergeFactor) calls.

Default value is 10.

Lucene Java and Luke (Lucene Index Toolbox - » http://www.getopt.org/luke/) can also be used to optimize an index. Latest Luke release (v0.8) is based on Lucene v2.3 and compatible with current implementation of Zend_Search_Lucene component (ZF 1.6). Earlier versions of Zend_Search_Lucene implementations need another versions of Java Lucene tools to be compatible:

  • ZF 1.5 - Java Lucene 2.1 (Luke tool v0.7.1 - http://www.getopt.org/luke/luke-0.7.1/)

  • ZF 1.0 - Java Lucene 1.4 - 2.1 (Luke tool v0.6 - http://www.getopt.org/luke/luke-0.6/)

Permissions

By default, index files are available for reading and writing by everyone.

It's possible to override this with the Zend_Search_Lucene_Storage_Directory_Filesystem::setDefaultFilePermissions() method:

// Get current default file permissions
$currentPermissions =
    Zend_Search_Lucene_Storage_Directory_Filesystem::getDefaultFilePermissions();

// Give read-writing permissions only for current user and group
Zend_Search_Lucene_Storage_Directory_Filesystem::setDefaultFilePermissions(0660);

Limitations

Index size

Index size is limited by 2GB for 32-bit platforms.

Use 64-bit platforms for larger indices.

Supported Filesystems

Zend_Search_Lucene uses flock() to provide concurrent searching, index updating and optimization.

According to the PHP » documentation, "flock() will not work on NFS and many other networked file systems".

Do not use networked file systems with Zend_Search_Lucene.

[1]The currently supported Lucene index file format is version 2.3 (starting from ZF 1.6).
Previous Next
Introduction to Zend Framework
Présentation
Installation
Zend_Acl
Introduction
Affiner les Contrôles d'Accès
Utilisation avancée
Zend_Amf
Introduction
Zend_Amf_Server
Zend_Auth
Introduction
Authentification avec une table de base de données
Authentification "Digest"
Adaptateur d'authentification HTTP
LDAP Authentication
Authentification OpenID
Zend_Cache
Introduction
Aspect théorique
Les frontends Zend_Cache
Les backends Zend_Cache
Zend_Captcha
Introduction
Opération Captcha
Adaptateurs Captcha
Zend_Config
Introduction
Aspect théorique
Zend_Config_Ini
Zend_Config_Xml
Zend_Config_Writer
Zend_Config_Writer
Zend_Console_Getopt
Introduction à Getopt
Déclarer les règles Getopt
Extraire les options et les arguments
Configurer Zend_Console_Getopt
Zend_Controller
Zend_Controller - Démarrage rapide
Fondations de Zend_Controller
Le contrôleur frontal (Front Controller)
L'objet Requête
Routeur Standard
Le dispatcheur
Contrôleurs d'action
Aides d'action (Helper)
Objet de réponse
Plugins
Utilisation de conventions de dossiers modulaires
Exceptions avec MVC
Migrer depuis des versions précédentes
Zend_Currency
Introduction à Zend_Currency
How to work with currencies
Migrer depuis des versions antérieures
Zend_Date
Introduction
Aspect théorique
Méthodes de base
Zend_Date API Overview
Créer des dates
Constants for General Date Functions
Exemples concrets
Zend_Db
Zend_Db_Adapter
Zend_Db_Statement
Zend_Db_Profiler
Zend_Db_Select
Zend_Db_Table
Zend_Db_Table_Row
Zend_Db_Table_Rowset
Relations Zend_Db_Table
Zend_Debug
Afficher des informations
Zend_Dojo
Introduction
Zend_Dojo_Data: dojo.data Envelopes
Les aides de vues Dojo
Les éléments de formulaire et les décorateurs Dojo
Zend_Dom
Introduction
Zend_Dom_Query
Zend_Exception
Utiliser les exceptions
Zend_Feed
Introduction
Importer des flux
Obtenir des flux à partir de pages Web
Consommer un flux RSS
Consommer un flux Atom
Consommer une entrée Atom particulière
Modifier la structure du flux ou des entrées
Classes personnalisées pour les flux et entrées
Zend_File
Zend_File_Transfer
Validateurs pour Zend_File_Transfer
Filtres pour Zend_File_Transfer
Migrer à partir des versions précédentes
Zend_Filter
Introduction
Classes de filtre standards
Chaînes de filtrage
Écriture de filtres
Zend_Filter_Input
Zend_Filter_Inflector
Zend_Form
Zend_Form
Zend_Form Quick Start
Creating Form Elements Using Zend_Form_Element
Creating Forms Using Zend_Form
Creating Custom Form Markup Using Zend_Form_Decorator
Standard Form Elements Shipped With Zend Framework
Standard Form Decorators Shipped With Zend Framework
Internationalization of Zend_Form
Advanced Zend_Form Usage
Zend_Gdata
Introduction to Gdata
Authentification par procédé AuthSub
Using the Book Search Data API
Authentification avec ClientLogin
Using Google Calendar
Using Google Documents List Data API
Using Google Health
Using Google Spreadsheets
Using Google Apps Provisioning
Using Google Base
Utilisation des albums Web Picasa
Using the YouTube Data API
Attraper les exceptions Gdata
Zend_Http
Zend_Http_Client - Introduction
Zend_Http_Client - Utilisation avancée
Zend_Http_Client - Adaptateurs de connexion
Zend_Http_Cookie and Zend_Http_CookieJar
Zend_Http_Response
Zend_InfoCard
Introduction
Zend_Json
Introduction
Utilisation de base
Objets JSON
XML to JSON conversion
Zend_Json_Server - JSON-RPC server
Zend_Layout
Introduction
Zend_Layout - Démarrage rapide
Zend_Layout options de configuration
Zend_Layout, utilisation avancée
Zend_Ldap
Introduction
Zend_Loader
Charger les fichiers et les classes dynamiquement
Chargeur de Plugins
Zend_Locale
Introduction
Using Zend_Locale
Normalization and Localization
Working with Dates and Times
Supported locales
Migrer à partir des versions précédentes
Zend_Log
Présentation
Rédacteurs (Writers)
Formateurs (mise en forme)
Filtres
Zend_Mail
Introduction
Envoyer des émail en utilisant SMTP
Envoyer plusieurs émail par connexion SMTP
Utiliser différents transports
Émail HTML
Fichiers joints
Ajouter des destinataires
Contrôler les limites MIME
Entêtes additionnelles
Jeux de caractères
Encodage
Authentification SMTP
Sécuriser les transports SMTP
Lire des émail
Zend_Measure
Introduction
Création d'une mesure
Récupérer des mesures
Manipuler des mesures
Types de mesures
Zend_Memory
Présentation
Manager de mémoire
Objet mémoire
Zend_Mime
Zend_Mime
Zend_Mime_Message
Zend_Mime_Part
Zend_OpenId
Introduction
Zend_OpenId_Consumer Basics
Zend_OpenId_Provider
Zend_Paginator
Introduction
Utilisation
Configuration
Utilisation avancée
Zend_Pdf
Introduction.
Créer et charger des documents PDF
Sauvegarder les changement dans un document PDF
Les pages d'un document
Dessiner
Informations du document et métadonnées.
Exemple d'utilisation du module Zend_Pdf
Zend_ProgressBar
Zend_ProgressBar
Zend_Registry
Utiliser le registre
Zend_Rest
Introduction
Zend_Rest_Client
Zend_Rest_Server
Zend_Search_Lucene
Overview
Building Indexes
Searching an Index
Query Language
Query Construction API
Jeu de caractères
Extensibility
Agir avec Lucene Java
Avancé
Best Practices
Zend_Server
Introduction
Zend_Server_Reflection
Zend_Service
Introduction
Zend_Service_Akismet
Zend_Service_Amazon
Zend_Service_Audioscrobbler
Zend_Service_Delicious
Zend_Service_Flickr
Zend_Service_Nirvanix
Zend_Service_ReCaptcha
Zend_Service_Simpy
Introduction
Zend_Service_StrikeIron
Zend_Service_StrikeIron: Bundled Services
Zend_Service_StrikeIron: Advanced Uses
Zend_Service_Technorati
Zend_Service_Twitter
Zend_Service_Yahoo
Zend_Session
Introduction
Usage basique
Utilisation avancée
Gestion générale de la session
Zend_Session_SaveHandler_DbTable
Zend_Soap
Zend_Soap_Server
Zend_Soap_Client
WSDL
Auto découverte
Zend_Test
Introduction
Zend_Test_PHPUnit
Zend_Text
Zend_Text_Figlet
Zend_Text_Table
Zend_TimeSync
Introduction
Utiliser Zend_TimeSync
Zend_Translate
Introduction
Adaptateurs pour Zend_Translate
Utiliser les adaptateurs de traduction
Migrer à partir des versions précédentes
Zend_Uri
Zend_Uri
Zend_Validate
Introduction
Classes de validation standard
Chaînes de validation
Écrire des validateurs
Zend_Version
Lire la version du Zend Framework
Zend_View
Introduction
Scripts de contrôleur
Scripts de vue
Aides de vue
Zend_View_Abstract
Zend_Wildfire
Zend_Wildfire
Zend_XmlRpc
Introduction
Zend_XmlRpc_Client
Zend_XmlRpc_Server
Configuration système requise par le Zend Framework
Version de PHP requise
Extensions PHP
Les composants du Zend Framework
Dépendances internes du Zend Framework
Convention de codage PHP du Zend Framework
Vue d'ensemble
Formatage des fichiers PHP
Conventions de nommage
Style de codage
Zend Framework Performance Guide
Introduction
Class Loading
Internationalisation (i18n) and Localisation (l10n)
View Rendering
Informations de copyright