Previous Next

Extensibility

Text Analysis

The Zend_Search_Lucene_Analysis_Analyzer class is used by the indexer to tokenize document text fields.

The Zend_Search_Lucene_Analysis_Analyzer::getDefault() and Zend_Search_Lucene_Analysis_Analyzer::setDefault() methods are used to get and set the default analyzer.

You can assign your own text analyzer or choose it from the set of predefined analyzers: Zend_Search_Lucene_Analysis_Analyzer_Common_Text and Zend_Search_Lucene_Analysis_Analyzer_Common_Text_CaseInsensitive (default). Both of them interpret tokens as sequences of letters. Zend_Search_Lucene_Analysis_Analyzer_Common_Text_CaseInsensitive converts all tokens to lower case.

To switch between analyzers:

Zend_Search_Lucene_Analysis_Analyzer::setDefault(
    new Zend_Search_Lucene_Analysis_Analyzer_Common_Text());
...
$index->addDocument($doc);

The Zend_Search_Lucene_Analysis_Analyzer_Common class is designed to be an ancestor of all user defined analyzers. User should only define the reset() and nextToken() methods, which takes its string from the $_input member and returns tokens one by one (a null value indicates the end of the stream).

The nextToken() method should call the normalize() method on each token. This will allow you to use token filters with your analyzer.

Here is an example of a custom analyzer, which accepts words with digits as terms:

Example #1 Custom text Analyzer.

/**
 * Here is a custom text analyser, which treats words with digits as
 * one term
 */

class My_Analyzer extends Zend_Search_Lucene_Analysis_Analyzer_Common
{
    private $_position;

    /**
     * Reset token stream
     */
    public function reset()
    {
        $this->_position = 0;
    }

    /**
     * Tokenization stream API
     * Get next token
     * Returns null at the end of stream
     *
     * @return Zend_Search_Lucene_Analysis_Token|null
     */
    public function nextToken()
    {
        if ($this->_input === null) {
            return null;
        }

        while ($this->_position < strlen($this->_input)) {
            // skip white space
            while ($this->_position < strlen($this->_input) &&
                   !ctype_alnum( $this->_input[$this->_position] )) {
                $this->_position++;
            }

            $termStartPosition = $this->_position;

            // read token
            while ($this->_position < strlen($this->_input) &&
                   ctype_alnum( $this->_input[$this->_position] )) {
                $this->_position++;
            }

            // Empty token, end of stream.
            if ($this->_position == $termStartPosition) {
                return null;
            }

            $token = new Zend_Search_Lucene_Analysis_Token(
                                      substr($this->_input,
                                             $termStartPosition,
                                             $this->_position -
                                             $termStartPosition),
                                      $termStartPosition,
                                      $this->_position);
            $token = $this->normalize($token);
            if ($token !== null) {
                return $token;
            }
            // Continue if token is skipped
        }

        return null;
    }
}

Zend_Search_Lucene_Analysis_Analyzer::setDefault(
    new My_Analyzer());

Tokens Filtering

The Zend_Search_Lucene_Analysis_Analyzer_Common analyzer also offers a token filtering mechanism.

The Zend_Search_Lucene_Analysis_TokenFilter class provides an abstract interface for such filters. Your own filters should extend this class either directly or indirectly.

Any custom filter must implement the normalize() method which may transform input token or signal that the current token should be skipped.

There are three filters already defined in the analysis subpackage:

  • Zend_Search_Lucene_Analysis_TokenFilter_LowerCase

  • Zend_Search_Lucene_Analysis_TokenFilter_ShortWords

  • Zend_Search_Lucene_Analysis_TokenFilter_StopWords

The LowerCase filter is already used for Zend_Search_Lucene_Analysis_Analyzer_Common_Text_CaseInsensitive analyzer by default.

The ShortWords and StopWords filters may be used with pre-defined or custom analyzers like this:

$stopWords = array('a', 'an', 'at', 'the', 'and', 'or', 'is', 'am');
$stopWordsFilter =
    new Zend_Search_Lucene_Analysis_TokenFilter_StopWords($stopWords);

$analyzer =
    new Zend_Search_Lucene_Analysis_Analyzer_Common_TextNum_CaseInsensitive();
$analyzer->addFilter($stopWordsFilter);

Zend_Search_Lucene_Analysis_Analyzer::setDefault($analyzer);
$shortWordsFilter = new Zend_Search_Lucene_Analysis_TokenFilter_ShortWords();

$analyzer =
    new Zend_Search_Lucene_Analysis_Analyzer_Common_TextNum_CaseInsensitive();
$analyzer->addFilter($shortWordsFilter);

Zend_Search_Lucene_Analysis_Analyzer::setDefault($analyzer);

The Zend_Search_Lucene_Analysis_TokenFilter_StopWords constructor takes an array of stop-words as an input. But stop-words may be also loaded from a file:

$stopWordsFilter = new Zend_Search_Lucene_Analysis_TokenFilter_StopWords();
$stopWordsFilter->loadFromFile($my_stopwords_file);

$analyzer =
   new Zend_Search_Lucene_Analysis_Analyzer_Common_TextNum_CaseInsensitive();
$analyzer->addFilter($stopWordsFilter);

Zend_Search_Lucene_Analysis_Analyzer::setDefault($analyzer);
This file should be a common text file with one word in each line. The '#' character marks a line as a comment.

The Zend_Search_Lucene_Analysis_TokenFilter_ShortWords constructor has one optional argument. This is the word length limit, set by default to 2.

Scoring Algorithms

The score of a document d for a query q is defined as follows:

score(q,d) = sum( tf(t in d) * idf(t) * getBoost(t.field in d) * lengthNorm(t.field in d) ) * coord(q,d) * queryNorm(q)

tf(t in d) - Zend_Search_Lucene_Search_Similarity::tf($freq) - a score factor based on the frequency of a term or phrase in a document.

idf(t) - Zend_Search_Lucene_Search_Similarity::tf($term, $reader) - a score factor for a simple term with the specified index.

getBoost(t.field in d) - the boost factor for the term field.

lengthNorm($term) - the normalization value for a field given the total number of terms contained in a field. This value is stored within the index. These values, together with field boosts, are stored in an index and multiplied into scores for hits on each field by the search code.

Matches in longer fields are less precise, so implementations of this method usually return smaller values when numTokens is large, and larger values when numTokens is small.

coord(q,d) - Zend_Search_Lucene_Search_Similarity::coord($overlap, $maxOverlap) - a score factor based on the fraction of all query terms that a document contains.

The presence of a large portion of the query terms indicates a better match with the query, so implementations of this method usually return larger values when the ratio between these parameters is large and smaller values when the ratio between them is small.

queryNorm(q) - the normalization value for a query given the sum of the squared weights of each of the query terms. This value is then multiplied into the weight of each query term.

This does not affect ranking, but rather just attempts to make scores from different queries comparable.

The scoring algorithm can be customized by defining your own Similarity class. To do this extend the Zend_Search_Lucene_Search_Similarity class as defined below, then use the Zend_Search_Lucene_Search_Similarity::setDefault($similarity); method to set it as default.

class MySimilarity extends Zend_Search_Lucene_Search_Similarity {
    public function lengthNorm($fieldName, $numTerms) {
        return 1.0/sqrt($numTerms);
    }

    public function queryNorm($sumOfSquaredWeights) {
        return 1.0/sqrt($sumOfSquaredWeights);
    }

    public function tf($freq) {
        return sqrt($freq);
    }

    /**
     * It's not used now. Computes the amount of a sloppy phrase match,
     * based on an edit distance.
     */
    public function sloppyFreq($distance) {
        return 1.0;
    }

    public function idfFreq($docFreq, $numDocs) {
        return log($numDocs/(float)($docFreq+1)) + 1.0;
    }

    public function coord($overlap, $maxOverlap) {
        return $overlap/(float)$maxOverlap;
    }
}

$mySimilarity = new MySimilarity();
Zend_Search_Lucene_Search_Similarity::setDefault($mySimilarity);

Storage Containers

The abstract class Zend_Search_Lucene_Storage_Directory defines directory functionality.

The Zend_Search_Lucene constructor uses either a string or a Zend_Search_Lucene_Storage_Directory object as an input.

The Zend_Search_Lucene_Storage_Directory_Filesystem class implements directory functionality for a file system.

If a string is used as an input for the Zend_Search_Lucene constructor, then the index reader (Zend_Search_Lucene object) treats it as a file system path and instantiates the Zend_Search_Lucene_Storage_Directory_Filesystem object.

You can define your own directory implementation by extending the Zend_Search_Lucene_Storage_Directory class.

Zend_Search_Lucene_Storage_Directory methods:

abstract class Zend_Search_Lucene_Storage_Directory {
/**
 * Closes the store.
 *
 * @return void
 */
abstract function close();

/**
 * Creates a new, empty file in the directory with the given $filename.
 *
 * @param string $name
 * @return void
 */
abstract function createFile($filename);

/**
 * Removes an existing $filename in the directory.
 *
 * @param string $filename
 * @return void
 */
abstract function deleteFile($filename);

/**
 * Returns true if a file with the given $filename exists.
 *
 * @param string $filename
 * @return boolean
 */
abstract function fileExists($filename);

/**
 * Returns the length of a $filename in the directory.
 *
 * @param string $filename
 * @return integer
 */
abstract function fileLength($filename);

/**
 * Returns the UNIX timestamp $filename was last modified.
 *
 * @param string $filename
 * @return integer
 */
abstract function fileModified($filename);

/**
 * Renames an existing file in the directory.
 *
 * @param string $from
 * @param string $to
 * @return void
 */
abstract function renameFile($from, $to);

/**
 * Sets the modified time of $filename to now.
 *
 * @param string $filename
 * @return void
 */
abstract function touchFile($filename);

/**
 * Returns a Zend_Search_Lucene_Storage_File object for a given
 * $filename in the directory.
 *
 * @param string $filename
 * @return Zend_Search_Lucene_Storage_File
 */
abstract function getFileObject($filename);

}

The getFileObject($filename) method of a Zend_Search_Lucene_Storage_Directory instance returns a Zend_Search_Lucene_Storage_File object.

The Zend_Search_Lucene_Storage_File abstract class implements file abstraction and index file reading primitives.

You must also extend Zend_Search_Lucene_Storage_File for your directory implementation.

Only two methods of Zend_Search_Lucene_Storage_File must be overridden in your implementation:

class MyFile extends Zend_Search_Lucene_Storage_File {
    /**
     * Sets the file position indicator and advances the file pointer.
     * The new position, measured in bytes from the beginning of the file,
     * is obtained by adding offset to the position specified by whence,
     * whose values are defined as follows:
     * SEEK_SET - Set position equal to offset bytes.
     * SEEK_CUR - Set position to current location plus offset.
     * SEEK_END - Set position to end-of-file plus offset. (To move to
     * a position before the end-of-file, you need to pass a negative value
     * in offset.)
     * Upon success, returns 0; otherwise, returns -1
     *
     * @param integer $offset
     * @param integer $whence
     * @return integer
     */
    public function seek($offset, $whence=SEEK_SET) {
        ...
    }

    /**
     * Read a $length bytes from the file and advance the file pointer.
     *
     * @param integer $length
     * @return string
     */
    protected function _fread($length=1) {
        ...
    }
}
Previous Next
Introduction to Zend Framework
Présentation
Installation
Zend_Acl
Introduction
Affiner les Contrôles d'Accès
Utilisation avancée
Zend_Amf
Introduction
Zend_Amf_Server
Zend_Application
Introduction
Zend_Application Quick Start
Théorie générale
Exemples
Core Functionality
Available Resource Plugins
Zend_Auth
Introduction
Authentification avec une table de base de données
Authentification "Digest"
Adaptateur d'authentification HTTP
LDAP Authentication
Authentification OpenID
Zend_Cache
Introduction
Aspect théorique
Les frontends Zend_Cache
Les backends Zend_Cache
Zend_Captcha
Introduction
Opération Captcha
Adaptateurs CAPTCHA
Zend_CodeGenerator
Introduction
Exemples Zend_CodeGenerator
Zend_CodeGenerator Réference
Zend_Config
Introduction
Aspect théorique
Zend_Config_Ini
Zend_Config_Xml
Zend_Config_Writer
Zend_Config_Writer
Zend_Console_Getopt
Introduction
Déclarer les règles Getopt
Extraire les options et les arguments
Configurer Zend_Console_Getopt
Zend_Controller
Zend_Controller - Démarrage rapide
Fondations de Zend_Controller
Le contrôleur frontal (Front Controller)
L'objet Requête
Routeur Standard
Le distributeur
Contrôleurs d'action
Aides d'action (Helper)
Objet de réponse
Plugins
Utilisation de conventions de dossiers modulaires
Exceptions avec MVC
Migrer depuis des versions précédentes
Zend_Currency
Introduction à Zend_Currency
How to Work with Currencies
Migrer depuis des versions antérieures
Zend_Date
Introduction
Aspect théorique
Méthodes de base
Zend_Date API Overview
Créer des dates
Constants for General Date Functions
Exemples concrets
Zend_Db
Zend_Db_Adapter
Zend_Db_Statement
Zend_Db_Profiler
Zend_Db_Select
Zend_Db_Table
Zend_Db_Table_Row
Zend_Db_Table_Rowset
Relations Zend_Db_Table
Zend_Debug
Afficher des informations
Zend_Dojo
Introduction
Zend_Dojo_Data: dojo.data Envelopes
Les aides de vues Dojo
Les éléments de formulaire et les décorateurs Dojo
Zend_Dom
Introduction
Zend_Dom_Query
Zend_Exception
Utiliser les exceptions
Zend_Feed
Introduction
Importer des flux
Obtenir des flux à partir de pages Web
Consommer un flux RSS
Consommer un flux Atom
Consommer une entrée Atom particulière
Modifier la structure du flux ou des entrées
Classes personnalisées pour les flux et entrées
Zend_File
Zend_File_Transfer
Validateurs pour Zend_File_Transfer
Filtres pour Zend_File_Transfer
Migrer à partir des versions précédentes
Zend_Filter
Introduction
Classes de filtre standards
Chaînes de filtrage
Écriture de filtres
Zend_Filter_Input
Zend_Filter_Inflector
Zend_Form
Zend_Form
Zend_Form démarrage rapide
Creating Form Elements Using Zend_Form_Element
Creating Forms Using Zend_Form
Créer un visuel personnalisé en utilisant Zend_Form_Decorator
Standard Form Elements Shipped With Zend Framework
Décorateurs standards fournis avec Zend Framework
Internationaliser un formulaire Zend_Form
Advanced Zend_Form Usage
Zend_Gdata
Introduction
Authentification par procédé AuthSub
Using the Book Search Data API
Authentification avec ClientLogin
Using Google Calendar
Using Google Documents List Data API
Using Google Health
Using Google Spreadsheets
Using Google Apps Provisioning
Using Google Base
Utilisation des albums Web Picasa
Using the YouTube Data API
Attraper les exceptions Gdata
Zend_Http
Introduction
Zend_Http_Client - Utilisation avancée
Zend_Http_Client - Adaptateurs de connexion
Zend_Http_Cookie and Zend_Http_CookieJar
Zend_Http_Response
Zend_InfoCard
Introduction
Zend_Json
Introduction
Utilisation de base
Utilisation avancée de Zend_Json
XML to JSON conversion
Zend_Json_Server - JSON-RPC server
Zend_Layout
Introduction
Zend_Layout - Démarrage rapide
Zend_Layout options de configuration
Zend_Layout, utilisation avancée
Zend_Ldap
Introduction
Zend_Loader
Charger les fichiers et les classes dynamiquement
L'autoloader
Autoloaders de ressources
Chargeur de Plugins
Zend_Locale
Introduction
Using Zend_Locale
Normalization and Localization
Working with Dates and Times
Supported locales
Migrer à partir des versions précédentes
Zend_Log
Présentation
Rédacteurs (Writers)
Formateurs (mise en forme)
Filtres
Zend_Mail
Introduction
Envoyer des émail en utilisant SMTP
Envoyer plusieurs émail par connexion SMTP
Utiliser différents transports
Émail HTML
Fichiers joints
Ajouter des destinataires
Contrôler les limites MIME
En-têtes additionnels
Jeux de caractères
Encodage
Authentification SMTP
Sécuriser les transports SMTP
Lire des émail
Zend_Measure
Introduction
Création d'une mesure
Récupérer des mesures
Manipuler des mesures
Types de mesures
Zend_Memory
Présentation
Manager de mémoire
Objet mémoire
Zend_Mime
Zend_Mime
Zend_Mime_Message
Zend_Mime_Part
Zend_Navigation
Introduction
Pages
Containers
Zend_OpenId
Introduction
Zend_OpenId_Consumer Basics
Zend_OpenId_Provider
Zend_Paginator
Introduction
Utilisation
Configuration
Utilisation avancée
Zend_Pdf
Introduction
Créer et charger des documents PDF
Sauvegarder les changement dans un document PDF
Les pages d'un document
Dessiner
Informations du document et métadonnées
Exemple d'utilisation du module Zend_Pdf
Zend_ProgressBar
Zend_ProgressBar
Zend_Reflection
Introduction
Zend_Reflection Exemples
Réference de Zend_Reflection
Zend_Registry
Utiliser le registre
Zend_Rest
Introduction
Zend_Rest_Client
Zend_Rest_Server
Zend_Search_Lucene
Vue d'ensemble
Créer des index
Searching an Index
Query Language
Query Construction API
Jeu de caractères
Extensibility
Agir avec Lucene Java
Avancé
Bonnes pratiques
Zend_Server
Introduction
Zend_Server_Reflection
Zend_Service
Introduction
Zend_Service_Akismet
Zend_Service_Amazon
Zend_Service_Amazon_Ec2
Zend_Service_Amazon_Ec2: Instances
Zend_Service_Amazon_Ec2: Windows Instances
Zend_Service_Amazon_Ec2: Reserved Instances
Zend_Service_Amazon_Ec2: CloudWatch Monitoring
Zend_Service_Amazon_Ec2: Amazon Machine Images (AMI)
Zend_Service_Amazon_Ec2: Elastic Block Stroage (EBS)
Zend_Service_Amazon_Ec2: Elastic IP Addresses
Zend_Service_Amazon_Ec2: Keypairs
Zend_Service_Amazon_Ec2: Regions and Availability Zones
Zend_Service_Amazon_Ec2: Security Groups
Zend_Service_Amazon_S3
Zend_Service_Audioscrobbler
Zend_Service_Delicious
Zend_Service_Flickr
Zend_Service_Nirvanix
Zend_Service_ReCaptcha
Zend_Service_Simpy
Introduction
Zend_Service_StrikeIron
Zend_Service_StrikeIron: Bundled Services
Zend_Service_StrikeIron: Advanced Uses
Zend_Service_Technorati
Zend_Service_Twitter
Zend_Service_Yahoo
Zend_Session
Introduction
Usage basique
Utilisation avancée
Gestion générale de la session
Zend_Session_SaveHandler_DbTable
Zend_Soap
Zend_Soap_Server
Zend_Soap_Client
WSDL
Auto découverte
Zend_Tag
Introduction
Zend_Tag_Cloud
Zend_Test
Introduction
Zend_Test_PHPUnit
Zend_Text
Zend_Text_Figlet
Zend_Text_Table
Zend_TimeSync
Introduction
Utiliser Zend_TimeSync
Zend_Tool_Framework
Introduction
Using the CLI Tool
Architecture
Creating Providers to use with Zend_Tool_Framework
Shipped System Providers
Zend_Tool_Project
Zend_Tool_Project Introduction
Create A Project
Zend Tool Project Providers
Zend_Translate
Introduction
Adaptateurs pour Zend_Translate
Utiliser les adaptateurs de traduction
Migrer à partir des versions précédentes
Zend_Uri
Zend_Uri
Zend_Validate
Introduction
Classes de validation standard
Chaînes de validation
Écrire des validateurs
Validation Messages
Zend_Version
Lire la version de Zend Framework
Zend_View
Introduction
Scripts de contrôleur
Scripts de vue
Aides de vue
Zend_View_Abstract
Migration depuis les versions précédentes
Zend_Wildfire
Zend_Wildfire
Zend_XmlRpc
Introduction
Zend_XmlRpc_Client
Zend_XmlRpc_Server
Configuration système requise par Zend Framework
Introduction
Convention de codage PHP de Zend Framework
Vue d'ensemble
Formatage des fichiers PHP
Conventions de nommage
Style de codage
Zend Framework Performance Guide
Introduction
Chargement des classes
Zend_Db Performance
Internationalisation (i18n) and Localisation (l10n)
View Rendering
Informations de copyright