Dirichlet space

From formulasearchengine
Revision as of 11:52, 18 December 2011 by en>Qetuth (more specific stub type)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Ranking SVM is an application of Support vector machine, which is used to solve certain ranking problems. The algorithm of ranking SVM was published by Torsten Joachims by 2003.[1] The original purpose of Ranking SVM is to improve the performance of the internet search engine. However, it was found that Ranking SVM also can be used to solve other problems such as Rank SIFT.[2]

Description

Ranking SVM, one of the pair-wise ranking methods, which is used to adaptively sort the web-pages by their relationships (how relevant) to a specific query. A mapping function is required to define such relationship. The mapping function projects each data pair (inquire and clicked web-page) onto a feature space. These features combined with user’s click-through data (which implies page ranks for a specific query) can be considered as the training data for machine learning algorithms.

Generally, Ranking SVM includes three steps in the training period:

  1. It maps the similarities between queries and the clicked pages onto certain feature space.
  2. It calculates the distances between any two of the vectors obtained in step 1.
  3. It forms optimization problem which is similar to SVM classification and solve such problem with the regular SVM solver.

Background

Ranking Method

Suppose is a data set containing C elements ci. r is a ranking method applied to . Then the r in can be represented as a C by C asymmetric binary matrix. If the rank of ci is higher than the rank of cj, i.e. rci<rcj, the corresponding position of this matrix is set to value of "1". Otherwise the element in that position will be set as the value "0".

Kendall’s Tau [3][4]

Kendall's Tau also refers to Kendall tau rank correlation coefficient, which is commonly used to compare two ranking methods for the same data set.

Suppose r1 and r2 are two ranking method applied to data set , the Kendall's Tau between r1 and r2 can be represented as follows:

τ(r1,r2)=PQP+Q=12QP+Q

where P is the number of the same elements in the upper triangular parts of matrices of r1 and r2, Q is the number of the different elements in the upper triangular parts of matrices of r1 and r2. The diagonals of the matrices are not included in the upper triangular part stated above.

Information Retrieval Quality [5][6][7]

Information retrieval quality is usually evaluated by the following three measurements:

  1. Precision
  2. Recall
  3. Average Precision

For an specific query to a database, let Prelevant be the set of relevant information elements in the database and Pretrieved be the set of the retrieved information elements. Then the above three measurements can be represented as follows:

Precision=|PrelevantPretrieved||Pretrieved|;Recall=|PrelevantPretrieved||Prelevant|;AveragePrecision=01Prec(Recall)dRecall,

where Prec(Recall) is the Precision function of Recall.

Let r* and rf(q) be the expected and proposed ranking methods of a database respectively, the lower bound of Average Precision of method rf(q) can be represented as follows:

AvgPrec(rf(q))1R[Q+(R+12)]1(i=1Ri)2

where Q is the number of different elements in the upper triangular parts of matrices of r* and rf(q) and R is the number of relevant elements in the data set.

SVM Classifier [8]

Suppose (xi,yi) is the element of a training data set, where xi is the feature vector (with information about features) and yi is the label(which classifies the category of xi). An typical SVM classifier for such data set can be defined as the solution of the following optimization problem.

minimize:V(w,ξ)=12ww+CFξiσs.t.σ0;yi(wxi+b)1ξiσ;wherebisascalar;yi{1,1};ξi0;

The solution of the above optimization problem can be represented as a linear combination of the feature vectors xis.

w*=iαiyixi

where αi is the coefficients to be determined.

Ranking SVM algorithm

Loss Function

Let τP(f) be the Kendall's tau between expected ranking method r* and proposed method rf(q), it can be proved that maximizing τP(f) helps to minimize the lower bound of the Average Precision of rf(q).

  • Expected Loss Function [9]

The negative τP(f) can be selected as the loss function to minimize the lower bound of Average Precision of rf(q) Lexpected=τP(f)=τ(rf(q),r*)dPr(q,r*)

where Pr(q,r*) is the statistical distribution of r* to certain query q.

  • Empirical Loss Function

Since the expected loss function is not applicable, the following empirical loss function is selected for the training data in practice.

Lempirical=τS(f)=1ni=1nτ(rf(qi),ri*)

Collecting training data

n i.i.d. queries are applied to a database and each query corresponds to a ranking method. So The training data set has n elements. Each elements containing a query and the corresponding ranking method.

Feature Space

Labelled points in feature space

A mapping function Φ(q,d)[10][11] is required to map each query and the element of database to a feature space. Then each point in the feature space is labelled with certain rank by ranking method.

Optimization problem

The points generated by the training data are in the feature space, which also carry the rank information (the labels). These labeled points can be used to find the boundary (classifier) that specifies the order of them. In the linear case, such boundary (classifier) is a vector.

Suppose ci and cj are two elements in the database and denote (ci,cj)r if the rank of ci is higher than cj in certain ranking method r. Let vector w be the linear classifier candidate in the feature space. Then the ranking problem can be translated to the following SVM classification problem.Note that one ranking method corresponds to one query.

minimize:V(w,ξ)=12ww+Constantξi,j,ks.t.ξi,j,k0(ci,cj)rk*w(Φ(q1,ci)Φ(q1,cj))1ξi,j,1;...w(Φ(qn,ci)Φ(qn,cj))1ξi,j,n;wherek{1,2,...n},i,j{1,2,...}.

The above optimization problem is identical to the classical SVM classification problem, which is the reason why this algorithm is called Ranking-SVM.

W candidate
Not a w candidate

Retrieval Function

The optimal vector w* obtained by the training sample is

w*=αk,l*Φ(qk,ci)

So the retrieval function could be formed based on such optimal classifier.
For new query q, the retrieval function first projects all elements of the database to the feature space. Then it orders these feature points by the values of their inner products with the optimal vector. And the rank of each feature point is the rank of the corresponding element of database for the query q.

Application of Ranking SVM

Ranking SVM can be applied to rank the pages according to the query. The algorithm can be trained using click-through data, where consists of the following three parts:

  1. Query.
  2. Present ranking of search results
  3. Search results clicked on by user

The combination of 2 and 3 cannot provide full training data order which is needed to apply the full SVM algorithm. Instead, it provides a part of the ranking information of the training data. So the algorithm can be slightly revised as follows.

minimize:V(w,ξ)=12ww+Contantξi,j,ks.t.ξi,j,k0(ci,cj)rk'w(Φ(q1,ci)Φ(q1,cj))1ξi,j,1;...w(Φ(qn,ci)Φ(qn,cj))1ξi,j,n;wherek{1,2,...n},i,j{1,2,...}.

The method r does not provide ranking information of the whole dataset, it's a subset of the full ranking method. So the condition of optimization problem becomes more relax compared with the original Ranking-SVM.

References

43 year old Petroleum Engineer Harry from Deep River, usually spends time with hobbies and interests like renting movies, property developers in singapore new condominium and vehicle racing. Constantly enjoys going to destinations like Camino Real de Tierra Adentro.

  1. Joachims, T. (2003), "Optimizing Search Engines using Clickthrough Data", Proceedings of the ACM Conference on Knowledge Discovery and Data Mining
  2. Bing Li; Rong Xiao; Zhiwei Li; Rui Cai; Bao-Liang Lu; Lei Zhang; "Rank-SIFT: Learning to rank repeatable local interest points",Computer Vision and Pattern Recognition (CVPR), 2011
  3. M.Kemeny . Rank Correlation Methods, Hafner, 1955
  4. A.Mood, F. Graybill, and D. Boes. Introduction to the Theory of Statistics. McGraw-Hill, 3rd edition, 1974
  5. J. Kemeny and L. Snell. Mathematical Models in THE Social Sciences. Ginn & Co. 1962
  6. Y. Yao. Measuring retrieval effectiveness based on user preference of documents. Journal of the American Society for Information Science, 46(2): 133-145, 1995.
  7. R.Baeza- Yates and B. Ribeiro-Neto. Modern Information Retrieval. Addison- Wesley-Longman, Harlow, UK, May 1999
  8. C. Cortes and V.N Vapnik. Support-vector networks. Machine Learning Journal, 20: 273-297,1995
  9. V.Vapnik. Statistical Learning Theory. WILEY, Chichester,GB,1998
  10. N.Fuhr. Optimum polynomial retrieval functions based on the probability ranking principle. ACM TRANSACTIONS on Information Systems, 7(3): 183-204
  11. N.Fuhr, S.Hartmann, G.Lustig, M.Schwantner, K.Tzeras,and G.Knorz. Air/x - a rule-based multistage indexing system for large subject fields. In RIAO,1991