Please post your solutions to the Apr 28 In-Class Exercise to this thread.

Best,

Chris

DFR_california_3 = (log (1 + (104 / 500000)) + (1 * log (1 + (3 / 3))) log(1 + (500000 / 104))) / ((1 * log (1 + (3 / 3))) + 1) = 0.00030005 + (12.231 / 2) = 6.1155<br> <br> DFR_business_3 = (log (1 + (501 / 500000)) + (1 * log (1 + (3 / 3))) log(1 + (500000 / 501))) / ((1 * log (1 + (3 / 3))) + 1) = 0.001445 + (9.964 / 2) = 4.982<br> <br> DFR_tax_3 = (log (1 + (254 / 500000)) + (1 * log (1 + (3 / 3))) log(1 + (500000 / 254))) / ((1 * log (1 + (3 / 3))) + 1) = 0.0007327 + (10.944 / 2) = 5.472<br> <br> DFR_california_4 = (log (1 + (104 / 500000)) + (1 * log (1 + (3 / 4))) log(1 + (500000 / 104))) / ((1 * log (1 + (3 / 4))) + 1) = 0.00030005 + ((0.807412.231) / 1.8074) = 5.46382<br> <br> DFR_business_4 = (log (1 + (501 / 500000)) + (1 * log (1 + (3 / 4))) log(1 + (500000 / 501))) / ((1 * log (1 + (3 / 4))) + 1) = 0.001445 + ((0.80749.964) / 1.8074) = 4.45111<br> <br> DFR_tax_4 = (log (1 + (254 / 500000)) + (1 * log (1 + (3 / 4))) log(1 + (500000 / 254))) / ((1 * log (1 + (3 / 4))) + 1) = 0.0007327 + ((0.807410.944) / 1.8074) = 4.88889<br> <br> DFR_return = (log (1 + (607 / 500000)) + (1 * log (1 + (3 / 4))) log(1 + (500000 / 607))) / ((1 * log (1 + (3 / 4))) + 1) = 0.0017504 + ((0.80749.688) / 1.8074) = 4.32781<br> <br><br>

1. DFR_california_business_tax = DFR_california_3 * DFR_business_3 * DFR_tax_3 = 6.1155 * 4.982 * 5.472 = 166.7177<br><br>
2. DFR_california_business_tax_return = DFR_california_4 * DFR_business_4 * DFR_tax_4 * DFR_return = 5.46382 * 4.45111 * 4.88889 * 4.32781 = 514.56846<br>

Given N = 500000, l_avg = 3, l_t = 104, 501, 254, 607 and noticing that f_t,d for each term is 1:

Query: "California Business Tax" (l_d = 3)

f'_t,d = f_t,d * log(1 + l_avg/l_d) = 1 * log(1 + 3/3) = 1

For each term, (1-P_2)(-log P_1) = (log(1 + l_t/N) + f'_t,d * log(1 + N/l_t))/(f'_t,d + 1) with document length normalization

"California"

(log(1 + 104/500000) + 1 * log(1 + 500000/104))/(1 + 1) = (0.000300049364453 + 12.2314289005)/2 = 6.11586447493

"Business"

(log(1 + 501/500000) + 1 * log(1 + 500000/501))/(1 + 1) = (0.0014448566786 + 9.96434663281)/2 = 4.98289574474

"Tax"

(log(1 + 254/500000) + 1 * log(1 + 500000/254))/(1 + 1) = (0.000732702989965 + 10.9436165855)/2 = 5.47217464424

DFR Score = 6.11586447493 + 4.98289574474 + 5.47217464424 = 16.5709348639

Query: "California Business Tax Return" (l_d = 4)

f'_t,d = f_t,d * log(1 + l_avg/l_d) = 1 * log(1 + 3/4) = 0.807354922058

For each term, (1-P_2)(-log P_1) = (log(1 + l_t/N) + f'_t,d * log(1 + N/l_t))/(f'_t,d + 1) with document length normalization

"California"

(log(1 + 104/500000) + 0.807354922058 * log(1 + 500000/104))/(0.807354922058 + 1) = (0.000300049364453 + 0.807354922058 * 12.2314289005)/1.80735492206 = 5.4640094513

"Business"

(log(1 + 501/500000) + 0.807354922058 * log(1 + 500000/501))/(0.807354922058 + 1) = (0.0014448566786 + 0.807354922058 * 9.96434663281)/1.80735492206 = 4.45192532887

"Tax"

(log(1 + 254/500000) + 0.807354922058 * log(1 + 500000/254))/(0.807354922058 + 1) = (0.000732702989965 + 0.807354922058 * 10.9436165855)/1.80735492206 = 4.88897632145

"Return"

(log(1 + 607/500000) + 0.807354922058 * log(1 + 500000/607))/(0.807354922058 + 1) = (0.00175036952018 + 0.807354922058 * 9.68776623259)/1.80735492206 = 4.32854445226

DFR Score = 5.4640094513 + 4.45192532887 + 4.88897632145 + 4.32854445226 = 19.1334555539

<nowiki>

Important constants: -The average length of a query in our corpus of queries is 3 = l_avg -The number of occurrences of the terms "California", "Business", "Tax", and "Return" in the whole corpus are respectively 104, 501, 254, 607 = l_t for each respective terms

DFR score of a query is given by: Sum of every t in q(q_t * (1-P_2,t) * (-logP_1,t))

This can be simplified to: Sum of every t in q(q_t * (log(1+l/t_N)+ f_t,d * log(1 + N/l_t)) / (f_t,d + 1) )

we can substitute f_t,d with the normalized version given by: f'_t,d = f_t,d * log(1 + l_avg/l_d)

This gives us the final equation for DFR score as: Sum of every t in q(q_t * (log(1+l/t_N)+ f'_t,d * log(1 + N/l_t)) / (f'_t,d + 1) )

The two queries are:

• California Business Tax
• California Business Tax Return

In this example the document is the query q_t is 1 for every term in both queries since each word appears once in each query. f'_t,d for the first query for every term is 1 * log(1 + 3/3) = 1 f'_t,d for the second query for every term is 1 * log(1 + 3/4) = log(3/4)

1st query: 1 * (log(1+104/500,000)+ 1 * log(1 + 500,000/104)) / (1 + 1) = 6.116 1 * (log(1+501/500,000)+ 1 * log(1 + 500,000/501)) / (1 + 1) = 4.983 1 * (log(1+254/500,000)+ 1 * log(1 + 500,000/254)) / (1 + 1) = 5.472

DFR score for query 1 is 6.116 + 4.983 + 5.472 = 16.570934864

2st query: 1 * (log(1+104/500,000)+ log(3/4) * log(1 + 500,000/104)) / (log(3/4) + 1) = 5.464 1 * (log(1+501/500,000)+ log(3/4) * log(1 + 500,000/501)) / (log(3/4) + 1) = 4.452 1 * (log(1+254/500,000)+ log(3/4) * log(1 + 500,000/254)) / (log(3/4) + 1) = 4.889 1 * (log(1+607/500,000)+ log(3/4) * log(1 + 500,000/607)) / (log(3/4) + 1) = 4.329

DFR score for query 2 is 5.464 + 4.452 + 4.889 + 4.329 = 19.1334555539

</nowiki>

DFR score = Sum of every t in q(q_t * (log(1+l/t_N)+ f'_t,d * log(1 + N/l_t)) / (f'_t,d + 1) )

f'_t,d = f_t,d * log(1 + l_avg/l_d)

First Query = "California Business Tax"

For term california in Query "California Business Tax"

1* log(1 + 104/500000) + 1* log(1+ 500000/104) / (1+1)

log(1.0002) + log(4808.69) /2

12.231/2 = 6.1155

For term business in Query "California Business Tax"

1log(1 + 501/500000) + 1 log(1+ 500000/501) / (1+1)

log(1.0010) + log(999) /2 = 4.983

For term tax in Query "California Business Tax"

1log(1 + 254/500000) + 1 log(1+ 500000/254) / (1+1)

log(1.0005) + log(1969.5) /2 = 5.472

First query DFR score = 6.115 + 4.983 + 5.472 = 16.571

Second Query = "California Business Tax Return" For term california in Query "California Business Tax Return"

f'(t, d) = 1.log(1+3/4) = 0.8074 For term business in Query "California Business Tax Return"

1* log(1 + 104/500000) + 0.8074* log(1+ 500000/104) / (0.8074+1) = 5.464

For term business in Query "California Business Tax Return"

1log(1 + 501/500000) + 0.8074 log(1+ 500000/501) / (0.8074+1) = 4.452

For term tax in Query "California Business Tax Return"

1log(1 + 254/500000) + 0.8074 log(1+ 500000/254) / (0.8074+1) = 4.889

For term return in Query "California Business Tax Return"

1log(1 + 254/500000) + 0.8074 log(1+ 500000/254) / (0.8074+1) = 4.329

Second query DFR score = 5.464 + 4.452 + 4.889 + 4.329 = 19.133

N = Size of corpus = 500000

l_avg = Average length of a query in our corpus of queries = 3

l_t = Number of occurrences of the terms "California", "Business", "Tax" and "Return" = 104, 501, 254, 607

DFR score = Sum of every term t in query q = (q_t x (1-P_2,t) x (-logP_1,t))

Or, DFR score = (q_t x (log(1+l/t_N)+ f_t,d x log(1 + N/l_t)) / (f_t,d + 1))

Substituting f_t,d with their normalized version: f'_t,d = f_t,d * log(1 + l_avg/l_d)

We get, DFR score = Sum of every t in q(q_t * (log(1+l/t_N)+ f'_t,d * log(1 + N/l_t)) / (f'_t,d + 1) )

a) For "California Business Tax"

f'_t,d for the first query for every term is 1 * log(1 + 3/3) = 1

DFR = Sum of:

1 * (log(1+104/500,000)+ 1 * log(1 + 500,000/104)) / (1 + 1) = 6.116

1 * (log(1+501/500,000)+ 1 * log(1 + 500,000/501)) / (1 + 1) = 4.983

1 * (log(1+254/500,000)+ 1 * log(1 + 500,000/254)) / (1 + 1) = 5.472

DFR score = 6.116 + 4.983 + 5.472 = 16.571

b) For "California Business Tax Return":

f'_t,d for the second query for every term is 1 * log(1 + 3/4) = log(3/4)

DFR = Sum of:

1 * (log(1+104/500,000)+ log(3/4) * log(1 + 500,000/104)) / (log(3/4) + 1) = 5.464

1 * (log(1+501/500,000)+ log(3/4) * log(1 + 500,000/501)) / (log(3/4) + 1) = 4.452

1 * (log(1+254/500,000)+ log(3/4) * log(1 + 500,000/254)) / (log(3/4) + 1) = 4.889

1 * (log(1+607/500,000)+ log(3/4) * log(1 + 500,000/607)) / (log(3/4) + 1) = 4.329

DFR score = 5.464 + 4.452 + 4.889 + 4.329 = 19.133

N = Size of corpus => 500000 l_avg = Average length of a query in our corpus of queries => 3 l_t = Number of occurrences of the terms "California", "Business", "Tax" and "Return" = 104, 501, 254, 607

DFR score = Sum of every term t in query q = (q_t x (1-P_2,t) x (-logP_1,t)) Or, DFR score = (q_t x (log(1+l/t_N)+ f_t,d x log(1 + N/l_t)) / (f_t,d + 1))

f'_t,d = f_t,d * log(1 + l_avg/l_d) DFR score = Sum of every t in q(q_t * (log(1+l/t_N)+ f'_t,d * log(1 + N/l_t)) / (f'_t,d + 1) )

California Business Tax: f'_t,d for every term is 1 * log(1 + 3/3) = 1 1 * (log(1+104/500,000)+ 1 * log(1 + 500,000/104)) / (1 + 1) = 6.116 1 * (log(1+501/500,000)+ 1 * log(1 + 500,000/501)) / (1 + 1) = 4.983 1 * (log(1+254/500,000)+ 1 * log(1 + 500,000/254)) / (1 + 1) = 5.472

DFR score(query 1) is 6.116 + 4.983 + 5.472 = 16.570934864

California Business Tax Return: f'_t,d for every term is 1 * log(1 + 3/4) = log(3/4)

1 * (log(1+104/500,000)+ log(3/4) * log(1 + 500,000/104)) / (log(3/4) + 1) = 5.464 1 * (log(1+501/500,000)+ log(3/4) * log(1 + 500,000/501)) / (log(3/4) + 1) = 4.452 1 * (log(1+254/500,000)+ log(3/4) * log(1 + 500,000/254)) / (log(3/4) + 1) = 4.889 1 * (log(1+607/500,000)+ log(3/4) * log(1 + 500,000/607)) / (log(3/4) + 1) = 4.329

DFR score(query 2) is 5.464 + 4.452 + 4.889 + 4.329 = 19.1334555539

Size of corpus(N) = 500000

Average length of a query in our corpus of queries (l_avg ) = 3

Number of occurrences of the terms (l_t) "California", "Business", "Tax" and "Return" in the whole corpus are respectively 104, 501, 254, 607

DFR score = Sum of every term t in query q = (q_t x (1-P_2,t) x (-logP_1,t)) Or DFR score = (q_t x (log(1+l/t_N)+ f_t,d x log(1 + N/l_t)) / (f_t,d + 1))

f'_t,d = f_t,d * log(1 + l_avg/l_d) so, DFR score = Sum of every t in q(q_t * (log(1+l/t_N)+ f'_t,d * log(1 + N/l_t)) / (f'_t,d + 1) )

a) Query: "California Business Tax"

For term California: 1 * (log(1+104/500,000)+ 1 * log(1 + 500,000/104)) / (1 + 1) = 6.116

For term Business: 1 * (log(1+501/500,000)+ 1 * log(1 + 500,000/501)) / (1 + 1) = 4.983

For term Tax: 1 * (log(1+254/500,000)+ 1 * log(1 + 500,000/254)) / (1 + 1) = 5.472

DFR score for the query California Business Tax = 6.116 + 4.983 + 5.472 = 16.571

b) Query: "California Business Tax Return":

For term California: 1 * (log(1+104/500,000)+ log(3/4) * log(1 + 500,000/104)) / (log(3/4) + 1) = 5.464

For term Business: 1 * (log(1+501/500,000)+ log(3/4) * log(1 + 500,000/501)) / (log(3/4) + 1) = 4.452

For term Tax: 1 * (log(1+254/500,000)+ log(3/4) * log(1 + 500,000/254)) / (log(3/4) + 1) = 4.889

For term Return: 1 * (log(1+607/500,000)+ log(3/4) * log(1 + 500,000/607)) / (log(3/4) + 1) = 4.329

DFR score for the query California Business Tax Return = 5.464 + 4.452 + 4.889 + 4.329 = 19.133

<nowiki> corpus = 500,000

ave = 3

california = 104 business = 501 tax = 254 return = 607

DFR = (1-P2)(-logP1)

Q1. california business tax

california log(1 + 104/500,000) + ftd log(1 + 500,000/104) / (ftd + 1) ftd = ftd * log(1 + 3/3) = ftd * 1 = 1

business log(1 + 501/500,000) + ftd log(1 + 500,000/501) / (ftd + 1) ftd = ftd * log(1 + 3/3) = ftd * 1 = 1

tax log(1 + 254/500,000) + ftd log(1 + 500,000/254) / (ftd + 1) ftd = ftd * log(1 + 3/3) = ftd * 1 = 1

Sum (california business tax) = 6.116 + 4.983 + 5.472 = 16.571

Q2. california business tax return

california log(1 + 104/500,000) + ftd log(1 + 500,000/104) / (ftd + 1) ftd = ftd * log(1+3/4) = .807

business log(1 + 501/500,000) + ftd log(1 + 500,000/501) / (ftd + 1) ftd = ftd * log(1+3/4) = .807

tax log(1 + 254/500,000) + ftd log(1 + 500,000/254) / (ftd + 1) ftd = ftd * log(1+3/4) = .807

return log(1 + 607/500,000) + ftd log(1 + 500,000/607) / (ftd + 1) ftd = ftd * log(1+3/4) = .807

Sum (california business tax return) = 5.464 + 4.452 + 4.889 + 4.329 = 19.133 </nowiki>

<nowiki> corpus_queries = N = 500,000

lavg = 3

occurences = lt california = 104 business = 501 tax = 254 return = 607

DFR = (1 - P)(-log(P)) = log(1 + lt/N) + f_td log(1 + N/lt) / f_td + 1

f_td = f_td * log(1 + lavg/ld)

1. california business tax

ld = 3

f_td = f_td * log(1 + 3/3) = f_td * 1 = 1 * 1 = 1

california DFR = log(1 + 104/500,000) + f_td log(1 + 500,000/104) / (f_td + 1) = log(1 + 104/500,000) + 1 * log(1 + 500,000/104) / (1 + 1) = 12.2317 / 2 = 6.1158 6.116

business DFR = log(1 + 501/500,000) + f_td log(1 + 500,000/501) / (f_td + 1) = log(1 + 501/500,000) + 1 * log(1 + 500,000/501) / (1 + 1) = 9.9657 / 2 = 4.9828 4.983

tax DFR = log(1 + 254/500,000) + f_td log(1 + 500,000/254) / (f_td + 1) = log(1 + 254/500,000) + 1 * log(1 + 500,000/254) / (1 + 1) = 10.9443 / 2 = 5.4721 5.472

Sum(california business tax) = california + business + tax = 6.116 + 4.983 + 5.472 = 16.571

1. california business tax return

ld = 4

f'_td = f_td * log(1 + 3/4) = f_td * 0.807 = 1 * 0.807 = 0.807

california DFR = log(1 + 104/500,000) + f_td log(1 + 500,000/104) / (f_td + 1) = log(1 + 104/500,000) + 0.807 * log(1 + 500,000/104) / (0.807 + 1) = 9.8710 / 1.807 = 5.4626 5.463

business DFR = log(1 + 501/500,000) + f_td log(1 + 500,000/501) / (f_td + 1) = log(1 + 501/500,000) + 0.807 * log(1 + 500,000/501) / (0.807 + 1) = 8.0426 / 1.807 = 4.4508 4.451

tax DFR = log(1 + 254/500,000) + f_td log(1 + 500,000/254) / (f_td + 1) = log(1 + 254/500,000) + 0.807 * log(1 + 500,000/254) / (0.807 + 1) = 8.8322 / 1.807 = 4.8877 4.888

return DFR = log(1 + 607/500,000) + f_td log(1 + 500,000/607) / (f_td + 1) = log(1 + 607/500,000) + 0.807 * log(1 + 500,000/607) / (0.807 + 1) = 7.8197 / 1.807 = 4.3274 4.327

Sum(california business tax return) = california + business + tax + return = 5.463 + 4.451 + 4.888 + 4.327 = 19.129

</nowiki>