Please post your solution to the May 12 In-Class Exercise Thread.

Best,

Chris

Kevin, Sriramm, Mustafa<br>

1. Set a threshold so that, from the whole corpus, we will only return 100 documents (P@100). Since X docs have score > N-X, we can say that, if X is 100, our threshold will be 1000 - 100 = 900.<br>
2. Next, look at each batch of 100; for a given batch, see what documents have score greater than 900. For the documents Y with score over 900 in the batch, let rel_Y be the (human-determined) relevant documents. The precision for the batch is |rel_Y| / |Y|.<br>
3. Compute the average of these precision scores over all batches. That is the aggregate precision @ 100.

Total number of documents = N = 1000

Batch size = X = 100

Scoring function = N - X = 900

precision_list = []

for each batch in corpus:

• results_list = []
• relevant_list = []
• for each doc in batch:
  • If the score for doc is greater than 900:
  • Add them to results_list
  • Check if they are relevant and add them to relevant_list
• Finally, precision for this batch = len(relevant_list)/len(results_list)
• Append precision value to precision_list

Once we have gone through the whole loop, average the precision scores in the precision_list

P@100

N = 1000

Batch size = 100

10 batches

if X is 100, then there are 100 documents with a score over 900, so our threshold is 900

Then, for each batch, find all documents that have a score greater than 900, the precision for this batch is the number of relevant documents in the batch over the total number of documents returned

Finally, the average of all these precisions is the aggregate P@100

def aggregate_at_k(N = 1000, k = 100, X = 100): t = N - X precision_scores = [] for batch in corpus: Y = [] rel_Y = [] for document in batch: if score(document) > t: Y.append(document) if relevant(document): rel_Y.append(document) batch_precision = len(rel_Y)/len(Y) precision_scores.append(batch_precision) return average(precision_scores)

In this example we are letting X be 100 So , N-X = 900

Now, we check for every doc if the score is greater than 900 and we also check if that doc is relevant. If the score is greater than 900 then we add it to the results list. If the doc is relevant then it also entered in the relevant_results list.

The precision for a batch is calculated by length of results list/length of relevant_results list.

Similarly, precision of all the batches can be calculated and all batch precisions can stored in batchwise_precision list.

For N=1000 and batch=100, <br> threshold=N-X=1000-100=900 -> Only 100 documents will have a score over 900 <br><br> For each batch, store documents with score > threshold,<br> Calculate precision for each batch:<br> Precision = percentage of the result that is relevant = <br>[relevant AND result]/result <br><br> Aggregate P@100 score = average(precision for each batch)

To compute the aggregate P@100 for a corpus of N=1000 with batch size 100, first have to calculate which documents have a score of N - X => 900. We look at each batch of 100 documents and store the documents with a score over 900. We calculate the precision of each batch by diving the human determined relevant documents in the stored batch over 100. We do that for every batch and then the average precision of the 10 batches is the aggregate P@100.

Total number of documents N = 1000 X = 100 N-X = 1000-100 = 900 for each batch, for each document find documents that have score greater than 900 and add to the results list check relevance add them to relevance find precision by the number of relevant documents in the batch over the total number of documents store precision finally, find the average precision to find aggregate precision@100

<nowiki> Give a concrete procedure for computing aggregate P@100 If N = 1000 documents, corpus; X = 100 documents, batch; N / X = 1000 / 100 = 10 batches; N - X = 1000 - 100 = 900, from scoring function;

Steps:

1. Compute threshold.

N - X = 1000 - 100 = 900 documents. This shows 100 documents have a score over 900.

1. Iterate each batch (10 batches).
2. Iterate each document in batch.
3. Find document with a score over 900.
4. Determine relevance of found document.
5. After iterating every document, calculate precision of the batch.
6. After iterating every batch, calculate average precision of all batches.
7. Result is computing aggregate P@100.

</nowiki>