Hey Everyone,

Hw5 is up!

Best, Chris

Professor,

While constructing index, are we going to use stem tokenization method only? Thank you!

Yes. Best, Chris

Hi Professor,

I'm not sure how to represent the gamma codes in binary form. Should we just store it as string? If yes, would't that take more space than the original representation?

Thanks, Kushal

Use the PHP pack, unpack, ord, chr as well as the operators &, | ~, etc to make sure you are manipulating bits not bytes. The Yioop file Utility.php has PHP code for some encoding and decoding operations related to some codes (not gamma codes though), so you can use it for inspiration. In particular, it does have code for encoding/decoding delta lists.

Hi Professor,

When I do query, I have a term t which occurs in document 11, 45 and 66. Can I load all info related to those three documents to memory?

I prefer this way since I can manipulate related document info in memory instead of fseek frequency on disk. But I would like to check whether this way is allowed.

Thank you!

Bo Li

Hello Professor, In the question 2, how many stages of reducers are expected?

Thank you, Dhruven Vora

@bo,

Can I load all info related to those three documents to memory? Are you asking if you can read three entries from the document map into memory at the same time? You can assume that a single document map entry can be read into memory in one go. I wouldn't assume more than that. In practice, you can assume there is some number k entries you can read into memory in one go. For this assignment let's keep k=1.

@dhruven

You don't have to do problem two in a single round. If it is more convenient to break it into a small number of rounds, do so.

I am asking whether I could read document information into memory. That is. all term-frequency information of these three docs.