Quora Scraper

Hi! What exactly went wrong, were some answers missing in the results or you just couldn't get the collected answers at all? I have seen your run, to see answers for a particular questions you have to go to your run, click Storage tab, then Key-Value Store tab and there will be a lot of files for each question with the corresponding answers. Let me now if it wasn't your problem

Thank you for the answer! I just thought the answers would be in the downloaded JSON file and did not know about the storage tab. what is the best way to download the full JSON with questions and for each question the related answers?

Well, there is special API for the store (you should be able to find it somewhere in that tab), so you can query endpoint for all keys and then for each of those keys query the endpoint to get the data for that particular key. But if you wait, I think I'll just modify the scraper to additionaly create a dataset with all the answers, I think it'll be much more convenient

As for the questions dataset, there should be a green button to download in your desired format

That would be awesome, getting the answers with the questions datasets would be so much more convenient. Right now I am using the API to fetch for each question the corresponding record in the storage, but creating records in a database is so much slower this way. Getting everything from the initial scrape would be awesome!

Hi! I've added a parameter to the input to create an answer dataset. If you turn it on, you should see a dataset prefixed with ANSWERS-... in Datasets tab, and you'll be able to join answers with questions by qid.