Pokemap GO – Gotta scrape them all (Pokescrap released today)

Busy week-end due to the Valentine event, but I still found time to release Pokescrap. Pokescrap scrapes pokemap websites, such as SGPokemap and others.

Just as visiting the website with a browser, Pokescrap will request data about Pokémon spawns in a certain area and store the results in a SQLite database.

The Pokescrap code

The Python code is build upon Scrapy, a very powerful framework for extracting data from websites. Scrapy is extensible, as a consequence it is rather trivial to extend Pokescrap to send an email when Lapras spawns in the area, or to send a notification to your phone. These capabilities will be added in the near future.

Hiding in the shadows

When needed, you can configure Pokescrap to use an army of random proxies to hide the fact that you are scraping the website without permission. The next version will also allow to use randomized User-Agent strings.

The User-Agent HTTP header defines a specific tool or browser and this could be used to block the scraping process. Making the header random, makes this much more difficult. At this time however, the tool is not blocked.

What can I do with the data?

Use it for research! Find out which area spawns rare Pokémon, or at what time a specific point becomes active (most of the spawn points become only active at a specific minute on the hour). Deeper research could couple data to Open Street Map features and find out how Niantic creates the Pokémon biomes.

Isn’t this against the terms of service (TOS) of Niantic?

Probably, possibly, maybe, maybe not. Who reads these anyway? Pokescrap does not connect to the Niantic servers. It might be called cheating, if the data is abused to gain a personal advantage, but I don’t feel that this is worth discussing.

Tons of people use the maps directly. Go ahead, and take a look on the Pokémon Singapore Facebook page. 80,000 plus members freely share  information found on SGPokemap, boast about their endeavors using bots or with GPS spoofing. Pokescrap will not make a difference for the worse.

Gimme, gimme

You can find the source code (open source MIT license) on the Jeerbees github page.

Please do also visit my list of current Pokémon nests in Singapore, Enjoy the hunt!