Python Flask Scrapy API¶
Arachne provides a wrapper around your scrapy spider to run them through a flask app. All you have to do is setup your SPIDER_SETTINGS
in the settings file. You can view the source code on github.
Arachne is powered by Flask, Twisted and the Scrapy package
Why not scrapyd?¶
The best code is no code at all, I have tried to write as little code as possible to maintain my scrapy projects but when I started off using scrapy daemon, it was incredibly harder to maintain. My project grew to over 100 spiders and then I started to break my projects into smaller tiny pieces when I got to the bare roots of it then lo and behold, this project is its creation.
Installation¶
To install Arachne from pip use the command
pip install -i https://pypi.python.org/pypi/Arachne