This page shows how to install Scrapy on a Redhat based linux system. I experienced some lib missing problem when installing on on an Redhat EC2 instance and spend time to figure out. So I record it here if anyone else needs similar stuff.

sudo su
yum groupinstall 'Development Tools'
yum install -y libffi-devel libxslt-devel libxml2-devel openssl-devel python-devel mysql-devel

The above commands prepared necessary environment to compile the project. Now we are ready to install Scrapy with

pip install scrapy

This command will install the scrapy. Note that when it is successful, we still need to configure the path to make the scrapy command work.

export PATH=$PATH:/usr/local/bin

Now we are ready to go. run scrapy!

scrapy crawl yourspider