Steps to install Scrapy.

The following steps assumes that you already have python installed. To make your installation easy the steps recommend using easy_install tool.

Install Scrapy

1. Download and install easy_install tool from cheeseshop .
2. Easy install “zope.interface”
3. Download and install Twisted 10.2.0 from Twisted site
4. Easy install “lxml”
5. Easy install “pyOpenSSL”
6. Easy install “Scrapy”

Test your Scrapy installation

1. Within python interpreter, import the following batteries and there should not be any errors

1. import zope.interface
2. import twisted
3. import lxml
4. import OpenSSL
5. import scrapy

2. If you need to find the version of a particular battery then use module attribute .version or .__version__ as below

print twisted.version ==> [twisted, version 10.2.0]

Steps to install Scrapy.

Split a list using python zip function

I just found this trick. Say if you want to split a list with i number of elements into lists with j number of elements. It can easily be done with one liner using the python zip function. You just need to unzip the list specifying the “j”.
See python documentation on zip for more information on this.

Gist: https://gist.github.com/rajaramanathan/979dd880767fd051927d

Split a list using python zip function

Combine items in two list to form dictionary(map)

People love python because it has solutions built in for problems most frequently faced by developers. Python’s zip is one such thing.

Gist: https://gist.github.com/rajaramanathan/a4e1b5b3531fd2b2cfab

You would most probably need this in scenarios like csv file reading where the first row in the file is list of column names and subsequent are values. You know well that it is better to use map to read/write the values in a row like row[colName] = x or x = row[colName] rather than index like row[1] etc.,

Note: In case of csv file reading/writing, python does have a out-of-the-box solution: DictReader and so you don’t even have to do” even” this one liner 🙂 above…

Combine items in two list to form dictionary(map)

Python objects and types

I’m one of the guy who needs to know the internals of a machine before using it. I can’t help it but that is my very nature. The following article that I refer to is not for faint heart (meaning beginners to programming or python) and is by far the best i’ve found ever that describes the internals of python(at least 2.x) and i’ve to admit that i never ever used python 3 or found a need for using 3.x.

If you are curious about how python types, object or descriptors or attributes or methods work internally then read this .

My recommendation is to save the pdf version since this guy never wrote a book (which would have been an excellent one for experts or intermediates) and not sure how long his/her site will be up.

Python objects and types