Navigating Python

A tutorial to glue other tutorials

Or Python at 30,000 feet

Yann Tambouret

A Brief Introduction to Python

  • What is python
  • distributions

Python in Action

  • Syntax tour: variables, code blocks
  • Installing packages
  • conda (and some about virtualenv)

Python for Research

  • ipython, ipython notebooks
  • numpy, scipy, pandas and matplotlib
  • sympy, theano

Next Steps

Python is interactive

In [1]:
x = 100
In [2]:
x**2
Out[2]:
10000
In [3]:
y = x**2 - 22
y
Out[3]:
9978

Python is Python 2.7, Python 3.3, Python 3.4 ...

Python 3

  • introduces subtle improvements
  • involves changes that are sometimes incompatable with older code
  • is easy to migrate to
In [4]:
from __future__ import print_function
# This is how you print in python 2
#print "hello world"
# and now in python 3 
print ('hello world')
hello world

Python 2 works

  • with all scientific packages
  • for now, but no new features are being added

Python 3 is the way to go, unless you have to use Python 2

Python is really written in C, Java, .net, ...

  • Cpython -- python built with C under the hood
  • Jython -- python built upon Java
  • IronPython -- python built upon .net

cpython is what people are usually talking about.

Python Distributions

python.org -- just python

  • pypi.python.org -- access to many other packages

 Anaconda (Continuum) -- manages installations with conda tool

  • free to everyone
  • you can't/don't use virtualenv
  • you can pay for optimized/optimization packages

Enthought -- manages installing extra packages with Canopy GUI

  • free to academics
  • requires an account

Python(x,y) -- a distribution I'm not very familiar with

Python on SCC

module load anaconda/2.0.0
python
Python 2.7.7 |Anaconda 2.0.0 (64-bit)| (default, Jun  2 2014, 12:34:02)
[GCC 4.1.2 20080704 (Red Hat 4.1.2-54)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://binstar.org
>>>

Variables are labels for info

In [5]:
x = 10
x = "ten"
print(type(x))
<type 'str'>

In [6]:
y = ["this", "is", 1, "list"]
z = y
In [7]:
print("y is", y)
print("z is", z)
y is ['this', 'is', 1, 'list']
z is ['this', 'is', 1, 'list']

In [8]:
z[2] = "a"
print("y is now", y)
print("so is z:", z)
y is now ['this', 'is', 'a', 'list']
so is z: ['this', 'is', 'a', 'list']

For more info, read this article

everything is an object

In [9]:
print(y, "is of type", type(y))
['this', 'is', 'a', 'list'] is of type <type 'list'>

In [10]:
print("it has", len(y), "elements")
it has 4 elements

In [11]:
y.append("!")
print("and now", len(y))
and now 5

In [12]:
print(y.__add__)
print(y.__len__)
print(y.append)
print(y.insert)
<method-wrapper '__add__' of list object at 0x7f1c24e1db48>
<method-wrapper '__len__' of list object at 0x7f1c24e1db48>
<built-in method append of list object at 0x7f1c24e1db48>
<built-in method insert of list object at 0x7f1c24e1db48>

In [13]:
dir(y)
Out[13]:
['__add__',
 '__class__',
 '__contains__',
 '__delattr__',
 '__delitem__',
 '__delslice__',
 '__doc__',
 '__eq__',
 '__format__',
 '__ge__',
 '__getattribute__',
 '__getitem__',
 '__getslice__',
 '__gt__',
 '__hash__',
 '__iadd__',
 '__imul__',
 '__init__',
 '__iter__',
 '__le__',
 '__len__',
 '__lt__',
 '__mul__',
 '__ne__',
 '__new__',
 '__reduce__',
 '__reduce_ex__',
 '__repr__',
 '__reversed__',
 '__rmul__',
 '__setattr__',
 '__setitem__',
 '__setslice__',
 '__sizeof__',
 '__str__',
 '__subclasshook__',
 'append',
 'count',
 'extend',
 'index',
 'insert',
 'pop',
 'remove',
 'reverse',
 'sort']

Code Blocks

python code, then colon ':', then consistently indented lines of more python code

In [14]:
user_age = 10
AGE_LIMIT = 18
if user_age >= AGE_LIMIT:
    print("you may enter")
    print("welcome to the club")
else:
    print("please come back in", AGE_LIMIT - user_age, "years")
print("Now to our next example!")
please come back in 8 years
Now to our next example!

In [15]:
for item in y:
    print("next item is", item)
next item is this
next item is is
next item is a
next item is list
next item is !

In [16]:
def this_is_a_function(arg1, arg2):
    print("arugment 1 is", arg1)
    print("argument 2 is", arg2)
    if arg1 < 10:
        print("argument 1 is small, fyi")
        
this_is_a_function(1, 'other one')
arugment 1 is 1
argument 2 is other one
argument 1 is small, fyi

Classes

define you're own objects with classes

  • class keyword and code block
  • object is initialized with __init__ method
  • first variable to all methods is the object itself, named self by convention
In [17]:
class name:
    def __init__(self, first, last):
        self.first = first.capitalize()
        self.last = last.capitalize()
        
    def __str__(self):
        return self.first + " " + self.last
    
    def is_yann(self):
        return self.first == "Yann"
        
joe = name("Joe", "smith")
yann = name("yann", "tambouret")
print(joe, "is yann?", joe.is_yann())
print(yann, "is yann?", yann.is_yann())
Joe Smith is yann? False
Yann Tambouret is yann? True

modules, packages, scripts

  • a module is a single file of python code read into another python program
  • a package is a directory of python modules; when you import a package, __init__.py is really loaded
  • a script is just a file contain python code.
In [18]:
import numpy
print('numpy.arange:', numpy.arange(10))
from numpy import arange
print('arange:', arange(10))
from numpy import random
print('what is random anyway?', type(random))
numpy.arange: [0 1 2 3 4 5 6 7 8 9]
arange: [0 1 2 3 4 5 6 7 8 9]
what is random anyway? <type 'module'>

How to code?

  • just a text editor: vi, emacs, notepad
  • or a fancy text editor: notepad++ (Windows) or sublime text (Mac)
  • or an ide: Spyder or Eclipse + Pydev or many others
  • ipython notebooks are very handy too (more on this later)

pypi.python.org

An online repository for packages, with instructions to automatically install them.

  • easy_install, comes with python to install packages

    easy_install --user translate

  • pip is the recommended way (easy_install --user pip if it's missing)

    pip intall --user translate

conda

A tool for creating new python environments.

  • need a different version
  • need to try buggy code
  • need to check what your code depends on (start with just python, install dependencies till code works)

A conda example

conda create -n demo python==3.4 scipy
Fetching package metadata: ..
Solving package specifications: .
Package plan for installation in environment /home/yannpaul/anaconda3/envs/demo:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    numpy-1.8.2                |           py34_0         7.2 MB
    python-3.4.0               |                0        21.5 MB
    ------------------------------------------------------------
                                           Total:        28.6 MB

The following packages will be linked:

    package                    |            build
    ---------------------------|-----------------
    numpy-1.8.2                |           py34_0   hard-link
    openssl-1.0.1h             |                0   hard-link
    python-3.4.0               |                0   hard-link
    readline-6.2               |                2   hard-link
    scipy-0.14.0               |       np18py34_0   hard-link
    sqlite-3.8.4.1             |                0   hard-link
    system-5.8                 |                1   hard-link
    tk-8.5.15                  |                0   hard-link
    zlib-1.2.7                 |                0   hard-link

Proceed ([y]/n)? y

Fetching packages ...
numpy-1.8.2-py 100% |################################| Time: 0:00:07 940.24 kB/s
python-3.4.0-0 100% |################################| Time: 0:00:14   1.59 MB/s
Extracting packages ...
[      COMPLETE      ] |##################################################| 100%
Linking packages ...
[      COMPLETE      ] |##################################################| 100%
#
# To activate this environment, use:
# $ source activate demo
#
# To deactivate this environment, use:
# $ source deactivate
#

virtualenv

NOTE: This will not work with anaconda; use conda instead.

A tool to clone a python environment.

User: "Hey Yann, please help me xyz package".

Yann: "Ok, let me make a new environment with virtualenv to try this out"

virtualenv some_directory_name
source some_directory_name/bin/activate
which python
 some_directory_name/bin/python
which pip
 some_directory_name/bin/pip
pip install xyz
... doing work with xyz ...
deactivate

use "--system-site-packages" to inherit already installed packages:

virtualenv --system-site-packages some_directory_name
source some_directory_name/bin/activate
pip install --upgrade numpy

Press "down arrow" for a demo, or "right arrow" to continue...

virtualenv Demo

First, make sure correct python is loaded

[tuta30@scc1 ~]$ module load python/2.7.7
[tuta30@scc1 ~]$ which python
/share/pkg/python/2.7.7/install/bin/python

Setup venv

Next make a new virtualenv named venv

[tuta30@scc1 ~]$ virtualenv venv
New python executable in venv/bin/python
Installing setuptools, pip...done.

Activate venv

Next, activate it by sourcing the activate script

[tuta30@scc1 ~]$ source venv/bin/activate
(venv)[tuta30@scc1 ~]$ 

You'll notice that you're prompt has changed; it now matches the name of your virtualenv.

Install google translate package

Then try to install the translate package.

(venv)[tuta30@scc1 ~]$ pip install translate
Downloading/unpacking translate
  Downloading translate-0.0.6.tar.gz
  Running setup.py (path:/usr4/tutorial/tuta30/venv/build/translate/setup.py) egg_info for package translate

Installing collected packages: translate
  Running setup.py install for translate
    changing mode of build/scripts-2.7/translate from 644 to 755

    changing mode of /usr4/tutorial/tuta30/venv/bin/translate to 755
Successfully installed translate
Cleaning up...

And give it a try

Try translate -h to see how it works, or just follow this example:

(venv)[tuta30@scc1 ~]$ translate -t fr "Hello World"
Bonjour tout le monde
(venv)[tuta30@scc1 ~]$ deactivate
[tuta30@scc1 ~]$

Python for Research

  • ipython
  • ipython notebooks
  • numpy, scipy, pandas and matplotlib
  • sympy, theano

ipython

ipython replaces python; it improves upon interactive python.

  • easier to edit long instructions
  • easier to get help
  • easier to navigate file system
  • etc.

ipython notebook

A web interface that looks like Mathmatica. See nbviewer for many examples.

tuta31@scc1 ~]$ ipython notebook
2014-06-11 11:55:27.555 [NotebookApp] Created profile dir: u'/usr4/tutorial/tuta31/.ipython/profile_default'
2014-06-11 11:55:27.564 [NotebookApp] Using MathJax from CDN: http://cdn.mathjax.org/mathjax/latest/MathJax.js
2014-06-11 11:55:27.611 [NotebookApp] The port 8888 is already in use, trying another random port.
2014-06-11 11:55:27.611 [NotebookApp] The port 8889 is already in use, trying another random port.
2014-06-11 11:55:27.611 [NotebookApp] The port 8890 is already in use, trying another random port.
2014-06-11 11:55:27.611 [NotebookApp] Serving notebooks from local directory: /usr4/tutorial/tuta31
2014-06-11 11:55:27.612 [NotebookApp] 0 active kernels
2014-06-11 11:55:27.612 [NotebookApp] The IPython Notebook is running at: http://localhost:8891/
2014-06-11 11:55:27.612 [NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
/usr/bin/xdg-open: line 402: htmlview: command not found
INFO:tornado.access:302 GET / (127.0.0.1) 0.78ms

Numpy

Fast array manipuation in python.

In [19]:
import numpy as np
x = np.arange(10)
y = np.sin(x)
In [20]:
zlist = [1,2,3,4]
zlist.append(100)
z = np.array(zlist)
In [21]:
print(x)
print(y)
print(z)
[0 1 2 3 4 5 6 7 8 9]
[ 0.          0.84147098  0.90929743  0.14112001 -0.7568025  -0.95892427
 -0.2794155   0.6569866   0.98935825  0.41211849]
[  1   2   3   4 100]

A tutorial to use from here: Numpy Tutorial

scipy

Access to frequently needed numerical methods: Scipy Tutorial

pandas

Dataframes, like in R, but in Python (and well a ton of other stuff).

sympy

Symbolic math in python: Official Tutorial

theano

Sort of a symbolic math, numerical methods hybrid: Documentation

pymc

Monte-Carlo Bayesian analysis. Bayesian Methods for Hackers

scikit.learn and scikit.image

Machine learning and image processing libraries.

In [22]:
import pandas as pd
with open('data/fake_accidents.csv') as fake_csv:
    recs = pd.read_csv(fake_csv)
print(recs)
print(recs.accidents)
   day     state  accidents  fines
0    1  New York         75   1150
1    1      Ohio         42    NaN
2    2  New York         58    775
3    3  New York         71    899
4    3      Ohio         11    172
5    4      Ohio         41    349
6    4  New York         51    701
7    5      Ohio        NaN    300
8    5  New York         62    819
9    6  New York         67    850

[10 rows x 4 columns]
0    75
1    42
2    58
3    71
4    11
5    41
6    51
7   NaN
8    62
9    67
Name: accidents, dtype: float64

In [23]:
# the following "magic" loads matplotlib and sets plots to render inline
%pylab inline
Populating the interactive namespace from numpy and matplotlib

matplotlib

The plotting library for python.

In [24]:
plot(x, y)
Out[24]:
[<matplotlib.lines.Line2D at 0x7f1bff0cc650>]
In [25]:
recs = recs.sort('accidents')
recs.plot('accidents', 'fines')
Out[25]:
<matplotlib.axes.AxesSubplot at 0x7f1bff0f6bd0>

Faster Python

  • cprofile to profile code
  • use numpy, scipy, etc., when you can
  • use Cython to make C from your python code, or to access C, C++ code in python (for example)
  • use f2py to make Fortran code accessable in Python

A Python Subway map

From Matt Harrison's blog

Next Steps

A first step would be to follow a tutorial, like the official one, or Code Academy's.

Then aim to write a small program.

From Stackoverflow, you might consider a series of possible next steps.

Also, read some blogs.