Parsing Excel with Python

Cos sometimes you don't want to deal with xml, csv, json or any other text format.  Check out https://openpyxl.readthedocs.org/en/latest/.

Design Patterns

I generally avoid buying computer books because they become outdated very quickly.  However, Design Patterns by Gamma, Helm, Johnson, and Vlissides is an exception (probably due to its general nature):

http://www.amazon.com/Design-Patterns-Elements-Reusable-Object-Oriented/dp/0201633612/ref=sr_1_1?s=books&ie=UTF8&qid=1411492186&sr=1-1&keywords=design+patterns

 This book makes me rethink how I approach object oriented programming.

Setting the PYTHONPATH with the Powershell permanently

I actually got a lot of tips on how to do this through this article:

http://technet.microsoft.com/en-us/library/ff730964.aspx

If you have the .NET Framework, you can do the following:

> [Environment]::SetEnvironmentVariable("PYTHONPATH", "D:/mypath", "User")

In order to see the changes, you need to open a new Powershell.

Setting the PYTHONPATH with Powershell

Powershell is the Windows hacker's best friend until it isn't.  Actually, I really do enjoy the power you get from Powershell.  However, figuring out how to set my python path in the shell was a frustrating experience that I hope to spare other users from.

PS C:\Users\spearsc> Get-ChildItem Env:PYTHONPATH

Name                                    Value
----                                        -----
PYTHONPATH                     r:/env


PS C:\Users\spearsc> $env:PYTHONPATH = "D:/localenv"
PS C:\Users\spearsc> Get-ChildItem Env:PYTHONPATH

Name                           Value
----                               -----
PYTHONPATH           D:/localenv

There you have it!  Pretty sure this setting is not permanent, which is fine for testing purposes.

Dealing with the error message: "Cross origin requests are only supported for HTTP."

I know I have dealt with this before, but I found a Stack Overflow thread that does a pretty good job of explaining this problem.  Plus, the thread shows you how to use Python to fire up a simple web server to get around the issue:

http://stackoverflow.com/questions/20041656/xmlhttprequest-cannot-load-file-cross-origin-requests-are-only-supported-for-ht

Multiprocessing Versus Threading in Python

I keep forgetting the difference between multiprocessing and threading in Python.  This Stack Overflow thread explains the difference between the two pretty well:

http://stackoverflow.com/questions/3044580/multiprocessing-vs-threading-python

Pushing to a remote bitbucket repository with Mercurial

Ugh.  This is a lot harder than it ought to be.  I am finding that a lot of activities with Mercurial is just easier to do through the command line instead of TortoiseHg.

If you want to push to your bitbucket repo, type this command in your directory:

> hg push https://username@bitbucket.org/username/someproject

However, the smarter way is to go into the .hg directory and modify the hgrc file like so:

[paths]
default-push = https://username@bitbucket.org/username/someproject

Now just type 'hg push' at the command prompt.

If you do not see the hgrc file, then you will have to make one!

Copying Text To And From the Command Prompt in Microsoft

When programming in a Microsoft development environment, I actually find the Command Prompt to be fairly useful.  However, I keep forgetting how to copy and paste text to it.  This page shows you how:

https://www.microsoft.com/resources/documentation/windows/xp/all/proddocs/en-us/windows_dos_copy.mspx?mfr=true

Managing Multiple Versions of Python with pyenv

I have been recently working on a project that needed to be tested on multiple versions of Python.  I found pyenv to be invaluable:

https://github.com/yyuu/pyenv

This project was apparently forked rbenv and ruby-build and works about the same.

I also found this article useful:

http://dtucker.co.uk/hack/installing-pyenv-on-mac-osx.html

Python + Gmail = Email Adventures!

Today I learned how to send email using Gmail as a service provider.  I learned from reading this thread:

http://stackoverflow.com/questions/10147455/trying-to-send-email-gmail-as-mail-provider-using-python

Location of bower_components directory in AngularJS phone tutorial

I'm actually enjoying the Angular JS tutorial: http://docs.angularjs.org/tutorial

However, I have a sneaking suspicion that the bower_components directory should be located on the same level as the app directory (not inside of it).

christohersmbp2:angular-phonecat christopherspears$ ll | awk '{print $9}'

LICENSE

README.md

app/

bower.json

bower_components/ <- moved it here

node_modules/

npm-debug.log

package.json

scripts/

test/

Am I wrong?

Location of the Procfile

Keep your Procfile in your project's top directory!  Otherwise, gunicorn (the server running your app) will not be able to see the Procfile!

/Users/christopherspears/PyDevel/tapeworm_django/

Procfile

README.md

requirements.txt

tapeworm/

  __init__.py

  __init__.pyc

  drawings/

  manage.py

  tapeworm/

  templates/

Deploying a Python Django app to Heroku

Today I decided to try to deploy my app to Heroku.  Fortunately, I found a pretty good tutorial that shows me how to do this:

https://devcenter.heroku.com/articles/getting-started-with-django

I will be detailing my trials and tribulations as I attempt to do this.  My first issue was that I could not get foreman to work.  The gem foreman is not part of the django-toolbelt.  It is part of the Heroku ToolBelt.  I fixed this by uninstalling and reinstalling the gem.  I then discovered that the Procfile I created needed to be at the same level as the manage.py file.  Otherwise, I would get a bunch of this:

(tapeworm_django)christohersmbp2:tapeworm_django christopherspears$ foreman start

11:18:52 web.1  | started with pid 2122

11:18:52 web.1  | 2014-03-28 11:18:52 [2122] [INFO] Starting gunicorn 18.0

11:18:52 web.1  | 2014-03-28 11:18:52 [2122] [INFO] Listening at: http://0.0.0.0:5000 (2122)

11:18:52 web.1  | 2014-03-28 11:18:52 [2122] [INFO] Using worker: sync

11:18:52 web.1  | 2014-03-28 11:18:52 [2125] [INFO] Booting worker with pid: 2125

11:18:52 web.1  | 2014-03-28 11:18:52 [2125] [ERROR] Exception in worker process:

11:18:52 web.1  | Traceback (most recent call last):

11:18:52 web.1  |   File "/Users/christopherspears/.virtualenvs/tapeworm_django/lib/python2.7/site-packages/gunicorn/arbiter.py", line 495, in spawn_worker

11:18:52 web.1  |     worker.init_process()

11:18:52 web.1  |   File "/Users/christopherspears/.virtualenvs/tapeworm_django/lib/python2.7/site-packages/gunicorn/workers/base.py", line 106, in init_process

11:18:52 web.1  |     self.wsgi = self.app.wsgi()

11:18:52 web.1  |   File "/Users/christopherspears/.virtualenvs/tapeworm_django/lib/python2.7/site-packages/gunicorn/app/base.py", line 114, in wsgi

11:18:52 web.1  |     self.callable = self.load()

11:18:52 web.1  |   File "/Users/christopherspears/.virtualenvs/tapeworm_django/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 62, in load

11:18:52 web.1  |     return self.load_wsgiapp()

11:18:52 web.1  |   File "/Users/christopherspears/.virtualenvs/tapeworm_django/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 49, in load_wsgiapp

11:18:52 web.1  |     return util.import_app(self.app_uri)

11:18:52 web.1  |   File "/Users/christopherspears/.virtualenvs/tapeworm_django/lib/python2.7/site-packages/gunicorn/util.py", line 354, in import_app

11:18:52 web.1  |     __import__(module)

11:18:52 web.1  | ImportError: No module named tapeworm.wsgi

11:18:52 web.1  | Traceback (most recent call last):

11:18:52 web.1  |   File "/Users/christopherspears/.virtualenvs/tapeworm_django/lib/python2.7/site-packages/gunicorn/arbiter.py", line 495, in spawn_worker

11:18:52 web.1  |     worker.init_process()

11:18:52 web.1  |   File "/Users/christopherspears/.virtualenvs/tapeworm_django/lib/python2.7/site-packages/gunicorn/workers/base.py", line 106, in init_process

11:18:52 web.1  |     self.wsgi = self.app.wsgi()

11:18:52 web.1  |   File "/Users/christopherspears/.virtualenvs/tapeworm_django/lib/python2.7/site-packages/gunicorn/app/base.py", line 114, in wsgi

11:18:52 web.1  |     self.callable = self.load()

11:18:52 web.1  |   File "/Users/christopherspears/.virtualenvs/tapeworm_django/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 62, in load

11:18:52 web.1  |     return self.load_wsgiapp()

11:18:52 web.1  |   File "/Users/christopherspears/.virtualenvs/tapeworm_django/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 49, in load_wsgiapp

11:18:52 web.1  |     return util.import_app(self.app_uri)

11:18:52 web.1  |   File "/Users/christopherspears/.virtualenvs/tapeworm_django/lib/python2.7/site-packages/gunicorn/util.py", line 354, in import_app

11:18:52 web.1  |     __import__(module)

11:18:52 web.1  | ImportError: No module named tapeworm.wsgi

11:18:52 web.1  | 2014-03-28 11:18:52 [2125] [INFO] Worker exiting (pid: 2125)

11:18:52 web.1  | 2014-03-28 11:18:52 [2122] [INFO] Shutting down: Master

11:18:52 web.1  | 2014-03-28 11:18:52 [2122] [INFO] Reason: Worker failed to boot.

11:18:52 web.1  | exited with code 3

11:18:52 system | sending SIGTERM to all processes

By the way, I always forget to do this when creating a Heroku app:

(tapeworm_django)christohersmbp2:tapeworm_django christopherspears$ heroku apps:create tapeworm

Creating tapeworm... done, stack is cedar

http://tapeworm.herokuapp.com/ | git@heroku.com:tapeworm.git

Git remote heroku added

Otherwise, Heroku is going to give your app a name like "Flowing Downstream Leaves", which is annoying.  I managed to successfully push my app to Heroku.  The next step is to try to turn it on.

Django Json Field

Been looking for a good json field for Django.  I hope this will work:

https://github.com/bradjasper/django-jsonfield

Creating a requirements.txt file

You can create a requirements.txt file for a Python project with pip:

(tapeworm_django)christohersmbp2:tapeworm_django christopherspears$ pip freeze > requirements.txt

(tapeworm_django)christohersmbp2:tapeworm_django christopherspears$ cat requirements.txt 

Django==1.6.2

psycopg2==2.5.2

wsgiref==0.1.2

Setting environmental variables in virtualenv

I am taking a stab at my first Django project.  As with all web apps, I am concerned with security, so I decided to set my SECRET_KEY as an environmental variable.  Fortunately, this is not too hard to do if you have virtualenvwrapper.  Here is a pretty good Stack Overflow thread about that:

http://stackoverflow.com/questions/9554087/setting-an-environment-variable-in-virtualenv

After reading the thread multiple times, I did the following. First I located the bin directory of the virtual environment I was using ($HOME/.virtualenvs/tapeworm_django/bin).  Inside I found the postactivate hook, which is an executable file and modified it like so:

#!/bin/bash

# This hook is run after this virtualenv is activated.

export SECRET_KEY=myKey

Then I found the predactivate hook and added the following:

#!/bin/bash

# This hook is run before this virtualenv is deactivated.

unset SECRET_KEY

Now will the SECRET_KEY variable will be set and unset when I decided to work on the tapeworm_django virtual environment:

christohersmbp2:tapeworm_django christopherspears$ workon tapeworm_django

(tapeworm_django)christohersmbp2:tapeworm_django christopherspears$ echo $SECRET_KEY

myKey

(tapeworm_django)christohersmbp2:tapeworm_django christopherspears$ deactivate

christohersmbp2:tapeworm_django christopherspears$ echo $SECRET_KEY

christohersmbp2:tapeworm_django christopherspears$

Now in settings.py file of the project configuration directory, I set secret key like so:

SECRET_KEY =  os.environ['SECRET_KEY']

Comparing Three Python Web Frameworks

I have been spending a lot of time noodling around with different Python web frameworks.  Specifically, I have played with Django, Bottle, and Pyramid.  Here are my thoughts on them.  Since the first MVC framework I learned is Rails, I will be comparing some of them to Rails.

1. Django

Basically Python's answer to Rails (or vice versa?).  Django is a powerful, fully developed, and very structured framework that can be used to make professional web apps.  If you want a job, this is the one to learn.  One significant difference between Rails and Django is that you work on Django project, which can include multiple apps, where in Rails you just work on one app.  This promotes code reuse because if you are clever you can use one Django app in several projects.

Models are called models in both Django and Rails.  What are called controllers in Rails are called views in Django, and views in Rails are called templates.  Django does not have a concept of scaffolding to speed up setting up routes like Rails does.  You have to write all of the routes out.

2. Bottle

Bottle interests me because I like the idea of carrying an app in my pocket.  Seriously, the code for the Bottle framework is contained in one file!  One way to install Bottle is to just download the bottle.py file.  My guess is that the Ruby equivalent is Sinatra.

Bottle is very minimal, and you have to build everything from the ground up, which can be liberating or frustrating depending on your personality, unlike Django, which gives you a lot of bells and whistles out of the box.

3. Pyramid

Pyramid wants you to have your cake and eat it too.  You can write a very minimal web app (like the "Hello World" app you write in one of Pyramid's tutorials), or you can write a larger app using a directory structure similar to Django.  Flexibility seems to be the trademark of this framework.

Install twisted in a virtual environment on OS X Mavericks

A recruiter suggested that I look at twisted before having an interview with a client.  Strangely enough, learning how to install twisted in a virtual environment on OS X Mavericks is not easy probably because some of twisted is installed in OS X anyway.  However, I was not using the system python.  I was using the python I installed with brew, so I kept getting this error message:

ImportError: No module named twisted

After poking around a little bit, I finally broke down and asked Stack Overflow:

http://stackoverflow.com/questions/22585393/using-twisted-on-os-x-mavericks/22585567?noredirect=1#22585567

First, I made a virtual environment called try_twisted, and then I installed twisted with pip:

(try_twisted)christohersmbp2:try_twisted christopherspears$ pip install twisted

Now twisted works:

(try_twisted)christohersmbp2:try_twisted christopherspears$ python

Python 2.7.5 (default, Aug 25 2013, 00:04:04) 

[GCC 4.2.1 Compatible Apple LLVM 5.0 (clang-500.0.68)] on darwin

Type "help", "copyright", "credits" or "license" for more information.

>>> import twisted

>>> 

“Cross origin requests are only supported for HTTP.”

Got a really fun error message today when trying to load a html file with Paper.js using file:///.  Turns out for I needed to actually spin up a server to load the file.  The explanation why can be found here:

http://stackoverflow.com/questions/10752055/cross-origin-requests-are-only-supported-for-http-error-but-im-loading-a-co

Spinning up a server is really easy thanks to the magic of Python:

python -m SimpleHTTPServer

Fixing mkvirtualenv after installing OS X Mavericks

As expected, installing OS X Mavericks caused a few of my hacker tools to break.  The most annoying issue was getting virtualenv and virtualenvwrapper to work again.  Figured it out after 48 hours:

http://stackoverflow.com/questions/22238170/trouble-using-mkvirtualenv-after-installing-os-x-mavericks

Yes, dealing with !@#! like this is a pain, but I did learn a lot by fixing this.

Bottle Tutorials

Finally finished the Bottle tutorial:

http://bottlepy.org/docs/dev/tutorial.html

Involved more reading than I thought, so I want to work on a more hands on tutorial next.  This looks good:

http://bottlepy.org/docs/dev/tutorial_app.html

The app you end up building is the ever popular to do list.

Learning Ember

Finally got around to finishing this tutorial: http://emberjs.com/guides/getting-started/

Of course, now I need to brush up on my Angular.

Django project to work on?

I'm getting to the point where I am trying to think of a Django project to work on.  Any hints?  My only requirement is that it should not be a to do list!

Differences Between Rails And Django

As I launch into the world of Python/Django, I am going to be comparing and contrasting it to Ruby/Rails.  One significant difference is that in Rails you work on just one app.  When you work on Django, you create a project that contains multiple apps, and in fact, if you are smart, you can then reuse apps in different projects.  Of course, with great power comes great responsibility.  You will need to wire up the apps in the project.

Django puts what I consider controllers in an app's views.py file.  What I consider views are put in a template folder.  Both frameworks have template engines.  Rails uses embedded Ruby.  Django has a template engine that looks like embedded Python.  

From what I can see, Django does not have any scaffolding.  Strangely enough, I do not miss it.  I guess the frameworks reflect the philosophy of each language.  Ruby has multiple ways to do tasks and has many shortcuts.  In Python, you can only do a task one way, and that language seems to value clarity over shortcuts.  I have heard that packages might exist that allows scaffolding, but I would have to do some research.

I am sure I will be able to find more differences and similarities as I travel further down this path.

Tango With Django

Going to check out this online Django book:

http://www.tangowithdjango.com/

CoffeKlatsch is finished!

Finished my first ember.js project.  Feel free to checkout my Github repo:

https://github.com/cspears2002/coffeeklatsch

Here is a screen grab of the SPA to show off the styling:

The styling was done using Twitter Bootstrap.  I might consider hosting this on AWS, but right now, I feel like relaxing.

CoffeeKlatsch has CRUD!

I implemented CRUD in CoffeeKlatsch!  Check out the code at:

https://github.com/cspears2002/coffeeklatsch

Now for some styling!

CoffeeKlatsch

Wow.  Ember.js has a REALLY steep learning curve.  Here is what I have accomplished so far:

https://github.com/cspears2002/coffeeklatsch

One problem I have been having has been getting my Ember.js apps to work on JSFiddle, which has been a tremendous pain.  Finally got this up and running:

http://jsfiddle.net/cspears2002/y8MT3/

Right now users can display and add coffee prices.  I am still on working on adding the ability to edit and delete.

Learning Ember.js

Ember.js is like Angular on steroids.  Strangely enough, some of the code reminds me of Python microframeworks like Bottle.

Understanding Python Decorators

As I learn more about web frameworks in Python, I am getting more exposure to Python decorators.  Decorators seem easy to use, but I have a hard time understanding them.  Hoping this article will clear them up:

http://simeonfranklin.com/blog/2012/jul/1/python-decorators-in-12-steps/

Question About Rails Asset Pipeline

From staring at my old code and poking around the web, I figured out that the Rails Asset Pipeline loads up all of your assets (Javascript, CSS, etc.) and smashes them together into a manifest file for fast and easy digestion.  When does this happen?  When your app starts up?

Scraping Success!

Managed to send out a Spider using Scrapy to scrape a website and save the results in a JSON file!  Now I just need to work on some formatting for the JSON file.

Writing A Spider In Python

Even though I successfully scraped a website with Ruby and Nokogiri, I decided to continue explore scraping websites with Python and Scrapy.  Today I wrote my first spider in Python.  You can watch my progress in my github repository:

https://github.com/cspears2002/scrapestat

I need to look up deploying Python to Heroku:

https://devcenter.heroku.com/articles/getting-started-with-python

Actually, deploying to Heroku might not make any sense.  This is not a web app.

Running A Quick Webserver With Python

Python never ceases to amaze me.  Here is how to run a quick web server using Python from the command line:

unknownc8e0eb148153:D3 christopherspears$ python -m SimpleHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...
127.0.0.1 - - [10/Jan/2014 10:53:35] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [10/Jan/2014 10:53:36] "GET /data.tsv HTTP/1.1" 200 -
127.0.0.1 - - [10/Jan/2014 10:53:47] "GET / HTTP/1.1" 200 -
127.0.0.1 - - [10/Jan/2014 10:53:47] "GET /data.tsv HTTP/1.1" 200 -

127.0.0.1 - - [10/Jan/2014 10:54:13] "GET / HTTP/1.1" 200 -

As you can see, the server is serving up the data.tsv file in the directory.  Here are some helpful links:

http://www.linuxjournal.com/content/tech-tip-really-simple-http-server-python

http://stackoverflow.com/questions/12647196/how-do-i-shut-down-a-python-simplehttpserver

Scrapy is a framework

Weird.  Scrapy is actually a framework.  I feel like I am working in Django or Ruby on Rails again.

Scraping A Website With Python

Going to explore scraping a website with Python and Scrapy (http://scrapy.org/).

Python Installation Odds And Ends

Forgot to add this to the bottom of my .bash_profile file:

#Python

export PATH=/usr/local/share/python:$PATH

My site-packages folder is installed here:
/usr/local/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages

Which virtual environment am I working on?

Here is how to find out (as well as set and exit the virtual environment):

Christophers-MacBook-Pro-2:scrapestat christopherspears$ workon

scrapestat
Christophers-MacBook-Pro-2:scrapestat christopherspears$ workon scrapestat
(scrapestat)Christophers-MacBook-Pro-2:scrapestat christopherspears$ echo $VIRTUAL_ENV
/Users/christopherspears/.virtualenvs/scrapestat
(scrapestat)Christophers-MacBook-Pro-2:scrapestat christopherspears$ deactivate
Christophers-MacBook-Pro-2:scrapestat christopherspears$ echo $VIRTUAL_ENV

Setting Up virtualenvwrapper on Mac OS X

The software virtualenvwrapper is a wrapper around virtualenv that has a lot of neat functions that allows the user to manage multiple virtual environments.  I decided to install the wrapper today.

First I ran the following:

$ pip install virtualenvwrapper

Then inside of the .bash_profile file inside of my home directory I added the following lines.

# virtualenvwrapper stuff
export WORKON_HOME=$HOME/.virtualenvs
export PROJECT_HOME=$HOME/PyDevel

source /usr/local/bin/virtualenvwrapper.sh

Finally I sourced .bash_profile:

$ source ~/.bash_profile

Now I have a directory called .virtualenvs/ inside of my home directory.  Inside of my home directory, I create the 'PyDevel' directory.

Then I created my project's directory:

Christophers-MacBook-Pro-2:~ christopherspears$ mkproject scrapestat
New python executable in scrapestat/bin/python2.7
Also creating executable in scrapestat/bin/python
Installing Setuptools..............................................................................................................................................................................................................................done.
Installing Pip.....................................................................................................................................................................................................................................................................................................................................done.
Creating /Users/christopherspears/PyDevel/scrapestat

Setting project for scrapestat to /Users/christopherspears/PyDevel/scrapestat