Friday, May 24, 2013

Django and sql azure, wtf?


Chapter 5 of the Django Book - Models

I thought about just skipping this one as I don't anticipate needing a db for the schedule app, but decided I might as well follow through, it'll probably come in handy soon enough.

So step 1 I created a sql database in azure to use, and now I need a sql engine for python.

I found https://pypi.python.org/pypi/django-pyodbc-azure/1.0.5 which told me to install pyodbc first, which was googleable enough, and available for both 2.7 and 3.x. But the 2.7 version said it couldn't install bc I don't have 2.7 in the registry…hmm.  After some hints from Stack Overflow, I checked the python version I have for 2.7 and it's the 32 bit version, which won't be found by the 64 bit installer. Ran the 32 bit installer and done. Then it said to use pip to install itself, which I didn't have, but SO led me to this great page: http://www.lfd.uci.edu/~gohlke/pythonlibs/#pip…but after running that, pip still wasn't recognised on the command line. Another useful note on SO said it installed to python/scripts instead of just to /python, so I had to add that to my path. And now I get 'failed to create process'. Tried in admin mode, same error. Google was initially not forthcoming about this error, but then turned up this: http://www.maphew.com/Python/python-fixit-snippets/ which suggested that it was unable to find python. Perhaps it is confused by my python27 option? The location certainly hasn't changed since I installed it. I uninstalled and reinstalled to be sure, but that didn't help.

Checking the Event Viewer to see where it's trying to find python might help -nope, can't see any relevant messages.

I went back and installed Distribute, as it is mentioned in the same place as where I got pip. No change.

I found this gui for pip that lets you switch between active python installers - that might help, because it could still be being caused by my dual 2 and 3 installation? https://sites.google.com/site/pydatalog/python/pip-for-windows but it also failed. But it did point out that I have no pip.exe in c:\python27 - which is odd because I do have a Removepip.exe? Turns out pip.exe is in c:\python27\scripts, but moving it just got me back to failed to create process. Hmmmmmm.

So I went back to the site, and noticed it had a downloadable zip. So I downloaded that, and ran setup.py install, and it appears to have installed something? But not a full package. In fact I can't even see a real pyodbc package...

But just to make sure it isn't installed, let's try following the instructions on the pyodbc-azure site. And nope, I get jango.core.exceptions.ImproperlyConfigured: 'sql_server.pyodbc' isn't an available database backend.
 
So, that was a couple of unproductive hours, and I give up. I will instead attempt to follow this tutorial (http://www.windowsazure.com/en-us/develop/python/tutorials/web-app-with-blob-storage/) for connecting a python app to azure storage, which should be perfectly adequate for anything.  And to get through the Django book I can just go ahead and use sqllite.

Getting started with sqllite: http://sqlite.org/sqlite.html
Easy!
Quickly created and connected to a database, created a model, turned it into tables, added data to the tables, and started querying the data out of it
  • All rows
  • Filter: Field = x, field contains x (see Appendix C for field == x, or < x?)
  • Get(): A single object instead of a list ( filter returns a list) uses same options as filter
  • Order_by
  • Slicing - works just like a python list!
  • Use .update(x=1) instead of pub.name='a' to get more efficient sql translations (setting a field directly will cause all fields to be saved over)
  • .delete() can be called on an object or a query set (result of .filter()) Django tries to save you from accidentally deleting everyhting by requiring the odd syntax x.objects.all().delete() to do that
 

Thursday, May 23, 2013

It really worked!


Chapter 4 of the Django book got off to a poor start when I followed their instructions to run 'python manage.py shell', and unexpectedly got a syntax error from one of the core django files. It seemed unlikely there was an actual issue with the file (didn't I even run it myself yesterday?), and after googling the error one of the results mentioned 'of course, I have python 3' so I remembered that I have python 2 and 3 installed, but the django installation came with python 2. I updated my path and the python 2 executable so I could call 'python2' to get python 2 running, leaving my default of 3 still there, and running under python 2 succeeded.

However this does mean I'll have to check my app, which was written in 3, for compatibility with 2 - at the very least all those print statements will be wrong :/ It also means that I can't run my python programs in Sublime Text because it is in Python 3.
Whoa. Actually the only change I had to make was remove the 'newline' option from csv.reader. Way easier than expected!

Useful introductions in the first half of this chapter: templates and contexts, template variables, accessing lists, dicts, and methods through dots, basic logic tags available in templates, filters, comments. Further details can be found in the appendix, which they recommend you read so as to familiarise yourself with all the available options.

And now part 2 - using a template to create a view.

Step 1 - put the full path to your templates files in the settings file. That's easy enough locally, but I'll have to look up what it will be on azure. For now I guess I can just use os.path of the settings file (prompted by their example) but that might not be where I want it to live.

Step 2 - update the views to call templates. Easy enough.

Step 3 - template inheritance. Awesome!

I followed along with code as they went, creating my own side example of attendee schedules to make sure I was getting it - everything worked as expected, it felt really simple but quite powerful. I committed it all to Azure and that worked straight off as well.

Wednesday, May 22, 2013

Success: Django hello world on azure!



Something made me think of verifying all the django packages in my github repository, as well as my local repository, and I realised that the whole sql folder was missing. I tried to add the files in it manually and got ' did not match any files' - so obviously there was something wrong with my .gitignore file…I looked through it for everything python related, and then noticed a 'sql' entry. Doh, it was ignoring the whole folder. Removed that line, commit, deploy, and voila!

Now to figure out manage.py…

So, found a tutorial.
Step 1: run manage.py runserver
Result: error "ImportError: No module named 'django'
Fixed!

And now, found an introduction to django - a whole book, apparently. Which seems good as I have no idea how to get my existing app running through django. http://www.djangobook.com/en/2.0/chapter02.html

Was good that they included the test 'import django' instructions, as it turned out I didn't have any of it in my PYTHONPATH - it was all just in my path.

I'll ignore the database section for now, because I don't actually plan to use a database for this at all.  I've already done the create project steps while setting up the site with azure. So I'm straight through to chapter 3: setting up webpages!

Creating views.py looks a little familiar from when I did the first half of that google app engine course - I guess I really should have finished it :)

Pleasantly surprised by the 'django automatically detects code changes' note, I was expecting to have to restart the server :)

Doh, and an import error. Foolishly copying and pasting, of course I don't have a module called mysite.views, I gave it a much more descriptive name! And - got the hello world page going.

Ooh, I bet this will come in handy at some point: 
At any point in your view, temporarily insert an assert False to trigger the error page.


So, half past twelve and end of a chapter - time to commit, confirm it all works on azure as well as my box, and go practice hindi.  And it all works exactly the same! A successful evening.

Tuesday, May 21, 2013

New project: python+django+azure


Dad emailed me a few weeks ago asking what's the best way to assign 200 conference attendees to workshops, where there are 10 workshops each running 3 times at the conference (all at the same time). So I said I'd write an app that would take a csv file of (attendee, pref, ... prefN) and spit out a csv file of attendee, slot1, slot2, slot3). Easy, I got a naive version up on github a few days later.


Then I realised that it would be kind of a nightmare packaging this app to send to dad for him to try and run and then me remotely debugging for him, and getting him to use an updated version. So the obvious solution is to put it online and have him access it as a web app. This has the side benefit of getting me to actually try building a real website again.

Since I've been meaning to try setting stuff up on azure forever, and the app is arleady in python, I chose a python site on azure. Since I don't know what I'm doing I looked for a tutorial, and the first one suggested I use Django. I'm not actually quite clear on what Django will handle and what I could do without it, but for now I'll go with it. Following the tutorial at
http://www.windowsazure.com/en-us/develop/python/tutorials/web-sites-with-django/ was super straightforward, although since I already had a github account set up I didn't even need to do the deploy with git, when I chose github as my source control it started just auto deploying the latest copy of master. Unfortunately at the end, instead of their nice example page, I had a server error :/

Possibly it's because I renamed my app SimplySchedule instead of the suggested DjangoApplication - I may have missed a place that used the original name? Seems odd though.

Soo….an early opportunity to learn how to check out the logs and troubleshoot! Starting here: http://www.windowsazure.com/en-us/develop/net/best-practices/troubleshooting-web-sites/
So I set a new deployment credential to use with ftp and git - fails the first time when I try and get the ftp site, you need to use the domain as well. The logs don't seem that helpful:



Aha - having turned on detailed logging, I got this info
D:\Python27\python.exe - The FastCGI process exceeded configured request timeout
Module
   FastCgiModule
Notification
   ExecuteRequestHandler
Handler
   FastCGI_SimplySchedule_ccdfe77c-40fe-4aec-84e4-bb292a25a799
Error Code
   0x80070102
Requested URL
Physical Path
   C:\DWASFiles\Sites\SimplySchedule\VirtualDirectory0\site\wwwroot
Logon Method
   Anonymous
Logon User
   Anonymous


So, it is at least getting to the site and running the fastcgi handler is setup…

I went and doublechecked all the config options, and restarted the site. When I tried visiting it again I got this:

ImportError at /
cannot import name sql
Request Method:
GET
Request URL:
Django Version:
1.4
Exception Type:
ImportError
Exception Value:
cannot import name sql
Exception Location:
D:\home\site\wwwroot\site-packages\django\db\models\deletion.py in <module>, line 5
Python Executable:
D:\Python27\python.exe
Python Version:
2.7.3
Python Path:
['D:\\Python27\\Scripts',
 'D:\\home\\site\\wwwroot\\SimplySchedule',
 'D:\\home\\site\\wwwroot\\site-packages',
 'D:\\Windows\\SYSTEM32\\python27.zip',
 'D:\\Python27\\DLLs',
 'D:\\Python27\\lib',
 'D:\\Python27\\lib\\plat-win',
 'D:\\Python27\\lib\\lib-tk',
 'D:\\Python27',
 'D:\\Python27\\lib\\site-packages']
Server time:
Tue, 21 May 2013 06:19:54 +0000


Environment:


Request Method: GET

Django Version: 1.4
Python Version: 2.7.3
Installed Applications:
('django.contrib.auth',
 'django.contrib.contenttypes',
 'django.contrib.sessions',a
 'django.contrib.sites',
 'django.contrib.messages',
 'django.contrib.staticfiles')
Installed Middleware:
('django.middleware.common.CommonMiddleware',
 'django.contrib.sessions.middleware.SessionMiddleware',
 'django.middleware.csrf.CsrfViewMiddleware',
 'django.contrib.auth.middleware.AuthenticationMiddleware',
 'django.contrib.messages.middleware.MessageMiddleware')


Traceback:
File "D:\home\site\wwwroot\site-packages\django\core\handlers\base.py" in get_response
  89.                     response = middleware_method(request)
File "D:\home\site\wwwroot\site-packages\django\contrib\sessions\middleware.py" in process_request
  10.         engine = import_module(settings.SESSION_ENGINE)
File "D:\home\site\wwwroot\site-packages\django\utils\importlib.py" in import_module
  35.     __import__(name)
File "D:\home\site\wwwroot\site-packages\django\contrib\sessions\backends\db.py" in <module>
  77. from django.contrib.sessions.models import Session
File "D:\home\site\wwwroot\site-packages\django\contrib\sessions\models.py" in <module>
  1. from django.db import models
File "D:\home\site\wwwroot\site-packages\django\db\models\__init__.py" in <module>
  5. from django.db.models.query import Q
File "D:\home\site\wwwroot\site-packages\django\db\models\query.py" in <module>
  13. from django.db.models.deletion import Collector
File "D:\home\site\wwwroot\site-packages\django\db\models\deletion.py" in <module>
  5. from django.db.models import signals, sql

Exception Type: ImportError at /
Exception Value: cannot import name sql

Welp, after a little blind web searching, I can't find anyone else hitting this error - I think. There is a mention of someone else who went through the tutorial and saw a 500 internal server error in a comment on this Stack Overflow question,: http://stackoverflow.com/questions/10990706/azure-free-website-with-python-django-not-running/15757329#15757329
But a 500 error could be lots of things, as shown by this question: http://stackoverflow.com/questions/12554725/azure-sdk-django-visual-studio-2012-publish-to-azure-succeeds-but-i-get-5?rq=1

So, I asked my question at the disqus comments on the tutorial - will wait a day for a response before going to Stack Overflow I guess