Django and sql azure, wtf?

Chapter 5 of the Django Book - Models

I thought about just skipping this one as I don't anticipate needing a db for the schedule app, but decided I might as well follow through, it'll probably come in handy soon enough.

So step 1 I created a sql database in azure to use, and now I need a sql engine for python.

I found https://pypi.python.org/pypi/django-pyodbc-azure/1.0.5 which told me to install pyodbc first, which was googleable enough, and available for both 2.7 and 3.x. But the 2.7 version said it couldn't install bc I don't have 2.7 in the registry…hmm.  After some hints from Stack Overflow, I checked the python version I have for 2.7 and it's the 32 bit version, which won't be found by the 64 bit installer. Ran the 32 bit installer and done. Then it said to use pip to install itself, which I didn't have, but SO led me to this great page: http://www.lfd.uci.edu/~gohlke/pythonlibs/#pip…but after running that, pip still wasn't recognised on the command line. Another useful note on SO said it installed to python/scripts instead of just to /python, so I had to add that to my path. And now I get 'failed to create process'. Tried in admin mode, same error. Google was initially not forthcoming about this error, but then turned up this: http://www.maphew.com/Python/python-fixit-snippets/ which suggested that it was unable to find python. Perhaps it is confused by my python27 option? The location certainly hasn't changed since I installed it. I uninstalled and reinstalled to be sure, but that didn't help.

Checking the Event Viewer to see where it's trying to find python might help -nope, can't see any relevant messages.

I went back and installed Distribute, as it is mentioned in the same place as where I got pip. No change.

I found this gui for pip that lets you switch between active python installers - that might help, because it could still be being caused by my dual 2 and 3 installation? https://sites.google.com/site/pydatalog/python/pip-for-windows but it also failed. But it did point out that I have no pip.exe in c:\python27 - which is odd because I do have a Removepip.exe? Turns out pip.exe is in c:\python27\scripts, but moving it just got me back to failed to create process. Hmmmmmm.

So I went back to the site, and noticed it had a downloadable zip. So I downloaded that, and ran setup.py install, and it appears to have installed something? But not a full package. In fact I can't even see a real pyodbc package...

But just to make sure it isn't installed, let's try following the instructions on the pyodbc-azure site. And nope, I get jango.core.exceptions.ImproperlyConfigured: 'sql_server.pyodbc' isn't an available database backend.

 

So, that was a couple of unproductive hours, and I give up. I will instead attempt to follow this tutorial (http://www.windowsazure.com/en-us/develop/python/tutorials/web-app-with-blob-storage/) for connecting a python app to azure storage, which should be perfectly adequate for anything.  And to get through the Django book I can just go ahead and use sqllite.

Getting started with sqllite: http://sqlite.org/sqlite.html

Easy!

Quickly created and connected to a database, created a model, turned it into tables, added data to the tables, and started querying the data out of it

• All rows
• Filter: Field = x, field contains x (see Appendix C for field == x, or < x?)
• Get(): A single object instead of a list ( filter returns a list) uses same options as filter
• Order_by
• Slicing - works just like a python list!
• Use .update(x=1) instead of pub.name='a' to get more efficient sql translations (setting a field directly will cause all fields to be saved over)
• .delete() can be called on an object or a query set (result of .filter()) Django tries to save you from accidentally deleting everyhting by requiring the odd syntax x.objects.all().delete() to do that

 

It really worked!

Chapter 4 of the Django book got off to a poor start when I followed their instructions to run 'python manage.py shell', and unexpectedly got a syntax error from one of the core django files. It seemed unlikely there was an actual issue with the file (didn't I even run it myself yesterday?), and after googling the error one of the results mentioned 'of course, I have python 3' so I remembered that I have python 2 and 3 installed, but the django installation came with python 2. I updated my path and the python 2 executable so I could call 'python2' to get python 2 running, leaving my default of 3 still there, and running under python 2 succeeded.

However this does mean I'll have to check my app, which was written in 3, for compatibility with 2 - at the very least all those print statements will be wrong :/ It also means that I can't run my python programs in Sublime Text because it is in Python 3.

Whoa. Actually the only change I had to make was remove the 'newline' option from csv.reader. Way easier than expected!

Useful introductions in the first half of this chapter: templates and contexts, template variables, accessing lists, dicts, and methods through dots, basic logic tags available in templates, filters, comments. Further details can be found in the appendix, which they recommend you read so as to familiarise yourself with all the available options.

And now part 2 - using a template to create a view.

Step 1 - put the full path to your templates files in the settings file. That's easy enough locally, but I'll have to look up what it will be on azure. For now I guess I can just use os.path of the settings file (prompted by their example) but that might not be where I want it to live.

Step 2 - update the views to call templates. Easy enough.

Step 3 - template inheritance. Awesome!

I followed along with code as they went, creating my own side example of attendee schedules to make sure I was getting it - everything worked as expected, it felt really simple but quite powerful. I committed it all to Azure and that worked straight off as well.

Success: Django hello world on azure!

Something made me think of verifying all the django packages in my github repository, as well as my local repository, and I realised that the whole sql folder was missing. I tried to add the files in it manually and got ' did not match any files' - so obviously there was something wrong with my .gitignore file…I looked through it for everything python related, and then noticed a 'sql' entry. Doh, it was ignoring the whole folder. Removed that line, commit, deploy, and voila!

Now to figure out manage.py…

So, found a tutorial.

Step 1: run manage.py runserver

Result: error "ImportError: No module named 'django'

Googling implies a path error : http://stackoverflow.com/questions/5428704/manage-py-runserver-importerror-no-module-named-mysite

Fixed!

And now, found an introduction to django - a whole book, apparently. Which seems good as I have no idea how to get my existing app running through django. http://www.djangobook.com/en/2.0/chapter02.html

Was good that they included the test 'import django' instructions, as it turned out I didn't have any of it in my PYTHONPATH - it was all just in my path.

I'll ignore the database section for now, because I don't actually plan to use a database for this at all.  I've already done the create project steps while setting up the site with azure. So I'm straight through to chapter 3: setting up webpages!

Creating views.py looks a little familiar from when I did the first half of that google app engine course - I guess I really should have finished it :)

Pleasantly surprised by the 'django automatically detects code changes' note, I was expecting to have to restart the server :)

Doh, and an import error. Foolishly copying and pasting, of course I don't have a module called mysite.views, I gave it a much more descriptive name! And - got the hello world page going.

Ooh, I bet this will come in handy at some point: 

At any point in your view, temporarily insert an assert False to trigger the error page.

From <http://www.djangobook.com/en/2.0/chapter03.html>

So, half past twelve and end of a chapter - time to commit, confirm it all works on azure as well as my box, and go practice hindi.  And it all works exactly the same! A successful evening.

New project: python+django+azure

Dad emailed me a few weeks ago asking what's the best way to assign 200 conference attendees to workshops, where there are 10 workshops each running 3 times at the conference (all at the same time). So I said I'd write an app that would take a csv file of (attendee, pref, ... prefN) and spit out a csv file of attendee, slot1, slot2, slot3). Easy, I got a naive version up on github a few days later.


Then I realised that it would be kind of a nightmare packaging this app to send to dad for him to try and run and then me remotely debugging for him, and getting him to use an updated version. So the obvious solution is to put it online and have him access it as a web app. This has the side benefit of getting me to actually try building a real website again.

Since I've been meaning to try setting stuff up on azure forever, and the app is arleady in python, I chose a python site on azure. Since I don't know what I'm doing I looked for a tutorial, and the first one suggested I use Django. I'm not actually quite clear on what Django will handle and what I could do without it, but for now I'll go with it. Following the tutorial at
http://www.windowsazure.com/en-us/develop/python/tutorials/web-sites-with-django/ was super straightforward, although since I already had a github account set up I didn't even need to do the deploy with git, when I chose github as my source control it started just auto deploying the latest copy of master. Unfortunately at the end, instead of their nice example page, I had a server error :/

Possibly it's because I renamed my app SimplySchedule instead of the suggested DjangoApplication - I may have missed a place that used the original name? Seems odd though.

Soo….an early opportunity to learn how to check out the logs and troubleshoot! Starting here: http://www.windowsazure.com/en-us/develop/net/best-practices/troubleshooting-web-sites/
So I set a new deployment credential to use with ftp and git - fails the first time when I try and get the ftp site, you need to use the domain as well. The logs don't seem that helpful:



Aha - having turned on detailed logging, I got this info

D:\Python27\python.exe - The FastCGI process exceeded configured request timeout

Module	FastCgiModule
Notification	ExecuteRequestHandler
Handler	FastCGI_SimplySchedule_ccdfe77c-40fe-4aec-84e4-bb292a25a799
Error Code	0x80070102

Requested URL	http://simplyschedule.azurewebsites.net:80/
Physical Path	C:\DWASFiles\Sites\SimplySchedule\VirtualDirectory0\site\wwwroot
Logon Method	Anonymous
Logon User	Anonymous

So, it is at least getting to the site and running the fastcgi handler is setup…

I went and doublechecked all the config options, and restarted the site. When I tried visiting it again I got this:

ImportError at /

cannot import name sql

Request Method:	GET
Request URL:	http://simplyschedule.azurewebsites.net/
Django Version:	1.4
Exception Type:	ImportError
Exception Value:	cannot import name sql
Exception Location:	D:\home\site\wwwroot\site-packages\django\db\models\deletion.py in <module>, line 5
Python Executable:	D:\Python27\python.exe
Python Version:	2.7.3
Python Path:	['D:\\Python27\\Scripts',  'D:\\home\\site\\wwwroot\\SimplySchedule',  'D:\\home\\site\\wwwroot\\site-packages',  'D:\\Windows\\SYSTEM32\\python27.zip',  'D:\\Python27\\DLLs',  'D:\\Python27\\lib',  'D:\\Python27\\lib\\plat-win',  'D:\\Python27\\lib\\lib-tk',  'D:\\Python27',  'D:\\Python27\\lib\\site-packages']
Server time:	Tue, 21 May 2013 06:19:54 +0000

Environment:

Request Method: GET

Request URL: http://simplyschedule.azurewebsites.net/

Django Version: 1.4

Python Version: 2.7.3

Installed Applications:

('django.contrib.auth',

 'django.contrib.contenttypes',

 'django.contrib.sessions',a

 'django.contrib.sites',

 'django.contrib.messages',

 'django.contrib.staticfiles')

Installed Middleware:

('django.middleware.common.CommonMiddleware',

 'django.contrib.sessions.middleware.SessionMiddleware',

 'django.middleware.csrf.CsrfViewMiddleware',

 'django.contrib.auth.middleware.AuthenticationMiddleware',

 'django.contrib.messages.middleware.MessageMiddleware')

Traceback:

File "D:\home\site\wwwroot\site-packages\django\core\handlers\base.py" in get_response

  89.                     response = middleware_method(request)

File "D:\home\site\wwwroot\site-packages\django\contrib\sessions\middleware.py" in process_request

  10.         engine = import_module(settings.SESSION_ENGINE)

File "D:\home\site\wwwroot\site-packages\django\utils\importlib.py" in import_module

  35.     __import__(name)

File "D:\home\site\wwwroot\site-packages\django\contrib\sessions\backends\db.py" in <module>

  77. from django.contrib.sessions.models import Session

File "D:\home\site\wwwroot\site-packages\django\contrib\sessions\models.py" in <module>

  1. from django.db import models

File "D:\home\site\wwwroot\site-packages\django\db\models\__init__.py" in <module>

  5. from django.db.models.query import Q

File "D:\home\site\wwwroot\site-packages\django\db\models\query.py" in <module>

  13. from django.db.models.deletion import Collector

File "D:\home\site\wwwroot\site-packages\django\db\models\deletion.py" in <module>

  5. from django.db.models import signals, sql

Exception Type: ImportError at /

Exception Value: cannot import name sql

Welp, after a little blind web searching, I can't find anyone else hitting this error - I think. There is a mention of someone else who went through the tutorial and saw a 500 internal server error in a comment on this Stack Overflow question,: http://stackoverflow.com/questions/10990706/azure-free-website-with-python-django-not-running/15757329#15757329
But a 500 error could be lots of things, as shown by this question: http://stackoverflow.com/questions/12554725/azure-sdk-django-visual-studio-2012-publish-to-azure-succeeds-but-i-get-5?rq=1

So, I asked my question at the disqus comments on the tutorial - will wait a day for a response before going to Stack Overflow I guess

Home debugging, part 2: can't create a new user account on windows 7, 'the user profile service service failed the logon'.

The next home network maintenance task took a little longer to solve. My mother wanted to create her own user account on the shared PC. She knew how to do this, but the new account kept giving an error when she tried to log on with it: 'the user profile service service failed the logon'. Cryptic.

It sounded easy. I was confident that I'd solve it in minutes. I googled it, and found several recommendations to delete the corrupt user profile/reg key and boom. But why would the newly created profile be corrupt? Turns out, it's not: it's not even created until you log on with it for the first time. So there was no profile to delete.

I tried turning off UAC, and creating a new account with admin rights instead of standard, but none of this worked. Eventually, I found this Microsoft Answers thread: http://answers.microsoft.com/en-us/windows/forum/windows_7-security/i-cannot-create-a-new-user-account-windows-7-the/dee340f7-5da9-402b-9e99-293dbd249a53?page=2 which pinpointed the problem as an access failure while copying data to the new user profile. I felt a little silly for not thinking to look through the event logs earlier in the process, but here is the step by step guide to fixing this error if you're having the same problem:

- open the Event Viewer (Start->Run->type "Event Viewer") and look at the Application logs


-  Now find any Warnings (the Error messages are likely telling you that the user profile didn't work, which you already know) and click on each of them to read the details about it

 

 

 

- You should be able to find a warning message that says 'Windows cannot copy file x to location C:\Users\Temp...' 

 

- Find the file mentioned by typing in the full directory location given to the address bar of Windows Explorer. (You can't browse to it because it will be hidden). 

 

WARNING: deleting or changing some files will break your computer. If you don't know what the file mentioned is, you should find someone who knows more than you do.

 

- If you know what you are doing, now go ahead and delete or give universal write access on the file. Et voila, your new user account should log on successfully.

Home debugging, part 1: hacked wifi?

And now for something completely different!

I'm at home in Australia for a few weeks* visiting my parents, so obviously I've been doing a little home PC maintenance. Between my dad, sister and brother they are able to pretty well manage everything, so it hasn't been too bad.

My most stunning accomplishment has been setting up a Windows Homegroup so that they can all send documents to the printer from their laptops and the second desktop, instead of kicking off anyone currently using the PC with the printer for a minute.

More interestingly, a couple months ago, my sister rang me early in the morning to say that she thought the wifi had been hacked. I rolled my eyes and asked what was happening, and she said that overnight, all the phones (which get carried in and out of the house, and therefore off the wireless network) had stopped connecting to the local wifi and said the password was incorrect. The laptops which had stayed in the house were still connected, but the laptop that had left the house that day was also unable to connect. When I directed her to the router admin page, which according to Dad's notes still had the default password, she was unable to log in. It appeared that she was right, and someone had indeed managed to change the password on their wifi network. I gave her instructions for resetting the router and setting the network back up, and left her to it.

Fast forward: I've been home for about a week. I'm testing some phone stuff on two different phones. As each phone sits idle, it disconnects from the wireless network, then seamlessly reconnects when I wake it up. Until one doesn't. It says the password is wrong. Then the other phone won't reconnect either. I go find my sisters iPhone and it also won't connect - password incorrect. (I have no interest in disconnecting my laptop to test it as well.) It appears that whatever happened before, has happened again.I go find my sister's documentation from that incident: she didn't change any of the passwords involved, including the router sitting on the default password. So yea, I guess it's not inconceivable that someone got into it again? I reset the router, set back up the network and the voip line, and this time change the router password.

I still find it a little unbelievable that someone would have gotten into their wireless network twice, in this random corner of the suburbs, but I can't come up with another explanation.


*Yes, it was incredibly inconvenient for my house to flood days before I left on an international trip. 

Conflicts and setbacks

So, ran into a few challenges recently.

I have mentioned that I work at Microsoft, but here it becomes relevant that I specifically work on the Windows Phone marketplace: the experience you have when you install an app as a user. As part of my job, I was involved in investigating and diagnosing <a href="http://windowsteamblog.com/windows_phone/b/windowsphone/archive/2012/08/14/can-t-download-an-app-here-s-why.aspx">the recent problem installing app updates</a>, and I also know the team involved in managing app certification, so I was well aware of the subsequent <a href="http://windowsteamblog.com/windows_phone/b/wpdev/archive/2012/08/14/app-publishing-temporarily-on-hold.aspx">delays in app publishing</a>.

This was pretty inconvenient for me personally, as I submitted the PAX Digital Assistant update for Prime to the marketplace only the day before certification was temporarily halted. I waited for my app to make its way through certification once everything resumed, and kept waiting. I checked at work and knew there were major delays catching up with the queue of apps that had built up, and so it wasn't unbelievable that mine hadn't made it through yet. I knew that I could contact the guys who ran the process and have my app fast tracked, but I couldn't decide if that was reasonable or not. In hindsight, I think it would have been: I had a good justification with the event actually happening, and other developers do contact a developer evangelist for similar deadlines. However I decided that it felt a little unfair to use my internal position like that, so I didn't do anything, and my app didn't make it through publishing until September 6, well after the useful moment had passed. Now I feel like I inconvenienced a bunch of people and made myself look flaky and unreliable, all for no good reason. Damn.

On the worse side, four days before PAX my apartment was flooded by a burst pipe, and my laptop was destroyed. As detailed in my last post, I did have a remote backup for the PAX app - but I hadn't done incremental uploads to the server of the last couple days of work I'd done before finishing it for Prime, so that work is now gone forever. The Flashcard app I'd created is also completely gone, as I hadn't bothered backing it up to the server at all (not a huge loss, but a foolish mistake). Could have been worse, of course: a month earlier the entire PAX app was stored only on my local machine and would have been gone. But could have been better as well. I need to make the incremental backup to the server as easy as the local incremental backup.