Deploying New Relic on Azure

I created a New Relic account a while ago, but never got around to deploying it, and then when I tried to log in again it told me
a) my account was not verified, I needed to click the verification link in the email
b) the verification link in the email was not valid
I emailed support and never heard back, so just gave up on it.

Today I saw they were doing a new promotion of access to Tutes+, which looked useful, so I figured it was time to sign up for a new account (as the old one was completely empty anyway, easier to abandon it than work through support). I signed up for a new account and chose to deploy the python agent. They provided some pretty easy-looking instructions, and I went to open up the django local copy of my python site, and it wouldn't run. Somehow, I have no idea why, I seem not to have python2.7 on my machine any more. So I had to reinstall that, following the steps at http://docs.python-guide.org/en/latest/starting/install/win/. Since I was resetting it all, I figured I'd try better practices this time and installed virtualenv as well.

C:\Dropbox\code\optimization\SimplySchedule>virtualenv --python=C:\Python27\python27.exe scheduler

(env)>pip install newrelic

....
   ***************************************************************************

   WARNING: The optional C extension components of the Python agent could
   not be compiled. This can occur where a compiler is not present on the
   target system or the Python installation does not have the corresponding
   developer package installed. The Python agent will instead be installed
   without the extensions. The consequence of this is that although the
   Python agent will still run, JSON encoding/decoding speedups will not be
   available, nor will some of the non core features of the Python agent.

   INFO: Only pure Python agent was installed.

   ***************************************************************************
....

I guess I probably don't have a C compiler installed, now I feel bad.

but....then it said I needed to relaunch the app with a different init string. Or, since I don't have access to actually do the launching, I need to edit the wsgi file.

modify the WSGI script file/module for your web application and insert at the start of that file:

import newrelic.agent
newrelic.agent.initialize('newrelic.ini')

I had no idea where that was at first, but after some googling (and temporary confusion between this and the wfastcgi.py file) I found the wsgi file in django.core and added those lines, but got an import error trying to start the server. I eventually realised that I had installed newrelic for python3, not python2 (sigh) but while noodling around looking for hints on that, I found a more detailed new relic setup guide which mentioned you should edit the newrelic.ini file to choose a real log file location and set the name of the app. So I did pip2 install newrelic, copied the files to the site-packages directory, and python27 manage.py runserver : everything works!

While I was fiddling with everything, I also took the time to set up a slightly more useful index/home page for the site. Then when I went to push to github for deployment (which is super nice) the local repository wasn't associated with the remote - maybe I lost all the old settings when I installed windows 8.1 over the preview? So I also had to reassociate my local copy of the repository with github to push to that - and it's running with the new pages :)

File locations on azure

Spent the last few days getting everything working in azure with correct folder permissions and so on, leaning heavily on stack overflow for hints. First I found that the templates didn't work in azure because I hardcoded the template locations to c:\users\jafitzge…, had to figure out how to put a location that would work in both the dev and production environments

• http://stackoverflow.com/questions/3038459/django-template-path
• http://stackoverflow.com/questions/6098275/django-problem-with-template-paths

Then I got permissions errors trying to write my result files and debug files

[Errno 2] No such file or directory: 'SampleData2.csv'

[Errno 13] Permission denied: 'scheduledebug10.log'

I tried looking up azure debugging options to see if there was a standard location for log files, etc. http://msdn.microsoft.com/en-us/magazine/ff714589.aspx is only about .net logging options apparently?

http://social.msdn.microsoft.com/Forums/en-US/windowsazuredevelopment/thread/7733a3d8-e80f-4ac9-b285-bf9c652896fd is promising, how do I write to a file in python on azure? Answer leads to

http://msdn.microsoft.com/en-us/library/ee758708.aspx which talks about using the Azure Managed Library to access Local Resources.

http://stackoverflow.com/questions/15146545/ioeerror-in-python-no-such-file-or-directory suggested I needed to change from writing to the current working directory, and instead write to the module directory.  Locally the file is at C:\Users\...\SimplySchedule which is PROJECT_PATH.

Finally fixed all the file permission issues by using the correct folders, and extracted the file path calculations to a couple functions so I could be sure I was doing it the same way everywhere.

Now working on serving the saved schedule file as a download, which seems like should be done using the MEDIA settings. Checking out the django docs again at https://docs.djangoproject.com/en/dev/topics/files/ means I should change the file save location from

import os.path

BASE = os.path.dirname(os.path.abspath(__file__))

To

>>> from django.core.files.storage import default_storage
>>> from django.core.files.base import ContentFile

>>> path = default_storage.save('/path/to/file', ContentFile('new content'))

From <https://docs.djangoproject.com/en/dev/topics/files/>

This stuff looks relevant: http://stackoverflow.com/questions/883313/django-excel-xlwt/883351#883351

This implies I should be saving my debug files to STATIC, not media - that makes sense! http://stackoverflow.com/questions/6619464/django-serving-from-media-but-not-static?rq=1

Also looks interesting for when I create an actual model for all the data I'm throwing around: saving an uploaded file as a model: https://docs.djangoproject.com/en/dev/topics/http/file-uploads/

Now I tried creating public facing 404 and 500 templates and setting Debug=False but it didn't seem to have taken hold when I deployed, although it was working as expected locally.  However when I went back to look at it again a couple hours later it was working as expected. Possibly just a caching issue. 

So, current state - I have all the functionality requested available,  although the site itself looks ugly as hell and super amateur. Still waiting for dad to go check it out and give some feedback, and until then it's time to get some PAX Aus app preparation done.

New project: python+django+azure

Dad emailed me a few weeks ago asking what's the best way to assign 200 conference attendees to workshops, where there are 10 workshops each running 3 times at the conference (all at the same time). So I said I'd write an app that would take a csv file of (attendee, pref, ... prefN) and spit out a csv file of attendee, slot1, slot2, slot3). Easy, I got a naive version up on github a few days later.


Then I realised that it would be kind of a nightmare packaging this app to send to dad for him to try and run and then me remotely debugging for him, and getting him to use an updated version. So the obvious solution is to put it online and have him access it as a web app. This has the side benefit of getting me to actually try building a real website again.

Since I've been meaning to try setting stuff up on azure forever, and the app is arleady in python, I chose a python site on azure. Since I don't know what I'm doing I looked for a tutorial, and the first one suggested I use Django. I'm not actually quite clear on what Django will handle and what I could do without it, but for now I'll go with it. Following the tutorial at
http://www.windowsazure.com/en-us/develop/python/tutorials/web-sites-with-django/ was super straightforward, although since I already had a github account set up I didn't even need to do the deploy with git, when I chose github as my source control it started just auto deploying the latest copy of master. Unfortunately at the end, instead of their nice example page, I had a server error :/

Possibly it's because I renamed my app SimplySchedule instead of the suggested DjangoApplication - I may have missed a place that used the original name? Seems odd though.

Soo….an early opportunity to learn how to check out the logs and troubleshoot! Starting here: http://www.windowsazure.com/en-us/develop/net/best-practices/troubleshooting-web-sites/
So I set a new deployment credential to use with ftp and git - fails the first time when I try and get the ftp site, you need to use the domain as well. The logs don't seem that helpful:



Aha - having turned on detailed logging, I got this info

D:\Python27\python.exe - The FastCGI process exceeded configured request timeout

Module	FastCgiModule
Notification	ExecuteRequestHandler
Handler	FastCGI_SimplySchedule_ccdfe77c-40fe-4aec-84e4-bb292a25a799
Error Code	0x80070102

Requested URL	http://simplyschedule.azurewebsites.net:80/
Physical Path	C:\DWASFiles\Sites\SimplySchedule\VirtualDirectory0\site\wwwroot
Logon Method	Anonymous
Logon User	Anonymous

So, it is at least getting to the site and running the fastcgi handler is setup…

I went and doublechecked all the config options, and restarted the site. When I tried visiting it again I got this:

ImportError at /

cannot import name sql

Request Method:	GET
Request URL:	http://simplyschedule.azurewebsites.net/
Django Version:	1.4
Exception Type:	ImportError
Exception Value:	cannot import name sql
Exception Location:	D:\home\site\wwwroot\site-packages\django\db\models\deletion.py in <module>, line 5
Python Executable:	D:\Python27\python.exe
Python Version:	2.7.3
Python Path:	['D:\\Python27\\Scripts',  'D:\\home\\site\\wwwroot\\SimplySchedule',  'D:\\home\\site\\wwwroot\\site-packages',  'D:\\Windows\\SYSTEM32\\python27.zip',  'D:\\Python27\\DLLs',  'D:\\Python27\\lib',  'D:\\Python27\\lib\\plat-win',  'D:\\Python27\\lib\\lib-tk',  'D:\\Python27',  'D:\\Python27\\lib\\site-packages']
Server time:	Tue, 21 May 2013 06:19:54 +0000

Environment:

Request Method: GET

Request URL: http://simplyschedule.azurewebsites.net/

Django Version: 1.4

Python Version: 2.7.3

Installed Applications:

('django.contrib.auth',

 'django.contrib.contenttypes',

 'django.contrib.sessions',a

 'django.contrib.sites',

 'django.contrib.messages',

 'django.contrib.staticfiles')

Installed Middleware:

('django.middleware.common.CommonMiddleware',

 'django.contrib.sessions.middleware.SessionMiddleware',

 'django.middleware.csrf.CsrfViewMiddleware',

 'django.contrib.auth.middleware.AuthenticationMiddleware',

 'django.contrib.messages.middleware.MessageMiddleware')

Traceback:

File "D:\home\site\wwwroot\site-packages\django\core\handlers\base.py" in get_response

  89.                     response = middleware_method(request)

File "D:\home\site\wwwroot\site-packages\django\contrib\sessions\middleware.py" in process_request

  10.         engine = import_module(settings.SESSION_ENGINE)

File "D:\home\site\wwwroot\site-packages\django\utils\importlib.py" in import_module

  35.     __import__(name)

File "D:\home\site\wwwroot\site-packages\django\contrib\sessions\backends\db.py" in <module>

  77. from django.contrib.sessions.models import Session

File "D:\home\site\wwwroot\site-packages\django\contrib\sessions\models.py" in <module>

  1. from django.db import models

File "D:\home\site\wwwroot\site-packages\django\db\models\__init__.py" in <module>

  5. from django.db.models.query import Q

File "D:\home\site\wwwroot\site-packages\django\db\models\query.py" in <module>

  13. from django.db.models.deletion import Collector

File "D:\home\site\wwwroot\site-packages\django\db\models\deletion.py" in <module>

  5. from django.db.models import signals, sql

Exception Type: ImportError at /

Exception Value: cannot import name sql

Welp, after a little blind web searching, I can't find anyone else hitting this error - I think. There is a mention of someone else who went through the tutorial and saw a 500 internal server error in a comment on this Stack Overflow question,: http://stackoverflow.com/questions/10990706/azure-free-website-with-python-django-not-running/15757329#15757329
But a 500 error could be lots of things, as shown by this question: http://stackoverflow.com/questions/12554725/azure-sdk-django-visual-studio-2012-publish-to-azure-succeeds-but-i-get-5?rq=1

So, I asked my question at the disqus comments on the tutorial - will wait a day for a response before going to Stack Overflow I guess