Automated Calendar Planning

My initial interest in AI planning comes from a promise I made to a friend years ago, which I was unable to keep. He was an amateur theater director, and he was complaining about the time and difficulty of building a rehearsal schedule, where constraints included
- actors available at different times
- rehearsal slots of different length
- scenes of different length
- scenes that contained varying combinations of actors
- a hard constraint that each scene be rehearsed at least once before the play opened (I think he actually said at least twice, ideally three times)
- a soft constraint that scenes would ideally be rehearsed in order
- the reluctant possibility that if it were impossible to have all the actors for a scene together for long enough, you could provide stand-ins to allow others to rehearse

As an over-ambitious junior programing student, I said I could build something that would do that! Unfortunately (although perhaps unsurprisingly), it turned out to be quite a bit more difficult than I had imagined, and I never got around to it. Last year, I started looking into the problem again and I did build a much simplified version for an unrelated scenario (given x conference attendees who have expressed preferences from 1-y for each of y conference workshops held over z timeslots, which attendees should go to which workshops?) but the Automated Theater Rehearsal Planner was still beyond me. In late January, as I poked through the list of uncompleted projects, I once again came across this one and decided to look seriously into it - as a professional programmer, surely this is something I can do! And in a serendipitous moment, while looking up resources on scheduling algorithms I discovered that this course had begun a few days earlier, so I signed up for it and this is now an active project that I hope to get into a useful state this year.

Some background info on the problem for reference

To understand the general domain of scheduling, I started at the broadest relevant page, http://en.wikipedia.org/wiki/Automated_planning_and_scheduling, and after looking through it selected these relevant links
- http://en.wikipedia.org/wiki/Preference-based_planning, which is relevant because of the soft constraints listed, such as preferring to rehearse in order and have all actors in a scene together
- http://en.wikipedia.org/wiki/Scheduling_(computing) (if one considers the director or the rehearsal space to be the resource being scheduled) which led to http://en.wikipedia.org/wiki/Job_Shop_Scheduling and eventually on to http://en.wikipedia.org/wiki/Genetic_algorithm_scheduling and finally to http://en.wikipedia.org/wiki/Scheduling_(computing)#Scheduling_optimization_problems and to this flow shop scheduling demonstration app: http://posh-wolf.herokuapp.com/ (although flow-shop appears more restricted than my problem). 


Since it seemed likely that at least a broadly similar tool existed, I searched for free tools to try out in order to clarify what I needed to build exactly. (Or if perhaps it already existed). When searching for basic 'scheduling tools' or 'resource planning', the results tended to be about processes, such as scheduling a factory line and supplier logistics for organising delivery schedules - it may turn out that this is the same problem, but they don't quite seem to match. When I searched for 'rehearsals' (or similar), there were no specifically relevant results, however I found that there is a ton of software available for employee scheduling. Since it seemed that this should have significant overlap with the problem I'm looking at, it seemed worth installing a couple of the freeware options to check them out.
http://www.kappix.com/download.htm
http://www.simulation.co.uk/

I found that the basic issue with these tools is that employee scheduling operates on the assumption that you have a fixed number of slots to fill for each session, and are moving the people into the slots (and in the more complex tools, matching the people to the required tasks for each slot). For the rehearsals, I have a variable number of slots, based on how many people can attend, and I want to see which scenes (tasks) are relevant to the largest number of these people. 

Now, for the Creative Challenge project in the class: I'm going to get my scheduler app into a user-friendly state and post it to the class.

File locations on azure

Spent the last few days getting everything working in azure with correct folder permissions and so on, leaning heavily on stack overflow for hints. First I found that the templates didn't work in azure because I hardcoded the template locations to c:\users\jafitzge…, had to figure out how to put a location that would work in both the dev and production environments

• http://stackoverflow.com/questions/3038459/django-template-path
• http://stackoverflow.com/questions/6098275/django-problem-with-template-paths

Then I got permissions errors trying to write my result files and debug files

[Errno 2] No such file or directory: 'SampleData2.csv'

[Errno 13] Permission denied: 'scheduledebug10.log'

I tried looking up azure debugging options to see if there was a standard location for log files, etc. http://msdn.microsoft.com/en-us/magazine/ff714589.aspx is only about .net logging options apparently?

http://social.msdn.microsoft.com/Forums/en-US/windowsazuredevelopment/thread/7733a3d8-e80f-4ac9-b285-bf9c652896fd is promising, how do I write to a file in python on azure? Answer leads to

http://msdn.microsoft.com/en-us/library/ee758708.aspx which talks about using the Azure Managed Library to access Local Resources.

http://stackoverflow.com/questions/15146545/ioeerror-in-python-no-such-file-or-directory suggested I needed to change from writing to the current working directory, and instead write to the module directory.  Locally the file is at C:\Users\...\SimplySchedule which is PROJECT_PATH.

Finally fixed all the file permission issues by using the correct folders, and extracted the file path calculations to a couple functions so I could be sure I was doing it the same way everywhere.

Now working on serving the saved schedule file as a download, which seems like should be done using the MEDIA settings. Checking out the django docs again at https://docs.djangoproject.com/en/dev/topics/files/ means I should change the file save location from

import os.path

BASE = os.path.dirname(os.path.abspath(__file__))

To

>>> from django.core.files.storage import default_storage
>>> from django.core.files.base import ContentFile

>>> path = default_storage.save('/path/to/file', ContentFile('new content'))

From <https://docs.djangoproject.com/en/dev/topics/files/>

This stuff looks relevant: http://stackoverflow.com/questions/883313/django-excel-xlwt/883351#883351

This implies I should be saving my debug files to STATIC, not media - that makes sense! http://stackoverflow.com/questions/6619464/django-serving-from-media-but-not-static?rq=1

Also looks interesting for when I create an actual model for all the data I'm throwing around: saving an uploaded file as a model: https://docs.djangoproject.com/en/dev/topics/http/file-uploads/

Now I tried creating public facing 404 and 500 templates and setting Debug=False but it didn't seem to have taken hold when I deployed, although it was working as expected locally.  However when I went back to look at it again a couple hours later it was working as expected. Possibly just a caching issue. 

So, current state - I have all the functionality requested available,  although the site itself looks ugly as hell and super amateur. Still waiting for dad to go check it out and give some feedback, and until then it's time to get some PAX Aus app preparation done.

Success: Django hello world on azure!

Something made me think of verifying all the django packages in my github repository, as well as my local repository, and I realised that the whole sql folder was missing. I tried to add the files in it manually and got ' did not match any files' - so obviously there was something wrong with my .gitignore file…I looked through it for everything python related, and then noticed a 'sql' entry. Doh, it was ignoring the whole folder. Removed that line, commit, deploy, and voila!

Now to figure out manage.py…

So, found a tutorial.

Step 1: run manage.py runserver

Result: error "ImportError: No module named 'django'

Googling implies a path error : http://stackoverflow.com/questions/5428704/manage-py-runserver-importerror-no-module-named-mysite

Fixed!

And now, found an introduction to django - a whole book, apparently. Which seems good as I have no idea how to get my existing app running through django. http://www.djangobook.com/en/2.0/chapter02.html

Was good that they included the test 'import django' instructions, as it turned out I didn't have any of it in my PYTHONPATH - it was all just in my path.

I'll ignore the database section for now, because I don't actually plan to use a database for this at all.  I've already done the create project steps while setting up the site with azure. So I'm straight through to chapter 3: setting up webpages!

Creating views.py looks a little familiar from when I did the first half of that google app engine course - I guess I really should have finished it :)

Pleasantly surprised by the 'django automatically detects code changes' note, I was expecting to have to restart the server :)

Doh, and an import error. Foolishly copying and pasting, of course I don't have a module called mysite.views, I gave it a much more descriptive name! And - got the hello world page going.

Ooh, I bet this will come in handy at some point: 

At any point in your view, temporarily insert an assert False to trigger the error page.

From <http://www.djangobook.com/en/2.0/chapter03.html>

So, half past twelve and end of a chapter - time to commit, confirm it all works on azure as well as my box, and go practice hindi.  And it all works exactly the same! A successful evening.

New project: python+django+azure

Dad emailed me a few weeks ago asking what's the best way to assign 200 conference attendees to workshops, where there are 10 workshops each running 3 times at the conference (all at the same time). So I said I'd write an app that would take a csv file of (attendee, pref, ... prefN) and spit out a csv file of attendee, slot1, slot2, slot3). Easy, I got a naive version up on github a few days later.


Then I realised that it would be kind of a nightmare packaging this app to send to dad for him to try and run and then me remotely debugging for him, and getting him to use an updated version. So the obvious solution is to put it online and have him access it as a web app. This has the side benefit of getting me to actually try building a real website again.

Since I've been meaning to try setting stuff up on azure forever, and the app is arleady in python, I chose a python site on azure. Since I don't know what I'm doing I looked for a tutorial, and the first one suggested I use Django. I'm not actually quite clear on what Django will handle and what I could do without it, but for now I'll go with it. Following the tutorial at
http://www.windowsazure.com/en-us/develop/python/tutorials/web-sites-with-django/ was super straightforward, although since I already had a github account set up I didn't even need to do the deploy with git, when I chose github as my source control it started just auto deploying the latest copy of master. Unfortunately at the end, instead of their nice example page, I had a server error :/

Possibly it's because I renamed my app SimplySchedule instead of the suggested DjangoApplication - I may have missed a place that used the original name? Seems odd though.

Soo….an early opportunity to learn how to check out the logs and troubleshoot! Starting here: http://www.windowsazure.com/en-us/develop/net/best-practices/troubleshooting-web-sites/
So I set a new deployment credential to use with ftp and git - fails the first time when I try and get the ftp site, you need to use the domain as well. The logs don't seem that helpful:



Aha - having turned on detailed logging, I got this info

D:\Python27\python.exe - The FastCGI process exceeded configured request timeout

Module	FastCgiModule
Notification	ExecuteRequestHandler
Handler	FastCGI_SimplySchedule_ccdfe77c-40fe-4aec-84e4-bb292a25a799
Error Code	0x80070102

Requested URL	http://simplyschedule.azurewebsites.net:80/
Physical Path	C:\DWASFiles\Sites\SimplySchedule\VirtualDirectory0\site\wwwroot
Logon Method	Anonymous
Logon User	Anonymous

So, it is at least getting to the site and running the fastcgi handler is setup…

I went and doublechecked all the config options, and restarted the site. When I tried visiting it again I got this:

ImportError at /

cannot import name sql

Request Method:	GET
Request URL:	http://simplyschedule.azurewebsites.net/
Django Version:	1.4
Exception Type:	ImportError
Exception Value:	cannot import name sql
Exception Location:	D:\home\site\wwwroot\site-packages\django\db\models\deletion.py in <module>, line 5
Python Executable:	D:\Python27\python.exe
Python Version:	2.7.3
Python Path:	['D:\\Python27\\Scripts',  'D:\\home\\site\\wwwroot\\SimplySchedule',  'D:\\home\\site\\wwwroot\\site-packages',  'D:\\Windows\\SYSTEM32\\python27.zip',  'D:\\Python27\\DLLs',  'D:\\Python27\\lib',  'D:\\Python27\\lib\\plat-win',  'D:\\Python27\\lib\\lib-tk',  'D:\\Python27',  'D:\\Python27\\lib\\site-packages']
Server time:	Tue, 21 May 2013 06:19:54 +0000

Environment:

Request Method: GET

Request URL: http://simplyschedule.azurewebsites.net/

Django Version: 1.4

Python Version: 2.7.3

Installed Applications:

('django.contrib.auth',

 'django.contrib.contenttypes',

 'django.contrib.sessions',a

 'django.contrib.sites',

 'django.contrib.messages',

 'django.contrib.staticfiles')

Installed Middleware:

('django.middleware.common.CommonMiddleware',

 'django.contrib.sessions.middleware.SessionMiddleware',

 'django.middleware.csrf.CsrfViewMiddleware',

 'django.contrib.auth.middleware.AuthenticationMiddleware',

 'django.contrib.messages.middleware.MessageMiddleware')

Traceback:

File "D:\home\site\wwwroot\site-packages\django\core\handlers\base.py" in get_response

  89.                     response = middleware_method(request)

File "D:\home\site\wwwroot\site-packages\django\contrib\sessions\middleware.py" in process_request

  10.         engine = import_module(settings.SESSION_ENGINE)

File "D:\home\site\wwwroot\site-packages\django\utils\importlib.py" in import_module

  35.     __import__(name)

File "D:\home\site\wwwroot\site-packages\django\contrib\sessions\backends\db.py" in <module>

  77. from django.contrib.sessions.models import Session

File "D:\home\site\wwwroot\site-packages\django\contrib\sessions\models.py" in <module>

  1. from django.db import models

File "D:\home\site\wwwroot\site-packages\django\db\models\__init__.py" in <module>

  5. from django.db.models.query import Q

File "D:\home\site\wwwroot\site-packages\django\db\models\query.py" in <module>

  13. from django.db.models.deletion import Collector

File "D:\home\site\wwwroot\site-packages\django\db\models\deletion.py" in <module>

  5. from django.db.models import signals, sql

Exception Type: ImportError at /

Exception Value: cannot import name sql

Welp, after a little blind web searching, I can't find anyone else hitting this error - I think. There is a mention of someone else who went through the tutorial and saw a 500 internal server error in a comment on this Stack Overflow question,: http://stackoverflow.com/questions/10990706/azure-free-website-with-python-django-not-running/15757329#15757329
But a 500 error could be lots of things, as shown by this question: http://stackoverflow.com/questions/12554725/azure-sdk-django-visual-studio-2012-publish-to-azure-succeeds-but-i-get-5?rq=1

So, I asked my question at the disqus comments on the tutorial - will wait a day for a response before going to Stack Overflow I guess