Tuesday, December 17, 2013

sharing app data for the live tile

So, the recommended way to share data between the app and the periodic task is by files in the isolated storage - perfect, that was my plan!

BUT, my periodic task project doesn't by default have access to the classes defined in the app - which will make it tricky to read and parse the schedule xml :/ And I can't make it reference that project, because it 'is not supported by a Background Agent'.

Clearly, I need to extract shared functionality to a library of some supported type, and reference it from both projects. I'll need
 - event object definition
 - schedule parser

Basically this is all the bits contained in my Model definition, which makes me feel like I must have structured it in a reasonable way :) Unfortunately, once I moved them, I found that I had a lot of View-related code in there, calling App.Resources, etc, which wouldn't work in a shared library :/ Well, I guess the cleanup required here will make the code better altogether, so it's worth doing!

I used this post as an example to get me started: 

Specifically, I have an SQLite db which is instantiated in the App.xaml.cs class, which I can't refer to from my library - and is probably the wrong place anyway. But, the DB seems to use the Assembly name to access it: and isn't the assembly name going to be different between the main app and the background task? There's not much reference on the web, but a couple of people say that it isn't possible to use sqlite in a background task. I was worried that I would have to update my model to use the db for the full list, and back to the flat file for the individual schedule, which would be a shame - but I decided to go ahead with the refactoring and try it - and everything ran! It didn't work, quite - it always said 0 events found - but it managed to instantiate the db connection and run through all the code without panicking, so must be most of the way there.

Unfortunately, I found that the database had starting renaming all events to "1.0" when they were stored. I figured out that this was because I had added a new field to the Event object, which was messing up the Insert statement. Once that was solved, I started hitting exceptions when the scheduled task tried to read the db - I figured that was because the db file was locked, because it was never being closed by the main process. I cleaned up the DBHelper file a little, turning it into a singleton and making sure that every operation which called Open() also called Close(). At this point, I decided should probably create an Interface for the DBHelper, which would make testing easier and also make migration easier if I switch away from SQLite at some point.  I recently started actually trying to use some of Visual Studio's refactoring abilities, so here I did right-click on the class name -> Refactor -> Extract Interface. Then select which of the public methods I wanted in the Interface, and Done! Pretty nice.

Unfortunately, when I ran the app again, the database locking was even worse - even the original app process could never open it, always getting the same error. I decided that I did need a real mutex. I found the outlines in another stack overflow answer and added a check/hold/release mutext everywhere I had the open/close db work. Running with this made it clear that actually, it wasn't a deadlock - even when I turned off the background agent, I was getting 'unable to open database' errors every time. Clearly, the actual problem had been introduced when I updated the DB schema. I could validate with the Isolated Storage Explorer that it was being copied into storage correctly. I stepped through the code and verified that the db was being opened successfully on first contact - but then when it attempted to execute a query, the code tried to open a dbname of "Events.sqlite-journal", which threw a storage exception because it didn't exist.

I followed advice (from Stack Overflow, obvs) to create the journal file back at the start with the actual db file - but still got an error when trying to open it. To narrow down the problem, I tried opening the file straight after creating it - and still got the error. I think I should have been able to figure this out from there, but instead I went and searched to get a list of basic causes for this error - and the very first answer was 'you haven't closed the stream that was returned when you created the file'. Duh, I mean, really. Added a file.Close(), ran the app, everything works! Well, except a Null Reference Exception when I tried to add an event to my schedule, but lots of things worked. Some, at least. More than earlier today.

(this post was written as-it-happened, between december 17 and 26. Very useful for tracking what I had done when I was only doing random half-hours of work!)

Thursday, December 12, 2013

generating a live tile

I'm working on adding a live tile to my PAX app, to show the next item in the schedule. Finally got 'a' live tile working, with a couple headaches along the way.

1. Because I still have a 7.0 project, the nice simple API for 'ActiveTiles.FirstOrDefault' doesn't exist, so you have to use an Enumerator on the ActiveTiles set, and apparently I am still hazy on Enumerators so forgot to get next before trying to use the current tile.

2. I'm really making life more difficult for myself by continuing to target 7.0 devices, but I don't want to give up on them yet. But I do really want to offer a wide tile option to those who can use it! Fortunately Telerik offers a LiveTileHelper in their tools (which I already use and pay for) which handles the 'does this device support x' aspect of it, I haven't incorporated that yet but will definitely use it. I should probably read through all their docs to see what else there is, because I only found out about this by someone mentioning it in a stack overflow response about tiles.

3. Because I want the front of the tile to display a bunch of text, I'm going to have to generate a bitmap containing that text. I found a few examples on stack overflow , but somehow when I ran my app the generated image was not applied. I checked the tile docs and they mentioned that setting the BackgroundImage property to a null Uri would just be ignored, but I used the Isolated Storage Explorer to verify that the image was created and in the file system exactly where I expected. After going back to doublecheck anything I'd missed in the sample code, I noticed that their code looked like
BackgroundImage = new Uri("isostore:/Shared/ShellContent/Image.jpg", UriKind.Absolute), 
whereas I had 
data.BackgroundImage = new Uri("/Images/Tiles/paxTile.png", UriKind.Relative); 
data.BackBackgroundImage = new Uri("/Images/Melbourne/melbTile.png", UriKind.Relative);
curiously enough the background image worked, but not the front image. I changed the front image to be in the format isostore:file, Absolute, and it worked! Still not sure why the requirements are different for the front and back - I left a comment on one of the MSDN pages for setting Live Tiles asking about this (and for a note to be added to the docs somewhere).

Now I just have to design the tile layout and actually make it look decent. Here's the plan so far:

            // __________________
            //                               (x)      // <- if you have more than one session on your schedule for the
             //                                        // next hour, this is how many. The tile just shows details on one.
            // my great session...            // max 14 chars? check font size
            //           rest of name?
            // time                                   // in users culture
            // __PAX Digital Assistant__

            // BACK
            // My session name is...
            //    12:30: Unicorn
            // Come play with my f...
            //    12.30: Cat.
            // Leave Me Alone Wit...
            //    13:00: Main Theater
            // _______________________

            // layout wide:
            // My session name is the funniest thing you eve...
            //    12:30: Unicorn
            // Come play with my friends while I make fun of y...
            //    12.30: Cat.
            // Leave Me Alone With My Collection of Pokemon...
            //    13:00: Main Theater

Wednesday, November 27, 2013

Cannot access x.y.z.dll because it is being used by another process

Spent a while today trying to figure out why I couldn't run an app, which kept giving me the build error that a dll was being used by another process. I found this super nice tip which can help avoid a bunch of procmon/etc fiddling:

1. open a command line
2. go to the directory containing your locked dll
3. >tasklist /m my.locked.dll

via http://edwin.vriethoff.net/2009/01/26/find-out-which-process-is-locking-a-dll-without-extra-tools/

Monday, November 18, 2013

[TestMethod] public static void IAmInvisible()

Just spent a little time debugging some MSTests I was trying to add to a project, which just weren't showing up in the Test Explorer. Most of the 'help these tests don't show up' info out there is because someone is trying to add a new project, but in this case I was adding a new class to an existing project.

Eventually I tried adding a new set of tests identical to some working tests, and editing them bit by bit to look like my failing ones. Perhaps it should have been more obvious, but the offending difference was that I had marked the new tests as Static - that doesn't seem to be allowed, and I can't even find that restriction mentioned anywhere. I guess most people never try it?

Tuesday, October 15, 2013

Phone devs love this one weird trick.

Just came across a really sweet fiddler trick to make life a little simpler for windows phone development: set up an autoresponder rule that serves the FiddlerRoot certificate from a nice short url, to save a little effort in the process of re-installing it every time you restart the emulator.


Saturday, June 8, 2013

File locations on azure

Spent the last few days getting everything working in azure with correct folder permissions and so on, leaning heavily on stack overflow for hints. First I found that the templates didn't work in azure because I hardcoded the template locations to c:\users\jafitzge…, had to figure out how to put a location that would work in both the dev and production environments

Then I got permissions errors trying to write my result files and debug files
[Errno 2] No such file or directory: 'SampleData2.csv'
[Errno 13] Permission denied: 'scheduledebug10.log'

I tried looking up azure debugging options to see if there was a standard location for log files, etc. http://msdn.microsoft.com/en-us/magazine/ff714589.aspx is only about .net logging options apparently?
http://social.msdn.microsoft.com/Forums/en-US/windowsazuredevelopment/thread/7733a3d8-e80f-4ac9-b285-bf9c652896fd is promising, how do I write to a file in python on azure? Answer leads to
http://msdn.microsoft.com/en-us/library/ee758708.aspx which talks about using the Azure Managed Library to access Local Resources.

http://stackoverflow.com/questions/15146545/ioeerror-in-python-no-such-file-or-directory suggested I needed to change from writing to the current working directory, and instead write to the module directory.  Locally the file is at C:\Users\...\SimplySchedule which is PROJECT_PATH.

Finally fixed all the file permission issues by using the correct folders, and extracted the file path calculations to a couple functions so I could be sure I was doing it the same way everywhere.

Now working on serving the saved schedule file as a download, which seems like should be done using the MEDIA settings. Checking out the django docs again at https://docs.djangoproject.com/en/dev/topics/files/ means I should change the file save location from

import os.path
BASE = os.path.dirname(os.path.abspath(__file__))

>>> from django.core.files.storage import default_storage
>>> from django.core.files.base import ContentFile
>>> path = default_storage.save('/path/to/file', ContentFile('new content'))

This implies I should be saving my debug files to STATIC, not media - that makes sense! http://stackoverflow.com/questions/6619464/django-serving-from-media-but-not-static?rq=1

Also looks interesting for when I create an actual model for all the data I'm throwing around: saving an uploaded file as a model: https://docs.djangoproject.com/en/dev/topics/http/file-uploads/

Now I tried creating public facing 404 and 500 templates and setting Debug=False but it didn't seem to have taken hold when I deployed, although it was working as expected locally.  However when I went back to look at it again a couple hours later it was working as expected. Possibly just a caching issue. 

So, current state - I have all the functionality requested available,  although the site itself looks ugly as hell and super amateur. Still waiting for dad to go check it out and give some feedback, and until then it's time to get some PAX Aus app preparation done.

Friday, May 24, 2013

Django and sql azure, wtf?

Chapter 5 of the Django Book - Models

I thought about just skipping this one as I don't anticipate needing a db for the schedule app, but decided I might as well follow through, it'll probably come in handy soon enough.

So step 1 I created a sql database in azure to use, and now I need a sql engine for python.

I found https://pypi.python.org/pypi/django-pyodbc-azure/1.0.5 which told me to install pyodbc first, which was googleable enough, and available for both 2.7 and 3.x. But the 2.7 version said it couldn't install bc I don't have 2.7 in the registry…hmm.  After some hints from Stack Overflow, I checked the python version I have for 2.7 and it's the 32 bit version, which won't be found by the 64 bit installer. Ran the 32 bit installer and done. Then it said to use pip to install itself, which I didn't have, but SO led me to this great page: http://www.lfd.uci.edu/~gohlke/pythonlibs/#pip…but after running that, pip still wasn't recognised on the command line. Another useful note on SO said it installed to python/scripts instead of just to /python, so I had to add that to my path. And now I get 'failed to create process'. Tried in admin mode, same error. Google was initially not forthcoming about this error, but then turned up this: http://www.maphew.com/Python/python-fixit-snippets/ which suggested that it was unable to find python. Perhaps it is confused by my python27 option? The location certainly hasn't changed since I installed it. I uninstalled and reinstalled to be sure, but that didn't help.

Checking the Event Viewer to see where it's trying to find python might help -nope, can't see any relevant messages.

I went back and installed Distribute, as it is mentioned in the same place as where I got pip. No change.

I found this gui for pip that lets you switch between active python installers - that might help, because it could still be being caused by my dual 2 and 3 installation? https://sites.google.com/site/pydatalog/python/pip-for-windows but it also failed. But it did point out that I have no pip.exe in c:\python27 - which is odd because I do have a Removepip.exe? Turns out pip.exe is in c:\python27\scripts, but moving it just got me back to failed to create process. Hmmmmmm.

So I went back to the site, and noticed it had a downloadable zip. So I downloaded that, and ran setup.py install, and it appears to have installed something? But not a full package. In fact I can't even see a real pyodbc package...

But just to make sure it isn't installed, let's try following the instructions on the pyodbc-azure site. And nope, I get jango.core.exceptions.ImproperlyConfigured: 'sql_server.pyodbc' isn't an available database backend.
So, that was a couple of unproductive hours, and I give up. I will instead attempt to follow this tutorial (http://www.windowsazure.com/en-us/develop/python/tutorials/web-app-with-blob-storage/) for connecting a python app to azure storage, which should be perfectly adequate for anything.  And to get through the Django book I can just go ahead and use sqllite.

Getting started with sqllite: http://sqlite.org/sqlite.html
Quickly created and connected to a database, created a model, turned it into tables, added data to the tables, and started querying the data out of it
  • All rows
  • Filter: Field = x, field contains x (see Appendix C for field == x, or < x?)
  • Get(): A single object instead of a list ( filter returns a list) uses same options as filter
  • Order_by
  • Slicing - works just like a python list!
  • Use .update(x=1) instead of pub.name='a' to get more efficient sql translations (setting a field directly will cause all fields to be saved over)
  • .delete() can be called on an object or a query set (result of .filter()) Django tries to save you from accidentally deleting everyhting by requiring the odd syntax x.objects.all().delete() to do that

Thursday, May 23, 2013

It really worked!

Chapter 4 of the Django book got off to a poor start when I followed their instructions to run 'python manage.py shell', and unexpectedly got a syntax error from one of the core django files. It seemed unlikely there was an actual issue with the file (didn't I even run it myself yesterday?), and after googling the error one of the results mentioned 'of course, I have python 3' so I remembered that I have python 2 and 3 installed, but the django installation came with python 2. I updated my path and the python 2 executable so I could call 'python2' to get python 2 running, leaving my default of 3 still there, and running under python 2 succeeded.

However this does mean I'll have to check my app, which was written in 3, for compatibility with 2 - at the very least all those print statements will be wrong :/ It also means that I can't run my python programs in Sublime Text because it is in Python 3.
Whoa. Actually the only change I had to make was remove the 'newline' option from csv.reader. Way easier than expected!

Useful introductions in the first half of this chapter: templates and contexts, template variables, accessing lists, dicts, and methods through dots, basic logic tags available in templates, filters, comments. Further details can be found in the appendix, which they recommend you read so as to familiarise yourself with all the available options.

And now part 2 - using a template to create a view.

Step 1 - put the full path to your templates files in the settings file. That's easy enough locally, but I'll have to look up what it will be on azure. For now I guess I can just use os.path of the settings file (prompted by their example) but that might not be where I want it to live.

Step 2 - update the views to call templates. Easy enough.

Step 3 - template inheritance. Awesome!

I followed along with code as they went, creating my own side example of attendee schedules to make sure I was getting it - everything worked as expected, it felt really simple but quite powerful. I committed it all to Azure and that worked straight off as well.

Wednesday, May 22, 2013

Success: Django hello world on azure!

Something made me think of verifying all the django packages in my github repository, as well as my local repository, and I realised that the whole sql folder was missing. I tried to add the files in it manually and got ' did not match any files' - so obviously there was something wrong with my .gitignore file…I looked through it for everything python related, and then noticed a 'sql' entry. Doh, it was ignoring the whole folder. Removed that line, commit, deploy, and voila!

Now to figure out manage.py…

So, found a tutorial.
Step 1: run manage.py runserver
Result: error "ImportError: No module named 'django'

And now, found an introduction to django - a whole book, apparently. Which seems good as I have no idea how to get my existing app running through django. http://www.djangobook.com/en/2.0/chapter02.html

Was good that they included the test 'import django' instructions, as it turned out I didn't have any of it in my PYTHONPATH - it was all just in my path.

I'll ignore the database section for now, because I don't actually plan to use a database for this at all.  I've already done the create project steps while setting up the site with azure. So I'm straight through to chapter 3: setting up webpages!

Creating views.py looks a little familiar from when I did the first half of that google app engine course - I guess I really should have finished it :)

Pleasantly surprised by the 'django automatically detects code changes' note, I was expecting to have to restart the server :)

Doh, and an import error. Foolishly copying and pasting, of course I don't have a module called mysite.views, I gave it a much more descriptive name! And - got the hello world page going.

Ooh, I bet this will come in handy at some point: 
At any point in your view, temporarily insert an assert False to trigger the error page.

So, half past twelve and end of a chapter - time to commit, confirm it all works on azure as well as my box, and go practice hindi.  And it all works exactly the same! A successful evening.

Tuesday, May 21, 2013

New project: python+django+azure

Dad emailed me a few weeks ago asking what's the best way to assign 200 conference attendees to workshops, where there are 10 workshops each running 3 times at the conference (all at the same time). So I said I'd write an app that would take a csv file of (attendee, pref, ... prefN) and spit out a csv file of attendee, slot1, slot2, slot3). Easy, I got a naive version up on github a few days later.

Then I realised that it would be kind of a nightmare packaging this app to send to dad for him to try and run and then me remotely debugging for him, and getting him to use an updated version. So the obvious solution is to put it online and have him access it as a web app. This has the side benefit of getting me to actually try building a real website again.

Since I've been meaning to try setting stuff up on azure forever, and the app is arleady in python, I chose a python site on azure. Since I don't know what I'm doing I looked for a tutorial, and the first one suggested I use Django. I'm not actually quite clear on what Django will handle and what I could do without it, but for now I'll go with it. Following the tutorial at
http://www.windowsazure.com/en-us/develop/python/tutorials/web-sites-with-django/ was super straightforward, although since I already had a github account set up I didn't even need to do the deploy with git, when I chose github as my source control it started just auto deploying the latest copy of master. Unfortunately at the end, instead of their nice example page, I had a server error :/

Possibly it's because I renamed my app SimplySchedule instead of the suggested DjangoApplication - I may have missed a place that used the original name? Seems odd though.

Soo….an early opportunity to learn how to check out the logs and troubleshoot! Starting here: http://www.windowsazure.com/en-us/develop/net/best-practices/troubleshooting-web-sites/
So I set a new deployment credential to use with ftp and git - fails the first time when I try and get the ftp site, you need to use the domain as well. The logs don't seem that helpful:

Aha - having turned on detailed logging, I got this info
D:\Python27\python.exe - The FastCGI process exceeded configured request timeout
Error Code
Requested URL
Physical Path
Logon Method
Logon User

So, it is at least getting to the site and running the fastcgi handler is setup…

I went and doublechecked all the config options, and restarted the site. When I tried visiting it again I got this:

ImportError at /
cannot import name sql
Request Method:
Request URL:
Django Version:
Exception Type:
Exception Value:
cannot import name sql
Exception Location:
D:\home\site\wwwroot\site-packages\django\db\models\deletion.py in <module>, line 5
Python Executable:
Python Version:
Python Path:
Server time:
Tue, 21 May 2013 06:19:54 +0000


Request Method: GET

Django Version: 1.4
Python Version: 2.7.3
Installed Applications:
Installed Middleware:

File "D:\home\site\wwwroot\site-packages\django\core\handlers\base.py" in get_response
  89.                     response = middleware_method(request)
File "D:\home\site\wwwroot\site-packages\django\contrib\sessions\middleware.py" in process_request
  10.         engine = import_module(settings.SESSION_ENGINE)
File "D:\home\site\wwwroot\site-packages\django\utils\importlib.py" in import_module
  35.     __import__(name)
File "D:\home\site\wwwroot\site-packages\django\contrib\sessions\backends\db.py" in <module>
  77. from django.contrib.sessions.models import Session
File "D:\home\site\wwwroot\site-packages\django\contrib\sessions\models.py" in <module>
  1. from django.db import models
File "D:\home\site\wwwroot\site-packages\django\db\models\__init__.py" in <module>
  5. from django.db.models.query import Q
File "D:\home\site\wwwroot\site-packages\django\db\models\query.py" in <module>
  13. from django.db.models.deletion import Collector
File "D:\home\site\wwwroot\site-packages\django\db\models\deletion.py" in <module>
  5. from django.db.models import signals, sql

Exception Type: ImportError at /
Exception Value: cannot import name sql

Welp, after a little blind web searching, I can't find anyone else hitting this error - I think. There is a mention of someone else who went through the tutorial and saw a 500 internal server error in a comment on this Stack Overflow question,: http://stackoverflow.com/questions/10990706/azure-free-website-with-python-django-not-running/15757329#15757329
But a 500 error could be lots of things, as shown by this question: http://stackoverflow.com/questions/12554725/azure-sdk-django-visual-studio-2012-publish-to-azure-succeeds-but-i-get-5?rq=1

So, I asked my question at the disqus comments on the tutorial - will wait a day for a response before going to Stack Overflow I guess