SQLite woes, cont.

At the end of my last post, I could launch the app and view events, but then got a Null Reference Exception while saving the events into the db. Oddly enough, debugging through the code showed that I was getting a SQLite error code 11 on *every second event*, and then getting a Null Reference Exception after a set of events.The call stack showed:

{System.NullReferenceException: NullReferenceException
   at Community.CsharpSqlite.Sqlite3.sqlite3BtreeClose(Btree& p)
   at Community.CsharpSqlite.Sqlite3.sqlite3_close(sqlite3 db)
   at SQLiteClient.SQLiteConnection.Dispose()
   at PAX7.Utilicode.DBHelper.Close()
   at PAX7.Utilicode.DBHelper.Finalize()}

perhaps I'm calling close twice? I added a breakpoint in the DBHelper.Close() method to check this. I also wrapped the code inside the method in a try-catch-throw to Little Watson block to avoid crashing the app with this exception.

{System.NullReferenceException: NullReferenceException
   at Community.CsharpSqlite.Sqlite3.sqlite3FindIndex(sqlite3 db, String zName, String zDb)
   at Community.CsharpSqlite.Sqlite3.sqlite3StartTable(Parse pParse, Token pName1, Token pName2, Int32 isTemp, Int32 isView, Int32 isVirtual, Int32 noErr)
   at Community.CsharpSqlite.Sqlite3.yy_reduce(yyParser yypParser, Int32 yyruleno)
   at Community.CsharpSqlite.Sqlite3.sqlite3Parser(yyParser yyp, Int32 yymajor, Token yyminor, Parse pParse)
   at Community.CsharpSqlite.Sqlite3.sqlite3RunParser(Parse pParse, String zSql, String& pzErrMsg)
   at Community.CsharpSqlite.Sqlite3.sqlite3Prepare(sqlite3 db, String zSql, Int32 nBytes, Int32 saveSqlFlag, Vdbe pReprepare, Vdbe& ppStmt, String& pzTail)
   at Community.CsharpSqlite.Sqlite3.sqlite3LockAndPrepare(sqlite3 db, String zSql, Int32 nBytes, Int32 saveSqlFlag, Vdbe pOld, Vdbe& ppStmt, String& pzTail)
   at Community.CsharpSqlite.Sqlite3.sqlite3_prepare(sqlite3 db, String zSql, Int32 nBytes, Vdbe& ppStmt, String& pzTail)
   at Community.CsharpSqlite.Sqlite3.sqlite3_exec(sqlite3 db, String zSql, dxCallback xCallback, Object pArg, String& pzErrMsg)
   at Community.CsharpSqlite.Sqlite3.sqlite3InitCallback(Object pInit, Int64 argc, Object p2, Object NotUsed)
   at Community.CsharpSqlite.Sqlite3.sqlite3InitOne(sqlite3 db, Int32 iDb, String& pzErrMsg)
   at Community.CsharpSqlite.Sqlite3.sqlite3Init(sqlite3 db, String& pzErrMsg)
   at Community.CsharpSqlite.Sqlite3.sqlite3ReadSchema(Parse pParse)
   at Community.CsharpSqlite.Sqlite3.sqlite3LocateTable(Parse pParse, Int32 isView, String zName, String zDbase)
   at Community.CsharpSqlite.Sqlite3.sqlite3SrcListLookup(Parse pParse, SrcList pSrc)
   at Community.CsharpSqlite.Sqlite3.sqlite3Insert(Parse pParse, SrcList pTabList, ExprList pList, Select pSelect, IdList pColumn, Int32 onError)
   at Community.CsharpSqlite.Sqlite3.sqlite3Insert(Parse pParse, SrcList pTabList, ExprList pList, Int32 null_4, IdList pColumn, Int32 onError)
   at Community.CsharpSqlite.Sqlite3.yy_reduce(yyParser yypParser, Int32 yyruleno)
   at Community.CsharpSqlite.Sqlite3.sqlite3Parser(yyParser yyp, Int32 yymajor, Token yyminor, Parse pParse)
   at Community.CsharpSqlite.Sqlite3.sqlite3RunParser(Parse pParse, String zSql, String& pzErrMsg)
   at Community.CsharpSqlite.Sqlite3.sqlite3Prepare(sqlite3 db, String zSql, Int32 nBytes, Int32 saveSqlFlag, Vdbe pReprepare, Vdbe& ppStmt, String& pzTail)
   at Community.CsharpSqlite.Sqlite3.sqlite3LockAndPrepare(sqlite3 db, String zSql, Int32 nBytes, Int32 saveSqlFlag, Vdbe pOld, Vdbe& ppStmt, String& pzTail)
   at Community.CsharpSqlite.Sqlite3.sqlite3_prepare_v2(sqlite3 db, String zSql, Int32 nBytes, Vdbe& ppStmt, Int32 dummy)
   at SQLiteClient.SQLiteCommand.ExecuteNonQuery[T](T toInsert)
   at PAX7.Utilicode.DBHelper.Insert[T](T obj, String statement)
   at PAX7.Model.Schedule.sqlInsertEvent(Event newEvent)}


this one
{System.NullReferenceException: NullReferenceException
   at Community.CsharpSqlite.Sqlite3.DbHasProperty(sqlite3 D, Int32 I, UInt16 P)
   at Community.CsharpSqlite.Sqlite3.sqlite3Init(sqlite3 db, String& pzErrMsg)
   at Community.CsharpSqlite.Sqlite3.sqlite3ReadSchema(Parse pParse)
   at Community.CsharpSqlite.Sqlite3.sqlite3LocateTable(Parse pParse, Int32 isView, String zName, String zDbase)
   at Community.CsharpSqlite.Sqlite3.sqlite3SrcListLookup(Parse pParse, SrcList pSrc)
   at Community.CsharpSqlite.Sqlite3.sqlite3Insert(Parse pParse, SrcList pTabList, ExprList pList, Select pSelect, IdList pColumn, Int32 onError)
   at Community.CsharpSqlite.Sqlite3.sqlite3Insert(Parse pParse, SrcList pTabList, ExprList pList, Int32 null_4, IdList pColumn, Int32 onError)
   at Community.CsharpSqlite.Sqlite3.yy_reduce(yyParser yypParser, Int32 yyruleno)
   at Community.CsharpSqlite.Sqlite3.sqlite3Parser(yyParser yyp, Int32 yymajor, Token yyminor, Parse pParse)
   at Community.CsharpSqlite.Sqlite3.sqlite3RunParser(Parse pParse, String zSql, String& pzErrMsg)
   at Community.CsharpSqlite.Sqlite3.sqlite3Prepare(sqlite3 db, String zSql, Int32 nBytes, Int32 saveSqlFlag, Vdbe pReprepare, Vdbe& ppStmt, String& pzTail)
   at Community.CsharpSqlite.Sqlite3.sqlite3LockAndPrepare(sqlite3 db, String zSql, Int32 nBytes, Int32 saveSqlFlag, Vdbe pOld, Vdbe& ppStmt, String& pzTail)
   at Community.CsharpSqlite.Sqlite3.sqlite3_prepare_v2(sqlite3 db, String zSql, Int32 nBytes, Vdbe& ppStmt, Int32 dummy)
   at SQLiteClient.SQLiteCommand.ExecuteNonQuery[T](T toInsert)
   at PAX7.Utilicode.DBHelper.Insert[T](T obj, String statement)
   at PAX7.Model.Schedule.sqlInsertEvent(Event newEvent)}


{SQLiteClient.SQLiteException: Unknown error
   at SQLiteClient.SQLiteCommand.ExecuteNonQuery[T](T toInsert)
   at PAX7.Utilicode.DBHelper.Insert[T](T obj, String statement)}

{System.NullReferenceException: NullReferenceException
   at Community.CsharpSqlite.Sqlite3.ENC(sqlite3 db)
   at Community.CsharpSqlite.Sqlite3.sqlite3VdbeExec(Vdbe p)
   at Community.CsharpSqlite.Sqlite3.sqlite3Step(Vdbe p)
   at Community.CsharpSqlite.Sqlite3.sqlite3_step(Vdbe pStmt)
   at Community.CsharpSqlite.Sqlite3.sqlite3_exec(sqlite3 db, String zSql, dxCallback xCallback, Object pArg, String& pzErrMsg)
   at Community.CsharpSqlite.Sqlite3.sqlite3InitCallback(Object pInit, Int64 argc, Object p2, Object NotUsed)
   at Community.CsharpSqlite.Sqlite3.sqlite3InitOne(sqlite3 db, Int32 iDb, String& pzErrMsg)
   at Community.CsharpSqlite.Sqlite3.sqlite3Init(sqlite3 db, String& pzErrMsg)
   at Community.CsharpSqlite.Sqlite3.sqlite3ReadSchema(Parse pParse)
   at Community.CsharpSqlite.Sqlite3.sqlite3LocateTable(Parse pParse, Int32 isView, String zName, String zDbase)
   at Community.CsharpSqlite.Sqlite3.sqlite3SrcListLookup(Parse pParse, SrcList pSrc)
   at Community.CsharpSqlite.Sqlite3.sqlite3Insert(Parse pParse, SrcList pTabList, ExprList pList, Select pSelect, IdList pColumn, Int32 onError)
   at Community.CsharpSqlite.Sqlite3.sqlite3Insert(Parse pParse, SrcList pTabList, ExprList pList, Int32 null_4, IdList pColumn, Int32 onError)
   at Community.CsharpSqlite.Sqlite3.yy_reduce(yyParser yypParser, Int32 yyruleno)
   at Community.CsharpSqlite.Sqlite3.sqlite3Parser(yyParser yyp, Int32 yymajor, Token yyminor, Parse pParse)
   at Community.CsharpSqlite.Sqlite3.sqlite3RunParser(Parse pParse, String zSql, String& pzErrMsg)
   at Community.CsharpSqlite.Sqlite3.sqlite3Prepare(sqlite3 db, String zSql, Int32 nBytes, Int32 saveSqlFlag, Vdbe pReprepare, Vdbe& ppStmt, String& pzTail)
   at Community.CsharpSqlite.Sqlite3.sqlite3LockAndPrepare(sqlite3 db, String zSql, Int32 nBytes, Int32 saveSqlFlag, Vdbe pOld, Vdbe& ppStmt, String& pzTail)
   at Community.CsharpSqlite.Sqlite3.sqlite3_prepare_v2(sqlite3 db, String zSql, Int32 nBytes, Vdbe& ppStmt, Int32 dummy)
   at SQLiteClient.SQLiteCommand.ExecuteNonQuery[T](T toInsert)
   at PAX7.Utilicode.DBHelper.Insert[T](T obj, String statement)
   at PAX7.Model.Schedule.sqlInsertEvent(Event newEvent)}



and this, seriously?

{SQLiteClient.SQLiteException: SQL logic error or missing database
   at SQLiteClient.SQLiteCommand.ExecuteNonQuery[T](T toInsert)
   at PAX7.Utilicode.DBHelper.Insert[T](T obj, String statement)}

+        ex    {SQLiteClient.SQLiteException: no such table: events
   at SQLiteClient.SQLiteCommand.ExecuteNonQuery[T](T toInsert)
   at PAX7.Utilicode.DBHelper.Insert[T](T obj, String statement)}    System.Exception {SQLiteClient.SQLiteException}

+        $exception    {System.ObjectDisposedException: Cannot access a closed file.
   at System.IO.__Error.FileNotOpen()
   at System.IO.FileStream.get_Length()
   at System.IO.IsolatedStorage.IsolatedStorageFileStream.get_Length()
   at Community.CsharpSqlite.Sqlite3.sqlite3_fileSize(sqlite3_file id, Int32& pSize)
   at Community.CsharpSqlite.Sqlite3.sqlite3OsFileSize(sqlite3_file id, Int32& pSize)
   at Community.CsharpSqlite.Sqlite3.sqlite3PagerPagecount(Pager pPager, Int32& pnPage)
   at Community.CsharpSqlite.Sqlite3.hasHotJournal(Pager pPager, Int32& pExists)
   at Community.CsharpSqlite.Sqlite3.sqlite3PagerSharedLock(Pager pPager)
   at Community.CsharpSqlite.Sqlite3.lockBtree(BtShared pBt)
   at Community.CsharpSqlite.Sqlite3.sqlite3BtreeBeginTrans(Btree p, Int32 wrflag)
   at Community.CsharpSqlite.Sqlite3.schemaIsValid(Parse pParse)
   at Community.CsharpSqlite.Sqlite3.sqlite3Prepare(sqlite3 db, String zSql, Int32 nBytes, Int32 saveSqlFlag, Vdbe pReprepare, Vdbe& ppStmt, String& pzTail)
   at Community.CsharpSqlite.Sqlite3.sqlite3LockAndPrepare(sqlite3 db, String zSql, Int32 nBytes, Int32 saveSqlFlag, Vdbe pOld, Vdbe& ppStmt, String& pzTail)
   at Community.CsharpSqlite.Sqlite3.sqlite3_prepare_v2(sqlite3 db, String zSql, Int32 nBytes, Vdbe& ppStmt, Int32 dummy)
   at SQLiteClient.SQLiteCommand.ExecuteNonQuery[T](T toInsert)
   at PAX7.Utilicode.DBHelper.Insert[T](T obj, String statement)
   at PAX7.Model.Schedule.sqlInsertEvent(Event newEvent)}    System.Exception {System.ObjectDisposedException}


while events from all four days are visible on first load, only events on friday are saved back to the database (inspected with isolated storage explorer). It has correctly saved the star and reminder for the event I added to my schedule, however.

sharing app data for the live tile

So, the recommended way to share data between the app and the periodic task is by files in the isolated storage - perfect, that was my plan!

BUT, my periodic task project doesn't by default have access to the classes defined in the app - which will make it tricky to read and parse the schedule xml :/ And I can't make it reference that project, because it 'is not supported by a Background Agent'.

Clearly, I need to extract shared functionality to a library of some supported type, and reference it from both projects. I'll need
 - event object definition
 - schedule parser

Basically this is all the bits contained in my Model definition, which makes me feel like I must have structured it in a reasonable way :) Unfortunately, once I moved them, I found that I had a lot of View-related code in there, calling App.Resources, etc, which wouldn't work in a shared library :/ Well, I guess the cleanup required here will make the code better altogether, so it's worth doing!

I used this post as an example to get me started: 
http://www.31a2ba2a-b718-11dc-8314-0800200c9a66.com/2011/11/this-is-continuation-of-this-post.html

Specifically, I have an SQLite db which is instantiated in the App.xaml.cs class, which I can't refer to from my library - and is probably the wrong place anyway. But, the DB seems to use the Assembly name to access it: and isn't the assembly name going to be different between the main app and the background task? There's not much reference on the web, but a couple of people say that it isn't possible to use sqlite in a background task. I was worried that I would have to update my model to use the db for the full list, and back to the flat file for the individual schedule, which would be a shame - but I decided to go ahead with the refactoring and try it - and everything ran! It didn't work, quite - it always said 0 events found - but it managed to instantiate the db connection and run through all the code without panicking, so must be most of the way there.

Unfortunately, I found that the database had starting renaming all events to "1.0" when they were stored. I figured out that this was because I had added a new field to the Event object, which was messing up the Insert statement. Once that was solved, I started hitting exceptions when the scheduled task tried to read the db - I figured that was because the db file was locked, because it was never being closed by the main process. I cleaned up the DBHelper file a little, turning it into a singleton and making sure that every operation which called Open() also called Close(). At this point, I decided should probably create an Interface for the DBHelper, which would make testing easier and also make migration easier if I switch away from SQLite at some point.  I recently started actually trying to use some of Visual Studio's refactoring abilities, so here I did right-click on the class name -> Refactor -> Extract Interface. Then select which of the public methods I wanted in the Interface, and Done! Pretty nice.

Unfortunately, when I ran the app again, the database locking was even worse - even the original app process could never open it, always getting the same error. I decided that I did need a real mutex. I found the outlines in another stack overflow answer and added a check/hold/release mutext everywhere I had the open/close db work. Running with this made it clear that actually, it wasn't a deadlock - even when I turned off the background agent, I was getting 'unable to open database' errors every time. Clearly, the actual problem had been introduced when I updated the DB schema. I could validate with the Isolated Storage Explorer that it was being copied into storage correctly. I stepped through the code and verified that the db was being opened successfully on first contact - but then when it attempted to execute a query, the code tried to open a dbname of "Events.sqlite-journal", which threw a storage exception because it didn't exist.

I followed advice (from Stack Overflow, obvs) to create the journal file back at the start with the actual db file - but still got an error when trying to open it. To narrow down the problem, I tried opening the file straight after creating it - and still got the error. I think I should have been able to figure this out from there, but instead I went and searched to get a list of basic causes for this error - and the very first answer was 'you haven't closed the stream that was returned when you created the file'. Duh, I mean, really. Added a file.Close(), ran the app, everything works! Well, except a Null Reference Exception when I tried to add an event to my schedule, but lots of things worked. Some, at least. More than earlier today.

(this post was written as-it-happened, between december 17 and 26. Very useful for tracking what I had done when I was only doing random half-hours of work!)

generating a live tile

I'm working on adding a live tile to my PAX app, to show the next item in the schedule. Finally got 'a' live tile working, with a couple headaches along the way.

1. Because I still have a 7.0 project, the nice simple API for 'ActiveTiles.FirstOrDefault' doesn't exist, so you have to use an Enumerator on the ActiveTiles set, and apparently I am still hazy on Enumerators so forgot to get next before trying to use the current tile.

2. I'm really making life more difficult for myself by continuing to target 7.0 devices, but I don't want to give up on them yet. But I do really want to offer a wide tile option to those who can use it! Fortunately Telerik offers a LiveTileHelper in their tools (which I already use and pay for) which handles the 'does this device support x' aspect of it, I haven't incorporated that yet but will definitely use it. I should probably read through all their docs to see what else there is, because I only found out about this by someone mentioning it in a stack overflow response about tiles.

3. Because I want the front of the tile to display a bunch of text, I'm going to have to generate a bitmap containing that text. I found a few examples on stack overflow , but somehow when I ran my app the generated image was not applied. I checked the tile docs and they mentioned that setting the BackgroundImage property to a null Uri would just be ignored, but I used the Isolated Storage Explorer to verify that the image was created and in the file system exactly where I expected. After going back to doublecheck anything I'd missed in the sample code, I noticed that their code looked like

BackgroundImage = new Uri("isostore:/Shared/ShellContent/Image.jpg", UriKind.Absolute), 

whereas I had 

data.BackgroundImage = new Uri("/Images/Tiles/paxTile.png", UriKind.Relative); 

data.BackBackgroundImage = new Uri("/Images/Melbourne/melbTile.png", UriKind.Relative);

curiously enough the background image worked, but not the front image. I changed the front image to be in the format isostore:file, Absolute, and it worked! Still not sure why the requirements are different for the front and back - I left a comment on one of the MSDN pages for setting Live Tiles asking about this (and for a note to be added to the docs somewhere).


Now I just have to design the tile layout and actually make it look decent. Here's the plan so far:

            // __________________
            //                               (x)      // <- if you have more than one session on your schedule for the
             //                                        // next hour, this is how many. The tile just shows details on one.
            // my great session...            // max 14 chars? check font size
            //           rest of name?
            // time                                   // in users culture
            //
            // __PAX Digital Assistant__

            // BACK
            //______________________
            //
            // My session name is...
            //    12:30: Unicorn
            // Come play with my f...
            //    12.30: Cat.
            // Leave Me Alone Wit...
            //    13:00: Main Theater
            // _______________________

            // layout wide:
            //__________________________________________
            //
            // My session name is the funniest thing you eve...
            //    12:30: Unicorn
            // Come play with my friends while I make fun of y...
            //    12.30: Cat.
            // Leave Me Alone With My Collection of Pokemon...
            //    13:00: Main Theater
            //__________________________________________

Cannot access x.y.z.dll because it is being used by another process

Spent a while today trying to figure out why I couldn't run an app, which kept giving me the build error that a dll was being used by another process. I found this super nice tip which can help avoid a bunch of procmon/etc fiddling:

1. open a command line
2. go to the directory containing your locked dll
3. >tasklist /m my.locked.dll

via http://edwin.vriethoff.net/2009/01/26/find-out-which-process-is-locking-a-dll-without-extra-tools/

[TestMethod] public static void IAmInvisible()

Just spent a little time debugging some MSTests I was trying to add to a project, which just weren't showing up in the Test Explorer. Most of the 'help these tests don't show up' info out there is because someone is trying to add a new project, but in this case I was adding a new class to an existing project.

Eventually I tried adding a new set of tests identical to some working tests, and editing them bit by bit to look like my failing ones. Perhaps it should have been more obvious, but the offending difference was that I had marked the new tests as Static - that doesn't seem to be allowed, and I can't even find that restriction mentioned anywhere. I guess most people never try it?

Phone devs love this one weird trick.

Just came across a really sweet fiddler trick to make life a little simpler for windows phone development: set up an autoresponder rule that serves the FiddlerRoot certificate from a nice short url, to save a little effort in the process of re-installing it every time you restart the emulator.

http://rikk.it/2013/07/capture-windows-phone-8-network-traffic-with-fiddler/

File locations on azure

Spent the last few days getting everything working in azure with correct folder permissions and so on, leaning heavily on stack overflow for hints. First I found that the templates didn't work in azure because I hardcoded the template locations to c:\users\jafitzge…, had to figure out how to put a location that would work in both the dev and production environments

• http://stackoverflow.com/questions/3038459/django-template-path
• http://stackoverflow.com/questions/6098275/django-problem-with-template-paths

Then I got permissions errors trying to write my result files and debug files

[Errno 2] No such file or directory: 'SampleData2.csv'

[Errno 13] Permission denied: 'scheduledebug10.log'

I tried looking up azure debugging options to see if there was a standard location for log files, etc. http://msdn.microsoft.com/en-us/magazine/ff714589.aspx is only about .net logging options apparently?

http://social.msdn.microsoft.com/Forums/en-US/windowsazuredevelopment/thread/7733a3d8-e80f-4ac9-b285-bf9c652896fd is promising, how do I write to a file in python on azure? Answer leads to

http://msdn.microsoft.com/en-us/library/ee758708.aspx which talks about using the Azure Managed Library to access Local Resources.

http://stackoverflow.com/questions/15146545/ioeerror-in-python-no-such-file-or-directory suggested I needed to change from writing to the current working directory, and instead write to the module directory.  Locally the file is at C:\Users\...\SimplySchedule which is PROJECT_PATH.

Finally fixed all the file permission issues by using the correct folders, and extracted the file path calculations to a couple functions so I could be sure I was doing it the same way everywhere.

Now working on serving the saved schedule file as a download, which seems like should be done using the MEDIA settings. Checking out the django docs again at https://docs.djangoproject.com/en/dev/topics/files/ means I should change the file save location from

import os.path

BASE = os.path.dirname(os.path.abspath(__file__))

To

>>> from django.core.files.storage import default_storage
>>> from django.core.files.base import ContentFile

>>> path = default_storage.save('/path/to/file', ContentFile('new content'))

From <https://docs.djangoproject.com/en/dev/topics/files/>

This stuff looks relevant: http://stackoverflow.com/questions/883313/django-excel-xlwt/883351#883351

This implies I should be saving my debug files to STATIC, not media - that makes sense! http://stackoverflow.com/questions/6619464/django-serving-from-media-but-not-static?rq=1

Also looks interesting for when I create an actual model for all the data I'm throwing around: saving an uploaded file as a model: https://docs.djangoproject.com/en/dev/topics/http/file-uploads/

Now I tried creating public facing 404 and 500 templates and setting Debug=False but it didn't seem to have taken hold when I deployed, although it was working as expected locally.  However when I went back to look at it again a couple hours later it was working as expected. Possibly just a caching issue. 

So, current state - I have all the functionality requested available,  although the site itself looks ugly as hell and super amateur. Still waiting for dad to go check it out and give some feedback, and until then it's time to get some PAX Aus app preparation done.