SQLite woes, cont.

At the end of my last post, I could launch the app and view events, but then got a Null Reference Exception while saving the events into the db. Oddly enough, debugging through the code showed that I was getting a SQLite error code 11 on *every second event*, and then getting a Null Reference Exception after a set of events.The call stack showed:

{System.NullReferenceException: NullReferenceException
   at Community.CsharpSqlite.Sqlite3.sqlite3BtreeClose(Btree& p)
   at Community.CsharpSqlite.Sqlite3.sqlite3_close(sqlite3 db)
   at SQLiteClient.SQLiteConnection.Dispose()
   at PAX7.Utilicode.DBHelper.Close()
   at PAX7.Utilicode.DBHelper.Finalize()}

perhaps I'm calling close twice? I added a breakpoint in the DBHelper.Close() method to check this. I also wrapped the code inside the method in a try-catch-throw to Little Watson block to avoid crashing the app with this exception.

{System.NullReferenceException: NullReferenceException
   at Community.CsharpSqlite.Sqlite3.sqlite3FindIndex(sqlite3 db, String zName, String zDb)
   at Community.CsharpSqlite.Sqlite3.sqlite3StartTable(Parse pParse, Token pName1, Token pName2, Int32 isTemp, Int32 isView, Int32 isVirtual, Int32 noErr)
   at Community.CsharpSqlite.Sqlite3.yy_reduce(yyParser yypParser, Int32 yyruleno)
   at Community.CsharpSqlite.Sqlite3.sqlite3Parser(yyParser yyp, Int32 yymajor, Token yyminor, Parse pParse)
   at Community.CsharpSqlite.Sqlite3.sqlite3RunParser(Parse pParse, String zSql, String& pzErrMsg)
   at Community.CsharpSqlite.Sqlite3.sqlite3Prepare(sqlite3 db, String zSql, Int32 nBytes, Int32 saveSqlFlag, Vdbe pReprepare, Vdbe& ppStmt, String& pzTail)
   at Community.CsharpSqlite.Sqlite3.sqlite3LockAndPrepare(sqlite3 db, String zSql, Int32 nBytes, Int32 saveSqlFlag, Vdbe pOld, Vdbe& ppStmt, String& pzTail)
   at Community.CsharpSqlite.Sqlite3.sqlite3_prepare(sqlite3 db, String zSql, Int32 nBytes, Vdbe& ppStmt, String& pzTail)
   at Community.CsharpSqlite.Sqlite3.sqlite3_exec(sqlite3 db, String zSql, dxCallback xCallback, Object pArg, String& pzErrMsg)
   at Community.CsharpSqlite.Sqlite3.sqlite3InitCallback(Object pInit, Int64 argc, Object p2, Object NotUsed)
   at Community.CsharpSqlite.Sqlite3.sqlite3InitOne(sqlite3 db, Int32 iDb, String& pzErrMsg)
   at Community.CsharpSqlite.Sqlite3.sqlite3Init(sqlite3 db, String& pzErrMsg)
   at Community.CsharpSqlite.Sqlite3.sqlite3ReadSchema(Parse pParse)
   at Community.CsharpSqlite.Sqlite3.sqlite3LocateTable(Parse pParse, Int32 isView, String zName, String zDbase)
   at Community.CsharpSqlite.Sqlite3.sqlite3SrcListLookup(Parse pParse, SrcList pSrc)
   at Community.CsharpSqlite.Sqlite3.sqlite3Insert(Parse pParse, SrcList pTabList, ExprList pList, Select pSelect, IdList pColumn, Int32 onError)
   at Community.CsharpSqlite.Sqlite3.sqlite3Insert(Parse pParse, SrcList pTabList, ExprList pList, Int32 null_4, IdList pColumn, Int32 onError)
   at Community.CsharpSqlite.Sqlite3.yy_reduce(yyParser yypParser, Int32 yyruleno)
   at Community.CsharpSqlite.Sqlite3.sqlite3Parser(yyParser yyp, Int32 yymajor, Token yyminor, Parse pParse)
   at Community.CsharpSqlite.Sqlite3.sqlite3RunParser(Parse pParse, String zSql, String& pzErrMsg)
   at Community.CsharpSqlite.Sqlite3.sqlite3Prepare(sqlite3 db, String zSql, Int32 nBytes, Int32 saveSqlFlag, Vdbe pReprepare, Vdbe& ppStmt, String& pzTail)
   at Community.CsharpSqlite.Sqlite3.sqlite3LockAndPrepare(sqlite3 db, String zSql, Int32 nBytes, Int32 saveSqlFlag, Vdbe pOld, Vdbe& ppStmt, String& pzTail)
   at Community.CsharpSqlite.Sqlite3.sqlite3_prepare_v2(sqlite3 db, String zSql, Int32 nBytes, Vdbe& ppStmt, Int32 dummy)
   at SQLiteClient.SQLiteCommand.ExecuteNonQuery[T](T toInsert)
   at PAX7.Utilicode.DBHelper.Insert[T](T obj, String statement)
   at PAX7.Model.Schedule.sqlInsertEvent(Event newEvent)}


this one
{System.NullReferenceException: NullReferenceException
   at Community.CsharpSqlite.Sqlite3.DbHasProperty(sqlite3 D, Int32 I, UInt16 P)
   at Community.CsharpSqlite.Sqlite3.sqlite3Init(sqlite3 db, String& pzErrMsg)
   at Community.CsharpSqlite.Sqlite3.sqlite3ReadSchema(Parse pParse)
   at Community.CsharpSqlite.Sqlite3.sqlite3LocateTable(Parse pParse, Int32 isView, String zName, String zDbase)
   at Community.CsharpSqlite.Sqlite3.sqlite3SrcListLookup(Parse pParse, SrcList pSrc)
   at Community.CsharpSqlite.Sqlite3.sqlite3Insert(Parse pParse, SrcList pTabList, ExprList pList, Select pSelect, IdList pColumn, Int32 onError)
   at Community.CsharpSqlite.Sqlite3.sqlite3Insert(Parse pParse, SrcList pTabList, ExprList pList, Int32 null_4, IdList pColumn, Int32 onError)
   at Community.CsharpSqlite.Sqlite3.yy_reduce(yyParser yypParser, Int32 yyruleno)
   at Community.CsharpSqlite.Sqlite3.sqlite3Parser(yyParser yyp, Int32 yymajor, Token yyminor, Parse pParse)
   at Community.CsharpSqlite.Sqlite3.sqlite3RunParser(Parse pParse, String zSql, String& pzErrMsg)
   at Community.CsharpSqlite.Sqlite3.sqlite3Prepare(sqlite3 db, String zSql, Int32 nBytes, Int32 saveSqlFlag, Vdbe pReprepare, Vdbe& ppStmt, String& pzTail)
   at Community.CsharpSqlite.Sqlite3.sqlite3LockAndPrepare(sqlite3 db, String zSql, Int32 nBytes, Int32 saveSqlFlag, Vdbe pOld, Vdbe& ppStmt, String& pzTail)
   at Community.CsharpSqlite.Sqlite3.sqlite3_prepare_v2(sqlite3 db, String zSql, Int32 nBytes, Vdbe& ppStmt, Int32 dummy)
   at SQLiteClient.SQLiteCommand.ExecuteNonQuery[T](T toInsert)
   at PAX7.Utilicode.DBHelper.Insert[T](T obj, String statement)
   at PAX7.Model.Schedule.sqlInsertEvent(Event newEvent)}


{SQLiteClient.SQLiteException: Unknown error
   at SQLiteClient.SQLiteCommand.ExecuteNonQuery[T](T toInsert)
   at PAX7.Utilicode.DBHelper.Insert[T](T obj, String statement)}

{System.NullReferenceException: NullReferenceException
   at Community.CsharpSqlite.Sqlite3.ENC(sqlite3 db)
   at Community.CsharpSqlite.Sqlite3.sqlite3VdbeExec(Vdbe p)
   at Community.CsharpSqlite.Sqlite3.sqlite3Step(Vdbe p)
   at Community.CsharpSqlite.Sqlite3.sqlite3_step(Vdbe pStmt)
   at Community.CsharpSqlite.Sqlite3.sqlite3_exec(sqlite3 db, String zSql, dxCallback xCallback, Object pArg, String& pzErrMsg)
   at Community.CsharpSqlite.Sqlite3.sqlite3InitCallback(Object pInit, Int64 argc, Object p2, Object NotUsed)
   at Community.CsharpSqlite.Sqlite3.sqlite3InitOne(sqlite3 db, Int32 iDb, String& pzErrMsg)
   at Community.CsharpSqlite.Sqlite3.sqlite3Init(sqlite3 db, String& pzErrMsg)
   at Community.CsharpSqlite.Sqlite3.sqlite3ReadSchema(Parse pParse)
   at Community.CsharpSqlite.Sqlite3.sqlite3LocateTable(Parse pParse, Int32 isView, String zName, String zDbase)
   at Community.CsharpSqlite.Sqlite3.sqlite3SrcListLookup(Parse pParse, SrcList pSrc)
   at Community.CsharpSqlite.Sqlite3.sqlite3Insert(Parse pParse, SrcList pTabList, ExprList pList, Select pSelect, IdList pColumn, Int32 onError)
   at Community.CsharpSqlite.Sqlite3.sqlite3Insert(Parse pParse, SrcList pTabList, ExprList pList, Int32 null_4, IdList pColumn, Int32 onError)
   at Community.CsharpSqlite.Sqlite3.yy_reduce(yyParser yypParser, Int32 yyruleno)
   at Community.CsharpSqlite.Sqlite3.sqlite3Parser(yyParser yyp, Int32 yymajor, Token yyminor, Parse pParse)
   at Community.CsharpSqlite.Sqlite3.sqlite3RunParser(Parse pParse, String zSql, String& pzErrMsg)
   at Community.CsharpSqlite.Sqlite3.sqlite3Prepare(sqlite3 db, String zSql, Int32 nBytes, Int32 saveSqlFlag, Vdbe pReprepare, Vdbe& ppStmt, String& pzTail)
   at Community.CsharpSqlite.Sqlite3.sqlite3LockAndPrepare(sqlite3 db, String zSql, Int32 nBytes, Int32 saveSqlFlag, Vdbe pOld, Vdbe& ppStmt, String& pzTail)
   at Community.CsharpSqlite.Sqlite3.sqlite3_prepare_v2(sqlite3 db, String zSql, Int32 nBytes, Vdbe& ppStmt, Int32 dummy)
   at SQLiteClient.SQLiteCommand.ExecuteNonQuery[T](T toInsert)
   at PAX7.Utilicode.DBHelper.Insert[T](T obj, String statement)
   at PAX7.Model.Schedule.sqlInsertEvent(Event newEvent)}



and this, seriously?

{SQLiteClient.SQLiteException: SQL logic error or missing database
   at SQLiteClient.SQLiteCommand.ExecuteNonQuery[T](T toInsert)
   at PAX7.Utilicode.DBHelper.Insert[T](T obj, String statement)}

+        ex    {SQLiteClient.SQLiteException: no such table: events
   at SQLiteClient.SQLiteCommand.ExecuteNonQuery[T](T toInsert)
   at PAX7.Utilicode.DBHelper.Insert[T](T obj, String statement)}    System.Exception {SQLiteClient.SQLiteException}

+        $exception    {System.ObjectDisposedException: Cannot access a closed file.
   at System.IO.__Error.FileNotOpen()
   at System.IO.FileStream.get_Length()
   at System.IO.IsolatedStorage.IsolatedStorageFileStream.get_Length()
   at Community.CsharpSqlite.Sqlite3.sqlite3_fileSize(sqlite3_file id, Int32& pSize)
   at Community.CsharpSqlite.Sqlite3.sqlite3OsFileSize(sqlite3_file id, Int32& pSize)
   at Community.CsharpSqlite.Sqlite3.sqlite3PagerPagecount(Pager pPager, Int32& pnPage)
   at Community.CsharpSqlite.Sqlite3.hasHotJournal(Pager pPager, Int32& pExists)
   at Community.CsharpSqlite.Sqlite3.sqlite3PagerSharedLock(Pager pPager)
   at Community.CsharpSqlite.Sqlite3.lockBtree(BtShared pBt)
   at Community.CsharpSqlite.Sqlite3.sqlite3BtreeBeginTrans(Btree p, Int32 wrflag)
   at Community.CsharpSqlite.Sqlite3.schemaIsValid(Parse pParse)
   at Community.CsharpSqlite.Sqlite3.sqlite3Prepare(sqlite3 db, String zSql, Int32 nBytes, Int32 saveSqlFlag, Vdbe pReprepare, Vdbe& ppStmt, String& pzTail)
   at Community.CsharpSqlite.Sqlite3.sqlite3LockAndPrepare(sqlite3 db, String zSql, Int32 nBytes, Int32 saveSqlFlag, Vdbe pOld, Vdbe& ppStmt, String& pzTail)
   at Community.CsharpSqlite.Sqlite3.sqlite3_prepare_v2(sqlite3 db, String zSql, Int32 nBytes, Vdbe& ppStmt, Int32 dummy)
   at SQLiteClient.SQLiteCommand.ExecuteNonQuery[T](T toInsert)
   at PAX7.Utilicode.DBHelper.Insert[T](T obj, String statement)
   at PAX7.Model.Schedule.sqlInsertEvent(Event newEvent)}    System.Exception {System.ObjectDisposedException}


while events from all four days are visible on first load, only events on friday are saved back to the database (inspected with isolated storage explorer). It has correctly saved the star and reminder for the event I added to my schedule, however.

sharing app data for the live tile

So, the recommended way to share data between the app and the periodic task is by files in the isolated storage - perfect, that was my plan!

BUT, my periodic task project doesn't by default have access to the classes defined in the app - which will make it tricky to read and parse the schedule xml :/ And I can't make it reference that project, because it 'is not supported by a Background Agent'.

Clearly, I need to extract shared functionality to a library of some supported type, and reference it from both projects. I'll need
 - event object definition
 - schedule parser

Basically this is all the bits contained in my Model definition, which makes me feel like I must have structured it in a reasonable way :) Unfortunately, once I moved them, I found that I had a lot of View-related code in there, calling App.Resources, etc, which wouldn't work in a shared library :/ Well, I guess the cleanup required here will make the code better altogether, so it's worth doing!

I used this post as an example to get me started: 
http://www.31a2ba2a-b718-11dc-8314-0800200c9a66.com/2011/11/this-is-continuation-of-this-post.html

Specifically, I have an SQLite db which is instantiated in the App.xaml.cs class, which I can't refer to from my library - and is probably the wrong place anyway. But, the DB seems to use the Assembly name to access it: and isn't the assembly name going to be different between the main app and the background task? There's not much reference on the web, but a couple of people say that it isn't possible to use sqlite in a background task. I was worried that I would have to update my model to use the db for the full list, and back to the flat file for the individual schedule, which would be a shame - but I decided to go ahead with the refactoring and try it - and everything ran! It didn't work, quite - it always said 0 events found - but it managed to instantiate the db connection and run through all the code without panicking, so must be most of the way there.

Unfortunately, I found that the database had starting renaming all events to "1.0" when they were stored. I figured out that this was because I had added a new field to the Event object, which was messing up the Insert statement. Once that was solved, I started hitting exceptions when the scheduled task tried to read the db - I figured that was because the db file was locked, because it was never being closed by the main process. I cleaned up the DBHelper file a little, turning it into a singleton and making sure that every operation which called Open() also called Close(). At this point, I decided should probably create an Interface for the DBHelper, which would make testing easier and also make migration easier if I switch away from SQLite at some point.  I recently started actually trying to use some of Visual Studio's refactoring abilities, so here I did right-click on the class name -> Refactor -> Extract Interface. Then select which of the public methods I wanted in the Interface, and Done! Pretty nice.

Unfortunately, when I ran the app again, the database locking was even worse - even the original app process could never open it, always getting the same error. I decided that I did need a real mutex. I found the outlines in another stack overflow answer and added a check/hold/release mutext everywhere I had the open/close db work. Running with this made it clear that actually, it wasn't a deadlock - even when I turned off the background agent, I was getting 'unable to open database' errors every time. Clearly, the actual problem had been introduced when I updated the DB schema. I could validate with the Isolated Storage Explorer that it was being copied into storage correctly. I stepped through the code and verified that the db was being opened successfully on first contact - but then when it attempted to execute a query, the code tried to open a dbname of "Events.sqlite-journal", which threw a storage exception because it didn't exist.

I followed advice (from Stack Overflow, obvs) to create the journal file back at the start with the actual db file - but still got an error when trying to open it. To narrow down the problem, I tried opening the file straight after creating it - and still got the error. I think I should have been able to figure this out from there, but instead I went and searched to get a list of basic causes for this error - and the very first answer was 'you haven't closed the stream that was returned when you created the file'. Duh, I mean, really. Added a file.Close(), ran the app, everything works! Well, except a Null Reference Exception when I tried to add an event to my schedule, but lots of things worked. Some, at least. More than earlier today.

(this post was written as-it-happened, between december 17 and 26. Very useful for tracking what I had done when I was only doing random half-hours of work!)

Django and sql azure, wtf?

Chapter 5 of the Django Book - Models

I thought about just skipping this one as I don't anticipate needing a db for the schedule app, but decided I might as well follow through, it'll probably come in handy soon enough.

So step 1 I created a sql database in azure to use, and now I need a sql engine for python.

I found https://pypi.python.org/pypi/django-pyodbc-azure/1.0.5 which told me to install pyodbc first, which was googleable enough, and available for both 2.7 and 3.x. But the 2.7 version said it couldn't install bc I don't have 2.7 in the registry…hmm.  After some hints from Stack Overflow, I checked the python version I have for 2.7 and it's the 32 bit version, which won't be found by the 64 bit installer. Ran the 32 bit installer and done. Then it said to use pip to install itself, which I didn't have, but SO led me to this great page: http://www.lfd.uci.edu/~gohlke/pythonlibs/#pip…but after running that, pip still wasn't recognised on the command line. Another useful note on SO said it installed to python/scripts instead of just to /python, so I had to add that to my path. And now I get 'failed to create process'. Tried in admin mode, same error. Google was initially not forthcoming about this error, but then turned up this: http://www.maphew.com/Python/python-fixit-snippets/ which suggested that it was unable to find python. Perhaps it is confused by my python27 option? The location certainly hasn't changed since I installed it. I uninstalled and reinstalled to be sure, but that didn't help.

Checking the Event Viewer to see where it's trying to find python might help -nope, can't see any relevant messages.

I went back and installed Distribute, as it is mentioned in the same place as where I got pip. No change.

I found this gui for pip that lets you switch between active python installers - that might help, because it could still be being caused by my dual 2 and 3 installation? https://sites.google.com/site/pydatalog/python/pip-for-windows but it also failed. But it did point out that I have no pip.exe in c:\python27 - which is odd because I do have a Removepip.exe? Turns out pip.exe is in c:\python27\scripts, but moving it just got me back to failed to create process. Hmmmmmm.

So I went back to the site, and noticed it had a downloadable zip. So I downloaded that, and ran setup.py install, and it appears to have installed something? But not a full package. In fact I can't even see a real pyodbc package...

But just to make sure it isn't installed, let's try following the instructions on the pyodbc-azure site. And nope, I get jango.core.exceptions.ImproperlyConfigured: 'sql_server.pyodbc' isn't an available database backend.

 

So, that was a couple of unproductive hours, and I give up. I will instead attempt to follow this tutorial (http://www.windowsazure.com/en-us/develop/python/tutorials/web-app-with-blob-storage/) for connecting a python app to azure storage, which should be perfectly adequate for anything.  And to get through the Django book I can just go ahead and use sqllite.

Getting started with sqllite: http://sqlite.org/sqlite.html

Easy!

Quickly created and connected to a database, created a model, turned it into tables, added data to the tables, and started querying the data out of it

• All rows
• Filter: Field = x, field contains x (see Appendix C for field == x, or < x?)
• Get(): A single object instead of a list ( filter returns a list) uses same options as filter
• Order_by
• Slicing - works just like a python list!
• Use .update(x=1) instead of pub.name='a' to get more efficient sql translations (setting a field directly will cause all fields to be saved over)
• .delete() can be called on an object or a query set (result of .filter()) Django tries to save you from accidentally deleting everyhting by requiring the odd syntax x.objects.all().delete() to do that