CodeTender: A Brief History

I was lucky to be involved in building a micro-services platform with Vasil Kovatchev. Vasil is a great software architect and overall great guy. One concept he introduced was the “bartender” script. The idea was to build a working micro-service with a placeholder name that won’t conflict with any code. The first such template was called “Pangalactic Gargleblaster”. The bartender script replaces placeholders in the code (“Pangalactic” and “Gargleblaster”) and does some other stuff to make the service work with the platform. The bartender script is a bash script and the “other stuff” is…well…more bash script. We were quickly serving up not just Pangalactic Gargleblasters, but also Flaming Volcanos and Tequila Sunrises and Whisky Sours. I fell in love with this concept, but I don’t, however, love the bash script since it is very tightly coupled to our ecosystem. Since node.js is a de facto standard for dev utilities (no offense, Python), I set out to build a CLI with these basic requirements:

  • Clone a git repo or copy a folder to a specified folder
  • Display prompts to collect values to replace placeholders with
  • Run scripts before or after replacing placeholders

Since “bartender” is a bit too common of a name, I went with “CodeTender”. That was March of 2018 (according to my commit history). Fast forward a few years and version 0.0.4-alpha was still full of bugs and my somewhat ambitious issue list was collecting dust along with the code. A somewhat unrelated thread with Vasil and another colleague reminded me of CodeTender so I set out to get it working. A few minutes (hours maybe) later, and the basics were working. So of course, time to use it.

I have been playing with Home Assistant lately and wanted to start playing with custom dashboard cards. I found a boilerplate project and decided why just clone the repo when I could get CodeTender working even better and make a Home Assistant custom card template? So after about another week, I have burned down a lot of my issue list and added some cool features. It is still very alpha, but it’s close to ready for prime time.

I will formally launch CodeTender in a future post, but for anyone interested, you can check it out on my GitHub.

Don’t Dispose Your Own EF Connections

I’m working on upgrading a framework to dotnet core so I am moving from .Net 2.x conventions to netstandard 2.2. Our code was using DbContext.Database.Connection to get DB connections for custom SQL. I needed to switch to DbContext.Database.GetDbConnection(). I made the wrong assumption that GetDbConnection() was a factory method and returned a new connection every time. Therefore I made sure I was disposing of each connection. Tests immediately started failing with “System.InvalidOperationException: ‘The ConnectionString property has not been initialized.'” After investing way too much time due to the complexity of the framework and my own stubbornness, I narrowed the issue down to the following scenario:

    using (var conn = context.Database.GetDbConnection())
    {
      conn.Open();
      using (var cmd = conn.CreateCommand())
      {
        cmd.CommandText = "SELECT * FROM sys.databases";
        cmd.ExecuteNonQuery();
      }
    }

    using (var conn = context.Database.GetDbConnection())
    {
      conn.Open();
      using (var cmd = conn.CreateCommand())
      {
        cmd.CommandText = "SELECT * FROM sys.databases";
        cmd.ExecuteNonQuery();
      }
    }

The real issue is the second call to GetDbConnection(). This does not in fact return a new instance, it appears to return the previous connection and the ConnectionString property has been set to an empty string causing the exception about ConnectionString not being initialized. You can test this yourself with the following:

    var conn2 = context.Database.GetDbConnection();
    Console.WriteLine(conn2.ConnectionString);
    conn2.Dispose();
    Console.WriteLine(conn2.ConnectionString);

The fix is to simply not dispose of your connections or commands. As indicated in this issue comment, disposing of the context will dispose of any connections created using GetDbConnection(). Therefore the correct implementation of this use case is as follows:

  using (var context = new MyContext())
  {
    var conn = context.Database.GetDbConnection();
    conn.Open();
    var cmd = conn.CreateCommand();
    cmd.CommandText = "SELECT * FROM sys.databases";
    cmd.ExecuteNonQuery();
    conn.Close();

    var conn2 = context.Database.GetDbConnection();
    conn2.Open();
    cmd = conn2.CreateCommand();
    cmd.CommandText = "SELECT * FROM sys.databases";
    cmd.ExecuteNonQuery();
  }
horse zebra and pony

Look for Horses, Not Zebras

I have two quotes from old friends that have stuck in my head in certain situations.  The first is, “dancing is like standing still, only faster.”  As a runner, I have adapted that to, “running is like falling down, only slower.”  The second is, “when you hear hoofbeats, think of horses, not zebras.”  As a programmer, I have found there is not much better advice than this second quote.

You would think that the more experience you have, the more likely you are to look for the obvious answer.  I’ve found that the opposite is true – while you are going to make “stupid” mistakes less frequently, the majority of your mistakes are still going to be “stupid”. Unfortunately with great power comes great ego, and the most difficult bugs are caused by the occurrence of something that was “never going to happen”.  I happened to stumble through 3 of these self-induced hair-pulling-out exercises in a matter of days.  To make matters worse, I don’t really code at work anymore.  This was a hobby project that I can only work on for an hour or two each night.

First was the infinite loop (and subsequent stack overflow) – one of the few errors that can’t be captured with standard error logging techniques.  This problem only happened on the production server so instead of doing the obvious and duplicating the data down to dev so I could debug, I decided to add debug logging until I found the problem.  This was the first symptom of my bravado.  My application is a state machine “engine” that runs in a loop. As things happen and the state changes, the loop eventually completes.  I narrowed down the problem to the state where the error occurred but had trouble getting closer than that.  I was convinced that some dark magic was at play and some data anomaly so I spent a few nights adding debugging, publishing, running, repeating.

Finally, I stopped looking for zebras.  Instead of continuing to try to zero in on the line of code through trial and error addition of debug logging, I put 3 lines of code in the most obvious places the problem could occur – the 3 while loops that could get called inside my main loop. That immediately pointed me to the loop in question so then I threw an exception when a condition causing an infinite loop occurred.  It probably took me all of 5 minutes to determine the state that caused the problem.

So next I had to figure out how the code got into that state.  All of the information I needed to solve the rest of the problem, I saw within minutes – and completely ignored.  I saw that the state in question could only come from the database because the key data was never set anywhere by code.  Therefore, the code that allowed the infinite loop condition must have been in the code that loaded the data from the database.  However, rather than looking right where all indications pointed, I looked for zebras.  Since I knew how to find the problem, I could now duplicate it on my dev box and debug.  That meant I could spend time inspecting variables at breakpoints to see why the problem was happening.  After all, this infinite loop was caused by a recursive linked-list relationship – something that was “never going to happen”.  There couldn’t have been an obvious reason for something like this to happen so I had dig for a needle in the haystack instead of following the thread attached to the needle.  In spite of myself, I did eventually figure out that the problem could absolutely be solved when the data was loaded into the engine.  Technically this is after the data is extracted from the database, but the recursive situation doesn’t matter until it gets to the engine so that’s where I put the fix to truncate the recursive linked list since the duplicate reference shouldn’t have been there anyway.

Fast forward a few days.  With my problem solved, I was able to run my engine for a few days and then started to look at the output.  Then I stumbled on a new error – “An error occurred while reading from the store provider’s data reader. See the inner exception for details” and the inner exception was “Arithmetic overflow error for data type tinyint, value = 2288”.  So, the expert programmer has managed to break the Entity Framework.  Of course I checked the “obvious” problem – that my model had the wrong data type.  Then I went off the deep end.  Unfortunately, there are people who have had legitimate problems with EF or linq-to-SQL mapping parameters incorrectly (2288 was a parameter, not a database value).  I converted my linq expression into a SQL command and then realized I was once again looking for zebras.  Instead of looking at the columns in my database that would map to tinyint, I assumed the problem was in the Entity Framework – code written by Microsoft and used by thousands if not millions of people – and not in MY code.  When I stepped through my code and looked at the parameters, I noticed there was a parameter that should’t have even been there.  That was my problem – I passed the parameters into my method in the wrong order!  Instead of passing “int, null, int, true” I was passing “int, int, null, true”.  The 2nd and 3rd parameters were both nullable ints so as far as the compiler was concerned this was all good.  Unfortunately it meant it passed my very large primary key into a parameter that was expecting a much smaller value – something that should have been very obvious from the error.

This post has been sitting in draft for so long that I have now had several more “events”.  The most recent was while doing a demo on AngularJS directives with my team at work.  I was trying to illustrate scope isolation and no matter what I tried I couldn’t figure out why the attribute from the directive specified in my HTML wasn’t making its way to my link function in the directive code.  When I turned off scope isolation and pointed at the parent scope, everything worked fine.  Even my more Angular-savvy team members were stumped.  Then came the “duh” moment – I actually had 2 directives nested inside each other.  I was looking at the code in the inner directive and passing data into the outer directive.  The HTML of the inner directive wasn’t passing the value from the parent.  Yet another zebra chase.

So in conclusion, drop the attitude!  When you have problems, they are most likely the most basic and amateur issues.  Reign in those horses and let the zebras smack you in the head, don’t go looking for them.

UPDATE: 29 January 2019

I have done it again! I was working on a new framework feature and couldn’t get my test to work. I just KNEW it was a problem with the third-party software I was using. More specifically, I wasn’t sure what I was trying to do was even supported. Then somehow I managed to get everything to work directly with said software. After days of struggling, I discovered I was writing the final portion of my test immediately after testing deleting the exact thing I am trying to test. As usual, my code was doing EXACTLY what I told it to do.

On a side note, since the original post, I was introduced to the concept of rubber duck code reviews. I highly recommend this and your coworkers will thank me.