More robust verification

Fragile: relying on certain behavior between action and response, and using this to know when to validate the response

Better: make no assumptions about the behavior. Instead, continuously check for a valid response for a reasonable amount of time.

e.g.:

Fragile:
- perform action
- wait for loading ui (if the devs change or remove loading ui, this will break)
- validate response

Better:
- perform action
- check for valid response
- wait a second or two
- check again...for 5-10 seconds

The second is only invalidated if the response changes (less likely). The former is invalidated if the loading UI changes (apparently, any time the PM/devs get bored).

TimeStamp100nSec is DateTime FileTime

The TimeStamp100nSec value of CounterSamples is in the same format you get from DateTime.ToFileTime() (or maybe DateTime.ToFileTimeUtc() in some situations, not sure).

This was not made clear to me in any documentation I could find, but some example code half-way down this page spells it out.

To go from TimeStamp100nSec to a DateTime object, use DateTime.FromFileTime(timestamp100nsec).

Division of labor for automation

Devs should maintain robust task library of functions that perform and validate user interactions (e.g. "Select", "Resize"), so they can easily update these functions at the same time the behavior changes in the build.

Test should maintain the scenarios that use these basic actions. This way, they can stay above the nitty gritty of changes to buttons, DOM, etc, and focus on building a large library of interesting/eclectic scenarios and test files.

This is especially bad with actively changing features -- if test is responsible for the tasklibrary as well, then they get stuck churning code just to keep the most basic functionality tests up to date. This is made worse if dev hates the automation system test uses...

Easily sleep/restart/shutdown Windows via remote desktop

To easily shutdown/restart/sleep a windows machine via remote desktop, click on the desktop and press Alt+F4. This will bring up the standard Windows shutdown prompt (where you can choose shutdown, restart, sleep, etc). (I didn't know that!)

More info here.

VSTS data bindings and unicode

I have a test with a data source (settings.csv) and a data binding (mysetting). Settings.csv exists, and there's an entry for 'mysetting', but VSTS complains:
Error...Could not run Web test 'test' on agent 'agent': Could not access table 'settings#csv' in data source 'settingsSource' of test '12345678-abcd-abcd-abcd-12345678abcd': No value given for one or more required parameters.

Turns out it's because the csv file is Unicode. The easiest way to fix this (for me, at any rate), is Powershell:
cat settings.csv | out-file temp.csv -encoding ascii
Compare the sizes of settings.csv and temp.csv with ls. If temp.csv is smaller, then settings.csv was unicode.
mv temp.csv settings.csv -force

The truly maddening thing is that VS seems to be saving out as Unicode itself, so I have to edit my csv files elsewhere.

Increasing VSTS limit on old test run results

By default, Visual Studio Team System will only save 25 of your test run results. To increase this number:

1. Tools -> Options -> Test Tools -> Test Execution


2. Change the value after "Limit number of old Test Results to:"

Note: you still may have a practical limit if you use the freely included SQL Express (default). I think the artificially imposed limit is something like 5GB, not sure.

The "Uncanny valley" of automation

Automation is not the panacea I thought it was.

Implemented incorrectly, an automation system will waste more of your time than it saves. Sure, you won't have to run through the same checklists over and over, but your time will instead be spent tracking down a disproportionately large number of false positives. At first, I thought: "Hey, at least it's code! It's got to be more fun than just using the product, right?" Now I'm not so sure.

In robotics and computer graphics, there's a problem known as the "uncanny valley": when you get closer to modeling realistic human behavior, the results can be unsettling. I propose that there's a similar problem with automation: the closer you get to trying to simulate human behavior, the more problematic the results. (A bit of a stretch, and due to an entirely different set of problems, but bear with me).

Humans are good at dealing with the abstract. Take, for instance, the hard to read text you're supposed to identify when signing up for something (it's called a "CAPTCHA"). Reading these is (usually) easy for a human; we look at the blurry distorted mess and see letters. This is something we are good at. Writing a program that can read those things, on the other hand, is a difficult computer science problem.

Conversely, say you had to create dozens of accounts somewhere (perhaps as part of testing an online service). Completely filling out the name, address, interests, secret question, etc, is boring, repetitive, and prone to mistakes. These sort of tasks are not our forte. However, it's easy to write a script that fills in all of the fields for you. (Especially with a web page, since you can interact directly with the DOM).

This example illustrates the balance between manual and automatic tasks. Humans are good at abstract reasoning and big picture. Computers are good at repetition and precision (note I didn't say accuracy :-P)

Human users have no problem with slight changes in design or layout (and sometimes they won't even notice); computers tend to go apeshit. This is what causes so many of the automation failures you'll be tracking down. Maybe the browser started minimized, or another window popped up in front of it and stole focus (automatic updates, anyone?). Maybe the page took a few seconds longer to load, and the expected items weren't there when the script checked for them. Maybe the designer moved a button or changed a label. Or, maybe your test itself was wrong! There may be a legitimate outcome that you, author of the test, didn't think of. (When tests are code, they can have bugs too!)

Where computers shine are precisely defined problems. This suits them well for testing behind the scenes, where the input and output is not so abstract. If I send this packet, does the server give back that response? If I call this function with this value, do I get back that answer? What makes this sort of testing and verification so boring (and difficult) for humans is precisely what makes it so perfect for computers.

Another area where automation shines is tools. Running a series of installers and patches...opening up dozens of web browsers to a series of long, convoluted URLs...finding all the logs from a certain period of time, spread over dozens of computers, compressing them, copying them to a central repository, and emailing all interested parties about their availability...these are the sort of mindless, error prone tasks that waste tester time and are just begging to be automated.

The take away from this is that automated tests are most useful when they augment human testing, not try to replace it. Automation should simplify the lives of human testers -- by taking over the tasks humans are inherently bad at -- so they can focus on what they do well: finding problems in the user experience.

If SharePoint Service Instances are "Provisioning" forever...

Note to self: Make sure the SPTimer service is running on all machines in your SharePoint farm. If things seem "paused" (ie: service instances are "provisioning" forever), this may be the culprit.