VBScript to Remap your Outlook Profiles to a new server.

Recently recovered from an incident involving the slow and painful death of a MS Exchange Server. Right smack in the middle of the day, with some 800 mailboxes on that one host. We restored the mail stores to another Exchange Server, and ran a hack script to go and correct the homeMDB, homeMTA, and msExchMailServer attributes in the AD to point to the new host. That was the easy part. Then we had all these users with outlook profiles pointing to a dead server. Outlook may be resilient in picking the right server when you point it to the wrong server, but if the wrong server (to which is is pointed) is not available - as in if the server it’s pointed at is DEAD, then it doesn’t know to get the new info from the AD to point you to the correct one. This, of course, is exactly what happened to me an my ilk. We had some hundreds of users without profiles pointing to the right host. You’ve heard of Autoconfig for Outlook 2007, but that only works for NEW profiles, not existing ones suddenly pointing to a dead host. So what to do? Manually have all the users repoint their profiles by talking them through the Mail Control Panel widgets? NO! That’s expensive. Instead, write a script and deploy it as a login script.

Why a Login script? Well, because this script affects the HKEY_CURRENT_USER registry hive which only exists for a particular user when that user is logged on. At least for all intents and purposes that’s true. I think it may be possible to somehow enumerate all users on the HKEY_USERS hive, but I didn’t go that far. I apologize up front for not going the extra mile on this one, but hey, look! A free script.

NOTE: If you copy and paste it, and get a compile error, check for abberant the new-line characters caused by the squishing of my code by the text rendering on the page.

Quick, list the dbs and Users!

I’ve been orchestrating a move from an old SQL 2000 database to a SQL 2k5 DB host. Most of the dbs are in use somewhere by some application or other, and I can’t tell by looking at it just what they are. One of the things I’ll need to do is contact all of the users of each one and ask them to tell me what the impact is, as well as inform them that they’ll need to swap a hostname out in the connection string. No problem, I can just click through all 50 databases on the host and look at all the users, and write them down, and send an email to them all asking what gives… The trouble is, half the time they have no idea what db’s are on the host, and what if any they have to do with it. I didn’t feel like spending about 4 hours on this, so I quickly tossed off a script that generates some SQL to figure out the real list, without getting too complicated here. In case you find it useful, here’s the script to generate the second script:

After that runs it will generate in the output window (or the console if you use OSQL or some other command line tool to run it,) this form of output. I pipe it to another file, and change the last ‘union’ to a ‘go’ and we’re done. I probably should have also excluded the user dbo, but I didn’t you can if you want.

Rich Programmer Food: Highly recommended reading

http://steve-yegge.blogspot.com/2007/06/rich-programmer-food.html

I have been preaching things like this for a long time, but Steve here is a bit more fluent about explaining it. I’ll not try to turn you off to the ideas presented here by summarizing it before you read it. Honestly, it’s the perfect expression of a relatively amorphous emotion I’ve had about compilers and their place in the world of CS for a long time, (but obviously I can’t take credit for the quality of Steve’s writing, of course.)

Make yourself read it though, please, if you never do another thing for me as long as you know me. (Halo_Four, if you’re reading this, this means you too.)

Dirty Data?

I’ve heard from all sides lately about how there is a growing problem with ‘Dirty Data’…

What is Dirty Data? According to IBM, and various other folks, Dirty Data is data that is extracted from the real world that cannot be nicely integrated into relational models because it’s not organized, or doesn’t fit a particular model. Luckily for us, there are various products out there to ‘clean it’ or techniques recommended by the various database manufactures to squeeze ‘clean’ over ‘dirty data.’

First of all, WHAT?! Do we work for computers or do they work for us? There is no such thing as dirty data, the fact of the matter is that if your system pukes because the data isn’t in the right format, then you’re using the wrong system to process it. The real world isn’t just ‘full of dirty data,’ it’s full of data, and computers are here to help us process it. It’s completely ridiculous for the database companies to say that my data is dirty.

Another sore point with me is the fact that they say that performance problems can stem from ‘unreasonable relationship tracking.’ I want to know too many facets of my data, and it’s not ‘database correct’ of me to want my data and to be able to eat it too, from every angle, in real time. Pardon me, but the fact that YOUR relational database doesn’t handle this kind of scenario well doesn’t mean that my request is unreasonable. If I want to be able to efficiently track back to the original record by any one of any number of my many to one or many to many, or one to many relationships, that’s my requirement, not your place to tell me that it’s unreasonable of me to expect that it can be done.

The problem is that relational database proponents, and in particular the big three commercial implementors, are all engaging in this ‘you can’t do that, in real life’ and ‘its your fault our apps are slow,’ propaganda, and I won’t stand for it.

Just as an example, and I’m sure it’s not the only way, I’m using Relavance, an associative model engine which has none of these limitations.

Even Relavance aside, or even the entire associative model aside, it’s still really stupid of any real computer scientist to think, and especially to decree that certain problems are just not to be solved using computers because they are not well suited to modern technology. If that were the attitude that people had throughout history, then we’d still be toggling in code on the front panels of computers the size of Olympus Mons. Microsoft may be the market leader, and Oracle and IBM may sell a lot of bits and bytes, and tow very heavy loads in the marketplace, but they are not here to tell me what is and is not possible in computer science based on their own models of the universe.

</rant>

As I said before…

I mentioned in the middle of one of my other posts that I was preparing an article on Ruby. I sort of alluded to the fact that I’d post a link, so I suppose I’ll do that now, (although it’s more likely that you’ve come to this blog FROM the article than the other way around.) Ruby for C# Geeks (DevX.com) It’s not really all that in-depth, but just to make C# folks aware of what else is out there.

The Associative Model of Knowledge

Forget everything you know about storing data. Particularly the bits about retrieving it after you’ve stored it. I said forget it, and I know you didn’t. So, try again, forget it.

Rather than try to start by comparing this method to what you already know how to do, I’m going to just pretend that you’ve always wondered how to store data, and never have done it before.

Applications acquire data. They also retrieve data and display it for users, or do something otherwise useful with it. Sometimes it’s the users who give the data to the application directly. Actually that happens quite a lot.

We store different bits of information in different places, and we do so for the purpose of organizing things that go together. We go to all of this bother because we’re likely to want to get them back out together. What that means to you and me as developers is that we spend quite a lot of time thinking about the kinds of things we want to get out of the system before we go putting them in. Arguably we try to think of everything we’d like to get out of it beforehand, and that’s how we’ll know we’re done with our model – when it provides us with a place and method to put everything we need to know into the storage and get it back out on demand. We’d use this information of what we need to get out later as the basis for a ’schema.’ A layout of how all the data looks when it’s just sitting around waiting for someone to get it back out again.

Great! That sounds simple enough! Ah, but life isn’t so forgiving, and neither is programming. Today we think we know what we want, and that’s it, but tomorrow, that all might change. In fact, if you ask enough pointy haired bosses about what they want from the data they’re paying you to store in a database, then you’re likely to get a few things in common, and a bunch of special requests. Some of what they want is ’stuff’ that needs to be in there, but also, they want ‘ways to organize things’ on the way out especially. Some people would call these differing requirements views, or reports.

Fine, so we can’t possibly think of all of the kinds of views and reports, so we might as well just not think about them ahead of time, right? We’ll just store everything as generally as possible while leaving some things together that obviously go together in pretty much all cases, and that way, we can let the person getting things out tell us what they want at that time, and never worry about it again. Especially not that would push the application delivery date back! Right? WRONG!

When we store data, each thing we’d like to know about is called a Concept. And all of the different aspects of the concepts are different sorts of ‘lights in which we’d like to view these concepts,’ which we might term a Context. So in life we have concepts, or things or ideas of interest, and contexts, which are the glasses through which we view these concepts.

When we want to store a concept in our knowledge store, we put it in there, and designate it to be accessible in a particular context. Of course it’s not really friendly to the user to just pull every concept out with every time we want to retrieve information, so the idea of limiting a concept to a particular context seems to make a lot of sense. We do this in our everyday conversation, even sometimes a single word can mean many things! Surely that’s an idea we’d have difficulty describing in a file… Or is it?

Lucky for us, Everyone in the Universe uses Associative Knowledgebases, and everyone knows that in an associative knowledgebase, you add concepts to the base, associated with a particular context, and from there you are able to retrieve concepts relative to the particular context of reference. You can have a single concept be availabe in as many contexts as are needed, and you don’t need to keep putting multiple representations of the same concept in over and over again to get the job done. Each concept has a bit of data that goes with it, and it can be associated to other concepts to represent what sorts of ideas it shares with its associates. Now an association isn’t just a one type deal, you can associate on different ‘axes’ of association to represent different aspects of what the two concepts have in common. Who knows, two concept items may also be associated to the same context, so when you go to retrieve information relative to a context, you’d get them both anyway! As you may have picked up, each ‘concept’ can be represented by a single ‘item’ or a set of ‘items’, which are tiny units of free-floating information. Every item is associated to at least one context, and some associated to thousands of contexts, just depending on what you need from it. Each item is universally addressable by it’s own unique four ordinal vector. Each item stores its own value, as well as the vectors of each of its associates, grouped by which axis they happen to be associated on. The items can be associated to other items, or to entire contexts, it’s completely up to us! We can associate multiple items on the same axis or split them up, and there is virtually no limit to the number of associates we can have in any particular axis. (Could be anywhere from zero to several million or even sometimes several billion other concepts.) Since we have this wildy variable method of associating one uniquely addressable item with any number of uniquely addressible items, on any number of axes, it’s lucky for us that we don’t need to know how many of what kind of associates we can have right up front. Even items in the same context can have different numbers of associates, and there is no negative consequence to the other items. It’s cool that nobody decided that we’d have to allocate null associates to represent items without associates, because when we got over a hundred, well that would just be ridiculous! We certainly can’t easily convert items into something that fits completely into an Excel spreadsheet! That’s quite alright, because we have a lot at our disposal now. (And if we REALLY like Excel spreadsheets, then we can limit the information coming out in such a way that it fits, complete all of the totally unnatural quirkiness of any spreadsheet, some empty columns in some rows, and flat tuples that are all but meaningless without some formal explanation of what they are and what they mean — like any Excel spreadsheet, pointy haired bosses love those.)

Of course we can predetermine some of the ways in which we’ll have to establish associations based on the context we wish to use to represent our particular concept, and some of them will be common to all items in the same context. These we’ll call ‘explicit mappings’ of information. For example if you’re adding the concept of an Apple to the Fruit context, you’ll likely want to also associate it to a member of the Plants context to indicate what it grows on (we can even say that this association exists on the ‘Fruit Grows On’ axis.) These are simple, and the rules are always the same.

We also have the ability to examine concepts we wish to associate with a context AS THEY ARE BEING ADDED TO THE BASE and apply a set of rules to them to determine if, while we’re at it, we ought to associate them with any other concepts or contexts. In this way, we can define a list of rules which dynamically determine to which other contexts and concepts we would like to associate with this concept, based on well… what it is. Contexts and concepts don’t have a set structure. They can be anything we like them to be, and represent any aspect we’d like them to. For example, sure we have a context called Fruit which is associated to all of the things that you and I call “Fruit” like Apples, Pears, and Bananas (a personal favorite of mine!) but you can also talk about a context that’s more abstract like ‘Found in Africa’. So all of the things we know about that are ‘found in Africa’ can be associated to the Found in Africa context. That means we can have some of the Fruits, and some of the Plants all be associated to the same context of ‘Found in Africa’, so when we ask the Knowledge base for what sort of concepts it knows about in the context of ‘found in Africa’ we get both some Fruits and some Plants. Isn’t data storage great!?!

Well, pointy haired bosses are never satisfied, so when they see you can do such wonderful cartwheels with individually simple data, they still want more.

“I’d like to be able to find what sort of Fruits and Plants we can find in Africa that are legal to export to the United States.”

Just when we thought we’d thought of everything, we now have to add something else!

“Ok,” we tell the boss, “No Problem!” It’s a good thing that we don’t have to completely redesign the knowledge base just to add a new context for some concepts we already have! We can just create a new context called “Exportable To the US” and associate the proper concepts with that context as well. Great!

So, it’s also relatively simple now to construct ‘queries’ for this kind of data store. When we write the application to show the boss what sorts of Fruits and Plants are Exportable to Africa, then we just have to retrieve all of the items associated with the context of Exportable to the US and also Found in Africa, and then make sure that the results are either Fruits or Plants. It’s rather like drawing a venn diagram. Since we used rules to make the associations on the way into the storage system, we don’t have to do anything but grab the ready and waiting list of associates we need on the way out! Sure it takes a tiny bit of effort when the concepts are inserted into the store for the rules to figure out where they are to be associated, but that all runs asynchronously, and we didn’t care about it when we added it. Only now when the boss comes to get his answers does the effort really matter! So, with almost no effort at all, we’ve given the boss what he wants. We can only hope that there aren’t too many more bosses to satisfy today, because we were supposed to go out for a drink after work. It’s a good thing everyone uses Associative Knowledgebases, or who knows how long we’d be here?!?

A Post Facto I forgot to mention, this is a description of a working system. It’s called Relavance, and it really does all of the wonderful things I just described as if they were hypothetical.

HEY WOULD-BE COMPILER WRITERS!

I’m sorry for the capital title, but this is big. It’s old news technically, but for me, it’s huge. I found an article that actually shows from step one to end how to write your own scripting language. I was going to write one of these articles myself, but now that I’ve been beaten to the punch by about 8 years, I think I feel a little relieved. Check out this wonderfully not new, but still awesome article by Jan Niestadt at http://www.flipcode.com/articles/scripting_issue01.shtml

I’m so impressed it’s ridiculous. What a great world we have folks! Free compiler writing advice for everyone! If you deny yourself this lesson in Computer Science… well then what are you doing twiddling around reading obscure little blogs like mine? You should just go back to selling insurance or whatever subject you started your degree on in the first place.

An Open Letter to Terence Parr

Mr Parr,
You never met me, but I’m very well acquainted with you and your work. You’ve produced a set of good tools, and I’ll have to admit that I’ve been inspired by your endeavors. If it weren’t for you and ANTLR, I’d not know what a parser is. I’d never have spent all the time frustrating myself trying to figure out how to write a compiler, or even an interpreter. Overall I can’t blame you for firing my imagination enough to make me want to pursue the now very real possibility of implementing my own language based solutions, but I do blame you for one thing: your famous quote of quotes:

“Why spend 5 days writing by hand what you can spend 5 years of your life automating?”

Yeah that’s right. It’s all your fault. I was so profoundly wounded by that sentiment that it drove me up the wall. It carried over into everything I’ve been working on. I’ve somehow had it in my head that I should never write a line of code by hand again, other than to generate other code, or even code to generate code generators. While this has the potential to be wildly productive… oh say… five years from now, it’s making me look silly to everyone else who’s ripping out RADD apps left and right in front of me! I refuse to wire up a UserControl manually because I just KNOW that I can write a tool to generate it for me, and I absolutely abhor the concept of designing a web page layout IN THE ACTUAL PAGE because surely there is a way for me to abstract it to meta data that will allow me to rattle off three lines of script that will generate a page that will handle any and every single type of input that will ever be devised by man and his future ilk.

I guess the part that I’ve been missing, and, really I have no excuse for it, you’ve provided all the wisdom that I should have needed right there after the ‘when you can spend’, is that part about the ‘five years of your life.’ I mean, I’ve read it enough, and said it enough, and it’s right there in ones and zeros. I should have figured out what that meant by now. But no. Sadly, I’m still on the quest to automate everything, and Terence, it’s all your fault! You corruptor of simple minds!

Sincerely,
Dave

Phew! Finally. I got that out of my system.

Why’s (poignant) guide to ruby

You will read this. Yes, you will finish it.

All of it. So it is written. So it shall be done.

Go forth, learn, and laugh.

And after you're done, then go here.

Users Make Awful Robots!

Ok folks, I have a lesson learned for you. I wouldn’t say that I’m a perfectionist, but this was a crazy experience that’s left a long lasting impression on me.
A week after declaring an app I wrote “ready for production” I got a list of 10 items the client considered to be a bug. Nine of these turned out to be minor, quick fix, standalone bugs. “Standalone bugs,” you ask? One of the things that’s bad about knowing the list of flaws in your system is that you’re certain of where it will break, and what circumstances will cause it to flaw or crash entirely. One of the good things about knowing the list of flaws is that you’re certain of where it will break, and what circumstances will cause it to flaw or crash entirely. With every fix you will always introduce the uncertainty that your new code will break something else that seemed to have been working before. So, check ten items off the list, and you’re still not sure that your list of bugs is at zero. I did just that, and got burned pretty badly.  The final item on the list turned out to affect the ability to edit existing data, and I looked really stupid.  I wouldn’t have myself ever discovered the bug, because it arose out of them using the application in such a way I wouldn’t have ever imagined.  They claimed that they do this all the time, and needed to be able to, so I went down that road.
There are millions of ways to try and combat this, unit testing, very granular design (with unit testing) and so on, but still you will always leave out something. Some circumstance your client will undoubtedly trigger, perhaps more often than any other state, the deadly sequence of events or data items and you’ll have a mess on your hands at some point.

A good, and arguably more intelligent, friend of mine sums up the scenario in the cute yet true phrase that “users make awful robots.” If they’d only do the same things with your app that you’d written tests for, there would never be any issues. Of course, that’s never going to be the case, and it’s so painfully obvious to anyone that does this on a regular basis, that I might not even say that… but I had to learn this lesson… hard, so I believe it deserves mention.
Before I get into it here, and start making quick enemies, I’m not saying that you will let some bugs “live forever,” just that you may need to leave some at the time of relase.

So, when do we fix it before release, and when do we keep it on the ‘good list’ of bugs at launch time? I can’t say that I have an easy answer for you, but there are some basic points that I like to keep in mind.

How big of an impact on the apps overall function is this bug?

If your bug doesn’t really do anything bad other than annoy you, then it may be one to keep and study for a service release. If your product produces some official printout as a deliverable to a government agency (which in my case was exactly what the product does) you can’t leave anything affecting the printouts in there or you might wind up in court, or send your customer up the creek, but … There are times when a bug that doesn’t officially screw anything up still warrants fixing before release. Logging for example, or error reporting. Those types of bugs will throw off your ability to fix others down the line… I think it’s pretty clear that I can’t go into a list of yes’s and no’s on this, but you get the idea. My general rule of thumb is, does this affect the ultimate function of my application, and does this obscure my view of what else is happening in my application? If the answer is no, then you can possibly live with it.

How easy is it to trigger the bug

This one is a little more complicated than it sounds… Each client may indeed use your code for slightly, or entirely different things than each other, so if the bug is common to a certain category of clients, and not to others, then you might want to give a little less weight to the ‘easy to trigger’ factor. It need not necessarily be a true average. (Math and debugging don’t always mesh, in my experience. Though many have tried to make a mathematical model to describe everything from development time to phase iteration, sometimes you just have to squeeze a bit of reality over your numbers.) If the use of your code though is fairly universal, and the feature that’s affected, or the exact circumstances required to trigger it are pretty obscure, (I mean in real use, not simply on paper) then you can probably put this one off.

Is this going to impact existing data

One thing that I find myself running into is a change that doesn’t really break anything properly, but will leave previous versions of the data created with the old code either in an arbitrary or undertermined state or leave behind extraneous bits altogether. For one example, one thing to consider is “Data Determinism”. Does your code use the data itself to determine how to store the incoming information (or modify existing information), or is the method of input what determines it’s persistent representation? If you choose the former, you’ll have to account for regressively changing your old data to reflect the new structure, if the latter, you might be ok, but still have to check very thoroughly…

The ultimate question

Does this make people not want to use my product over an available alternative? If you happen to be microsoft, you probably don’t have to think much about this one, but for the rest of us, this is probably the most critical thing to consider. You only get good answers for this one from doing careful beta testing. Alpha testing will of course leave you with a bunch of ‘of course not, this is our great product’ but the beta people, by definition, won’t be employed by you and will tell you if they are turned off enough by whatever the bug happens to be. You may want to construct this sort of question directly for your beta testers. Give them a list of popular, or similar alternatives, and try to ask something that seems heartfelt and honesty inspiring like ‘Considering your experiences with our product, which of the following products are you most likely to choose: Alternative B, Alternative A, Our Product’ If you get people NOT choosing yours, call them back a second time and ask why. Putting it on the same survey may make them not want to answer truthfully in order to avoid having to fill out a ‘please explain box.’ (I’m guilty of that myself…)

Anyway, I know I haven’t given a lot of instruction, but at least I hope that I have illustrated why sometimes it might not be worth your while to try and perfect a product before it goes out the door. Especially if you’re writing something groundbreaking and novel, and possibly a competitor is working on it at the same time… You don’t want to let your perfectionism to allow your competition to become a ‘de facto’ standard, because getting someone to invest in your product after they’ve already shelled out for a similar solution is worse than pulling healthy teeth — even if yours really does happen to be better. (For example apple vs microsoft…)