Early Christmas this year!

Visual Studio 2008 drops this week as an RTM. I’m going to be checking the website about every 3 hours now. Whee!

C# Template for GoldParser Builder and the Morozov Engine

I decided to give a shot to creating a template for Gold Parser Builder that implements a C# instance of a parser making use of the Morozov Engine, which is known in the Gold Parser world as the fastest of the C# engines.

Thanks to Devin Cook for making this all possible by creating Gold Parser Builder, and also to Vladimir Morozov for his contribution of the great Gold engine for us C# geeks. (a few of the structure-like features of this template were.. uh.. borrowed from the Calitha template, but nothing really ‘codey’, just ’structurey’)

You can get the template here, in case you’re interested: http://davedolan.com/downloads/C-SharpMorozovNet2.pgt

Updated I forgot to add the definition of SymbolException and RuleException to the original download (it was correct in the version that went to the list, I just got a little happy cleaning up the script and … chopped two things too many. It is FIXED now.

I’ve also added a .Net 1.x compatible version of this template: http://davedolan.com/downloads/C-SharpMorozovNet1.pgt

Don’t even ask about .Net 3, and if you were thinking ‘what? why not?’ then perhaps you should not be making parsers, even with templates

Lock Free Coding Addendum

Ok so something strikes me… the only thing that makes sense in multi client lock free coding is to have a shared, un-orderd data structure between all process elements. This strikes me as a little odd because you have to ensure the thread safety of this shared structure… I’m going to have to read more on this, because I’m now throughly confused… Maybe lock free isn’t the best methodology for client-server paradigms (that don’t broadcast data to each client simultaneously, and must maintain open ‘indexed’ tcp connections) Sure you can safeguard a collection, like the hash structure of tcp handles… but… don’t you have to put them in critical regions first? This is what i get for learning this stuff in my “spare time”… you can see how often I get that.

Comment added After the Fact: I apologize for sounding like such an idiot on this subject, I’m just trying to get my head into the multi-core ready ideas, and I’m having a bit of a time at it… All of your emails and comments are very helpful. So Thanks, and I’m trying, I’m really not as dumb as my (lack of) conceptualization of Lock-Freeness would suggest.

Lock Free Coding

So I’ve been looking around. One of the things i’ve been messing with is trying to determine how to take advantage of an I/O Completion Port strategy for a client server program.

What’s an I/O Completion Port? Well, it’s basically a queue that fires an event when something is either added to it or comes out of it. The benefit of them is that you can have code wired up through delegates (assuming we’re talking about C#) to respond in some way when a ‘message’ comes into the queue. It can eliminate the need for blocking the thread, or coding an endless loop (though under the hood I guess most every program is some kind of endless loop.)

The benefit of this is I can treat my processors as a pool of thread runners, so for example, when an event fires, I can spawn another thread (provided I’m not at the limit yet) to go handle it, and it will execute in parallel on the other core while the one that captured the input is still, well… capturing input and handing off to other threads. Actually you can have several different kinds of ‘listeners’ hooked up to a completion port each doing something when a different conditional factor is met based on the thing that’s going into or out of the queue.

Problem… HOW do I get result messages back to clients after I’m done processing them? I can easily see how if your ‘result’ message is something that say, prints something to the console… But what if it’s a TCP/Socket where the other endpoint is in egypt and I somehow now have to phone home telling it that it’s ok to pick up the results… I can say, sure… the client can block, and the server can block a thread waiting on the output, but, then we have less of an advantage to being able to use the whole IOCP model to being with… So… I know I’m just missing something, and if anyone out there might have an idea of what that is let me know.

I’ve run into things like COmega, and the CCR, and even this nice little opensource gig on CodePlex, but in all of the examples, the data is like a Multiple input, single output spigot, (printing to the console or broadcasting a single message to multiple clients polling a shared memory queue) What am I missing?

In case I seem extraordinarily dense, Lets say I want to run a database server on the back end, and have my clients connect to an IOCP based command processor, which returns data to the caller. When each client issues a query, I want to get the data back that applies to the client, but having the client keep open a listener, seems odd, and having the client poll around all over ALSO seems odd. Asynch just doesn’t seem like it’s a good idea when we’re trying to get data back from the operation… So then, how do I take advantage of multi core machines if they’re never any better than lock block and stock systems?

I must be Slow… (DLR)

Ok, well I was so anxious to hear about the DLR, that I forgot to look around for articles on it. I missed this one [The One True Object (Part One)] by a month.

And this one [First DLR Talk On The Web - Live (pre-recorded!)].

And, wow, I was really sleeping folks, because this one [A Dynamic Runtime DLR] is even days older than those two.

I’m not just a link poster, honest. I do a lot of writing too, it’s just that I’m most definitely not an expert on this subject, even though I’m trying here. I’m so happy stuff is finally coming out, but I just don’t have all day to play like I wish I did, so I have to rely on blogisms (which I have to admit, coming straight from the library authors’ desk is great,) but I have a feeling they’re saving the best for a book or something. Oh yeah, and they’re probably coding. Go figure.

I’m not so sure that I would be able to work under those conditions. Picture yourself in this scenario: It’s like the world is watching with ticking stop watches, just waiting for you to stop reading that google, I mean… er… msn.. news article. “Check your email later, man, we’re waiting for the DLR with examples and doco!!!” Yeah. See?

So my hats off to Jim for taking the time to write about this. Whatever happened to that John Lam cat? Oh yeah, he’s a rock star now! (In all seriousness, I’d probably live in a hole if I were hired off to Microsoft on such a high profile project.)

In my own various attempts to write a ‘dynamic’ language of my own, I’ve always tried to do what the folks do in C++ land.. they make tagged unions or wrappers for data types and objects, and then have some sort of syntax tree for dealing with the cases of each… Even take a look at our friend TreeCC, which is, at least for it’s day, one of the most excellent tools for generating AST nodes en mass (though not necessarily for dynamic languages, but just anyway for argument) it even does the tagged union thing. So naturally I hadn’t even considered the possibility that building actual CLR types (a la the one true object) in .NET meta-data and IL is the way to go… I just figured I ought to write my own ‘runtime support’ which I figured must include a type system that marshals to the real type system. Hah. This is where I learned a valuable lesson. “Look at the source to IronPython.”

In particular, I’d take a look at the illustration in UserTypes.cs. The one comment that sticks out nice and blonde at a brunette convention:

“UserType represents the type of new-style Python classes (which can inherit from built-in types). ”

Who would have thought that some of the most interesting information in a comment would be the aside in parenthesis?

Programmers at the old folks home in 2065

“Back in my day… We typed in complete sentences… uphill both ways.”

The DLR for .NET?

A post in ZDNet points out that Microsoft is releasing a dynamic language layer intended to run on the CLR. With the acquisition of John Lam and Jim Hugunin, they pretty much have their own little market cornerd… perhaps this is why Mr Lam wasn’t releasing any more drops of RubyCLR of late…

Very cool, and I’ll be very excited to have a look at this!

My Quest for a Language

That’s it. I’ve about had it. I’ve been working with C# now for a long time, and there are many things I like about it. But the structure for structure’s sake, lack of easy to implement multiple dispatch, and other such nonsense is starting to eat me for breakfast.

It’s time for a change, or at least some more education and investigation. So… in the weeks of past I’ve been feverishly investigating new and exciting frontiers. I’ll admit this is more than just looking for a language to use, it’s and investigation of techniques and methodology as much as it is a search for the golden fleece of utilitarian bliss.

I’ve made a few stops of late, and I’m now going to describe what I’ve found. Before I delve right in, I want to make you aware of my initial biases: I’m not looking for something to interface with Relational Databases. I write code for Relavance knowledge bases now, and I don’t intend to crush my thoughts back into flat tabular form anytime soon. So I have a few avenues that are already open for me to get data to and from Relavance. One is a .NET API written by my friend and collegue Dylan Currier, of Levitronics in Canada called AbstractThought. Another is a COM interface to a set of VB6 DLL’s written by the Inventors and Masters of Relavance technology (we’ll call them RE and PT for now) and company. These are temporary limitations, but I’m working within them at the very… My ideal scenario is to implement another multi-user server head as a TCP/IP front for a virtual machine, using lock-free I/O completion ports to simulate prioritized multitasking. The Virtual machine is based on a stack machine design, so the nodes therein are to be of the dynamic variety. I will either implement some kind of Tagged Union structure to represent a stack node as a Variant of sorts, or do some fancy conversion algorithms to get things hopping with multiple types of nodes. The parser, well that’s the easy part, I’m going to write the grammar for Gold Parser Builder, and have the parser emit instructions for my VM. I plan to use either the Calitha Engine or the Morozov engine as the parser implementation. My old implementation of this language actually hard compiled the code to MSIL assemblies, but I want to be able to run it in lower security environments, (read Commercial Hosting,) as well as avail TCP/IP raw (instead of .NET remoting as it stands at the moment.) And lets face it, a query language should not have to build DLLS for every operation That’s just silly.

First Stop: Ruby. I can’t get too much into the details here because I’m saving my material for an expose on Ruby for C# developers that I’m publishing for DevX.com. I’ll post a link when it comes up. Suffice it to say that Ruby has many nifty features such as dynamism and continuations as well as a simplified syntax, but COM interop is at best difficult, and at worst, slower than molases. And it’s not quite .NET ready yet because Mr. John Lam is a VERY busy employee of Microsoft now, and understandably doesn’t get as much time to pump this sucker out at the moment as he once did.

Second Stop: Forth. Forth is very nice for implementing a stack machine because, well, it is a stack machine, and it’s syntax is stack oriented. The only problem I have with it is that I’d have to implement my own embeddable version of it in .NET to allow for my extensions, so I just decided to keep looking for the moment. I haven’t ruled it out, but I can’t say I’m leaning toward it. Simple translation at the parser layer would be efficient in that step, but a stack over a stack over a stack machine seems to be pushing the envelope a little in the terms of runtime efficiency. I do admit that I haven’t actually run a performance test to justify my thinking on this one.

The third stop: Well this is sort of more than a stop, it’s a miniature journey of it’s own… Functional Languages. I’ve taken the time to learn the basics of LISP and SML (thanks LtU!) and even had a small crack at Scheme. These languages are beautiful to be sure, and elegant beyond words, but being so, they are difficult to make an imperative run on top of, at least with my current level of experience with them. I have to say that I have very much respect for the folks who make these their bread and butter, and indeed I can learn a lot from their examples. Unfortunately I am an aging unix hacker turned .NET coder, and my quickly aging C (not ++) rooted skills aren’t sharp enough to absorb all that I need to in this realm in time to put together something useful, not to mention efficient in a Functional language. I will say that with the combination of SML.net and LSharp, there are two very good embeddable options for interoperating with the .NET API AbstractThought.

So, then, what about languages LIKE c#?

Stop 4 was a very close call, and I almost stopped here for good: Nemerle. It’s a META-programming extension to C#, and apart from some slick built-in shortcuts and macros (compile-time macros are one of the key features of the language!) it pretty much is C# masked by a few nicer syntacticly sweet constructs. Still a tiny bit rigid for what I want, as in want to be able to use yesterday.

So then I found it… The damned near holy grail for a frustrated C# coder such as myself… Boo. No, I’m not trying to scare you (Yeah, see how funny I am? That was hilarious.) it’s the name of the language. It’s a very nice hybrid of C# and Python (what?) It’s much like C# in syntax, if you could take out 80% of the non-code jibberish that you have to write in C#, but it’s also SO much more. It has closures (which don’t have to be confusing like anonymous methods in Whidbey) and the real kickers: Optional Duck Typing and Native support for Multiple Dispatch. I can’t say for certain that I’ll be here forever, or even a very long time, but the wonderful syntactic terseness that’s inspired by Python, and the type conventions of C# rolled into an easy to read language make this sucker a promising candidate. The language is an allstar cast of my favorite features of Python (short syntax for classes, optionally based on whitespace-as-blocks) the ability to declare Modules (a common feature of modern languages of late fame to aggregate static utility-ish functions into a structure that make sense on it’s own without having instances.) Free floating functions which can be used as objects (like functors, but even better - Closures), and what appears to the common observer to be LISP-Like List contruction (actually a masked IList structures like an array list, but the native syntactical support and conventions encourage the use in a more sensical and natural way than simply instantiating an ArrayList and casting it to IList). Last, but not least among the spectacular feature set is the very active development and improvement of the language which is quickly bringing it in line with other fully Whidbey-savvy languages through the incorporation of Generics. The list moves several times a day, and the bugfix discovery to resolution rate of late is very good. Possibly best of all, it works on Mono and is natively supported in SharpDevelop, which makes hacking it very user friendly for all of you superior operating systems fans.

So for now… Boo is Best, and I’m working through the conversion of some of my more murky code base to it. I’m testing my chops out by implementing a manual DAL over relavance for object persistence in a particular application, but once I get this off the ground in earnest, I’m going to implement the virtual machine a-here.

On a side note: investigation for a query language is a tough business, there are a lot of things to consider, or at least there were until I found SQLite.org. I of course don’t really want to implement SQL proper becasue it doesn’t lend itself very nicely to associative methodology, but I do like having most of the idioms I’ll be using exemplified, not the least of which being how to use a virtual machine to query a knowledge store.

On an even Sider note: REBOL was recommended to me by one of my unwitting mentors and it has a lot of potential. I will need a little time to get my head around the idea of using a simple syntax language that pretty much does everything under the sun. Of particular note, the guy who invented REBOL also happens to be the genius behind the magical simplicity of yore, the Amiga.

The Myth that “Features” must be used…

I’ve been doing a lot of research reading and experimentaion lately in the field of compiler writing and language theory, and I keep running across a lot of miscellaneous blog entries or responding comments that make assertions somewhere on the order of “But since we’re using c# we should be using properties for everything that can possibly have them, because otherwise, what is c# for?” In essense they’re suggesting that “since you can use properties, then you should.” I don’t get it… other than default applicability in databinding and XmlSerialization (sometimes these things aren’t needed, believe it or not!) In my estimation, properties, when you don’t have access to the source code can be ambiguous as to whether or not they are just setters or getters… And Fields, well if you aren’t executing code on them, and aren’t firing events when they are modified, then it’s perfectly acceptable to use just plain old public fields! Call me a heretic or something… but you can always change a field to a property later, and without taking away the usability of it.

There are very good reasons for using properties though, like implementing interfaces, but lets not forget that methods can go in interfaces too! So what’s wrong with having a method called get_Name() and another set_Name()? (Ignoring the fact that properties are effectively methods anyway,) I think that’s perfectly clear what they do in that case, and the only possible drawback is the fact that they don’t get recognized as first class units of state by the XmlSerializer and databinders. I don’t think it’s very likely that my Lexer is going to need to be serialized, and the same goes for my Compiler class…

There is a time and a place for every tool, and indeed the features of that tool, including c#’s properties, but they aren’t required just because they exist, lets not be ridiculous.

This brings me round to another point I like to make… it’s not correct for someone to assert users of any one particular programming language are of any higher or lesser degree ‘real programmers.’ I like C the best myself, but C# seems to let me do a lot of what I need, and in a windows environment, I can get around with it pretty nicely. I would also posit to you the reader that even VB6 can be used as a good tool, and even by REAL programmers. Computer Scientists who are actually devotees to the field know that once you learn your first 5 or 6 languages, the rest are easy! That being said, since you know six or seven languages, and most of the time more than that, as a computer scientist or a programmer, it’s up to you to pick the language that suits your familiarity and the environment you’re coding for, in the best way. It’s wholly possible to intermingle some C# classes with some vb.net for a corporate website project… but do you think being stubborn in the use of C# over VB.net is actually producing anything of value, especially when the maintainer of the overall project two years from know will be primarily comparing your code to that of 4 or 5 other VB.net programmers who started on the project before you got there? There will be a definite gear switch involved in even trying to debug the code you’re writing. I’m not a purist advocate of any one language, and I think it’s ridiculous to hear (and I have heard) for example a Java programmer tell me that they would not use parameterized queries against the database because they were ‘Java Programmers not SQL programmers.’
Perhaps I’m biased on this issue, because I love everything about coding. I like the syntax, I like learning new, old, archaic, and esoteric languages, and I even like writing them. I very much enjoy .net on windows, as well as some lower level C hacking (and even nasm — but only once) on a linux box. Scripting? Sure! Everything from ksh to vbscript! It’s all in the name of being resourceful and applying tools from whatever toolbox you find most useful in solving a particular problem. A programmer is someone who engineers a solution to a problem that adequately makes use of the available resources, one of which is TIME, and I’m not just talking about execution time. Development time is the most expensive bit of writing code, especially when using truely arbitrary conventions. (This is NOT a reference to dependency injection which actually can be quite productive as a methodology. I’m merely jabbing at those who like to say “I’m a(n) X language developer not a Y language kiddie!”)

Wow, I rant a lot. Sorry about that. (I’m only saying that to be polite.)

Let me finish this up with something constructive and point to the site from which I have grabbed the most useful information on programming languages:

http://lambda-the-ultimate.org/

It’s affectionately known by many as LtU. So far, for me, it really is the ultimate.

EDIT: Ok I thought of one more good reason where you would want to make sure that your classes use properties and not fields… If you intend to allow derived classes to override your top level behavior. But, for sealed or Static classes though … you get the idea

By the way…

I just wanted to make it known that my home grown query language was called OPQL (object persistence query language) BEFORE JPQL (java persistence Query Language) was known. So there. I claim the naming originality, despite the apparent similarity to the nomenclature of a feature of java’s new iteration. Phew, I feel better now.