Microsoft with the Visual Studio 2005 Beta has really perverted the beta process as we know it into something that has little to do with what used to pass as traditional beta testing. In the past, beta testing was the idea of final testing of a product by end users (who in this case are developers) – usually a selective sample of the end users along with people who have been selected to provide quality feedback and have a deep understanding of the tool. By the time a beta rolled around the product generally was pretty much feature complete and in a fairly stable state of operation, and approaching the release cycle.

With Visual Studio .NET 2005 and the whole Whidbey wave (and even previously with the release of .NET 1.0 and VS.NET 2000)  Microsoft has turned the beta process into little more than a public marketing program. Microsoft’s beta/marketing process starts as soon as a product ships and immediately pushes the next technology coming. I call this perpetual beta: The beta of the currently released product stops and immediately you go into the alpha/beta/marketing of the next product version in the cycle. Immediately there’s a perception that the current, just released technology is obsolete and lacking in features or having other shortcomings that surely will be addressed by the forthcoming version.

When the new technology is first announced it looks like it’ll be usually 1 year to a year and half away. Of course that never actually happens. It’ll end up being closer to 2-3 years before the product sees RTM, that people are potentially holding off development in anticipation of the new release and its ‘vastly improved’ features.

What makes things worse is that the computer press will pick this up and immediately flood technical articles with this new and exciting technology with ultimately worthless and premature articles that will be obsolete a few months later. It’s a self-fulfilling prophecy cycle for the MS marketing folks and everybody who is ever so closely involved with that wave. Never is there really time to absorb the current technology fully, expand and get a solid platform. Instead the platform is in constant flux. While Whidbey certainly brings many very useful enhancements to the .NET framework, a lot of those changes will have significant impact on how we build applications. For those that have built up frameworks and libraries that will mean a significant amount of reworking to take advantage of the new improved feature set.

Look at Whidbey and it sure bears this out. You can look at this ‘preview’ process as a good thing or a bad thing. The good thing about it is that you do get to play with the technology early on so you can get a feel of what it will do and can start planning for it. The bad news is that it gives a false impression of things to come that make the current development landscape unstable. I work with a variety of customers and just about everywhere I go I hear the question: “Should we build this application with the current version or start building now with the current beta release?” A number of customers have actually gone as far as building complete systems on the new technologies even though they at the time were not even in beta. Sure enough all this broke when the actual beta release came out (fixable, but still a major chunk of time was required to get it there).

The problem is that if a new technology is available, no matter how many labels of warning are put onto it about it not being production worthy, some people will accept this code base as good enough and build with it anyway. For the rest of typical developers who don’t follow the very latest of the Microsoft party line they do get the highlights of this or that beta release and it gives an impression that the final release is not that far off. I in many cases this influences decisions being made on development with current tools. In addition new technology that gets pushed out too soon has the annoying side effect of pointing out weaknesses in the existing platform which only exaggerates this problem of ‘insecurity’ about building an application with the current tools.

All of this is the typical FUDD effect. Fear and confusion over what to learn and what to use. The next version will fix all of your concerns you might have about the shortcomings of the current version. And imagine that there will be another fairly vast learning curve to absorbing all the new stuff that is coming and changes the way applications get developed. This coming wave may not change the fundamentals drastically but there are SO MANY new things coming it’s going to take a long time to absorb and figure out how to integrate all this stuff into existing and even new applications.

And all the features - if it’s not in this beta it’ll surely be in the next. In the meantime a lot projects in the real world sit on hold, waiting out the cycle. Many in this community have wondered why there’s been a relatively slow uptake rate for .NET. You can bet that this effect has a fair influence on this behavior.

The Beta Process

As I see it there are number of serious flaws in the beta process itself as well. Beta is now a year long or longer process of a product that is still in heavy flux. This means that the purpose of beta – to test a fairly complete version of a product – is not really being served at all. Instead Microsoft uses the beta to see what features need to be added, removed or modified due to public consensus while not really handling the process of bug fixing based on customer feedback. If you look at Whidbey Beta 1 and the follow on Refresh releases you can see this happening.

Public input, or lack therof, is another odd by-product of this funky ‘beta’ process. A while back when the public beta was released Microsoft actually closed the ‘official’ beta newsgroups and opened up public general access newsgroups for this process. While the public groups are a good thing to allow the greater public access to information Microsoft also stopped taking responsibility for anything posted there. In fact, Microsoft never really has done even a remotely decent job of dealing with feedback that has been posted on the beta newsgroups and message boards. If you’re very lucky somebody from the appropriate team would look at the posts and comment. But even that minimal action was rare and official style discussions of a feature/bug problem were extremely rare. In the public newsgroups this process has almost completely disappeared.

Making a beta public and then removing direct beta involvement for ‘official’ beta testers is also a bad idea. While a public beta provides potentially many more testing sites, it also increases the signal to noise ratio dramatically with unqualified feedback or feedback that is so obvious or previously handled that it has no bearing. While it sounds a little elitist there is a reason that some developers get invited to be beta testers – usually testers are chosen from a variety of groups and for their involvement and previous efforts that have proven to provide useful feedback. A public beta can be useful at the end of a beta cycle for final testing and feedback but this early on it’s not something that should be called a beta.

This is not to say that the ‘general’ public couldn’t have good feedback, but adding a public group makes the sample group so big that it will become impossible for Microsoft to actually deal with all incoming requests in anything but an automated and non-personal fashion, which is yet another problem with Microsoft’s beta testing for VS.NET and .NET 2.0.

Microsoft now states that much of their input from community comes from BLOGs and commentary and related posts on community boards etc. While that’s well and good, it’s a very scattered approach to solicit feedback and one that doesn’t lend itself at all to consistent input from those involved in the beta.

I’ve been part of many betas in my time and I can tell you that a beta process without feedback is a waste of my time. Of course for Microsoft it’s a bonus as they don’t have to do anything. But is it really a bonus? Feedback provides a better understanding of user scenarios, which as we all know Microsoft developers are not always on top of. There are too many features that are cool on the surface, but lack critical features or lack a good front end to take advantage of them without writing reams of code that could be handled much easier. Much of Whidbey’s functionality actually is based on this simple fact. It takes customer feedback to drive this however.

For a model of what a beta process should look like the VS.NET teams should review how the Visual FoxPro team runs its betas. VFP is handling most bug reports directly through the newsgroups rather than a ‘bug reporting’ system that sucks. Ask anybody who was involved in the VFP betas about how they feel about their involvement and you will get an answer that says that the process made them feel as part of the process and being able to contribute meaningful feedback. In this context meaningful feedback often means a dialog and backup questions on submitted bugs. Nothing of the sort happens with the VS.NET beta. Bugs are supposed to be logged into a bug reporting system and that’s that. Never to be heard of again. You can’t check on status and worse other people who might have something to contribute or even voice an opinion on the bug or suggestion never know about it. This way the beta process becomes a decidedly non-community process that serves neither the people involved in the process nor Microsoft.

While VS.NET is vastly more complex than Visual FoxPro, the VFP beta team that deals with the newsgroups is also very small but extremely efficient. The process involves picking up the reports and checking them out, following up if necessary and returning bug ids ones confirmed. It’s not uncommon for a rapport to go back and forth through 10 messages or more and that’s the way a bug report should work. Often times coming up with bug scenarios is not easy and having a public place to discuss these issues can spawn discussion by other members involved in addition to the dedicated testers at Microsoft. Maybe that’s just me, but that seems a logical way to run a beta. Yes, it takes some resources, but these resources can and should involve the members of the various teams. It’s their product and they should be aware directly of the issues that are being encountered by the target group who will be using the product. For all the talk of ‘community involvement’ of the various teams this is one place where there is very little of it. I for one think this is where community involvement has real value for both customers and Microsoft for encouraging a feeling of partnership in this process.

This process doesn’t have to be exactly like the VFP beta either. For example, I could think of a Web interface that would allow posting bugs in the format MS requires, but also allowing comments and follow up on these bugs in an interactive environment that’s viewable by everybody involved in the beta. Or better yet, post those reports into an appropriate newsgroup or Web board for discussion. How hard could that be and it would be a huge step forward in providing the necessary feedback loop? Just seeing what bugs and comments get reported can be enormously beneficial in understanding (for other developers) how the product is being used and I can’t tell you how many times I’ve learned heaps from following a bug report of another developer all the way through. This system must be easy to get to and at least as easy to use as a newsreader or Web Message Board (heck you can even use a Web Message Board for this entire situation, simply by reposting bug reports as a message).

Of course all of this would require a bigger time commitment from Microsoft developers or at least the beta leads to moderate and interface with the community and that may very well be the problem.

This is a beta?

I like many others am fairly excited about the features that Whidbey will bring. But I’ve also been pretty disappointed with the quality of the current beta. In no way would I call this release a beta. It’s buggy, slow, and uses ridiculous amounts of memory with a number of core features not working at all. While I expected that (given preview releases) it’s not something that should be called a beta. The quality of the beta is not much above the preview releases that preceeded it. Although Microsoft goes through a much more rigorous testing process for full beta releases as opposed to the community previews, but even so these betas are lacking significantly in terms of stability and in the case of VS.NET usability.

The problem is that VS.NET and the .NET framework is such a huge system and there are 100’s of teams involved. Trying to coordinate all of this is a monumental process. Visual Studio especially is.

It’s kind of ironic how Microsoft is advocating distributed, atomic systems, yet VS.NET of all things is the epidemy of a monolithic Windows application that is tightly coupled to every part of the operating system, IIS, SQL Server 2005, heck even the .NET version. If there’s one application that’s most likely to hose your system on install or uninstall it is Visual Studio!!!

Just think of the install that takes well over an hour to run locally just to configure your system. If anything breaks, heaven help us if you’re trying to fix it without rebuilding the machine. Something is wrong in this process. The cycle is so volatile because of all the interconnections between the tool, the products and systems. This is scary. I can just see trying to integrate systems a few years old and running different sets of services that aren’t the latest and greatest. That might be a lot of fun once all of these components get out of sync and no longer work together. It also makes being a beta tester difficult. Where do you start to report bugs? With all the things wrong you'd never get anything done if you start reporting everything...

I personally think that a beta should not be released until a product approaches stability. If feature sets are still moving around (as they are at this point) a release should not even be considered a beta. Recently Microsoft announced that beta 2 will now not happen until February or March 2005 – more than 6 months (at least) since the release of beta 1 and beta 2 will have significantly changed features. Whidbey is now scheduled for the end of 2005. Another year, when originally the end of this year was the ‘late’ target date.

I’m all for Microsoft taking their time getting it right and making sure the product is as good and stable as it can be. This is priority One. But I can’t agree with the policy of over-hyping the new technology so early on when it distracts developers so significantly with what at this point is little more than vaporware that's diverting valuable resources (time!) from developers who should be getting work done ...