Josh Ledgard from Microsoft posted an extensive rebuttal to my Beta Perversion entry from a while back.  Josh brings up some interesting points from the opposite end. But overall I have to say that he's just not 'getting it'!

There really two issues that the discussion boils down to:

• Previewing the software
  
• And beta testing

Previewing the software is quite different than the actual beta process which is designed to help iron out problems and inconsistencies. While there's some similarity the difference is usually in the amount of involvement by the beta program provider - ie. Microsoft.

My contention is that Microsoft is doing a lousy job interacting with people here. While Previews are clearly not production quality and are meant for previewing and experimenting, with a product the size of VS.NET I find it almost too much to sit down figure out how something works without documentation, without support and somebody to ask that's involved. Then find out that the code and concepts we spend hours/days/weeks figuring out have been pulled or changed so it's completely different in the next build.

This is where a well laid out beta process helps tons because there's a sort of community around the new stuff. Not only people struggling the same as you but the people who are building the stuff and have the insight that you often need in order to provide decent feedback. I can bitch about how something doesn't do what I need it do, but not really understand how this is supposed to work.

Maybe I'm not as adventurous as I used to be . I like to play with new technology and to figure things out. But I hate to do it when I can almost feel the technology flowing out from under me as changes are made. In my experience betas have been products that you can start being productive with and along with some comfortability in the knowledge that things won't completely get turned upside down on you. How can you realistically test anything if a new build comes along a few weeks later that breaks everything?

I understand the need to get good feedback up front, but in a lot of cases architects and engineers need to make those choices. Too much feedback is a killer too. Look at what's happening with the ASP.NET CodeBeside model which is turning into a total mess because everybody is asking for something different to the point of all but making the old model (CodeBehind) unworkable. This stuff has changed over and over again and every CTP is significantly different.

This is the first time I've heard someone say it's a bad thing that we are finally giving people a chance to play with a product early enough to effect the end result. I would also disagree with the statement that we "don't really handle the process of bug fixing based on customer feedback". The statistics from the early betaplace and Product feedback centers tell a different story. On the feedback center alone almost 6000 bugs from customers have been filed against us and these bugs are being fixed at the same rate with similar numbers to bugs that are filled internally. Customers now have a way to raise bugs against our pre-release software with the same voice as the internal developers in testers. I'll go as far as to say that customers are getting better responses than internal testers on the issues they raise... and that's a good thing. Also, 6000 bugs doesn't generate as much noise internally as you might imagine.

I'm not suggesting that you're not getting bug reports, but I am suggesting you'd get a whole lot more and possibly more detailed and insightful ones with better two-way interaction. You'd learn about customer usage scenarios that might otherwise be lost. Beta Place product feedback is fine, but it's not a feedback loop. It goes in and is gone as far as we on the outside are concerned.

Heck if you've gotten 6000 bugs (over what period of time - since beta started? Since the alphas), that seems tiny compared to the amount of people using the betas. THAT should tell you something too...

I'll disagree here. The foxpro model works because of the limited scope of the product, the customer base, and small internal team size. Yes, the Foxpro team all have their hearts in the right location (to make customers feel connected to the process), but some of the methodologies don't scale well outside of the small focused product and team.

Well, I guess we disagree. While the scope is definitely different once you break out the process into groups that are responsible for each part of the product the same model can apply. It's just that instead of one single beta you have a whole bunch of sub betas. This is how this handled at MS anyway (or appears to anyway).

Why does this matter? Ok, I'll give an example. I've built a number of add-ins in VS.2003. The process has been a nightmare with tons of wasted time on inconsistencies and a model that is not well documented, buggy and well, very inconsistent across the tool in .NET. As you might expect I have plenty of feedback I could give you. But there's nowhere to go to actually provide this feedback. If I post this in a public newsgroup it surely will not do anything. If I post to product feedback it will be too vague because it's not a bug report. You see there's no place for 'ideas' beyond bugs. This is part of a beta (or maybe even earlier). Sometimes this process might not even involve using the product in question... Sure I can probably ask some of my contacts inside of Microsoft to put me in touch with somebody on that team. Weeks will go by until this happens though - by that time I will barely remember my point ... the point is, this should be easier and not require backroads and knowing the right person.

The product feedback center has some issues that, if addressed, could make it 100x better. If the firefox team can handle their bug influx from 10 million customers in bugzilla, then I imagine we'll be able to scale to our Visual Studio developers with the right tool. I make no claims that the Feedback Center is perfect today or that it will be tomorrow. Only that we will be dedicated to improving the process of handling customer bug submissions. While I can't give exact numbers... trust me, the number of customer bugs submitted is not anywhere near the number of bugs that are submitted internally. We will be able to keep up for a long time.

That's great, but not really what I'm talking about. Sheer numbers is one thing (an important one) but the process of how bugs are handled. All I can tell you is if you've never worked on a beta where you have two-way communication of bugs you can't really appreciate the importance of this process especially for tough technical issues that usually can't be bugged properly in a one-way setting.

I'm not sure which build of the product you are using, but if we simply only labeled the later releases as a "beta" it seems that a lot of your concerns would go away. In the end it really sounds like your concerns are ones that could primarily be addressed with proper messaging rather than serious changes to how we are opening up the process earlier in the release cycle.

I've installed many builds along the way. Which is one problem . Two hour installs, plus first clearing the last install is not exactly to be taken lightly (well, Ok, you can use VPC with a clean image which makes it a little easier).

But most of it about the test/bug reporting process. I guess there's not much that can be done about the hype. I just wish the hype would hold off until we're somewhat closer to release. The first wave of hype (starting prior to PDC) will be more than than 2 and a half years removed by the time Whidbey ships.