VFP String Concatenation Issue in Loops

I got bit again by a nasty string concatenation problem in VFP. This isn't the first time I've hit this particular issue and I've known about it for years, but it's one that's easy to miss and so from time to time I hit it again. The problem occurs when you have string concatenation code that involves an expression that returns results from a function and where the function often returns a large amount of text.

For example, the following code - taken from Web Connection's wwWebRepeater control - fails:

   LOCAL lnX, lnCount

   lnCount = this.ChildControls.Count

   FOR lnX = 1 TO lnCount

       LOCAL loControl

       loControl = this.ChildControls.aItems(lnX,2)

       IF !ISNULL(loControl)

          lcOutput = lcOutput + loControl.Render()

       ENDIF

The code fails to properly build your concatenated string. The code in question is string concatenation where the code is appending to an existing string by appending an expression - typically a class method that is called to provide additional data. In each iteration of the loop lcOutput is set with the result of loControl.Render() rather than the concatenation of lcOutput + loControl.Render().

This makes for some interesting debugging - you step over the code with no problems but it takes a while to actually grok what the debugger is telling you - it's not the debugger truncating your string but VFP actually failing to concatenate.

This is an insidious bug in that sometimes the code works as expected and at other times it fails with creating the incorrect string format. The only way to really detect this problem is to step through with the debugger and see that the result is incorrect.

Luckily the workaround for this is fairly easy: you can create a variable and assign that to the concatenation expression. The following code produces the expected result:

   LOCAL lnX, lnCount, lcT

   lnCount = this.ChildControls.Count

   FOR lnX = 1 TO lnCount

       LOCAL loControl

       loControl = this.ChildControls.aItems(lnX,2)

       IF !ISNULL(loControl)

          lcT = loControl.Render()

          lcOutput = lcOutput + lcT

       ENDIF

   ENDFOR

This fixes the issue, but of course it results in the string being copied to a variable first, which depending on how much output is generated may be an expensive operation.

In Web Connection I had to use code like this in a lot of places and just last week I ran into this nasty 'bug' again while working with a client in a place where I missed the issue. Oddly, the problem occurred in a fairly high traffic area of the framework, so I'm surprised that nobody has actually hit this particular issue before. It's possible the behavior varies slightly with different versions of VFP 9 (ie. RTM, SP1, SP2) - I believe some of these behaviors were actually addressed in SP1 and later.

This 'behavior' was introduced with the string optimizations in VFP 7 which drastically improved string concatenation performance in VFP back then. When concatenating strings VFP will usually allocate more memory than it needs to concatenate the current string value to optimize string allocation. So rather than having to copy strings all the time for every concatenation, it allocates a bigger buffer and loads the content of the right had side of the concatenation directly into the end of the string buffer which improves string performance drastically.

I can only guess on why this doesn't always work with functions but my guess is that if the buffer size exceeds the pre allocated buffer that the string gets recreated and somewhere in that process the new string pointer for the result, rather than the concatenation is returned.

No matter what the problem is, it's a bug that you should be aware of since it's likely to be hit by a wide variety of code.