How to work around Visual FoxPro's 16 Megabyte String Limit

If you take a look at the Visual FoxPro documentation and the System Limits you find that FoxPro's string length limit is somewhere around ~16mb. The following is from the FoxPro Documentation:

Maximum # of characters per character string or memory variable: 16,777,184

Now 16mb seems like a lot of data, but in certain environments like Web applications it's not uncommon to send or receive data larger than 16 megs. In fact, last week I got a message from a user of our Client Tools lamenting the fact that the HTTP Upload functionality does not allow for uploads larger than 16 megs. One of his applications is trying to occasionally upload rather huge files to a server using our wwHttp class. At the time I did not have a good solution for him due to the 16meg limit.

What does the 16mb Limit really mean?

The FoxPro documentation actually is in fact not quite accurate! You definitely can get strings much larger than 16megs into FoxPro. For example you can load up a huge file like the Office 2010 download from MSDN like this:

lcFile = FILETOSTR("e:\downloads\en_office_professional_plus_2010_x86_515486.exe")
? LEN(lcFile)  && 681mb

The size of this string: 681,876,016 bytes or 681megs! Ok that's a little extreme ?? Just because you can doesn't mean you should!

But to my surprise that worked just fine; you can load up a really huge string in VFP if necessary. But... when you get over 16mb the behavior of strings changes and you can't do all the things you normally do with strings.

Other operations do not work for example, the following which creates an 18mb string fails:

lcString = REPLICATE("1234567890",1800000)

with "String is too long to fit".

However the following which creates a 25mb string does work:

lcString = REPLICATE("1234567890",1500000) && < 16 megs
lcString = lcString + REPLICATE("1234567890",1000000)
? LEN(lcString)  && 25,000,000

The following is almost identical except it copies the longer string to another string which does not work:

lcString = REPLICATE("1234567890",1500000) 
lcNewString = lcString + REPLICATE("1234567890",1000000)

And that my friends is the real sticking point with large strings:

You can create strings larger than 16 megs, but once they get bigger than 16megs you can no longer assign the >16mb string to a new variable or operation that creates a new variable. Any operation that requires the string to be copied (even internally in FoxPro's engine) will not work.

That might sound easy to avoid, but it's actually tough to do because the vast majority of FoxPro string operations mutate the string and produce a new string. If you pass string to mutable methods it's very likely that they are actually copied into temporary variables or added to another variable in a simulated buffer, and that is typically where large strings fail.

So what can we learn from this:

Doesn't Work:

• Assigning a 16mb+ FoxPro string to another string fails
• FoxPro function like REPLICATE(), STRTRAN() can't work on strings larger than 16mb

Works:

• Assigning a 16m+ directly to a variable works ie. lcOutput = FileToStr(lcLargeFile)
• Adding a <16mb string to a 16mb+ string works - ie. lcOutput = lcOutput + " more text"

The trick that can make let you build larger than 16mb strings is to use a fixed variable or property and keep incrementing it with strings that are smaller than 16mb, but don't do anything else with it. If you need to perform operational code on the string try to do it before you add it to the large string.

This may not always be possible. Sometimes you need to do something to the whole string, but in many cases it's possible to do things like trimming the string pieces before, replacing text in the segments rather than in the whole string.

There are limitations, but knowing that if you work with a single string instance that can grow allows for many use cases that you might have thought previously impossible due to the string size limit.

Caveat Emptor

Just because you can, doesn't mean you should! Creating strings larger than 16mb can create severe memory pressure in FoxPro and that makes it much more likely that FoxPro will crash with out of memory errors. FoxPro is notoriously greedy grabbing memory and not releasing it for a long time until Windows gets feisty about it. If you can keep large memory usage to a single method and release in the method as the memory then never gets moved to the heap. If you do use big strings make very sure you release them as quickly as possible or write them to a file. Always explicitly null out the values as soon as possible to mark the memory available again for FoxPro to use and then FoxPro to give back to Windows.

Real World Scenario

This actually worked well for me in the wwHttp class and the POST issue for larger than 16meg files but not string. Internally wwHttp uses a .cPostBuffer property to hold the POST data. The failure was occurring in the send code which would copy the string to a temporary string and get the size, then pass that to the WinInet APIs. The fix for this was fairly easy: Rather than creating the temporary variables (which were redundant anyway) I simply used the class property directly throughout the code without any hand off and voila, now wwHttp supports POSTs for greater than 16 megs.

The code I use is kinda ugly because it's doing lots of string concatenation to build up the Post buffer. Something along these lines like this excerpt from wwHttp::AddPostKey:

************************************************************************
* wwHTTP :: AddPostKey
*********************************
***  Function: Adds POST variables to the HTTP request
***    Assume: depends on nHTTPPostMode setting
***      Pass: 
***    Return:
************************************************************************
FUNCTION AddPostKey(tcKey, tcValue, llFileName)
LOCAL lcOldAlias
tcKey=IIF(VARTYPE(tcKey)="C",tcKey,"")
tcValue=IIF(VARTYPE(tcValue)="C",tcValue,"")


IF tcKey="RESET" OR PCOUNT() = 0
   THIS.cPostBuffer = ""
   RETURN
ENDIF

*** If we post a raw buffer swap parms
IF PCOUNT() < 2
   tcValue = tcKey
   tcKey = ""
ENDIF

IF !EMPTY(tcKey)
   DO CASE
    *** Url Encoded
    CASE THIS.nhttppostmode = 1         
         THIS.cPostBuffer = this.cPostBuffer + IIF(!EMPTY(this.cPostBuffer),"&","") + ;
                            tcKey +"="+ URLEncode(tcValue) 
      *** Multi-part formvars and file
    CASE this.nHttpPostMode = 2
      *** Check for File Flag -  HTTP File Upload - Second parm is filename
      IF llFileName
           THIS.cPostBuffer = THIS.cPostBuffer + "--" + MULTIPART_BOUNDARY + CRLF + ;
            [Content-Disposition: form-data; name="]+tcKey+["; filename="] + JUSTFNAME(tcValue) + ["]+CRLF+CRLF
         this.cPostBuffer = this.cPostBuffer + FILETOSTR(FULLPATH(tcValue))
         this.cPostBuffer = this.cPostBuffer + CRLF
      ELSE
           this.cPostBuffer = this.cPostBuffer +"--" + MULTIPART_BOUNDARY + CRLF + ;
            [Content-Disposition: form-data; name="]+tcKey+["]+CRLF+CRLF
         this.cPostBuffer = this.cPostBuffer + tcValue
      ENDIF
   ENDCASE
ELSE
   *** If there's no Key post the raw buffer
   this.cPostBuffer = this.cPostBuffer +tcValue
ENDIF

ENDFUNC

AddPostKey() can accept either a string value or a filename to load from. The file loading works by accepting the filename and then directly loading the file from within the function:

this.cPostBuffer = this.cPostBuffer + FILETOSTR(FULLPATH(tcValue))

This works fine because the file is directly loaded up into the buffer with no intermediate string variable.

You cannot however pass a string that is greater than 16 megs into this function because the code that adds the key basically does this with the tcValue parameter:

this.cPostBuffer = this.cPostBuffer + tcValue

which is assigning the larger than 16 meg string (tcValue in this case) to another variable and as discussed earlier that fails with String too long to fit. Using a string to buffer your output to build up a larger string, there's no workaround for adding a larger than 16 meg string to another variable or buffer using variables. So my code now works with files loaded from disk, but not string parameters

Good, but not good enough!

Files for Large Buffers

Based on the earlier examples I showed we know that we can easily load up massive content from a file. Thus FILETOSTR() offers an easy way to serve large files. Knowing that it's possible to build stream like class that allows you to accumulate string content in a file and then later retrieve it. To do this I created a wwFileStream class. Using the class looks like this:

*** Load library
DO wwapi

*** Create 20 meg string
lcString = REPLICATE("1234567890",1500000)
lcString = lcString + REPLICATE("1234567890",500000)

*** Create a stream
loStream = CREATEOBJECT("wwFileStream")

*** Write the 20meg  string
loStream.Write(lcString)

*** Add some more string data
loStream.WriteLine("...added content")

*** Now write a 16meg+ to the buffer as well
loStream.WriteFile("e:\downloads\ActiveReports3_5100158.zip")

*** Works
lcLongString = loStream.ToString()

*** 55+ megs
? loStream.nLength
? LEN(lcLongString)

*** Clear the file (auto when released)
loStream.Dispose()

Using this mechanism you can build up very large strings from files or strings regardless of what the size of the string is.

How wwFileStream works

Internally wwFileStream opens a low level file and tracks the handle. Each Write() operation does an FWRITE() to disk and the handle is released when the class goes out of scope.

The class implementation is pretty straight forward:

*************************************************************
DEFINE CLASS wwFileStream AS Custom
*************************************************************
*: Author: Rick Strahl
*:         (c) West Wind Technologies, 2012
*:Contact: http://www.west-wind.com
*:Created: 01/04/2012
*************************************************************

nHandle = 0
cFileName = "" 
nLength = 0


************************************************************************
*  Init
****************************************
FUNCTION Init()

this.cFileName = SYS(2023)  + "\" +  SYS(2015) + ".txt"
this.nHandle = FCREATE(this.cFileName)
this.nLength = 0

ENDFUNC
*   Init

************************************************************************
*  Destroy
****************************************
FUNCTION Destroy()
this.Dispose()
ENDFUNC
*   Destroy

************************************************************************
*  Dispose
****************************************
FUNCTION Dispose()

IF THIS.nHandle > 0
   TRY
   FCLOSE(this.nHandle)
   DELETE FILE (this.cFileName)
   CATCH
   ENDTRY
ENDIF
this.nLength = 0
ENDFUNC
*   Destroy

************************************************************************
*  Write
****************************************
FUNCTION Write(lcContent)
THIS.nLength = THIS.nLength + LEN(lcContent)
FWRITE(this.nHandle,lcContent)
ENDFUNC
*   Write

************************************************************************
*  WriteLine
****************************************
FUNCTION WriteLine(lcContent)
this.Write(lcContent)
this.Write(CHR(13) + CHR(10))
ENDFUNC
*   WriteLine

************************************************************************
*  WriteFile
****************************************
FUNCTION WriteFile(lcFileName)
lcFileName = FULLPATH(lcFileName)
this.Write(FILETOSTR( lcFileName ))
ENDFUNC
*   WriteFile

************************************************************************
*  ToString()
****************************************
FUNCTION ToString()
LOCAL lcOutput

FCLOSE(this.nHandle)
lcOutput = FILETOSTR(this.cFileName)

*** Reopen the file
this.nHandle = FOPEN(this.cFileName,1)
FSEEK(this.nHandle,0,2)

RETURN lcOutput
ENDFUNC
*   ToString()


************************************************************************
*  Clear
****************************************
FUNCTION Clear()

THIS.Dispose()
THIS.Init()

ENDFUNC
*   Clear

ENDDEFINE
*EOC wwFileStream 

The code is fairly self explanatory. The class creates a file in the temp folder and saves the handle. Any write operation then uses the file handle to FWRITE() either a string or the output from FILETOSTR(). ToString() can be called to retrieve the file, which closes the file, reads it then reopens it and points to the end. When the class is released the handle is closed and the handle released.

Using this class makes it easy to create large strings and hold onto them. The additional advantage is that memory usage is kept low as strings are loaded up only briefly and then immediately written to file and can be released. So if you're dealing with very large strings a class like this is actually highly recommended. In fact Web Connection uses this same approach for file based application output.

A matching MemoryStream Class

While the FileStream class works, it does have some overhead compared to memory based operation especially when you're dealing with small amounts of data. In the wwHttp class for example, I would not want to create a new wwFileStream for each POST operation. 99% of POST ops are going to be light weight, so it makes sense to only use the wwFileStream class selectively.

In order to do this I also created a wwMemoryStream class which has the same interface as wwFileStream and which uses a simple string property on the class to hold data. Since the classes have the same interface they are interchangable in use which makes them easily swappable.

The code for wwMemoryStream looks like this:

DEFINE CLASS wwMemoryStream AS Custom
*************************************************************
*: Author: Rick Strahl
*:         (c) West Wind Technologies, 2012
*:Contact: http://www.west-wind.com
*:Created: 01/05/2012
*************************************************************

cOutput = ""
nLength = 0

************************************************************************
*  Destroy
****************************************
FUNCTION Destroy()
THIS.Dispose()
ENDFUNC
*   Destroy

************************************************************************
*  Dispose
****************************************
FUNCTION Dispose()
this.cOutput = ""
this.nLength = 0
ENDFUNC
*   Dispose

************************************************************************
*  Clear
****************************************
FUNCTION Clear()
this.cOutput = ""
this.nLength = 0
ENDFUNC
*   Clear

************************************************************************
*  Write
****************************************
FUNCTION Write(lcContent)
this.nLength = this.nLength + LEN(lcContent)
this.cOutput = this.cOutput + lcContent
ENDFUNC
*   Write

************************************************************************
*  WriteLine
****************************************
FUNCTION WriteLine(lcContent)
this.Write(lcContent)
this.Write(CRLF)
ENDFUNC
*   WriteLine

************************************************************************
*  WriteFile
****************************************
FUNCTION WriteFile(lcFileName)
this.Write(FILETOSTR( FULLPATH(lcFileName) ))
ENDFUNC
*   WriteFile

************************************************************************
*  ToString()
****************************************
FUNCTION ToString()
RETURN this.cOutput
ENDFUNC
*   ToString()

ENDDEFINE
*EOC wwMemoryStream 

This way the user can easily chose which of the streams to use simply by specifying:

IF VARTYPE(this.oPostStream) != "O"
   this.oPostStream = CREATEOBJECT(this.cPostStreamClass)
ENDIF

What's also nice about this approach is that the mechanism becomes extensible. If you want to store POST vars in another storage format you can simply create another subclass that implements the same methods and now can store your post variables in an INI file or in structured storage etc. Unlikely scenario for POST data, but very useful for other potential data storage scenarios.

BTW, the wwFileStream class is also a fairly useful generic file output tool. If you ever need to write output to files it provides a real easy OO way to do so, cleaning up after itself when you close it. I've used classes like (wwResponseFile) for years in various applications that need to create file output. It's very useful in many situations.

Summary

Even though Visual FoxPro has a 16 meg string limit, you now have some tools in your arsenal to work around this limit and work with larger strings. While you can work with larger strings, keep in mind that once you go past 16 megs you can't assign that string to anything else. It also gets much harder (and slower) to use string manipulation functions on that string once you're beyond VFP's legal limit.

Still, nice to know that the limit is not a final one, and there are ways to work around it.

Resources

• Making Web Connection Work with Response Output Greater than 16mb
• wwStreams.prg and Sample

this post created and published with Markdown Monster