How to work around Visual FoxPro's 16 Megabyte String Limit

If you take a look at the Visual FoxPro documentation and the System Limits you find that FoxPro's string length limit is somewhere around ~16mb. The following is from the FoxPro Documentation:

Maximum # of characters per character string or memory variable: 16,777,184

Now 16mb seems like a lot of data, but in certain environments like Web applications it's not uncommon to send or receive data larger than 16 megs. In fact, last week I got a message from a user of our Client Tools lamenting the fact that the HTTP Upload functionality does not allow for uploads larger than 16 megs. One of his applications is trying to occasionally upload rather huge files to a server using our wwHttp class. At the time I did not have a good solution for him due to the 16meg limit.

What does the 16meg Limit really mean?

The FoxPro documentation actually is not quite accurate! You can actually get strings much larger than 16megs into FoxPro. For example you can load up a huge file like the Office 2010 download from MSDN like this:

lcFile = FILETOSTR("e:\downloads\en_office_professional_plus_2010_x86_515486.exe")
? LEN(lcFile)

The size of this string: 681,876,016 bytes or 681megs! Ok that's a little extreme ?? but to my surprise that worked just fine; you can load up a really huge string in VFP if you need to. But when you get over 16megs the behavior of strings changes and you can't do all the things you normally do with strings.

Other operations do not work for example, the following which creates an 18meg string fails:

lcString = REPLICATE("1234567890",1800000)

with “String is too long to fit”.

However the following which creates a 25 meg string does work:

lcString = REPLICATE("1234567890",1500000) && < 16 megs
lcString = lcString + REPLICATE("1234567890",1000000)
? LEN(lcString)  && 25,000,000

The following is almost identical except it copies the longer string to another string which does not work:

lcString = REPLICATE("1234567890",1500000) 
lcNewString = lcString + REPLICATE("1234567890",1000000)

And that my friends is the real sticking point with large strings:

You can create strings larger than 16 megs, but once they get bigger than 16megs you can no longer assign them to a new variable. Any operation that requires the string to be copied (even internally in FoxPro's engine) will not work.

That might sound easy to avoid but it's actually tough to do. If you pass string to mutable methods it's very likely that they are actually copied into temporary variables or added to another variable in a simulated buffer, and that is typically where large strings fail.

So what can we learn from this:

Doesn't Work:

• Assigning a massive FoxPro string to another string fails
• FoxPro commands that mutate strings like REPLICATE(), STRTRAN() can't create output larger than 16 megs

Works:

• Assigning a massive string from a file using FileToStr() works
• Adding to the same large string - ie. lcOutput = lcOutput + " more text" - works
• Calling methods that manipulate the string work as long as the same string is assigned

There are some limitations but knowing that if you work with a single string instance that can grow large is actually good news. If you're careful with how you use strings in FoxPro you can get around the 16 meg string limit.

This actually worked well for me in the wwHttp class and the POST issue for larger than 16meg files but not string. Internally wwHttp uses a .cPostBuffer property to hold the POST data. The failure was occurring in the send code which would copy the string to a temporary string and get the size, then pass that to the WinInet APIs. The fix for this was fairly easy: Rather than creating the temporary variables (which were redundant anyway) I simply used the class property directly throughout the code without any hand off and voila, now wwHttp supports POSTs for greater than 16 megs.

The code I use is kinda ugly because it's doing lots of string concatenation to build up the Post buffer. Something along these lines like this excerpt from wwHttp::AddPostKey:

************************************************************************
* wwHTTP :: AddPostKey
*********************************
***  Function: Adds POST variables to the HTTP request
***    Assume: depends on nHTTPPostMode setting
***      Pass: 
***    Return:
************************************************************************
FUNCTION AddPostKey(tcKey, tcValue, llFileName)
LOCAL lcOldAlias
tcKey=IIF(VARTYPE(tcKey)="C",tcKey,"")
tcValue=IIF(VARTYPE(tcValue)="C",tcValue,"")


IF tcKey="RESET" OR PCOUNT() = 0
   THIS.cPostBuffer = ""
   RETURN
ENDIF

*** If we post a raw buffer swap parms
IF PCOUNT() < 2
   tcValue = tcKey
   tcKey = ""
ENDIF

IF !EMPTY(tcKey)
   DO CASE
    *** Url Encoded
    CASE THIS.nhttppostmode = 1         
         THIS.cPostBuffer = this.cPostBuffer + IIF(!EMPTY(this.cPostBuffer),"&","") + ;
                            tcKey +"="+ URLEncode(tcValue) 
      *** Multi-part formvars and file
    CASE this.nHttpPostMode = 2
      *** Check for File Flag -  HTTP File Upload - Second parm is filename
      IF llFileName
           THIS.cPostBuffer = THIS.cPostBuffer + "--" + MULTIPART_BOUNDARY + CRLF + ;
            [Content-Disposition: form-data; name="]+tcKey+["; filename="] + JUSTFNAME(tcValue) + ["]+CRLF+CRLF
         this.cPostBuffer = this.cPostBuffer + FILETOSTR(FULLPATH(tcValue))
         this.cPostBuffer = this.cPostBuffer + CRLF
      ELSE
           this.cPostBuffer = this.cPostBuffer +"--" + MULTIPART_BOUNDARY + CRLF + ;
            [Content-Disposition: form-data; name="]+tcKey+["]+CRLF+CRLF
         this.cPostBuffer = this.cPostBuffer + tcValue
      ENDIF
   ENDCASE
ELSE
   *** If there's no Key post the raw buffer
   this.cPostBuffer = this.cPostBuffer +tcValue
ENDIF

ENDFUNC

AddPostKey() can accept either a string value or a filename to load from. The file loading works by accepting the filename and then directly loading the file from within the function:

this.cPostBuffer = this.cPostBuffer + FILETOSTR(FULLPATH(tcValue))

This works fine because the file is directly loaded up into the buffer with no intermediate string variable.

You cannot however pass a string that is greater than 16 megs into this function because the code that adds the key basically does this with the tcValue parameter:

this.cPostBuffer = this.cPostBuffer + tcValue

which is assigning the larger than 16 meg string (tcValue in this case) to another variable and as discussed earlier that fails with String too long to fit. Using a string to buffer your output to build up a larger string, there's no workaround for adding a larger than 16 meg string to another variable or buffer using variables. So my code now works with files loaded from disk, but not string parameters

Good, but not good enough!

Files for Large Buffers

Based on the earlier examples I showed we know that we can easily load up massive content from a file. Thus FILETOSTR() offers an easy way to serve large files. Knowing that it's possible to build stream like class that allows you to accumulate string content in a file and then later retrieve it. To do this I created a wwFileStream class. Using the class looks like this:

*** Load library
DO wwapi

*** Create 20 meg string
lcString = REPLICATE("1234567890",1500000)
lcString = lcString + REPLICATE("1234567890",500000)

*** Create a stream
loStream = CREATEOBJECT("wwFileStream")

*** Write the 20meg  string
loStream.Write(lcString)

*** Add some more string data
loStream.WriteLine("...added content")

*** Now write a 16meg+ to the buffer as well
loStream.WriteFile("e:\downloads\ActiveReports3_5100158.zip")

*** Works
lcLongString = loStream.ToString()

*** 55+ megs
? loStream.nLength
? LEN(lcLongString)

*** Clear the file (auto when released)
loStream.Dispose()

Using this mechanism you can build up very large strings from files or strings regardless of what the size of the string is.

How wwFileStream works

Internally wwFileStream opens a low level file and tracks the handle. Each Write() operation does an FWRITE() to disk and the handle is released when the class goes out of scope.

The class implementation is pretty straight forward:

*************************************************************
DEFINE CLASS wwFileStream AS Custom
*************************************************************
*: Author: Rick Strahl
*:         (c) West Wind Technologies, 2012
*:Contact: http://www.west-wind.com
*:Created: 01/04/2012
*************************************************************

nHandle = 0
cFileName = "" 
nLength = 0


************************************************************************
*  Init
****************************************
FUNCTION Init()

this.cFileName = SYS(2023)  + "\" +  SYS(2015) + ".txt"
this.nHandle = FCREATE(this.cFileName)
this.nLength = 0

ENDFUNC
*   Init

************************************************************************
*  Destroy
****************************************
FUNCTION Destroy()
this.Dispose()
ENDFUNC
*   Destroy

************************************************************************
*  Dispose
****************************************
FUNCTION Dispose()

IF THIS.nHandle > 0
   TRY
   FCLOSE(this.nHandle)
   DELETE FILE (this.cFileName)
   CATCH
   ENDTRY
ENDIF
this.nLength = 0
ENDFUNC
*   Destroy

************************************************************************
*  Write
****************************************
FUNCTION Write(lcContent)
THIS.nLength = THIS.nLength + LEN(lcContent)
FWRITE(this.nHandle,lcContent)
ENDFUNC
*   Write

************************************************************************
*  WriteLine
****************************************
FUNCTION WriteLine(lcContent)
this.Write(lcContent)
this.Write(CHR(13) + CHR(10))
ENDFUNC
*   WriteLine

************************************************************************
*  WriteFile
****************************************
FUNCTION WriteFile(lcFileName)
lcFileName = FULLPATH(lcFileName)
this.Write(FILETOSTR( lcFileName ))
ENDFUNC
*   WriteFile

************************************************************************
*  ToString()
****************************************
FUNCTION ToString()
LOCAL lcOutput

FCLOSE(this.nHandle)
lcOutput = FILETOSTR(this.cFileName)

*** Reopen the file
this.nHandle = FOPEN(this.cFileName,1)
FSEEK(this.nHandle,0,2)

RETURN lcOutput
ENDFUNC
*   ToString()


************************************************************************
*  Clear
****************************************
FUNCTION Clear()

THIS.Dispose()
THIS.Init()

ENDFUNC
*   Clear

ENDDEFINE
*EOC wwFileStream 

The code is fairly self explanatory. The class creates a file in the temp folder and saves the handle. Any write operation then uses the file handle to FWRITE() either a string or the output from FILETOSTR(). ToString() can be called to retrieve the file, which closes the file, reads it then reopens it and points to the end. When the class is released the handle is closed and the handle released.

Using this class makes it easy to create large strings and hold onto them. The additional advantage is that memory usage is kept low as strings are loaded up only briefly and then immediately written to file and can be released. So if you're dealing with very large strings a class like this is actually highly recommended. In fact Web Connection uses this same approach for file based application output.

A matching MemoryStream Class

While the FileStream class works, it does have some overhead compared to memory based operation especially when you're dealing with small amounts of data. In the wwHttp class for example, I would not want to create a new wwFileStream for each POST operation. 99% of POST ops are going to be light weight, so it makes sense to only use the wwFileStream class selectively.

In order to do this I also created a wwMemoryStream class which has the same interface as wwFileStream and which uses a simple string property on the class to hold data. Since the classes have the same interface they are interchangable in use which makes them easily swappable.

The code for wwMemoryStream looks like this:

DEFINE CLASS wwMemoryStream AS Custom
*************************************************************
*: Author: Rick Strahl
*:         (c) West Wind Technologies, 2012
*:Contact: http://www.west-wind.com
*:Created: 01/05/2012
*************************************************************

cOutput = ""
nLength = 0

************************************************************************
*  Destroy
****************************************
FUNCTION Destroy()
THIS.Dispose()
ENDFUNC
*   Destroy

************************************************************************
*  Dispose
****************************************
FUNCTION Dispose()
this.cOutput = ""
this.nLength = 0
ENDFUNC
*   Dispose

************************************************************************
*  Clear
****************************************
FUNCTION Clear()
this.cOutput = ""
this.nLength = 0
ENDFUNC
*   Clear

************************************************************************
*  Write
****************************************
FUNCTION Write(lcContent)
this.nLength = this.nLength + LEN(lcContent)
this.cOutput = this.cOutput + lcContent
ENDFUNC
*   Write

************************************************************************
*  WriteLine
****************************************
FUNCTION WriteLine(lcContent)
this.Write(lcContent)
this.Write(CRLF)
ENDFUNC
*   WriteLine

************************************************************************
*  WriteFile
****************************************
FUNCTION WriteFile(lcFileName)
this.Write(FILETOSTR( FULLPATH(lcFileName) ))
ENDFUNC
*   WriteFile

************************************************************************
*  ToString()
****************************************
FUNCTION ToString()
RETURN this.cOutput
ENDFUNC
*   ToString()

ENDDEFINE
*EOC wwMemoryStream 

This way the user can easily chose which of the streams to use simply by specifying:

IF VARTYPE(this.oPostStream) != "O"
   this.oPostStream = CREATEOBJECT(this.cPostStreamClass)
ENDIF

What's also nice about this approach is that the mechanism becomes extensible. If you want to store POST vars in another storage format you can simply create another subclass that implements the same methods and now can store your post variables in an INI file or in structured storage etc. Unlikely scenario for POST data, but very useful for other potential data storage scenarios.

BTW, the wwFileStream class is also a fairly useful generic file output tool. If you ever need to write output to files it provides a real easy OO way to do so, cleaning up after itself when you close it. I've used classes like (wwResponseFile) for years in various applications that need to create file output. It's very useful in many situations.

Summary

Even though Visual FoxPro has a 16 meg string limit, you now have some tools in your arsenal to work around this limit and work with larger strings. While you can work with larger strings, keep in mind that once you go past 16 megs you can't assign that string to anything else. It also gets much harder (and slower) to string manipulation on that string once you're beyond VFP's legal limit.

Still it's nice to know that the limit is not a final one and there are ways to work around it.

Resources

• wwStreams.prg and Sample

this post created and published with Markdown Monster