Rick Strahl's FoxPro and Web Connection Web Log
White Papers | Products | Message Board | News |
 

Creating Truly Unique Ids in FoxPro


No comments
Monday, February 13, 2017, 3:54:07 PM

Generating ids is a common thing for Database applications. Whether it's for unique identifiers to records in a database, whether you need to send a unique, non-guessable email link to a customer or create a temporary file, unique IDs are common in software development of all kinds.

Why not Sys(2013)?

FoxPro internally includes a not so unique id generation routine in SYS(2015):

? SYS(2015)
* _4UJ0VDHVX

This works for some things as long as they are internal to the application. But there are a lots of problems with this approach:

  • The values are easily guessable as they are based on sequential timestamps
  • Not unique across machines
  • The id value is too short
  • Duplication rate can be very high

SYS(2015)'s original purpose was internal to Foxpro for generating unique procedure names for generated code for some of the FoxPro tools. It worked fine because when it was created we had a single application running. Within a single application Ids are unique, but as soon as you throw in multiple applications either on the same machine or the network SYS(2015) is no longer able to even remotely guarantee unique ids.

For anything across processes or machines SYS(2015) is unacceptable. This can be mitigated somewhat by adding process or thread Ids to the string, but still there is too much possibility of conflict. Because the actual ID (minus the leading _) is only 9 character, the chance for duplication is also pretty high once the timestamp ‘rounds’ around. If you account for different timezones and multiple machines you find that IDs are not anywhere near 'unique'.

Guids

One way to ensure you generate truly unique IDs is to generate GUIDs. Guids are guaranteed to be unique across time and space (machines) as they are based on an algorithm that is based on a timestamp and a machine's MacId. Guids are safe and relatively easy to generate even in FoxPro:

FUNCTION CreateGUID
************************************************************************
* wwapi::CreateGUID
********************
***    Author: Rick Strahl, West Wind Technologies
***            http://www.west-wind.com/
***  Modified: 01/26/98
***  Function: Creates a globally unique identifier using Win32
***            COM services. The vlaue is guaranteed to be unique
***    Format: {9F47F480-9641-11D1-A3D0-00600889F23B}
               if llRaw .T. binary string is returned
***    Return: GUID as a string or "" if the function failed 
*************************************************************************
LPARAMETERS llRaw
LOCAL lcStruc_GUID, lcGUID, lnSize

DECLARE INTEGER CoCreateGuid ;
  IN Ole32.dll ;
  STRING @lcGUIDStruc
  
DECLARE INTEGER StringFromGUID2 ;
  IN Ole32.dll ;
  STRING cGUIDStruc, ;
  STRING @cGUID, ;
  LONG nSize
  
*** Simulate GUID strcuture with a string
lcStruc_GUID = REPLICATE(" ",16) 
lcGUID = REPLICATE(" ",80)
lnSize = LEN(lcGUID) / 2
IF CoCreateGuid(@lcStruc_GUID) # 0
   RETURN ""
ENDIF

IF llRaw
   RETURN lcStruc_GUID
ENDIF   

*** Now convert the structure to the GUID string
IF StringFromGUID2(lcStruc_GUID,@lcGuid,lnSize) = 0
  RETURN ""
ENDIF

*** String is UniCode so we must convert to ANSI
RETURN  StrConv(LEFT(lcGUID,76),6)
* Eof CreateGUID

To use these Guid routines:

? CreateGuid()
* {344986DB-D674-42BD-9A2E-A7833B190E05}

? CreateGuid(.t.)
*‰öÓeî‹èK“§Y‚þ¡I

? LOWER(CHRTRAN(CreateGuid(),"{}-"))
* 6823a6f0af7040318964e74cc8a78833

The middle one represents binary characters. Typically you wouldn't use that except for direct storage to a binary field (using CAST() most likely). The last value is what I recommend if you use GUIDs in any sort of user facing scenario. Using lowercase values makes it much easier to read the long value.

Guids are safe and guaranteed to be unique, but they are big. Even if you strip out the {}- from the string, it's still 32 characters. The binary value is 16 bytes, which is better, but for FoxPro data the last thing you'd want to do is use binary data for a field especially an indexable one.

Guids as keys also are a problem because they are truly random. There's very little commonality between one GUID and another, so any indexing scheme can't really pack GUIDs. Coupled with the large string size GUID indexes tend to be larger than other indexes.

Creating custom Variations off of Guids

In West Wind Web Connection we've had to use unique ids for a long time. Session tables in particular - with their potentially high volume insert/read operations - have always needed to have unique values that were unique across machines. I've gone through a number of iterations with this starting originally with SYS(2015) plus tacked on process and threadIds plus random characters.

But more recently (with Web Connection 6.0) I switched to using subsets of Guids and finally more recently I built a new routine that can strip down a Guid to a 16 character string safely.

How do you fit a 32 character string into 16 characters? Simple: GUIDs use hex value notation which means that the number of characters used is actually twice the number of bytes involved in the actual GUID binary. If you break down that each byte value into the full alphabet, digits and perhaps a few symbol characters you can get pretty close to representing GUIDs in full. Note you still lose some fidelity here - because we're shoehorning 255 hex values down to about 70, but in my testing running 10 million guids in a single run and over a billion in aggregate, I've not been able to generate any duplicates. That doesn't mean it can't happen but it's very, very unlikely. If you need 100% guarantees then stick with Guids - otherwise this variation is good enough.

The current routine that West Wind Web Connection and the West Wind Client Tools (versions 6.10 and later) use is this:

************************************************************************
*  GetUniqueId
****************************************
***    Author: Rick Strahl, West Wind Technologies
***            http://www.west-wind.com/
***  Function: Create a unique ID based on a Guid spread over 
***            full alpha, digit and some symbols
***    Assume:
***      Pass: lnLength = length between 8 and 16 - 16 is full Guid
***    Return:
************************************************************************
FUNCTION GetUniqueId(lnLength,llIncludeSymbols,lcAdditionalChars)
LOCAL lcChars, lcGuid, lcId, lnX, lcHex, lnHex, lcGuidBinary

IF VARTYPE(lnLength) # "N" 
   lnLength = 16
ENDIF
IF lnLength < 8
   lnLength = 8
ENDIF
IF lnLength > 16
   lnLength = 16
ENDIF   
IF EMPTY(lcAdditionalChars)
   lcAdditionalChars = ""
ENDIF   

lcChars = "abcdefghijkmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890" + IIF(llIncludeSymbols,"!@#$%&*+-?","") + lcAdditionalChars
lcGuidBinary = REPLICATE(" ",16) 

DECLARE INTEGER CoCreateGuid ;
  IN Ole32.dll ;
  STRING @lcGUIDStruc
CoCreateGuid(@lcGuidBinary)

lcId = ""
FOR lnX = 1 TO lnLength
   lnHex = ASC(SUBSTR(lcGuidBinary,lnX,1)) % LEN(lcChars)
   lcId = lcId + SUBSTR(lcChars,lnHex + 1,1)   
ENDFOR

RETURN lcID
*   GetUniqueId

The routine bascially grabs a new Guid and the breaks the Guid's bytes out into values that are provided as part of a ‘string array’ - a string of allowable characters. The code loops through all the bytes and pushes them into a new string based on the 'string array'.

To use this:

? GetUniqueId()   && Full 16 chars
* z9h8snad4dwe18sk

? GetUniqueId(8)  && Minimum
* cxip5nre

? GetUniqueId(20) && stripped to 16
* pfdogee6kd29978j

Note that you can pass a parameter for the number of characters to generate for the ID. The more characters you choose the more reliable the id up to 16. For small local scenarios 8 characters are going to be enough. Again running in tests I was unable to generate duplicate IDs in a single run, although running in the billion operations I managed to generate a total of 2 dupes. That's very low and goes up significantly as you add more characters.

This routine replaces my existing GetUniqueId() routine in Web Connection. The main change here is that the actual string generated is a lot shorter than the old routine. The old one required 15 characters up to 32. Here we require 8 up to 16. If a string greater than 16 is requested only 16 characters are returned. This should be Ok for backwards compatibility with VarChar(x) types in the DB the values will just work and with char(x) extra spaces fill out the number.

Performance

These routines are not blazingly fast especially if compared to SYS(2015). Most of this is due to the complexity of GUID generation and the FoxPro interop required to call it as well as the limited character iteration support in FoxPro - using SUBSTR() to iterate over each character in a string is very very slow. Interop in FoxPro has a bit of overhead and the routines require Unicode to ANSI conversions internally. Still on my machine I generate 10,000 ids in 2.5 seconds which puts the creation time roughly at 1/4 millisecond which is acceptable for a nearly unique, reasonably sized Id. By comparison though, SYS(2015) took less than a quarter second for those same 10,000 generations.

Summary

Remember that this last routine is not 100% guaranteed to be unique - but it's pretty close. If you need 100% guaranteed unique IDs stick with full GUIDs. Personally I feel pretty confident that there won't be any dupes with the GetUniqueId() routine even if I have a fully distributed application where data is entered in multiple locations.

There you have it - a few ways to generate unique IDs in FoxPro. Enjoy.

this post created with Markdown Monster


Web Connection 6.10 Release Notes


No comments
Wednesday, February 1, 2017, 3:25:41 PM

West Wind Web Connection Logo

West Wind Web Connection 6.10 has been released today and there are a a number of enhancements and a few new features in this update release. There are also a couple of breaking changes, so be sure to read those if you are upgrading from previous versions.

I'm going to break down the changes into Web features that are specific to Web Connection and General Purpose features that concern the more general Internet and Client Tools functionality, that will eventually also find their way into West Wind Internet And Client Tools.

This is a fairly long post - if you just want the quick and dirty list go to

Otherwise here's more detail:

Web Features

Let's start with the Web Features that are relevant for building Web applications with Web Connection.

Html Encoded Script Expressions: <%: %>

When building Web content it's important that you Html encode content. Html content encoding essentially makes HTML text safe, and is a huge step towards preventing cross site scripting attacks because script tags are encoded as text, rather than being left as executable HTML tags that run inside of your document.

Html Encoding takes angle brackets (< and >), quotes, ampersands and a few other characters and turns them into HTML safe entities that are not evaluated. By doing so you prevent script injection, so if anybody tries to inject a script tag like <script>alert("Gotcha");</script> into your content that attempt will be foiled.

Web Connection has always included an EncodeHtml() function that can be used so you could always do:

<%= EncodeHtml(poOrder.Notes) %>

With Web Connection 6.10 there now is an easier way to do the same thing:

<%: poOrder.Notes %>

So rather than having to explicitly specify the EncodeHtml() function using <%: expression %> does the same thing. The syntax is compatible with ASP.NET WebForms which also uses that same syntax for encoded content.

It's a nice convenience feature and I recommend you use it on all of your script expression tags, except where you explicitly do not want it! In most cases where you display user entered input, you want to HTML encoded text.

Updated Visual Studio Addin

We've updated the West Wind Web Connection Visual Studio add in and added it to the Visual Studio Gallery.

This means the addin can now be installed from Visual Studio itself via Tools -> Extensions and Updates. Because it's an installed extension in Visual Studio and it lives in the Extension Gallery, the addin can now automatically update itself if an update is available. It should show on the Updates tab in the Extension Manager and on the Visual Studio Notifications list.

The new addin also supports Visual Studio 2017 which brings a number of very cool productivity enhancements and a much more lightweight Web development experience.

wwRequest::GetLogicalPath() now returns Proxied and UrlRewrite Urls

The Request.GetLogicalPath() now properly returns the active URL the user sees, even if the URL was rewritten by tooling like IIS UrlRewrite or and internal proxy redirection.

For example, if you are using UrlRewrite to route extensionless URLs to a Web Connection Process class (UrlRewriteHandler) you now get:

** Original Url is: http://localhost/albumviewer/api/album/516
lcUrl = Request.GetLogicalPath()
* lcUrl  =  /albumviewer/api/album/516
*   not  =  /api/UrlRwriteHandler.av  (redirected url)

lcRedirectedUrl = Request.ServerVariables("PATH_TRANSLATED")
* lcUrl  =  /api/UrlRwriteHandler.av  (redirected url)

Previously GetLogicalPath() would always return the redirected path only.

If you are using URL redirection on your Web Server, you probably know that when you rewrite a URL on the server to a new location the original URL is lost to the redirected target URL.

A typical example for Url Rewriting is to rewrite Extensionless Urls to a specific Web Connection Url. For example, check out this request trace from an Extensionless URL in the AlbumViewer:

The original URL is:

/albumviewer/api/album/516

and it's rewritten to:

/albumviewer/api/RewriteHandler.av

The RewriteHandler is Web Connection's route handler that gets fired when a re-written request is found and you can overwrite this handler to route requests to the appropriate handler. The most common thing to do is simply route a method of the current class. You can find out more about this process in the documentation.

If you are logging requests or otherwise want to find out what the original URL the user sees in the address bar is, you had to explicitly look at the HTTP_X_ORIGINAL_URL header value in Request.ServerVariables().

Web Connection 6.10 now always returns the original URL when a request is proxied. The logic internally first checks for the proxy path and if found uses that. If not found that then PATH_INFO variable is returned as before.

This is important for applications that generate URLs on the fly and need to figure out relative paths or fully qualified to embed into the page or to send out as email links for example. It's a minor feature, but an important one for those of you that use URLRewrite.

Admin Script Compilation

The Admin page can now properly compile MVC Style Script pages via the Admin page and this link:

This operation can run through either a single folder or all folders of your site and find all matching files you specify via the wildcard and recompile the script files.

While this feature was there previously it didn't actually work with the new scripting engine, and it didn't support recursive compilation.

Using this option can allow you to run with pre-compiled scripts if you didn't explicitly run through all scripts and upload them to your site.

There's also a new wwScripting::CompileAspScript() method that lets you compile and individual script. The above script compilation features use this method to handle the script compilation. You can look at WCSCompile() in wwServer.prg if you want to see how that works.

wwUserSecurity Password Encryption

We've added the ability to encrypt passwords in the wwUserSecurity class by setting the new cPasswordEncryptionKey property to a string value used as the hash encryptionkey.

If the cPasswordEncryptionKey property is set, any SaveUser() operation on the object causes the password property to be encrypted if it is not encrypted yet. Encrypted passwords are post-scripted with ~~ to detect whether the field is encrypted or not.

The password encryption uses id salted SHA512 hashing to produce the password hash used in the user security table.

By default cPasswordEncryptionKey is empty so no password encryption occurs unless you explicitly specify it.

If you plan on using this feature I would highly recommed that you sublcass the wwUserSecurity class and set the cPasswordEncryptionKey as part of the class. How you set the value is up to you, whether it's simply a static value you assign or whether you retrieve the key from some known save location like Azure Key Storage or likewise service.

Password Hashing is One-Way

Please note that once you encrypt passwords you can't retrieve them for users. Hashing is basically a one way trip and any authentication that compares passwords hashes an input value to match stored password hash. The only way to ‘fix’ a password for a user if they've lost it, is for them to create a new one.

wwUserSecurity Structure Changes

As part of the updates for encrypted passwords we also made some changes to the structure of the wwUserSecurity table. These changes go beyond the password field, but since we had to make changes anyway we updated the table to use VARCHAR characters for all text fields.

The new structure is:

CREATE CURSOR usersecurity ;
(    PK          V (40),;
   USERNAME    V (80),;
   PASSWORD    V (80),;
   FULLNAME    V (80),;
   MAPPEDID    V (40),;
   EMAIL       M ,;
   NOTES       M ,;
   PROPERTIES  M ,;
   LOG         M ,;
   LEVEL       Y ,;
   ADMIN       L ,;
   CREATED     T ,;
   LASTON      T ,;
   LOGONCOUNT  I ,;
   ACTIVE      L ,;
   EXPIRESON   D )

If updating from the old version you should also run the following command to trim white spaces off the fields:

REPLACE ALL PK with TRIM(PK), ;
            USERNAME WITH TRIM(USERNAME),;
            PASSWORD WITH TRIM(PASSWORD),;
            FULLNAME WITH TRIM(FULLNAME),;
            MAPPEDID WITH TRIM(MAPPEDID)

Breaking Change

The change above is a breaking change and you have to ensure you change the database structure of any existing tables to match this new schema.

Updated to a new Markdown Parser Library

Markdown conversion was introduced in Web Connection 6.0. Markdown is a simple text editing format that generates HTML output using a very simple markup language that can be easily typed as text. Markdown is awesome to use instead of text input as it allows simple markup like bold and italic text, lists, headers and so in with a very text like format that doesn't require a special editor. Adding simple interactive editing features like a toolbar is also pretty easy to accomplish just with some simple javascript.

Web Connection's Markdown support comes via the MarkdownParser class and more typically through the Markdown function that it exposes. To parse Markdown you can simply do:

lcMarkdown = "This is some **bold** and *italic* text"
lcHtml = Markdown(lcMarkdown)

More commonly though you're likely to use markdown in your HTML pages to write out rich content. For example on the message board each message's body in a thread is displayed with:

<div class="message-list-body">
    <%= Markdown(loMessage.oMessage.Body) %>
</div>

or if you have a custom configuration options for formatting the Markdown:

<div class="message-list-body">
    <%= poMdParser.Parse(loMessage.oMessage.Body) %>
</div>

In Web Connection 6.10 we've switched from the .NET CommonMark.NET package to the MarkDig parser. MarkDig supports Github flavored markdown, automatic URL linking, and a slew of other standards that sit on top of markdown out of the box that in older versions we had to implement on our own. Besides the simplicity Markdig also is much easier to extend and quite a bit faster especially since it can perform most of the add-on operations we needed to do in FoxPro previously now internally.

Breaking Change

This is a breaking change and in order to use the Markdown Features in Web Connection 6.10 you need to make sure you include the Markdig.dll with your Web Connection distribution. This replaces the CommonMarkNet.dll that was previously used.

General Purpose Features

The following features are focused on the general purpose library portion of Web Connection. These are also features that will show up in future versions of West Wind Client Tools.

SFTP Support with the wwSFTP Class

One of the most requested features in both Web Connection and Client Tools over the years has been support for secure FTP. Secure FTP is a tricky thing to provide as there are several standards and because the built-in Windows library that Web Connection uses - WinINET - doesn't support any secure FTP features.

In Web Connection 6.10 there's now support for SFTP which is FTP over SSH via the wwSftp Class. The class is based on the familiar wwFTP class and the interface to send and receive files remains the same as with the original wwFtp class.

loFtp = CREATEOBJECT("wwSftp")
loFtp.nFtpPort = 23

lcHost = "127.0.0.1"
lnPort = 23
lcUsername = "tester"
lcPassword = "password"

*** Download
lcOutputFile = ".\tests\sailbig.jpg"
DELETE FILE lcOutputFile

lnResult = loFtp.FtpGetFile(lcHost,"sailbig.jpg",".\tests\sailbig.jpg",1,lcUsername,lcPassword)

this.AssertTrue(lnResult == 0,loFtp.cErrorMsg)
this.AssertTrue(FILE(lcOutputFile))


*** Upload a file
lcSourceFile = ".\tests\sailbig.jpg"
lcTargetFile = "Sailbig2.jpg"

lnResult = loFtp.FtpSendFile(lcHost,lcSourceFile,lcTargetFile,lcUsername,lcPassword)
this.AssertTrue(lnResult == 0,loFtp.cErrorMsg)

There are both high level (all in one upload/download file functions) and low level functions. The low level function require that you open a connection explicitly and fire each operation, potentially multiple times. Again this maps the existing old wwFtp functionality:

loFtp = CREATEOBJECT("wwSftp")

loFtp.cFtpServer =  "127.0.0.1"
loFtp.nFtpPort = 23
loFtp.cUsername = "tester"
loFtp.cPassword = "password"

loFtp.FtpConnect()

*** Change to a specific folder - convenience only - you can reference relative paths
this.AssertTrue(loFtp.FtpSetDirectory("subfolder"),loFtp.cErrorMsg)

*** Create a new directory
this.AssertTrue(loFtp.FtpCreateDirectory("SubFolder2"),loFtp.cErrorMsg)

*** Send a file into the new directory
this.AssertTrue(loFtp.FtpSendFileEx(".\tests\sailbig.jpg","subfolder2/sailbig.jpg")==0,loFtp.cErrorMsg)

*** Download the file just uploaded
this.AssertTrue(loFtp.FtpGetFileEx("subfolder2/sailbig.jpg",".\tests\sailbig2.jpg")==0,loFtp.cErrorMsg)

*** Delete it
this.AssertTrue(loFtp.FtpDeleteFile("subfolder2/sailbig.jpg")==0,loFtp.cErrorMsg)

*** And delete the folder
this.AssertTrue(loFtp.FtpRemoveDirectory("Subfolder2"),loFtp.cErrorMsg)

No FTPS support

Note that this feature does not support FTPS which is yet another protocol that uses TLS over FTP which is considerably less common than the SFTP over SSH protocol implemented by this class.

wwUtils::GetUniqueId()

This routine lets you generate semi to fully unique IDs based on GUIDs. You can specify a size between 15 and 32 characters with 32 characters preserving the full fidelity of a GUID. Any value smaller gives you somewhat unique (definitely a lot more unique than SYS(2015) though) values.

wwUtils::SplitString()

This function provides the same functionality as ALINES() but adds the important abililty to split very long lines (that exceed 254 characters) with MEMLINES() into additional lines.

This is critical for any sort of code parsing library that generates code with string literals which cannot exceed FoxPro's literal string limit.

This function is used internally in wwScripting and webPageParser and the Markdown parser but can be useful for anything else that needs to generate code output.

wwEncryption::ComputeHash() adds HMAC Hash Algorithms

wwEncryption::ComputeHash() adds HMACSHA1, HMACSHA256, HMACSHA384 and HMACSHA512 algorithms. HMAC algorithms use complex salting cycles to add complexity and delay to generated hashes using an industry standard and repeatable algorithm.

Note the HMAC related functions require that you specify a Salt value for the hash.

wwDotnetBridge now supports TLS 1.2

wwDotnetBridge fires up a new instance of the .NET Runtime inside of Visual FoxPro when it launches and as such any configuration set for the app has to be set as well. A number of people using wwDotnetBridge ran into problems with HTTPS content not working correctly because older versions of .NET don't default to support TLS 1.2.

In Web Connection 6.10 we always enable TLS 1.2 support (and conversely disable obsolete and insecure SSL3 support). The old version only allows support for TLS 1.0. This affects any HTTP clients whether directly using HTTP clients or using libraries (such as Credit Card Processing APIs, REST APIS etc.) that use HTTPS internally.

Summary

Phew, a lot of stuff in this release. There are also a number of small bug fixes and minor performance tweaks based on Message Board discussions in the last few months.

As always, we actively encourage feedback, so if you run into a bug or have a feature request, let us know by posting a message in the support forum.

this post created with Markdown Monster


Syntax Errors in the FoxPro Editor caused by Extended Characters


1 comment
Wednesday, January 25, 2017, 10:46:20 AM

I keep running into weird errors, when trying to save program files in the FoxPro editor on occasion. For some reason I end up with errors like the following:

What could be wrong with that line of code? Even if the variable name was invalid the compiler should always be able to process a simple expression like this, right?

10 minutes later and after checking the code around it, I finally figure out that I have to...

Watch those Control Characters

If you look closely at the line of code highlighted, you'll notice that there's an extra space at the end of that line. Or rather what appears to be an extra space.

As it turns out that's no space - it's an invisible control character sequence that I accidentally inserted by way of my Visual Studio biased fingers :-) I pressed Ctrl-Shift-B to build my project (a Visual Studio key combo) which in turn embedded a character combination into the editor. That invisible character is interpreted as an extra character on the line of code and so the line actually becomes invalid.

It becomes obvious if you take the text and paste it into another editor like Sublime Text:

The result: The code doesn't compile and you get the above error. Remove the extra character an life is good again.

Moral of the Story

This happens to me on a regular basis. The FoxPro editor is notorious for its crappy handling of control characters - even those it knows about like Ctrl-S. For example, if you hit Ctrl-S multiple times in a row quickly the first time will save, while subsequent Ctrl-S combos will inject characters into the document.

Ctrl-S Failure

This also causes syntax errors which often get left in the document, but at least this one you can see and the compiler can point you at the offending line or code block.

Other key combos though - like my Ctrl-Shift-B compile twitch are more difficult to catch when the compiler complains because they are invisible and it looks like there's nothing wrong.

The ultimate moral of the story is: If you see an error that clearly isn't an error make sure it's not just an extra character that snuck into the document.

this post created with Markdown Monster


Route Web Connection Scripts, Templates and Pages to a single Process Class


No comments
Sunday, January 15, 2017, 2:51:44 PM

West Wind Web Connection supports a number of different 'frameworks' for generating HTML. There are two styles of MVC (Model View Controller) renderers and an ASP.NET like Forms based interface for creating rich FoxPro code based HTML pages.

These engines are available:

  • Web Connection Scripts (wcs by default)
    HTML templates that compile into FoxPro PRG files and are executed as code

  • Templates
    Simple templates that work well with embedded expressions and self contained code blocks that are evaluated on the fly (ie. no compilation).

  • Web Control Framework Pages
    These are ASP.NET style pages that use an object based approach to describing controls on a page.

Both Scripts and Templates work for MVC style applications where you have controller code in a class method, and the template/script is the view that displays the model data you accumulated in the controller code to display in the script or template.

Web Control Framework pages are more complex, compiling HTML controls into component classes that are coordinated and rendered by the page framework.

Only one Default nPageScriptMode

When you set up a new Web Connection process class that process class can associate with a default nPageScriptMode:

  • 1 - Template
  • 2 - Web Control Framework
  • 3 - Script

The default is 3 - Script.

The default determines what happens when you access a page like NonMethodPage.wwd for example, where there is no matching method in the Process class. If there is no NonMethodPage method in the class, Web Connection falls back to the default nPageScriptMode to decide what to do with the script. In this case, wwd will be passed to the default nPageScriptMode which is 3 - Script which compiles the script into a small PRG/FXP and it runs.

Handling more than one nPageScriptMode

So I just said that there's only one default, right? And that is so, but... you can actually tweak the behavior by explicitly setting the nPageScriptMode based on parsing rules in your code.

If I wanted to have multiple script maps for a single process method I might set up wwd, wwds, wwdx extensions for templates, scripts and pages respectively.

I then have to make sure I route all of these extensions in my MyServer::Process():

CASE lcExtension == "WWD" OR lcExtension == "WWDS" OR lcExtension == "WWDX"
    DO wwDemo with THIS

actually this is enough if SET EXACT OFF as it is by default:

CASE lcExtension = "WWD"
    DO wwDemo with THIS

this makes sure all requests with these extensions are routed to my process class.

In the process class I can then handle the extension routing in the OnProcessInit():

FUNCTION OnProcessInit()

* ... other init code

lcExt = UPPER(JUSTEXT(this.oRequest.GetPhysicalPath()))
THIS.nPageScriptMode = 3
DO CASE 
   CASE lcExt == "WWD"
      this.nPageScriptMode = 1
   CASE lcExt == "WWDS"
      this.nPageScriptMode = 3
   CASE lcExt == "WWDX"
      this.nPageScriptMode = 2
ENDCASE

Because OnProcessInit() fires early in the process pipeline you can change the page scriptmode which is used in wwProcess::RouteRequest() to find the right handler. Checked this out and it works great using a single process class to route requests for each extension to the appropriate handler.

And voila - you can now handle wwd pages as templates, wwds pages as scripts and wwdx pages as Web Control Framework pages.

You can make the logic more complex as well and look at other parts of the request. Perhaps extensionless URLs or query strings etc. - you have full access to the Request object to do as you please to figure out what gets routed where.

It's a cool little trick for your Web Connection applications.

this post created with Markdown Monster


Visual Studio Gets Support for FoxPro Files


3 comments
Wednesday, November 16, 2016, 2:26:15 PM

Visual Studio 2015 now has support for FoxPro files by way of a very cool add-in from Mads Kristensen. Mads is a a Program Manager on the Visual Studio Tools team and the prime driver behind extensions for ASP.NET tooling and he has created a cool extension that can take existing Textmate style syntax files and render them as syntax highlighted documents. Thanks to Matt Slay who a while back created a Sublime Text FoxPro language extension, Mads was able to add FoxPro as a language to the extension language pack.

End result: You can now view and edit FoxPro files with syntax coloring in Visual Studio natively. Keep in mind this basic syntax coloring support, not a full featured editor with auto-complete. Just the basics...

Here's what this looks like:

You can find out more and install the extension from here:

or use the Visual Studio Extension Manager to add it directly from within Visual Studio.

You'll also want to install the File Icons extension to show appropriate icons for your FoxPro files:

Cool!

It's nice to have native support for FoxPro right in Visual Studio, so you can open and edit FoxPro files right from the Visual Studio project.

This works great when working with Web Connection, so that you can quickly open PRG files and edit them in-place. If you're Web Control Pages and you keep your PRG files with the script page file, the two files live side by side and you can edit them.

It's also useful for checking out generated script template PRG files while debugging code and you can also open Process class files (manually) in the editor and see your process class code side by side with a template.

Keep in mind the features are limited to syntax coloring. There's no auto-complete support, so you may still want to use the Web Connection add-in to explicitly open FoxPro files in the FoxPro editor.

Other Editors with Web Connection?

FWIW, you don't have to use Visual Studio or FoxPro either. If you want a different editor for your PRG files - like Sublime text as I've done for years - you can use the Open With... option to specify a default editor to open files with. You can map PRG files there to whatever editor you like including the VFP IDE.

Once done double clicking opens the file in the specified editor. I've been using Sublime for years because it fires up quickly, but now I may stick with the built in editor simply for the fact that everything is one place.

Web Connection and Editing FoxPro Files

Note Web Connection also supports opening Code behind files using the Web Connection Visual Studio Addin:

Web Control Pages automatically open the code behind file. For script and template pages a special page directive allows you to specify what file should be opened - notice the highlight in the code file above:

<% * SourceFile="~/../../fox/samples/wwdemo/wwdemo.prg" %>

which lets you specify what file is opened. Here it points at the Web Connection wwProcess class handler file as a relative path to the current document.

The editor that's used can be configured in web.config using the following settings:

<configuration>
    <webConnectionVisualStudio>
        <add key="FoxProEditor" value="" />
        <add key="FoxProEditorAlternate" value="C:\Program Files\Sublime Text 3\sublime_text.exe" />
    </webConnectionVisualStudio>
</configuration>  

An empty string value means that the VFP IDE is used, otherwise the file specified by the path is used with the filename passed as parameter.

Any way you slice it - there are now quite a few options for opening FoxPro code in a decent editor - you get to choose what works best for you.


Custom Row and Column Rendering in Web Connection with HtmlDataGrid()


No comments
Thursday, June 30, 2016, 1:13:33 AM

DataGrids are popular use case for FoxPro developers who are obsessed with use data grids for data display. Personally I'm not a huge fan of grids, unless absolutely necessary, but for some use cases grids using HTML tables are the only way to go to display a lot of data on a form.

In this post I'll describe using the wwHtmlHelper HtmlDataGrid() function which makes it super easy to create grids, and specifically demonstrate how you can create customized displays for rows or columns to highlight content.

HtmlDataGrid() basics

HTMLDataGrid() is a wwHtmlHelper function that is used to render data from tables or collections/arrays into an HTML table. It uses the new Web Connection 6.0 Bootstrap styling to create attractive grid displays fairly easily.

The function supports a couple of different operational modes:

  • Auto-Column generation from a cursor
  • Custom Column Definitions

Auto Column Generation

The former is super easy - it simply takes a cursor and renders it.

SELECT albums.pk,albums.title, albums.year, ;
       artists.pk as artistpk, artists.artistName  ;
    FROM albums, artists ;
    ORDER BY albums.title ;
	WHERE albums.ArtistPk = artists.Pk AND ;
	      artists.ArtistName = ?lcFilter ;	
    INTO CURSOR TQuery

* ... other header rendering

*** Render the Grid
Response.Write( HtmlDataGrid("TQuery") )

* ... other page rendering

Alternately you can also capture the HTML in your controller code:

PRIVATE pcHtml
pcHtml = HtmlDataGrid("TQuery") 

and you can then render the HTML inside of a template or script page:

<div class="gridcontainer">
    <%= pcHtml %>
</div>

In either case, this produces a simple table list of all the records:

If you want a little more control you can add a HtmlDataGridConfig object to configure a few options like paging and styling of the table:

loConfig = CREATEOBJECT("HtmlDataGridConfig")
loConfig.PageSize = 10             && 10 items per page
loConfig.PagerButtonCount = 5      && Max page number buttons in pager

Response.Write( HtmlDataGrid("TQuery",loConfig) )

which turns the grid into a paged grid:

Auto-column generation is very easy but it's fairly limited. As you can see you can't easily control what fields are displayed (the pks are showing) unless you modify your query to return only the fields you want, and the titles are determined by the field names.

It works for quick and dirty stuff, but in most cases you'll want to use custom grid columns.

Custom Column Definitions

The better approach is to explicitly define your columns for the data grid which gives you a lot more control. When using columns you can add custom styling, apply formatting and even use code expressions to transform the content. For example, it's fairly easy to call other helpers for adding things like links or checkboxes, or use entire custom functions for rendering the content of individual columns.

The following example demonstrates a few custom features:

SELECT albums.pk,albums.title, albums.year, ;
       artists.pk as artistpk, artists.artistName ;
    FROM albums, artists ;
    ORDER BY albums.title ;
	WHERE albums.ArtistPk = artists.Pk AND ;
	      artists.ArtistName = ?lcFilter ;	
    INTO CURSOR TQuery

* ... other HTML generation
  
*** Use the HtmlDataGrid Helper function (easier)
loConfig = CREATEOBJECT("HtmlDataGridConfig")

*** Paging is a problem when you dynamically switch between the two modes
*** so leave out for now
loConfig.PageSize = 6

*** Add a column explicitly by craeating it first
*** Note this column uses an expression that is evaluated for each record
loColumn = loConfig.AddColumn([HtmlLink("ShowAlbum.wwd?id=" + TRANSFORM(Tquery.pk),Title)],"Album")
loColumn.Sortable = .T.
loColumn.SortExpression ="Upper(Title)"

loColumn = loConfig.AddColumn("ArtistName","Artist Name")
loColumn.Sortable = .T.
loColumn.SortExpression ="Upper(ArtistName)"
loConfig.AddColumn("Year","Year Released","N")

*** Add a checkbox column
loConfig.AddColumn("HtmlCheckBox([chkIsActive_] + TRANSFORM(Pk) ,[],Year > 2000)","Recently Produced")

lcHtml = HtmlDataGrid("Tquery",loConfig)

Response.Write(lcHtml)

which looks like this:

Notice that we have a link and a checkbox embedded into this display - using additional HTML helpers to render these items. You can embed any string expressions - including other HTML helpers which makes it easy to create more complex items in cells.

Additionally you can also create column output from expressions which can be either native FoxPro functions (for formatting for example), User Defined Functions, or class method calls.

To do this simply create the column expression with the function name:

loColumn = loConfig.AddColumn([Process.LinkAlbum()],"Album")

You can then implement this function on the Process class. Note it doesn't have to be a Process class method it just has to be something that's in scope called further down the call stack.

Customizing Column Display

In this function you can do whatever you want to do. So if you want to create a custom column that bolds the link you could do:

FUNCTION LinkAlbum()
LOCAL lcHtml
lcHtml = [<a href="ShowAlbum.wwd?id=] + TRANSFORM(TQuery.pk) + [" ]  +;
		 [  style="font-weight: bold; color: green;">] +;
         EncodeHtml(TQuery.Title) + [</a>]

RETURN lcHtml
ENDFUNC

Which now renders a bold green texted link.

Not very practical, but you get the idea - you can pretty much generate any HTML using a function that creates the HTML string output. It's very powerful.

Customizing the Column Styling

You can also access the Column's own styling and attributes using the ActiveColumnAttributeString property for the config object. For example to turn the Year column to a green background when the year is newer than 2000 you can use code like this:

loConfig.AddColumn("Process.YearColumn(loConfig)","Year Released","N")

Then create the handler:

FUNCTION YearColumn(loConfig)

IF (TQuery.Year > 2000)
   loConfig.ActiveColumnAttributeString = [ style="background: green; color: white" ]
ELSE
   loConfig.ActiveColumnAttributeString = [  ]
ENDIF
 
RETURN TRANSFORM(TQuery.Year)
ENDFUNC

The ActiveColumnAttributeString allows you to customize the <td> element with custom attributes which gives you full control over rendering.

Here's what the green boxed cell looks like:

Customizing Row Rendering

Finally it's also possible to affect row rendering using the OnBeforeRowRender() and OnAfterRowRender() handlers. These handlers also take expressions that have to return a string. Using these functions it's possible to get access to the row header and the individual column headers.

This is similar to the column customization except you use the RowAttributeString to customize the row.

First hook up the OnBeforeRowRender handler with the function expression:

loConfig.OnBeforeRowRender = "Process.RenderRowHeader(loConfig)"

Then implement the method:

FUNCTION RenderRowHeader(loConfig)

IF (TQuery.Year > 2000 )
	loConfig.RowAttributeString = [ class="highlight" ]
ELSE
    loConfig.RowAttributeString = []
ENDIF
    
RETURN ""

In this example, rather than setting an explicit style I simply assign a custom CSS class defined in a style sheet:

.highlight {
    background: #c6f1c6 !important;
}

which applies a light green background to the entire row. The !important forces the background to override other styles.

When you now render this grid you get:

Note that you have to always set the RowAttributeString to a value including the empty value, because when you set it that value stays for all subsequent requests. So effectively you have to reset it for each row.

Summary

As you can see you have a lot of control when rendering HtmlDataGrids with columns and values, even if the mechanism by which this works is not very obvious. But with a proper reference like this blog post it's easy to pick the right choice to customize your column or row render to match your needs exactly with very little code effort.


Calling JSON REST Services with FoxPro and wwJsonServiceClient


3 comments
Thursday, May 26, 2016, 11:54:21 PM

A few days ago I released version 6.0 of the West Wind Client Tools which includes a little gem of a class called wwJsonServiceClient that makes it super easy to call JSON REST services. This class is part of the West Wind Client Tools as well as West Wind Web Connection and it makes it a snap to call JSON services to receive and send data.

Calling REST Services

REST services are becoming more and more popular and are starting to crowd out SOAP based services as the vehicle to share raw data over the Internet. Where SOAP is a very heavy XML based protocol that requires a very strict format, REST generally uses much simpler structures to push data over the wire. Data is usually encoded as JSON rather than XML. JSON is essentially a way to express JavaScript values, objects and arrays as a literal string value, which is easy to create and parse using standard libraries. Like XML, JSON is a serialization format, but unlike XML it skips all the formality and provides just the data.

If you're retrieving data from online social services or 'new Web' companies, you're going to encounter REST and JSON APIs generally. The benefit of REST JSON services is that generally they are much simpler than SOAP counterparts as you don't have worry about XML data mapping descreprancies. You don't need any special software to access a REST service - there's no SOAP client that has to parse a SOAP message and call. You simply have a documented endpoint and a set structure of data to send in (if any) and a set structure of data that is returned. All you need to access a REST service is an HTTP client (like wwHttp) and if the data is in JSON format a JSON serializer (like wwJsonSerializer).

On the server side things are also easier because again there's not additional contract that needs to be created. To create a REST service you build plain HTTP endpoints and return a JSON value or object. If you're using FoxPro in Web Connection the wwRestProcess class makes it easy to create FoxPro based REST services by simply creating classes that take a single input parameter and return a single result value. The wwRestProcess class handles all the logistics of publishing that data.

What is wwJsonServiceClient?

You can think of wwJsonServiceClient as a simple way to call a REST service. At it's simplest it's a high level wrapper class that wraps the wwHttp and wwJsonSerializer to call JSON services via HTTP and handles the JSON serialization and deserialization.

In a nutshell it reduces calling a REST service to a single line of code.

Making a GET Request

Let's try it out. The following calls a sample site to retrieve a list of Music Albums from an online AlbumViewer sample I created for last year's Web Connection Training.

This first request is a simple GET command that retrieves the list of albums like so:

do wwhttp
do wwJsonSerializer

loProxy = CREATEOBJECT("wwJsonServiceClient")

*** Make the service call
loAlbums = loProxy.CallService("http://albumviewerswf.west-wind.com/api/albums",
                               "","GET")
? loProxy.cErrorMsg

lnCount = loAlbums.Count

* ? loAlbums.Item(1).Title

FOR EACH loAlbum in loAlbums 
   ? loAlbum.Title  + ;
     " by " + loAlbum.Artist.ArtistName + ;
     " (" +  TRANSFORM(loAlbum.Tracks.Count) + " tracks)"
   FOR EACH loTrack IN loAlbum.Tracks
		? "  " + loTrack.SongName
   ENDFOR
ENDFOR

The key is this line:

loAlbums = loProxy.CallService("http://albumviewerswf.west-wind.com/api/albums","","GET")

which makes an HTTP GET request to the server with not data to send to the server (empty second parameter). The last two parameters are actually not required - if no data is passed GET is assumed for the HTTP verb. You've just simplified a service call to single line of code.

POSTing data to a server

Along the same lines you can also post data to a server by passing a value, object or array to the server as the second parameter and changing the HTTP verb.

loArtist = CREATEOBJECT("Empty")
ADDPROPERTY(loArtist,"Id","2")
ADDPROPERTY(loArtist,"ArtistName","Accept")
ADDPROPERTY(loArtist,"Description","Old school German Metal band")
ADDPROPERTY(loArtist,"ImageUrl","http://cps-static.rovicorp.com/3/JPG_400/MI0001/389/MI0001389322.jpg?partner=allrovi.com")
ADDPROPERTY(loArtist,"AmazonUrl","")

loProxy = CREATEOBJECT("wwJsonServiceClient")
loArtist = loProxy.CallService("http://albumviewerswf.west-wind.com/api/artist",loArtist,"PUT")

IF  loProxy.lError
   ? loProxy.cErrorMsg
   RETURN
ENDIF

? loArtist.Albums.Count

Now in this case the call fails because you actually have to log in first, but if you hook up an HTTP proxy like Fiddler you you'll see the following was sent to the server:

PUT http://albumviewerswf.west-wind.com/api/artist HTTP/1.1
Content-Type: application/json
User-Agent: West Wind Internet Protocols 6
Host: albumviewerswf.west-wind.com
Content-Length: 194
Pragma: no-cache
Cookie: _ga=GA1.2.1516538738.1449808058; albv=C03E01F1F67F15336

{ 
 "amazonurl":"",
 "artistname":"Accept",
 "description":"Old school German Metal band",
 "id":"2",
 "imageurl":"http://cps-static.rovicorp.com/3/JPG_400/MI0001/389/MI0001389322.jpg?partner=allrovi.com"
}

So you can see that the service client is generating an HTTP request with a JSON payload from your method call.

Dealing with FoxPro Case Limitations

Now it turns out that the service actually expects all the property names to be proper case. FoxPro's property retrieval API unfortunately does not natively support preserving case (except if you use MemberData which is a pain in the ass) and so the serializer by default turns all property names into lower case.

If you need to communicate with a service that requires either proper or camel case (which is common for JSON services) you can override property names explicitly.

loProxy = CREATEOBJECT("wwJsonServiceClient")

*** Create a custom serializer that overrides property names
loSer = CREATEOBJECT("wwJsonSerializer")
loSer.PropertyNameOverrides = "Id,ArtistName,Description,ImageUrl,AmazonUrl"
loProxy.CreateSerializer(loSer)

loArtist = loProxy.CallService("http://albumviewerswf.west-wind.com/api/artist",loArtist,"PUT")

IF  loProxy.lError
   ? loProxy.cErrorMsg
   RETURN
ENDIF

? loArtist.Albums.Count

Now the data is sent with proper names:

{ 
 "AmazonUrl":"",
 "Artistname":"Accept",
 "Description":"Old school German Metal band",
 "Id":"2",
 "ImageUrl":"http://cps-static.rovicorp.com/3/JPG_400/MI0001/389/MI0001389322.jpg?partner=allrovi.com"
}

These overrides are applied globally to all properties, child properties and array items so this addresses the FoxPro type name limitations.

HTTP Verbs

The above example used a PUT command to update an existing record. POST is typically used to add a new record, there's also DELETE and HEAD and a few other verbs. The verbs are usually determined by the service interface which should be provided by the documentation for the service.

Verbs can be used to differentiate operations on the same URL. For example, in the Item Service above I can use the Artist endpoint with GET and an ID to retrieve an artist. With POST a new artist is added, PUT updates an artist and DELETE removes an artist all from the same http://albumviewerswf.west-wind.com/api/artist Url. This is a common pattern and this approach lets you work with that.

Make it Better

Ok, so the above works, but you really don't want to scatter code like this into your application logic. It's a much better idea to abstract the service implementation into its own class with methods for each of the operations you want to perform.

Instead I recommend that when you access a service, you create a class that inherits from wwJsonServiceClient and then implement methods that make the service call. This can simply error handling and puts all the code related to the service - including any possible pre and post processing you might do for each call - into a single class.

So if we refactor our two service functions we might end up with a class like this:

******************************************************
DEFINE CLASS AlbumViewerService as wwJsonServiceClient
******************************************************

************************************************************************
*  Init
****************************************
FUNCTION Init()

DODEFAULT()

*** Ensure a serializer exists
this.CreateSerializer()

ENDFUNC

************************************************************************
*  GetAlbums
****************************************
FUNCTION GetAlbums()
LOCAL loAlbums

loAlbums = THIS.CallService("http://albumviewerswf.west-wind.com/api/albums","","GET")

*** Error message is already set
IF THIS.lError
   RETURN null
ENDIF   

RETURN loAlbums
ENDFUNC


************************************************************************
*  UpdateArtist
****************************************
FUNCTION UpdateArtist(loArtist)
LOCAL loArtist

this.oSerializer.PropertyNameOverrides = "Id,ArtistName,Description,ImageUrl,AmazonUrl"

loArtist = THIS.CallService("http://albumviewerswf.west-wind.com/api/artist",loArtist,"PUT")

*** Reset overridden properties
this.oSerializer.PropertyNameOverrides = ""

IF THIS.lError
   RETURN null
ENDIF   

RETURN loArtist
ENDFUNC


ENDDEFINE

By doing this you're abstracting all the logic required to deal with the service in this class, so the application doesn't need to see this low level interaction.

To use the service and call both methods the front end code you might use inside of your application looks like this:

*** Create our CUSTOM service
loProxy = CREATEOBJECT("AlbumViewerService")

*** call the wrapper method
loAlbums = loProxy.GetAlbums()

lnCount = loAlbums.Count
* Do something with the data


loArtist = CREATEOBJECT("Empty")
ADDPROPERTY(loArtist,"Id","2")
ADDPROPERTY(loArtist,"ArtistName","Accept")
ADDPROPERTY(loArtist,"Description","Old school German Metal band")
ADDPROPERTY(loArtist,"ImageUrl","http://cps-static.rovicorp.com/3/JPG_400/MI0001/389/MI0001389322.jpg?partner=allrovi.com")
ADDPROPERTY(loArtist,"AmazonUrl","")

loAlbum = loProxy.UpdateArtist(loArtist)
IF  loProxy.lError
   ? loProxy.cErrorMsg
   RETURN
ENDIF

? loArtist.Albums.Count

RETURN 

This is much cleaner as you are calling simple methods that describe what's happening. If something goes wrong, you have one place to go to look for the failures and the service becomes now reusable.

Summary

REST services are becoming much more common and although they are easy to call if you have an HTTP client and JSON serializer, wwJsonSerializer offers a more simpler, more abstracted and consistent way of doing it for you. Errors are caught and passed back to you and it reduces the service call to essentially a single line of code potentially with some extra configuration if you need to post data to the server that requires proper casing.

Give wwJsonSerializer a try. It's available in West Wind Client Tools 6.0 and West Wind Web Connection, but the Web Connection version is not quite up to date and is missing the CreateSerializer() method. This will be addressed in the forthcoming v6.05 release due out shortly.


Web Connection 6.0 RTM is here


No comments
Monday, April 4, 2016, 12:00:00 AM

It's been a long journey but I'm happy to announce that Web Connection 6.0 RTM is finally here.

To be honest it's taken a lot longer than I anticipated to do all the things I had set out to do with this release. The main brunt was to clean up the rough edges of the framework and the tooling and to make it much easier to build, deploy, update and manage Web Connection applications, which traditionally has been one of its weak points. Because there are lots of externalities in this process, it took a long time to fine tune and test and re-rest to make sure that these processes are streamlined as much as possible.

Functionally, most of the features I set out to provide were complete at the Southwest Fox conference last October, but all the plumbing around the UI updates and documentation updates and some of the infrastructure features ended up taking a long time to get just right and baked, and then run through some extensive testing in a few live applications. So here we are 5 months later…

But now that time is past and Web Connection 6.0 RTM is now available:

Web Connection 6.0 is a paid upgrade from previous versions unless you purchased Web Connection 5.x after April 1st, 2015 in which case you get to upgrade for free. If you previously purchased an upgrade to the full version of Web Connection 6.0, you should have received an update download notice via email – if not contact me privately. If you own any previous version of Web Connection (yes – all the way back to v1 in 1995!) you can take advantage of upgrading to the latest version of Web Connection. And if you hurry and purchase an upgrade before the end of the week you can still save 10% from the Beta discount that we offered during the pre-release cycle.

What's New?

Web Connection 6.0 is a major update to Web Connection. There are many, many new features and enhancements in this release. But at the same time, the new version has very little impact on backwards compatibility. There are very few, and very minor breaking changes between v5 and v6, so existing 5.x application will run under 6.0 with just about no changes which you can find in a section on breaking changes (look for the yellow box at the end of the change list) of the What's new topic in the documentation.

If you check out the What's New list in the documentation, you'll find that Web Connection 6.0 has a lot of new features and enhancements. There are a handful of big features like the new project system, the administration features and the new Scripting enhancements that I'll cover below. The rest are mostly API enhancements or extensions that are small enhancement to address very specific needs.

New Project System

Probably the most noticeable new feature is the new project system in 6.0. In the past Web Connection didn't really have the concept of 'project' as you simply worked out of the Web Connection folder or you come up with your own strategy for isolating your application specific code. While you could always structure your projects any way you wanted, Web Connection itself never imposed any structure to a project. It still doesn't force you into a particular structure but newly created projects now create a very specific and repeatable folder structure that builds a self-contained and portable structure that I call a 'project' (which is simply files and folders).

In the past I've always shied away from making this particular change, because managing paths in a development environment is difficult and because Web Connection apps depend on finding the the Web Connection code there were always issues. In this release I've overcome my reservations about this and addressed the pathing issues with generated configuration and startup files that can ensure proper dependency discovery. If you leave things in default locations things will just work – otherwise making a path change in config.fpw or setpaths.prg will get you on your way.

So in Web Connection 6.0 there's the concept of a project layout, where a new project is created into a self-contained folder structure. The project gets a top level folder, and subfolders for \web, \deploy and optionally \data. A related feature allows use of relative paths in the server configuration files so that you can basically set up your dependencies entirely based on relative paths. Because projects have a known structure the New Project and New Process Wizards can make assumptions where related files live and automatically and without user interaction create a fully portable application layout.

FolderStructure

By default new projects are created under a WebConnectionProjects folder, but that's not really important. Each project has its own self contained folder hierarchy that includes both the source code (Deploy) and Web directories (Web). Upon configuration the Web folder is configured as a Web virtual folder or root site, in place right in this folder.

There are few huge benefits to this:

  • Portable Application Structure
    This means you can copy your project to a new location and simply recreate/configure the virtual in the new folder and the application should simply work. This also means that you can take an project folder and push it to the server and with very minimal configuration make it work.
  • Easier Configurability
    Because the default project structure is known, the configuration Wizards can be much simpler and not ask a bunch of previously confusing (to new users) questions. For example for a new project you now specify the path to the main project folder and any script maps you want to create and that's it. No more pointing at DLLs, temp folder location or web paths – files just get copied and associations created.
  • Automated Configuration
    Because of the known dependency locations, it's now possible to auto-generate a configuration script that can completely auto-configure a typical Web Connection project. When you create a new project now Web Connection creates a PRG based script file that automatically compiles into your server EXE that creates the virtual directory, Application Pool, Scriptmap(s) and sets folder permissions for your project. When you deploy your application to a live server you can now do MyApp.exe CONFIG and this automated script will automagically configure your application. The script can be customizes as it's just a PRG file and based on the project relative default settings. You can customize this script if you want to use non-default paths, or add to it and handle application specific install features (creating data folders, mapping drives, copying data etc.).

To me personally this has been a big win. Over the weekend I moved over my Web Connection servers to Version 6.0 and in the process I moved the folder structures of several applications to the this model. I was able to take a generated configuration script and modify 3 lines of code to make it work against my new setup. I was able to let the script configure my Web and security setup in minutes rather than the typical 15-30 minutes.

Automatic Configuration Scripts

When you create a new Web Connection application, an automated Web server configuration script is generated as part of the creation process. The script creates configures IIS, creates scriptmaps, sets application paths and folder permissions. The script is generated as a PRG file called MyApp_ServerConfig.prg which also gets added to your project and can be invoked when you run the main application with a CONFIG parameter. Because it's a PRG file you can look at it, modify it and add functionality to it if you want.

The script is compiled into your server, so from a command prompt you can do:

MyApp.exe CONFIG

and that configuration script is fired.

The script can be configured interactively via MyApp.ini and the [ServerConfig] which allows you to specify the IIS instance to configure on, the virtual directory and scriptmaps to create:

[ServerConfig]
Virtual=testproject
ScriptMaps=wc,wcs,wcsx,wwd,tp1
IISPath=IIS://localhost/w3svc/1/root

The IIS path is the IIS Web site instance which above is 1 or the default instance. If you configure another site, you can look up the instance ID and replace it.

The configuration script is just FoxPro code, that you can look at and if you choose modify. The default implementation uses the default project structure to assume where things need to go.

A typical configuration script looks like this:

************************************************************************
*  Testproject_ServerConfig
****************************************
***  Function: Templated Installation routine that can configure the
***            Web Server for you from this file.
***
***            You can modify this script to fit your exact needs
***            
***    Assume: Called from TestprojectMain.prg startup code
***            with CONFGI parameter
***
***            Launch with from the Windows Command Line
***            Testproject.exe config
***            or with a specific IIS site/virtual
***            Testproject.exe config "IIS://localhost/w3svc/2/root"
***      Pass: lcIISPath  -  IIS Configuration Path (optional)
***                          IIS://localhost/w3svc/1/root
************************************************************************
LPARAMETERS lcIISPath
 
DO wwUtils    
SET CLASSLIB TO WebServer ADDITIVE
SET CLASSLIB TO wwXML ADDITIVE
 
IF !IsAdmin() 
   MESSAGEBOX("Admin privileges are required to configure the Web Server." + CHR(13) +;
              "Please make sure you run this exe using 'Run As Administrator'",;
              48,"Testproject Server Configuration")
   RETURN
ENDIF
 
*** Try to read from Testproject.ini [ServerConfig] section
loApi = CREATEOBJECT("wwAPI")
lcIniPath = FULLPATH("Testproject.ini")
lcVirtual = loApi.GetProfileString(lcIniPath,"ServerConfig","Virtual")
IF ISNULL(lcVirtual)
  lcVirtual = "TestProject"
ENDIF
lcScriptMaps = loApi.GetProfileString(lcIniPath,"ServerConfig","ScriptMaps")
IF ISNULLOREMPTY(lcScriptMaps)
  lcScriptMaps = "wc,wcs,wcsx,tp1"  
ENDIF
IF EMPTY(lcIISPath)
  lcIISPath = loApi.GetProfileString(lcIniPath,"ServerConfig","IISPath")
  IF ISNULLOREMPTY(lcIISPath)
     *** Typically this is the root site path
    lcIISPath = "IIS://localhost/w3svc/1/root"
  ENDIF
ENDIF  
 
 
*** Other relative configurable settings
lcVirtualPath = LOWER(FULLPATH("..\Web"))
lcScriptPath = lcVirtualPath + "\bin\wc.dll"
lcTempPath = LOWER(FULLPATH(".\temp"))
lcApplicationPool = "WebConnection"
lcServerMode = "IIS7HANDLER"     && "IIS7" (ISAPI) / IISEXPRESS
 
 
loWebServer = CREATEOBJECT("wwWebServer")
loWebServer.cServerType = UPPER(lcServerMode)
loWebServer.cApplicationPool = lcApplicationPool
IF !EMPTY(lcIISPath)
   loWebServer.cIISVirtualPath = lcIISPath
ENDIF
 
WAIT WINDOW NOWAIT "Creating virtual directory " + lcVirtual + "..."
*** Create the virtual directory
IF !loWebServer.CreateVirtual(lcVirtual,lcVirtualPath)   
   WAIT WINDOW TIMEOUT 5 "Couldn't create virtual."
   RETURN
ENDIF
 
 
*** Create the Script Maps
lnMaps = ALINES(laMaps,lcScriptMaps,1 + 4,",")
FOR lnx=1 TO lnMaps
    lcMap = laMaps[lnX]
    WAIT WINDOW NOWAIT "Creating Scriptmap " + lcMap + "..."
      llResult = loWebServer.CreateScriptMap(lcMap, lcScriptPath)        
    IF !llResult
       WAIT WINDOW TIMEOUT 2 "Failed to create scriptmap " + lcMap
    ENDIF
ENDFOR
 
 
WAIT WINDOW NOWAIT "Setting folder permissions..."
 
lcAnonymousUserName = ""
loVirtual = GETOBJECT(lcIISPath)
lcAnonymousUserName = loVirtual.AnonymousUserName
loVirtual = .NULL.
 
*** Set access on the Web directory -  should match Application Pool identity
SetAcl(lcVirtualPath,"Administrators","F",.T.)
SetAcl(lcVirtualPath,"Interactive","F",.T.)
SetAcl(lcVirtualPath,"NETWORKSERVICE","F",.T.)
* SetAcl(lcVirtualPath,"OtherUser","F",.T.,"username","password")
 
*** IUSR Anonymous Access
IF !EMPTY(lcAnonymousUserName)
    llResult = SetAcl(lcVirtualPath,lcAnonymousUserName,"R",.T.)
 
    *** No unauthorized access to admin folder
    lcAdminPath = ADDBS(lcVirtualPath) + "Admin"
    IF DIRECTORY(lcAdminPath)
          llResult = SetAcl(lcAdminPath,lcAnonymousUserName,"N",.t.)
    ENDIF
ENDIF
 
*** Set access on the Temp directory - should match Application Pool Identity
SetAcl(lcTempPath,"Administrators","F",.T.)
SetAcl(lcVirtualPath,"Interactive","F",.T.)
SetAcl(lcTempPath,"NETWORK SERVICE","F",.T.)
* SetAcl(lcTempPath,"OtherUser","F",.T.,"username","password")
 
*** COM Server Registration
IF FILE("Testproject.exe")
   RUN /n4 "Testproject.exe" /regserver
 
   *** Optionally set DCOM permission - only set if needed
   *** requires that DComLaunchPermissions.exe is available
   * DCOMLaunchPermissions("Testproject.TestprojectServer","INTERACTIVE")
   * DCOMLaunchPermissions("Testproject.TestprojectServer","username","password")
ENDIF
 
WAIT WINDOW Nowait "Configuration completed..."

As you can see Web Connection includes all the features needed to configure servers via code and the default generated PRG file just uses these features and applies it to the default project structure using the relative paths to figure out where things need to be configured.

The bottom line is this: You can move your project to a new location and run this script and your application will simply run. Likewise you can deploy an application to the server and assuming the Web Server is installed and has the base configuration required, you can run MyApp.exe CONFIG and your application is configured and ready to go.

This is a big change from previous versions were you either had to manually configure or else run the confusing Configuration Wizard. Best of all you can review the script above and get a good idea what configuration settings are made to get your application to run and you can modify the script to make changes or add additional configuration options.

Seeing this work for the first is almost magical – the script just runs for a few seconds and once done you can start your server, browser your site and you're off to the races.

Deploying Applications

Web Connection 6.0 can also help you deploy applications. Configuration is one thing but if you need to deploy and update applications you still have to get your files to the server.

Build.bat

First a new Web Connection project includes a build.bat file that creates a \build folder that contains all the project dependencies that a Web Connection application typically needs to deploy. The script picks up all dependent binary DLLs, the main binary file (your exe) and configuration file (.ini) and dumps those files into a the \build folder and zips everything up (if 7zip is in your path).

Here's what the build folder looks like:

BuildFolder

You can then push that file to the server and unpack it there for installation. You might have to add additional files for your installation, but this is a quick way to get all the dependencies that Web Connection typically needs in one quick swoop.

File Upload

The Web Connection .NET Handler includes a new feature to upload files into the application's \temp path on the server. Which is a poor man's way of copying files to the server if RDP file transfer is too slow (as it usually is) or you don't have an FTP server running on your site:

UploadFile

Just make sure you set the appropriate request limits in web.config to allow your Web server to accept large file uploads – temporarily if necessary (you can bump it back down after you're done uploading).

This is very useful if you need to upload say the Visual FoxPro runtimes, or your application or application data.

Updating Applications

Web Connection has always included tools to allow you to hotswap running applications by uploading new binary files and swapping them while the application is running. In the screen shot above the Upload Server Exe and Update Server Exe links accomplish this using the ExeFile and UpdateExeFile configuration keys. The way this works is that you can upload a new binary with the Upload link, which uploads the file to the file name specfied in UpdateExeFile. The Update Server link then shuts down all running servers and copies the new EXE file and restarts all servers.

This functionality is not new but it has been streamlined in Web Connection 6.0. The feature now works both with COM and file based Messaging. Additionally the hotswapping routine now gives you detailed info on the uploaded file including file size and version number so you can validate that the file made it and got swapped out. The upload process is now integrated into the Web Connection .NET Handler so you don't have to have a Web Connection server running in order to upload the file. This means you can use this feature to push an initial EXE to the server.

Finally new projects generate a bld_MyApp.prg file that automates this process. This bld file can be used to compile your project and show errors but if you do:

DO BLD_MyApp with .T.

The script will try to upload your binary file to your server. For this to work you have set a couple of values in the PRG file to point at the server where your application will be updated.

This is a great way to update the binaries in your application in a few minutes, causing a mere few seconds of downtime on your server only as your servers are hotswapped. I use this feature all the time to update my code online and it takes all but a minute to get new bianaries uploaded and hotswapped on my live server.

All of the features I've highlighted so far are administrative features meant to make it easier to manage Web Connection applications. For my own workflows I can tell you that these simple improvements have made the process of managing a Web Connection application considerably easier and more predictable. I hope you find these lifecycle features as useful as I have.

Modern Styling and Mobile Friendly Layout

Web Connection's existing base styling in Web Connection 5.0 was getting a bit - quaint. Initially created in the early 2000's the styling is showing it's date – it's looking decidedly old school. Not only that but the existing styling was completely custom, designed by a non-UI person (me) as best I could. The CSS and base markup was pretty stable, but by now the style is definitely very dated looking.

Web Connection 6.0 takes a different approach by relying – by default – on a popular, external CSS libraries designed by real designers and used widely by many developers for building Web applications.  Web Connection 6.0 uses Bootstrap with a somewhat customized base theme and FontAwesome for its default theming to provide a more consistent and much more popular design.

Web Connection uses these base libraries and then layers a relatively small customized UI on top of it. So when you create new pages in Visual Studio or use the default script templates, you get this base customized Bootstrap UI.

To be clear, this is just a default to get you started with. As before you can still build completely custom CSS and use it with your Web Connection applications, but if you are not graphically inclined you can build reasonably nice looking and mobile friendly applications relatively easily. Although Web Connection provides a slightly customized Bootstrap layout, all the default Bootstrap styles and components are available to you to use so it's possible to completely override our custom styling with your own.

Web Connection's customization essentially throws out Bootstraps default nav header,  which doesn't work very well on mobile and provides a slide in side menu via the bars on the left in the screen shot below:

TimetrakkerDesktop 

The default styling is mobile friendly as long as you stick to the general bootstrap design guidelines (using content containers and the grid layout). Here's the same form displayed on my iPhone 6. The first picture is plain the second with the animated slide in menu opened.

TimetrakkerMobileTimetrakkerMobileMenu

(this Web Connection sample is available on BitBucket if you want to check out a more comprehensive Web Connection sample app).

Visual Studio Page Templates

This sample, uses mostly the default styling that comes out of the box with Web Connection. If you are using Visual Studio, there a number of templates you can use to create various types of pages that include this default styling:

PageTemplates 

Use of Visual Studio is completely optional – you can use any editor you chose, but you do get a few benefits in Visual Studio. One are the templates above which make it really easy to create new pages that have all the standard styling and base page layout applied.

There's also a Web Connection Visual Studio Add-in that allows you to preview script and template pages  in your browser and that optionally allows you to show FoxPro code (if you have SourceFile="<file>" tag in your source file)

If you want to use other editors you totally can – I personally use Sublime Text with Matt Slay's Sublime FoxPro extension quite frequently, but any HTML editor like WebStorm, Atom, VS Code, Notepad++ will work as well. You won't get automatic templates, but if you want to cut and paste you can find the Visual Studio templates in the \VisualStudio\PageTemplates folder. If you drill into the various template folders you can find the default.* pages that contain the template and you can cut and paste from there.

As a side note you can also create your own Visual Studio templates – simply create a new zip file and use one of the other templates as a base and simply rename the manifest and final file names to create your own custom templates you can load in Visual Studio. Along the same lines if you're ambitious you can also create custom templates for Sublime, Atom or VS Code in much the same way.

MVC Style Development Improvements

Web Connection has long supported MVC style development using code based logic that can drive scripts or templates. This was one of the big features in Web Connection 2.0 in 1998. Nowadays MVC style frameworks like ASP.NET MVC, Ruby on Rails or Express on Node are the norm for server side Web development. In version 6.0 Web Connection gets a major updates for the Script and Template engines which now support master Layout pages that let you create a top level page template into which other content pages load, partial pages, which allow you to break large and complex pages into smaller pages and sections, which allow you to inject content from content pages into the master pages. These concepts are common in other MVC frameworks and they make it much easier to build complex applications as you don't have to repeat the same markup code on each page. Instead layout pages can take the brunt of page 'chrome' with various content areas that are filled by the content pages.

For those of you not familiar with scripts and templates in Web Connection, scripts and templates allow you to mix HTML markup and FoxPro code. The following demonstrates using a FoxPro SCAN to run through a cursor created by a controller method in the application's Process class:

ScriptPage

MVC style development is the most popular mechanism of development with Web Connection from what I've observed what people are using and so this is why there has been renewed focus on this development style.

JSON Service Improvements

Web Connection 6.0 now has a dedicated RestProcess class that makes it super easy to create JSON REST services that can take JSON input and generate JSON output. To create JSON endpoints you simply add methods to the RestProcess subclass and the methods simply accept a single parameter (which can be a complex object) and returns back a result value. The input parameter is parsed from JSON to a FoxPro object and the result value, object, collection or cursor is turned into an JSON (or XML) document.

The New Project and New Process Wizards now give you options to explicitly create a standard Web application or a REST service.

NewRestProcessClass2

wwJsonService now also includes .cPropertyNameOverrides which gets around FoxPro's limitation of reflecting property names only in lower case. Using a comma delimited list of property names you can override properties with proper case so you can create JSON objects that match a specific signature. wwJsonSerializer has been improved to handle collections better and has been updated to deal better deal with illegal FoxPro property names that are supported in JavaScript.

JSON Client Features

There's also a new wwJsonServiceClient class that makes it much easier to call JSON services. Using this class you can create methods for each service call and the methods can call a CallService base class helper method to actually call the Web based REST service method. You specify a URL and a single value or object parameter which is serialized and sent as JSON to the server. The JSON result that comes back is automatically parsed back into a FoxPro value or object. wwJsonSerializer now has a .FormattedOutput  flag to create pretty printed JSON output.

New wwHtmlHelpers

In Web Connection 5.50 Web Connection introduced a new library of HtmlHelpers that provide rich control functionality by way to of functions that can render certain controls and control values. These 'control' functions can be used with MVC style scripts or in code to generate HTML. The various input controls like textbox, textarea, checkbox, radio etc. all support databinding making it easier to bind data to these controls.

For example, here are the HtmlTextArea and HtmlBindingError helpers handling databinding for bound business object entity:

htmlhelper

While  the code is a bit verbose this code provides two way databinding support – the control is smart enough to display values from the model on a GET operation and display the posted back values when a POST/PUT operation is active. Likewise the checkbox and radio helpers know how to deal with the undermined state issue you run into with this painful HTML controls that don't differentiate between unchecked and not set values.

There are a number of new Html Helpers:

  • HtmlBindingError
  • HtmlDataTextBox (bootstrap datepicker)
  • HtmlErrorDisplay (bootstrap alert box)

There are also a number of updates to the HtmlDataGrid component.

Markdown Support

Markdown has become massively popular as an easy data entry format for HTML that can be written without actually writing HTML markup. Basically the text format uses very simple technical editing style markup directives using plain characters to describe common markup operations. This markdown text can then be parsed down to HTML. For most free form text data entry operations using plain text like this is much more efficient than using a rich HTML editor. If you're new to Markdown check out my Markdown Basics document I publish in West Wind Html Help Builder.

Web Connection now includes a markdown parser that can turn Markdown into HTML very quickly, using a .NET Markdown component (CommonMark). A new Markdownparser class provides for parsing Markdown to html and there's also an easy Markdown() function that can be easily used in templates (assuming you've loaded Markdownparser.prg).

For scripts and templates you can also directly write markdown text into the document markup using <markdown>This is **Markdown Text**</markdown> syntax in templates and scripts which is automatically expanded.

Web Connection .NET Handler Updates

The Web Connection Handler also has a number of big updates. The most noticable is that the UI now uses an _AdminTemplate that allows for styling of the admin page. Rather than the bland generic page you now get a branded page like the rest of the application:

Likewise error and status messages use the same _AdminTemplate.html to display their content. You can customize this template with your own branding in your applications if you choose.

Hot Swapping of Live Servers for both COM and File Based

The .NET Handler now also internally handles hot-swapping of both COM and File Based servers (previously this worked only with COM). The handler can shut down all servers, put requests on hold and hot-swap server exes, then restart instances. The upload operation is now part of the .NET Handler, rather than the Web Connection server which means you can actually do first time uploads to the server of a new server and you're not dependent on a functioning EXE server to make the actual copy operation.

Updated Documentation

The Web Connection documentation has been overhauled significantly.  Web Connection's docs are very big but a lot of the general purpose and intro content has been updated for version 6.0. A lot of old stuff has either been cleaned up and updated, or thrown out when irrelevant.

documentation

New Walk through Tutorials

There are also several new walk through tutorials and I recommend if you're new that you go through these to understand how Web Connection works.

Fast and Mobile Friendly

The online documentation and Help file has been re-built with the latest version of West Wind Html Help Builder and provides a much more modern,  livelier, interactive browsing experience. Topics load much faster and the new search feature is nearly instant, simply filtering the topic list. The new design has responsive layout that is mobile friendly, and no longer uses ugly frames.

Quick Search

The search is feature is a key improvement and I really recommend you take advantage of it.

The topic search box is very fast in looking up topics, as is all client side and simply filters the list of topcis based on your search terms. This means you can type partial text and it will match. Typing IIS for example quickly shows you the IIS and IIS Express configuration topics as well as a few knowledge base topics. Typing Expand will bring up ExpandTemplate(), ExpandScript() and ExpandPage() topics you can then jump to directly. This sounds pretty ordinary, but the search is nearly instant which makes it very efficient to browse the documentation quickly and find what you need.

If you use Web Connection, create a browser shortcut to the help file now and use it whenever you need to look up any Web Connection related function or feature. I guarentee you, it'll make you more productive – I know it has done that for me!

New Message Board Sample

With all the new MVC and scripting features in Web Connection 6.0 I needed to build another full featured sample, and this version's target was the Message Board which has been completely re-written from the ground up using business objects and MVC style development using Process/Controller classes and Scripts. The result is a is a much cleaner implementation that is mobile friendly and much smoother and faster compared to the old frames based implementation. The new message board includes support for the popular Markdown text format for writing messages and includes a rich editor that makes it easy to create messages with attached images and nicely formatted code snippets.

If you haven't stopped by the message board recently, check it out to see all the changes and please join the conversation – I hope to make this place more active and get more people involved to keep this community alive.

The message board is a sample application that ships with all of its source code in the Web Connection package and you're free to run this message board on your own site.

I've also made the source code available on Github to allow people to get involved and get access to the latest versions and potentially add features or submit pull requests.

And so much more

There is much more of course, but here are a few more small tidbits:

There's now better support for extensionless urls, which are common for REST services that use 'noun' based URL endpoints (like customer, order etc). There are a number of new jQuery plugins: jQuery-resizable, jQuery-watcher, a generic debounce() function that lets you throttle event handling on a timed basis. The .NET Handler's COM loading has been drastically improved. COM servers should now load up twice as fast as before.

The Request object has many new functions that provide better access to file uploads with multiple files, collecting form variables into a collection,  Request.UnbindFormVars() lets you unbind form variables to an object and provides you a collection of validation errors (if any). Response.ExpandScript/ExpandTemplate() now default to the current script if no template path is provided. You can also use ~\ virtual paths to specify a web relative script name (ie. ~\admin\CustomerAdmin.wcs).

wwDotnetBridge has many, many improvements to allow accessing more and more features of the .NET type system from Visual FoxPro. There's now support for calling any .NET method asynchronously including the ability to be called back when the method completes.

If you're interested in everything new check the large What's New Topic link:

There are tons of small enhancements. New support functions, small improvements to others and a lot of updates.

Summary

It's been a long journey involving a lot of very tedious work, but it's been worth it even if these changes end up being mostly for my own use of Web Connection. The vast majority of the changes in this release address very specific usability scenarios that I think will be beneficial to just about anybody using Web Connection today and the new version is bound to make the development and especially the administration process much easier overall.

I know in my own work with several customers and also with the Web Connection sample app and the message board, it's noticeably easier to build and deploy applications with confidence. Because of the automated scripts the process of getting applications online and keep them updated is simply quicker and easier.

The new HTML Scripting features – Layout Pages in particular - make HTML development so much easier than previously because you can reuse so much of the standard page chrome without having to repeat yourself. This greatly reduces HTML you have to write. Couple that with the default templates (in Visual Studio) and it's extremely quick and productive to get new pages into the application.

I haven't been this excited about Web Connection in years and I wish I had spent the time to do this a few years back, but I simply did not have the time and resources to do it. In the last year I've finally set aside the time to make these necessary changes happen and you're looking at the end result of it.

And I'm not done yet either. There are a still handful of improvements that didn't make the cut for the RTM release, but that will be showing up in updates in the near future, so stay tuned for more cool stuff coming down the line.

I hope you find these updates as useful as I have. Enjoy.

Aloha.


Outlook Email Automation–Watch out for Administrator Access


No comments
Sunday, November 29, 2015, 10:38:36 PM

I was looking into creating prefilled emails using Outlook Automation earlier today and ran into an unexpected snag. The requirement was to display a pre-filled email that contains recipient, subject, body and one or more attachments and then display the prefilled email in Outlook.

This is fairly straightforward to do using COM Automation:

TRY
    *** NOTE: This fails if you run as Administrator (rather than the active user)
    loOutlook = GETOBJECT(,"Outlook.Application")
CATCH
ENDTRY
 
IF VARTYPE(loOutlook) != "O"
   loOutlook = CREATEOBJECT("Outlook.Application")
ENDIF
 
IF VARTYPE(loOutlook) != "O"
   MESSAGEBOX("Couldn't create Outlook instance")
   RETURN
ENDIF
 
loItem = loOutlook.CreateItem(0)
loItem.Body = "Hello World"
loItem.Subject = "New Test Message"
loItem.Recipients.Add("rstrahl@west-wind.com")
 
*** Add you files to attach here
loItem.Attachments.Add("c:\sailbig.jpg")
 
loItem.Display()
 
RETURN

Note that Outlook – as most Office Applications – is a Singleton object that expects to run only one instance. So if Outlook is already running you can use CREATEOBJECT() to create a new instance. Instead, you have to attach to an already running instance using GETOBJECT().

Gotcha: Running as Administrator makes GETOBJECT() fail

I typically run FoxPro as an Administrator because I frequently build COM objects, which requires that you run as a full administrator in order to write COM registration to the registry.

However, when running the above code, it turns out the GETOBJECT() call to capture Outlook.Application fails. It works fine when you run as a non-admin user or even as an Admin user when User Account Control (UAC) is enabled on the machine and your account is effectively running as a non Admin account.

But when you explicitly run as an Administrator either using Run As Administrator when you start the app, or from ShortCut properties, or if you have UAC disabled on the machine, you'll find that the GETOBJECT() call fails with:

OLE error code 0x800401e3: Operation unavailable.

The reason for this is that when you run as Administrator you are actually running a different user account (Administrator – duh) and that account can't actually access the running instance of Outlook that is already running on your desktop. The only workaround I could find is to ensure both Outlook and your application run in the same execution context. So it works if you run your app without administrative rights, or if you run your app as administrator as well as Outlook.

As you might expect it took me a long time to figure out WTF was going on here. According to all examples I've seen Outlook should be accessible with GETOBJECT(). It wasn't until I tried using wwDotnetBridge and COM Interop doing the same thing with .NET and getting the same response in FoxPro but not in LinqPad when I realized it must have something to do with the actual runtime environment. Sure enough – once I started VFP without Run as Administrator option, the code works.


String Tokenizing based on StrExtract()


2 comments
Thursday, November 19, 2015, 6:18:43 PM

I've been building a number of solutions lately that relied heavily on parsing text. One thing that seems to come up repeatedly is the need to split strings but making sure that certain string tokens are excluded. For example, a recent MarkDown parser I've built for Help Builder needs to make sure it first excludes all code snippets, then performs standard parsing then puts the code snippets back for custom parsing.

Another scenario is when Help Builder imports .NET classes and it has to deal with generic parameters. Typically parameters are parsed via commas to separate them, but .NET generics may add commas as part of generic parameter lists.

Both of those scenarios require that code be parsed by first pulling out a token from a string and replacing it with a placeholder, then performing some other operation and then putting the the original value back.

For me this has become common enough that I decided I could really use a couple helpers for this. Here are two functions that help with this:

************************************************************************
*  TokenizeString
****************************************
***  Function: Tokenizes a string based on an extraction string and
***            returns the tokens as a collection. 
***    Assume: Pass the source string by reference to update it
***            with token delimiters.
***            Extraction is done with case insensitivity
***      Pass:  @lcSource   -  Source string - pass by reference
***             lcStart     -  Extract start string
***             lcEnd       -  Extract End String
***             lcDelimiter -  Delimiter embedded into string
***                            #@# (default) produces:
***                            #@#<sequence Number>#@#   
***    Return: Collection of tokens
************************************************************************
FUNCTION TokenizeString(lcSource,lcStart,lcEnd,lcDelimiter)
LOCAL loTokens, lcExtract
 
IF EMPTY(lcDelimiter)
   lcDelimiter = "#@#"
ENDIF
 
loTokens = CREATEOBJECT("Collection")
 
lnX = 1
DO WHILE .T.
    lcExtract = STREXTRACT(lcSource,lcStart,lcEnd,1,1+4)
    IF EMPTY(lcExtract)
       EXIT
    ENDIF
    loTokens.Add(lcExtract)
    
    lcSource = STRTRAN(lcSource,lcExtract,lcDelimiter + TRANSFORM(lnx) + lcDelimiter)
    lnx = lnx + 1 
ENDDO
 
RETURN loTokens
ENDFUNC
*   TokenizeString
 
************************************************************************
*  DetokenizeString
****************************************
***  Function: Detokenizes an individual value of the string
***    Assume:
***      Pass:  lcString    - Value that contains a token
***             loTokens    - Collection of tokens
***             lcDelimiter - Delimiter for token id
***    Return: detokenized string or original value if no token
************************************************************************
FUNCTION DetokenizeString(lcString,loTokens,lcDelimiter)
LOCAL lnId, loTokens as Collection
 
IF EMPTY(lcDelimiter)
  lcDelimiter = "#@#"
ENDIF
 
DO WHILE .T.
    lnId = VAL(STREXTRACT(lcString,lcDelimiter,lcDelimiter))
    IF lnId < 1
       EXIT
    ENDIF   
    lcString = STRTRAN(lcString,lcDelimiter + TRANSFORM(lnId) + lcDelimiter,loTokens.Item(lnId))
ENDDO
 
RETURN lcString
ENDFUNC
*   DetokenizeString

TokenizeString() basically picks out anything between one or more start and end delimiter and returns a collection of these values (tokens). If you pass the source string in by reference the source is modified to embed token place holders into the the passed string replacing the extracted values.

You can then use DetokenizeString() to detokenize either individual string values or the entire tokenized string.

This allows you to basically work on the string without the tokenized values contained in it which can be useful if the tokenized text requires separate processing or interferes with the string processing of the original string.

An Example – .NET Generic Parameter Parsing

Here's an example of the comma delimited list of parameters I mentioned above. Assume I have a list of comma delimited parameters that needs to be parsed:

DO wwutils
CLEAR
 
lcParameters = "IEnumerable<Field,bool> List, Field field, List<Field,int> fieldList"
? "Original: " 
? lcParameters
?
*** Creates tokens in the lcSource String and returns a collection of the 
*** tokens.
loTokens = TokenizeString(@lcParameters,"<",">")
 
? lcParameters
* IEnumerable#@#1#@# List, Field field, List#@#2#@# fieldList
 
FOR lnX = 1 TO loTokens.Count
   ? loTokens[lnX]
ENDFOR
?
? "Tokenized string: " + lcParameters
?
? "Parsed parameters:"
*** Now parse the parameters
lnCount = ALINES(laParms,lcParameters,",")
FOR lnX = 1 TO lnCount
   *** Detokenize indvidual parameters
   laParms[lnX] = DetokenizeString(laParms[lnX],loTokens)
   ? laParms[lnX]
ENDFOR
 
?
? "Detokenized String (should be same as original):"
*** or you can detokenize the entire string at once
? DetokenizeString(lcParameters,loTokens)

IEnumerable<Field,bool> List, Field field, List<Field,int> fieldList

Notice that this list contains generic parameters embedded in the < > brackets so I can't just run ALINES() on this list. The following code strips out the generic parameters first, then parses the list then adds the token back in. The Tokenization allows picking out a subset of substrings and replace them with tokens so additional parsing can be done without the noise of the generic parameters in brackets that would otherwise break the parse logic. This is quite common in text parsing where you often deal with patterns that you are matching – and trying to avoid edge cases where the pattern breaks down. This is where I've found tokenization super useful.

Specialized Use Cases

In Help Builder I have tons of use cases where this applies as documents are rendered: In parsing code snippets out of documents for parsing because the code snippets are rendered 'raw' while the rest of the document gets rendered as encoded context. Links that require special fixup before being embedded into the document – the tokenization allows easy capture of the links, replacing the the captured token value and writing it back out with a new value. In Web Connection the various template parsers do something very similar with expressions and code blocks that get pulled out of the document then injected back in later as expanded values.

There are lots of variations of how you can use these tokens effectively.

This isn't the sort of thing you run into all the time, but for me it's been surprisingly frequent that I've had to do stuff like this and while this isn't terribly difficult to do manually, it's very verbose code that's ugly to write as part of an application. These two functions greatly simplify application code as it's shrunk to a couple of simple helper functions.

Maybe some of you will find this useful though…




© Rick Strahl, West Wind Technologies, 2004 - 2017