Release Notes DPT V3R0

Release Notes for DPT Version 3.0

Release date: 29th April 2011

Dedicated to William and Kate, future king and queen, married today.


Version 3.0 contains several large chunks of functionality which have been in development for years, and have now finally made it up to production standard and off the workbench. There is no unifying theme except to make all this good stuff available.

For anybody who has been trying out the beta versions, the main new things since the last one are the File Wizard and Java API, which are included for the first time.

Contents


Database Binary Large Object Fields (BLOBs) and "Table E"

DPT now allows string fields in the database to be arbitrary in length, an approximate emulation of the Model 204 feature.

A BLOB field is created by using the new field attribute:

DEFINE FIELD REMARKS.TEXT (BLOB)

The DBA guide contains a detailed section describing the implementation and usage of BLOB fields on DPT.

File structure compatibility
This is mentioned because on Model 204 files needed to be recreated to get a table E, and then became inaccessible to earlier versions. On DPT the maintenance of BLOB fields does involve some new FCT information, but the first time a file is opened with version 3.xx these items are configured. There is no explicit action you need to take, such as recreating the file, and the FICREATE parameter is still "B" as it was for version 2. Files that have been opened with version 3.xx can still be used with version 2.xx. Records with BLOB data when accessed with older DPT versions just appear to contain the BLOB descriptor, i.e. a cryptic string item.


File Wizard Utility

This is a simple but useful new application for browsing DPT database files just as if they were text files in notepad. No more having to log onto a DPT host and issue line commands just to look at a few records of data. The application was written to test and to showcase the new Java API (see below), so it provides the ability to perform most database functions one way or another, including field and file admin, but it won't be offended if all you do is use it to browse data and change the odd record here or there.

Ideally you would associate dptwizard.exe with ".DPT" files, which allows double-click on files in Windows Explorer to open them.

The File Wizard is a Java application, wrapped as a Windows executable. In the unlikely event that you don't have a Java runtime virtual machine installed, you'll be prompted to get one when invoking the application for the first time.

Full details are contained in the File Wizard User Guide, which is also accessible via F1 inside the application. The picture below shows the default appearance after double clicking on the sample TEAMS.DPT file which comes with the download.


Fast Unloads, Loads and Reorgs

This version contains a new group of features designed to speed up routine file loading and maintenance operations on DPT, much like the Sirius tools do on Model 204. Despite having the same purpose, the DPT features are controlled by entirely different commands, command options and parameters. They are not an emulation of Sirius fast load/unload.

The DPT manuals contain detailed information on the new commands, parameters etc., with the most important new chapters being as follows.

Benchmarks
Here are some rough timings for the same 5 million records that were loaded in previous load benchmarks, e.g. V 2.14. All times in seconds.

Points of interest with this data:

# Run type Data Indexes Total Remarks
-Unloads
1 Extract data via formatted User Language image 146 - 146 FLOAT fields in FLOAT image items (not possible from M204)
2 (V3) Fast unload (compressed) 23 20 43 -
3 Extract data via User Language PAI 602 - 602 See sample program in DEMOPROC
4 (V3) Fast unload (PAI format) 64 44 108 Data extract equivalent to #3
- Loads
5 User Language image from #1, single-step deferred index updates 336 116 452 Index time is just the final chunk merges plus btree builds
6 (V3) Fast load of data from #2, single-step deferred index updates 181 121 302 ""
7 (V3) Fast load of everything from #2 37 82 119 Index time is TAPEI reads plus btree builds
8 User Language "PAI" input from #3, single-step deferred index updates 1798 115 1913 as #5
9 (V3) Fast load of data from #4, single-step deferred index updates 244 121 365 ""
10 (V3) Fast load of everything from #4 101 120 221 as #7


Java API and Other API Enhancements/Changes

Java wrappers
The Database API is now accessible from Java using the downloadable JAR of API classes which mirror the base C++ API. This whole area is covered in the Database API guide.

Sample source code is available as a separate download of the File Wizard project (see above).

Database queries issued in text form
It is now possible to do arbitrarily complex database finds via a new "User Language-like" interface of the API, as an alternative to constructing them in one of the other C++ style ways. This function is useful when end users are offered the ability to issue database queries via API programs. An example is shown below (and more details are available here).

void fd (APIDatabaseFileContext file) 
{
//Piecemeal style:
  APIFindSpecification cb("COLOR", FD_EQ, "BLUE");
  APIFindSpecification mf("MAKE", FD_EQ, "FORD");
  APIFindSpecification mt("MAKE", FD_EQ, "TOYOTA");
  APIFindSpecification b2("BODY", FD_EQ, "2DR");
  APIFindSpecification b4("BODY", FD_EQ, "4DR");

  file.FindRecords(cb & (mf | mt & (b2 | b4)));

//Text style:
  file.FindRecords("COLOR = BLUE AND (MAKE = FORD OR MAKE = TOYOTA AND (BODY = 2DR OR 4DR)))";
}

Access to BLOBs

Otherwise BLOB fields behave just like STRING fields, only longer. Attempts to use values >255 where not allowed (everywhere except BLOB fields) will throw exceptions at a slightly later stage than before - i.e. no longer at parameter construction time. You can also access the BLOB descriptor if you want. See the API guide section on field access.

Miscellaneous API


Other Version 3.0 stuff unrelated to the above

Miscellaneous fixes

  • REDEFINE FIELD from FLOAT to INVISIBLE now converts all data values correctly.
  • Follow-on fix from V2.30 where expressions containing patterns were made to treat right bracket as ending the expression rather than being part of the pattern. That fix should also have been applied to find expressions too, such as FD (NAME IS LIKE *).
  • When sorting by a multiply-occurring field without the EACH option, and later in the same request accessing occurrences after the first, the later occurrences do not sometimes appear to be missing from the record.
  • No more "sick record" error if you sort a set containing non-existent records.
  • Better error message now with certain invalid %variable INITIAL expressions previously allowed through but evaluating to null string.

    Miscellaneous enhancements

  • The pattern matcher, as used across the system in numerous places, can now process certain simple leading wildcard searches much more efficiently, by translating them internally into the equivalent of a $SCAN function. '*ABC', 'ABC*' and '*ABC*' are examples. This is obviously useful in any case but is the only way to give the system a fighting chance when such patterns are used (you know you want to) in table B searches against BLOB fields.
  • The "batch2" userid can be in mixed case.
  • The webserver built-in =COMMAND script can be disabled with a new access-control flag.

    Multi-CPU support

  • A new read-only system parameter, NCPUS.
  • General deferred update processing can now make use of the parallelism enabled as part of the fast load project, via LOADTHRD.

    Sequential files - NOCRLF option

  • Previously User Language IMAGE READ could not always handle input files containing floating point data, because of the possibility of CRLF (X'0D0A') sequences embedded within the FP bit patterns. This is no longer a restriction so long as fixed length records and the new NOCRLF option are used.

    Demo code - fun stuff

  • The emulator download now contains a full-screen User Language game to steal away some of your free time. Just type "MINE" at the chevron (or use the shortcut in the installed MINEPROC directory). It's mouse-sensitive - click your way to the top of the high score table!


    Version 2.xx Quick Recap

    For anyone who wasn't paying close attention during the lengthy 2.xx era, lots of improvements were made to emulation accuracy, quite a few bugs were reported and fixed, and some new features were added too. Here is a short list of the top 10 most exe-catching new features from 2.01 to 2.30, in chronological order.

    Full details can be found in the V2.xx release notes.