Getting started using DPT with Python

Introduction

This document describes how to create a Python wrapper for the DPT API using SWIG.

This document assumes that you do not have a Visual C++ compiler fit to build the Python wrapper. The free download versions are not fit. If you do have a fit Visual C++ version then you may be happy with 'here are instructions for installing SWIG; some SWIG interface files; and a Makefile that could be converted to use Visual C++ to create a Python wrapper for the DPT API'.

History

At about the time DPT version 1.0 was released I was working on a chess database program and wishing for the ability to manipulate indexes in the style of Model 204 User Language. This work was being done on a FreeBSD installation with regular but not frequent ports to Microsoft Windows. The DPT website (www.dptoolkit.com) said the DPT API could be made available if anyone wanted it. The DPT Host and Client run under Wine (www.winehq.com) on FreeBSD so the same would be true for the DPT API. When the time came to tackle the task the decision was to build a Python wrapper for the DPT API with SWIG.

So one might end up running versions of Python for Microsoft Windows under Wine on FreeBSD or Linux or OS X, where native versions of Python are available, just to use DPT API in these environments.

Compiling the DPT API was the major problem because some of the runtime libraries used are not supplied with the versions of Visual C++ available for free download. My diversion into Visual C++ version 8 took up more time than it deserved. The dptdb.lib build (see projects\dptdb.mak in the DPT API sources) was first done successfully with Visual C++ version 6, using the Winelib runtime libraries on a Microsoft Windows installation, but Wine documentation (www.winehq.com) advice is that this be for development only. The solution was to use the MinGW / MSYS combination described in the SWIG documentation as an alternative to building SWIG from source using Visual C++.

MinGW uses the gcc compiler and it turned out that a fairly small number of changes to the DPT API source were needed to modify statements unique to the Visual C++ dialect before a successful build of the Python wrapper using gcc occurred. These are in DPT API version 2.06.

Building the Python wrapper was easy, allowing for this being my first use of SWIG, except for having to add a few small C++ classes to the SWIG interface file as C or C++ programming is new for me as well.

Prerequisites

The version numbers quoted are those I used. Check the package documentation for compatibility if you have a later version. Earlier versions may or may not work.

DPT_V2R06_SOURCE_DBMS (www.dptoolkit.com)

MinGW (www.mingw.org)

MSYS 1.0.10 (www.mingw.org)

Python 2.4.4 (www.python.org)

SWIG 1.3.39 (www.swig.org)

See the list of files in Instructions when choosing the files to download from www.mingw.org.

SWIG 1.3.39 or later is required for compatibility with Python 3.

Instructions

The C drive is used throughout these instructions. Although I have done an installation using these instructions exactly, except for the version of SWIG, I use an installation with most of the stuff on the D drive. These instructions have been updated to refer to SWIG 1.3.39 (from SWIG 1.3.31). The default build uses SWIG 1.3.39.

Make folder c:\mingw

Copy these files to c:\mingw

binutils-2.15.91-20040904-1.tar.gz

gcc-core-3.4.2-20040916-1.tar.gz

gcc-g++-3.4.2-20040916-1.tar.gz

mingw-runtime-3.11.tar.gz

mingw-utils-0.3.tar.gz

w32api-3.8.tar.gz

Run MSYS-1.0.10.exe to install Minimal SYStem in c:\msys

Answer 'y' to the 'Do you wish to continue with the post install?' question

Answer 'y' to the 'Do you have MinGW installed?' question

Type 'c:/mingw' in answer to the 'Where is your MinGW installation?' question. The response is that the installation cannot be found and that installation will continue as if the answer to the 'Do you have MinGW installed?' question were 'n'. However it does say what to do later once the MinGW installation has been completed.

Press ENTER to continue.

You are warned to rename the version of make in the MinGW-1.1 installation to 'mingw32-make.exe'.

Press ENTER to continue.

Press ENTER to continue when the pause command is echoed to the terminal.

The installation completes.

Start an MSYS session by clicking the msys entry in the MinGW group on the Start menu.

Type the command 'less /etc/fstab.sample'. The windows path to the /etc folder is c:\msys\1.0\etc if you prefer to use your favourite text editor to view the fstab.sample file. A sample fstab file is displayed (quit from the display by typing 'q'). What is needed right now is an fstab file containing just the 'c:/mingw /mingw' line. Create it or copy and edit fstab.sample. Check the result with the command 'less /etc/fstab'.

Type the command 'cd /mingw'.

Type the command 'ls'. The six files copied earlier to c:\mingw should be listed. If the list contains no files it is likely that the c:\msys\1.0\mingw folder is being listed which means the fstab file is wrong somehow.

Complete the installation of MinGW by typing the following commands in the MSYS session. A list of the files extracted is generated.

tar -xvzf binutils-2.15.91-20040904-1.tar.gz

tar -xvzf gcc-core-3.4.2-20040916-1.tar.gz

tar -xvzf gcc-g++-3.4.2-20040916-1.tar.gz

tar -xvzf mingw-runtime-3.11.tar.gz

tar -xvzf mingw-utils-0.3.tar.gz

tar -xvzf w32api-3.8.tar.gz

Logout of the MSYS session by typing 'logout' or 'CTRL + d'.

You will now find that the warning to rename make.exe does not apply for this version of MinGW because the file does not exist.

Extract files from swigwin-1.3.39.zip to c:\swigwin-1.3.39.

Run python-2.4.4.msi to install Python 2.4 (many of you will have done this as you wish already).

I chose the 'just for me' option; install everything; and the 'compile .py' files option on the Advanced button while doing the steps in these instructions.

Building the SWIG wrapper for Python

Extract files from DPT_VnRnn_SOURCE_DBMS.ZIP (VnRnn > V2R12).

These instructions assume the build is done in the extracted directory DPT_VnRnn_SOURCE_DBMS.

Start an MSYS session by clicking the msys entry in the MinGW group on the Start menu.

Change the current working directory with

'cd DPT_VnRnn_SOURCE_DBMS/”sample projects”/”DPT with Python”/mingw_port'

or similar depending on location of DPT_VnRnn_SOURCE_DBMS or build environment.

Type the command

'dptsetup.py'

to build the Python wrapper. After a successful build the folder python (in mingw_port) will exist containing the following files:

_dptapi.pyd (about 5.2Mb with compilation option -O0)

dptapi.py (about 77kb)

dptapi.py and _dptapi.pyd form the Python wrapper for the DPT API described in the DPT documentation.

Type the command

'dptsetup.py install'

to install the Python wrappers in site-packages/dptdb. After a successful install two additional files are created in folder python and all are copied to site-packages/dptdb:

__init__.py

version.py

Alternatively type the command

'dptsetup.py install'

to do it all in one go.

Logout of the MSYS session by typing

'logout' or 'CTRL + d'.

You are now finished with MSYS MinGW and SWIG until you want to install a new release of DPT API.

Then you may wish to use the command

'dptsetup.py uninstall'

to remove this release of DPT API from site-packages before installing the next.

The Makefile in mingw_port assumes the folder structure introduced at V2R12 and the use of Python 2.5 but some flexibility is provided. Thus the command

'c:python24/pythonw dptsetup.py install'

installs the wrappers if just Python 2.4 is installed.

The command

'dptsetup.py clean'

removes all files and folders created in mingw_port by the build process.

Using the Python wrappers for DPT

Refer to the DPT API documentation for details of how to use the API. You will have to translate the C++ to Python.

The dptapi module wraps the API classes (APIDatabaseServices for example). The underlying classes (DatabaseServices for example) are not available in the Python interface provided by dptapi.

Note that none of MSYS, MinGW, SWIG, or even DPT Host and DPT Client need be installed to use the Python wrappers.

dptapi module

The Python SWIG interface file dptapi_python.i deals with the following problems.

Four constants have been renamed because the DPT names, with their similarity to the User Language names, are not acceptable Python names.

FD_POINT$ becomes FD_POINT

FD_SET$ becomes FD_SET

FD_FILE$ becomes FD_FILE

FD_NOT_POINT$ becomes FD_NOT_POINT

One class method has been renamed to prevent it being ignored and thus not available.

The operator! method in the APIFindSpecification class becomes the __invert__ method of the wrapper class

Three additional classes are defined to solve specific problems in calling particular DPT API methods. They are BoolPtr; IntPtr; and StdStringPtr. These were defined as the need for them arose and it is reasonable to suppose that other cases await discovery.

BoolPtr - the known use for this is when opening a file in deferred update mode.

bptr = dptapi.BoolPtr()

dbserv = dptapi.APIDatabaseServices(...)

cs = dptapi.APIContextSpecification(...)

...

oc = dbserv.OpenContext(cs, False, bptr, ..., ...)

IntPtr and StdStringPtr - the known use for these is in a APIReadableRecord.GetNextFVPair(...) call.

fieldname = dptapi.StdStringPtr()

count = dptapi.IntPtr()

fv = dptapi.APIFieldValue()

v = []

while record.GetNextFVPair(fieldname, fv, count):

v.append((fieldname.value(), fv.ExtractString()))

The methods AdvanceToNextFVPair(...) LastAdvancedFieldName(...) and LastAdvancedFieldValue(...) added at version 2 release 6 between them allow GetNextFVPair(...), and thus IntPtr and StdStringPtr, to be avoided.

The APIRoundedDouble class has been extended with methods to convert float values for read and write operations on the sequential files. These methods are more conveniently located than the DPT originals. Their names are pyCastToRoundedDouble and pyCastToString.

When dptapi is first imported APIRoundedDouble_SetNumRangeThrowOption(True) is called so that attempts to assign a value that is out of range to a float raise an exception rather than silently set the value to 0.

Sample Code

Much unnecessary stuff has been removed at V2R14 following the 1 step load changes (indeed most of it does build any more). This section will be changed when sample use of 1 step load have been prepared.

Folder dptdbsamples contains module rmstorerecordapi, demonstrating the use of the dptapi module to do deferred updates and rmstorerecordssapi to demonstrate single step deferred updates. Folder dptdbtests contains Python scripts that use rmstorerecordapi and rmstorerecordssapi to do deferred updates and a UL script to view the results in a DPT Client session. Run dptdbtests/pydpt.py to see the effect.

The DU_FORMAT_NOPAD || DU_FORMAT_NOCRLF and DU_FORMAT_DEFAULT options are demonstrated along with non-deferred updates and single step deferred updates.

If RoundedDouble_SetNumRangeThrowOption(False) is called any attempt to assign an illegal value to a float results in the assignment of 0 (to emulate UL behaviour as stated in DPT documentation). The sample code does, but need not, detect EOF on numeric deferred update sequential files by catching the exception generated trying to convert a zero length string to a float.

The sample code takes about 40 seconds to do the updates non-deferred; about 20 seconds using DU_FORMAT_DEFAULT and the dptapi module; and about 14 seconds in all cases using the dpt module. The chess database (see History) run used to test deferred updates without intermediate sequential files takes 8.5 minutes compared with 15 minutes taken by the dptapi module in DU_FORMAT_DEFAULT mode; so perhaps the sample runs are not big enough to show up any differences.

Contact

The author of this guide to getting started using DPT with Python can be contacted at roger.marsh@btinternet.com