Components History

Old history Components Top
 

3 December 2019

dpt3.0-dptdb-0.7.3

The project structure is improved.

The only things installed are the modules built with SWIG, a minimal __init__.py, and versions.py containing some constants recording version information.

The setup script does a 'wine python -V', if not on Microsft Windows, to verify the Python version: it seems failure of this command under Wine is a sure sign the dptdb package would not be usable, even if build and install is successful.

The dptdb package is not usable with Python 3.8, even if installation succeeded (it does on Microsoft Windows but not on at least one FreeBSD configuration).

Problems of this kind seem absent when installing on Pythons earlier than 3.6.

Pythons 3.6 and 3.7 under Wine seems fine on FreeBSD 11.3 i386 but not on FreeBSD 12.1 amd64.

 
 

3 December 2019

dpt3.0-dptdb-0.6.7

The changes in dpt3.0-dptdb-0.7.3 are in dpt3.0-dptdb-0.6.7 too.

 
 

24 November 2019

dpt3.0-dptdb-0.7.2

Thanks to askubuntu.com/questions/678288/how-to-install-python3-in-wine for a simple workaround for this problem which started with Python 3.4.

The setup script is modified to work with recent Pythons installed in default 'user-only' locations: between Python 3.4 and Python 3.5 the default changed. On Wine assume Python was installed in [user]/AppData if [user]/AppData/Local/Programs does not exist.

While doing this, spotted and fixed the setup script problem which prevented builds as mentioned in earlier notes: it was not OS specific, let alone release specific.

The default SWIG is changed from swigwin-2.0.8 to swigwin-4.0.1 because builds with Python 3.8 work with the newer version but not the older one.

Attempts to install dptdb in Pythons 3.7, and 3.8, under Wine on FreeBSD run into a Wine error: the error is different on FreeBSD 11.3 i386 (64-bit hardware) with the emulators/wine port, and FreeBSD 12.1 amd64 with the emulators/i386-wine port.

Attempts to install dptdb in Python 3.6 on FreeBSD 12.1 amd64 with the emulators/i386-wine port run into a Wine error.

 
 

24 November 2019

dpt3.0-dptdb-0.6.6

Changes in dpt3.0-dptdb-0.7.1 and dpt3.0-dptdb-0.7.2 are applied to dpt3.0-dptdb-0.6.5, which is intended for use with versions of MinGW earlier than MinGW-6.3.0, such as MinGW-5.3.0, where dpt3.0-dptdb-0.7.2 does not work.

At MinGW-5.3.0 builds with Python 3.8 or Python 3.7 do not work.

At MinGW-5.3.0 builds with Python 3.6 work.

At MinGW-6.3.0 builds with Python 3.7 or Python 3.6 work. However the presence of three warnings that 'throw will always call terminate' implies the build should not be used because patches in the dpt3.0-dptdb-0.7 series are there to prevent throw calling terminate.

 
 

11 November 2019

solentware-base-4.0

Major changes to make interface to dptdb the same as the interface to apsw, bsddb3, and sqlite3. Projects which use solentware-base no longer need to define one class for dptdb access functions and another for the rest. One function class and mixers to link it with apsw, bsddb3, dptdb, and sqlite3, will do.

A wrapper for dptdb's APIFoundSet class supplies the '&', '^', and '|', operators. Applying these operators to a pair of APIFoundSet wrappers creates an APIRecordList wrapper.

A wrapper for dptdb's APIRecordList class supplies the '&', '^', '|', '&=', '^=', and '|=', operators.

The equivalents of APIFoundSet and APIRecordList in the class for apsw, bsddb3, and sqlite3, already have the '&', '^', '|', '&=', '^=', and '|=', operators. These classes are changed to use wrappers to fit what has to be done for dptdb. (Strictly speaking, previous versions of solentware-base only had an equivalent for APIRecordList: the opportunity to provide an equivalent for APIFoundSet was taken.)

Cursor first, last, next, and prev, operations on foundsets and recordlists which contain no segments return None rather than raise an IndexError exception. This fits with the behaviour when all segments in a foundset or recordlist contain no records, where None is returned.

All ...Primary and ...Secondary classes in the apsw, bsddb3, and sqlite3, interfaces are removed: being replaced by direct use of DB() object methods in Berkeley DB and Connection().cursor() operations in SQLite3 (via either apsw or sqlite3).

The make_recordset_* methods defined for the Berkeley DB and SQLite3 parts of solentware-base are renamed recordlist_* and added to the dptdb part of solentware-base.

The recordlist_range() method is modified to handle ranges limited at both ends.

Databases created on Berkeley DB before solentware-grid-4.0 put each Berkeley DB database in a separate file while databases created on Berkeley DB at solentware-grid-4.0 put all the Berkeley DB databases in the same file.

The naming convention for SQLite3 tables and Berkeley DB databases is changed at solentware-base-4.0 from earlier versions.

The segment size, in bytes, of a non-dptdb (DPT) database can be set, at creation, to a value between 500 and 8192 inclusive. The default is 4000. The segment size of an existing database cannot be changed.

A segment record on a non-DPT database contains either a list of record numbers, or a bitmap of record numbers, relative to the base of the segment. Before solentware-base-4.0 these two types of segment record were held on separate Berkeley DB databases. They are now held on the same Berkeley DB database. The same table is, and was, used on SQLite3 databases.

SQLite3 and Berkeley DB databases created before solentware-base-4.0 must be upgraded using the tools.upgrade_3_to_4 module. An application will usually provide a wrapper for this tool which specifies the FileSpec to use.

Support for read-only access to dBase files, and text files treated as 'one record per line' databases is removed. This includes csv and zipped text files.

Project directory structure changed from solentware-base to solentware-base/solentware-base where the implementation moves to solentware-base/solentware-base and project descriptions and setup.py remain in solentware-base.

The terms DPT and dptdb are used in these notes about solentware-base-4.0. dptdb is used when reference to the Python module built with SWIG to expose the DPT API is implied.

Some history detail may be relevant. Solentware-base was originally just for Berkeley DB and the ...Primary and ...Secondary classes made sense. The dptdb interface to DPT was created after the DPT project was published: there was neither room nor need for equivalents of the ...Primary and ...Secondary classes. An equivalent for DPT's bitmapped record numbers was added to Berkeley DB databases. The bsddb module was not transfered from Python 2 to Python 3 so bsddb3 is used instead. Practical problems building bsddb3 from source on Microsoft Windows led to creating an interface to SQLite3 via the sqlite3 module for solentware-base, copying the existing design. Then apsw was added to the mix. The apsw and sqlite3 modules have different purposes and this might lead to differences in behaviour even if both are using the same version of SQLite3.

 
 

11 November 2019

solentware-grid-2.0

Changes to fit modifications in new version of dependency: solentware-base.

Project directory structure changed from solentware-grid to solentware-grid/solentware-grid where the implementation moves to solentware-grid/solentware-grid and project descriptions and setup.py remain in solentware-grid.

 
 

11 November 2019

solentware-misc-1.2

Use ImportError not ModuleNotFoundError so solentware-misc works on Python 3.5.

Project directory structure changed from solentware-misc to solentware-misc/solentware-misc where the implementation moves to solentware-misc/solentware-misc and project descriptions and setup.py remain in solentware-misc.

 
 

11 November 2019

emailstore-1.4

New version of dependency: solentware-misc.

Project directory structure changed from emailstore to emailstore/emailstore where the implementation moves to emailstore/emailstore and project descriptions and setup.py remain in emailstore.

 
 

11 November 2019

emailextract-0.7

Use ImportError not ModuleNotFoundError so emailextract works on Python 3.5.

Project directory structure changed from emailextract to emailextract/emailextract where the implementation moves to emailextract/emailextract and project descriptions and setup.py remain in emailextract.

 
 

11 November 2019

dpt3.0-dptdb-0.7.1

Remove the & ^ | &= ^= |= operators, introduced in dpt3.0-dptdb-0.6.3, because they do not work. Their intended job is done by wrappers of the APIRecordList and APIFoundSet classes introduced in solentware-base-4.0.

Adding to the note for dpt3.0-dptdb-0.7.0 the build does not work on FreeBSD amd64 12.0.

 
 

05 March 2019

solentware-base-3.0.1

Fix two serious problems which may arise after deleting some records from a database and then adding some more records.

One affects just bsddb3 and occurs when an index which references a deleted record is changed from a bitmap to a list, or the list is deleted, and later additions re-use the deleted bitmap or list. The file of the index was not noted and another file could be misled to corrupt it's indicies. Sometimes one delete and one insert would be enough to cause corruption, at others several hundred of each would be needed, and some sequences would clear the problem.

The other affects apsw, bsddb3, and sqlite3, and occurs when a record is deleted from a file with more than one segment, 32536 records, followed by adding enough records to fill the highest segment in any file. The file of the index was not noted and another file could be misled to search for spare record numbers which do not exist before starting on a new segment.

 
 

24 November 2018

emailextract-0.6

Add ability to extract text from *.txt *.csv *.ods *.odt *.docx and *.xlsx files which are wrapped in an attachment, MIME type application/ms-tnef, to an email. The filename of the attachment is usually winmail.dat.

 
 

21 November 2018

emailstore-1.3.1

Fix problem which prevented multiple emails with same 'From:' and 'Date:' headers being handled correctly.

 
 

18 November 2018

emailextract-0.5

New version of dependency: solentware-grid.

 
 

18 November 2018

emailstore-1.3

New version of dependency: solentware-grid.

 
 

18 November 2018

solentware-grid-1.2

Changes in API of runtime dependency solentware-base.

 
 

18 November 2018

solentware-base-3.0

The naming convention for SQLite3 Tables is changed to be consistent with the equivalent one for Berkeley DB Databases. Both are now consistent with DPT's distinction between Files and Fields, and things which would be DPT Fields can have the same name if they would appear in different DPT Files.

A tool is added to convert existing SQLite3 Table and Index names, used in solentware-base-2.n, to the names expected in solentware-base-3.n.

An unused module which depends on tkinter is removed. Now solentware-base does not depend on the presence of tkinter in the Python installation.

Database specifications may be lists of field names for each file name in a (Python) dictionary. Defaults are supplied for items which would be put in a full specification: non-default database names for these fields and files.

Some attributes and properties in classes used with Berkeley DB and SQLite3 are renamed to give a better indication of their purpose and their equivalents in classes used with other database engines.

 
 

17 November 2018

solentware-misc-1.1.1

Package README file brought up to date.

 
 

17 November 2018

solentware-grid-1.1.1

Package README file brought up to date.

 
 

17 November 2018

solentware-base-2.1.1

Package README file brought up to date.

 
 

17 November 2018

emailextract-0.4.1

Package README file brought up to date.

 
 

17 November 2018

emailstore-1.2.1

Package README file brought up to date.

 
 

8 November 2018

solentware-misc-1.1

Change project name from rmappsup to solentware-misc prior to upload to PyPI, but continue the version number sequence.

Miscellaneous is a good name but too long.

 
 

8 November 2018

solentware-grid-1.1

Change project name from gridsup to solentware-grid prior to upload to PyPI, but continue the version number sequence.

 
 

8 November 2018

solentware-base-2.1

Change project name from basesup to solentware-base prior to upload to PyPI, but continue the version number sequence.

 
 

8 November 2018

emailextract-0.4

New versions of dependencies: gridsup renamed as solentware-grid and rmappsup renamed as solentware-misc.

 
 

8 November 2018

emailstore-1.2

New versions of dependencies: gridsup renamed as solentware-grid and rmappsup renamed as solentware-misc.

 
 

05 October 2018

dpt3.0-dptdb-0.7.0

Use MinGW run on Wine on Unix-like systems to build dptdb, so the build is done like on Microsoft Windows.

In particular stop using the mingw32-g++ compiler from the devel/mingw32 port on FreeBSD.

The build works on FreeBSD i386 but not on FreeBSD amd64. (Both 11.2)

The build works on Debian amd64 (9.5.0) but I was unable to install a working Python under Wine on openSUSE x86_64 (15.0).

Added 11 November 2019. The & ^ | &= ^= |= operators introduced in dpt3.0-dptdb-0.6.3 do not work: they were intended to do the job done by wrappers of the APIRecordList and APIFoundSet classes introduced in solentware-base-4.0. They are removed in dpt3.0-dptdb-0.7.1.

 
 

05 October 2018

dpt3.0-dptdb-0.6.5

Adapted from dpt3.0-dptdb-0.7.0 to work with versions of MinGW earlier than MinGW-6.3.0 with which dpt3.0-dptdb-0.7.0 does not work.

This means not doing four patches related to the c++11 standard.

Added 11 November 2019. The & ^ | &= ^= |= operators introduced in dpt3.0-dptdb-0.6.3 do not work: they were intended to do the job done by wrappers of the APIRecordList and APIFoundSet classes introduced in solentware-base-4.0.

 
 

29 September 2018

dpt3.0-dptdb-0.6.4

This version is compatible MinGW-6.3.0.

This version is incompatible with MinGW-5.3.0, MinGW-4.9.3, and MinGW-4.8.1.

I guess the known compatibilities mean this version is incompatible with MinGW versions before 6.0.0.

Use dpt3.0-dptdb-0.6.3, the previous version, if necessary with earlier versions of MinGW.

Added 11 November 2019. The & ^ | &= ^= |= operators introduced in dpt3.0-dptdb-0.6.3 do not work: they were intended to do the job done by wrappers of the APIRecordList and APIFoundSet classes introduced in solentware-base-4.0.

 
 

29 September 2018

dpt3.0-dptdb-0.6.3

Extend DPT's APIRecordList class with & ^ | &= ^= |= operators using SWIG to wrap appropriate sequences of the existing Place and Remove methods. Code using these operators will not work on earlier versions of dptdb. (See note added 11 November 2019 below.)

This version is compatible with MinGW-5.3.0, MinGW-4.9.3, and MinGW-4.8.1.

This version is incompatible MinGW-6.3.0.

I guess the known compatibilities mean this version is incompatible with MinGW version 6.0.0 and later.

This version can be installed using Wine on FreeBSD but is not usable on at least one recent combination of FreeBSD base and ports tree. It is not usable on FreeBSD 11.2 base with a ports tree downloaded in September 2018. It is usable on FreeBSD 10.3 base with the ports tree distributed on the 10.3 *.iso release image.

Added 11 November 2019. The & ^ | &= ^= |= operators do not work: they were intended to do the job done by wrappers of the APIRecordList and APIFoundSet classes introduced in solentware-base-4.0.

 
 

31 August 2018

basesup-2.0.2

Fix a problem which prevented large updates using the sqlite3 interface to SQLite3. The apsw interface to SQLite3 did not suffer the problem.

Fix a problem which prevented large updates using the bsddb3 interface to Berkeley DB.

Similar problems in old versions of basesup were fixed too:

basesup-1.0.1

Fix a problem which prevented large updates using the bsddb3 interface to Berkeley DB.

basesup-0.21.8

Fix a problem which prevented large updates using the apsw interface to SQLite3. The sqlite3 interface to SQLite3 did not suffer the problem.

gridsup-0.19.12 rmappsup-0.38.13 emailstore-0.4.17

New versions of dependencies.

 
 

29 June 2018

basesup-2.0.1

Fix a typo which prevented one of the lower-level classes being used directly.

 
 

11 December 2017

emailextract-0.3

Add option to extract text from Open Document Format (*.ods) attachments to an email with Python's xml module. Gnumeric's ssconvert utility will still be used if it is installed, rather than the xml module.

Add option to extract text from Office Open XML (*.xlsx) attachments to an email with xlsx2csv, a third party Python package. Gnumeric's ssconvert utility will still be used if it is installed, rather than the xlsx2csv package.

Text extracted by Gnumeric and the two alternatives is equally valid, but usually incompatible, as far EmailExtract is concerned.

Gnumeric stopped provided installers for Microsoft Windows in August 2014.

Add option to extract text from Portable Document Format (*.pdf) attachments to an email with pdfminer3k, a third party Python package. Xpdf's pdftotext utility will still be used if it is installed, rather than the pdfminer3k package.

The pdftotext utility is used to get lines of text as they appear on the page, columns are ignored. The same is done with pdfminer3k as far as possible but the way 'page layout' is specified leaves room for columns to be preferred.

Text extracted by pdftotext and pdfminer3k is equally valid, but usually incompatible, as far EmailExtract is concerned.

Text is extracted noticably quicker by pdftotext than pdfminer3k.

 
 

2 December 2017

emailextract-0.2

Add text extraction from Open Document Format (*.odt) and Office Open XML (*.docx) attachments to an email.

Thanks to https://github.com/deanmalmgren/textract and http://etienned.github.io/posts/extract-text-from-word-docs-simply for which pieces to take from the XML structures.

 
 

29 November 2017

emailextract-0.1

Extract text from emails in a directory with one email per file. Attachments containing text, PDF, or spreadsheets compatible with Gnumeric, are supported.

The emailextract package was part of ChessResults, which is split into two parts. Stuff specific to ChessResults remains there and the rest is now in Emailextract.

ChessResults needs this package.

 
 

emailstore-1.1

Copy selected emails to a directory, one email per file.

The emailstore package has existed for some time but was not mentioned on the website.

ChessResults needs this package.

 
 

4 October 2017

gridsup-1.0.1

The gui.minorbases package is now present in the distribution files.

ChessResults needs this package.

 
 

2 October 2017

basesup-2.0

This version of basesup provides the bitmapped record number interface extracted from basesup-0.21.7.

See basesup-1.0 for the non-bitmapped record number version.

The size of each database segment is reduced from 65536 to 32768 records, halving the size of each bitmap and reducing the memory needed to add large numbers of records to a database while deferring index updates until a segment has been filled with records.

Databases created with this version of basesup are not compatible with any earlier version of basesup.

 
 

2 October 2017

basesup-1.0

This version of basesup provides the non-bitmapped record number interface extracted from basesup-0.21.7.

See basesup-2.0 for the bitmapped record number version.

Databases created with this version of basesup are not compatible with versions of basesup other than the basesup-1 series.

The query language 'like' clause can now be evaluated using indicies, and a new 'starts' clause indicates the evaluation can be limited to the range of index entries starting with the given value.

Error reporting functions and customized dialogues are moved to gridsup and rmappsup.

Modules supporting dBaseIII, CSV, and text, files are moved from gridsup and rmappsup.

 
 

gridsup-1.0

The core error reporting is moved from basesup.

Modules supporting dBaseIII, CSV, and text, files are moved to basesup.

Minor adjustment to tidy-up when closing an edit or delete record dialogue.

 
 

rmappsup-1.0

The customized dialogues are moved from basesup.

Modules supporting dBaseIII, CSV, and text, files are moved to basesup.

 
 

31 January 2017

basesup-0.21.7

Fix a problem using lookup tables in bitmapped record number versions of apsw, bsddb3, and sqlite3, database interfaces.

Only records on the first segment of the lookup table could be accessed. The segment number is now used when calculating the entry in the lookup table.

gridsup-0.19.11 rmappsup-0.38.12

New versions of dependencies.

 
 

23 December 2016

basesup-0.21.6

Fix a major problem in query language processing. Constructs including clauses like 'field from value1 to value2' or 'field not gt value' can now be processed.

A clause which follows a mis-spelt boolean operator, 'and', 'or', or 'nor', is now ignored as a consequence. Thus 'f1 eq v1 ant f1 like v2 or f2 is v3' is treated as 'f1 eq v2 or f2 is v3' for example, having spelt 'and' as 'ant'.

gridsup-0.19.10 rmappsup-0.38.11

New versions of dependencies.

 
 

2 October 2016

rmappsup-0.38.10

Restore binding to one record selection widget so 'move to' occurs in grid. Other scrolling was not affected.

 
 

28 September 2016

basesup-0.21.5

Fix crash when closing a bsddb3 database. Problem introduced at basesup-0.21. Typically seen when closing a database rather than quitting the application.

gridsup-0.19.9 rmappsup-0.38.9

New versions of dependencies.

 
 

18 September 2016

basesup-0.21.4

Fix incompatibility with bitarray-0.8.1 introduced at basesup-0.19. The problem occurred only if bitarray-0-8.1 is installed.

gridsup-0.19.8 rmappsup-0.38.8

New versions of dependencies.

 
 

14 September 2016

basesup-0.21.3 gridsup-0.19.7 rmappsup-0.38.7

Documentation changes only.

 
 

9 September 2016

basesup-0.21.2

Add versions of query language modules which work with the DPT database engine.

gridsup-0.19.6 rmappsup-0.38.6

New versions of dependencies.

 
 

7 September 2016

basesup-0.21.1

Management of DPT cursors when dealing with partial keys brought into line with a change, made some time ago, to the interfaces to the apsw, bsddb3, and sqlite3, database engines.

gridsup-0.19.5 rmappsup-0.38.5

New versions of dependencies.

 
 

5 September 2016

basesup-0.21

Obsolte modules for deferred update using bsddb3 removed.

Starting a new segment in deferred update is now detected, avoiding much unnecessary checking for existing index records to be spliced.

The method to decide when chunks of deferred updates are done is changed to allow dependant packages to choose the chunk-size based on an estimate of available memory. All earlier code related to problems encountered on OpenBSD is removed: there are ways of allocating more, or less, memory but test whether one of them has been done rather than assume either way. The default remains to do a chunk at a segment boundary (every 65536 records).

Hash indexes can be used via bsddb3 where appropriate, as well as btree indexes.

Management of new DPT cursors changed so when first use is next() or prev() the effect is first() or last(): this had always been the intended behaviour.

gridsup-0.19.4 rmappsup-0.38.4

New versions of dependencies.

 
 

7 August 2016

See Old history for previous changes.