Old Change History

Notes PGN ChessTab Top

Changes are listed in reverse chronological order. Followed by version comments and general comments added from time to time.



19 July 2016

ChessTab version 0.39.6 released.

No changes in chess functions.

The basesup component is changed. Databases updated by this version of ChessTab cannot be accessed by earlier versions.


15 July 2016

ChessTab version 0.39.5 released.

No changes in chess functions.

A fault in the basesup component is fixed.


13 July 2016

ChessTab version 0.39.4 released.

No changes in chess functions.

A fault in the pgn component is fixed.


12 July 2016

ChessTab version 0.39.3 released.

No changes in chess functions.

The pgn component produces a different set of values for indexing games by position, so this version of ChessTab is not compatible with earlier versions.


30 May 2016

ChessTab version 0.39.2 released.

No changes in chess functions.

A fault in the basesup component is fixed.


20 May 2016

ChessTab version 0.39.1 released.

Chess engine assessments of variations are displayed, for example, as +0.33 for advantage to white and -0.33 for advantage to black.

The side to move is stated in first line of analysis.


17 April 2016

ChessTab version 0.39 released.

Selection rules for lists of games can be saved and used as needed.

The Order menu is renamed Select and adjusted to fit the additional search functions.


5 April 2016

ChessTab version 0.38.1 released.

A major fault in the sqlite3 part of the basesup component is fixed.


14 November 2015

ChessTab version 0.38 released.

Analysis generated by chess engines for a position is written to the database if one is open. The generated analysis for a position from all chess engines is displayed whenever the position is displayed, even if no chess engine is running. The depth and number of variations for a position on the database is never reduced.

A toggle between showing all items and just one item is added to the Tools menu. When the item is a game it's board, score, and any analysis, are shown.


26 October 2015

ChessTab version 0.37.1 released.

Fix problem which caused crash when attempting to display an item for first time with scrollbars or analysis hidden. Switching between visible and hidden on items already displayed is not affected.


15 October 2015

ChessTab version 0.37 released.

An interface to chess engines which support the Universal Chess Interface (UCI) is added. The current positions in displayed games and repertoires are passed to a chess engine and the analysis is displayed when ready. The analysis can be played through.

Several chess engines, all running on the same computer as ChessTab, can be used at the same time.

The Chess Merida and Chess Motif fonts may not work on platforms other than Microsoft Windows. These fonts are 'microsoft-symbol'. Chess Cases and Chess Lucena do work on FreeBSD 10.1 and Microsoft Windows. The font defaults have been adjusted to be platform dependent, meaning Windows or not-Windows.

A number of corrections and minor improvements were made.

When using the bsddb3 database engine interface, importing a small number of games into a new empty database fails. The workaround is close the new database, then open it and import the games. Small is much less than 1000 games here, and 200 games seems not small enough to have the problem.

The other database engine interfaces, including the default sqlite3, are not affected.

See comment dated 15 October 2015 on the uci package used but not yet listed in components.


15 July 2015

ChessTab version 0.36 released.

No changes in chess functions.

The changes introduced at releases 0.35.1 and 0.35.2 to fix a problem on OpenBSD 5.7 i386 using the sqlite3 database engine do not achieve anything. They are replaced by a version of import which limits the number of games imported to the first 32768, no matter how many games are in the PGN file.

The problem did not occur on FreeBSD or Microsoft Windows.

It is not known what happens on other Operating Systems.

The unlimited import version is used by the chesstab/chessgames.py module.

The limited import version is used by the chesstab/chessgames_sqlite3chunk.py module.


15 July 2015

ChessTab version 0.35.4 released.

No changes in chess functions.

Several common cases of PGN file imports to sqlite3 databases now get done a lot quicker than before.


30 June 2015

ChessTab version 0.35.3 released.

No changes in chess functions.

A fault in the sqlite3 part of the basesup component is fixed.

Imports to databases which already contain games were almost certain to fail. Unlike the previous version, 0.35.2, this problem occurred under all operating systems.

Imports to an empty database were not affected by the problem.


30 June 2015

ChessTab version 0.35.2 released.

No changes in chess functions.

Large imports on the Python 3.4 package for OpenBSD 5.7 i386 using the sqlite3 database engine now run to completion.

The problem did not occur on FreeBSD or Microsoft Windows.


18 June 2015

ChessTab version 0.35.1 released.

No changes in chess functions.

The import functions now work on the Python 3.4 package for OpenBSD 5.7 i386. The post mentioned in comments suggests a mis-match of compilation options for Tcl/Tk and Python leads to the problem.


14 June 2015

ChessTab version 0.35 released.

No changes in chess functions.

The technique used to serialize data for storage on a database has been changed.

Use the export and import actions, on the old and new versions respectively, to convert databases to the new storage format.


20 May 2015

ChessTab version 0.34.1 released.

No changes in chess functions.

A fault in the basesup component is fixed.


18 May 2015

ChessTab version 0.34 released.

No changes in chess functions.

Changes in the basesup component allow large PGN files to be imported to an sqlite3 database a lot quicker than before, but still significantly slower than to bsddb3 or DPT databases.

A fault in the pgn component is fixed.


13 February 2015

ChessTab version 0.33.1 released.

No changes in chess functions.

Contact details moved from README files to new CONTACT files in all products.

The README file will be used as the description of a package if put on PyPI. The author's email is given separately when uploading the package, but not displayed, so it is sensible to remove that email address from the README file.


12 February 2015

ChessTab version 0.33 released.

No changes in chess functions.

All products use setuptools rather than distutils as the package installer.


29 September 2014

ChessTab version 0.32 released.

No changes in chess functions. One of the components modified.


22 June 2014

ChessTab version 0.31 released.

No changes in chess functions. One of the components modified.


31 August 2013

ChessTab version 0.30 released.

Runs on Python 3.3. Number of reasons for not going to version 1.0 reduced from 6 to 2.


20 July 2013

Modifications to Python 3 versions of components of ChessTab.

Implementation improvements. See general comments dated 20 July 2013.


29 June 2013

ChessTab version 0.25.2 released.

Provide game exporting. See general comments dated 15 June 2013.


15 June 2013

Python 3 versions of components of ChessTab available.

These require Python 3.3 or later. Some may work on earlier Python 3 versions.

Visit Other Products. Also via Downloads page.

Databases created by Python 2 versions of chesstab cannot be used by Python 3 versions. Or vice-versa. See general comments dated 15 June 2013.


4 June 2013

ChessTab version 0.25.1 released.

Fix some faults spotted during conversion to Python 3.


29 September 2012

ChessTab version 0.25 released.

Prepare for conversion to Python 3.


31 July 2012

ChessTab version 0.24 released.

Change version number as a consequence of correcting a problem in the gridsup package included in the ChessTab installers.

No changes to the chesstab package included in ChessTab.


24 March 2012

ChessTab version 0.23 released.

Naming convention for releases, and associated help text, changed.

Correct a problem using sqlite3.


10 December 2011

ChessTab version 0.22 released.

sqlite3 can be used if components installed from source distributions.

Correct two problems.


20 November 2011

ChessTab version 0.21 released.

Correct some problems.


10 November 2011

ChessTab version 0.20 released.

Correct some problems. The most important being absence, in release 0-19, of update actions for single games via keyboard.


6 November 2011

ChessTab version 0.19 released.

Upgraded to use DPT API version 3 release 0.


6 November 2011

ChessTab version 0.18 released.

The size of a PGN import is estimated and the planned increase in file size reported. New options are provided to increase the file size in steps relevant to the estimated size of the import.

Provide pointer equivalents for most keyboard actions.

Log errors to a file in the database folder to avoid offending Microsoft's User Access Control (UAC).

Placed directly into old versions because version 0.19 was created immediately using DPT version 3 release 0 instead of DPT version 2 release 25.


24 August 2011

ChessTab version 0.17 released.

Withdrawn shortly after because it was not obvious how to deal with PGN imports that failed because the database was full.


Version 0.39.6

New component versions basesup-0.20 gridsup-0.19.3 rmappsup-0.38.3.

Version 0.39.5

New component versions basesup-0.19.2 gridsup-0.19.2 rmappsup-0.38.2.

Version 0.39.4

New component version pgn-0.10.1.

A Numeric Annotation Glyph followed by a Game Termination Marker with no whitespace between them caused ChessTab to fail abruptly.

Version 0.39.3

New component version pgn-0.10.

Handling of errors in PGN files is improved, including edited games saved with incomplete moves at the end of recursive annotation variations. Generation of values to index games by position is simplified and closer to Forsyth Edwards Notation, at the expense of compatibility with earlier versions.

The export and import functions can be used to transfer databases from earlier versions.

Version 0.39.2

New component versions basesup-0.19.1 gridsup-0.19.1 rmappsup-0.38.1.

Version 0.39.1

New component versions basesup-0.19 gridsup-0.19 rmappsup-0.38.

Positioning slider in scrollbar for sorted list of games now works, after changes in basesup.

Version 0.39

New component versions basesup-0.18.1 gridsup-0.18.2 rmappsup-0.37.3.

Some additions to basesup and gridsup in support of changes to chesstab.

Version 0.38.1

New component versions basesup-0.18 gridsup-0.18.1 pgn-0.9.1 rmappsup-0.37.2.

The unit tests of basesup, a dependency of chesstab, for the sqlite3 interface related to deleting records in the bitmapped record number classes usually fail (or sometimes pass) when run under Python 3.4 or 3.5. These tests always pass using Python 3.3, and these tests always pass using the apsw insterface on Python 3.3, 3.4, and 3.5.

Delete record in chesstab has not failed at all using any of these combinations and it is assumed the problem lies in the code running the test, not the code being tested.

The sqlite3 module is part of the Python distribution, but apsw is not and has to be installed separately.

An invalid PGN file is no longer exported when a recursive annoation variation starts at the end of a line: left parenthesis omitted if it was last character on line. Importing PGN files with missing left parethesis saves an error record rather than causing chesstab to crash.

Version 0.38

New component version pgn-0.9.

The machinery to parse import format PGN generated from chess engine analysis is moved to the pgn component.

Version 0.37.1

No changes to dependency components.

Version 0.37

New component versions basesup-0.17.1 gridsup-0.18 pgn-0.8.3 rmappsup-0.37.1.


Indexes created by the bitmapped record number version of the bsddb3 database engine interface when importing games were likely to be incorrect. The typing error which allows the problem has been corrected.

The other database engine interfaces, including the default sqlite3, are not affected.

Inserting new games, or editing existing games, by typing the game score are not affected.

Some code made redundant at release 0.35 is removed.


Internal adjustments which do not affect functions provided.


Allow for attempts to import games from files that do not contain any games according to validation against the PGN specification.

It is beyond the scope of the pgn component to prevent such imports: which will occur if requested after the estimates have been produced. The entire file will appear as one game with errors if the import is done.


No changes except new versions of dependencies.

Version 0.36

New component versions basesup-0.17 gridsup-0.17 rmappsup-0.37.

The apsw database engine, an alternative to Python's sqlite3, is added after being used to investigate the problem with sqlite3 on OpenBSD by comparison. It suffers the problem too, and reacts the same way in the solution.

If both apsw and sqlite3 are installed, apsw is used because sqlite3 is part of the Python installation and extra action is needed to install apsw.

The problem also occurs on OpenBSD 5.7 i386 using the bsddb3 database engine, but a solution is not provided.

The modules which were used in Microsoft Windows executables generated with py2exe are removed.

Version 0.35.4

New component versions basesup-0.16.2 gridsup-0.16.2 rmappsup-0.36.2.

Indexes dropped and re-created when doing bulk imports to sqlite3 databases.

Imports to empty sqlite3 databases are faster. Imports of more than between 65537 and 131072 games, depending on the size and history of the database, are usually quicker. The time taken to import less than 65537 games is not affected.

On sufficiently large databases, a comparatively small import may be slower. It is not known what sufficiently large means, but a database of 2 million games is nowhere near large enough because comparatively small is less than 65537 games.

Version 0.35.3

New component versions basesup-0.16.1 gridsup-0.16.1 rmappsup-0.36.1.

A problem in the basesup bitmapped record import interface to Python's sqlite3 module is fixed. If a record is being inserted into a segment which contained records before the import started, existing index records were not located so they could be updated. The problem seems to have been there always.

Version 0.35.2

No changes to dependency components.

Version 0.35.1

No changes to dependency components.

Version 0.35

New component versions basesup-0.16 gridsup-0.16 rmappsup-0.36.

Python's repr() and ast.literal_eval() replace both Python's pickle.dumps() and pickle.loads() and some functions in the ChessTab product when serializing a Python object for storage on a database or reconstructing it. It is no longer possible to opt out of using the serialization.

Databases created using earlier versions of these products cannot be accessed. In practice this means ChessTab databases must be converted to the new format by using the export actions in the old version and the import actions in this version.

ChessTab is not affected by the import problem mentioned for Results in 'Other Products'.

Version 0.34.1

New component versions basesup-0.15.1 gridsup-0.15.1 rmappsup-0.35.3.

A problem in the basesup bitmapped record interface to Python's sqlite3 module is fixed. If a record is being deleted, any index value refering to this record which refers to exactly two records loses the reference to the other record instead. The problem seems to have been there always.

Version 0.34

New component versions pgn-0.8.2 basesup-0.15 gridsup-0.15 rmappsup-0.35.2.

Version 0.33.1

New component versions pgn-0.8.1 basesup-0.14.1 dpt3.0-dptdb-0.6.1 gridsup-0.14.1 rmappsup-0.35.1.

Version 0.33

New component versions pgn-0.8 basesup-0.14 gridsup-0.14 rmappsup-0.35 including dpt3.0-dptdb-0.6 done at 8 February change.

The Microsoft Windows binary installers for ChessTab, along with it's versions of source distributions, have been removed. The setuptools install will download and install all the components needed when chesstab-0.33 is installed.

Version 0.30

Opening repertoires are supported.

Export games in PGN format added.

ChessTab runs on Python 3.3.

sqlite3 becomes the default database interface.

Application installer (InstallChessTab.exe) no longer provided. Python 3.3 or later must be installed to run ChessTab.

Python Package installers provided for dptdb.

Version 0.25.2

Export games in database as text added. Recursive Annotation Variation (RAV) to last move in a RAV is now inserted in correct location.

Version 0.25.1

The pointer equivalents of typing game results now work.

Version 0.25

The 2to3 conversion tool suggests many changes to coding style along with those necessary for the code to work at Python 3. The non-essential changes that also work on Python 2.7 are applied in this version.

An option to install the Microsoft Visual C++ 2008 Redistributable is added. You may not need to install this if other software has already installed it or an equivalent product.

Version 0.24

Scrolling any of the widgets up or down by a single line works again after being broken in version 0.23. Scrolling up or down by the number of lines displayed in a widget was not affected.

Version 0.23

Help | About text adjusted to match differences between the installation methods.

Naming convention adjusted to indicate Python version dependency more precisely.

Edit dialogues changed to commit transactions explicitly in sqlite3.

Version 0.22

The sqlite3 database engine can be used but importing large numbers of games using sqlite3 is not a good idea at present.

A couple of problems editing partial positions corrected.

Two modules moved from rmappsup to gridsup because their sqlite3 equivalent is put there directly.

Version 0.21

Problems with colouring rows when moving pointer though selected row or selected row through row under pointer corrected.

Problems displaying records when using the Insert or Delete keys or pointer equivalents (F11 not affected) after closing and reopening a database corrected.

Some things which should be in gridsup rather than chesstab moved there.

Version 0.20

The update code for adding, deleting, and editing single games restored after being removed in error.

A relatively minor problem with colouring rows for games which had been displayed corrected.

Version 0.19

The example large PGN import used, 1.5 million games, now takes just over 20 hours rather than the 24 hours still quoted in the help documents.

An obscure problem in the program chesstab_winedptmulti.exe has been fixed. This program is intended for use under Wine on PCs with less than 500Mb memory when importing more than 5000 games. Prior to this version the presence of a null value in a PGN Tag would cause an import run from this program to fail. For example [ Source "" ].

Version 0.18

DPT is a good choice for handling the indexing of a chess game database. But the objectives of the DPT project constrain DPT to use files that do not grow as required. Thus a chess database has to be big enough to hold the games imported from a PGN file before the import starts.

The number of bytes in the PGN file is not good enough as a predictor of the required size of the DPT database (or extra space when adding to an existing database).

The number of games, positions, and pieces in those positions, in the PGN file is a good predictor of the required size of the DPT database. Getting the exact counts takes about 25% of time used to do an import. So the size of a PGN import is estimated to save time on large imports.

Version 0.17

Earlier versions were points at which work on ChessTab stopped for a while or when changes to a component package were made for reasons external to ChessTab.

ChessTab has a 0.n version number because several things deny it production status:

ChessTab does not communicate with Chess Engines and collate the position evaluations returned.

ChessTab does not provide facilities to search a database using an Opening Repertoire.

ChessTab does not provide facilities to export games in Portable Game Notation (PGN) format.

DPT is Windows only. ChessTab does run under Wine on non-Windows platforms but the visual experience is not as good as seen when running chesstab on a native Python.

ChessTab does not run on Python 3.

Support for Berkeley DB is not included in Python 3 (a third party package is available) and sqlite3 is not yet supported by ChessTab.


6 November 2011

Six reasons are given (Version 0.17 above) for denying production status to ChessTab. The intention is to deal with these in the following order:

Add support for the Python interface to sqlite3.

Convert to Python 3.

Find a solution to the problems raised because DPT is Windows only. Ideally this means removing the restriction that DPT is Windows only. But making ChessTab run nicely under Wine may be what is possible in practice.

Implement the three ChessTab facilities in no particular order. But the perceived ease of implementation suggests the order will be Opening Repertoire, PGN export, and Chess Engine interface.

An SQL interface has been added to Berkeley DB 11g using a derivative of sqlite3 with the same interface. An interesting possibility is that Berkeley DB has been, or will be, modified to do index searches at the speed achieved in DPT.

10 November 2011

The blunder with database updates does not affect importing games. While releases 0-18 and 0-19 are broken, they do not do any damage to the chess databases (unless doing nothing counts as damage). It is the missing facilities that matter.

Release 0-19 will be removed soon. Release 0-18 will be patched (to 0-18-1) to match release 0-20 apart from the dptdb component.

10 December 2011

ChessTab can be moved to Python 3 now that sqlite3 can be used as the cross-platform database engine to fill hole left by removal of Berkeley DB support from core Python.

Importing games is much slower using the sqlite3 database engine but optimisation has not been investigated (the disk I/O light is more or less permanently on).

The sqlite3 module takes priority over the bsddb module, both are supported at Python 2 but only sqlite3 is supported at Python 3. The bsddb3 module is available for Python 2 and Python 3 and takes priority over the sqlite3 module. On Microsoft Windows the dptdb module takes priority over the sqlite3 module.

Releases 0-19, 0-20, and 0-21 are withdrawn on publication of Release 0-22. Release 0-18 will be patched (to 0-18-1) to match release 0-22 apart from the dptdb component.

24 March 2012

The default action on ending a transaction in sqlite3 is to rollback (backout) any changes to the database. The default action on ending a transaction in DPT is to commit any changes to the database. The transaction capabilities of Berkeley DB are not used. If an error occurs during a transaction in DPT the default action is backout (rollback).

The 'other products' page, accessed via the 'download' page is added to the website. Some of the products use the same packages as ChessTab. Adjustments to the source code repository were done to ease maintenance and the visible consequence is the change in naming convention.

14 September 2012

The intention was that release 0-25 be the first one to work on Python 3. The volume of non-essential change along with some non-trivial changes needed in one of the 'other products' made it reasonable to get the non-essential changes out of the way before releasing the conversion.

29 September 2012

The option to install Microsoft Visual C++ 2008 Redistributable should have been present from the start. Python documentation suggests it is possible to embed the redistributable in the application (licences permitting), which would avoid the dialogue with the Microsoft installer. I could not get this to happen and the whole installation area has to change at Python 3 anyway.

15 June 2013

At Python 2 strings {type('') is str} and bytes {type(b'') is str} are equivalent. Unicode strings {type(u'') is unicode} are different.

At Python 3 strings {type('') is str} and unicode strings {type(u'') is str} are equivalent. Bytes {type(b'') is bytes} are different.

Sloppy assumptions about the equivalence of strings and bytes made it seem impossible, and at least not practical, to produce Python 3 versions of chesstab capable of accessing databases generated by Python 2 versions of chesstab. The PGN export facility will be used to solve this problem.

The bit-mapped record numbers option added to the interface to bsddb3 in basesup, which follows the example of DPT, is in theory a solution to the problem that DPT is Windows only (comment 6 November 2011). But packages and installers for bsddb3 on Python 3 are not yet common, and while building bsddb3 from source on FreeBSD (and presumably other BSDs and Linux distributions) is almost trivial, it seems not so trivial on Windows.

sqlite3 is cross-platform but can be unacceptably slow for some functions compared with the bit-mapped options when dealing with large databases.

29 June 2013

Export games as text added to chesstab so that databases can be moved from Python 2 to Python 3.

Deferred update, following the example of DPT, now works on the bit-mapped record numbers option added to the interface to bsddb3 in basesup in Python 3 when adding to a database which already contains records. The absence of the relevant code was seen when testing the sorting of chess games by chesstab for export in PGN format.

20 July 2013

Partial position searches evaluated using bit-mapped record numbers on bsddb3 under Python 3.

It is now reasonable to save the search answers obtained using bsddb3, and this is done. The initial evaluation is still about an order of magnitude slower than the DPT version, but a lot quicker than before.

31 August 2013

Partial position searches evaluated using bit-mapped record numbers on sqlite3 under Python 3.

SQL statements cannot be used to get at the records in the natural way. But sqlite3 is being used because it is the only cross-platform database interface in the Python distribution, not because SQL is the language.

InstallChessTab.exe is withdrawn because it has not proved possible to make it work at Python 3.3 when built with either py2exe or cx_Freeze.

Both bsddb3 and bitarray are provided as Python Packages which should build and install on all platforms if the necessary toolsets are installed. But I have not succeeded in building bsddb3 on Windows.

Response times are quickest with dptdb, sometimes significantly, but there is a risk of having to do database administration tasks, increasing size in particular, while such risks are practically absent with bsddb3 and sqlite3. The bitarray module may, or may not, give noticeable improvement in response times for bsddb3 and sqlite3.

8 February 2015

The DPT website, www.dptoolkit.com, expired on 15 January 2015. At time of writing that website had not been re-registered for any purpose.

All links to www.dptoolkit.com have been removed from www.solentware.co.uk but the product references remain in place, sometimes edited to say the DPT website has expired.

The author tells me DPT will be put on a repository at some point, citing sourceforge as an example.

The current dptdb distributions are replaced by a source distribution which downloads the zip files containing a recent version of the DPT API source code and documentation from www.solentware.co.uk.

In PyPI terms the development status is described as '7 Inactive'.

18 May 2015

The bit-mapped record numbers option is now the only one supported in the interface to sqlite3. At some point the non bit-mapped option will be removed from the bsddb3 interface as well. The distinction is not relevant to the DPT interface.

At present importing 1.5 million games to an empty bsddb3 or DPT database takes about 7.5 hours, but about 11.5 hours to an empty sqlite3 database. The sqlite3 time can be reduced to about 7.5 hours too, by creating the indexes after the import: which is how the other two manage it. Simply dropping and creating indexes will not do when updating a populated database because bitmaps have to be spliced together. DPT was built to handle the splicing efficiently, and hopefully transplanting the solution in the bsddb3 interface will be good enough.

14 June 2015

The non bit-mapped option has been restored to the sqlite3 interface because Results, see Other Products, uses it but cannot use the bit-mapped option yet. Use of the non bit-mapped option has not been restored to ChessTab.

ChessTab always uses a separate process to import anything: so it does not suffer the threading (DPT or sqlite3) and record locking (DPT) problems met by Results.

ChessTab was designed to work with DPT: so has to use a separate process where the use of transaction backout can be disabled to do potentially large volume imports more quickly. The equivalent in sqlite3 does not need a different process or thread, but the user interface can be kept active by doing so. The bsddb3 version of transaction backout is not used at all, but probably should be.

The significant change for ChessTab is not opting out of the serialization rather than the different technique.

18 June 2015

The pydoc-g-exhausts-the-stack-td223979.html document at openbsd-archive.7691,n7.nabble.com described the problem encountered in ChessTab. It seems the choice of --enabled-threads option leads to the problem because adjusting ChessTab to avoid doing User Interface actions (tkinter calls) in the wrong thread stopped the problem occurring.

The adjustments seem to be compatible with the choices made in FreeBSD Ports and in Microsoft installers for Python and Tcl/Tk.

30 June 2015

A MemoryError exception (insufficient memory) was raised after processing a few thousand games when importing on OpenBSD 5.7 using the sqlite3 database engine. Splitting the task into chunks of 5000 games, each done in a new process after the preceding one finishes, seems to allow the task to be completed. An import of 2 million games, not chunked, took about 17.7 hours on identical hardware using FreeBSD but it is estimated, from progress so far, that the chunked import will take at least 5 days on OpenBSD (job terminated!).

Splitting into chunks did not eliminate the MemoryError exceptions entirely: three chunks failed during first 20% of the 2 million game import with 2Gb memory available. It is assumed the failures would not happen with more memory.

The problem fixed at version 0.35.3, published on same day, was found while testing version 0.35.2 on FreeBSD. In hindsight it is a surprise the error fixed at 0.35.3 did not occur until six hours into the run. The import, when split into chunks, will take just as long on FreeBSD as on OpenBSD.

The 2 million game import to an empty database, without chunking, was done on these hardware and operating system combinations differing only in disk type:

Windows, rotating disk, DPT database engine: 11.5 hours.

Windows, rotating disk, sqlite3 database engine: 78 hours.

FreeBSD, solid state disk, sqlite3 database engine: 17.7 hours.

FreeBSD, solid state disk, bsddb3 database engine: not done yet. (~12 hours maybe by comparison with 18 May 2015 comment.)

6 July 2015

The bitmap splicing problem cited in the comment of 18 May 2015 as the reason for not dropping indexes while importing to an sqlite3 database has been resolved for some common cases. The high segment, and the next one, are filled before the indexes are dropped. The next one is included to guarantee that the common imports of up to a few thousand games are never affected by excessive overheads of dropping and creating indexes for comparatively large databases.

The 2 million game import to an empty database, without chunking, was done on this hardware and operating system combination:

FreeBSD, rotating disk, sqlite3 database engine: 29.3 hours.

It is assumed the time on Windows would be similar, if not exactly the same, so dropping then re-creating the indexes gets the import done in less than half the time compared with that given in the comment of 30 June 2015.

15 July 2015

The earlier attempts to avoid the MemoryError exception, which happens only on OpenBSD as far as known, do not work. Only the first chunk succeeds in committing any changes. All subsequent chunks suffer the MemoryError exception and backout (rollback) any changes.

The only solution known is do a single chunk, then reboot the machine, then do the next chunk.

The MemoryError exception occurs well after game 40000, but before game 65536, for typical PGN files on a machine with 2Gb memory. 65536 is the database segment size, and the optimum chunk size is the largest factor of the segment size which does not suffer the MemoryError exception. That is 32678. Given this, a segment is filled by two chunks, with the first chunk done in half the time taken by the second chunk. If more than two chunks were needed to fill the segment, only the first chunk would take half the time of each later chunk.

I found the apsw interface while looking for plausible causes of the problem. It suffered the MemoryError exception on OpenBSD too, which suggested a cause using Sqlite3 relative to OpenBSD. But perhaps not because it turns out the bsddb3 interface suffers the MemoryError exception on OpenBSD too.

Maybe my code relies on a configuration option not used when building the OpenBSD Python package. There are no obvious knobs to turn in the OpenBSD Python port when building from source. Eventually I may try ignoring even the OpenBSD Python port and build directly from Python.org sources using the knob settings in the FreeBSD Python port if possible.

The work with apsw did identify the places where explicit 'begin transaction's are needed: which will be useful when adding transactions using the bsddb3 interface.

The OpenBSD 5.7 i386 apsw package turned out to be broken: the Cursor class is missing the fetchone() method. The OpenBSD port of apsw is consistent here. Using the source from the apsw home page worked fine.

26 July 2015

pydbitkit is added to the Other Products page because the modules implementing the client-server conversation will be adapted to provide communication between ChessTab and chess engines.

This marks the start of doing something on communicating with chess engines: see the comment dated 6 November 2011 and the changes note dated 30 August 2013 for version 0.30 on things to do in ChessTab.

15 October 2015

Communicating with chess engines running on other computers is not supported at release 0.37 of ChessTab.

Saving the analysis on the database is not done either. This will be added later, along with additional search functions based on PGN Tags and changes to the User Interface so each list of things or each game can occupy the whole of ChessTab's display area.

Saving the analysis will be done before communicating with chess engines on other computers. This order seems sensible, and the easier task gets done first.

The uci package provides the interface to chess engines. It is meant to work closely with the clientserver package to get analysis done by chess engines on other computers. Fortunately it proved fairly easy to break the dependency on clientserver so chess engines can be used on the same computer as ChessTab without installing the unfinished clientserver package. The clientserver and uci packages are not listed under components, but the uci package is on the website for the installer to download.

14 November 2015

At present all chess engine analysis done while a database is open is saved on the database. This will be changed so just repertoire game analysis is saved automatically, and game analysis is saved on request. It is not clear what the rules should be yet: maybe data storage is cheap enough that "save everything always" is fine.

Nothing done on additional search functions yet.

5 April 2016

I do not know why the unit tests for deleting records using the sqlite3 interface usually fail at Python 3.4 and 3,5, but always pass at Python 3.3; nor why the tests always pass using the apsw interface. It is possible to split the tests into smaller pieces such that they always pass, but that seems like cheating.