Change History

Temporary backups while doing an import can take many minutes. Large imports to a database can take several hours: plenty of time for some incident to force a repeat of work already done. Small imports can go wrong: hopefully restoring from backup is a lot quicker than repeating the imports to reconstruct the database.

The changes prompted by pylint, pycodestyle, and pydocstyle, led to changing the version number to 6.0 rather than 5.1 because there are a number of module name changes and public method and attribute name changes.

There are a number of fixes included but even the largest of these, which involved the addition of a new dependency solentware_bind, could reasonably have been done within changing the version to 5.1.

The distutils package is deprecated at Python 3.10 and scheduled for removal at Python 3.12. The Python Packaging Authority assumes the use of pip supported by setuptools and packaging. The use of the setup.py module provided by setuptools, which replaces the version in distutils, is deprecated in the pip package.

Thus the 'python setup.py ...' command is replaced by the 'python -m pip ...' command in the installation instructions in ChessTab's README. At time of writing both commands work for installing ChessTab.

Games can be displayed in the main widget by 'F11', or in a separate widget by 'Enter', for example. If the separate widget is closed while a related dialogue is displayed that dialogue is now destroyed too. Previously separate action was needed to destroy the dialogue.

Starting Recursive Annotation Variations (RAV) and comments on a new line, together with indenting the moves, looks nicer in my opinion.

Moves are wrapped to a new line depending on the width of the display area. If a sequence of 20 fullmoves occur without a RAV or comment the next fullmove is forced to a new line. This seems a good compromise between tidiness and response times navigating large games. See comments dated 15 November 2019 about response times.

When moves are deleted while editing a game, the move numbers are not deleted leading to move numbers referencing no move and possibly new moves out of sequence with their move numbers. The move numbers are ignored when updating the game so all will be correct when the game is opened in a new widget. While editing, the game may look less untidy if move numbers are turned off.

It is not clear when deleting games from a Berkely DB database on OpenBSD became a problem, or if it has always been one. The reported error led to finding the maximum allowed locks is the problem. Games with a small number of moves can be deleted but at some point the extra indicies assocatied with more moves need too many locks.

The default value of 'maxlocks' is 1000 on OpenBSD; but 0, zero, on FreeBSD where there is no problem. Setting 'maxlocks' to 0, zero, on OpenBSD does not fix the problem, but setting to 10000 has been high enough so far.

The ability to set 'maxlocks' for OpenBSD has been added to solentware-base and this is used in ChessTab to set it to 10000.

Modules are added to ChessTab which use the new modules in solentware-base supporting the UnQLite, Vedis, dbm.gnu, and dbm.ndbm, database engines.

None perform as well in ChessTab as the other database engines; so some protection against accidental use is given by requiring command line options to create or open databases with them.

ChessTab version changed from 3 to 4 for three reasons. The table naming convention for databases using the SQLite3 database engine or the Berkeley DB database engine is changed. When using the Berkeley DB database engine all tables are on one file rather than each table being on a separate file. Much code specific to a particular database engine is moved to the appropriate module in solentware-base (see Components page): several chesstab modules are retired and replaced by modules mixing classes providing ChessTab logic with solentware-base modules providing the database interface.

(Here, table names are equivalent to names of primary and secondary databases in Berkeley DB terminology.)

Upgrade of a ChessTab database using Berkeley DB will take a lot longer than a database, containing the same data, using SQLite3 because all the data is moved to a new location.

Some details of the code changes are given in comments dated 1 November 2019.

ChessTab version changed from 2 to 3 because the table naming convention for databases using the SQLite3 database engine is changed.

Filtering a sorted list of games now displays the filtered list immediately rather than an empty list which is replaced with the correct list after the first scrolling action.

A warning is given about a program crash which will occur, unless the specified avoiding action is taken, when an '*' is included in the filter if the database engine is SQLite3.

The basesup, gridsup, pgn, rmappsup, and uci, dependencies have been renamed solentware-base, solentware-grid, pgn-read, solentware-misc, and uci-net.

Python3.6 or later should be used to run ChessTab if the database is accessed with the sqlite3 interface.

This version was released so downloading the version of the gridsup dependency with a part missing can be avoided. ChessTab does not need the missing part of gridsup. In other words ChessTab-2.0 works exactly like this version.

This version has not replaced version 2.0 as the current version on PyPI.

The scope of the chess engine analysis functions is extended to allow use of chess engines running on other computers. Chess engines running on the same computer as ChessTab may be treated like this too. The functions to run chess engines have been moved to the new uci package which also provides a way of communicating with these engines.

Firewalls may prevent communication with chess engines on other computers unless you change settings to allow it.

ChessTab cannot run under Wine at time of writing for a variety of reasons, see Comments dated 2 October 2017, but can now get analysis done by chess engines running under Wine without itself running under that Wine environment.

ChessTab uses Chess Query Language (CQL) as a way of expressing database searches. ChessTab does not attempt to replicate the report generation done by the program provided with the CQL specification. The CQL website provided version 5.1 of the query language at time of writing (the obvious links in the contemporary Wikipedia entry for CQL were to version 3.01 which is significantly different).

At present ChessTab supports CQL's 'piece designators' and 'transform filters'. This is sufficient to replace the partial position specifications used by earlier versions of ChessTab.

The indicies which supported partial positions are extended and modified to support CQL. The number of index references per typical game is doubled, with noticable effect on updating a game and significant effect on the time taken to import large numbers of games to a database (see Comments dated 2 October 2017).

Fix problem which prevented exception reports being written to error log when importing chess games, and prevented the occurrence being reported in the progress window for the import.

Serious problems in the query language processing of the basesup component have been fixed.

The simplest clauses, like 'field le value', were fine. When a negation was added, 'field not eq value' for example, or a range clause was used, 'field above value1 to value2' for example, the request did not work.

Mis-spelling the 'and', 'or', and 'nor', keywords remains a problem. The query is processed but the clause following the mis-spelt keyword is ignored: 'White eq Jones ot White like Smi or Date is 2003.01.24' is treated as 'White eq Smi or Date is 2003.01.24' having typed 'ot' for 'or' for example.

Fix the problems in analyze multiple games or analyse checkmate and avoid the work-around.

The sign either problem had occurred was new analysis would no longer appear while navigating games. The counts shown by menu option 'Engines | Position Queues' would be stuck at some probably non-zero values.

The work-around was quit the engines and restart them (and try to avoid doing whatever caused the problem).

Add a menu action and keyboard equivalent for the O-O-O move.

When typing O-O-O there is usually a point where O-O, itself a possible legal move, has been typed. The change allows O-O-O to be input directly in cases where both O-O and O-O-O are legal moves. It remains possible, for example, to type O--O and then fill in the third O to get O-O-O.

Put correct version numbers in list of installed packages in "Help | About".

Analysis of all positions in a game or repertoire may be requested. All active engines when the request is made are used. The order in which a games' positions are analysed will vary between runs of ChessTab, so analysis for the board position could appear almost immediately or after some longer time.

The commands to start chess engines may be started from a list on a database in addition to a 'run program' dialogue.

A couple of faults in translating moves in the analysis produced by chess engines to PGN are fixed.

ChessTab does not use the functions changed in dependencies.

Error detection and reporting improved. Popup menus advertize the insert, edit, and delete actions correctly.

All previous versions used the value given in the White and Black tags as a player's name for indexing purposes. This version accepts multiple player names, in either tag, separated by colons as stated in the PGN specification. See comment dated 18 September 2016.

Consultation games must not be added to a database using earlier versions.

The selection rule "White like : or Black like :" will give an empty list of games if there are no consultation games on a database: if the earlier version of ChessTab supports selection rules.

You may need to export all the games using the earlier version of ChessTab, then import them to a new database using ChessTab-0.41. Certainly if the earlier version of ChessTab is before 0.40, but only if the database contains consultation games otherwise.

The program crashes related to using bitarray-0.8.1 were introduced in changes to the basesup component concurrent with ChessTab-0.39.1 and are fixed by changes to the basesup component concurrent with ChessTab-0.40.5.

The 'Move To' and 'Filter' buttons apply the rules introduced in ChessTab-0.40 on commas and periods to the search value entered. No need to remember not to type commas and periods which appear in player's names.

The amount of text in game selection rules is likely too variable for the list to be easily read, so show the rule's name only.

The list of games matching a position in a game or repertoire, or a partial position is always displayed, rather than being left blank until an action which causes the list to be refreshed is done. Colour highlighting is corrected in cases where a game, repertoire, selection rule, or partial position is closed while the database is closed. Closing a database, and then all the games, repertoires, and partial positions, leaves a blank screen rather than a screen showing some or all the list of games matching a partial position.

Redundant lists of items removed from object docstrings.

The restrictions on evaluating selection rules when using the DPT database engine have been removed as a consequence of the concurrent changes in the basesup component.

Hide selection, repertoire, and position, actions when using the DPT database engine do not crash program as a consequence of the concurrent changes in the basesup component.

Evaluation of selection rules when using the DPT database engine do not crash program. An information dialogue states evaluation has been suppresed.

Revert to starting each variation of chess engine analysis on a new line.

New component versions pgn-0.10.2 basesup-0.21 gridsup-0.19.4 rmappsup-0.38.4.

Paragraph removed because change left out in error. See comment dated 18 September 2016.

The values in the White and Black tags are stored as given but sequences of periods, commas, and spaces, are replaced by a single space for indexing by player name. PGN files often use consistent but conflicting ways of following the layout recommended in the PGN specification for commas and periods in these tags. The change attempts to achieve more consistent indexing of games from different sources.

See these nabble.com and stackexchange.com forum questions for ways to increase the memory available to processes on OpenBSD.

5 September 2016

A 2 million game deferred update using sqlite3 takes over 9.5 hours on FreeBSD, 15.5 hours on MS Windows XP, and 45 hours on OpenBSD. The differences are likely due to the memory available. Runs using bsddb3 on FreeBSD and OpenBSD take similar times to the sqlite3 runs.

DPT takes 11.5 hours to do the 2 million game deferred update on Windows XP, but 150 hours under Wine on FreeBSD. DPT seems to be unsupported now, but the scale of the discrepancy makes it worth mentioning. DPT's way of determining memory use does not work under Wine on FreeBSD and the work-around turns out to be slow.

Python's multiprocessing.Queue is used to communicate with chess engines but it does not work under Wine. Wine HQ Bugzilla bug 17195 and Python Issue Tracker issue 9099 are the closest references found which could be relevant.

When one of the chess fonts, Chess Cases for example, is added to the font directory for Wine in FreeBSD, ~/.wine/drive_c/windows/Fonts, it gets used as the default font for Text widgets created by the MS Windows version of Python tkinter. The same happens if instead the font is put in /usr/local/share/fonts so it is available for the FreeBSD version of Python as well.

But the font does not get used as the default font for Text widgets created by the FreeBSD version of Python. Nor does it get used in this way by the MS Windows version of Python tkinter running on MS Windows when the font is installed on MS Windows.

Versions used are FreeBSD 10.1, Wine 1.8.1, and MS Windows XP, Python 3.3.

Reading about fonts in the Tcl/Tk manual suggests the effect is not an error, but a consequence of the font selection rules.

The problem is resolved within ChessTab by stating the font required explicitely under Wine, taking advantage of the multiprocessing.Queue problem. However under Wine ChessTab is best run in a dedicated user, with the font installed in ~/.wine/drive_c/windows/Fonts but not in /usr/local/share/fonts, to avoid polluting the widgets of other programs running under Wine with the chess fonts.

Concurrent changes in the pgn package turned the minor problem of putting a space at the start of each variation's line into the major one of not putting each variation on a separate line. Fixing the major problem revealed the minor problem had been fixed too.

7 September 2016

The selection rule evaluator was written to use data stored using the apsw, bsddb3, or sqlite3 database interfaces without regard for the differences to the DPT database engine. Hence the problems evaluating selection rules when using the DPT database engine.

14 September 2016

The module and class docstrings look nicer when viewed with pydoc now the lists have been removed, but otherwise their content is the same.

18 September 2016

This paragraph was removed from the "Version 0.40" description because the change was left out in error: "All previous versions used the value given in the White and Black tags as a player's name for indexing purposes. This version accepts multiple player names, in either tag, separated by colons as stated in the PGN specification.". This change is applied in "Version 0.41".

18 October 2016

It became clear that both a ucinewgame and a setoption clear hash command are needed between analysis of each position by a chess engine to make the analysis consistent between runs of ChessTab. The random order in which positions are analysed is assumed to be why setoption clear hash is needed as well.

Positions are not sorted into game order for analysis because navigation of any displayed game or repertoire causes the most recent position displayed to be put at the front of the queue for analysis. It is not yet clear how Recursive Annotation Variations should be fitted into game order for analysis.

13 November 2016

It is possible to claim all the conditions mentioned in the comment dated 6 November 2011, see old history, have been met. DPT is no longer supported but the important element in DPT, bitmapped record numbers, has been transplanted to the interfaces to sqlite3 and bsddb3. Display of chess engine analysis is supported but the chess engines must be running on the same computer as ChessTab.

Problems found in ChessTab-0.41.5 will be fixed in ChessTab-0.41.6 and so forth, but new work will appear in ChessTab-0.42 and beyond. All earlier versions of ChessTab which survived previous culls have been removed from the website.

These things will be done before replacing ChessTab-0.41 as the stable version:

ChessTab-0.41.5 will be put on PyPI shortly.

2 October 2017

CQL distinguishes the moves played in a game from variations, a distinction not made in the partial position scheme. The decision to support this distinction forces a change to the content of index values, to refer to the variation indexed, making ChessTab-1.0 incompatible with earlier versions. Thus a new major version number is needed.

ChessTab-1.0 does not run under Wine at time of writing on my environment for these reasons:

The reason for running ChessTab under Wine is to use the dptdb database interface away from Microsoft Windows.

The number of index entries per game is more than doubled, from about 1800 to about 4200, to support CQL statements. On a computer with a 3Ghz CPU with 2Gb 800MHz DDR2 SDRAM and a SSD importing about 1.5 million games took about 826 minutes on ChessTab-2.0 compared with about 397 minutes on the previous version. This gives an estimate of about 75 minutes to do everything except the indexing, which does not seem inconsistent with what my notes of several years ago say about timings on a slower computer but I would have liked 120 minutes as the estimate here.

29 June 2018

The fix applied today to basesup-2.0 does not affect ChessTab, which will move on to use a later basesup version at a convenient time.

31 August 2018

A misleading error report is made at the at the end of game imports done by ChessTab-2 versions with the sqlite3 interface on Python3.5 or earlier. The import has completed successfully but the last line in the error report is 'sqlite3.OperationalError: cannot commit - no transaction is active'.

All the work to convert the original ChessTab-0 to ChessTab-1 and ChessTab-2 was done using Python3.6 and, for convenience, comparisons runs were done with apsw on Python3.5 and sqlite3 on Python3.6. For this change the combinations were apsw on Python3.6 and sqlite3 on Python3.5 and the problem got noticed.

ChessTab-0 treats the apsw and sqlite3 interfaces as unrelated code which happens to look very similar. ChessTab-2 treats the apsw and sqlite3 interfaces as the same code which uses either the apsw module or the sqlite3 module.

All the differences are implemented in basesup versions, and mainly means not using the features added by sqlite3 to standard SQLite in basesup-2 versions. The definition of a transaction in sqlite3 changes between Python3.5 and Python3.6, and ChessTab-2 needs the definition used at Python3.6.

Python3.5 and earlier are removed from the list of Programming Languages in setup classifiers.

8 November 2018

The dependencies were renamed because the old names seem poor for use on PyPI.

An '*' character used in a filter string seems to get treated as a 0-n wildcard character when used with the SQLite3 database engine, but as a literal when used with the Berkeley DB or DPT database engines. The behaviour with SQLite3 breaks the DataGrid class in the solentware-grid component but fortunately the program crash is delayed till the next scrolling action giving time for a warning and avoiding action.

2 October 2019

The problem fixed at ChessTab-3.0.6 was found after importing the PGN file referenced in the 'General Chat | 5949 moves : is this a maximum ?' thread on English Chess Forum at https://www.ecforum.org.uk in the post by Jesper Norgaard dated Thursday September 19, 2019 5:25 pm.

At time of writng my browser gives the thread reference as: https://www.ecforum.org.uk/viewtopic.php?f=2&t=10411 and the thread gives the PGN file reference as: https://drive.google.com/open?id=1v2V20kyQ-cyfAgGfc4okeV4xgY16aHRo and a reference to a discussion of the maximum number of moves as: http://www.chesschat.org/showthread.php?12548-Longest-possible-chess-game.

Importing the game into an empty database takes about 10 seconds. Probably reasonable as the game is indexed by 11790 positions and, on average, 19 pieces per position. Displaying the game from the database and navigating from one position to another takes a very long time: up to 3 hours for each action!

Moves '2692. Qb2c3', '4449... Qg1f2', and '5591. Qe6e7', were the only examples of this type of movetext found while investigating the problem.

1 November 2019

The table name changes for SQLite3 and Berkeley DB ChessTab databases, and the merging of tables for Berkeley DB ChessTab databases, were prepared first but delivery of ChessTab-4.0 depended on the successful implementation of '&', '^', '|', '&=', '^=', and '|=', operators for DPT's APIRecordSet class. The interfaces to SQLite3 and Berkeley DB databases already used shared code, in particular for these operations on their equivalents of APIRecordSet, and the interface to DPT had to be able to use this code too.

Two versions of APIRecordSet exist. APIFoundSet is immutable. APIRecordList can be changed using the Place() and Remove() methods; and these methods can be used to make the six operators used in the other interfaces.

The intial attempt to provide '&', '^', '|', '&=', '^=', and '|=', operators in the interface to DPT built with SWIG did not work, although these operators still exist in the DPT interface available for download.

It proved possible to provide the '&', '^', '|', '&=', '^=', and '|=', operators in the code that uses the DPT interface. Then use of shared code in all the interfaces just fell into place.

15 November 2019

The moves of a game are normally displayed as a single line, which will wrap depending on the size of the widget in which it appears.

The documentation for Tcl/Tk warns about the performance impact of large lines when tagged extensivly. Typical games of chess do not lead to large lines, and it was the longest possible game, see note 2 October 2019, which gave an idea on what large is in this context.

Displaying each full-move on it's own line counts as a bug-fix, hence ChessTab version 4.0.1, while changing things to always display games in a multi-line structure is definitely a new major release, ChessTab version 5.0 perhaps.

14 April 2020

Support for UnQLite and Vedis databases was added to test the tree implemented in the solentware-base.core.tree module: or how can a tree be implemented in a key-value structure. ChessTab seems to work but importing large numbers of games in one go does not seem possible. In favourable conditions half a million can be imported with 8G memory.

The Python dbm.gnu and dbm.ndbm databases were added, because it was now quick and easy to do so, to see if the absence of transaction support makes a difference. Not really, because dbm.gnu slows down by a factor of 3 after half a million games, possibly with more to come, while Vedis just runs out of memory.

These four engines are kept against the possibility replacing ordered keys by non-ordered keys will improve the performance in ChessTab.

28 April 2020

It is assumed the absence of problems, with maximum locks, importing games on OpenBSD is because transactions are not used. Editing games may not be a problem either because only the changed indicies are updated. The problem is exposed by running the 'castling options correction' tool because here the entire game, including all indicies, has to be deleted with code from one version of ChessTab and re-inserted with another version.

26 July 2020

The change in documentation style at CQL version 6.0.4 compared with version 5.1 made it practical to parse all filters. It is now clear there are many filters which ChessTab will never be able to evaluate because it is necessary to play through the game. The best ChessTab can do is reduce the number of games which have to be processed by CQL.

Empty square piece designators such as _d3 are now tranlated to the full form 'not squaremove eq 2320d3A and not squaremove eq 2320d3a' rather than the abbreviated form 'not squaremove eq 2320d3A or 2320d3a'. The abbreviated form does not work because there is a problem in the query languange provided by solentware-base.

2 September 2020

The rewrite of pgn-read at version 2.0 was prompted by 'http://portablegamenotation.com/LittlePGN.html', which I cannot access at time of writing. It included a sample game with lots of things wrong compared with the PGN standard. The game is in the pgn-read unittests as Little_01.pgn along with the minimum changes which allow the GameTextPGN class in pgn-read to handle it.

ChessTab-4.3 does not use GameTextPGN so it decides Little_01.pgn and it's derivatives have PGN errors.

On Microsoft Windows double click is needed to activate scrolling in the list of games matching a position when done over a row listing a game, but on OpenBSD a single click is sufficient.

A mixture of Text and Label widgets is used to display the lists of games, and the mix is changed between ChessTab versions 4.2 and 4.3 after remaining the same since the beginning. For example at version 4.2 the Tags column in the main list of games was a Text widget and needs the double click but is a Label now. The list of games matching a position uses a Text widget for each game at version 4.3 and it can be annoying to avoid the double click by clicking on the header row or the empty area.

Before ChessTab version 4.3 any Text widgets were to the right in a row, and scrolling was usually needed to get there, and it never became obvious the problem existed.

20 October 2020

Investigating why ChessTab's game and repertoire editor was not handling incomplete moves as expected led to finding the problem in pgn-read.

2 January 2021

The 'crafty' PGN files which used to be available at ftp://ftp.cis.uab.edu/pub/hyatt/pgn were a good source of noise between game scores in PGN format, usually in the form of email headers.

The collection of California games at www.chessdryad.com/games/CalBase.pgn has a good number of instances of the '--' move in PGN movetext. Most are in recursive annotation variations but a few are in the game line. References justifying use of '--' are hard to find, well I found it so, but the existence of 'chess.com/forum/view/general/what-is-a-null-move-and-how-to-make-it' gives '--' plausibility as an accepted extension to the PGN standard. '--' is mentioned as a minor point in a comment on 'reddit.com/r/chess/comments/47fexz/towards_a_replacement_for_the_pgn_format'.

ChessTab turns '--' and following moves in the variation into a '{...}' comment to avoid indexing the game for these moves. At some point in future I may change the indexing code to ignore '--' and following moves so these moves can be stored as found. One problem is some sources use '--' as a placeholder for an unknown move which may cause a move actually played later in a game to look like a stupid or illegal move.

3 January 2021

The first thing I do after publishing ChessTab-4.3.2 is concatenate some PGN files downloaded from 4ncl.co.uk and find the first game from the second and subsequent files marked as invalid PGN because of the end of file marker, Ctrl-Z, at the end of the preceding file. 4NCL PGN files published before 2012, which I downloaded many years ago, do not have the end of file marker.

18 November 2022

Most of the pylint suggestions on coding style were implemented, except for those saying 'too many ..' or 'too few ..' of something. Perhaps I chose the wrong way of dividing stuff up into classes, but the choices seemed reasonable back then (and still do).

Attribute names were given a "_" prefix unless pylint objected about access to a private attribute. Thinking the attribute should be public was not allowed as a reason to make it public.

There was, and is, a risk that accepting pylint's view on names of less than three characters has introduced as yet undetected problems. Quite a lot of changes were devoted to this.

31 March 2023

The Tcl via tkinter interface to Berkeley DB is a solution to the problem on OpenBSD where the available versions of Berkeley DB as ports are 3.1.17 and 4.6.21. Neither are supported by PyPI package berkeleydb, and bsddb3 which does support them is not supported at Python 3.10 or later.

The PyPI package lmdb was noticed, and appreciated, while working on the Tcl interface. It is seen as a long-term replacement for Berkeley DB as far as ChessTab is concerned.

The Python interface to Symas LMMD, lmdb, is based on PyPI package cffi, which sparked interest in rescuing the interface to DPT via cffi. The Swig interface to DPT stopped working at Python 3.8.

8 October 2023

Little progress on interface to DPT via cffi. The Swig interface to DPT has been revived on Developer Command Prompt for Visual Studio 2017 and Msys2. The x86 version seems fine but there are several unresolved problems with the x64 version.

15 July 2024

The large import strategy introduced at ChessTab 8.0 is not appropriate for the gnu.dbm, ndbm, unqlite, and vedis, database engines because they do not support BTree.

I have not yet seen a way of making the large import strategy usable on the DPT database engine. Partly because deferred index updates have to be done with transactions turned off, and at least hindered by DPT's fastload feature not working in x64 compilations.

I have not yet tried resuming an import after cutting power to the box running the import. It is unlikely I will test this on an OpenBSD box, but will see what happens on either or both FreeBSD and Windows 10.