Old Change History

New component versions basesup-0.20 gridsup-0.19.3 rmappsup-0.38.3.

New component versions basesup-0.19.2 gridsup-0.19.2 rmappsup-0.38.2.

New component version pgn-0.10.1.

A Numeric Annotation Glyph followed by a Game Termination Marker with no whitespace between them caused ChessTab to fail abruptly.

New component version pgn-0.10.

Handling of errors in PGN files is improved, including edited games saved with incomplete moves at the end of recursive annotation variations. Generation of values to index games by position is simplified and closer to Forsyth Edwards Notation, at the expense of compatibility with earlier versions.

The export and import functions can be used to transfer databases from earlier versions.

New component versions basesup-0.19.1 gridsup-0.19.1 rmappsup-0.38.1.

New component versions basesup-0.19 gridsup-0.19 rmappsup-0.38.

Positioning slider in scrollbar for sorted list of games now works, after changes in basesup.

New component versions basesup-0.18.1 gridsup-0.18.2 rmappsup-0.37.3.

Some additions to basesup and gridsup in support of changes to chesstab.

New component versions basesup-0.18 gridsup-0.18.1 pgn-0.9.1 rmappsup-0.37.2.

The unit tests of basesup, a dependency of chesstab, for the sqlite3 interface related to deleting records in the bitmapped record number classes usually fail (or sometimes pass) when run under Python 3.4 or 3.5. These tests always pass using Python 3.3, and these tests always pass using the apsw insterface on Python 3.3, 3.4, and 3.5.

Delete record in chesstab has not failed at all using any of these combinations and it is assumed the problem lies in the code running the test, not the code being tested.

The sqlite3 module is part of the Python distribution, but apsw is not and has to be installed separately.

An invalid PGN file is no longer exported when a recursive annoation variation starts at the end of a line: left parenthesis omitted if it was last character on line. Importing PGN files with missing left parethesis saves an error record rather than causing chesstab to crash.

New component version pgn-0.9.

The machinery to parse import format PGN generated from chess engine analysis is moved to the pgn component.

No changes to dependency components.

New component versions basesup-0.17.1 gridsup-0.18 pgn-0.8.3 rmappsup-0.37.1.

basesup:

Indexes created by the bitmapped record number version of the bsddb3 database engine interface when importing games were likely to be incorrect. The typing error which allows the problem has been corrected.

The other database engine interfaces, including the default sqlite3, are not affected.

Inserting new games, or editing existing games, by typing the game score are not affected.

Some code made redundant at release 0.35 is removed.

gridsup:

Internal adjustments which do not affect functions provided.

pgn:

Allow for attempts to import games from files that do not contain any games according to validation against the PGN specification.

It is beyond the scope of the pgn component to prevent such imports: which will occur if requested after the estimates have been produced. The entire file will appear as one game with errors if the import is done.

rmappsup:

No changes except new versions of dependencies.

New component versions basesup-0.17 gridsup-0.17 rmappsup-0.37.

The apsw database engine, an alternative to Python's sqlite3, is added after being used to investigate the problem with sqlite3 on OpenBSD by comparison. It suffers the problem too, and reacts the same way in the solution.

If both apsw and sqlite3 are installed, apsw is used because sqlite3 is part of the Python installation and extra action is needed to install apsw.

The problem also occurs on OpenBSD 5.7 i386 using the bsddb3 database engine, but a solution is not provided.

The modules which were used in Microsoft Windows executables generated with py2exe are removed.

New component versions basesup-0.16.2 gridsup-0.16.2 rmappsup-0.36.2.

Indexes dropped and re-created when doing bulk imports to sqlite3 databases.

Imports to empty sqlite3 databases are faster. Imports of more than between 65537 and 131072 games, depending on the size and history of the database, are usually quicker. The time taken to import less than 65537 games is not affected.

On sufficiently large databases, a comparatively small import may be slower. It is not known what sufficiently large means, but a database of 2 million games is nowhere near large enough because comparatively small is less than 65537 games.

New component versions basesup-0.16.1 gridsup-0.16.1 rmappsup-0.36.1.

A problem in the basesup bitmapped record import interface to Python's sqlite3 module is fixed. If a record is being inserted into a segment which contained records before the import started, existing index records were not located so they could be updated. The problem seems to have been there always.

No changes to dependency components.

No changes to dependency components.

New component versions basesup-0.16 gridsup-0.16 rmappsup-0.36.

Python's repr() and ast.literal_eval() replace both Python's pickle.dumps() and pickle.loads() and some functions in the ChessTab product when serializing a Python object for storage on a database or reconstructing it. It is no longer possible to opt out of using the serialization.

Databases created using earlier versions of these products cannot be accessed. In practice this means ChessTab databases must be converted to the new format by using the export actions in the old version and the import actions in this version.

ChessTab is not affected by the import problem mentioned for Results in 'Other Products'.

New component versions basesup-0.15.1 gridsup-0.15.1 rmappsup-0.35.3.

A problem in the basesup bitmapped record interface to Python's sqlite3 module is fixed. If a record is being deleted, any index value refering to this record which refers to exactly two records loses the reference to the other record instead. The problem seems to have been there always.

New component versions pgn-0.8.2 basesup-0.15 gridsup-0.15 rmappsup-0.35.2.

New component versions pgn-0.8.1 basesup-0.14.1 dpt3.0-dptdb-0.6.1 gridsup-0.14.1 rmappsup-0.35.1.

New component versions pgn-0.8 basesup-0.14 gridsup-0.14 rmappsup-0.35 including dpt3.0-dptdb-0.6 done at 8 February change.

The Microsoft Windows binary installers for ChessTab, along with it's versions of source distributions, have been removed. The setuptools install will download and install all the components needed when chesstab-0.33 is installed.

Opening repertoires are supported.

Export games in PGN format added.

ChessTab runs on Python 3.3.

sqlite3 becomes the default database interface.

Application installer (InstallChessTab.exe) no longer provided. Python 3.3 or later must be installed to run ChessTab.

Python Package installers provided for dptdb.

Export games in database as text added. Recursive Annotation Variation (RAV) to last move in a RAV is now inserted in correct location.

The pointer equivalents of typing game results now work.

The 2to3 conversion tool suggests many changes to coding style along with those necessary for the code to work at Python 3. The non-essential changes that also work on Python 2.7 are applied in this version.

An option to install the Microsoft Visual C++ 2008 Redistributable is added. You may not need to install this if other software has already installed it or an equivalent product.

Scrolling any of the widgets up or down by a single line works again after being broken in version 0.23. Scrolling up or down by the number of lines displayed in a widget was not affected.

Help | About text adjusted to match differences between the installation methods.

Naming convention adjusted to indicate Python version dependency more precisely.

Edit dialogues changed to commit transactions explicitly in sqlite3.

The sqlite3 database engine can be used but importing large numbers of games using sqlite3 is not a good idea at present.

A couple of problems editing partial positions corrected.

Two modules moved from rmappsup to gridsup because their sqlite3 equivalent is put there directly.

Problems with colouring rows when moving pointer though selected row or selected row through row under pointer corrected.

Problems displaying records when using the Insert or Delete keys or pointer equivalents (F11 not affected) after closing and reopening a database corrected.

Some things which should be in gridsup rather than chesstab moved there.

The update code for adding, deleting, and editing single games restored after being removed in error.

A relatively minor problem with colouring rows for games which had been displayed corrected.

The example large PGN import used, 1.5 million games, now takes just over 20 hours rather than the 24 hours still quoted in the help documents.

An obscure problem in the program chesstab_winedptmulti.exe has been fixed. This program is intended for use under Wine on PCs with less than 500Mb memory when importing more than 5000 games. Prior to this version the presence of a null value in a PGN Tag would cause an import run from this program to fail. For example [ Source "" ].

DPT is a good choice for handling the indexing of a chess game database. But the objectives of the DPT project constrain DPT to use files that do not grow as required. Thus a chess database has to be big enough to hold the games imported from a PGN file before the import starts.

The number of bytes in the PGN file is not good enough as a predictor of the required size of the DPT database (or extra space when adding to an existing database).

The number of games, positions, and pieces in those positions, in the PGN file is a good predictor of the required size of the DPT database. Getting the exact counts takes about 25% of time used to do an import. So the size of a PGN import is estimated to save time on large imports.

Earlier versions were points at which work on ChessTab stopped for a while or when changes to a component package were made for reasons external to ChessTab.

ChessTab has a 0.n version number because several things deny it production status:

6 November 2011

Six reasons are given (Version 0.17 above) for denying production status to ChessTab. The intention is to deal with these in the following order:

An SQL interface has been added to Berkeley DB 11g using a derivative of sqlite3 with the same interface. An interesting possibility is that Berkeley DB has been, or will be, modified to do index searches at the speed achieved in DPT.

10 November 2011

The blunder with database updates does not affect importing games. While releases 0-18 and 0-19 are broken, they do not do any damage to the chess databases (unless doing nothing counts as damage). It is the missing facilities that matter.

Release 0-19 will be removed soon. Release 0-18 will be patched (to 0-18-1) to match release 0-20 apart from the dptdb component.

10 December 2011

ChessTab can be moved to Python 3 now that sqlite3 can be used as the cross-platform database engine to fill hole left by removal of Berkeley DB support from core Python.

Importing games is much slower using the sqlite3 database engine but optimisation has not been investigated (the disk I/O light is more or less permanently on).

The sqlite3 module takes priority over the bsddb module, both are supported at Python 2 but only sqlite3 is supported at Python 3. The bsddb3 module is available for Python 2 and Python 3 and takes priority over the sqlite3 module. On Microsoft Windows the dptdb module takes priority over the sqlite3 module.

Releases 0-19, 0-20, and 0-21 are withdrawn on publication of Release 0-22. Release 0-18 will be patched (to 0-18-1) to match release 0-22 apart from the dptdb component.

24 March 2012

The default action on ending a transaction in sqlite3 is to rollback (backout) any changes to the database. The default action on ending a transaction in DPT is to commit any changes to the database. The transaction capabilities of Berkeley DB are not used. If an error occurs during a transaction in DPT the default action is backout (rollback).

The 'other products' page, accessed via the 'download' page is added to the website. Some of the products use the same packages as ChessTab. Adjustments to the source code repository were done to ease maintenance and the visible consequence is the change in naming convention.

14 September 2012

The intention was that release 0-25 be the first one to work on Python 3. The volume of non-essential change along with some non-trivial changes needed in one of the 'other products' made it reasonable to get the non-essential changes out of the way before releasing the conversion.

29 September 2012

The option to install Microsoft Visual C++ 2008 Redistributable should have been present from the start. Python documentation suggests it is possible to embed the redistributable in the application (licences permitting), which would avoid the dialogue with the Microsoft installer. I could not get this to happen and the whole installation area has to change at Python 3 anyway.

15 June 2013

At Python 2 strings {type('') is str} and bytes {type(b'') is str} are equivalent. Unicode strings {type(u'') is unicode} are different.

At Python 3 strings {type('') is str} and unicode strings {type(u'') is str} are equivalent. Bytes {type(b'') is bytes} are different.

Sloppy assumptions about the equivalence of strings and bytes made it seem impossible, and at least not practical, to produce Python 3 versions of chesstab capable of accessing databases generated by Python 2 versions of chesstab. The PGN export facility will be used to solve this problem.

The bit-mapped record numbers option added to the interface to bsddb3 in basesup, which follows the example of DPT, is in theory a solution to the problem that DPT is Windows only (comment 6 November 2011). But packages and installers for bsddb3 on Python 3 are not yet common, and while building bsddb3 from source on FreeBSD (and presumably other BSDs and Linux distributions) is almost trivial, it seems not so trivial on Windows.

sqlite3 is cross-platform but can be unacceptably slow for some functions compared with the bit-mapped options when dealing with large databases.

29 June 2013

Export games as text added to chesstab so that databases can be moved from Python 2 to Python 3.

Deferred update, following the example of DPT, now works on the bit-mapped record numbers option added to the interface to bsddb3 in basesup in Python 3 when adding to a database which already contains records. The absence of the relevant code was seen when testing the sorting of chess games by chesstab for export in PGN format.

20 July 2013

Partial position searches evaluated using bit-mapped record numbers on bsddb3 under Python 3.

It is now reasonable to save the search answers obtained using bsddb3, and this is done. The initial evaluation is still about an order of magnitude slower than the DPT version, but a lot quicker than before.

31 August 2013

Partial position searches evaluated using bit-mapped record numbers on sqlite3 under Python 3.

SQL statements cannot be used to get at the records in the natural way. But sqlite3 is being used because it is the only cross-platform database interface in the Python distribution, not because SQL is the language.

InstallChessTab.exe is withdrawn because it has not proved possible to make it work at Python 3.3 when built with either py2exe or cx_Freeze.

Both bsddb3 and bitarray are provided as Python Packages which should build and install on all platforms if the necessary toolsets are installed. But I have not succeeded in building bsddb3 on Windows.

Response times are quickest with dptdb, sometimes significantly, but there is a risk of having to do database administration tasks, increasing size in particular, while such risks are practically absent with bsddb3 and sqlite3. The bitarray module may, or may not, give noticeable improvement in response times for bsddb3 and sqlite3.

8 February 2015

The DPT website, www.dptoolkit.com, expired on 15 January 2015. At time of writing that website had not been re-registered for any purpose.

All links to www.dptoolkit.com have been removed from www.solentware.co.uk but the product references remain in place, sometimes edited to say the DPT website has expired.

The author tells me DPT will be put on a repository at some point, citing sourceforge as an example.

The current dptdb distributions are replaced by a source distribution which downloads the zip files containing a recent version of the DPT API source code and documentation from www.solentware.co.uk.

In PyPI terms the development status is described as '7 Inactive'.

18 May 2015

The bit-mapped record numbers option is now the only one supported in the interface to sqlite3. At some point the non bit-mapped option will be removed from the bsddb3 interface as well. The distinction is not relevant to the DPT interface.

At present importing 1.5 million games to an empty bsddb3 or DPT database takes about 7.5 hours, but about 11.5 hours to an empty sqlite3 database. The sqlite3 time can be reduced to about 7.5 hours too, by creating the indexes after the import: which is how the other two manage it. Simply dropping and creating indexes will not do when updating a populated database because bitmaps have to be spliced together. DPT was built to handle the splicing efficiently, and hopefully transplanting the solution in the bsddb3 interface will be good enough.

14 June 2015

The non bit-mapped option has been restored to the sqlite3 interface because Results, see Other Products, uses it but cannot use the bit-mapped option yet. Use of the non bit-mapped option has not been restored to ChessTab.

ChessTab always uses a separate process to import anything: so it does not suffer the threading (DPT or sqlite3) and record locking (DPT) problems met by Results.

ChessTab was designed to work with DPT: so has to use a separate process where the use of transaction backout can be disabled to do potentially large volume imports more quickly. The equivalent in sqlite3 does not need a different process or thread, but the user interface can be kept active by doing so. The bsddb3 version of transaction backout is not used at all, but probably should be.

The significant change for ChessTab is not opting out of the serialization rather than the different technique.

18 June 2015

The pydoc-g-exhausts-the-stack-td223979.html document at openbsd-archive.7691,n7.nabble.com described the problem encountered in ChessTab. It seems the choice of --enabled-threads option leads to the problem because adjusting ChessTab to avoid doing User Interface actions (tkinter calls) in the wrong thread stopped the problem occurring.

The adjustments seem to be compatible with the choices made in FreeBSD Ports and in Microsoft installers for Python and Tcl/Tk.

30 June 2015

A MemoryError exception (insufficient memory) was raised after processing a few thousand games when importing on OpenBSD 5.7 using the sqlite3 database engine. Splitting the task into chunks of 5000 games, each done in a new process after the preceding one finishes, seems to allow the task to be completed. An import of 2 million games, not chunked, took about 17.7 hours on identical hardware using FreeBSD but it is estimated, from progress so far, that the chunked import will take at least 5 days on OpenBSD (job terminated!).

Splitting into chunks did not eliminate the MemoryError exceptions entirely: three chunks failed during first 20% of the 2 million game import with 2Gb memory available. It is assumed the failures would not happen with more memory.

The problem fixed at version 0.35.3, published on same day, was found while testing version 0.35.2 on FreeBSD. In hindsight it is a surprise the error fixed at 0.35.3 did not occur until six hours into the run. The import, when split into chunks, will take just as long on FreeBSD as on OpenBSD.

The 2 million game import to an empty database, without chunking, was done on these hardware and operating system combinations differing only in disk type:

Windows, rotating disk, DPT database engine: 11.5 hours.

Windows, rotating disk, sqlite3 database engine: 78 hours.

FreeBSD, solid state disk, sqlite3 database engine: 17.7 hours.

FreeBSD, solid state disk, bsddb3 database engine: not done yet. (~12 hours maybe by comparison with 18 May 2015 comment.)

6 July 2015

The bitmap splicing problem cited in the comment of 18 May 2015 as the reason for not dropping indexes while importing to an sqlite3 database has been resolved for some common cases. The high segment, and the next one, are filled before the indexes are dropped. The next one is included to guarantee that the common imports of up to a few thousand games are never affected by excessive overheads of dropping and creating indexes for comparatively large databases.

The 2 million game import to an empty database, without chunking, was done on this hardware and operating system combination:

FreeBSD, rotating disk, sqlite3 database engine: 29.3 hours.

It is assumed the time on Windows would be similar, if not exactly the same, so dropping then re-creating the indexes gets the import done in less than half the time compared with that given in the comment of 30 June 2015.

15 July 2015

The earlier attempts to avoid the MemoryError exception, which happens only on OpenBSD as far as known, do not work. Only the first chunk succeeds in committing any changes. All subsequent chunks suffer the MemoryError exception and backout (rollback) any changes.

The only solution known is do a single chunk, then reboot the machine, then do the next chunk.

The MemoryError exception occurs well after game 40000, but before game 65536, for typical PGN files on a machine with 2Gb memory. 65536 is the database segment size, and the optimum chunk size is the largest factor of the segment size which does not suffer the MemoryError exception. That is 32678. Given this, a segment is filled by two chunks, with the first chunk done in half the time taken by the second chunk. If more than two chunks were needed to fill the segment, only the first chunk would take half the time of each later chunk.

I found the apsw interface while looking for plausible causes of the problem. It suffered the MemoryError exception on OpenBSD too, which suggested a cause using Sqlite3 relative to OpenBSD. But perhaps not because it turns out the bsddb3 interface suffers the MemoryError exception on OpenBSD too.

Maybe my code relies on a configuration option not used when building the OpenBSD Python package. There are no obvious knobs to turn in the OpenBSD Python port when building from source. Eventually I may try ignoring even the OpenBSD Python port and build directly from Python.org sources using the knob settings in the FreeBSD Python port if possible.

The work with apsw did identify the places where explicit 'begin transaction's are needed: which will be useful when adding transactions using the bsddb3 interface.

The OpenBSD 5.7 i386 apsw package turned out to be broken: the Cursor class is missing the fetchone() method. The OpenBSD port of apsw is consistent here. Using the source from the apsw home page worked fine.

26 July 2015

pydbitkit is added to the Other Products page because the modules implementing the client-server conversation will be adapted to provide communication between ChessTab and chess engines.

This marks the start of doing something on communicating with chess engines: see the comment dated 6 November 2011 and the changes note dated 30 August 2013 for version 0.30 on things to do in ChessTab.

15 October 2015

Communicating with chess engines running on other computers is not supported at release 0.37 of ChessTab.

Saving the analysis on the database is not done either. This will be added later, along with additional search functions based on PGN Tags and changes to the User Interface so each list of things or each game can occupy the whole of ChessTab's display area.

Saving the analysis will be done before communicating with chess engines on other computers. This order seems sensible, and the easier task gets done first.

The uci package provides the interface to chess engines. It is meant to work closely with the clientserver package to get analysis done by chess engines on other computers. Fortunately it proved fairly easy to break the dependency on clientserver so chess engines can be used on the same computer as ChessTab without installing the unfinished clientserver package. The clientserver and uci packages are not listed under components, but the uci package is on the website for the installer to download.

14 November 2015

At present all chess engine analysis done while a database is open is saved on the database. This will be changed so just repertoire game analysis is saved automatically, and game analysis is saved on request. It is not clear what the rules should be yet: maybe data storage is cheap enough that "save everything always" is fine.

Nothing done on additional search functions yet.

5 April 2016

I do not know why the unit tests for deleting records using the sqlite3 interface usually fail at Python 3.4 and 3,5, but always pass at Python 3.3; nor why the tests always pass using the apsw interface. It is possible to split the tests into smaller pieces such that they always pass, but that seems like cheating.