BeniBela online


18th September 2022

I have updated my oss packages. All the changes were already in the public repositories, but some packages released here had not been updated in years:

  • Internet Tools now have complete support for XPath 3.1 and XQuery 3.1, and also parts of 4.0. Since XPath 3.1 supports querying JSON in a standard way, I am deprecating most of the JSONiq implementation. XML/HTML serialization now produces strings that correspond byte-for-byte to test cases from the XQuery test suite. In the non-standard extensions there is now a function to change document nodes and one to submit HTML forms. It is a huge update with 900 commits, far too many to present them here. The package includes a changelog listing each commit.
  • BBUtils: I am moving away from procedural programming with raw pointers to something more object oriented. Hence, I added a string view, array list, a better string builder and codepoint enumerator for string processing. The objects are stored on the stack, so this is more efficient than ordinary heap-allocated classes.
  • Big Decimal Math: fix some string parsing and float conversion corner cases.
  • TTree­List­View: fix that you cannot change options during form creation on gtk2.
  • TCommand­Line­Reader: more flexible syntax and restored Delphi compatibility.
  • HAMT: some utility functions.

I have also added actual version numbers to the packages. This might be the last update that still supports FPC 3.0.4.

12th March 2022

A long time since the last update!
Meanwhile, over a year ago, I finished my PhD thesis and became a doctor.

26th November 2017

I have modernized the BBUtils and Internet Tools: Lengths are stored in SizeInt, so they will support >2GB data structures on 64-bit systems. If you were using functions that return lengths in var parameters, you need to adjust your variables as well, or it will not compile.
So far the units assumed that all strings are UTF-8 encoded, which was fine on FPC 2, but could return non-sense with the codepage aware strings of FPC 3 when used on non-utf8 strings. Now all functions in bbutils either use DefaultSystemCodePage encoded strings, when they do not care about the encoding or UTF8String when they return UTF-8 strings. The Internet Tools will convert input files to DefaultSystemCodePage rather than to UTF-8, and most string functions will switch between 1-bit encoding and UTF-8 depending on the default codepage. However, some functions like fn:normalize-unicode have no meaningful purpose for non-UTF8 strings, so they will just assume their input is still UTF-8.
Furthermore, the BButils do not support Delphi anymore. I always tried to keep it compilable with Delphi 4, but that is now nearly 20 years old. The Internet Tools now require FPC 3.
Last but not least their test cases are now run on Travis.

26th March 2017

Time for an update of the Internet Tools that does not involve XQuery:
Downloaded data is now returned through a callback function, so it can be written directly in the destination buffer without copying, no matter if the destination is a string, stream, or file.
The various url/hex encoding functions TInternetAccess.urlEncodeData, xquery.urlHexEncode, bbutils.strEscapeToHex are merged and the default escaping mode follows html5.
The cookie handling is updated to RFC 6265 and the cookie origin rules are actually enforced, i.e. a cookie set by one server will not be sent to another server

29th January 2017

I have a new way of sorting my mails.

It is the ten year anniversary of VideLibri: 4183 commits, 88 kLOC256 kLOC when counting test cases, etc., says Open Hub, 30 000 test cases, 18 different supported library systems/APIs used by at least hundreds of libraries. And yet there is still no English translation.

And to whoever owns can you please stop posting 1000 spam entries to my guestbook every day? They are being auto blocked anyway. 30375 blocked comments this year.

3rd December 2016

It seems people have a hard time knowing, which functions they can use in Xidel / Internet Tools, even though most are described in the XPath/XQuery, EXPath, or JSONiq standards. Therefore I have assembled a complete list of all implemented functions.

25th November 2016

I have made my XQuery interpreter much faster, after profiling it several times over the XQuery Test Suite. Some key improvements are:

  • The query is fully parsed before being evaluated. Previously some parts were left as a string and only parsed during the evaluation, e.g. axes names. Although this change makes it of course slower when running many queries with untaken branches.
  • Some specific query rewriting, e.g.: a and b to if (a) then boolean(b) else false(). In XPath conditionally filtering and index access of a sequence are the same, so there always has been a query optimizer to detect constant filters. But I had not marked last() as constant, so it was evaluated for every element. Marking it as a constant was the biggest speed-up, O(n), although it only affects queries using last() on sequences with ten thousands of values.
  • Local variables are no longer accessed by name but by their stack position. Although it still pushes and compares their names for debugging, as the speed difference of that was negligible.
  • {$IMPLICITEXCEPTIONS OFF} sprinkled around everything. Otherwise, FreePascal surrounds every function using non-const interfaces, arrays, or strings with an implicit exception block, including a long jump. Setting a long jump on every interpreted instruction definitely hurts performance. Unfortunately, there are still places, where exceptions actually might occur and the directive is not applicable, which means either I need to stop using functions (->byte code) or stop using fpc interfaces.
  • Avoidance of memory allocations: First temporary arrays are replaced by one big stack that only grows during the evaluation, avoiding reallocation. Secondly freed value instances are cached on each thread, so values can be created from the cache without any memory allocation. This also avoids a constructor call, which is especially helpful for interface instances, because they have two VMTs initialized by the constructor.
  • At most places, it does not use interfaces anymore, but pointers to interfaces, which is weird, because interfaces are only used here as a replacement for smart pointers. For the stack and list, it was also helpful to bypass the automatic ref counting, by treating the smartpointers as normal pointers and memmoving them around. Having interfaces clearly was a mistake.
  • Node names are hashed and searched by hash, which skips most nodes that do not match the queried name without ever actually comparing the name.
  • All namespaces URLs are cached and normalized to one specific string instance. So if a node in the correct namespace is found, it never needs to compare the namespace URLs itself, because they are the same pointer. Although I had to write my own function for string comparisons, before this had any effect.

It also somewhat breaks backward compatibility, e.g. by replacing my old TEncoding data type with fpc 3.0's TSystemCodepage.

Another huge problem with Pascal is the lack of any well-designed collection library. When it did not have generics, everyone reimplemented collections to contain one specific type. Then it got generics and every collection was reimplemented while keeping the old ones. Then Delphi got generics, and every collection was implemented again Delphi-compatible. In the end, there are a dozen collections of which neither works well.
Here I have compared all the available hash maps.

20th August 2016

I have changed the webpage structure slightly. If you had set any links to me, you might need to update them.

13th June 2016

My Internet Tools (and thus Xidel and VideLibri) have now a complete XPath/XQuery 3.0 implementation. It has probably become one of the most standard conformant interpreters out there. The next goals would be performance improvements and support for html5 as well as the still preliminary XPath/XQuery 3.1 standard. Remember the survey, if it one day should default to 3.1 instead of JSONiq.

20th April 2016

"How do you handle JSON in the XQuery world?", was a question that just arose on the XML mailing list and is quite relevant here. Two competing standards were made, the real XQuery 3.1 currently being specified by the W3C, and an earlier standard JSONiq, which exists in two versions, a standalone version and a JSONiq extension to XQuery 3.0. My Xidel is based on the JSONiq standard but might be updated to XQuery 3.1, when it is finished.
Let $object be an object, $objects a sequence of objects, $key a key (which can be any value in XQuery 3.1, but is converted to a string in JSONiq or current Xidel), $array an object, and so on...
Object values can be accessed in the following ways:

  • The dot operator $object.key is of course the most natural to access JSON object fields and familiar to every JavaScript developer. However, if you want to use it in XQuery, you encounter a problem: XQuery variables names are XML names and thus can contain dots, as in let $ := 123 return $
    Hence this notation is neither allowed in XQuery 3.1 nor the JSONiq-XQuery-extension.
    Standalone JSONiq wants to be JavaScript-like, so it takes another approach and forbids dots in variables names, making the standard syntax legal and unambiguous. It treats . as a "dot operator" and allows computed keys as $objects.(any expression), queries for non-alphanumeric keys like $objects."special key name", or keys referenced by another variable $objects.$indirect.
    The last two cases are also supported in Xidel, as they are unambiguous. A variable name cannot contain " or $, and $objects. cannot be a single variable, as it is not followed by an operator. $objects.(any expression) is not allowed, as the variable $objects. could be a function that is called. Other unambiguous and allowed cases are after the end of parentheses, e.g. ($objects).key or {..object constructor..}.key. A long time ago, before VideLibri became XQuery compatible, I did not care about these details and just used the $objects.key syntax. From that time Xidel still has a legacy mode in which it checks, if a variable with name $objects.key exists. If it exists the expression returns the value of that variable, otherwise it searches for a variable $objects containing an object. But this mode is disabled in the default settings.
  • The functional notation $object("key"): It is perhaps the most natural syntax in XQuery, you treat the object as a function that you can call with a key to read the corresponding value. Therefore it is allowed in all XQuery variants, but not in standalone JSONiq.
    In XQuery 3.1 it is an error to use this notation on non-JSON types (non-functions) or sequences of multiple objects, in the JSONiq extension or Xidel it returns an empty sequence. Those two also allow $objects() to retrieve all keys of an object.
  • A lookup operator $objects?key. The functional notation is not always easy to read, so the W3C has added the ?-lookup operator to XQuery 3.1. It can be used similarly to JSONiq's . operator, allowing $object?key or $objects?(calculated key). Latter form can query multiple objects and multiple keys as ($object1, $object2, ...)?("key1", "key2", ..). If the left side is omitted, the current context item is used for it. You can also use $objects?* to get all values in the object. It is an error to use it on non-JSON types.
    I have not come around to implement this in Xidel, but later versions might support it.
  • A XPath like query $objects/key or $objects!key. This was the original XPath syntax to query XML, so it seems to be a good syntax to query JSON, too. Thus I have added this syntax to Xidel, but none of the other languages support it. Like W3C's ? it can be used for multiple queries ($obj1, $obj2)/(key1, key2) or to retrieve all values in the object $objects/*. It can also recursively search for a key in all descendant objects with $objects//key. For compatibility with XQuery, it raises an error if used on non-JSON (non-XML).
  • Helper functions: All variants have a bunch of helper functions. For example JSONiq (extension/Xidel) have keys($objects) (or jn:keys($objects)) to get the keys of all objects, ignoring non-objects. They also have values($objects) (libjn:values($objects)) to get all the values. XQuery 3.1 has map:keys($object) for the keys, map:get($object, $key) to get a single value, or map:size($object) for the number of keys in an object. These functions raise an error for anything that is not a single object.
For arrays, a similar number of notations exists. Surprisingly to JavaScript developers the first element in the array has the index 1.
  • The JavaScript notation $array[index]: This is already used in XQuery to get the element of a sequence, and an array is not a sequence. So this syntax is sadly not supported anywhere.
    However, standalone JSONiq has at least $arrays[] to get all values of one or more arrays, ignoring non-arrays.
  • The functional notation $array(index): Similar to objects, this syntax is allowed in all variants except standalone JSONiq. Again XQuery 3.1 raises an error on non-arrays (non-functions) or invalid indices, while JSONiq extension and Xidel return an empty sequence. Later also have $arrays() to get all values of the array.
  • The lookup operator $arrays?indices of XQuery 3.1 can also be used for arrays and returns the values with the corresponding indices. $arrays?* will return all the values in the array.
  • The $arrays[[index]] operator. Standalone JSONiq uses double brackets to get array elements, returning the empty sequence for invalid indices. This is not supported in Xidel and I see no reason to add it.
  • The helper functions: JSONiq has members($arrays) to get all values in a sequence of arrays, flatten($arrays) to get all values recursively and size($array) to get the size of a single array. In JSONiq extension or Xidel they are called jn:members, libjn:flatten and jn:size. XQuery 3.1 has similar functions array:flatten and array:size, as well as array:get($array, $index) to get a single value, which raise errors on non arrays.
Multiple notations also exist for creating objects and arrays:
  • { $key: $value, ... } is the normal JavaScript syntax to create objects and is supported in JSONiq and Xidel. $key and $value can be any value, but the key is cast to a string and raises an error if that conversion fails. {} logically returns an empty object. {| $objects |} can be used to merge multiple objects with distinct keys to a single object. A further JSONiq syntax not yet implemented in Xidel is { $key ?: $value } which omits the key/value pair, if the value is the empty sequence. In JSONiq XML values are copied, erasing the node identity and relationship to their ancestors.
  • map { $key: $value } the special syntax of XQuery 3.1 to create an object. The key is cast to an atomic value (not a string!) and the value can have any type.
  • [$value1, $value2, ...] the standard JavaScript syntax to create an array. It is supported in any language, although with different semantics. In XQuery [$value] always creates an array of size 1, containing the single value $value, even if it is a sequence like (1,2,3). In JSONiq and Xidel [$value] contains so many elements as $value, e.g. for the previous sequence the array will have size 3.
  • array { $value1, $value2, ... } the XQuery 3.1 syntax, which creates the same array as [$value1, $value2, ...] would create in JSONiq.
So to summarize the supported notations:
NotationXQuery 3.1JSONiq s.a.JSONiq ext.Xidel
Object access
$object.key, $object."key"noyesnoyes
$objects/keys, $objects!keysnononoyes
Array access
Object/array creation
{key: value, key2: value2, ...}noyesyesyes
map {key: value, key2: value2, ...}yesnonono
[value1, value2, ...]yesyesyesyes
array {value1, value2, ...}yesnonono

Btw, another advantage of my new webpage backend is that Xidel can dump all these crappy news in an RSS feed .

21st March 2016

I have migrated this webpage from my JAXMLP language to XQuery. It is a great example of how to generate a static webpage with Xidel. Although keeping the HTML as XQuery expression, loading it as a string, evaluating it with Xidel's eval function, and writing the output with an EXPath function is not standard XQuery. But surely other XQuery processors can evaluate XQuery, too.
JAXMLP worked on a byte stream and inserted XML elements parsed with the standard Java XML library, so the input had to be valid XML and the output could be anything. Xidel works on a tree representation, so it always creates valid HTML. Although Xidel's default parser accepts anything as input, unlike JAXMLP it cannot dump a CDATA section containing invalid X/HTML in an output stream, so I still had to change and correct the input files.
Three things were surprising: First that PHP's <?php doubles as XML processing instruction, so a PHP file can be valid XML. Secondly that JAXMLP's semantics was similar to XQuery's. {{$IF download/screenshot != ""}}..{{$/IF}} becomes {if ($download/screenshot) then .. else ()}, which even has the same number of characters. Third, that Xidel is 5 times faster than JAXMLP, despite the fact that JAXMLP just copies the data from the XML to the output, while an XQuery function in Xidel cannot change the immutable output, so it has to copy every node and children repeatedly in O(n^2). I blame the JVM overhead. On the very first start, Xidel was even 30 times faster.
The big advantage of Xidel is that I can now use all the overcomplicated XQuery functions on the webpage, like format-date to translate the dates between German and English. I hope this change did not break the IE5 compatibility, but I cannot find that IE version to test it anymore.

18th March 2016

Another update of the Internet Tools: All the format-* functions are now implemented, which gives almost 99% XQTS-conformance for XPath/XQuery 3.0, too. I did not plan to implement them as they are so silly, but then I was stuck in an airplane and too tired to work on anything else, so they seemed an ideal task. You can still see a few failing format-* tests in the report, one of the blocks with the most failures, but there the W3C just seems to have forgotten, what the functions are supposed to do.
Here is an example to show just how silly and overcomplicated they made a simple int-to-str conversion:
("0", "٠", "𝟘", "𝟎", "߀", "೦", "꧐", "༠", "i", "a", "w", "w;t", "w;o", "W", "W", "w;o", "w;o(-er)", "#,0") 
! format-integer(position(), ., if (position() >= last() - 2) then "de" else ())

7th February 2016

After way too many years I have finally completed my XPath 2.0 interpreter. It can now pass every test case in the XQuery test suite successfully, even the weird ones, e.g. parsing dates preceding the universe like New Year's Eve -25252734927766554-12-31.Except for 0.1% of the cases that are just invalid or fail due to reported bugs in other libraries. . Next goal is finalizing XQuery 1.0 ( mostly fails to properly resolve dependency cycles in function/variable declaration ) and then XQuery 3.0 (lacks some minor things and the integer/date formatting functions, which are way too overspecified. I stopped reading when they described the gendered integers of foreign languages.).
It now requires FLRE as new regular expression library. There is still a compile option to use Sorokin's old library, however, without FLRE it fails a lot of test cases.

11th November 2015

So TeXstudio lost the community vote, but it was still picked as Staff POTM. One of the few open-source projects that received a SourceForge Project of the Month award twice.

The new version of the Internet Tools has now a LINQ-like interface, e.g. you can write

query('1 to $_1', ['100']).filter('. mod 2 = 0').query('sum($_)')
to get the sum of all even numbers below 100.

16th August 2015

You can now vote:

  • If you rather want to use XPath or XQuery in Xidel (on Google Forms, so you need to login). Too bad that XQuery uses & as escape character, so URIs are not valid XQuery strings.
  • And for TeXstudio to earn the SourceForge Project of the Month award. (let's just hope SourceForge does not go offline again)

4th July 2015

It is astonishing how much the Internet affects the efficiency of programming. I did not have Internet access for some weeks, and everything went much faster without all those distractions. So I could implement the new XQuery 3.0 for-syntax, a query command as complex as the SELECT statement of SQL, in seemingly less time than it took to write this news entry.
Thence there is now a new release of Xidel and the Internet Tools that supports almost the entire XQuery 3.0 syntax.

Also, Xidel has now a page on a custom domain. Its source is triple hosted (github, bitbucket, and here; perhaps add Gitlabs, too?), like my other projects. (Except TeXstudio, it is too big. But people already made Github mirrors) SourceForge seems to have earned a really suspicious reputation lately. I was hosting projects there to have them on the most trustworthy source, not the least.

8th February 2015

On New Year's Eve, more than a year after the first attempt, I finally managed to evaluate the entire new XQuery Test Suite in all configurations from XPath 2 to XQuery 3 without aborting due to some unhandled exception. I made a really bad decision when implementing everything without error codes at the beginning, and the XQTS evaluation are far too complex.
Turns out the XQuery 3 support is still lacking, even after implementing Q{}, higher-order functions and spending the last months adding the error codes.
At least the higher-order functions seem to work perfectly in the new version. And the percentage of passed XPath 2 tests remained almost the same except for these codes.

13th October 2014

All my smaller open-source repositories are now also mirrored here and on Bitbucket, rather than only on GitHub (So it won't matter if Russia blocks GitHub again like last week). Although the real big ones are still solely on Sourceforge.
And I just noticed that my links on Delicious are still available there. I thought Yahoo had shutdown these servers years ago and the links were lost, but it seems they were just temporarily unavailable, while they sold the site. Perhaps time to fetch the links back, too.

23rd August 2014

If someone is curious where I was last month: I went to conferences (protip: try to avoid transit at JFK when flying across the Atlantic), and presented two papers: One about my PhD project (warning: pdf) and one about VideLibri/XQuery.

30th November 2013

(not translated)

Hier ist, was ich bisher so von meinen alten oder neuen Programmen gefunden habe; viele Spiele und Schulsachen:

  • Ein uraltes Mathelernspiel für Kopfrechentraining
  • Die vollständige Version von Mausitanien
  • Ein Kampf-Simulator für das "Stadt der Diebe"-Buch, damit man bei Monsterbegegnungen nicht selbst würfeln und rechnen muss
  • Defendrix, ein Spiel für die Windowskonsole mit einem ASCII Raumschiff. Sieht am ältesten aus, ist aber das neueste dieser Spiele. In alter Tradition ist es extrem schwer zu spielen
  • Ein Farbratespiel ähnlich wie Mastermind
  • Mein altes Chemieheft. Interessant nur, weil ich dafür ein eigenes Dateiformat geschrieben habe (aber html wäre besser gewesen)
  • Ein unvollständiger Chiffren-Decoder. Für die letzten Übungsblätter wäre er trotzdem nützlich gewesen.
  • Ein Referat über Euglena. Mit einem Java-Applet für eine virtuelle "Simulation" von dem Einzeller.

Das ist aber noch nicht alles.

19th November 2013

There was a bug in the string conversion of the big decimal math. It is fixed now. (very surprising since I had a test for this case. Seems like I forgot to run all the tests)

Also, the bbutils now work with Delphi (4), since there was a feature request for that...

18th November 2013

I moved my page to a new web hoster, where it has can astonishing 10 GB instead 30 MB quota.

Therefore I can now upload some things, which would have been far too big for the old hoster. Starting with my master thesis. (It is even in English, first English pdf here)

There is also an update of my internet tools. The biggest change is that the XQuery engine can now calculate with arbitrary precision decimal numbers. This replaces my old 65-bit arithmetics and allows you to use almost the full range of xs:decimal (well the full range would be infinite, but for that, you need more memory...). Now it also supports the newest JSONiq standard (a standard that changes every few months is a little bit annoying), some minor operators of the XPath/XQuery 3.0 standard (too busy to implement the full standard right now), and the type system was almost completely rewritten in preparation of xml schema support (meaning there is no xml schema support, but it could be added by just writing new source and not modifying the old one).
The arbitrary-precision bcd floats can also be used as a standalone unit, independent of the internet tools.

And a comment to the last news: TeXstudio actually won the sourceforge award!

13th July 2013

A simple update for the Internet Tools (with changes needed for VideLibri): <s>-elements (shortcut-elements to evaluate an XQuery/XPath statement) are allowed in multipage templates, and the XQuery engine supports more functions to change url-encoded data. Also, fixes the progress calculation of the synapse wrapper.

And as so often there is a new TeXstudio version (bug fix release) coming. By the way, you can vote for it to become the Sourceforge project of the month.

16th June 2013

I made mirrors of my projects on GitHub. Not all of them of course, do not want people to look at the source of the programs written during junior high school ;) Kind of strange to have a Mercurial repository on github through, and sad to surrender it all to a newcomer (this page is far older than github), but it is so popular nowadays...

Also, we released a new TeXstudio version (2.6) this week.

16th May 2013

Seems like I forgot to announce that I added JSONiq support to the Internet Tools some months ago. It can be used to process JSON-data in the templates, so you can access JSON and XML-APIs without having the application care about the type of the API.
There is also a new backend for Android to download/upload stuff there, using the Apache HttpComponents through JNI. It works fine on arm-Android and x86-Android. Regarding other platforms: someone confirmed to use it successfully on MacOSX with the newest Synapse version, and someone else wanted to use it on arm-WinCE and after fixing some issues (missing extended float, widechar as default), it compiles there as well. (but I did not receive feedback if he could actually run it).
I also added a function to the TTreeListView to place the record/columns items at arbitrary positions, which can be used to create a list which looks and scrolls like the standard Android listviews. It seems to run (very slowly through) on the emulator, but all the Standard components of Lazarus are so buggy there that I gave up on using Lazarus for Android. So I did not test further if the treelistview is actually usable on Android.

13th December 2012

I just found a version of my mathSys on an old usb stick (almost 7 years old, holy shit), and remembered that I wanted to put it back on my webpage, since it is still the best free, open-source (well, open-source for me back then. But now for all) plotter for families of curves of parametrized (or multivariable) functions I have ever seen. But some years ago googling showed that someone else had used that mathSys name before, and I wanted to avoid some stupid name conflicts, so I took mathSys down. (and went on to create texmakerx instead. Very clever). Anyways, mathSys is now called BBMultiPlot.
I updated the source to the new name, but cannot compile an updated binary, since I no longer use Delphi/Windows and am not even sure, if I found the correct version of the source. I tried to change the name of the binary in a hex editor, but that failed, since I had compressed it with upx to fit in the 30 mb website quota. An obsolete version of upx (0.8?) that the upx tool itself cannot decompress. And dumping it in ollydbg or some petools just produces a larger binary (with the entry point set to the jmp after popad) that crashes on startup. Does anyone have an idea?

6th November 2012

The Internet Tools support now XQuery, instead of only XPath 2!
And all the variant-classes in the API were replaced by ref-counted interfaces.

26th September 2012

Some updates:

  • Internet Tools: They do now (well, since a month) support css 3 selectors, standalone or within XPath expressions, and now have a new XPath function form that converts an html form into an object describing the corresponding http request. The internet access class lets you read/write the http header, and the html tree parser will choose the correct encoding if the Content-Type header and the http-equiv meta tag specify different encodings.
    I also forked the multipage query scripts/templates from VideLibri in a new class that is now part of the internettools, and made a command line tool Xidel that lets you use all the internet functionality of VideLibri/Internet Tools without requiring some stupid libraries (the book kind).
    And I wrote a GreaseMonkey script that can automatically generate the scraping templates (or most of them) for VideLibri/Xidel by selecting the values on the webpage in FireFox.
  • TCommandLineReader: my command line reader does now work on Linux. (it did not work before, because the rtl function I used to read the cmdline does not support single quotes on Linux! Quite ironically, since I wrote the command line reader, because the other parameter-reading rtl function does not support single quotes on Windows!)
    I also extended it to read the arguments from GET/POST-requests, instead of only the command line.
  • TeXstudio: many, many new features...

3rd May 2012

Another update to the internet tools parser. It is now almost standard compatible, up to mostly rounding issues and missing namespace/dtd/schema support and my extensions like variable assignments and (new) some kind of objects. And the template language itself allows you now to abbreviate common commands, e.g. <template:read/> can be written as {...} or <template:loop/> as *.

10th July 2011

I added a few constructs to the internet tools parsers, so that the pxp parser is now a almost standard compliant implementation of XPath 2. (did you know that

every $x in satisfies satisfies satisfies
is always true?)

25th March 2011

I added a rainbow color mode to the sirdslets game, although I didn't really want to include it because it makes the game to easy to play (i.e. actually playable) in the non sirdslets modes. But it looks nice.

26th February 2011

Since everything is now on SF with public commit logs, I didn't write any news here, but these are the changes of the last months:

  • regexgen: a new little program that is the inverse of an usual regex engine and generates all matches to a given regex. (you could use it as a replacement of John's non-standard rules if you known a certain structure :-)
  • bbutils: some bugs fixed, faster, and much more array functions. Latter are all trivial, but tedious to write for all cases, so the complete unit is now auto-generated from a Pascal template.
  • Internet Tools: Some interfaces are changed and some low-level html parsing bugs are fixed. PXPath: The $..;-variables are now part of the PXPath-syntax, so they can be replaced in the evaluation step and don't have to be written in quotes anymore (actually must not). And it is more similar to real XPath with support for all selectors and many more functions.
  • TreeListView: It doesn't set the scrollbar size to a negative value anymore if the view size is 0. (because stupid Lazarus refuses to open a form with negative sizes instead of clipping the value)
  • SIRDSlets: New backgrounds. (I think the image has to minimize the autocorrelation of the brightness of middle large points to generate the best 3d effect.)

11th October 2010

(not translated)

Huch, was ist denn mit VideLibri los? In den vier Jahren, die es auf dieser Seite lag, wurde es praktisch kein einziges Mal heruntergeladen, nun ist es drei Tage auf SourceForge und wurde schon 778 mal runtergeladen. Weitaus öfter als TexMakerX in der gleichen Zeit, obwohl es keine fremden Links darauf gibt, außer mehreren aus Japan (Und wen interessiert es in Japan?). Am Quellcode kann es nicht liegen, der wurde nämlich nicht heruntergeladen.

9th October 2010

(not translated)

VideLibri habe ich jetzt auf SourceForge verschoben; hier findet es ja niemand und mittlerweile ist es so groß geworden, dass die Webseite nicht genug Speicher dafür hat.

24th September 2010

I tried to write an automatic sourceforge bugtracker/forum responder, which should be trivial with my internet tools library, but it was surprisingly complicated. First, sourceforge does not seem to understand host names in the format hostname:port, which is the default format of synapse. Then sf responds with a lot of redirections which neither my library nor synapse followed. Sf also uses the wrong (not standard compliant, although widely used) http status code for these redirections. And finally, sf has banned the user agent "Mozilla 3.0 (compatible)" which was my default agent. It only works with "Mozilla/3.0 (compatible)". And everything is encrypted with https, so you could hardly debug it with Wireshark.
In the new update, I changed that, and you can now compile it on Windows, too.

9th September 2010

(not translated)

Heute habe ich noch eine Seminararbeit über Bildsegmentierung vom letzten Semester hochgeladen. Die Bilder musste ich allerdings stark komprimieren, damit der Text noch so gerade auf die Seite passt (obwohl sie sowieso schon 10 mb größer ist als erlaubt), eigentlich sehen sie also schöner aus.

5th September 2010

I broke the &szlig; handling in the latest bbutils/internettools update, now it is fixed and even supports all entities, including those of MathML.

29th August 2010

I made a new version of my template based html parser. It is now probably thousands times slower and needs a magnitude more memory.
The old version used an NFA combined with a very clever heuristic, to match the template to the html page directly with O(1) memory and O(n) time, the new version uses simple backtracking on a DOM-like tree -- like everyone else does it -- with O(n) memory and I-don't-know-how-much time (perhaps O(k^n)?).
But it still runs in a few milliseconds, it is easier to understand the templates, and most importantly, the matcher/parser can now prove that a matching is possible or not. I also split it into three separate classes which makes it easier to test.
Furthermore, I had to change with the parser the bbutils (functions renamed or removed), the TreeListView, and the search bar (A method removed to be LCL 27190 compatible).

26th August 2010

When I wanted to display the article below as "untranslated" on the German version, I noticed that JAXMLP did not support parentheses in if-conditions (perhaps because it would increase the implementation difficulty from trivial to easy). Anyways, I changed it and you can now use brackets there.

26th August 2010

I just started to rewrite my fpc html template parser (which of course involves hours of debugging and patching Lazarus, before I can start to program anything else) and found two bugs due to an invalid string->pchar conversion. And although this is an absolutely trivial conversion, no one seems to have examined and described, how to do it best. So, here is a detailed analysis:

Converting ansistring to pchar

Converting a ansistring to pchar should be pretty simple because the text itself is stored in almost the same way in both cases. If you have a variable str: string, there are three popular ways to convert it to a pchar: pchar(str), @str[1] and pchar(pointer(str)), but it is not obvious which is the best way to do the conversion.

If you don't want to bother with the technical details, you can always use pchar(str) and treat the result as a read-only pchar. But this is not the most efficient way to convert the string, and the following table summarizes the advantages (always green) and disadvantages (always red) of the other methods:

ExampleNo copyBranch freeNil awareRange safe$H- safe
Explicit p := pchar(str) yes no yes yes yes
Array like p := @str[1] yes without $Rno no no
Indirect p := pchar(pointer(str)) yesyes no yes yes

The meaning of the columns is:

  • No copy: None of the methods above copies the content of the string.
  • Branch free: yes, iff the conversion works without branching instructions (hidden ifs). Branching is always very slow and even if modern computers are fast enough that it usually does not matter, you sometimes have/want to make something as fast as possible.
  • Nil aware: yes, if it works if str=nil.
  • Range safe: no, iff the conversion can fail if you enable range checking with the $R+ or -Cr option.
    @str[1] will raise a range checking error if the string is empty because s[1] doesn't exist then.
  • $H- safe: no, if it also compiles with shortstrings. This section is only about the conversion of ansistrings to pchars and none of the methods supports shortstrings. However, sometimes you forget to enable the $H+ option and every string becomes a shortstring. If you then use @str[1], it can randomly crash, because shortstrings are not null-terminated. The other two methods are safe, because they will not compile.

So you should never use @str[1] to convert it to a pchar, and pchar(pointer(str)) only if you understand what nil aware means.
It is said that every ansistring is null-terminated, but this is not exactly true. There is one (and only one!) case in which an ansistring str is not null-terminated: If str is empty (str = ''). Then str is nil.
pchar(str) will detect this case and returns a pointer to a global #0 character, but pchar(pointer(str)) will return nil. So you can always use pchar(str), and pchar(pointer(str)) only if your string is not empty and you can prove this.
(and as a remark: if you ever get a string which is not '' and not null-terminated, you probably forget to enable $H+ or - less probably - something wrote to a wrong pointer. )

So why should you not always use pchar(str)?
Because it is much slower than the two other methods, since pchar(str) checks for the nil-case, and this is often unnecessary.

Some people also say pchar(str) calls UniqueString to reset the reference count of the string, but this is wrong. pchar(str) does not call any function.

To see how worse the check is, you have to look at the assembly of the program (generated with fpc2.4 on amd64, not affected by optimization levels (which is strange because the assignment to rax was duplicated)):


1    mov    rax,QWORD PTR string variable
2    test   rax,rax
3    jne    to 5
4    mov    rax, global #0 address
5    mov    QWORD PTR pchar variable,rax
p := pchar(pointer(str)) or p := @str[1] with $R-:
1    mov    rax,QWORD PTR string variable
2    mov    QWORD PTR pchar variable,rax
So the check requires three additional instructions, and one is a jump instruction which will be executed iff the string is not empty.

Benchmarking (100000000 * 8 times repeated) results in:
non-empty stringempty string
pchar(str)894 ms496 ms
pchar(pointer(str))325 ms325 ms, but wrong result
@str[1]325 ms4616 ms325 ms, but wrong resultcrash

This shows pchar(pointer(str)) is more than twice as fast as pchar(str) in the more common use case of non-empty strings.

And for completeness, also the $R+ @s[1] assembly:

1    mov    rbx, string variable
2    mov    rdi,rbx
3    call   0x421668 <fpc_ansistr_checkzero>
4    mov    rsi,0x1
5    mov    rdi, string variable
6    call   0x421680 <fpc_ansistr_checkrange>
7    mov    pchar variable,rbx
In all cases is possible to modify the resulting pchar, but if you do that you will modify all copies of the string (unless you call UniqueString before):
var s,t:string;
So only modify the pchar of an converted string if you created that string or have modified it before with string functions (so reference count = 1).

20th August 2010

I just noticed that Andrey V. Sorokin's site does not exist anymore. His class TRegExpr was the best regular expression parser for Delphi and FPC. Because you need this class to use my internet tools, I created a mirror of TRegExpr.

10th August 2010

A Sirdslets update is there: 3 new levels, translations, accelerators, and anaglyph rendering. The accelerators are like black holes, but independent of the ship position. The anaglyph rendering creates red/cyan images which you can see with 3d glasses. But they are a little bit confusing because most 3d glasses have a very bad channel separation and you will see everything three times.
And if you are interested in theoretical computer science and did not get the news: Deolalikar claims to have found proof for P!=NP using a combination of graphical models, FO(LFP) logic, and theoretical physics. If he is right, the third of the most famous problems is solved in just fifteen years (after Fermat and Poincaré), perhaps Riemann is going to be the next one.

28th May 2010

Today I uploaded my new SIRDSlets game which is one of the few games that you can play/see in real 3d without additional 3d glasses. It is also the first online game on this page, so you can play it without downloading it explicitily. However, although it is playable, it is not completely finished, but it is in that state for months and I don't have much time in the moment. Some things that are missing are additional levels and the opengl renderer from the last news, so it only has six levels that are software-rendered.

16th February 2010

There is a new update of my Internettools: It fixes two small bugs that prevented the receiving of cookies without attributes and the creation of http connections over the 8080 port. It also adds a global logging which is especially useful for debugging https. In the parser class, the pseudo-XPath-expressions can now extract regex-matching data during the evaluation of the expression instead only afterward like before. (real XPath-expressions does not support it at all)
Furthermore, some news has been moved in the archive and some uncompleted tools to the demo page.

7th February 2010

There is now a new demo page listing small programs which are useless but demonstrate interesting techniques or show nice films.
Aside from some recategorizated old programs, there are two new ones on this page, both based on lecture exercises and rendering real 3d. The first draws hardware-accelerated Single Image (Random Dot) Stereograms. SIRDS are flat pictures containing a 3d scene which you can only see by squinting. The other "Webcam-UCP" program creates traditional red/cyan anaglyphs. Furthermore, it tracks the position of the 3d glasses via a webcam and renders the scene in user centered-projection.

27th January 2010

(not translated)

Von VideLibri gibt es nun eine neue Version, die auch die Hochschulbibliothekk der RWTH Aachen unterstützt und vollständig Linux-kompatible ist, inklusive Menüeinträge und integrierte Autostartkonfiguration.
Nun die schlechten Nachrichten: Bei der BTH Aachen gibt es noch keine integrierte Verlängerung, da man nicht gleich nach einer Ausleihe verlängern darf, und ich somit weiß nicht feststellen kann, wie es funktioniert. Außerden haben alle Libero-Büchereien mal wieder eine neue Software installiert. Das heißt:
Sowohl die Aachener Stadtbibliothek wie auch die FHB Düsseldorf gehen vermutlich nicht mehr. Testen kann ich es nicht, weil ich momentan nicht in die Bibliothek kann. (und genau DAS ist der Grund, warum mal endlich jemand die Software benutzen sollte. Wenn mich jemand darauf hinweist und die neue Seite per Mail schickt, kann ich solche Fehler nämlich in einer Viertelstunde beseitigen, statt in zwei Wochen!)
Zumindest in der alten Windowsversion funktionieren die beiden definitiv nicht (Stb Aachen war da nie drin und die FHB ist umgezogen). Sollten sie in der Linuxversion funktionieren, kann man aber einfach die Dateien der Linuxversion in das Installationverzeichnis in Windows kopieren, dann läuft es dort genauso gut/schlecht.
Zudem habe ich die FHB Bochum gestrichen, da sie nicht mehr unter ihrer alten Internetadresse zu finden ist, ich die neue nirgends finden kann und sie sowieso nur im Programm war, weil sie einen gemeinsamen Server mit der FHB Düsseldorf hatte.

6th January 2010

There are some minor changes in all the packets below, mostly I forgot to disable the unittests (and you probably don't want to have test log results everywhere). And the TreeListView won't flicker anymore on gtk when you scroll horizontally (I'm pretty sure that it also didn't flicker when I uploaded the last version...) and icons in the item tree will be clipped if the column is too short.

31st December 2009

And more updates: I merged the three packages with the template based html parser, the auto update and my Wininetwrapper into an Internet Tools-Package and made them platform-independent. (the main change is that the wininet wrapper is now also a Synapse wrapper). This new package requires/contains now my bbutils in which I fixed a crash on 64-Bit. (and probably renamed some functions since the last published version).
Furthermore, I uploaded the pasdoc documentation from every (changed so far) package as online documentation and every archive has now a common structure. The root level of the zip contains a single directory named like the zip itself, which contains the data files, the changelog from Mercurial and, if available, the pasdoc documentation for that archive.

28th December 2009

So now I switched from Windows 32 bit to Linux 64 bit and need to update all the programs here which were theoretically platform-independent, but apparently didn't really work there.
The TreeListView is now a lot faster in LCL-GTK, because it will cache subsequent input events and draw all changes at once (actually it is a reported lcl bug that there are so many unnecessary events, but joining them is better anyway), and in every item you can now store a 64 bit value instead of a 32-bit value like before. I also changed the included SearchBar to set its size dependent on the currently used font. For both these components, I have created a Delphi and a Lazarus package, so you can place them on the form during the design time. Furthermore, my JAXML parser can now create files in different encodings and \\-escaping works again (it seems I forgot to actually run the unit tests on the last release).

5th October 2009

(not translated)

Ich habe jetzt mal meine Bachelorarbeit hochgeladen. Im Wesentlichen geht es darum, wie man aus mehreren gegebenen, sortierten Listen von gleichen Ereignissen herausfinden kann, welches dieselben sind.

16th September 2009

This week I finished and presented my bachelor thesis, so that I have now enough time to update this home page.
Today I added a Javascript that you can use to perform a Gaussian Elimination on a given matrix. Of course, there are already hundreds of such scripts online, but all of those I found, didn't work for the equations I wanted to solve, because they assumed that the matrix is quadratic and contains elements of a field instead of a ring. And most of them don't let you choose the order in which the operations are performed or allow neither import nor export of data. The only problem with my script is that Javascript is always a little bit slow, but anyway you don't solve equations with a dozen variables step-by-step. And it seems that no one downloads anything in the web2.0 era.
An interesting fact I noticed while implementing this is that you can consider the Euclidean algorithm for finding the gcd as a special case of the Gaussian Elimination on a 2x1-matrix of integers.

15th April 2009

There is now a new program: Sun-Simulator which changes the brightness and shade of color of the monitor during the day and year, so that its light matches the sunlight.
Additionally, I wrote a diagram component for Lazarus which shows float points in multiple ways. It is based on a model/view system (because I use qt a lot at the moment) which can be used to create interesting effects very easily. E.g.: you can synchronize the data points across different windows and add some other points in one of them at the same time.

9th April 2009

A problem with writing open-source software is that you often receive patches which are completely useless because they consist of 95 percent of changed whitespace. Therefore I wrote the command line tool Simplifydiff that removes all unnecessary data.
And all entries in the guestbook which are spam according to the heuristic are now removed without a backup. So far I received a copy as mail, but since the last thousand entries were really spam, this seems to be useless. (Actually, the spam entries are also completely useless because the guestbook has set the flags noindex, nofollow since the beginning!)

1st March 2009

Some time ago I started to write texts and even a book in LaTeX. Therefore I searched for a platform-independent, open-source editor, and found only Texmaker.
But since he has some bugs, I created some small patches, which got bigger and bigger, until they finally lead to the fork TexMakerX (on Sourceforge). The most important features I have added are interactive spell checking, code folding, and a text analysis.
On this web page, I only updated the overview text below and corrected the date of the last news from 18.Okt to 27.Nov. (version control systems are really useful...) (correction: The date of the VCS was wrong, but my CMS got it right).

Older news...


On this page, you can download some games and tools I wrote.
There are also useful sources and components, mainly for Delphi and Free Pascal.

Some information about me: I have programmed since I was twelve (that's why there are so many old downloads here), have studied computer science at the Heinrich-Heine-Universität Düsseldorf und RWTH Aachen und am now working on a PhD at the University of Lübeck.
I have won several prizes in the German national informatics competition and participated in different European and International Olympiads of Informatics. In the European ones, I also won some medals.

Here is a guestbook where you can tell me if you like or dislike something on this page. If you have any questions or suggestions you can also write me a mail.
mousing eating cheese made at the public libraries of Düsseldorf