Aix: Efficiently storing and querying chess game collections

The December set of games contains around 95 million games. And your query iterating over all positions takes 92 seconds. So that's one million games per second then, right? That's very impressive!

I'm wondering where the name is coming from.

The December set of games contains around 95 million games. And your query iterating over all positions takes 92 seconds. So that's one million games per second then, right? That's very impressive! I'm wondering where the name is coming from.

ProgramFOX edited

So that's one million games per second then, right?

Yes, although parallelized over 24 threads, so more like 42k/second/thread.

I'm wondering where the name is coming from.

The genus name for the wood duck is Aix, which is fitting because chess pieces are wood and this is a DuckDB extension. And it's also inspired by how DuckDB codenames its versions with ducks' species names (which is different from genus name, so no collision is possible, even if they pick the same duck then the version name would be Sponsa).

> So that's one million games per second then, right? Yes, although parallelized over 24 threads, so more like 42k/second/thread. > I'm wondering where the name is coming from. The genus name for the wood duck is Aix, which is fitting because chess pieces are wood and this is a DuckDB extension. And it's also inspired by how DuckDB codenames its versions with ducks' species names (which is different from genus name, so no collision is possible, even if they pick the same duck then the version name would be Sponsa).

MillenniumBug

edited

Pretty mind-bogglingly interesting stuff!

Now, I have zero experience with the Lichess DBs. But,I have had an idea lingering at the back of my brain for years&years without writing a sinlgle keystroke of code. Lazy, yes. It is about searching through the lichess Master games DB, looking for certain patterns* in the comp analysis.

Now, my idea of how to search is rather vague. I could query
[pseudo] both above 2700 ELO AND has comp analysis AND classical time ctrl
Now I assume the first two are indexed, but I am not sure if time ctrl is stored/indexed in the DB. But I would be surprised if comp analysis details - beyond bool exists - are indexed. (And time ctrl is of subordinate importance.)

I always thought I would naively let it brew overnight, over that 2700 subset , while simple searching for certain patterns sequentially. But your post made my head to start spinning, and I wonder if Aix would be a good companion, for [at least some parts of] this little hobby endeavour...

I guess a first step would be to Aix-ify the entire Masters DB...

Appreciate, your thoughts here. THX.

*if I ever get around to my DB project, I will write a post here.

Pretty mind-bogglingly interesting stuff! Now, I have zero experience with the Lichess DBs. But,I have had an idea lingering at the back of my brain for years&years without writing a sinlgle keystroke of code. Lazy, yes. It is about searching through the lichess Master games DB, looking for _certain patterns_* in the comp analysis. Now, my idea of how to search is rather vague. I could query `[pseudo] both above 2700 ELO AND has comp analysis AND classical time ctrl` Now I assume the first two are indexed, but I am not sure if time ctrl is stored/indexed in the DB. But I would be surprised if comp analysis details - beyond `bool exists` - are indexed. (And time ctrl is of subordinate importance.) I always thought I would naively let it brew overnight, over that 2700 subset , while simple searching for _certain patterns_ sequentially. But your post made my head to start spinning, and I wonder if Aix would be a good companion, for [at least some parts of] this little hobby endeavour... I guess a first step would be to Aix-ify the entire Masters DB... Appreciate, your thoughts here. THX. *if I ever get around to my DB project, I will write a post here.

ProgramFOX

@MillenniumBug said:

I guess a first step would be to Aix-ify the entire Masters DB...

Indeed, but as far as I know, the Lichess Masters DB is not available for download, which makes this step impossible. If a PGN were available, https://github.com/thomas-daniels/aix/tree/main/pgn-to-aix would be a starting point for conversion (currently that tool is tailored to the Lichess database exports, specifically for how PGN tags are parsed, but I do plan on making it more generically useful out-of-the-box).

@MillenniumBug said: > I guess a first step would be to Aix-ify the entire Masters DB... Indeed, but as far as I know, the Lichess Masters DB is not available for download, which makes this step impossible. If a PGN were available, https://github.com/thomas-daniels/aix/tree/main/pgn-to-aix would be a starting point for conversion (currently that tool is tailored to the Lichess database exports, specifically for how PGN tags are parsed, but I do plan on making it more generically useful out-of-the-box).