Chainle: A Practical Guide to Testing Semantic Similarity

Word embeddings can make “light” a plausible bridge between “sun” and “lamp.” They can also miss the joke, the local reference, or the second meaning that made your answer feel obvious.

That makes Chainle a useful lens for a much larger engineering question: when does semantic similarity behave like understanding, and how would we know when it does not? The Chainle game turns the question into a compact word connection game with visible inputs, outputs, and disagreements to inspect. It is also a concrete word association game: the player proposes a relationship, and the scorer exposes how a fixed representation treats it.

Chainle turns that question into five short decisions. A player enters one word to connect two or three clues, or a close meaning for a single clue. Under the hood, the version at Chainle uses fixed semantic data and a scoring rule; it does not ask a generative model to judge each submitted answer. That distinction makes it a useful case study for evaluating representations without treating an LLM's fluent explanation as ground truth.

What a word vector can tell you

A word embedding maps a term to a point in a numerical space. Words with related usage or concepts may occupy nearby regions, so a system can compare vectors with a measure such as cosine similarity. ConceptNet Numberbatch describes its term vectors as 300 floating-point values per entry and combines multiple sources of semantic information. Its project documentation is a useful place to inspect the representation and its limits.

The word may matters. A vector is a compact signal learned from data, not a dictionary definition, a proof, or a human explanation. A single vector for a word also has difficulty separating senses. “Light” can mean illumination, low weight, or a pale color. Contextual systems can represent a word differently in different sentences; a fixed word vector cannot make that distinction in the same way.

This is why a word connection game is more interesting than a nearest-neighbor demo. The task is not merely “find words similar to sun.” It asks whether one candidate relates reasonably well to every clue. An answer that is excellent for one clue and weak for another should not look like a balanced bridge. That makes a Chainle game a compact probe of aggregation as well as similarity, while keeping each input small enough to review.

Chainle as a concrete scoring example

In the current Chainle implementation, each clue is compared separately with a candidate word using fixed semantic data. The result is scaled against a reference connection for that clue. For a two-clue round, the two per-clue strengths are combined with a harmonic mean. In plain language, a weak side pulls the total down; a strong association with only one clue cannot carry the whole answer.

The three-clue round uses a related calculation with a limited adjustment to the weakest link. The interface rounds each visible connection into up to eight links, while the score uses more precise values. Two guesses can therefore show similar chains and still receive different points. The five rounds of this Chainle game offer up to 1,000 points each, but the number is a property of this version's scoring data and rules, not a universal measure of how meaningful a word is.

For example, sun + lamp → light is an easy teaching illustration: illumination links both clues. It is not a claim about a live puzzle or a measured score. Needle + spider → thread shows a different relation: one object uses thread, while the other produces it. A useful evaluator should be able to distinguish those paths from a candidate that only fits one side. The examples also show why a word association game needs a declared rubric: “related” can mean several different things. In a word connection game, make that rubric explicit before comparing vector similarity with human judgments.

You can inspect the actual rules and try the Chainle game yourself. Treat the resulting score as the system's estimate. A low score might mean your candidate is weak, that its intended sense is rare, or that the representation does not capture the connection you have in mind.

Illustrative example: one candidate word can connect two clues through a shared meaning.

A small evaluation you can reproduce

The most useful test is not a handful of impressive examples. Build a small, labeled set that contains both ordinary bridges and cases likely to expose failure modes.

1. Write the task down. Define what counts as a valid connection: a shared category, function, property, relation, or close meaning. Decide whether indirect associations count. Keep that definition stable during evaluation.
2. Create clue sets before choosing candidates. Include two-clue and three-clue cases, ambiguous words, multiple senses, less common meanings, and examples from different domains. Keep a held-out set of clue combinations that does not appear during tuning.
3. Collect more than one candidate per set. Include a clear bridge, a plausible but uneven bridge, a one-sided association, and an unrelated control. Randomize candidate order and hide model scores from human reviewers.
4. Ask reviewers to rate each clue separately. A simple scale such as 0 (no defensible link), 1 (indirect or strained), and 2 (clear link) makes it easier to see whether a candidate fails on one side. Ask for a short rationale and record disagreement instead of forcing a false consensus.
5. Compare systems on the same cases. Record human ratings, vector scores, and—if useful—an LLM judge's ratings in separate columns. An LLM can provide a useful comparison, but it is another system under test, not the source of truth.
6. Report errors, not just averages. Check ranking agreement, whether the best candidate appears near the top, and how often one weak clue is hidden by a strong one. Review false positives and false negatives by category.

Keep related clue families together when making train/test splits. Otherwise, near-duplicate examples can make a system look more general than it is. Publish the test definition, representation version, candidate-generation method, and scoring rule alongside any result. Without those details, a score is difficult to reproduce or interpret.

A simple interface makes the evaluation task easy to observe; this illustration is not a live test result.

Use this prompt as an annotation assistant

An LLM can help organize human review if you keep its role narrow. Give it one candidate at a time, use the same rubric for every case, and preserve a human decision column. This prompt can help review a Chainle game or another word association game; its output remains an annotation suggestion, not a label. The same prompt works for a small word connection game benchmark if every clue is still rated independently.

You are assisting with a word-connection evaluation. Do not decide whether an
answer is officially correct. Evaluate only the semantic relationship stated
in the rubric.

Rubric: A connection may be a shared category, function, property, relation,
or close meaning. Judge each clue independently. Do not let a strong connection
to one clue compensate for no plausible connection to another.

Clues: [clue A] | [clue B]
Candidate: [candidate word]

Return valid JSON only:
{
  "links": [
    {"clue": "[clue A]", "rating": 0, "relation": "", "rationale": ""},
    {"clue": "[clue B]", "rating": 0, "relation": "", "rationale": ""}
  ],
  "ambiguity": "",
  "needs_human_review": true
}

Ratings: 0 = no clear relationship, 1 = indirect or context-dependent, 2 =
clear relationship. If a clue has multiple common senses, name the sense used.
Mark needs_human_review true when the relationship depends on a rare, regional,
cultural, or disputed interpretation. Do not invent a source or a dictionary
definition.

Run the prompt on a sample, then have reviewers check whether its rationales are faithful and its ratings consistent. If the prompt changes halfway through, version it and rerun the comparison. Otherwise, you are comparing different annotation procedures as if they were one.

Where semantic scoring breaks

Polysemy: A single vector can blur distinct senses. A candidate may be close to one meaning of a clue while the player intended another.

Asymmetric fit: Two words can be close in one direction of the task but not equally useful as a bridge across all clues. Always inspect per-clue scores before aggregating them.

Cultural and domain gaps: A phrase, local reference, specialist term, or recent usage may be meaningful to people but sparse in the training sources.

Vocabulary coverage: A valid word can be missing from the allowed guess list. Separate input validation failures from semantic scoring failures in logs and evaluation reports. This distinction matters in any Chainle game because a rejected word never reaches the semantic scoring step.

Metric mismatch: Cosine similarity measures geometry in a particular representation. It is not automatically the same thing as human-perceived association, synonymy, or usefulness for a particular task.

These issues are not unique to Chainle or a word connection game. They appear anywhere a product turns a broad human concept into a numerical score: semantic search, recommendation, deduplication, and content moderation all depend on task-specific definitions and evidence. Thinking through a word association game can make those assumptions easier to see because each candidate and clue is visible.

What to take from the experiment

A good semantic evaluation has a clear rubric, held-out examples, independent human judgments, and visible failure cases. A leaderboard number alone cannot tell you whether your system found a genuine bridge or exploited an easy association. The Chainle game provides a small, legible example of the distinction between a similarity score and a human interpretation.

Chainle makes that trade-off tangible in a few minutes: read every clue, submit one word, and inspect how the per-clue connections shaped the result. Use the word association game as an interactive example, then apply the same evaluation discipline to a model you build: define what “related” means, test each component, and keep uncertainty visible.