research-evaluation

research-evaluation — evaluate before adopting

Turn "should we use ?" into a consistent, evidence-backed evaluation instead of an ad-hoc impression. Works for one target or a batch; generalizes to any repo.

The rubric (all nine, every target)

Weighting (tunable, not hard-coded)

Each dimension carries a weight read at eval time from gff feature flags under research-rubric.* (tiers none·low·medium·high·critical = 0·1·2·3·4; public defaults skew to value + security + adversarial + borrowable, with business modest). Read them with gff get research-rubric.weight.<dim> / gff list, and state the active weighting in the output so a reader sees the lens behind the verdict. Weightings are meant to be overridden per person/team via the gff user layer — see references/tuning.md. Two gate flags: research-rubric.require.adversarial and research-rubric.require.docker-demo.

Explaining terms (clickable, not wordier)

Link jargon to references/glossary.md on its first use in a doc — e.g. [bus factor](./references/glossary.md#bus-factor). A link adds no prose; it just makes the output learnable. Glossary covers bus factor, blast radius, supply-chain surface, fail-open, prompt cache, CCR, prompt injection, SSRF, MCP, CalVer, CVE, SBOM.

Procedure

1 — Pin the target list and the destination

• Collect the targets from the user (names may be fuzzy — "omni route", "headroom").
• Detect the repo's research home, in order: docs/mbo/ (use its templates/research.md if present, else the design template) → any project-local MBO dir → fall back to docs/research/ (create it). One slug per target.
• Locate before create at every step: existing design doc, existing issue (gh issue list --search "<slug> in:title"), existing index row. Re-runs must reconcile, not duplicate.

2 — Research (fan out for >1 target)

For each target, run a web-enabled research agent (parallel, background, one per target) that must:

1. Identify the canonical project: GitHub first, then docs/blogs. Try name variants; rule out name collisions explicitly. If no confident identification after honest searching → verdict NOT_FOUND, listing the searches tried and closest candidates — this is a valid terminal outcome, not a failure.
2. Gather evidence for all nine dimensions. Ground (f) in the repo API (stars, contributors, last push, open issues) and state the observation date. Mine the issue tracker for (c)/(d)/(e) — open bugs are the adversarial goldmine. For (h), diff the tool's capabilities against what we already run and ask which valuable gaps are cheaply buildable in-house. For (i), tag the ROI tiers. Read the active research-rubric.* weights via gff and weight the dimensions accordingly; link jargon to the glossary on first use.
3. Draft the (g) demo as concrete commands (compose file / docker run), sandboxed: throwaway dirs, localhost-only ports, never real credentials or OAuth tokens, full teardown. If docker genuinely can't work, write "no docker demo — skip".
4. Write the full dossier to a scratch file; return only a ≤15-line executive summary (verdict, URL, license, one-line value, top risk, demo feasible).

Context the agents need to judge (a): a short paragraph on our environment and what we already run (memory systems, orchestration, infra) — pass it in the prompt.

3 — Produce the artifacts

Per FOUND target: condense the dossier into the research design doc (designs/<slug>.md from the research template) ending in a Verdict: adopt / adopt selectively / park (gated on demo) / reject / reject-but-steal-the-pattern / reject-but-build-the-feature — one paragraph, grounded in the sections (including the (h) build-vs-adopt call). Per NOT_FOUND target: a stub doc recording the searches + candidates, state not-found.

4 — Track

• Create/update one issue per target (title <slug>: research evaluation — <name>, body = one-liner + design-doc path + verdict) and put the number in the doc.
• Register every target in the index (docs/mbo/index.md or the fallback's README): slug, doc link, issue, state (evaluated — adopt/reject, parked, not-found).
• Land the docs via the repo's normal review flow (here: a gss draft PR — confirm before push per gss rules).

5 — Demos (optional follow-up, on request or per verdict)

Run the (g) plans for targets whose verdict is gated on the demo (park) or where first-hand validation was requested. Capture evidence per the repo's show-and-tell convention (e.g. demos/<date>-<slug>/). Never run an installer on a real host before its container run.

Hard rules

• Docker-or-skip for demos — no "just try it on the host".
• Adversarial section is mandatory — an evaluation with no case-against is marketing, not research.
• Always run the (h) build-vs-adopt check — never conclude "adopt" without first asking whether the valuable features could be built into our own stack more simply.
• Dated observations — every quality/maintenance number carries the date.
• NOT_FOUND stops that target — record it and move on; don't force a match onto a name-collision.
• Never route real subscription/OAuth credentials into a tool under evaluation.