AI Skills Augmentation: Data Sample Dataset and Findings

Most enterprise AI deployments stall at the same point: the models are increasingly capable but don’t know your team’s taxonomy, your incident classification system, or your internal configuration vocabulary. They can read the logs. They can’t speak your language. Skill documents close that gap, and this dataset proves it at scale.

G2i built a curated 20-task data sample dataset for the SkillsBench evaluation program. Each task is a real-world engineering or DevOps scenario. An agent is paired with a skill document, works through the environment, and its output is scored against a test suite.

Validation ran four frontier models across all tasks, with and without skills loaded, producing 1,820 scored runs. The 20 submitted tasks were selected for reliable performance lift, cross-model consistency, and clean run histories (no flaky tests, no environment failures).

The 20-task portfolio covers five major domains:

Diversity was a deliberate design goal. Tasks vary across domain, the number of skills required (1 to 4+), environment complexity (minimal single-file setups to 16+ file environments with distractors), test count, and difficulty tier.

Tasks are graded using the official SkillsBench rubric (arXiv:2602.12670v1), which defines three tiers based on estimated human completion time for a median domain specialist working without AI assistance:

The data sample portfolio contains 19 medium tasks and 1 hard task. Easy tasks were excluded because models solve them without any skill augmentation. The skew toward medium over hard is also deliberate: tasks with zero baselines generate cleaner signal than hard tasks, where even the best models partially fail with skills loaded.

Skill documents produce large, consistent performance gains when the task follows the investigation pattern: the agent reads system state or logs, the skill teaches an enumerated vocabulary of failure modes, and the agent classifies its findings accordingly. This pattern accounts for the strongest results in the dataset.

Performance across all four models:

The portfolio is structured for immediate use in the SkillsBench data sample evaluation. Each task is self-contained and includes the scenario environment, the skill document, and the test suite. The grading and scoring methodology follows the SkillsBench standard, so results are directly comparable to other submissions in the program.

Beyond the data sample, the dataset illustrates a broader principle: the gap between what a model knows from training and what it needs to know to operate in a specific team’s environment is measurable and significant. The 20 tasks in this portfolio demonstrate that across security, DevOps, general software engineering, and infrastructure domains.