From a0025a42d962e8254960326c0dc6b63989a6d4d4 Mon Sep 17 00:00:00 2001 From: Paul Gauthier Date: Mon, 3 Jun 2024 11:23:12 -0700 Subject: [PATCH] copy --- _posts/2024-05-22-swe-bench-lite.md | 1 + 1 file changed, 1 insertion(+) diff --git a/_posts/2024-05-22-swe-bench-lite.md b/_posts/2024-05-22-swe-bench-lite.md index beec36388..8a553f296 100644 --- a/_posts/2024-05-22-swe-bench-lite.md +++ b/_posts/2024-05-22-swe-bench-lite.md @@ -414,6 +414,7 @@ The "aider agent" internally makes multiple "attempts" at solving the problem, but it picks and returns one single candidate solution. Only that one candidate solution is evaluated with the acceptance tests and contributes to the benchmark score. +Thus it is a pass@1 result. This is contrast to a pass@N result for N>1, where N attempts are made and all N solutions are evaluated by the acceptance tests.