From 8c9536bcdd95217050aedbb7d7bf566d907b1f4c Mon Sep 17 00:00:00 2001 From: Paul Gauthier Date: Sun, 17 Dec 2023 18:23:45 -0800 Subject: [PATCH] copy --- docs/unified-diffs.md | 15 ++++----------- 1 file changed, 4 insertions(+), 11 deletions(-) diff --git a/docs/unified-diffs.md b/docs/unified-diffs.md index 2f0a25604..c4ea92c83 100644 --- a/docs/unified-diffs.md +++ b/docs/unified-diffs.md @@ -27,9 +27,9 @@ This new laziness benchmark produced the following results with `gpt-4-1106-prev - **Aider's new unified diff edit format raised the score to 65%**. - **No benefit from the user being blind, without hands, tipping $2000 or fearing truncated code trauma.** These widely circulated folk remedies performed no better than baseline when added to the system prompt with aider's SEARCH/REPLACE edit format. Including *all* of them only scored at 15% -The older `gpt-4-0613` also did better on the laziness benchmark by using unified diffs. +The older `gpt-4-0613` also did better on the laziness benchmark using unified diffs. The benchmark was designed to work with large source code files, and -28% of them are too large to use with June GPT-4's 8k context window. +28% of them are too large to fit in June GPT-4's 8k context window. This significantly harmed the benchmark results. - **The June GPT-4's baseline was 26%** using aider's existing "SEARCH/REPLACE block" edit format. @@ -86,7 +86,6 @@ default output format of `git diff`: @@ -1,5 +1,5 @@ def main(args): # show a greeting - - print("Hello!") + print("Goodbye!") return @@ -149,7 +148,7 @@ numbers in editing formats, backed up by many quantitative benchmark experiments. You've probably ignored the line numbers in every diff you've seen? -So aider tells GPT not to include line numbers, +So aider tells GPT not to include them, and just interprets each hunk from the unified diffs as a search and replace operation: @@ -159,19 +158,17 @@ This diff: @@ ... @@ def main(args): # show a greeting - - print("Hello!") + print("Goodbye!") return ``` -Means we want to search the original source file for all the +Means we want to search the file for all the *space* ` ` and *minus* `-` lines from the hunk: ```python def main(args): # show a greeting - print("Hello!") return ``` @@ -181,7 +178,6 @@ And then replace them with all the *space* ` ` and *plus* `+` lines: ```python def main(args): # show a greeting - print("Goodbye!") return ``` @@ -267,7 +263,6 @@ import sys def main(args): # show a greeting - print("Hello!") return @@ -284,11 +279,9 @@ because of the missing comment. ```diff @@ ... @@ -def main(args): -- - print("Hello!") - return +def main(args): -+ + print("Goodbye!") + return ```