From e90642295dc5f7d52043c1ffdc8bb54fb504c3c2 Mon Sep 17 00:00:00 2001 From: Paul Gauthier Date: Thu, 15 Aug 2024 10:50:51 -0700 Subject: [PATCH] feat: Add code-in-json-benchmark.js file and update code-in-json.md post --- .../website/_includes/code-in-json-benchmark.js | 0 aider/website/_posts/2024-08-14-code-in-json.md | 16 +++++++--------- 2 files changed, 7 insertions(+), 9 deletions(-) create mode 100644 aider/website/_includes/code-in-json-benchmark.js diff --git a/aider/website/_includes/code-in-json-benchmark.js b/aider/website/_includes/code-in-json-benchmark.js new file mode 100644 index 000000000..e69de29bb diff --git a/aider/website/_posts/2024-08-14-code-in-json.md b/aider/website/_posts/2024-08-14-code-in-json.md index 69553ad52..fe6a63466 100644 --- a/aider/website/_posts/2024-08-14-code-in-json.md +++ b/aider/website/_posts/2024-08-14-code-in-json.md @@ -157,21 +157,19 @@ function createStripedCanvas(isStrict) { ## Abstract -The newest LLMs have support for returning properly formatted JSON responses, -making it easier for clients to parse complex responses. -This makes it tempting for AI coding applications to -use JSON replies to -receive code from LLMs. +Current LLMs have support for returning properly formatted JSON, +making it easier for clients to reliably parse complex responses. +It therefore seems attractive for +AI coding applications ask LLMs to return code in structure JSON replies. Unfortunately, LLMs write worse code when asked to wrap it in JSON, harming their ability to correctly solve coding tasks. On a variant of the aider code editing benchmark, -JSON-wrapping code +asking for JSON-wrapped code often significantly harms coding -performance -compared to returning code as plain text. +performance. This holds true across many top coding LLMs, -including OpenAI's new gpt-4o-2024-08-06 +including OpenAI's latest model gpt-4o-2024-08-06 which has strong JSON support. ## Introduction