dearwolf/aider
1
0
Fork 0
mirror of https://github.com/Aider-AI/aider.git synced 2025-06-01 02:05:00 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 9

feat: Add code-in-json-benchmark.js file and update code-in-json.md post

Browse source
This commit is contained in:
Paul Gauthier 2024-08-15 10:50:51 -07:00 • committed by Paul Gauthier (aider)
parent 8d4d549a98
commit e90642295d
2 changed files with 7 additions and 9 deletions
Download patch file Download diff file Expand all files Collapse all files

0
aider/website/_includes/code-in-json-benchmark.js Normal file
View file

16
aider/website/_posts/2024-08-14-code-in-json.md
View file

@ -157,21 +157,19 @@ function createStripedCanvas(isStrict) {
## Abstract ## Abstract
The newest LLMs have support for returning properly formatted JSON responses, Current LLMs have support for returning properly formatted JSON,
making it easier for clients to parse complex responses. making it easier for clients to reliably parse complex responses.
This makes it tempting for AI coding applications to It therefore seems attractive for
use JSON replies to AI coding applications ask LLMs to return code in structure JSON replies.
receive code from LLMs.
Unfortunately, Unfortunately,
LLMs write worse code when asked to wrap it in JSON, harming their ability LLMs write worse code when asked to wrap it in JSON, harming their ability
to correctly solve coding tasks. to correctly solve coding tasks.
On a variant of the aider code editing benchmark, On a variant of the aider code editing benchmark,
JSON-wrapping code asking for JSON-wrapped code
often significantly harms coding often significantly harms coding
performance performance.
compared to returning code as plain text.
This holds true across many top coding LLMs, This holds true across many top coding LLMs,
including OpenAI's new gpt-4o-2024-08-06 including OpenAI's latest model gpt-4o-2024-08-06
which has strong JSON support. which has strong JSON support.
## Introduction ## Introduction