dearwolf/aider
1
0
Fork 0
mirror of https://github.com/Aider-AI/aider.git synced 2025-06-01 18:25:00 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 5

copy

Browse source
This commit is contained in:
Paul Gauthier 2024-07-01 16:16:19 -03:00
parent 7396e3883e
commit 7c746fe901
2 changed files with 37 additions and 26 deletions
Download patch file Download diff file Expand all files Collapse all files

30
website/_data/refactor_leaderboard.yml
View file

@ -143,25 +143,25 @@
seconds_per_case: 67.8 seconds_per_case: 67.8
total_cost: 20.4889 total_cost: 20.4889
- dirname: 2024-07-01-18-30-33--refac-claude-3.5-sonnet-diff-not-lazy
- dirname: 2024-06-20-16-39-18--refac-claude-3.5-sonnet-diff
test_cases: 89 test_cases: 89
model: claude-3.5-sonnet (diff) model: claude-3.5-sonnet (diff)
edit_format: diff edit_format: diff
commit_hash: e5e07f9 commit_hash: 7396e38-dirty
pass_rate_1: 55.1 pass_rate_1: 64.0
percent_cases_well_formed: 70.8 percent_cases_well_formed: 76.4
error_outputs: 240 error_outputs: 176
num_malformed_responses: 54 num_malformed_responses: 39
num_with_malformed_responses: 26 num_with_malformed_responses: 21
user_asks: 10 user_asks: 11
lazy_comments: 2 lazy_comments: 2
syntax_errors: 0 syntax_errors: 4
indentation_errors: 3 indentation_errors: 0
exhausted_context_windows: 0 exhausted_context_windows: 0
test_timeouts: 0 test_timeouts: 0
command: aider --model openrouter/anthropic/claude-3.5-sonnet command: aider --model openrouter/anthropic/claude-3.5-sonnet
date: 2024-06-20 date: 2024-07-01
versions: 0.38.1-dev versions: 0.40.7-dev
seconds_per_case: 51.9 seconds_per_case: 42.8
total_cost: 0.0000 total_cost: 11.5242

33
website/_posts/2024-07-01-sonnet-not-lazy.md
View file

@ -1,26 +1,34 @@
--- ---
title: Sonnet is the opposite of lazy title: Sonnet is the opposite of lazy
excerpt: Claude 3.5 Sonnet represents a step change in AI coding. excerpt: Claude 3.5 Sonnet represents a step change in AI coding.
#highlight_image: /assets/linting.jpg highlight_image: /assets/sonnet-not-lazy.jpg
draft: true
nav_exclude: true nav_exclude: true
--- ---
[![sonnet is the opposite of lazy](/assets/sonnet-not-lazy.jpg)](https://aider.chat/assets/sonnet-not-lazy.jpg)
{% if page.date %} {% if page.date %}
<p class="post-date">{{ page.date | date: "%B %d, %Y" }}</p> <p class="post-date">{{ page.date | date: "%B %d, %Y" }}</p>
{% endif %} {% endif %}
# Sonnet is the opposite of lazy # Sonnet is the opposite of lazy
[![sonnet is the opposite of lazy](/assets/sonnet-not-lazy.jpg)](https://aider.chat/assets/sonnet-not-lazy.jpg)
Claude 3.5 Sonnet represents a step change Claude 3.5 Sonnet represents a step change
in AI coding. in AI coding.
It is so industrious, diligent and hard working that It is so industrious, diligent and hard working that
it has caused multiple problems for aider. it has caused multiple problems for aider.
It's been worth the effort to adapt aider to work well It's been worth the effort to adapt aider to work well
with Sonnet, with Sonnet,
because the result is surprisingly powerful. because the result is surprisingly powerful.
Sonnet's score on
[aider's refactoring benchmark](https://aider.chat/docs/leaderboards/#code-refactoring-leaderboard)
jumped from 55.1% up to 64.0%
as a result of the changes discussed below.
This moved Sonnet into second place, ahead of GPT-4o and
behind only Opus.
## Problems
Sonnet's amazing work ethic caused a few problems: Sonnet's amazing work ethic caused a few problems:
@ -31,7 +39,7 @@ on API responses, which truncates its coding in mid-stream.
2. Similarly, Sonnet can specify large sequences of edits in one go, 2. Similarly, Sonnet can specify large sequences of edits in one go,
like changing a majority of lines while refactoring a large file. like changing a majority of lines while refactoring a large file.
Again, this regularly triggered the 4k output limit Again, this regularly triggered the 4k output limit
and resulted in a failed edits. and resulted in failed edits.
3. Sonnet is not shy about quoting large chunks of an 3. Sonnet is not shy about quoting large chunks of an
existing file to perform a SEARCH & REPLACE edit across existing file to perform a SEARCH & REPLACE edit across
a long span of lines. a long span of lines.
@ -57,7 +65,7 @@ Problem (3) does cause some real downsides.
Faced with a few small changes spread far apart in Faced with a few small changes spread far apart in
a source file, a source file,
Sonnet would often prefer to do one giant SEARCH/REPLACE Sonnet would often prefer to do one giant SEARCH/REPLACE
operation of the ~entire file. operation of almost the entire file.
This wastes a tremendous amount of tokens, This wastes a tremendous amount of tokens,
time and money -- and risks hitting the 4k output limit. time and money -- and risks hitting the 4k output limit.
It would be far faster and less expensive to instead It would be far faster and less expensive to instead
@ -76,13 +84,16 @@ has specialized support for Claude 3.5 Sonnet:
- Aider allows Sonnet to produce as much code as it wants, - Aider allows Sonnet to produce as much code as it wants,
by automatically and seamlessly spreading the response by automatically and seamlessly spreading the response
out over a sequence of 4k token API responses. out over a sequence of 4k token API responses.
- Aider carefully prompts Sonnet to be concise and - Aider carefully prompts Sonnet to be concise when proposing
return only changing sections of code. code edits.
This reduces Sonnet's tendency to waste time, tokens and money This reduces Sonnet's tendency to waste time, tokens and money
returning large chunks of unchanging code. returning large chunks of unchanging code.
- Aider now uses `claude-3-5-sonnet-20240620` by default if `ANTHROPIC_API_KEY` is set in the environment. - Aider now uses Claude 3.5 Sonnet by default if the `ANTHROPIC_API_KEY` is set in the environment.
You can use aider with Sonnet like this: See
[aider's install instructions](https://aider.chat/docs/install.html)
for more details, but
you can get started quickly with aider and Sonnet like this:
``` ```
pip install aider-chat pip install aider-chat