89db1d8673
# Aider chat conversation:
USER: in each graph, give each model a second bar showing row.percent_cases_well_formed
ASSISTANT: Here are the changes to add a second bar for `row.percent_cases_well_formed` to each graph:
docs/leaderboards/index.md
```markdown
<<<<<<< SEARCH
var leaderboardData = {
labels: [],
datasets: [{
label: 'Percent correct on code editing tasks',
data: [],
backgroundColor: 'rgba(54, 162, 235, 0.2)',
borderColor: 'rgba(54, 162, 235, 1)',
borderWidth: 1
}]
};
{% for row in edit_sorted %}
leaderboardData.labels.push('{{ row.model }}');
leaderboardData.datasets[0].data.push({{ row.pass_rate_2 }});
{% endfor %}
=======
var leaderboardData = {
labels: [],
datasets: [{
label: 'Percent correct on code editing tasks',
data: [],
backgroundColor: 'rgba(54, 162, 235, 0.2)',
borderColor: 'rgba(54, 162, 235, 1)',
borderWidth: 1
},
{
label: 'Percent cases well formed',
data: [],
backgroundColor: 'rgba(255, 99, 132, 0.2)',
borderColor: 'rgba(255, 99, 132, 1)',
borderWidth: 1
}]
};
{% for row in edit_sorted %}
leaderboardData.labels.push('{{ row.model }}');
leaderboardData.datasets[0].data.push({{ row.pass_rate_2 }});
leaderboardData.datasets[1].data.push({{ row.percent_cases_well_formed }});
{% endfor %}
>>>>>>> REPLACE
```
docs/leaderboards/index.md
```markdown
<<<<<<< SEARCH
var leaderboardData = {
labels: [],
datasets: [{
label: 'Percent correct on code refactoring tasks',
data: [],
backgroundColor: 'rgba(54, 162, 235, 0.2)',
borderColor: 'rgba(54, 162, 235, 1)',
borderWidth: 1
}]
};
{% for row in refac_sorted %}
leaderboardData.labels.push('{{ row.model }}');
leaderboardData.datasets[0].data.push({{ row.pass_rate_1 }});
{% endfor %}
=======
var leaderboardData = {
labels: [],
datasets: [{
label: 'Percent correct on code refactoring tasks',
data: [],
backgroundColor: 'rgba(54, 162, 235, 0.2)',
borderColor: 'rgba(54, 162, 235, 1)',
borderWidth: 1
},
{
label: 'Percent cases well formed',
data: [],
backgroundColor: 'rgba(255, 99, 132, 0.2)',
borderColor: 'rgba(255, 99, 132, 1)',
borderWidth: 1
}]
};
{% for row in refac_sorted %}
leaderboardData.labels.push('{{ row.model }}');
leaderboardData.datasets[0].data.push({{ row.pass_rate_1 }});
leaderboardData.datasets[1].data.push({{ row.percent_cases_well_formed }});
{% endfor %}
>>>>>>> REPLACE
```
This adds a second dataset to each chart's `leaderboardData`, with a label of "Percent cases well formed". It sets different colors for the bars and borders.
Then in the loop that populates the data, it pushes the `row.percent_cases_well_formed` value into the second dataset.
With these changes, each model will have two bars - one for the pass rate and one for the percent of cases that were well formed.
7.3 KiB
Aider LLM leaderboards
Aider works best with LLMs which are good at editing code, not just good at writing code. Aider uses the system prompt to tell the LLM how to make edits to the existing code in your local git repo. Some LLMs are better than others at consistently following these instructions to successfully edit code.
Aider uses two benchmarks to measure an LLM's code editing ability:
- Aider's code editing benchmark asks the LLM to edit python source files to complete 133 small coding exercises. This benchmark measures the LLM's coding ability, but also whether it can consistently emit code edits in the format specified in the system prompt.
- Aider's refactoring benchmark asks the LLM to refactor 89 large methods from large python classes. This is a more challenging benchmark, which tests the model's ability to output long chunks of code without skipping sections or making mistakes. It was developed to provoke and measure GPT-4 Turbo's "lazy coding" habit.
The leaderboards below report the results from a number of popular LLMs, to help users select which models to use with aider. While aider can connect to almost any LLM it will work best with models that score well on the benchmarks.
Code editing leaderboard
{% assign edit_sorted = site.data.edit_leaderboard | sort: 'pass_rate_2' | reverse %} {% for row in edit_sorted %} {% endfor %}| Model | Percent correct | Command | Edit format |
|---|---|---|---|
| {{ row.model }} | {{ row.pass_rate_2 }}% | {{ row.command }} |
{{ row.edit_format }} |
Code refactoring leaderboard
The refactoring benchmark requires a large context window to work with large source files. Therefore, results are available for fewer models.
{% assign refac_sorted = site.data.refactor_leaderboard | sort: 'pass_rate_1' | reverse %} {% for row in refac_sorted %} {% endfor %}| Model | Percent correct | Command | Edit format |
|---|---|---|---|
| {{ row.model }} | {{ row.pass_rate_1 }}% | {{ row.command }} |
{{ row.edit_format }} |
Notes on the edit format
Aider uses different "edit formats" to collect code edits from different LLMs. The "whole" format is the easiest for an LLM to use, but it uses a lot of tokens and may limit how large a file can be edited. Models which can use one of the diff formats are much more efficient, using far fewer tokens. Models that use a diff-like format are able to edit larger files with less cost and without hitting token limits.
Aider is configured to use the best edit format for the popular OpenAI and Anthropic models and the other models recommended on the LLM page. For lesser known models aider will default to using the "whole" editing format since it is the easiest format for an LLM to use.
Contributing benchmark results
Contributions of benchmark results are welcome! See the benchmark README for information on running aider's code editing benchmark. Submit results by opening a PR with edits to the benchmark results data files.