dearwolf/aider
1
0
Fork 0
mirror of https://github.com/Aider-AI/aider.git synced 2025-05-29 00:35:00 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

copy

Browse source
This commit is contained in:
Paul Gauthier 2024-05-04 11:19:32 -07:00
parent fbb3749270
commit b74edcf350
1 changed files with 7 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

10
docs/leaderboard.md
View file

@ -9,10 +9,10 @@ so the LLM needs to be capable of reliably specifying how to edit code.
Aider uses two benchmarks
to measure an LLM's code editing ability:
- The [code editing benchmark](https://aider.chat/docs/benchmarks.html#the-benchmark) asks the LLM to edit python source files to complete 133 Exercism exercises.
- The [refactoring benchmark](https://github.com/paul-gauthier/refactor-benchmark) asks the LLM to refactor large methods from a large python source file. This is a more challenging benchmark, which tests the model's ability to output long chunks of code without skipping sections.
- The [code editing benchmark](https://aider.chat/docs/benchmarks.html#the-benchmark) asks the LLM to edit python source files to complete 133 Exercism exercises. This benchmark measures the LLM's ability to emit code edits according to the format aider specifies in the system prompt.
- The [refactoring benchmark](https://github.com/paul-gauthier/refactor-benchmark) asks the LLM to refactor 89 large methods from large python classes. This is a more challenging benchmark, which tests the model's ability to output long chunks of code without skipping sections. It was developed to provoke and measure GPT-4 Turbo's "lazy coding" habit.
These leaderboards report the results from a number of popular LLMs,
The leaderboards below report the results from a number of popular LLMs,
to help users select which models to use with aider.
While [aider can connect to almost any LLM](https://aider.chat/docs/llms.html)
it will work best with models that score well on the benchmarks.
@ -83,6 +83,10 @@ it will work best with models that score well on the benchmarks.
## Code refactoring leaderboard
The refactoring benchmark requires a large context window to
work with large source files.
Therefore, results are available for fewer models.
<table style="width: 90%; max-width: 800px; margin: auto; border-collapse: collapse; box-shadow: 0 2px 4px rgba(0,0,0,0.1); font-size: 14px;">
<thead style="background-color: #f2f2f2;">
<tr>