This commit is contained in:
Paul Gauthier 2024-07-25 11:20:08 +02:00
parent 33e47c7efd
commit d0ebc7a810

View file

@ -47,8 +47,8 @@ Llama 3.1 family of models,
which have performed well on many evals. which have performed well on many evals.
The flagship Llama 3.1 405B instruct only The flagship Llama 3.1 405B instruct only
secured #7 on aider's leaderboard. secured #7 on aider's leaderboard,
This is well behind frontier models like well behind frontier models like
Claude 3.5 Sonnet & GPT-4o. Claude 3.5 Sonnet & GPT-4o.
The 405B model can use SEARCH/REPLACE to efficiently The 405B model can use SEARCH/REPLACE to efficiently
@ -58,6 +58,9 @@ from 66% to 64%.
The smaller 70B model was competitive with GPT-3.5, while The smaller 70B model was competitive with GPT-3.5, while
the 8B model lags far behind. the 8B model lags far behind.
Both seem unable to reliably use SEARCH/REPLACE to edit files.
This limits them to editing smaller files that can
fit into their output token limit.
## Mistral Large 2 (2407) ## Mistral Large 2 (2407)