dearwolf/aider
1
0
Fork 0
mirror of https://github.com/Aider-AI/aider.git synced 2025-05-24 22:34:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 9
aider/docs/benchmarks-speed-1106.md
paul-gauthier a4e224e190
Update benchmarks-speed-1106.md
2023-11-08 12:30:56 -08:00

2.6 KiB
Raw Blame History

Speed benchmarks of GPT-4 Turbo and gpt-3.5-turbo-1106

benchmark results

OpenAI just released new versions of GPT-3.5 and GPT-4, and there's a lot of interest about their capabilities and performance. With that in mind, I've been benchmarking the new models.

Aider is an open source command line chat tool that lets you work with GPT to edit code in your local git repo. Aider relies on a code editing benchmark to quantitatively evaluate performance.

This is the latest in a series of reports that use the aider benchmarking suite to assess and compare the code editing capabilities of OpenAI's GPT models. You can review previous reports to get more background on aider's benchmark suite:

Speed

This report compares the speed of the various GPT models. Aider's benchmark measures the response time of the OpenAI chat completion endpoint each time it asks GPT to solve a programming exercise in the benchmark suite. These results measure only the time spent waiting for OpenAI to respond to the prompt. So they are measuring how fast these models can generate responses which primarily consist of source code.

Some observations:

  • GPT-3.5 got 6-11x faster. The gpt-3.5-turbo-1106 model is 6-11x faster than the June (0613) version which has been the default gpt-3.5-turbo model.
  • GPT-4 Turbo is 4-5x faster. The new gpt-4-1106-preview model is 4-5x faster than the June (0613) version which has been the default gpt-4 model.
  • The old March (0301) version of GPT-3.5 is actually faster than the June (0613) version. This was a surprising discovery.

These are preliminary results. OpenAI is enforcing very low rate limits on the new GPT-4 model. The rate limiting disrupts the benchmarking process, requiring it to run single threaded, pause and restart frequently. These anomolous conditions make it slow to benchmark the new model, and make it less reliable to compare the results with benchmark runs against the older model versions. Once the rate limits are relaxed I will do a clean run of the entire benchmark suite.