From e2af782b59396eda898953c67d3e2ed212989c81 Mon Sep 17 00:00:00 2001 From: Paul Gauthier Date: Sat, 1 Jul 2023 21:28:07 -0700 Subject: [PATCH] copy --- docs/benchmarks.md | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/docs/benchmarks.md b/docs/benchmarks.md index 2e242d37e..ca6511c0c 100644 --- a/docs/benchmarks.md +++ b/docs/benchmarks.md @@ -348,3 +348,16 @@ cause a large variance in the overall benchmark results. Based on these benchmark results, aider will continue to use the `whole` edit format for GPT-3.5, and `diff` for GPT-4. + +GPT-4 gets comparable results with the `diff` and `whole` edit formats, +but using `whole` significantly increases costs and latency compared to `diff`. + +The latency of streaming back the entire updated copy of each edited file +is the real challenge. The GPT-3.5 models are quite responsive, and can +stream back entire files at reasonable speed. +Aider displays a progress bar and +live diffs of the files as they stream in, +which helps pass the time. + +The GPT-4 models are much slower, and waiting for even small files +to be completely "retyped" on each request is probably unacceptable.