From 0a4a547c3df185f5a8e8fcf99b2a1ece2f0cf3f4 Mon Sep 17 00:00:00 2001 From: Paul Gauthier Date: Wed, 21 Jun 2023 18:25:55 -0700 Subject: [PATCH 1/6] Updated 3.5 vs 4 --- README.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 0e84fa594..b97f6aef0 100644 --- a/README.md +++ b/README.md @@ -125,9 +125,6 @@ aren't able to return code edits in a compact format. So aider has to ask GPT-3.5 to return a full copy of any code that needs to be edited. This rapidly uses up tokens and can hit the limits of the context window. -In practice, this means you can only use `gpt-3.5-turbo` to edit files that are -smaller than about 2k tokens (8k bytes). -The new `gpt-3.5-turbo-16k` model should be able to edit code up to 8k tokens (32k bytes). Aider disables the [repository map feature](https://aider.chat/docs/ctags.html) @@ -135,6 +132,17 @@ when used with GPT-3.5 models. The `gpt-3.5-turbo` context window is too small to include a repo map. Evaluation is still needed to determine if `gpt-3.5-turbo-16k` can make use of a repo map. +In practice, this means you can use aider to edit a set of source files +that total up to these sizes: + +| model | context size | max file size | max file size | +| ----------------- | -- | -----| -- | +| gpt-3.5-turbo | 4k tokens | 2k tokens | 8k bytes | +| gpt-3.5-turbo-16k | 16k tokens | 8k tokens | 32k bytes | +| gpt-4 | 8k tokens | 8k tokens | 32k bytes | +| gpt-4-32k | 32k tokens | 32k tokens | 128k bytes | + + ## Tips * Large changes are best performed as a sequence of thoughtful bite sized steps, where you plan out the approach and overall design. Walk GPT through changes like you might with a junior dev. Ask for a refactor to prepare, then ask for the actual change. Spend the time to ask for code quality/structure improvements. From 5a05370627fbcb823c47bb66bf3fa2c4c7b3de92 Mon Sep 17 00:00:00 2001 From: Paul Gauthier Date: Wed, 21 Jun 2023 18:27:47 -0700 Subject: [PATCH 2/6] Updated 3.5 vs 4 --- README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index b97f6aef0..c2e50d17d 100644 --- a/README.md +++ b/README.md @@ -135,12 +135,12 @@ Evaluation is still needed to determine if `gpt-3.5-turbo-16k` can make use of a In practice, this means you can use aider to edit a set of source files that total up to these sizes: -| model | context size | max file size | max file size | -| ----------------- | -- | -----| -- | -| gpt-3.5-turbo | 4k tokens | 2k tokens | 8k bytes | -| gpt-3.5-turbo-16k | 16k tokens | 8k tokens | 32k bytes | -| gpt-4 | 8k tokens | 8k tokens | 32k bytes | -| gpt-4-32k | 32k tokens | 32k tokens | 128k bytes | +| model | edit format | context size | max file size | max file size | +| ----------------- | -- | -- | -----| -- | +| gpt-3.5-turbo | whole file | 4k tokens | 2k tokens | 8k bytes | +| gpt-3.5-turbo-16k | whole file | 16k tokens | 8k tokens | 32k bytes | +| gpt-4 | diffs | 8k tokens | 8k tokens | 32k bytes | +| gpt-4-32k | diffs | 32k tokens | 32k tokens | 128k bytes | ## Tips From efd4e973ddcdd2bff80268bba2af6d30bdf81f64 Mon Sep 17 00:00:00 2001 From: Paul Gauthier Date: Wed, 21 Jun 2023 18:29:33 -0700 Subject: [PATCH 3/6] Updated 3.5 vs 4 --- README.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index c2e50d17d..086fd125a 100644 --- a/README.md +++ b/README.md @@ -114,16 +114,17 @@ You will probably get the best results with one of the GPT-4 models, because of their large context windows, adherance to system prompt instructions and greater competance at coding tasks. -The GPT-4 models are also able to use a +The GPT-4 models are able to structure code edits as simple "diffs" +and use a [repository map](https://aider.chat/docs/ctags.html) to improve their ability to make changes in larger codebases. The GPT-3.5 models are supported more experimentally and are limited to editing somewhat smaller codebases. They are less able to follow instructions and -aren't able to return code edits in a compact format. +aren't able to return code edits in a compact "diff" format. So aider has -to ask GPT-3.5 to return a full copy of any code that needs to be edited. +to ask GPT-3.5 to return a new copy of the "whole file" with edits included. This rapidly uses up tokens and can hit the limits of the context window. Aider disables the From 443f9bf2b90f6be67f8601f15b606f95227b4ecf Mon Sep 17 00:00:00 2001 From: Paul Gauthier Date: Wed, 21 Jun 2023 18:30:51 -0700 Subject: [PATCH 4/6] copy --- README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/README.md b/README.md index 086fd125a..1ddca2a5e 100644 --- a/README.md +++ b/README.md @@ -146,6 +146,8 @@ that total up to these sizes: ## Tips +* Think about which files need to be edited to make your change and add them to the chat. +Aider has some ability to help GPT figure out which files to edit all by itself, but the most effective approach is to explicitly add the needed files to the chat yourself. * Large changes are best performed as a sequence of thoughtful bite sized steps, where you plan out the approach and overall design. Walk GPT through changes like you might with a junior dev. Ask for a refactor to prepare, then ask for the actual change. Spend the time to ask for code quality/structure improvements. * Use Control-C to safely interrupt GPT if it isn't providing a useful response. The partial response remains in the conversation, so you can refer to it when you reply to GPT with more information or direction. * Use the `/run` command to run tests, linters, etc and show the output to GPT so it can fix any issues. From dc5b33164a0695a66d07799982d65c5f1bbcf57e Mon Sep 17 00:00:00 2001 From: Paul Gauthier Date: Wed, 21 Jun 2023 18:34:44 -0700 Subject: [PATCH 5/6] copy --- README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 1ddca2a5e..690ef56a8 100644 --- a/README.md +++ b/README.md @@ -134,7 +134,10 @@ The `gpt-3.5-turbo` context window is too small to include a repo map. Evaluation is still needed to determine if `gpt-3.5-turbo-16k` can make use of a repo map. In practice, this means you can use aider to edit a set of source files -that total up to these sizes: +that total up to the sizes below. +You can (and should) add just the specific set of files to the chat +that are relevant to the change you are requesting. +This minimizes your use of the context window, as well as costs. | model | edit format | context size | max file size | max file size | | ----------------- | -- | -- | -----| -- | From 088d81d54da325d307545dad8645f8d9b815508c Mon Sep 17 00:00:00 2001 From: Paul Gauthier Date: Wed, 21 Jun 2023 18:36:44 -0700 Subject: [PATCH 6/6] copy --- README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/README.md b/README.md index 690ef56a8..e8ca53995 100644 --- a/README.md +++ b/README.md @@ -139,12 +139,12 @@ You can (and should) add just the specific set of files to the chat that are relevant to the change you are requesting. This minimizes your use of the context window, as well as costs. -| model | edit format | context size | max file size | max file size | +| model | context size | edit format | max file size | max file size | | ----------------- | -- | -- | -----| -- | -| gpt-3.5-turbo | whole file | 4k tokens | 2k tokens | 8k bytes | -| gpt-3.5-turbo-16k | whole file | 16k tokens | 8k tokens | 32k bytes | -| gpt-4 | diffs | 8k tokens | 8k tokens | 32k bytes | -| gpt-4-32k | diffs | 32k tokens | 32k tokens | 128k bytes | +| gpt-3.5-turbo | 4k tokens | whole file | 2k tokens | 8k bytes | +| gpt-3.5-turbo-16k | 16k tokens | whole file | 8k tokens | 32k bytes | +| gpt-4 | 8k tokens | diffs | 8k tokens | 32k bytes | +| gpt-4-32k | 32k tokens | diffs | 32k tokens | 128k bytes | ## Tips