dearwolf/aider
1
0
Fork 0
mirror of https://github.com/Aider-AI/aider.git synced 2025-05-31 09:44:59 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 9

docs: Audit gemini pricing in 0325 benchmark results

Browse source
This commit is contained in:
Paul Gauthier 2025-05-07 09:28:52 -07:00 • committed by Paul Gauthier (aider)
parent d56ce3ae56
commit 4c871c6f50
1 changed files with 13 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

19
aider/website/_posts/2025-05-07-gemini-pricing.md
View file

@ -21,31 +21,38 @@ There are a couple of reasons for concern:
This note reviews and audits the original 0325 benchmark results to investigate the reported price.
The incorrect litellm database entry does not appear to have affected the aider benchmark.
Aider maintains and uses its own database of costs for some models, and it contained
the correct pricing at the time of the benchmark.
This was possible to confirm because
aider records the git commit hash of the aider repository that was used
to run each benchmark.
Confirmed that aider built and run from commit 0282574 honors `output_cost_per_token` from `aider/resources/model-metadata.json` by putting in an absurdly high value and benchmarking `gemini/gemini-2.5-pro-preview-03-25`
# Timeline
- 2025-04-04 19:54:45 UTC (Sat Apr 5 08:54:45 2025 +1300)
- Commit eda796d feat: Add metadata and settings for gemini-2.5-pro-preview-03-25
- Correct value `"output_cost_per_token": 0.000010` added to `aider/resources/model-metadata.json`
- Commit eda796d in aider.
- 2025-04-05 16:20:01 UTC (Sun Apr 6 00:20:01 2025 +0800)
- First litellm commit of `gemini/gemini-2.5-pro-preview-03-25` metadata, with incorrect price `"output_cost_per_token": 0.0000010`
- Commit cd0a1e6
- Commit cd0a1e6 in litellm.
- 2025-04-10 01:48:43 UTC (Wed Apr 9 18:48:43 2025 -0700)
- litellm commit updates `gemini/gemini-2.5-pro-preview-03-25` metadata, but not price
- commit ac4f32f
- Commit ac4f32f in litellm.
- 2025-04-12 04:55:50 UTC (2025-04-12-04-55-50 UTC)
- Benchmark performed
- Repo hash 0282574 recorded in benchmark results, without "dirty" indicating it was run on a clean checkout of the repo at commit 0282574.
- Aider repo hash 0282574 recorded in benchmark results, without "dirty" indicating it was run on a clean checkout of the repo at commit 0282574.
- Correct value `"output_cost_per_token": 0.000010` is in `aider/resources/model-metadata.json` at commit 0282574
- Confirmed that aider built and run from commit 0282574 honors `output_cost_per_token` from `aider/resources/model-metadata.json` by putting in an absurdly high value and benchmarking `gemini/gemini-2.5-pro-preview-03-25`
- 2025-04-12 15:06:39 UTC (Apr 12 08:06:39 2025 -0700)
- Benchmark results added to repo
- Commit 7fbeafa in aider.
- 2025-04-12 15:20:04 UTC (Sat Apr 12 19:20:04 2025 +0400)
- litellm commit fixes `gemini/gemini-2.5-pro-preview-03-25` price metadata to `"output_cost_per_token": 0.00001`
- commit 93037ea
- Commit 93037ea in litellm.