From 4c871c6f5021d67983eab43522a2172b62a2c8db Mon Sep 17 00:00:00 2001 From: Paul Gauthier Date: Wed, 7 May 2025 09:28:52 -0700 Subject: [PATCH] docs: Audit gemini pricing in 0325 benchmark results --- .../_posts/2025-05-07-gemini-pricing.md | 19 +++++++++++++------ 1 file changed, 13 insertions(+), 6 deletions(-) diff --git a/aider/website/_posts/2025-05-07-gemini-pricing.md b/aider/website/_posts/2025-05-07-gemini-pricing.md index 2c72161ee..b8da8629c 100644 --- a/aider/website/_posts/2025-05-07-gemini-pricing.md +++ b/aider/website/_posts/2025-05-07-gemini-pricing.md @@ -21,31 +21,38 @@ There are a couple of reasons for concern: This note reviews and audits the original 0325 benchmark results to investigate the reported price. +The incorrect litellm database entry does not appear to have affected the aider benchmark. +Aider maintains and uses its own database of costs for some models, and it contained +the correct pricing at the time of the benchmark. +This was possible to confirm because +aider records the git commit hash of the aider repository that was used +to run each benchmark. -Confirmed that aider built and run from commit 0282574 honors `output_cost_per_token` from `aider/resources/model-metadata.json` by putting in an absurdly high value and benchmarking `gemini/gemini-2.5-pro-preview-03-25` # Timeline - 2025-04-04 19:54:45 UTC (Sat Apr 5 08:54:45 2025 +1300) - - Commit eda796d feat: Add metadata and settings for gemini-2.5-pro-preview-03-25 - Correct value `"output_cost_per_token": 0.000010` added to `aider/resources/model-metadata.json` + - Commit eda796d in aider. - 2025-04-05 16:20:01 UTC (Sun Apr 6 00:20:01 2025 +0800) - First litellm commit of `gemini/gemini-2.5-pro-preview-03-25` metadata, with incorrect price `"output_cost_per_token": 0.0000010` - - Commit cd0a1e6 + - Commit cd0a1e6 in litellm. - 2025-04-10 01:48:43 UTC (Wed Apr 9 18:48:43 2025 -0700) - litellm commit updates `gemini/gemini-2.5-pro-preview-03-25` metadata, but not price - - commit ac4f32f + - Commit ac4f32f in litellm. - 2025-04-12 04:55:50 UTC (2025-04-12-04-55-50 UTC) - Benchmark performed - - Repo hash 0282574 recorded in benchmark results, without "dirty" indicating it was run on a clean checkout of the repo at commit 0282574. + - Aider repo hash 0282574 recorded in benchmark results, without "dirty" indicating it was run on a clean checkout of the repo at commit 0282574. - Correct value `"output_cost_per_token": 0.000010` is in `aider/resources/model-metadata.json` at commit 0282574 + - Confirmed that aider built and run from commit 0282574 honors `output_cost_per_token` from `aider/resources/model-metadata.json` by putting in an absurdly high value and benchmarking `gemini/gemini-2.5-pro-preview-03-25` - 2025-04-12 15:06:39 UTC (Apr 12 08:06:39 2025 -0700) - Benchmark results added to repo + - Commit 7fbeafa in aider. - 2025-04-12 15:20:04 UTC (Sat Apr 12 19:20:04 2025 +0400) - litellm commit fixes `gemini/gemini-2.5-pro-preview-03-25` price metadata to `"output_cost_per_token": 0.00001` - - commit 93037ea + - Commit 93037ea in litellm.