dearwolf/aider
1
0
Fork 0
mirror of https://github.com/Aider-AI/aider.git synced 2025-05-31 01:35:00 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 9

use swe-bench-lite.txt

Browse source
This commit is contained in:
Paul Gauthier 2024-05-23 13:56:17 -07:00
parent a395d9bb5c
commit 9d20b5f43a
2 changed files with 45 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

44
benchmark/swe-bench-lite.txt Normal file
View file

@ -0,0 +1,44 @@
model: gpt-3.5-turbo
validate_environment: {'keys_in_environment': True, 'missing_keys': []}
completion: Hello! How can I assist you today?
model: claude-3-sonnet-20240229
validate_environment: {'keys_in_environment': True, 'missing_keys': []}
completion: Hello! How can I assist you today?
model: gemini/gemini-1.5-pro-latest
validate_environment: {'keys_in_environment': False, 'missing_keys': []}
completion: Hello! 👋 How can I help you today? 😊
model: command-r-plus
validate_environment: {'keys_in_environment': False, 'missing_keys': []}
completion: Hello! How can I help you today?
erased environment
model: gpt-3.5-turbo
validate_environment: {'keys_in_environment': False, 'missing_keys': ['OPENAI_API_KEY']}
completion error: OpenAIException - Traceback (most recent call last):
File "/Users/gauthier/Projects/aider/.venv/lib/python3.11/site-packages/litellm/llms/openai.py", line 414, in completion
raise e
File "/Users/gauthier/Projects/aider/.venv/lib/python3.11/site-packages/litellm/llms/openai.py", line 350, in completion
openai_client = openai(
^^^^^^^
File "/Users/gauthier/Projects/aider/.venv/lib/python3.11/site-packages/openai/_client.py", line 104, in __init__
raise openaiError(
openai.openaiError: The api_key client option must be set either by passing api_key to the client or by setting the OPENAI_API_KEY environment variable
model: claude-3-sonnet-20240229
validate_environment: {'keys_in_environment': False, 'missing_keys': ['ANTHROPIC_API_KEY']}
completion error: Missing Anthropic API Key - A call is being made to anthropic but no key is set either in the environment variables or via params
model: gemini/gemini-1.5-pro-latest
validate_environment: {'keys_in_environment': False, 'missing_keys': []}
completion error: PalmException - Invalid api key
model: command-r-plus
validate_environment: {'keys_in_environment': False, 'missing_keys': []}
completion error: {"message":"no api key supplied"}

2
benchmark/swe_bench_lite.py
View file

@ -68,4 +68,4 @@ def plot_swe_bench_lite(data_file):
# Example usage
plot_swe_bench_lite("benchmark/tmp.txt")
plot_swe_bench_lite("benchmark/swe-bench-lite.txt")