mirror of
https://github.com/Aider-AI/aider.git
synced 2025-05-28 16:25:00 +00:00
c7ba0a5dba
# Context:
USER: don't combine the edit formats. plot a different bar for each model/edit_format combo. But group the bars for each model together.
ASSISTANT: To implement this change, we need to modify the `show_stats` function in the `benchmark/benchmark.py` file. We will use the pandas and matplotlib libraries to create the bar plot.
Here are the steps:
1. In the `show_stats` function, after creating the DataFrame `df`, sort it based on "model" and "edit_format".
2. Group the DataFrame by "model" and "edit_format" and plot a bar graph for "pass_rate_1".
Here are the changes:
```python
benchmark/benchmark.py
<<<<<<< ORIGINAL
def show_stats(dirnames):
rows = []
for dirname in dirnames:
row = summarize_results(dirname)
rows.append(vars(row))
df = pd.DataFrame.from_records(rows)
df.sort_values(by=["model", "edit_format"], inplace=True)
df_grouped = df.groupby("model")["pass_rate_1"].mean()
df_grouped.plot(kind='barh', figsize=(10, 6))
plt.xlabel('Pass Rate 1')
plt.ylabel('Model')
plt.title('Pass Rate 1 for each Model/Edit Format')
plt.show()
df.to_csv("tmp.benchmark.csv")
=======
def show_stats(dirnames):
rows = []
for dirname in dirnames:
row = summarize_results(dirname)
rows.append(vars(row))
df = pd.DataFrame.from_records(rows)
df.sort_values(by=["model", "edit_format"], inplace=True)
df_grouped = df.groupby(["model", "edit_format"])["pass_rate_1"].mean()
df_grouped.unstack().plot(kind='barh', figsize=(10, 6))
plt.xlabel('Pass Rate 1')
plt.ylabel('Model')
plt.title('Pass Rate 1 for each Model/Edit Format')
plt.legend(title='Edit Format')
plt.show()
df.to_csv("tmp.benchmark.csv")
>>>>>>> UPDATED
```
This will create a bar graph with the average of "pass_rate_1" for each "edit_format" within each model. The bars for each model are grouped together.
|
||
|---|---|---|
| .. | ||
| __init__.py | ||
| benchmark.py | ||
| docker.sh | ||
| docker_build.sh | ||
| Dockerfile | ||
| prompts.py | ||
| test_benchmark.py | ||