Commit graph

247 commits

Author SHA1 Message Date
Paul Gauthier
ab55b6ff0a Copy ORIGINAL_DNAME to dirname. 2023-11-08 10:07:27 -08:00
Paul Gauthier
59fed25cd1 copy 2023-11-08 08:36:48 -08:00
Paul Gauthier
2c83287c46 refac 2023-11-08 06:21:16 -08:00
Paul Gauthier
5da64a6abc copy 2023-11-07 07:22:26 -08:00
Paul Gauthier
dfb921c509 copy 2023-11-06 20:24:44 -08:00
Paul Gauthier
15a1c754a4 copy 2023-11-06 18:58:24 -08:00
Paul Gauthier
35754247b4 copy 2023-11-06 18:48:14 -08:00
Paul Gauthier
976fc7a836 update benchmarking script 2023-11-06 18:26:02 -08:00
Joshua Vial
91b7648f33 update model creation method 2023-09-09 12:06:34 +12:00
Paul Gauthier
2c53c153b8 Updated ORIGINAL prompting 2023-08-09 11:38:39 -03:00
Paul Gauthier
0aa2ff2cf9 roughed in cache; elide full dirnames from test results to make them deterministic 2023-08-09 11:30:13 -03:00
Paul Gauthier
2eeee67ee5 support --keywords 2023-08-09 11:12:27 -03:00
Paul Gauthier
c43d7e9a69 initial --diffs implementation 2023-08-08 17:40:32 -03:00
Paul Gauthier
568be3812b Fix last_keyboard_interrupt in benchmark 2023-07-20 13:25:15 -03:00
Paul Gauthier
edbec5498e update benchmark driver 2023-07-13 20:02:58 -07:00
Paul Gauthier
44311f4737 preparing for v0.8.0 2023-07-06 14:03:06 -07:00
Paul Gauthier
ba2c5d05da new svg 2023-07-02 10:45:14 -07:00
Paul Gauthier
4ea70bfdc0 new svg 2023-07-02 10:35:11 -07:00
Paul Gauthier
5e4ad5bf00 new svg 2023-07-02 10:28:39 -07:00
Paul Gauthier
8f4a823855 new svg 2023-07-02 10:26:18 -07:00
Paul Gauthier
2915acb9a6 new svg 2023-07-02 10:23:06 -07:00
Paul Gauthier
86ced88f8e new svg 2023-07-02 10:18:58 -07:00
Paul Gauthier
db18876db6 plot the repeat error bars 2023-07-02 07:10:54 -07:00
Paul Gauthier
a4ab262d96 new svg 2023-07-01 21:35:38 -07:00
Paul Gauthier
f548f88220 new svg 2023-07-01 21:31:54 -07:00
Paul Gauthier
b9f8ed47f4 copy 2023-07-01 17:50:48 -07:00
Paul Gauthier
8ef166478a graph layout 2023-07-01 17:18:14 -07:00
Paul Gauthier
3f0c752368 show 1st & 2nd coding attempts 2023-07-01 16:40:25 -07:00
Paul Gauthier
f27a869649 new svg 2023-07-01 16:23:09 -07:00
Paul Gauthier
bddc5a36f0 cleanup 2023-07-01 13:29:52 -07:00
Paul Gauthier
34b4ab199d show try 1 and 2 2023-07-01 13:28:36 -07:00
Paul Gauthier
6471b89033 return 0 2023-07-01 06:50:13 -07:00
Paul Gauthier
92e5f43abe intro 2023-06-30 20:42:54 -07:00
Paul Gauthier
5942057a9c introduction.md 2023-06-30 20:39:08 -07:00
Paul Gauthier
895d3d2d67 skip graph 2023-06-30 20:36:12 -07:00
Paul Gauthier
5310ed6faf filename 2023-06-30 20:31:17 -07:00
Paul Gauthier
c745a3156f instructions_append 2023-06-30 20:30:00 -07:00
Paul Gauthier
8a0909738d stars 2023-06-30 19:27:57 -07:00
Paul Gauthier
674bfb52c6 updated graph with larger fonts 2023-06-30 19:19:23 -07:00
Paul Gauthier
ca306b6a4a remove the 1 try results 2023-06-30 14:01:29 -07:00
Paul Gauthier
7c56363d86 copy 2023-06-30 13:07:24 -07:00
Paul Gauthier
f84e3d6ab7 show both tries; repeats 2023-06-30 10:20:06 -07:00
Paul Gauthier
a9cc3f81b2 nice graph 2023-06-30 06:44:37 -07:00
Paul Gauthier
cba75a3bfe prettier 2023-06-29 22:08:09 -07:00
Paul Gauthier
73f99996df aider: Set bar color/fill style for each edit-format in show_stats function using a color map.
# Context:
USER: how can i set the bar color/fill style for each edit-format
ASSISTANT: To set the bar color/fill style for each edit-format, you need to modify the `show_stats` function in the `benchmark/benchmark.py` file.

Here are the steps:

1. Define a dictionary mapping each edit-format to a specific color or fill style.
2. Use the `color` or `fill` parameter in the `plot` function to set the color or fill style for each bar.

Here is the code change:

```python
benchmark/benchmark.py
<<<<<<< ORIGINAL
    fig, ax = plt.subplots(figsize=(10, 6))
    df_grouped2.unstack().plot(kind="barh", ax=ax)
=======
    fig, ax = plt.subplots(figsize=(10, 6))
    color_map = {"edit_format1": "blue", "edit_format2": "green", "edit_format3": "red"}  # Define your color map here
    df_grouped2.unstack().plot(kind="barh", ax=ax, color=[color_map[i] for i in df_grouped2.index])
>>>>>>> UPDATED
```

Please replace `"edit_format1"`, `"edit_format2"`, and `"edit_format3"` with your actual edit formats, and `"blue"`, `"green"`, and `"red"` with your desired colors.
2023-06-29 20:56:58 -07:00
Paul Gauthier
a02cadbc36 tight 2023-06-29 20:56:12 -07:00
Paul Gauthier
490a59cba4 copy 2023-06-29 20:47:49 -07:00
Paul Gauthier
cf171bdd1e noop 2023-06-29 19:43:06 -07:00
Paul Gauthier
a577a76a44 check for dup model/edit rows 2023-06-29 19:11:28 -07:00
Paul Gauthier
0c9deaa118 aider: Sort the grouped columns by descending average.
# Context:
USER: sort the grouped columns by descending average
ASSISTANT: To sort the grouped columns by descending average, we need to modify the `show_stats` function in the `benchmark/benchmark.py` file. Specifically, we need to sort the `df_grouped` DataFrame by its values in descending order.

Here is the change:

```python
benchmark/benchmark.py
<<<<<<< ORIGINAL
    df_grouped = df.groupby(["model", "edit_format"])["pass_rate_1"].mean()
=======
    df_grouped = df.groupby(["model", "edit_format"])["pass_rate_1"].mean().sort_values(ascending=False)
>>>>>>> UPDATED
```

This change will sort the grouped DataFrame `df_grouped` by its values in descending order. The `ascending=False` argument in the `sort_values` function ensures that the sorting is done in descending order.
2023-06-29 19:05:16 -07:00