Commit graph

269 commits

Author SHA1 Message Date
Paul Gauthier
1a3d8c4015 wip 2024-08-20 17:45:40 -07:00
Paul Gauthier
821eae16ae copy 2024-08-19 20:54:10 -07:00
Paul Gauthier
e0a9044118 copy 2024-08-19 20:53:42 -07:00
Paul Gauthier
730d6e0e94 copy 2024-08-19 20:51:03 -07:00
Paul Gauthier
86a7a17d47 copy 2024-08-19 20:47:52 -07:00
Paul Gauthier
2944445340 copy 2024-08-19 20:44:48 -07:00
Paul Gauthier
b61b5f4b74 cleanup before merge 2024-08-16 11:35:30 -07:00
Paul Gauthier
3a2ac02024 Merge branch 'main' into json-coders 2024-08-15 12:15:07 -07:00
Paul Gauthier
822a8ab671 remove gpt-4o-mini from the gpt-4 trendline 2024-08-15 09:52:21 -07:00
Paul Gauthier (aider)
5ccdebf2c0 refactor: Extract color assignment logic into a separate function 2024-08-15 09:50:50 -07:00
Paul Gauthier
bac04a2a3d no lint 2024-08-15 06:10:46 -07:00
Paul Gauthier (aider)
0a3c6bfbe7 feat: Change blue color to light blue in plot_over_time function 2024-08-14 06:29:48 -07:00
Paul Gauthier (aider)
d2b4846b95 feat: Replace orange color with purple for "-4o" models 2024-08-14 06:29:13 -07:00
Paul Gauthier (aider)
fb0b348bec fix: Remove unused blue_points variable 2024-08-14 06:28:28 -07:00
Paul Gauthier (aider)
a7290be843 style: Apply linter formatting changes 2024-08-14 06:27:51 -07:00
Paul Gauthier (aider)
1cdbc76974 feat: Connect model family lines in over_time plot 2024-08-14 06:27:48 -07:00
Paul Gauthier
714fd45f4d fix: Update color logic and font size in over_time.py 2024-08-14 06:27:47 -07:00
Paul Gauthier (aider)
1f6cadcc66 style: Refactor conditional logic in color assignment 2024-08-14 06:22:51 -07:00
Paul Gauthier (aider)
c4f70d81b7 feat: add new color for all "-4o-" models except "gpt-4o-mini" 2024-08-14 06:22:48 -07:00
Paul Gauthier (aider)
1f59687e9d style: Format code with linter 2024-08-14 06:21:48 -07:00
Paul Gauthier (aider)
d8c8c51156 The commit message for these changes would be:
feat: Improve graph visualization and add debugging

The changes made in this commit include:

1. Adjusting the y-axis limit to 100 to accommodate the higher pass rate values.
2. Rotating the x-axis labels for better readability.
3. Adding debug print statements to track the progress of figure generation and display.
4. Increasing the figure size for better visibility.
5. Adding additional debugging to ensure the data is being plotted correctly.

These improvements should help with the visualization and debugging of the graph generation process.
2024-08-14 06:21:45 -07:00
Paul Gauthier (aider)
d94d5aa3fa style: format code according to linter rules 2024-08-14 06:20:36 -07:00
Paul Gauthier (aider)
d2479f30f7 fix: Add debug prints and check for empty data in over_time.py 2024-08-14 06:20:32 -07:00
Paul Gauthier
56975d02a1 fix: Update path to edit_leaderboard.yml file 2024-08-14 06:20:31 -07:00
Paul Gauthier
060c8ff89a override dotenv 2024-08-13 18:06:00 -07:00
Paul Gauthier
139f7992cb do not pass pretty to coder 2024-08-13 17:43:41 -07:00
Your Name
e799e89ff4 install aider with -e in benchmark docker 2024-07-28 17:23:25 -03:00
Paul Gauthier
e9bde684ae updated benchmark docker to use [dev] 2024-07-14 17:24:51 +01:00
Paul Gauthier
a8d4be441e Update benchmark dockerfile 2024-07-11 08:54:47 +01:00
Paul Gauthier
9fa70b4b05 Install the cpu-only version of torch 2024-07-09 09:37:10 +01:00
Paul Gauthier
3ff0c7ce35 copy 2024-06-02 09:28:56 -07:00
Paul Gauthier
2cb9a8ddc8 copy 2024-06-01 16:10:55 -07:00
Paul Gauthier
47a3cb8adf copy 2024-06-01 15:05:29 -07:00
Paul Gauthier
2febc663f3 copy 2024-06-01 14:48:12 -07:00
Paul Gauthier
bc4d39ddf2 copy 2024-06-01 11:46:53 -07:00
Paul Gauthier
26edbcc8f1 copy 2024-06-01 11:26:16 -07:00
Paul Gauthier
fcc62ebffc copy 2024-06-01 07:34:30 -07:00
Paul Gauthier
07d36b22c0 copy 2024-06-01 07:23:25 -07:00
Paul Gauthier
1b539e4466 rename 2024-06-01 07:04:12 -07:00
Paul Gauthier
7623b8e2e6 copy 2024-06-01 07:03:58 -07:00
Paul Gauthier
6a2d7e08c2 copy 2024-05-31 13:32:20 -07:00
Paul Gauthier
2c6b472946 copy 2024-05-31 11:28:38 -07:00
Paul Gauthier
676b644df4 aider: Added more whitespace below the graph to give more room for the note.
# Aider chat conversation:

USER: add more white space below the graph, to give more room for the note

ASSISTANT: Here is the change to add more white space below the graph:

benchmark/swe_bench_lite.py
```python
<<<<<<< SEARCH
    plt.tight_layout(pad=3.0)
=======
    plt.tight_layout(pad=3.0, rect=[0, 0.05, 1, 1])
>>>>>>> REPLACE
```

This adjusts the `tight_layout` call to reserve 5% of the figure height at the bottom for the note, using the `rect` parameter. The `rect` tuple specifies the left, bottom, right and top of the area the subplots should occupy as a fraction of figure width and height.
2024-05-31 11:26:27 -07:00
Paul Gauthier
38ac9de678 use underscores in graph fnames 2024-05-31 09:44:33 -07:00
Paul Gauthier
0120d434ff added draft article 2024-05-31 09:33:31 -07:00
Paul Gauthier
639121bb45 full swe bench graph 2024-05-31 08:16:49 -07:00
Paul Gauthier
20ee2bbf36 aider: Increase white space at the bottom of the plot to make room for the note by adjusting the plt.tight_layout call to include more padding.
# Aider chat conversation:

USER: add some white space at the bottom to make more room for the note

ASSISTANT: To add more white space at the bottom of the plot to make room for the note, we can adjust the `plt.tight_layout` call to include more padding.

Here is the *SEARCH/REPLACE* block:

benchmark/swe_bench_lite.py
```python
<<<<<<< SEARCH
    plt.tight_layout(pad=3.0)
=======
    plt.tight_layout(pad=5.0)
>>>>>>> REPLACE
```
2024-05-31 07:44:28 -07:00
Paul Gauthier
8bcebfa475 Added a note at the bottom of the graph in swe_bench_lite.py. 2024-05-31 07:44:27 -07:00
Paul Gauthier
ffec28cc15 aider: Added note at the bottom of the graph explaining the number of instances processed by the agent.
# Aider chat conversation:

USER: no, add it to the bottom of the graph

ASSISTANT: To add a note at the bottom of the graph explaining that (570) and (2294) refer to the number of instances processed by the agent, we can use the `text` function from Matplotlib to place the note at the bottom of the plot.

Here is the *SEARCH/REPLACE* block to make this change:

benchmark/swe_bench_lite.py
```python
<<<<<<< SEARCH
ax.xaxis.label.set_color(font_color)
=======
ax.xaxis.label.set_color(font_color)

# Add note at the bottom of the graph
note = "(570) and (2294) refer to the number of instances that were processed by the agent."
plt.figtext(0.5, 0.01, note, wrap=True, horizontalalignment='center', fontsize=10, color=font_color)
>>>>>>> REPLACE
```
2024-05-31 07:42:33 -07:00
Paul Gauthier
d017b9b23f aider: Add note explaining that (570) and (2294) refer to the number of instances processed by the agent.
# Aider chat conversation:

USER: add a note at the bottom that explains that (570) and (2294) refer to the number of instances that were processed by the agent

ASSISTANT: Here is the change to add a note at the bottom of the script explaining that (570) and (2294) refer to the number of instances processed by the agent.

benchmark/swe_bench_lite.py
```python
<<<<<<< SEARCH
ax.xaxis.label.set_color(font_color)
=======
ax.xaxis.label.set_color(font_color)

# Note: (570) and (2294) refer to the number of instances that were processed by the agent.
>>>>>>> REPLACE
```
2024-05-31 07:42:18 -07:00