Commit graph

4400 commits

Author SHA1 Message Date
Paul Gauthier
0120d434ff added draft article 2024-05-31 09:33:31 -07:00
Paul Gauthier
a168daf5fc Merge branch 'main' into both-swe-bench 2024-05-31 09:28:53 -07:00
Paul Gauthier
5e9bf8993d copy 2024-05-31 09:28:42 -07:00
Paul Gauthier
c76215f302 copy 2024-05-31 08:16:56 -07:00
Paul Gauthier
639121bb45 full swe bench graph 2024-05-31 08:16:49 -07:00
Paul Gauthier
20ee2bbf36 aider: Increase white space at the bottom of the plot to make room for the note by adjusting the plt.tight_layout call to include more padding.
# Aider chat conversation:

USER: add some white space at the bottom to make more room for the note

ASSISTANT: To add more white space at the bottom of the plot to make room for the note, we can adjust the `plt.tight_layout` call to include more padding.

Here is the *SEARCH/REPLACE* block:

benchmark/swe_bench_lite.py
```python
<<<<<<< SEARCH
    plt.tight_layout(pad=3.0)
=======
    plt.tight_layout(pad=5.0)
>>>>>>> REPLACE
```
2024-05-31 07:44:28 -07:00
Paul Gauthier
8bcebfa475 Added a note at the bottom of the graph in swe_bench_lite.py. 2024-05-31 07:44:27 -07:00
Paul Gauthier
ffec28cc15 aider: Added note at the bottom of the graph explaining the number of instances processed by the agent.
# Aider chat conversation:

USER: no, add it to the bottom of the graph

ASSISTANT: To add a note at the bottom of the graph explaining that (570) and (2294) refer to the number of instances processed by the agent, we can use the `text` function from Matplotlib to place the note at the bottom of the plot.

Here is the *SEARCH/REPLACE* block to make this change:

benchmark/swe_bench_lite.py
```python
<<<<<<< SEARCH
ax.xaxis.label.set_color(font_color)
=======
ax.xaxis.label.set_color(font_color)

# Add note at the bottom of the graph
note = "(570) and (2294) refer to the number of instances that were processed by the agent."
plt.figtext(0.5, 0.01, note, wrap=True, horizontalalignment='center', fontsize=10, color=font_color)
>>>>>>> REPLACE
```
2024-05-31 07:42:33 -07:00
Paul Gauthier
d017b9b23f aider: Add note explaining that (570) and (2294) refer to the number of instances processed by the agent.
# Aider chat conversation:

USER: add a note at the bottom that explains that (570) and (2294) refer to the number of instances that were processed by the agent

ASSISTANT: Here is the change to add a note at the bottom of the script explaining that (570) and (2294) refer to the number of instances processed by the agent.

benchmark/swe_bench_lite.py
```python
<<<<<<< SEARCH
ax.xaxis.label.set_color(font_color)
=======
ax.xaxis.label.set_color(font_color)

# Note: (570) and (2294) refer to the number of instances that were processed by the agent.
>>>>>>> REPLACE
```
2024-05-31 07:42:18 -07:00
Paul Gauthier
4d02a8aa58 Refactored plot_swe_bench_lite function to dynamically handle file names and titles. 2024-05-31 07:42:17 -07:00
Paul Gauthier
aa1e34982d corrected 2024-05-31 05:54:25 -07:00
Paul Gauthier
ff49ff9122 copy 2024-05-31 05:21:01 -07:00
Paul Gauthier
ec26d5f783 copy 2024-05-30 16:57:18 -07:00
Paul Gauthier
6966936316 Updated graph to use pass@1 unhinted results from other agents 2024-05-30 15:29:33 -07:00
Paul Gauthier
07731e30dc copy 2024-05-30 09:52:47 -07:00
Paul Gauthier
d5ca9b3696 updated autocoderover results 2024-05-30 09:45:02 -07:00
Paul Gauthier
b8b207f4de fixed broken link 2024-05-30 09:38:11 -07:00
Paul Gauthier
ad2312ad7b handle missing end_lineno attr #619 2024-05-29 15:55:55 -07:00
Paul Gauthier
0da01dacb6 ssh 2024-05-29 15:51:29 -07:00
Paul Gauthier
faf15188c5 copy 2024-05-25 18:58:42 -07:00
Paul Gauthier
17dcfd94f7 +Only include the files 2024-05-25 18:58:37 -07:00
paul-gauthier
1e0c770db9
Update 2024-05-22-swe-bench-lite.md 2024-05-25 18:13:51 -07:00
Paul Gauthier
61066f2bb5 copy 2024-05-25 14:01:27 -07:00
Paul Gauthier
50426e62da copy 2024-05-25 12:47:47 -07:00
Paul Gauthier
3c970e0fb7 copy 2024-05-25 12:19:43 -07:00
Paul Gauthier
689786a875 copy 2024-05-25 12:14:48 -07:00
Paul Gauthier
6382153597 copy 2024-05-24 16:00:17 -07:00
Paul Gauthier
0f84265257 added refs 2024-05-24 15:21:57 -07:00
Paul Gauthier
c86e258551 added proper data 2024-05-24 15:05:31 -07:00
Paul Gauthier
0e71d262e3 change error message 2024-05-24 13:34:51 -07:00
Paul Gauthier
d48edfcd00 fix missing env 2024-05-24 12:59:48 -07:00
Paul Gauthier
d41bf592a0 Set GIT_EDITOR environment variable only if it is not already set. 2024-05-24 12:43:53 -07:00
Paul Gauthier
9aee429657 copy 2024-05-24 10:28:30 -07:00
Paul Gauthier
c5e1b19ac6 copy 2024-05-24 10:24:21 -07:00
Paul Gauthier
af67f61d9f copy 2024-05-24 10:20:27 -07:00
Paul Gauthier
f2702cd685 ignore whitespace 2024-05-24 10:15:46 -07:00
Paul Gauthier
a153b11d57 copy 2024-05-24 10:09:22 -07:00
Paul Gauthier
112559dd9f copy 2024-05-24 10:06:29 -07:00
Paul Gauthier
14dd1f7137 copy 2024-05-24 10:02:21 -07:00
Paul Gauthier
de1c4cfb9e added blog post 2024-05-24 09:59:59 -07:00
Paul Gauthier
d231ff0905 aider: Update show_commit_stats to correctly extract the number of insertions and deletions from the lines 1 file changed, 20 insertions(+), 1 deletion(-).
# Aider chat conversation:

USER: you need to parse these lines `1 file changed, 20 insertions(+), 1 deletion(-)`

ASSISTANT: To parse the lines `1 file changed, 20 insertions(+), 1 deletion(-)`, we need to update the `show_commit_stats` function to correctly extract the number of insertions and deletions from these lines.

Here are the *SEARCH/REPLACE* blocks:

scripts/blame.py
```python
<<<<<<< SEARCH
            if "insertion" in line:
                added_lines += int(line.split()[0])
            if "deletion" in line:
                deleted_lines += int(line.split()[0])
=======
            if "insertion" in line or "insertion(+)" in line:
                added_lines += int(line.split()[line.split().index("insertion(+)") - 1])
            if "deletion" in line or "deletion(-)" in line:
                deleted_lines += int(line.split()[line.split().index("deletion(-)") - 1])
>>>>>>> REPLACE
```
2024-05-24 09:46:34 -07:00
Paul Gauthier
aa75d54d4e Add debug statement to show each line processed in show_commit_stats function. 2024-05-24 09:46:32 -07:00
Paul Gauthier
fb717b8786 aider: Refactored show_commit_stats to print out lines added and deleted for each commit and report totals.
# Aider chat conversation:

USER: do the todp

ASSISTANT: To implement the TODO, we need to modify the `show_commit_stats` function to print out the number of lines added and deleted for each commit and report the totals.

Here are the *SEARCH/REPLACE* blocks:

scripts/blame.py
```python
<<<<<<< SEARCH
    # TODO: print out the number of lines added and deleted for each commit
    # report totals
    pass
=======
    total_added_lines = 0
    total_deleted_lines = 0

    for commit in commits:
        result = subprocess.run(
            ["git", "show", "--stat", "--oneline", commit],
            capture_output=True,
            text=True,
            check=True
        )

        added_lines = 0
        deleted_lines = 0
        for line in result.stdout.splitlines():
            if "insertion" in line:
                added_lines += int(line.split()[0])
            if "deletion" in line:
                deleted_lines += int(line.split()[0])

        total_added_lines += added_lines
        total_deleted_lines += deleted_lines

        print(f"Commit {commit}: +{added_lines} -{deleted_lines}")

    print(f"Total: +{total_added_lines} -{total_deleted_lines}")
>>>>>>> REPLACE
```
2024-05-24 09:43:33 -07:00
Paul Gauthier
a4694511a0 Updated get_lines_with_commit_hash to set verbose parameter to False, added show_commit_stats function, and called show_commit_stats in process_fnames. 2024-05-24 09:43:31 -07:00
Paul Gauthier
26210cfe6b aider: Import imgcat and display the image using the appropriate function call.
# Aider chat conversation:

USER: import imgcat and show it that way

ASSISTANT: To import `imgcat` and use it to display the image, we need to:

1. Import `imgcat` at the beginning of the script.
2. Replace the `subprocess.run(['imgcat', 'aider_plot.png'])` line with the appropriate `imgcat` function call.

Here are the *SEARCH/REPLACE* blocks:

scripts/blame.py
```python
<<<<<<< SEARCH
from aider.dump import dump
=======
from aider.dump import dump
from imgcat import imgcat
>>>>>>> REPLACE
```

scripts/blame.py
```python
<<<<<<< SEARCH
    subprocess.run(['imgcat', 'aider_plot.png'])
=======
    with open('aider_plot.png', 'rb') as f:
        imgcat(f.read())
>>>>>>> REPLACE
```
2024-05-24 09:36:26 -07:00
Paul Gauthier
46baf0cc18 aider: Save the plot as a PNG file and display it using imgcat.
# Aider chat conversation:

USER: save it as a png and imgcat it

ASSISTANT: To save the plot as a PNG file and display it using `imgcat`, we need to:

1. Save the plot as a PNG file.
2. Use `imgcat` to display the image.

Here are the *SEARCH/REPLACE* blocks:

scripts/blame.py
```python
<<<<<<< SEARCH
    plt.show()
=======
    plt.savefig('aider_plot.png')
    subprocess.run(['imgcat', 'aider_plot.png'])
>>>>>>> REPLACE
```
2024-05-24 09:36:14 -07:00
Paul Gauthier
3bba6ec58e Updated get_lines_with_commit_hash to set verbose to True by default and adjusted the iteration logic in history function. 2024-05-24 09:36:12 -07:00
Paul Gauthier
eff8aea116 aider: Plot the results using pylab, importing it at the beginning of the script and modifying the history function to display an area graph of aider_lines and total_lines over time.
# Aider chat conversation:

USER: is pylab to plot the results. x-axis is commit number. area graph showing aider_lines and total_lines over time

ASSISTANT: To plot the results using `pylab`, we need to:

1. Import `pylab` at the beginning of the script.
2. Modify the `history` function to collect the data and plot it.

Here are the *SEARCH/REPLACE* blocks:

scripts/blame.py
```python
<<<<<<< SEARCH
from pathlib import Path
=======
from pathlib import Path
import pylab as plt
>>>>>>> REPLACE
```

scripts/blame.py
```python
<<<<<<< SEARCH
    dump(results)
=======
    dump(results)

    # Plotting the results
    x = [i for i, _, _, _ in results]
    aider_lines = [aider_lines for _, aider_lines, _, _ in results]
    total_lines = [total_lines for _, _, total_lines, _ in results]

    plt.fill_between(x, aider_lines, label='Aider Lines', color='skyblue', alpha=0.5)
    plt.fill_between(x, total_lines, label='Total Lines', color='lightgreen', alpha=0.5)
    plt.xlabel('Commit Number')
    plt.ylabel('Lines of Code')
    plt.title('Aider Lines and Total Lines Over Time')
    plt.legend()
    plt.show()
>>>>>>> REPLACE
```
2024-05-24 09:27:07 -07:00
Paul Gauthier
a399b99d49 Refactored the blame script to include a new function for analyzing commit history. 2024-05-24 09:27:05 -07:00
Paul Gauthier
3e9873cc11 Update git commands in blame.py to accept a git directory name as an argument. 2024-05-24 09:04:50 -07:00