Commit graph

4498 commits

Author SHA1 Message Date
Paul Gauthier
5e9bf8993d copy 2024-05-31 09:28:42 -07:00
Paul Gauthier
c76215f302 copy 2024-05-31 08:16:56 -07:00
Paul Gauthier
639121bb45 full swe bench graph 2024-05-31 08:16:49 -07:00
Paul Gauthier
20ee2bbf36 aider: Increase white space at the bottom of the plot to make room for the note by adjusting the plt.tight_layout call to include more padding.
# Aider chat conversation:

USER: add some white space at the bottom to make more room for the note

ASSISTANT: To add more white space at the bottom of the plot to make room for the note, we can adjust the `plt.tight_layout` call to include more padding.

Here is the *SEARCH/REPLACE* block:

benchmark/swe_bench_lite.py
```python
<<<<<<< SEARCH
    plt.tight_layout(pad=3.0)
=======
    plt.tight_layout(pad=5.0)
>>>>>>> REPLACE
```
2024-05-31 07:44:28 -07:00
Paul Gauthier
8bcebfa475 Added a note at the bottom of the graph in swe_bench_lite.py. 2024-05-31 07:44:27 -07:00
Paul Gauthier
ffec28cc15 aider: Added note at the bottom of the graph explaining the number of instances processed by the agent.
# Aider chat conversation:

USER: no, add it to the bottom of the graph

ASSISTANT: To add a note at the bottom of the graph explaining that (570) and (2294) refer to the number of instances processed by the agent, we can use the `text` function from Matplotlib to place the note at the bottom of the plot.

Here is the *SEARCH/REPLACE* block to make this change:

benchmark/swe_bench_lite.py
```python
<<<<<<< SEARCH
ax.xaxis.label.set_color(font_color)
=======
ax.xaxis.label.set_color(font_color)

# Add note at the bottom of the graph
note = "(570) and (2294) refer to the number of instances that were processed by the agent."
plt.figtext(0.5, 0.01, note, wrap=True, horizontalalignment='center', fontsize=10, color=font_color)
>>>>>>> REPLACE
```
2024-05-31 07:42:33 -07:00
Paul Gauthier
d017b9b23f aider: Add note explaining that (570) and (2294) refer to the number of instances processed by the agent.
# Aider chat conversation:

USER: add a note at the bottom that explains that (570) and (2294) refer to the number of instances that were processed by the agent

ASSISTANT: Here is the change to add a note at the bottom of the script explaining that (570) and (2294) refer to the number of instances processed by the agent.

benchmark/swe_bench_lite.py
```python
<<<<<<< SEARCH
ax.xaxis.label.set_color(font_color)
=======
ax.xaxis.label.set_color(font_color)

# Note: (570) and (2294) refer to the number of instances that were processed by the agent.
>>>>>>> REPLACE
```
2024-05-31 07:42:18 -07:00
Paul Gauthier
4d02a8aa58 Refactored plot_swe_bench_lite function to dynamically handle file names and titles. 2024-05-31 07:42:17 -07:00
Paul Gauthier
aa1e34982d corrected 2024-05-31 05:54:25 -07:00
Paul Gauthier
ff49ff9122 copy 2024-05-31 05:21:01 -07:00
Paul Gauthier
ec26d5f783 copy 2024-05-30 16:57:18 -07:00
Paul Gauthier
6966936316 Updated graph to use pass@1 unhinted results from other agents 2024-05-30 15:29:33 -07:00
Paul Gauthier
07731e30dc copy 2024-05-30 09:52:47 -07:00
Paul Gauthier
d5ca9b3696 updated autocoderover results 2024-05-30 09:45:02 -07:00
Paul Gauthier
b8b207f4de fixed broken link 2024-05-30 09:38:11 -07:00
Paul Gauthier
ad2312ad7b handle missing end_lineno attr #619 2024-05-29 15:55:55 -07:00
Paul Gauthier
0da01dacb6 ssh 2024-05-29 15:51:29 -07:00
Paul Gauthier
faf15188c5 copy 2024-05-25 18:58:42 -07:00
Paul Gauthier
17dcfd94f7 +Only include the files 2024-05-25 18:58:37 -07:00
paul-gauthier
1e0c770db9
Update 2024-05-22-swe-bench-lite.md 2024-05-25 18:13:51 -07:00
Paul Gauthier
61066f2bb5 copy 2024-05-25 14:01:27 -07:00
Paul Gauthier
50426e62da copy 2024-05-25 12:47:47 -07:00
Paul Gauthier
3c970e0fb7 copy 2024-05-25 12:19:43 -07:00
Paul Gauthier
689786a875 copy 2024-05-25 12:14:48 -07:00
Paul Gauthier
6382153597 copy 2024-05-24 16:00:17 -07:00
Paul Gauthier
0f84265257 added refs 2024-05-24 15:21:57 -07:00
Paul Gauthier
c86e258551 added proper data 2024-05-24 15:05:31 -07:00
Paul Gauthier
0e71d262e3 change error message 2024-05-24 13:34:51 -07:00
Paul Gauthier
d48edfcd00 fix missing env 2024-05-24 12:59:48 -07:00
Paul Gauthier
d41bf592a0 Set GIT_EDITOR environment variable only if it is not already set. 2024-05-24 12:43:53 -07:00
Paul Gauthier
9aee429657 copy 2024-05-24 10:28:30 -07:00
Paul Gauthier
c5e1b19ac6 copy 2024-05-24 10:24:21 -07:00
Paul Gauthier
af67f61d9f copy 2024-05-24 10:20:27 -07:00
Paul Gauthier
f2702cd685 ignore whitespace 2024-05-24 10:15:46 -07:00
Paul Gauthier
a153b11d57 copy 2024-05-24 10:09:22 -07:00
Paul Gauthier
112559dd9f copy 2024-05-24 10:06:29 -07:00
Paul Gauthier
14dd1f7137 copy 2024-05-24 10:02:21 -07:00
Paul Gauthier
de1c4cfb9e added blog post 2024-05-24 09:59:59 -07:00
Paul Gauthier
d231ff0905 aider: Update show_commit_stats to correctly extract the number of insertions and deletions from the lines 1 file changed, 20 insertions(+), 1 deletion(-).
# Aider chat conversation:

USER: you need to parse these lines `1 file changed, 20 insertions(+), 1 deletion(-)`

ASSISTANT: To parse the lines `1 file changed, 20 insertions(+), 1 deletion(-)`, we need to update the `show_commit_stats` function to correctly extract the number of insertions and deletions from these lines.

Here are the *SEARCH/REPLACE* blocks:

scripts/blame.py
```python
<<<<<<< SEARCH
            if "insertion" in line:
                added_lines += int(line.split()[0])
            if "deletion" in line:
                deleted_lines += int(line.split()[0])
=======
            if "insertion" in line or "insertion(+)" in line:
                added_lines += int(line.split()[line.split().index("insertion(+)") - 1])
            if "deletion" in line or "deletion(-)" in line:
                deleted_lines += int(line.split()[line.split().index("deletion(-)") - 1])
>>>>>>> REPLACE
```
2024-05-24 09:46:34 -07:00
Paul Gauthier
aa75d54d4e Add debug statement to show each line processed in show_commit_stats function. 2024-05-24 09:46:32 -07:00
Paul Gauthier
fb717b8786 aider: Refactored show_commit_stats to print out lines added and deleted for each commit and report totals.
# Aider chat conversation:

USER: do the todp

ASSISTANT: To implement the TODO, we need to modify the `show_commit_stats` function to print out the number of lines added and deleted for each commit and report the totals.

Here are the *SEARCH/REPLACE* blocks:

scripts/blame.py
```python
<<<<<<< SEARCH
    # TODO: print out the number of lines added and deleted for each commit
    # report totals
    pass
=======
    total_added_lines = 0
    total_deleted_lines = 0

    for commit in commits:
        result = subprocess.run(
            ["git", "show", "--stat", "--oneline", commit],
            capture_output=True,
            text=True,
            check=True
        )

        added_lines = 0
        deleted_lines = 0
        for line in result.stdout.splitlines():
            if "insertion" in line:
                added_lines += int(line.split()[0])
            if "deletion" in line:
                deleted_lines += int(line.split()[0])

        total_added_lines += added_lines
        total_deleted_lines += deleted_lines

        print(f"Commit {commit}: +{added_lines} -{deleted_lines}")

    print(f"Total: +{total_added_lines} -{total_deleted_lines}")
>>>>>>> REPLACE
```
2024-05-24 09:43:33 -07:00
Paul Gauthier
a4694511a0 Updated get_lines_with_commit_hash to set verbose parameter to False, added show_commit_stats function, and called show_commit_stats in process_fnames. 2024-05-24 09:43:31 -07:00
Paul Gauthier
26210cfe6b aider: Import imgcat and display the image using the appropriate function call.
# Aider chat conversation:

USER: import imgcat and show it that way

ASSISTANT: To import `imgcat` and use it to display the image, we need to:

1. Import `imgcat` at the beginning of the script.
2. Replace the `subprocess.run(['imgcat', 'aider_plot.png'])` line with the appropriate `imgcat` function call.

Here are the *SEARCH/REPLACE* blocks:

scripts/blame.py
```python
<<<<<<< SEARCH
from aider.dump import dump
=======
from aider.dump import dump
from imgcat import imgcat
>>>>>>> REPLACE
```

scripts/blame.py
```python
<<<<<<< SEARCH
    subprocess.run(['imgcat', 'aider_plot.png'])
=======
    with open('aider_plot.png', 'rb') as f:
        imgcat(f.read())
>>>>>>> REPLACE
```
2024-05-24 09:36:26 -07:00
Paul Gauthier
46baf0cc18 aider: Save the plot as a PNG file and display it using imgcat.
# Aider chat conversation:

USER: save it as a png and imgcat it

ASSISTANT: To save the plot as a PNG file and display it using `imgcat`, we need to:

1. Save the plot as a PNG file.
2. Use `imgcat` to display the image.

Here are the *SEARCH/REPLACE* blocks:

scripts/blame.py
```python
<<<<<<< SEARCH
    plt.show()
=======
    plt.savefig('aider_plot.png')
    subprocess.run(['imgcat', 'aider_plot.png'])
>>>>>>> REPLACE
```
2024-05-24 09:36:14 -07:00
Paul Gauthier
3bba6ec58e Updated get_lines_with_commit_hash to set verbose to True by default and adjusted the iteration logic in history function. 2024-05-24 09:36:12 -07:00
Paul Gauthier
eff8aea116 aider: Plot the results using pylab, importing it at the beginning of the script and modifying the history function to display an area graph of aider_lines and total_lines over time.
# Aider chat conversation:

USER: is pylab to plot the results. x-axis is commit number. area graph showing aider_lines and total_lines over time

ASSISTANT: To plot the results using `pylab`, we need to:

1. Import `pylab` at the beginning of the script.
2. Modify the `history` function to collect the data and plot it.

Here are the *SEARCH/REPLACE* blocks:

scripts/blame.py
```python
<<<<<<< SEARCH
from pathlib import Path
=======
from pathlib import Path
import pylab as plt
>>>>>>> REPLACE
```

scripts/blame.py
```python
<<<<<<< SEARCH
    dump(results)
=======
    dump(results)

    # Plotting the results
    x = [i for i, _, _, _ in results]
    aider_lines = [aider_lines for _, aider_lines, _, _ in results]
    total_lines = [total_lines for _, _, total_lines, _ in results]

    plt.fill_between(x, aider_lines, label='Aider Lines', color='skyblue', alpha=0.5)
    plt.fill_between(x, total_lines, label='Total Lines', color='lightgreen', alpha=0.5)
    plt.xlabel('Commit Number')
    plt.ylabel('Lines of Code')
    plt.title('Aider Lines and Total Lines Over Time')
    plt.legend()
    plt.show()
>>>>>>> REPLACE
```
2024-05-24 09:27:07 -07:00
Paul Gauthier
a399b99d49 Refactored the blame script to include a new function for analyzing commit history. 2024-05-24 09:27:05 -07:00
Paul Gauthier
3e9873cc11 Update git commands in blame.py to accept a git directory name as an argument. 2024-05-24 09:04:50 -07:00
Paul Gauthier
1cef96ce16 Refactored process function to process_fnames and added process_repo to handle repository level processing. 2024-05-24 09:04:41 -07:00
Paul Gauthier
293cc6c684 aider: Update main function to list and filter Python files if no filenames are provided.
# Aider chat conversation:

USER: if no files are provided on cmd line, do git ls-files and filter to python files

ASSISTANT: To implement the requested change, we need to modify the `main` function to use `git ls-files` to list all files in the repository and filter them to include only Python files if no filenames are provided on the command line.

Here is the *SEARCH/REPLACE* block:

```python
scripts/blame.py
```python
<<<<<<< SEARCH
    if len(sys.argv) < 2:
        print(f"Usage: {sys.argv[0]} <filename> ...")
        sys.exit(1)
=======
    if len(sys.argv) < 2:
        result = subprocess.run(
            ["git", "ls-files"],
            capture_output=True,
            text=True,
            check=True
        )
        fnames = [fname for fname in result.stdout.splitlines() if fname.endswith('.py')]
        if not fnames:
            print("No Python files found in the repository.")
            sys.exit(1)
    else:
        fnames = sys.argv[1:]
>>>>>>> REPLACE
```

This change ensures that if no filenames are provided on the command line, the script will list all files in the repository, filter them to include only Python files, and then process those files.
2024-05-24 08:41:23 -07:00