dearwolf/aider
1
0
Fork 0
mirror of https://github.com/Aider-AI/aider.git synced 2025-05-31 01:35:00 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 9

aider: Modified scrape_with_playwright method to wait for network to be idle before extracting content.

Browse source 
# Aider chat conversation:

USER: with playwright enabled, i get: You need to enable JavaScript to run this app.

ASSISTANT: To address the issue of JavaScript not being enabled, we need to ensure that Playwright is properly executing JavaScript on the page. This can be done by waiting for the page to load completely and ensuring that JavaScript is executed.

We will modify the `scrape_with_playwright` method to wait for the network to be idle and ensure that the page is fully loaded before extracting the content.

Here are the changes:

aider/scrape.py
```python
<<<<<<< SEARCH
            page.goto(url)
=======
            page.goto(url, wait_until="networkidle")
>>>>>>> REPLACE
```

This change will ensure that Playwright waits for the network to be idle, which typically means that all JavaScript has been executed and the page is fully loaded.
This commit is contained in:
Paul Gauthier 2024-06-11 06:35:00 -07:00
parent a17b81236c
commit 337891e739
1 changed files with 1 additions and 1 deletions
Download patch file Download diff file Expand all files Collapse all files

2
aider/scrape.py
View file

@ -81,7 +81,7 @@ class Scraper:
user_agent += " " + aider_user_agent
page = browser.new_page(user_agent=user_agent)
page.goto(url)
page.goto(url, wait_until="networkidle")
content = page.content()
browser.close()