The Scrapy Playwright Tutorial (2026)

Many citizenry assume that once you cansend requests and parse HTML with Scrapy, you can scrape any website reliably. It feels unproblematic, predictable, and sufficient for most tasks.

However, when I wasscraping a job list website, I noticed that somelistingsnever appear in myScrapy spider. While the page initiallycharge in the browser, the content was dynamicand only appeared afterscrolling or applying filters.

That & # 8217; s when I realized thatstandard Scrapycould not coverdynamic content, and my scriptsbe silently missingimportant data.

To solve this, I become to. By combining it withScrapy, I could finallyinteract with dynamic pages, delay for message to charge, and scratching datathat was previouslyinvisible.

pip establish scrapy-playwrightplaywright install

In this tutorial, I & # 8217; ll establish youstep by stephow to useScrapy Playwright to scrape modern websites reliably, include handlingdynamic substance, infinite scrolling, AJAX shout, and more.

What is Scrapy Playwright?

Scrapy is a framework used for extract data from site. It is project to crawl still pages, follow links across a situation, and export structure data in formats such as CSV, JSON, or databases.

Scrapy Playwright is an consolidation of Scrapy with Playwright. It permit Scrapy spiders to interact with modern web pages that use JavaScript, AJAX, or dynamic contented loading.

Unlike standard Scrapy, which only handles static HTML, Scrapy Playwright can wait for message to appear, fulfil scripts, and perform user-like interactions such as clicking buttons, fill forms, and scrolling Page.

Why You Should Use Scrapy Playwright?

Scrapy Playwright combines the best of both worlds. It extends Scrapy & # 8217; s power to handle modernistic, active web pages reliably.

Here & # 8217; s why to use Scrapy Playwright:

• Dynamic Content Rendering: Scrapy Playwright can detect and wait for elements rendered after JavaScript execution to ascertain your spider capture data that would otherwise be invisible.
• Precise Interaction Control: Beyond clicking buttons or occupy form, it allows conditional actions, such as spark events only when certain page states or data conditions are met.

• Session and Context Management: Maintain multiple browser context with independent cookies and storage, enable multi-user simulations or session-specific scratching without information pollution.

• Optimized : Abort unnecessary requests, control resourcefulness burden, and reduce bandwidth usage for quicker scratch of JavaScript-heavy sites.
• Advanced and Visibility: Capture detailed page logs, screenshots, and videos during scraping sessions to debug dynamic content or unexpected page behavior effectively.

• Infinite Scrolling and Handling: Automatically scroll or trigger events to load additional message while obviate redundant page reloads, control completeness without overload host.
• Custom JavaScript Execution: Inject scripts to cook or pull datum in slipway that traditional selectors can not handle, offering flexibility for extremely dynamical or synergistic sites.

Prerequisites to Install Scrapy Playwright

Before you start, you need to ready your surround. Ensuring the right frame-up will save time and prevent common error.

1. Python Installation

Scrapy Playwright requiresPython 3.7 or higher. If Python is not installed, download it from the official site. Verify the installation with:

python & # 8211; adaptation

2. Install Scrapy

Scrapy cater the core crawling and parsing capacity. Install it habituate pip:

pip install scrapy

Confirm the installation by lam:

scrapy version

3. Install Playwright and Scrapy Playwright

Playwright powerfulness browser automation for active substance. Install both Scrapy Playwright and the required browser binaries:

pip instal scrapy-playwrightplaywright install

The playwright install command downloads browser engines (Chromium, Firefox, WebKit) so your spiders can simulate real user interaction.

4. Optional: Virtual Environment

It is recommended to use aPython virtual environsto manage dependencies and avoid conflicts:

python -m venv venvsource venv/bin/activate # Linux/macOS
venvScriptsactivate # Windows

This ensures a clean environment for your Scrapy Playwright project.

Core Scraping Operations with Scrapy Playwright

Once your environment is ready, it & # 8217; s clip to start scrape. Scrapy Playwright combines Scrapy & # 8217; s crawling power with Playwright & # 8217; s browser automation, allowing you to interact with dynamic web pages as if you were a real user.

Below are the core use cases of Scrapy Playwright along with how to do it.

1. Opening a URL

In a standard Scrapy workflow, opening a URL means sending an and immediately parsing the returned response. This works well when the server deliver complete HTML that already control the datum of interest.

However, many mod websites do not revert fully populated HTML responses. Instead, the initial response often hold exclusively a basic page structure, while the literal content is loaded later through JavaScript. Because Scrapy does not fulfill client-side codification, it parse the page too early and ne'er sees the rendered data.

Scrapy Playwright changes this doings by opening the URL inside a existent browser. This grant JavaScript to execute and network postulation to complete before Scrapy begins parse.

importation scrapyclass ExampleSpider (scrapy.Spider):
gens = & # 8220; representative & # 8221;
def start_requests (self):
yield scrapy.Request (
url= & # 8221; https: //example.com & # 8221;,
meta= {& # 8220; playwright & # 8221;: True}

)

In this representative, the request is flagged to use Playwright. Scrapy waits until the browser polish rendering the page before passing the reaction to the wanderer. When data becomes available without changing selectors, it usually indicates that rendering, not crawling logic, was the absent step.

2. Extracting Text from Elements

Extracting schoolbook is unremarkably the simplest part of scraping. With standard Scrapy, once a response is receive, the HTML is assumed to be complete and ready to parse. However, this premiss separate on dynamic pages.

On many modern websites, the initial HTML loads foremost, while key elements appear simply after JavaScript complete escape or after additional network request complete. Because of this, the can exist in an incomplete state when descent starts.

Scrapy Playwright solves this by furnish the page in a browser before passing the response to Scrapy. This means the DOM you act with already includes dynamically loaded elements, not just the initial HTML cuticle.

The follow codeafter the page has been rendered:

async def parse (self, answer): title = response.css (& # 8220; h1: :text & # 8221;) .get ()
salary = response.css (& # 8220; .salary: :text & # 8221;) .get ()
output {& # 8220; title & # 8221;: rubric, & # 8220; salary & # 8221;: salary}

In this illustration, the parse method runs only after Playwright finishes laden the page. The selectors query the final DOM province, which includes elements created by JavaScript. The extracted value are then returned as structured data.

However, descent can yet neglect if it starts before the page hit a stable state. This usually shows up as miss value or partially populated fields, even though the data is seeable in the browser. When this hap, it indicates that extra waits or load-state checks are required before extracting datum.

3. Automating Form Interactions

Many website expose information only after user interaction. Search boxes, filters, and controls often trip JavaScript event rather than URL-based navigation. With standard Scrapy, these interactions can not be sham because no browser circumstance subsist.

SUSA automates exploratory testing with persona-driven behavior, catching bugs that scripted automation misses.

Because of this, try to modify query parameters or follow links often neglect to reproduce the doings see in the browser.

Scrapy Playwright enables spider to interact with page elements directly inside the browser, allow form-driven workflow to be automated reliably.

async def parse (self, response): page = response.meta [& # 8220; playwright_page & # 8221;]
await page.fill (& # 8220; input [name= & # 8217; keyword & # 8217;] & # 8221;, & # 8220; engineer & # 8221;)
await page.click (& # 8220; button [type= & # 8217; submit & # 8217;] & # 8221;)
await page.wait_for_load_state (& # 8220; networkidle & # 8221;)

Here, the spider occupy an remark battlefield, triggers a submit action, and waits until network activeness settles. This sequence mirrors how a existent exploiter interacts with the page. When solvent do not appear after submission, the issue is usually associate to missing waits or incomplete navigation rather than incorrect selectors.

4. Capturing Screenshots

When scraping dynamic websites, missing data is not invariably caused by faulty logic. Pages may betray taciturnly due to time issue, blocked resource, or incomplete interactions. Logs unaccompanied often do not discover what the browser actually furnish.

Scrapy Playwright allows screenshots to be captured at any point during execution, providing visual confirmation of the page state.

async def parse (self, response): page = response.meta [& # 8220; playwright_page & # 8221;]
await page.screenshot (path= & # 8221; page.png & # 8221;)

This captures the current browser scene exactly as Playwright sees it. When extracted datum does not correspond expectations, screenshots help sustain whether the page attain the intended province before parsing commence.

Scraping Dynamic and JavaScript-heavy Websites

Modern web applications trust heavily on JavaScript to fetch data, update layouts, and render content after the initial page load. Scraping these pages requires more than downloading HTML because critical information much appears only after hand execute.

5. Scraping JavaScript-rendered Pages

In a standard Scrapy crawl, the response represents exclusively what the server returns. This approach works when Page are server-rendered, but it miscarry when the visible content is injected by JavaScript after the page loads.

However, many mod websites return an almost empty-bellied HTML document and rely on client-side frameworks to fetch information and render constituent. When Scrapy parse such responses, chooser regress empty or uncompleted results because the DOM ne'er reaches its final state.

Scrapy Playwright resolves this by loading the page in a real browser surroundings before Scrapy processes the response. JavaScript action fully, network requests complete, and the DOM reflects what a user really sees.

fruit scrapy.Request (url= & # 8221; https: //example.com & # 8221;,
meta= {& # 8220; playwright & # 8221;: True}

)

This constellation assure that Scrapy get a rendered response instead of a raw HTML shell. When previously missing ingredient become accessible without changing selector, it indicates that JavaScript rendering was the blocking factor.

6. Waiting for Dynamic Elements

Even after JavaScript performance begin, individual elements may still load asynchronously based on user action, API responses, or stay scripts. Extracting data as soon as the page opens often results in missing thickening because the target element are not yet attached to the DOM.

Scrapy Playwright allows spider to wait explicitly for specific factor before parsing starts, which align extraction with the actual handiness of data.

takings scrapy.Request (url= & # 8221; https: //example.com & # 8221;,

meta={
& # 8220; playwright & # 8221;: True,
& # 8220; playwright_page_methods & # 8221;: [
PageMethod (& # 8220; wait_for_selector & # 8221;, & # 8220; .job-card & # 8221;)

]

}

)

This approach guarantee that parsing begins exclusively after the required elements exist. When selectors intermittently return None despite right logic, it usually indicates that the page structure was accessed before dynamical components finished provide.

7. Waiting for Page Load States

Some pages seem visually complete while still performing background employment such as fetching data, hydrate components, or update layouts. Relying exclusively on element front can be misleading because the page may still be transition between states.

provides finer control over when Scrapy should proceed.

PageMethod (& # 8220; wait_for_load_state & # 8221;, & # 8220; networkidle & # 8221;)

Waiting for a stable load state reduce the risk of scratch transient DOM structures that change moments after. When extracted value alter across runs without changes in selector, delayed network activity is often the underlying cause.

8. Waiting for a Specific Amount of Time

In certain cases, pages trigger delayed animations, polling-based updates, or timed content injections that do not expose reliable selectors or load signal. While expressed postponement are not ideal, controlled delays can act as a disengagement when no deterministic condition exists.

Scrapy Playwright allows timed waits to fit such conduct.

PageMethod (& # 8220; wait_for_timeout & # 8221;, 3000)

This technique should be expend sparingly because it introduce fixed latency and reduces efficiency. When no picker or load state consistently sign readiness, timed waits can stabilize extraction but should be combined with retries or validation logic to forfend masking deeper number.

9. Capturing AJAX Data

Many modern websites laden critical data through ground API ring rather than implant it directly in the HTML. In these cases, scrape the rendered DOM may work, but it often adds unneeded complexity and increase the risk of parsing frail markup.

Scrapy Playwright can intercept network traffic and capture AJAX responses directly, which allows spiders to extract structured information at its source.

PageMethod (& # 8220; route & # 8221;, & # 8220; * * /api/jobs * & # 8221;, lambda route, request: route.continue_ ())

By monitoring specific endpoints, it becomes possible to parse clean JSON reply instead of swear on UI-level selectors. When DOM extraction feels brittle or breaks after minor UI modification, the fundamental issue is oftentimes that the data originates from an API instead than the page itself.

10. Running Custom JavaScript Code

Some scraping scenarios require logic that can not be expressed through selectors alone. Pages may compute value client-side, metamorphose text dynamically, or disclose information exclusively through JavaScript variables.

Scrapy Playwright allows performance of custom JavaScript within the page circumstance, which enable direct access to in-memory data and browser APIs.

PageMethod (& # 8220; evaluate & # 8221;,
& # 8220; document.querySelectorAll (& # 8216; .job-card & # 8217;) .length & # 8221;

)

Executing JavaScript inside the browser removes the guesswork involved in reverse-engineering rendered yield. When extracted values appear inconsistent with what is seeable in the browser, it often show that the terminal data exists only within the JavaScript runtime.

11. Scrolling Infinite Pages

Infinite scrolling replaces traditional paging with dynamic message load triggered by scroll events. Scraping only the initial viewport outcome in incomplete datasets because additional items load progressively as the user ringlet.

Scrapy Playwright can sham scrolling behavior to trigger content loading repeatedly.

PageMethod (& # 8220; evaluate & # 8221;,
& # 8220; window.scrollBy (0, document.body.scrollHeight) & # 8221;

)

Scrolling must be paired with waiting to allow new message to load before continuing. When wanderer return a fixed number of records disregarding of page size, it typically means that scroll-driven requests were never trigger.

Optimizing Scrapy Playwright for Large-scale Scraping

As scraping workloads grow, habituate Playwright inside Scrapy require calculated optimization. Browser contexts, Page, and network asking can easily become bottlenecks if they are not deal carefully.

Let & # 8217; s understand how to scale Scrapy Playwright faithfully by controlling concurrence, trim unneeded browser work, and keep stability during long-running or high-volume creep.

12. Scraping Multiple Pages

Scraping a single page is rarely the end goal. Most real-world targets imply navigating across family pages, folio links, or dynamically generated URLs. When Playwright is introduced, it becomes important to control when browser mechanization is really required.

Scrapy Playwright allows mixing browser-driven requests with standard Scrapy requests so that only pages requiring JavaScript find the overhead of a browser.

yield scrapy.Request (url=next_page,
callback=self.parse,
meta= {& # 8220; playwright & # 8221;: True}

)

Overusing Playwright for every petition can slow crawls importantly. When performance pearl unexpectedly, the usual campaign is treat static and dynamic page the same instead of selectively enabling browser rendition.

13. Managing Playwright Sessions and Concurrency

At scale, each Playwright page consumes memory and CPU. Without session control, spiders can tire scheme resourcefulness or stall under load.

Scrapy Playwright cater unrelenting browser contexts through session direction, which countenance cookies, authentication state, and page datum to be reused safely across requests.

meta= {& # 8220; playwright & # 8221;: True,
& # 8220; playwright_context & # 8221;: & # 8220; session_1 & # 8221;

}

Reusing contexts reduces browser startups and stabilise long-running crawls. When spiders slow down over time or wreck after various hundred requests, unmanaged browser contexts are usually the underlying problem.

14. Running Playwright in Headless Mode

Headless performance is essential for production scraping where visual output is unneeded. Running browsers with a UI increases resource consumption and bound concurrence.

Playwright runs headlessly by default, but explicit configuration ensures coherent behavior across environments.

PLAYWRIGHT_LAUNCH_OPTIONS = {& # 8220; headless & # 8221;: True

}

If behavior differs between local and CI environs, the discrepancy frequently arrive from inconsistent browser launch settings instead than scraping logic.

15. Aborting Unwanted Requests

Dynamic pages ofttimes lade analytics scripts, ads, videos, and tracking pixels that are irrelevant to scraping. Allowing these postulation to action wastes bandwidth and slows page rendering.

Scrapy Playwright can block unnecessary resource types before they are downloaded.

PageMethod (& # 8220; route & # 8221;,
& # 8220; * * / * & # 8221;,
lambda route, request: route.abort ()
if request.resource_type in [& # 8220; picture & # 8221;, & # 8220; font & # 8221;, & # 8220; media & # 8221;]
else route.continue_ ()

)

Reducing mesh noise improves page constancy and crawl speed. When pages direct longer to load despite minimal scraping logic, exuberant third-party requests are often the secret cause.

16. Restarting Disconnected Browsers

Long-running crawls can encounter browser clangoring, memory wetting, or lose connections. Without recovery logic, a single browser failure can halt the full scratching.

Scrapy Playwright mechanically restart disunited browser when configured correctly, allowing spiders to continue without manual interposition.

PLAYWRIGHT_BROWSER_TYPE = & # 8220; chromium & # 8221;

If wanderer stop reply after extended runtimes, the subject is usually browser lifecycle management kinda than website blocking or parsing erroneousness.

How to Use Proxies with Scrapy Playwright (Without Getting Blocked)

When grate at scale, repeated requests from a individual IP often lead to throttling or blocking. facilitate distribute traffic, but using them with browser automation require more than setting an HTTP placeholder.

With Scrapy Playwright, proxies must be applied at thebrowser setting gradeso that page loads, JavaScript executing, and AJAX ring all originate from the same IP. This is essential for conserve session consistency on active website.

Scrapy Playwright handles this by pass proxy setting straightaway to the Playwright browser context.

yield scrapy.Request (url= & # 8221; https: //example.com/jobs & # 8221;,

meta={
& # 8220; playwright & # 8221;: True,
& # 8220; playwright_context_kwargs & # 8221;: {
& # 8220; proxy & # 8221;: {
& # 8220; server & # 8221;: & # 8220; http: //proxy-server:8000 & # 8221;,
& # 8220; username & # 8221;: & # 8220; user & # 8221;,
& # 8220; password & # 8221;: & # 8220; pass & # 8221;,

}

}

},
callback=self.parse

)

This shape ensures that the entire browser session uses the proxy, include scripts, plus, and background mesh phone.

Problems with procurator rarely fail loudly. Instead, they usually appear as pernicious data issue:

• Pages render, but key elements are missing.
• AJAX responses fail silently.
• CAPTCHAs appear intermittently despite low request volume.

These symptom typically indicate unstable proxies or discrepant IP usage during rendering.

To reduce blocking risk and meliorate constancy:

• Rotate proxies per browser context instead than per postulation.
• Avoid highly shared or low-quality proxy for JavaScript-heavy sites.
• Validate proxy conduct under existent browser weather before scaling.

How BrowserStack Can Help Validate Scrapy Playwright on Real Devices?

Scrapy Playwright scripts are commonly developed and try in controlled local environments. While this work for initial development, it frequently hides issues that appear only when scraping runs at scale or in different environments.

Mutual limitations when validating Scrapy Playwright locally include:

• Headless browsers behave differently from full browser.
• Dynamic contented loads inconsistently across browser engines.
• Layout or interaction differences on mobile viewports.
• Limited visibility into failure that occur intermittently.
• Resource constraints when running multiple browser sessions in parallel.

These limitation do it unmanageable to know whether failures are caused by scrape logic, browser conduct, or the execution environment itself.

addresses this by providing approach toreal browser and real devicesin cloud-hosted environs. Instead of relying on local machines or feign setups, Scrapy Playwright workflow can be formalise against actual browser engines, operating systems, and device conformation.

Key means BrowserStack fits into a Scrapy Playwright workflow include:

• Cross-browser validation: Confirm that Playwright-driven interaction behave consistently across Chromium, Firefox, and WebKit.
• : Validate scraping behavior on actual mobile and desktop browsers where layouts and JavaScript execution differ.
• : Review past test sessions to track coverage, failure, and execution patterns to identify flaky behavior and meliorate scraping reliability.
• : Run multiple Playwright sessions concurrently without managing local browser infrastructure.

Talk to an Expert

Conclusion

Scrapy Playwright makes it possible to scrape modern websites that bank on dynamic rendering, client-side JavaScript, and asynchronous data loading. By combine Scrapy & # 8217; s effective crawling and postulation handling with Playwright & # 8217; s browser automation, it enables reliable interaction with pages that would otherwise retrovert incomplete or inconsistent data.

As scratch workflows scale, validating behavior across different browsers and environment becomes critical. BrowserStack complements Scrapy Playwright by allowing scripts to be screen on real browser and device, helping uncover environment-specific issues early and improving the reliability of large-scale scraping trial.

Related Guides