Strict Hierarchical Timeouts

Oct 13, 2025 by magicalceylontours.com 130 views

Fix #2: Comprehensive Solution for Concurrency, Timeouts, and Resource Management

Introduction

Freezing issues in scripts can be frustrating and time-consuming to resolve. To address this problem, we will implement a comprehensive solution that covers concurrency, timeouts, and resource management. This approach will ensure that no single URL can block the entire process indefinitely and system resources are managed rigorously.

1. Strict Hierarchical Timeouts

One of the primary causes of freezing issues is the lack of timeouts in asynchronous operations. To prevent indefinite hanging, we will wrap every async operation with specific timeouts.

Modify process_url:

async def process_url(url: str) -> Set[str]:
    try:
        return await asyncio.wait_for(_process_url_with_timeout(), timeout=120)
    except asyncio.TimeoutError:
        print(f"Timeout processing {url}")
        return set()
    finally:
        # Ensure any cleanup here
        pass

By adding a timeout of 120 seconds to the process_url function, we ensure that the operation will not hang indefinitely. If the operation takes longer than 120 seconds, it will raise a TimeoutError, which we can handle accordingly.

2. Playwright Timeout Enforcement

Playwright operations also require granular timeouts to prevent indefinite hanging. We will add timeouts to every Playwright operation and handle cancellations properly.

Update _perform_extraction:

async def _perform_extraction(...):
    try:
        async with async_playwright() as p:
            browser = await p.chromium.launch(timeout=30000)  # 30s timeout
            # Add timeouts to every navigation step
            await asyncio.wait_for(page.goto(url), timeout=25)
    except Exception as e:
        # Handle cleanup
        await browser.close()

In the updated _perform_extraction function, we added a timeout of 30 seconds to the launch method and a timeout of 25 seconds to the goto method. This ensures that the operation will not hang indefinitely.

3. Concurrency Control with Semaphores

To prevent resource exhaustion, we will limit the number of simultaneous Playwright instances using semaphores.

Add at the top:

PLAYWRIGHT_SEMAPHORE = asyncio.Semaphore(3)  # Only 3 concurrent browsers

Modify Playwright calls:

async def extract_emails_playwright(url: str):
    async with PLAYWRIGHT_SEMAPHORE:
        return await _perform_extraction(url)

By using a semaphore with a value of 3, we ensure that only 3 concurrent Playwright instances are created. This prevents resource exhaustion and ensures that the operation runs smoothly.

4. Batch-Level Timeout Enforcement

To ensure that no batch runs longer than a maximum duration, we will add a timeout to the process_batch function.

Update process_batch:

async def process_batch(batch, results_dict):
    try:
        await asyncio.wait_for(
            _process_batch(batch, results_dict),
            timeout=180
        )
    except asyncio.TimeoutError:
        print("Batch timed out, cancelling tasks")
        # Cancel all tasks in the batch
        for _, task in tasks:
            task.cancel()

In the updated process_batch function, we added a timeout of 180 seconds to the _process_batch function. If the operation takes longer than 180 seconds, it will raise a TimeoutError, which we can handle accordingly.

5. Enhanced Cancellation Handling

To ensure proper cleanup when tasks are cancelled, we will add cancellation handling to the Playwright cleanup.

Modify Playwright cleanup:

async def _perform_extraction(...):
    try:
        # ... existing code ...
    except asyncio.CancelledError:
        print("Playwright task cancelled")
        await browser.close()
        raise

By adding cancellation handling to the Playwright cleanup, we ensure that the browser is properly closed when a task is cancelled.

6. Critical HTTP Extraction Fixes

To enforce strict timeouts in HTTP requests and add cancellation support, we will update the extract_emails_http function.

Update extract_emails_http:

async def extract_emails_http(url):
    try:
        async with aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=15)) as session:
            async with session.get(url) as response:
                content = await asyncio.wait_for(response.text(), timeout=10)
    except asyncio.TimeoutError:
        return set(), None

In the updated extract_emails_http function, we added a timeout of 15 seconds to the ClientSession and a timeout of 10 seconds to the response.text method. This ensures that the operation will not hang indefinitely.

7. Diagnostic Logging

To help identify bottlenecks, we will add detailed logging throughout the code.

Add logs throughout:

print(f"[DEBUG] Starting HTTP extraction for {url}")
# ... operation ...
print(f"[DEBUG] Finished HTTP extraction for {url}")

By adding logging throughout the code, we can identify exactly where delays occur and optimize the operation accordingly.

8. Avoid Retry Loops Without Timeouts

To prevent infinite loops in contact-finding logic, we will add per-request timeouts to the try_common_contact_urls function.

Update try_common_contact_urls:

async def try_common_contact_urls(...):
    async with aiohttp.ClientSession(timeout=aiohttp.ClientTimeout(total=3)) as session:
        for pattern in contact_patterns:
            try:
                await asyncio.wait_for(session.head(...), timeout=2)
            except asyncio.TimeoutError:
                continue

In the updated try_common_contact_urls function, we added a timeout of 3 seconds to the ClientSession and a timeout of 2 seconds to the session.head method. This ensures that the operation will not hang indefinitely.

9. Use `asyncio.gather` with Timeouts

To process batch URLs in parallel with a global timeout, we will use asyncio.gather with timeouts.

Update process_batch:

async def process_batch(...):
    tasks = [process_url(url) for url in cleaned_urls]
    done, pending = await asyncio.wait(tasks, timeout=180)
    for task in pending:
        task.cancel()
    # Process results from 'done'

In the updated process_batch function, we used asyncio.gather with a timeout of 180 seconds to process the batch URLs in parallel. If any of the tasks take longer than 180 seconds, they will be cancelled.

10. Playwright Configuration Tweaks

To improve stability, we will add the following flags to the Playwright launch.

Update Playwright launch:

browser = await p.chromium.launch(
    headless=True,
    args=[
        '--disable-gpu',
        '--single-process',  # Reduces resource usage
        '--no-zygote',
        '--disable-setuid-sandbox'
    ]
)

By adding these flags to the Playwright launch, we can improve stability and reduce resource usage.

Summary of Key Improvements

Hierarchical Timeouts: Every layer (batch/URL/request) has strict timeouts.
Resource Limits: Semaphores control Playwright instances; cleanup is enforced.
Cancellation Propagation: Properly handles CancelledError to abort stuck operations.
Diagnostic Logging: Helps identify exactly where delays occur.
Avoid Nested Retries: Prevents infinite loops in contact-finding logic.

By implementing these key improvements, we can ensure that no single URL can block the entire process indefinitely and system resources are managed rigorously.
Fix #2: Comprehensive Solution for Concurrency, Timeouts, and Resource Management - Q&A

Introduction

In our previous article, we discussed a comprehensive solution for concurrency, timeouts, and resource management to resolve freezing issues in scripts. In this article, we will answer some frequently asked questions (FAQs) related to this solution.

Q1: What is the main cause of freezing issues in scripts?

A1: The main cause of freezing issues in scripts is the lack of timeouts in asynchronous operations. When an operation takes longer than expected, it can cause the script to hang indefinitely.

Q2: How can I implement hierarchical timeouts in my script?

A2: To implement hierarchical timeouts, you can wrap every async operation with specific timeouts. For example, you can use the asyncio.wait_for function to set a timeout for a specific operation.

Q3: What is the purpose of using semaphores in concurrency control?

A3: Semaphores are used to limit the number of simultaneous Playwright instances, preventing resource exhaustion. By using a semaphore with a value of 3, for example, you can ensure that only 3 concurrent Playwright instances are created.

Q4: How can I handle cancellations properly in my script?

A4: To handle cancellations properly, you can use the asyncio.CancelledError exception to abort stuck operations. You can also use the task.cancel method to cancel tasks that are no longer needed.

Q5: What is the purpose of diagnostic logging in this solution?

A5: Diagnostic logging is used to help identify bottlenecks in the script. By adding logging statements throughout the code, you can identify exactly where delays occur and optimize the operation accordingly.

Q6: How can I avoid retry loops without timeouts in my script?

A6: To avoid retry loops without timeouts, you can add per-request timeouts to the try_common_contact_urls function. This ensures that the operation will not hang indefinitely.

Q7: What is the purpose of using `asyncio.gather` with timeouts in this solution?

A7: asyncio.gather is used to process batch URLs in parallel with a global timeout. By using asyncio.gather with a timeout of 180 seconds, for example, you can process the batch URLs in parallel while ensuring that the operation does not hang indefinitely.

Q8: How can I improve stability in my script using Playwright configuration tweaks?

A8: To improve stability, you can add the following flags to the Playwright launch: headless=True, args=['--disable-gpu', '--single-process', '--no-zygote', '--disable-setuid-sandbox']. These flags can help reduce resource usage and improve stability.

Q9: What are the key improvements in this solution?

A9: The key improvements in this solution are:

Hierarchical Timeouts: Every layer (batch/URL/request) has strict timeouts.
Resource Limits: Semaphores control Playwright instances; cleanup is enforced.
Cancellation Propagation: Properly handles CancelledError to abort stuck operations.
Diagnostic Logging: Helps identify exactly where delays occur.
Avoid Nested Retries: Prevents infinite loops in contact-finding logic.

By implementing these key improvements, you can ensure that no single URL can block the entire process indefinitely and system resources are managed rigorously.

Conclusion

In this article, we answered some frequently asked questions related to the comprehensive solution for concurrency, timeouts, and resource management. By implementing this solution, you can resolve freezing issues in your scripts and ensure that your operations run smoothly and efficiently.