POSThttps://api.aitronos.com/v1/scrape/analyze
Analyze a website to determine scraping strategy and complexity before scraping.
| Name | Type | Required | Description |
|---|---|---|---|
Authorization | string | Yes | Bearer token authentication |
Content-Type | string | Yes | Must be application/json |
| Field | Type | Required | Description |
|---|---|---|---|
url | string | Yes | Target URL to analyze |
Status: 200 OK
| Field | Type | Description |
|---|---|---|
url | string | Analyzed URL |
estimated_items | integer | Estimated number of items on page |
complexity | string | Site complexity: "low", "medium", or "high" |
requires_js | boolean | Whether JavaScript rendering is required |
is_spa | boolean | Whether site is a Single Page Application |
has_anti_bot | boolean | Whether anti-bot measures detected |
estimated_processing_time | number | Estimated processing time in seconds |
recommended_limits | object | Recommended max_items and timeout values |
recommended_engine | string | Recommended scraping engine |
pagination_detected | boolean | Whether pagination was detected |
robots_txt_restrictions | string | Any robots.txt restrictions (null if none) |
content_type | string | Content type from HTTP headers |
status_code | integer | HTTP status code |
Bash
- Bash
- Python
- JavaScript
curl -X POST https://api.aitronos.com/api/v1/scrape/analyze \
-H "X-API-Key: $FREDDY_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/products"}'Response 200 OK
{
"url": "https://example.com/products",
"estimated_items": 25,
"complexity": "medium",
"requires_js": true,
"is_spa": false,
"has_anti_bot": false,
"estimated_processing_time": 3.5,
"recommended_limits": {
"max_items": 50,
"timeout": 45
},
"recommended_engine": "browser",
"pagination_detected": true,
"robots_txt_restrictions": null,
"content_type": "text/html",
"status_code": 200
}