POSThttps://api.aitronos.com/v1/scrape/batch
Process multiple URLs in parallel with optimized resource management.
| Name | Type | Required | Description |
|---|---|---|---|
Authorization | string | Yes | Bearer token authentication |
Content-Type | string | Yes | Must be application/json |
urls array required
List of URLs to scrape (max 50).
schema object required
JSON schema for data extraction.
organization_id string optional
Organization ID for billing and access control.
options object optional
Batch processing options.
| Field | Type | Required | Description |
|---|---|---|---|
urls | array | Yes | List of URLs to scrape (max 50) |
schema | object | Yes | JSON schema for data extraction |
options | object | No | Batch processing options |
| Field | Type | Default | Description |
|---|---|---|---|
max_items_per_url | integer | 50 | Max items per URL (1-500) |
max_total_items | integer | 500 | Max total items across all URLs (1-5000) |
parallel_jobs | integer | 3 | Number of parallel jobs (1-10) |
timeout_per_url | integer | 30 | Timeout per URL in seconds (1-300) |
date_filter | object | null | Filter items by date |
llm_mode | string | "structured" | LLM processing mode |
Status: 200 OK
| Field | Type | Description |
|---|---|---|
batch_id | string | Unique batch identifier |
total_urls | integer | Total number of URLs |
jobs | array | Array of job objects |
progress | object | Progress information |
created_at | string | Batch creation timestamp |
| Field | Type | Description |
|---|---|---|
job_id | string | Unique job identifier |
url | string | Target URL |
status | string | Job status: "pending", "processing", "completed", "failed" |
| Field | Type | Description |
|---|---|---|
completed | integer | Number of completed jobs |
failed | integer | Number of failed jobs |
pending | integer | Number of pending jobs |
total_items_extracted | integer | Total items extracted so far |
Returns a JSON response indicating success or failure.
Bash
- Bash
- Python
- JavaScript
curl -X POST https://api.aitronos.com/v1/scrape/batch \
-H "X-API-Key: $FREDDY_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"urls": [
"https://example.com/page1",
"https://example.com/page2",
"https://example.com/page3"
],
"schema": {
"type": "object",
"properties": {
"title": {"type": "string"},
"content": {"type": "string"}
}
},
"options": {
"max_items_per_url": 50,
"parallel_jobs": 3
}
}'Response 200 OK
{
"batch_id": "batch_xyz789",
"total_urls": 3,
"jobs": [
{
"job_id": "job_abc123",
"url": "https://example.com/page1",
"status": "pending"
},
{
"job_id": "job_def456",
"url": "https://example.com/page2",
"status": "pending"
},
{
"job_id": "job_ghi789",
"url": "https://example.com/page3",
"status": "pending"
}
],
"progress": {
"completed": 0,
"failed": 0,
"pending": 3,
"total_items_extracted": 0
},
"created_at": "2024-12-16T10:30:00Z"
}