Skip to main content
Property hunting is time-consuming and competitive. This example shows how to build an automated system that continuously monitors letting agents’ websites for new properties matching your specific criteria, giving you a competitive advantage in fast-moving rental markets.

​
Use case overview

You’re a property investor or tenant looking for 3-bedroom flats in Edinburgh with specific requirements. Rather than manually checking multiple letting agents multiple times a day, you’ll create an automated scraper that:
  • Monitors letting agents’ websites for new properties
  • Filters results by your criteria (bedrooms, location, price range)
  • Extracts structured data including address, description, pricing, and availability
  • Runs on a schedule to catch new listings as soon as they appear

​
How it works

1

Create the scraping task

Define what data you want to extract and from which website.
2

Execute the task

Run the task with specific criteria to get current listings.
3

Schedule regular polling

Set up automated execution to catch new properties immediately.

​
Implementation

​
Step 1: Create the scraping task

First, create a task that can extract property data from a letting agent’s website. We’ll use Southside Management as an example. Let’s define our input schema, so we can filter the results by bedrooms/location/price range, and the output schema to ensure a consistent data structure in the application code. 
{
  "type": "object",
  "properties": {
    "min_bedrooms": {
      "type": "integer",
      "description": "Minimum number of bedrooms"
    },
    "max_bedrooms": {
      "type": "integer", 
      "description": "Maximum number of bedrooms"
    },
    "max_rent": {
      "type": "number",
      "description": "Maximum monthly rent in GBP"
    }
  },
  "required": ["min_bedrooms", "max_bedrooms", "max_rent"]
}
We can then create the task using the above schemas: 
curl -X POST 'https://api.indices.io/v1beta/tasks' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "task": "Find available-to-rent flats in Edinburgh. Extract the property ID, address, description, number of bedrooms and bathrooms, rental cost with payment period, availability date, and main image URL if available. Focus only on properties from this letting agency - all information needed is available on their website.",
    "website": "https://southsidemanagement.com/",
    "input_schema": "{\"type\": \"object\", \"properties\": {\"min_bedrooms\": {\"type\": \"integer\", \"description\": \"Minimum number of bedrooms\"}, \"max_bedrooms\": {\"type\": \"integer\", \"description\": \"Maximum number of bedrooms\"}, \"max_rent\": {\"type\": \"number\", \"description\": \"Maximum monthly rent in GBP\"}}, \"required\": [\"min_bedrooms\", \"max_bedrooms\", \"max_rent\"]}",
    "output_schema": "{\"type\": \"array\", \"items\": {\"type\": \"object\", \"properties\": {\"property_id\": {\"type\": \"string\", \"description\": \"Unique identifier from the website\"}, \"address\": {\"type\": \"string\", \"description\": \"Full property address\"}, \"description\": {\"type\": \"string\", \"description\": \"Property description\"}, \"bedrooms\": {\"type\": \"integer\", \"description\": \"Number of bedrooms\"}, \"bathrooms\": {\"type\": \"integer\", \"description\": \"Number of bathrooms\"}, \"rental_cost\": {\"type\": \"number\", \"description\": \"Monthly rental cost in GBP\"}, \"cost_period\": {\"type\": \"string\", \"description\": \"Payment period (monthly, weekly, etc.)\"}, \"available_from\": {\"type\": \"string\", \"description\": \"Date available to rent from\"}, \"image_url\": {\"type\": \"string\", \"description\": \"URL to main property image\"}, \"listing_url\": {\"type\": \"string\", \"description\": \"Link to full property details\"}}, \"required\": [\"property_id\", \"address\", \"description\", \"bedrooms\", \"bathrooms\", \"rental_cost\", \"cost_period\", \"available_from\"]}}}"
  }'
The API will immediately respond with an id and a status of not_ready.
Task creation response
{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "current_state": "not_ready",
  "task": "Find available-to-rent flats in Edinburgh...",
  "website": "https://southsidemanagement.com/",
  "input_schema": "...",
  "output_schema": "...",
  "arguments": null,
  "created_at": "2024-01-15T10:30:00Z",
  "updated_at": "2024-01-15T10:30:00Z"
}
It typically takes 1-3 minutes for a task to be ready for usage, but may take up to 10 minutes.

​
Step 2: Wait for task to be ready

Check the task status until it’s ready for use:
cURL
curl -X GET 'https://api.indices.io/v1beta/tasks/550e8400-e29b-41d4-a716-446655440000' \
  -H 'Authorization: Bearer YOUR_API_KEY'
Task Ready
{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "current_state": "ready",
  "task": "Find available-to-rent flats in Edinburgh...",
  "website": "https://southsidemanagement.com/",
  "input_schema": "...",
  "output_schema": "...",
  "arguments": {
    "min_bedrooms": {"type": "integer"},
    "max_bedrooms": {"type": "integer"}, 
    "max_rent": {"type": "number"}
  },
  "created_at": "2024-01-15T10:30:00Z",
  "updated_at": "2024-01-15T10:32:15Z"
}
You can programatically poll the /tasks endpoint to be notified when it’s ready: 
import requests
import asyncio

async def main():
    task_id = ... # task_id from previous step

    # poll the task until it's ready
    while True:
        response = requests.get(f"https://api.indices.io/v1beta/tasks/{task_id}")
        task = response.json()
        if task["current_state"] == "ready":
            break
        await asyncio.sleep(1)

​
Step 3: Execute the task with your criteria

Now run the task with your filtering criteria: 
curl -X POST 'https://api.indices.io/v1beta/runs' \
  -H 'Authorization: Bearer YOUR_API_KEY' \
  -H 'Content-Type: application/json' \
  -d '{
    "task_id": "550e8400-e29b-41d4-a716-446655440000",
    "arguments": {
      "min_bedrooms": 3,
      "max_bedrooms": 3,
      "max_rent": 2500
    }
  }'
Property Results
{
  "id": "run_123",
  "task_id": "550e8400-e29b-41d4-a716-446655440000",
  "arguments": {
    "min_bedrooms": 3,
    "max_bedrooms": 3,
    "max_rent": 2500
  },
  "result_json": "[{\"property_id\": \"SM_2024_001\", \"address\": \"45 Marchmont Road, Edinburgh EH9 1HU\", \"description\": \"Spacious 3-bedroom flat in the heart of Marchmont with period features, high ceilings, and modern kitchen. Close to university and city centre.\", \"bedrooms\": 3, \"bathrooms\": 2, \"rental_cost\": 2200, \"cost_period\": \"monthly\", \"available_from\": \"15th February 2024\", \"image_url\": \"https://southsidemanagement.com/images/SM_2024_001_main.jpg\", \"listing_url\": \"https://southsidemanagement.com/properties/SM_2024_001\"}, {\"property_id\": \"SM_2024_002\", \"address\": \"12 Bruntsfield Place, Edinburgh EH10 4HN\", \"description\": \"Modern 3-bedroom apartment with en-suite master bedroom, open plan living, and private balcony. Excellent transport links.\", \"bedrooms\": 3, \"bathrooms\": 2, \"rental_cost\": 2400, \"cost_period\": \"monthly\", \"available_from\": \"1st March 2024\", \"image_url\": \"https://southsidemanagement.com/images/SM_2024_002_main.jpg\", \"listing_url\": \"https://southsidemanagement.com/properties/SM_2024_002\"}]",
  "created_at": "2024-01-15T10:35:00Z",
  "finished_at": "2024-01-15T10:35:12Z"
}
You can reuse the task_id to run the task many times. Different runs can use the same or different criteria.

​
Setting up automated polling

To catch new properties as soon as they’re listed, set up a scheduled system to run your task regularly:
Python
import requests
import json
import time
from datetime import datetime

class PropertyPoller:
    def __init__(self, api_key, task_id):
        self.api_key = api_key
        self.task_id = task_id
        self.base_url = "https://api.indices.io"
        self.headers = {
            "Authorization": f"Bearer {api_key}",
            "Content-Type": "application/json"
        }
        self.seen_properties = set()
    
    def run_task(self, criteria):
        """Execute the scraping task with given criteria"""
        response = requests.post(
            f"{self.base_url}/runs",
            headers=self.headers,
            json={
                "task_id": self.task_id,
                "arguments": criteria
            }
        )
        response.raise_for_status()
        return response.json()
    
    def check_for_new_properties(self, criteria):
        """Check for new properties and return only unseen ones"""
        run_result = self.run_task(criteria)
        properties = json.loads(run_result["result_json"])
        
        new_properties = []
        for prop in properties:
            if prop["property_id"] not in self.seen_properties:
                new_properties.append(prop)
                self.seen_properties.add(prop["property_id"])
        
        return new_properties
    
    def poll_continuously(self, criteria, interval_minutes=30):
        """Poll for new properties at regular intervals"""
        print(f"Starting property polling every {interval_minutes} minutes...")
        
        while True:
            try:
                new_properties = self.check_for_new_properties(criteria)
                
                if new_properties:
                    print(f"Found {len(new_properties)} new properties!")
                    for prop in new_properties:
                        self.notify_new_property(prop)
                else:
                    print(f"No new properties found at {datetime.now()}")
                
                time.sleep(interval_minutes * 60)
                
            except Exception as e:
                print(f"Error during polling: {e}")
                time.sleep(60)  # Wait 1 minute before retrying
    
    def notify_new_property(self, property_data):
        """Handle new property notification (customize as needed)"""
        print(f"NEW PROPERTY: {property_data['address']}")
        print(f"Rent: £{property_data['rental_cost']}/{property_data['cost_period']}")
        print(f"Available: {property_data['available_from']}")
        print(f"Link: {property_data['listing_url']}")
        print("-" * 50)

# Usage
poller = PropertyPoller(
    api_key="your_api_key_here",
    task_id="550e8400-e29b-41d4-a716-446655440000"
)

criteria = {
    "min_bedrooms": 3,
    "max_bedrooms": 3,
    "max_rent": 2500
}

# Poll every 15 minutes
poller.poll_continuously(criteria, interval_minutes=15)

​
Advanced configurations

​
Multiple letting agents

Create separate tasks for different letting agents and poll them all: 
agents = [
    {
        "name": "Southside Management",
        "task_id": "550e8400-e29b-41d4-a716-446655440000"
    },
    {
        "name": "DJ Alexander", 
        "task_id": "550e8400-e29b-41d4-a716-446655440001"
    },
    {
        "name": "Citylets",
        "task_id": "550e8400-e29b-41d4-a716-446655440002"
    }
]

class MultiAgentPoller:
    def __init__(self, api_key, agents):
        self.api_key = api_key
        self.agents = agents
        self.pollers = {
            agent["name"]: PropertyPoller(api_key, agent["task_id"])
            for agent in agents
        }
    
    def poll_all_agents(self, criteria):
        all_new_properties = []
        for name, poller in self.pollers.items():
            try:
                new_props = poller.check_for_new_properties(criteria)
                for prop in new_props:
                    prop["agent"] = name
                all_new_properties.extend(new_props)
            except Exception as e:
                print(f"Error polling {name}: {e}")
        
        return all_new_properties

​
Smart notifications

Integrate with notification services for immediate alerts: 
import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart

def send_email_alert(property_data, recipient_email):
    """Send email notification for new property"""
    smtp_server = "smtp.gmail.com"
    smtp_port = 587
    sender_email = "[email protected]"
    sender_password = "your_app_password"
    
    message = MIMEMultipart()
    message["From"] = sender_email
    message["To"] = recipient_email
    message["Subject"] = f"New Property Alert: {property_data['address']}"
    
    body = f"""
    New property matching your criteria:
    
    Address: {property_data['address']}
    Bedrooms: {property_data['bedrooms']}
    Rent: £{property_data['rental_cost']}/{property_data['cost_period']}
    Available: {property_data['available_from']}
    
    Description: {property_data['description']}
    
    View details: {property_data['listing_url']}
    """
    
    message.attach(MIMEText(body, "plain"))
    
    with smtplib.SMTP(smtp_server, smtp_port) as server:
        server.starttls()
        server.login(sender_email, sender_password)
        server.send_message(message)

def send_slack_notification(property_data, webhook_url):
    """Send Slack notification for new property"""
    import requests
    
    payload = {
        "text": f"🏠 New Property Alert",
        "blocks": [
            {
                "type": "section",
                "text": {
                    "type": "mrkdwn",
                    "text": f"*{property_data['address']}*\n{property_data['bedrooms']} bed, {property_data['bathrooms']} bath\n£{property_data['rental_cost']}/{property_data['cost_period']}"
                }
            },
            {
                "type": "actions",
                "elements": [
                    {
                        "type": "button",
                        "text": {"type": "plain_text", "text": "View Details"},
                        "url": property_data['listing_url']
                    }
                ]
            }
        ]
    }
    
    requests.post(webhook_url, json=payload)

​
Best practices

Polling frequency

Poll frequently for competitive markets, but rate limit your requests to avoid being blocked.

Error handling

Implement retry logic with exponential backoff. Log failures and continue polling other letting agents if one fails.

Data storage

Store historical data to track market trends, price changes, and property availability patterns.

Filtering optimization

Start with broader criteria on API calls and shift filtering to your application logic. This reduces API calls while maintaining flexibility.

​
Troubleshooting

Cause: Invalid JSON schema in input_schema or output_schemaSolution: Validate your JSON schemas using an online validator before creating the task. Ensure all required fields are properly defined.
Cause: The website structure is too complex or has anti-bot protectionsSolution: Try simplifying your task description or contact support. Some websites may require special handling.
Cause: Your filtering criteria might be too restrictive, or the website has no matching propertiesSolution: Test with broader criteria first, then gradually narrow down. Check the website manually to confirm properties exist.
Cause: The website displays data inconsistently across different property listingsSolution: Make your output schema more flexible by making fields optional where appropriate, and handle data cleaning in your application.
Start with a single letting agent and perfect your polling setup before expanding to multiple agents. This makes debugging much easier.

​
Next steps

Once you have basic property polling working:
  1. Add data persistence - Store results in a database to track price changes and availability
  2. Build a dashboard - Create a web interface to view and manage your property alerts
  3. Implement ML filtering - Use machine learning to score properties based on your preferences
  4. Scale to multiple cities - Expand your monitoring to other locations and letting agents
  5. Add market analytics - Track rental market trends and identify investment opportunities