Security, privacy and responsible data collection
Siftyr collects public web data on behalf of our customers. This page explains how we protect your account and data, how long we keep it, who helps us run the service, and how we crawl the web responsibly.
Security overview
Encryption in transit
All traffic to the platform, the API and file downloads is served over TLS. Webhooks and storage destinations are reached over HTTPS / SFTP where the destination supports it.
Encryption at rest
Credentials you store with us — destinations (S3, FTP/SFTP, cloud storage, webhooks) and SSO client secrets — are encrypted field-by-field with AES-256-GCM before they are written to the database.
Signed, expiring download links
Every delivered file link is signed and expires after 7 days. Pull links use an unguessable per-source identifier that you can rotate by recreating the source.
Signed webhooks
Webhook deliveries carry an HMAC-SHA256 signature in the x-siftyr-signature header so your endpoint can verify they came from Siftyr.
Role-based access control
Managers, Trusted users and Viewers have distinct permissions; collections can be assigned per user. API keys are scoped to a user, hashed at rest, optionally expire, and are shown only once.
Audit log
Sign-ins, configuration changes, harvests, exports, deliveries and administrative actions are recorded in your organization's activity log.
Single sign-on
Organizations can sign in through their own OpenID Connect identity provider (PKCE, state and nonce validated) and require SSO for all users.
Account protection
Accounts lock after 5 failed sign-in attempts. Sessions are httpOnly cookies. The public API is rate limited to 100 requests per minute per user.
Data retention
Extracted records, generated exports and crawl logs are kept for 90 days by default. Each organization's Managers can set their own retention period between 7 and 3,650 days in Settings → Data retention. Data older than the period is permanently deleted by an automated retention job.
Harvest metadata (dates, record counts, statuses) is retained for insights and billing. When an organization closes its account, its data is deleted on request.
Sub-processors
| Sub-processor | Purpose | Location |
|---|---|---|
| Amazon Web Services | Application hosting, databases, file storage, backups, and sending and receiving email | Australia (Sydney) |
| Vercel | Hosting the marketing website and its DNS | Global network |
| Groq | AI features: page content, field names and data samples sent by the feature | United States |
We will notify customers before adding a sub-processor that processes customer data.
Responsible scraping
robots.txt respected by default
Crawlers honor robots.txt rules and crawl-delay directives. Overriding this requires a documented, reviewed justification on the extractor.
Per-domain rate limits
Requests to each site are spaced with per-domain delays and back off automatically when a site signals load (HTTP 429 / Retry-After).
Public data only
We do not crawl behind logins we don't own on behalf of the customer, and we don't bypass paywalls or access controls.
PII flagging
Extractor fields that contain personal data are flagged so they can be reviewed before the data is collected and delivered. We collect personal data only where the customer has a lawful basis to process it.
Contact
Questions about security or privacy, or want to report a vulnerability? Email support@siftyr.si. Website owners who want to discuss how we crawl their site can contact us at the same address.