DEV Community

Devil Scrapes profile picture

Devil Scrapes

View all public scrapers @ https://apify.com/devilscrapes

Joined Joined on 
The US sanctions list still ships with a 1980s DOS end-of-file byte
devil_scrapes profile

The US sanctions list still ships with a 1980s DOS end-of-file byte

#python #api #opendata #webscraping
Comments
4 min read
This API returns bank details for IBANs that are invalid
devil_scrapes profile

This API returns bank details for IBANs that are invalid

#api #fintech #python #webdev
Comments
4 min read
The Czech company register hands you an ID that is not an ID
devil_scrapes profile

The Czech company register hands you an ID that is not an ID

#api #python #webscraping #opendata
Comments
4 min read
GLEIF ships two status fields for one company, and they answer different questions
devil_scrapes profile

GLEIF ships two status fields for one company, and they answer different questions

#api #python #fintech #compliance
Comments
5 min read
Neither app store has a reviews API — Apple caps at 50, Google has no field names
devil_scrapes profile

Neither app store has a reviews API — Apple caps at 50, Google has no field names

#webscraping #python #api #mobile
Comments
5 min read
Craigslist's JSON-LD has no ID field — we join 290 of 325 listings by title alone
devil_scrapes profile

Craigslist's JSON-LD has no ID field — we join 290 of 325 listings by title alone

#webscraping #python #api #datascience
Comments
5 min read
Booking.com will happily give you a plausible price in the wrong currency
devil_scrapes profile

Booking.com will happily give you a plausible price in the wrong currency

#webscraping #python #playwright #api
Comments
5 min read
BBB.org's business data isn't in the HTML — it's in an analytics script tag
devil_scrapes profile

BBB.org's business data isn't in the HTML — it's in an analytics script tag

#webscraping #python #api #opendata
Comments
5 min read
Glassdoor runs a second, different anti-bot challenge on its Reviews page
devil_scrapes profile

Glassdoor runs a second, different anti-bot challenge on its Reviews page

#webscraping #python #nextjs #debugging
Comments
5 min read
Norway's official company register hard-stops at 10,000 results
devil_scrapes profile

Norway's official company register hard-stops at 10,000 results

#api #python #opendata #compliance
Comments
5 min read
Redfin silently drops your price filter unless you set one undocumented parameter
devil_scrapes profile

Redfin silently drops your price filter unless you set one undocumented parameter

#webscraping #python #api #debugging
Comments
5 min read
OfferUp's zipCode parameter does nothing — the real signal is a cookie
devil_scrapes profile

OfferUp's zipCode parameter does nothing — the real signal is a cookie

#webscraping #python #nextjs #api
Comments
5 min read
An unset proxy group cost us 75% of Glassdoor's customer runs
devil_scrapes profile

An unset proxy group cost us 75% of Glassdoor's customer runs

#webscraping #python #devops #api
Comments
5 min read
A finished scraper sat on a git branch for 19 days. Nothing noticed.
devil_scrapes profile

A finished scraper sat on a git branch for 19 days. Nothing noticed.

#webscraping #python #git #devops
Comments
5 min read
Your sitemap checker is probably parsing a 404 page that returned 200
devil_scrapes profile

Your sitemap checker is probably parsing a 404 page that returned 200

#seo #python #webscraping #api
Comments
6 min read
Greece is EL, Northern Ireland is XI, and Germany answers '---'
devil_scrapes profile

Greece is EL, Northern Ireland is XI, and Germany answers '---'

#api #python #webscraping #compliance
Comments
6 min read
A KRS number is not a number, and Poland's register has no 404
devil_scrapes profile

A KRS number is not a number, and Poland's register has no 404

#api #python #opendata #webscraping
Comments
6 min read
crt.sh has no pagination, and its wildcard is a percent sign you must spell %25
devil_scrapes profile

crt.sh has no pagination, and its wildcard is a percent sign you must spell %25

#security #python #webscraping #devops
Comments
5 min read
Chess.com splits one player across four endpoints, and only answers Firefox
devil_scrapes profile

Chess.com splits one player across four endpoints, and only answers Firefox

#api #python #webscraping #cloudflare
Comments
5 min read
The Location header is relative more often than your link checker thinks
devil_scrapes profile

The Location header is relative more often than your link checker thinks

#webscraping #python #seo #http
Comments
4 min read
Cloudflare's /security.txt 404s with Content-Type: text/plain
devil_scrapes profile

Cloudflare's /security.txt 404s with Content-Type: text/plain

#security #webscraping #python #seo
Comments
4 min read
A domain that doesn't exist returns HTTP 200 OK
devil_scrapes profile

A domain that doesn't exist returns HTTP 200 OK

#dns #api #python #devops
Comments
4 min read
Wikimedia blocks python-requests and waves through an empty User-Agent
devil_scrapes profile

Wikimedia blocks python-requests and waves through an empty User-Agent

#python #api #webscraping #debugging
Comments
4 min read
Docker Hub's API 403s Chrome and 200s Firefox — same URL, same second
devil_scrapes profile

Docker Hub's API 403s Chrome and 200s Firefox — same URL, same second

#docker #webscraping #python #api
Comments
4 min read
OData's worst failure isn't the 400 — it's the query that quietly matches nothing
devil_scrapes profile

OData's worst failure isn't the 400 — it's the query that quietly matches nothing

#api #python #webscraping #data
Comments
3 min read
The FDIC's API returns data.data, and ASSET is in thousands
devil_scrapes profile

The FDIC's API returns data.data, and ASSET is in thousands

#api #python #fintech #webscraping
Comments
4 min read
Our container reported SUCCEEDED, ran nothing, and scored 100% on every dashboard
devil_scrapes profile

Our container reported SUCCEEDED, ran nothing, and scored 100% on every dashboard

#python #docker #webscraping #devops
Comments
4 min read
UN Comtrade's free tier: 500 rows, no pagination, and a silent 429
devil_scrapes profile

UN Comtrade's free tier: 500 rows, no pagination, and a silent 429

#python #api #webscraping #datascience
Comments
4 min read
EDGAR's total count lies to you, and it says so
devil_scrapes profile

EDGAR's total count lies to you, and it says so

#python #api #webscraping #fintech
Comments
3 min read
Most EDGAR tools stop at the filing. The trades are one layer deeper.
devil_scrapes profile

Most EDGAR tools stop at the filing. The trades are one layer deeper.

#python #api #webscraping #fintech
Comments
3 min read
NHTSA decodes any 17 characters you give it, valid or not
devil_scrapes profile

NHTSA decodes any 17 characters you give it, valid or not

#python #api #webscraping #automotive
Comments
3 min read
ORCID's search API has a ceiling it never tells you about
devil_scrapes profile

ORCID's search API has a ceiling it never tells you about

#python #api #webscraping #opensource
Comments
4 min read
The NWS API makes you resolve a grid before it tells you the weather
devil_scrapes profile

The NWS API makes you resolve a grid before it tells you the weather

#python #api #webscraping #datascience
Comments
4 min read
Our Chrome fingerprint was the thing getting us blocked
devil_scrapes profile

Our Chrome fingerprint was the thing getting us blocked

#python #api #webscraping #fintech
Comments
3 min read
Four levels of JSON to say the word Leasehold
devil_scrapes profile

Four levels of JSON to say the word Leasehold

#python #api #opendata #realestate
Comments
4 min read
Your demo input has an expiry date (and your tests can't see it)
devil_scrapes profile

Your demo input has an expiry date (and your tests can't see it)

#python #api #webscraping #testing
Comments
4 min read
openFDA's docs say +AND+. Send that and you get a 500.
devil_scrapes profile

openFDA's docs say +AND+. Send that and you get a 500.

#python #api #healthcare #webscraping
Comments
4 min read
Our browser impersonation made the API return HTML (and 29 tests missed it)
devil_scrapes profile

Our browser impersonation made the API return HTML (and 29 tests missed it)

#python #api #testing #webscraping
Comments
3 min read
Don't scrape data.gov. Speak CKAN.
devil_scrapes profile

Don't scrape data.gov. Speak CKAN.

#python #api #opendata #webscraping
Comments
4 min read
The offset parameter that silently does nothing (and would have billed for it)
devil_scrapes profile

The offset parameter that silently does nothing (and would have billed for it)

#python #webscraping #seo #api
Comments
4 min read
rdap.org answers 302, and a client that doesn't follow it sees nothing
devil_scrapes profile

rdap.org answers 302, and a client that doesn't follow it sees nothing

#python #webscraping #dns #api
Comments
4 min read
255 App Store scrapers and not one of them tracks keyword rank
devil_scrapes profile

255 App Store scrapers and not one of them tracks keyword rank

#python #webscraping #aso #api
Comments
4 min read
A SUCCEEDED run with zero log lines: the probe that never executed
devil_scrapes profile

A SUCCEEDED run with zero log lines: the probe that never executed

#python #webscraping #debugging #ecommerce
Comments
4 min read
One retry budget, two unrelated failures: how a scraper blamed the wrong wall
devil_scrapes profile

One retry budget, two unrelated failures: how a scraper blamed the wrong wall

#python #webscraping #debugging #leadgen
Comments
4 min read
Two green runs an hour apart is not reliability if the scraper is 33% reliable
devil_scrapes profile

Two green runs an hour apart is not reliability if the scraper is 33% reliable

#python #webscraping #testing #debugging
Comments
4 min read
Same API standard, 57 rivals, nobody consolidated: harvesting every Socrata portal at once
devil_scrapes profile

Same API standard, 57 rivals, nobody consolidated: harvesting every Socrata portal at once

#opendata #api #python #webscraping
Comments
4 min read
The 404 Was Our URL, Not Idealista's Wall
devil_scrapes profile

The 404 Was Our URL, Not Idealista's Wall

#python #webscraping #realestate #api
Comments
4 min read
We debugged the same bug for 3 days because it was never actually deployed
devil_scrapes profile

We debugged the same bug for 3 days because it was never actually deployed

#webscraping #python #devops #debugging
Comments
7 min read
A SUCCEEDED run with 3 rows, and every row was fabricated
devil_scrapes profile

A SUCCEEDED run with 3 rows, and every row was fabricated

#webscraping #python #testing #api
Comments
6 min read
Open Brewery DB Has Two Ways to Say 'Zero Results'
devil_scrapes profile

Open Brewery DB Has Two Ways to Say 'Zero Results'

#python #api #opendata #webscraping
Comments
4 min read
UK Police's Crime API Returns 404 When the Answer Is Zero
devil_scrapes profile

UK Police's Crime API Returns 404 When the Answer Is Zero

#python #api #opendata #govtech
Comments
4 min read
The proxy wall that was a leftover environment variable
devil_scrapes profile

The proxy wall that was a leftover environment variable

#webscraping #python #debugging #devops
Comments
4 min read
USAspending.gov Returns a Fake 500 to Block Scrapers — Here's the Fix
devil_scrapes profile

USAspending.gov Returns a Fake 500 to Block Scrapers — Here's the Fix

#python #api #govtech #webscraping
Comments
3 min read
openFDA Tells You the Outcome Was 5
devil_scrapes profile

openFDA Tells You the Outcome Was 5

#api #python #opendata #healthcare
Comments
3 min read
Our Scraper Said Every EU Tender Was Titled "F"
devil_scrapes profile

Our Scraper Said Every EU Tender Was Titled "F"

#api #python #testing #webscraping
Comments
3 min read
An API With No Pagination Is a Billing Problem
devil_scrapes profile

An API With No Pagination Is a Billing Problem

#api #python #opendata #webscraping
Comments
3 min read
The French Registry Says 10,000 Results. It Means the Cap.
devil_scrapes profile

The French Registry Says 10,000 Results. It Means the Cap.

#api #python #opendata #webscraping
Comments
3 min read
One Medicare API, Four Datasets, Three Renamed Fields
devil_scrapes profile

One Medicare API, Four Datasets, Three Renamed Fields

#api #python #opendata #webscraping
Comments
4 min read
EPA ECHO Reports a Rejected Search as HTTP 200
devil_scrapes profile

EPA ECHO Reports a Rejected Search as HTTP 200

#python #api #opendata #webscraping
Comments
4 min read
UK Contracts Finder: how to scrape public-sector tenders without losing rows to bad GUID parsing
devil_scrapes profile

UK Contracts Finder: how to scrape public-sector tenders without losing rows to bad GUID parsing

#webscraping #python #opendata #api
Comments
3 min read
loading...