Adaptive batching
Large REST requests are downloaded in batches and recursively subdivided when a provider times out or rejects the request.
GIS DEVELOPER CASE STUDY · ARCGIS PRO · PYTHON · ARCPY · REST
Automated GIS Data Acquisition
A production-oriented ArcGIS Pro Python toolbox that turns a point, line, or polygon study area into a project-ready geodatabase by automating public GIS data acquisition, clipping, projection, provenance, QA, reporting, and delivery.
THE PROBLEM
Project setup should not mean repeating the same portal-by-portal workflow.
Environmental and infrastructure projects routinely require analysts to visit multiple agency portals, identify the right dataset, download it, clip it to a study area, project it, organize outputs, and document where it came from. The work is necessary, but much of it is repetitive and inconsistent.
8public-data source groups
2,695NHD flowlines acquired in 3m 57s during a 10-mile test
4automatic 3DEP tiles used after a server export limit
0manual portal downloads required during an automated run
THE SOLUTION
One study area. One toolbox. Multiple authoritative datasets.
The toolbox accepts a point, polyline, or polygon study area, applies an optional buffer, communicates directly with public ArcGIS REST and ImageServer services, and delivers standardized GIS outputs inside a project geodatabase.
Supported data
- FEMA NFHL flood hazard zones
- USFWS National Wetlands Inventory
- USGS hydrography flowlines and waterbodies
- USGS NLCD land cover
- USGS 3DEP elevation and approximately 10 m DEM
- U.S. Census state and county boundaries
- USDA NASS Cropland Data Layer
01Input
Study area and optional buffer.
02Prepare AOI
Validate, dissolve, buffer, and project as required.
03Acquire
Query authoritative REST and ImageServer sources.
04Process
Batch, clip, mosaic, project, and standardize.
05Deliver
Geodatabase, provenance log, QA report, and manifest.
ENGINEERING CHALLENGE
8000 × 8000A large B2H-style 20-mile study area exceeded the USGS 3DEP ImageServer export-size limit. A normal raster request could not complete.
THE FIX
Make the workflow adapt instead of fail.
I added an automatic raster fallback that detects service-size errors, recursively subdivides the AOI, downloads manageable raster tiles, mosaics the results, and projects the final raster to the study-area coordinate system. In testing, the failed 3DEP request automatically became four successful tiles and a single final raster.
GIS DEVELOPER FEATURES
Built as a resilient data-engineering workflow, not a one-off script.
Actual-AOI querying
NHD Flowline acquisition uses the actual study-area geometry and optimized 125-feature batches to reduce unnecessary candidate features and timeout risk.
Automatic tiling
ImageServer size-limit failures trigger AOI subdivision, tile acquisition, mosaicking, and reprojection without manual intervention.
Failure isolation
A temporary failure from one public provider is logged without terminating acquisition from the remaining selected sources.
Provenance + validation
Each run records source agency, service URL, year, feature or raster properties, CRS, runtime, acquisition status, and QA status.
Project-ready reporting
The workflow produces a geodatabase log, an HTML QA report, and a CSV acquisition manifest alongside the GIS deliverables.
ArcGIS ProPythonArcPyArcGIS REST APIImageServerFile GeodatabaseRaster ProcessingSpatial QA/QCGitGitHub
RESULT
A repetitive GIS setup task became a reusable, fault-tolerant acquisition system.
The project demonstrates GIS development beyond geoprocessing: API integration, adaptive error handling, vector and raster processing, provenance, QA, packaging, documentation, and versioned software delivery.