Reorganize project into scripts/, docs/, data/, output/ directories
Move all Python scripts to scripts/, documentation to docs/, raw input data to data/, and generated HTML/CSV outputs to output/. Update path references in 8 scripts to use Path(__file__).parent.parent as project root so they work correctly from the new location. Update README links and quick-start commands accordingly. Notebooks remain at root. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
61
README.md
61
README.md
@@ -4,8 +4,9 @@ A comprehensive geospatial research project investigating the spatial concentrat
|
|||||||
|
|
||||||
## Documentation
|
## Documentation
|
||||||
|
|
||||||
- **[Database Tables](database-tables.md)** - Complete database schema with table descriptions, column definitions, and SQL examples
|
- **[Database Tables](docs/database-tables.md)** - Complete database schema with table descriptions, column definitions, and SQL examples
|
||||||
- **[Research Ideas](research-ideas.md)** - Future research directions, data improvements, and potential collaborations
|
- **[Research Ideas](docs/research-ideas.md)** - Future research directions, data improvements, and potential collaborations
|
||||||
|
- **[SQL Queries](docs/query_legiscan_bills.sql)** - Pre-built legislative analysis queries
|
||||||
|
|
||||||
## Project Overview
|
## Project Overview
|
||||||
|
|
||||||
@@ -93,25 +94,25 @@ Facilities in DBSCAN clusters differ significantly from isolated sites:
|
|||||||
|
|
||||||
### Core Python Scripts
|
### Core Python Scripts
|
||||||
|
|
||||||
**Data Ingestion**
|
**Data Ingestion** (`scripts/`)
|
||||||
- `load_postgis_data_centers.py` - Load curated data center CSV into PostGIS
|
- `scripts/load_postgis_data_centers.py` - Load curated data center CSV into PostGIS
|
||||||
- `load_postgis_osm_data_centers.py` - Fetch OSM data centers via Overpass API
|
- `scripts/load_postgis_osm_data_centers.py` - Fetch OSM data centers via Overpass API
|
||||||
- `build_master_data_centers.py` - Deduplicate & merge curated + OSM sources
|
- `scripts/build_master_data_centers.py` - Deduplicate & merge curated + OSM sources
|
||||||
- `load_postgis_internet_cables.py` - Load submarine cables and landing points
|
- `scripts/load_postgis_internet_cables.py` - Load submarine cables and landing points
|
||||||
- `ingest_eia_energy_layers.py` - Ingest EIA energy data via API
|
- `scripts/ingest_eia_energy_layers.py` - Ingest EIA energy data via API
|
||||||
- `build_watershed_huc8_tables.py` - Load USGS HUC8 watersheds
|
- `scripts/build_watershed_huc8_tables.py` - Load USGS HUC8 watersheds
|
||||||
- `ingest_legiscan.py` - Download all US state/federal bills 2016–2026 via LegiScan API, tag for data center research topics
|
- `scripts/ingest_legiscan.py` - Download all US state/federal bills 2016–2026 via LegiScan API, tag for data center research topics
|
||||||
|
|
||||||
**Enrichment**
|
**Enrichment**
|
||||||
- `create_data_center_census_tract_table.py` - Join data centers to Census tracts with ACS demographics
|
- `scripts/create_data_center_census_tract_table.py` - Join data centers to Census tracts with ACS demographics
|
||||||
- `build_fcc_bdc_broadband_connection_table.py` - Build per-facility broadband provider table
|
- `scripts/build_fcc_bdc_broadband_connection_table.py` - Build per-facility broadband provider table
|
||||||
- `build_fcc_bdc_location_provider_aggregates.py` - Aggregate FCC BDC data by county/tract
|
- `scripts/build_fcc_bdc_location_provider_aggregates.py` - Aggregate FCC BDC data by county/tract
|
||||||
|
|
||||||
**Analysis**
|
**Analysis**
|
||||||
- `analyze_dc_tract_concentration.py` - Tract-level cost concentration analysis (Gini, HHI, demographic deltas)
|
- `scripts/analyze_dc_tract_concentration.py` - Tract-level cost concentration analysis (Gini, HHI, demographic deltas)
|
||||||
- `analyze_cables_concentration.py` - Test if data centers cluster near submarine cables
|
- `scripts/analyze_cables_concentration.py` - Test if data centers cluster near submarine cables
|
||||||
- `make_data_center_map.py` - Generate Leaflet map of data centers
|
- `scripts/make_data_center_map.py` - Generate Leaflet map of data centers
|
||||||
- `make_internet_cables_map.py` - Generate Leaflet map of data centers + cables
|
- `scripts/make_internet_cables_map.py` - Generate Leaflet map of data centers + cables
|
||||||
|
|
||||||
### Key Jupyter Notebooks
|
### Key Jupyter Notebooks
|
||||||
- `spatial_clustering_master_data_centers.ipynb` - DBSCAN clustering of data centers
|
- `spatial_clustering_master_data_centers.ipynb` - DBSCAN clustering of data centers
|
||||||
@@ -161,34 +162,34 @@ Credentials stored in `~/.zsh_secrets`, loaded via environment variables:
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
# 1. Load base data center data
|
# 1. Load base data center data
|
||||||
python3 load_postgis_data_centers.py
|
python3 scripts/load_postgis_data_centers.py
|
||||||
python3 load_postgis_osm_data_centers.py
|
python3 scripts/load_postgis_osm_data_centers.py
|
||||||
python3 build_master_data_centers.py
|
python3 scripts/build_master_data_centers.py
|
||||||
|
|
||||||
# 2. Enrich with context layers
|
# 2. Enrich with context layers
|
||||||
python3 create_data_center_census_tract_table.py --replace-final
|
python3 scripts/create_data_center_census_tract_table.py --replace-final
|
||||||
python3 load_postgis_internet_cables.py
|
python3 scripts/load_postgis_internet_cables.py
|
||||||
python3 ingest_eia_energy_layers.py --category power
|
python3 scripts/ingest_eia_energy_layers.py --category power
|
||||||
python3 build_watershed_huc8_tables.py
|
python3 scripts/build_watershed_huc8_tables.py
|
||||||
|
|
||||||
# 3. Run analyses
|
# 3. Run analyses
|
||||||
python3 analyze_dc_tract_concentration.py > output/tract_analysis.txt
|
python3 scripts/analyze_dc_tract_concentration.py > output/tract_analysis.txt
|
||||||
python3 analyze_cables_concentration.py > output/cables_analysis.txt
|
python3 scripts/analyze_cables_concentration.py > output/cables_analysis.txt
|
||||||
|
|
||||||
# 4. Execute notebooks
|
# 4. Execute notebooks
|
||||||
jupyter notebook cluster_analysis.ipynb
|
jupyter notebook cluster_analysis.ipynb
|
||||||
|
|
||||||
# 5. Load legislation (all states, 2016-2026)
|
# 5. Load legislation (all states, 2016-2026)
|
||||||
python3 ingest_legiscan.py --all
|
python3 scripts/ingest_legiscan.py --all
|
||||||
# Weekly refresh (skips unchanged sessions):
|
# Weekly refresh (skips unchanged sessions):
|
||||||
python3 ingest_legiscan.py --fetch --load
|
python3 scripts/ingest_legiscan.py --fetch --load
|
||||||
```
|
```
|
||||||
|
|
||||||
### Generate Maps
|
### Generate Maps
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
python3 make_data_center_map.py
|
python3 scripts/make_data_center_map.py
|
||||||
python3 make_internet_cables_map.py
|
python3 scripts/make_internet_cables_map.py
|
||||||
```
|
```
|
||||||
|
|
||||||
## Key Outputs
|
## Key Outputs
|
||||||
|
|||||||
|
Can't render this file because it is too large.
|
Reference in New Issue
Block a user