Hallo allemaal, ik ben bezig met het bouwen van een object herkennings dataset. Hiervoor gebruik ik de luchtfoto’s van pdok. Deze download ik door wms requests te doen met python. Een paar weken geleden ging dit heel goed met maar ongeveer een 502 error per 150 requests. Sinds een paar dagen zijn er echter veel meer 502 errors. Sommige luchtfoto’s kan ik zelfs na 20 pogingen niet downloaden.
Weet iemand wat hier aan de hand is? Heb ik misschien te veel requests gaan in totaal ofzo?
image_versions = ['2022_orthoHR','2022_ortho25']
wms = WebMapService('https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?request=GetCapabilities&service=wms')
def get_images_for_tiles(tile_coordinates, folder_name, image_versions):
for version in image_versions:
for count, [xmin,xmax,ymin,ymax] in enumerate(tile_coordinates):
x_range = (xmax - xmin) * 4
y_range = (ymax - ymin) * 4
if not os.path.exists(f'data/{folder_name}/{version}; {str(round(xmin+500))}, {str(round(ymin+500))}.jpeg'):
success = False
sleep_time = 5
while not success:
try:
img = wms.getmap(layers=[version],
srs='EPSG:28992',
bbox=[xmin, ymin, xmax, ymax],
size=(x_range, y_range),
format='image/jpeg',
timeout=60)
success = True
print(f'Request succeeded for tile {count}')
sleep_time = 5
except Exception as e:
print(f'Error on attempt {count}: {str(e)}')
time.sleep(sleep_time)
sleep_time = sleep_time * 3
if sleep_time > 45:
print(f'Sleep time exceeded 45 seconds, skipping tile {count}')
break
if success:
out = open(f'data/{folder_name}/{version}; {str(round(xmin+500))}, {str(round(ymin+500))}.jpeg', 'wb')
out.write(img.read())
out.close()
Hieronder volgt een voorbeeld van de output
Request succeeded for tile 4722
Request succeeded for tile 4723
Request succeeded for tile 4724
Request succeeded for tile 4725
Request succeeded for tile 4729
Request succeeded for tile 4730
Error on attempt 4732: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=169000%2C409000%2C170000%2C410000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Error on attempt 4732: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=169000%2C409000%2C170000%2C410000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Request succeeded for tile 4732
Request succeeded for tile 4734
Request succeeded for tile 4735
Request succeeded for tile 4736
Request succeeded for tile 4739
Request succeeded for tile 4740
Request succeeded for tile 4741
Request succeeded for tile 4743
Request succeeded for tile 4744
Request succeeded for tile 4745
Request succeeded for tile 4746
Request succeeded for tile 4747
Request succeeded for tile 4750
Request succeeded for tile 4751
Request succeeded for tile 4753
Request succeeded for tile 4754
Request succeeded for tile 4755
Request succeeded for tile 4756
Request succeeded for tile 4757
Request succeeded for tile 4758
Request succeeded for tile 4759
Request succeeded for tile 4760
Request succeeded for tile 4764
Request succeeded for tile 4765
Error on attempt 4766: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=260000%2C567000%2C261000%2C568000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Request succeeded for tile 4766
Error on attempt 4769: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=237000%2C566000%2C238000%2C567000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Request succeeded for tile 4769
Request succeeded for tile 4770
Request succeeded for tile 4771
Request succeeded for tile 4773
Request succeeded for tile 4774
Request succeeded for tile 4775
Request succeeded for tile 4776
Request succeeded for tile 4777
Request succeeded for tile 4783
Request succeeded for tile 4784
Request succeeded for tile 4785
Error on attempt 4786: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=150000%2C414000%2C151000%2C415000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Request succeeded for tile 4786
Request succeeded for tile 4787
Request succeeded for tile 4789
Request succeeded for tile 4790
Request succeeded for tile 4791
Request succeeded for tile 4793
Request succeeded for tile 4794
Request succeeded for tile 4795
Request succeeded for tile 4796
Request succeeded for tile 4797
Request succeeded for tile 4800
Request succeeded for tile 4801
Error on attempt 4802: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=157000%2C395000%2C158000%2C396000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Request succeeded for tile 4802
Request succeeded for tile 4803
Request succeeded for tile 4804
Request succeeded for tile 4806
Request succeeded for tile 4807
Request succeeded for tile 4808
Request succeeded for tile 4810
Request succeeded for tile 4812
Request succeeded for tile 4814
Request succeeded for tile 4816
Request succeeded for tile 4818
Request succeeded for tile 4819
Error on attempt 4822: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=171000%2C408000%2C172000%2C409000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Request succeeded for tile 4822
Request succeeded for tile 4824
Request succeeded for tile 4825
Request succeeded for tile 4826
Error on attempt 4827: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=182000%2C410000%2C183000%2C411000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Request succeeded for tile 4827
Request succeeded for tile 4828
Request succeeded for tile 4830
Request succeeded for tile 4832
Request succeeded for tile 4833
Request succeeded for tile 4835
Request succeeded for tile 4838
Error on attempt 4839: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=170000%2C407000%2C171000%2C408000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Error on attempt 4839: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=170000%2C407000%2C171000%2C408000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Request succeeded for tile 4839
Error on attempt 4846: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=147000%2C410000%2C148000%2C411000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Request succeeded for tile 4846
Request succeeded for tile 4847
Error on attempt 4848: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=181000%2C409000%2C182000%2C410000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Request succeeded for tile 4848
Request succeeded for tile 4849
Request succeeded for tile 4850
Request succeeded for tile 4851
Error on attempt 4852: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=153000%2C407000%2C154000%2C408000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Request succeeded for tile 4852
Request succeeded for tile 4853
Error on attempt 4854: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=162000%2C401000%2C163000%2C402000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Request succeeded for tile 4854
Request succeeded for tile 4855
Error on attempt 4861: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=174000%2C410000%2C175000%2C411000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Error on attempt 4861: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=174000%2C410000%2C175000%2C411000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Error on attempt 4861: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=174000%2C410000%2C175000%2C411000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Sleep time exceeded 45 seconds, skipping tile 4861
Request succeeded for tile 4863
Error on attempt 4867: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=173000%2C405000%2C174000%2C406000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Error on attempt 4867: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=173000%2C405000%2C174000%2C406000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Error on attempt 4867: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=173000%2C405000%2C174000%2C406000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Sleep time exceeded 45 seconds, skipping tile 4867
Error on attempt 4872: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=137000%2C415000%2C138000%2C416000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Error on attempt 4872: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=137000%2C415000%2C138000%2C416000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Error on attempt 4872: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=137000%2C415000%2C138000%2C416000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Sleep time exceeded 45 seconds, skipping tile 4872
Request succeeded for tile 4873
Error on attempt 4875: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=139000%2C374000%2C140000%2C375000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Request succeeded for tile 4875
Request succeeded for tile 4876
Request succeeded for tile 4877
Request succeeded for tile 4878
Error on attempt 4879: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=149000%2C374000%2C150000%2C375000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Request succeeded for tile 4879
Error on attempt 4880: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=149000%2C391000%2C150000%2C392000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Error on attempt 4880: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=149000%2C391000%2C150000%2C392000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Request succeeded for tile 4880
Error on attempt 4881: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=149000%2C389000%2C150000%2C390000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Error on attempt 4881: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=149000%2C389000%2C150000%2C390000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Error on attempt 4881: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=149000%2C389000%2C150000%2C390000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Sleep time exceeded 45 seconds, skipping tile 4881
Request succeeded for tile 4884
Request succeeded for tile 4885
Request succeeded for tile 4886
Request succeeded for tile 4888
Request succeeded for tile 4889
Request succeeded for tile 4890
Error on attempt 4891: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=155000%2C390000%2C156000%2C391000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Error on attempt 4891: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=155000%2C390000%2C156000%2C391000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Error on attempt 4891: 504 Server Error: Gateway Timeout for url: https://service.pdok.nl/hwh/luchtfotorgb/wms/v1_0?service=WMS&version=1.1.1&request=GetMap&layers=2022_orthoHR&styles=&width=4000&height=4000&srs=EPSG%3A28992&bbox=155000%2C390000%2C156000%2C391000&format=image%2Fjpeg&transparent=FALSE&bgcolor=0xFFFFFF&exceptions=application%2Fvnd.ogc.se_xml
Sleep time exceeded 45 seconds, skipping tile 4891
Request succeeded for tile 4892
Request succeeded for tile 4894
Request succeeded for tile 4895
Request succeeded for tile 4896