Web-scraping web comics

A template for web-scraping comics, using Piled Higher & Deeper as an example (in Python; if on windows, you'll need wget from Cygwin):

import urllib2
import re
import os
import sys

url = "http://www.phdcomics.com/comics/archive.php?comicid="

for id in range(int(sys.argv[1]), int(sys.argv[2])):
print `id`+"..."
try:
page = urllib2.urlopen(url+`id`).read()
img = re.search('img src=(http://www.phdcomics.com/comics/archive/.+?.gif)', page).group(1)
os.system('wget -nc '+img)
except urllib2.HTTPError:
print "No page_id " + `id`


Notes-to-self: sinusoidal curve fitting

One ppt
Another ppt

y = C + Asin(Bx - D)

A = (largest-smallest)/2 (determine the amplitude)
C = (largest+smallest)/2 (determine the vertical shift)
B = 2pi/P (P = period? delta time/distance?) (determine period?)
D = "must use A, B, C and a point (x,y)" (determine horizontal shift)

Looks like it requires 3 points. Maybe divide height-map into 3x3 "squares"?
- Can't find sin(x)*sin(z) that goes through them all, how to handle that?
- Some points will have to be left out

Maybe do it like this?
0 z 0
x x/z x
0 z 0
- Results in about half the points being left out...much of a problem? (increase height-map resolution)

Notes-to-self: next steps for sine-based terrain

(notes to self; not intended to be interesting/meaningful/relevant to anyone else)

So I have terrain based on sin(x)*sin(z) working pretty well (where "pretty well" is defined as 40-60+ fps on my iMac...the shader is too complicated to run on my laptop's integrated intel card, but more on that later...maybe). It would look alright if used for, say, an open stretch of ocean, but it's way too boring for something that's supposed to be actual land. So...what's next?

Well, first, some irritations with my current method. Chief among them is that I cannot change the landscape without changing both the code and the shader (the code sets up the blocks the shader carves away from). The main reason for this is that the formula (sin(x)*sin(z)) is hard-coded into the shader. Ideally, I could change just change code. (Actually, ideally I wouldn't have to change the code either; it would read the formula from a text file and everything else would be magically figured out.) The way I see to do the first part: generalize the formula in the shader so that it accepts parameters from the code (ie: a*sin(bx + c) + d).

So here's a thought for "what's next", which attempts to address several other problems I'm having at the moment: code hands block to shader, with 8 parameters. The shader will evaluate a*sin(bx+c)+d * e*sin(fz+g)+h on the block, in modelspace. This will simplify the shader, as it will not need to know anything about where in the world the object is, and make it more flexible (we just want the shader to make the GPU do math work for us; we don't want to simulate state with it).

The simplest use of this is to easily tweak the whole scene's frequency, amplitude, etc. from code. However, another possible (ab)use is stringing together blocks of different frequency or amplitude together to make a more interesting level. One problem that immediately comes to mind is differentiability: if I'm just stringing sections of sine waves together, they certainly won't be differentiable...but the question is, how bad will it look? Given reasonable constraints, will it look "good enough"?

Keeping the current semi-goal of smooth curvy terrain from a height map in mind: could I take a "square" of the height map, look at the differences, and find a chunk of sin(x)*sin(z) that, with appropriate a-h parameters as described above, approximates it? It seems feasible enough (in my tired state) to warrant further investigation.

How I fixed my problems with long downloads

Turns out my DD-WRT (v24 sp1) router's SPI firewall was blocking packets related to DHCP from my ISP. This caused DHCP connection to reset every 10 minutes, which stalled any downloads in progress and caused internet applications (Skype, online games) to behave oddly.

I fixed this by adding the following line to my firewall startup instructions:
iptables -I INPUT -p udp --sport 67 --dport 68 -j ACCEPT

Open router configuration page (http://10.0.0.1 for me) and go to Administration->Commands. Copy and paste the above into the "Command shell" text box, then click "Save Firewall".

From this thread.