• Shoveling Out, On Steroids

    Another dusting last night, an easy shovel job but the neighborhood looks really pretty, especially on my walk this morning. Anne went early to deal with her office’s walkways, then met Debbie for breakfast at the new breakfast place on Main Street (the Flying Egg, go there it’s pretty nice). Anyway — after I got up, and shoveled here — I texted to see if she needed help; she replied that the job was done and I should  come over and join them. Great start to the day, nice to see Debbie, and the point of my story was that it was beautiful out, with early-morning-rosy winter clouds, before it all morphed into a generic “sunny winter day,” which was nice in its own way but that early sky really was cool.

    On the home front, we got our new oven yesterday. It looks pretty nice and stainless-steel modern, the range is a bit more aggressive than our old one and, most important, the oven keeps the correct temperature. Too bad the delivery came while I was trying to sleep in — not too early really, but before 9:00, and I was trying to catch up on my sleep after a rough few days…

    I’ve had a bit of an eczema problem lately, and it really got crazy this week. We super-cleaned the house, I switched to baths instead of showers… and I broke down yesterday, went to my GP and got some prescription strength cortisone cream, as well as a Prednisone prescription. I’ve been warned about euphoria, mania etc as side effects, but nothing: I’ve basically been just putzing around the house today, though my skin is running through a fast-motion miracle cure so there’s that. I have an allergist appointment in the New Year, and I got a referral for a dermatologist from the GP. I’m going up in the attic soon to find the humidifier. Life goes on.

    Meantime, the mapping — rather, the fixing of the mapping scale-up problems — continues. I had problems with getting the elevation changes, and had to eventually abandon a QGIS solution, and build my own PostGIS function to get the “chainages.” The term is apparently a holdover from ancient surveyor days, where they used chains to measure distances; what I needed was a shapefile of points, set every 10 meters along each road in the database, but the new file had to refer back to the road database in a certain way, and the QGIS plugin just wasn’t flexible enough for what I needed. (My solution worked like a charm.) The next step was to use SAGA and my elevation data to give each point an elevation, which since the new chainages were themselves now in the database rather than a standalone file, the process was its own struggle learning experience, but it’s done now. Next up is generating the ascent/descent data, which I might decide to do in PostGIS as well — my current, PyQGIS-based method is run-all-night-check-results-in-the-morning slow. Tomorrow, or this weekend…

  • This Week Today

    Stopping by again…

    Mapping: I had, and still have, a few technical issues to deal with, but the full Lehigh Valley database is now in PostGIS, along with elevation data — bogus elevation data, that’s one of my technical issues — and the demo map can now route with the new database. But it’s got the slows, it’s got the slooowws… With about 3200 road segments in the “toy database,” it could route in about 1-2 seconds, but the full-map version took about 6 seconds per routing task — and there may be multiple routing tasks in each route, from start point, to via point and then through subsequent via points, and finally to the endpoint. Unacceptable!

    I did some searches online, and sure enough there are a lot of people complaining about pgRouting performance and looking to speed it up. The general consensus: there are a few things you can do, including tune your database, but the actual bottlenecks are the pgRouting algorithms. Some suggested using osm2po, another program that converts OpenStreetMap data for databases but can  also do routing: tried it and it’s blindingly fast – d’oh! (Unfortunately, I didn’t see much there in the way of customized, dynamic cost functions, so I can’t see how to turn it into the the answer I’m looking for.) I tried a bunch f the Postgres/PostGIS performance-tuning tips anyway, and they did seem to help a little.

    I eventually came across one potential solution: route only on a subset of the roads in the database, using a bounding box. For each pair of points to route between, I find the smallest rectangle that contains both, then expand it by 2000 meters in every direction (like a buffer zone); this is my bounding box, and the routing search is limited to the roads that touch or fall within that box. This seemed to do the trick: my routing times are back down to about 1-2 seconds.

    Except near — wait for it — those confounded bridges. The valley is broken up by the Lehigh river, with occasional bridges, and if there are no bridges within the bounding box for a route that needs to cross the river, no route will be found. Meanwhile, when routing points are on a diagonal, the bounding boxes are fairly big, but routing points that run mainly east-west or north-south produce long, skinny bounding boxes. I found a few “dead zones” where routes couldn’t be found, especially east-west ones north of Northampton, routes with skinny bounding boxes where the bridges are a little sparser. My original bounding boxes were expanded by a buffer that was only 1000 meters; I went to 2000 meters in an attempt to alleviate the bridge problem. This didn’t solve it entirely, but it did help, and there was no real performance hit going from 1000 to 2000 meters. I’ll probably look at distances between bridges, and revise my buffer zone to be just bigger than say, half that distance.

    Reading: I picked up Don DeLillo’s Underworld again, intending to just read the first part. I love the first chapter but never finished the book because I found the rest boring; now I am engrossed and don’t know what I was thinking back  then.

    Listening: WXPN has been playing “The 70’s, A-Z” this past week, every song they have in their library that was released in the Seventies, played in alphabetical order. We’ve been following along religiously, and it’s been fascinating and fun but they’re only up to “T,” and it gets wearing. Full disclosure: the radio is off right now…

    The only time they weren’t playing the 70’s was for their Friday “Free at Noon” concert at the Word Cafe, which this week featured Russ’s band Cherry. So, we went down to Philly with Ray and Lorraine, where we met Frank and Patricia, and Ben, and Gabby, and we all watched the show and then went out to lunch with Russ at the White Dog Cafe. As always, we spent a few minutes at Penn Books before the ride home. All the talk in Philly, among us and overheard on the street, was about the upcoming snow on Saturday…

    By the way, Saturday was Luminaria Night in Bethlehem, here is a photo of ours:

    candles in bags on sidewalk
    Luminaria Night

    One last thing: here is what I wrote ten years ago.

  • Updates on Various Things

    Just kicking back this morning, before going with Anne over to the Bike Co-op for the afternoon…

    Reading: I just finished N.K. Jemisin’s debut novel, The Hundred Thousand Kingdoms. I took to it well enough at the beginning, but it actually became a chore to read: I put it down for a week, and read the last third in two sittings, closing the book with a sense of relief yesterday. Strange because I really liked her award-winning “Broken Earth” trilogy, and the style and voice were very similar; Anne said that maybe the author worked a few bugs out of her writing between her debut and the trilogy, and that may be so but I didn’t really see it. All I can say is that I really recommend the trilogy, but don’t feel the same about this one. I think it’s also first of two, but it’ll be a while before I read the sequel.

    Two Hours Before The Mast: I did my usual Wednesday volunteering at the Canal Museum yesterday. The canal boat is now in dry-dock for the winter, and Scott E is trying to get as much maintenance done on it (especially things like painting) in the nicer weather as he can, so yesterday I helped prep the deck for staining. Mostly this meant sanding, and the sanding I did was mostly “trim work” with a small vibrating sander, near fixtures and in corners where the bigger unit couldn’t fit — I did this for about two hours until the little sander overheated and turned off.  I thought of it as “swabbing the deck,” but showed remarkable restraint and did not talk like a pirate.

    Mapping: The routing website is now essentially — well, not done done, but the functionality is pretty complete. It routes, with a few glitches (but I added error handling so it doesn’t just choke without apologizing), it modifies routes based on user preferences for hills and visible recommended streets, and it can export the route as GPX; the final steps for website usability are to add printing capabilities for the directions, and add some explanatory content. (Finishing the job means building the real database — and finding a place to put it online.) I’m pretty happy with how this came out so far, it’s actually fun to play with.

    Listening: Not to eMusic, that’s for sure. I’ve used them for years to purchase music, and once they were both a good deal at a flat 49 cents a song (with no DRM: download it and it’s yours), and a good source for whatever I was looking for. Then in about 2010, they bought into some of the more mainstream catalogs, changing their price structure — more popular stuff became more expensive, some songs required you to buy the entire album — to accommodate the new sources. This actually  drove away many of the better and more obscure labels, leaving eMusic no better than any other generic source, at least in terms of selection. Now the major labels are gone again (I think), and the catalogs are mostly things I don’t care about. So every month I pay $15, which gives me $17-$18 in credit to use or lose that month, and I hardly ever even check in anymore to see their new offerings — and whenever I go there to search for something specific, they don’t have it. It’s time to move on.

    On the Home Front: We are busy researching ovens, in preparation for our new purchase.

  • A Thanksgiving Miracle

    So, flash back to several days ago: Anne was cooking things in the oven, and they were just going …wrong: bread was burning, cakes were coming out uneven, the oven was just acting funny. A quick check with the thermometer and she confirmed that the oven’s thermostat was malfunctioning, and a look inside the oven showed one side wasn’t even firing. Our oven was on the fritz.

    Our oven was on the fritz, that is, three days or so before we hosted Thanksgiving. Maybe it’s just the thermostat, but the last time the repairman came out (to fix the thermostat if I remember correctly), he said we needed a new oven — if we even got a repairman out before the holiday we would probably not be able to get parts,  much less a new oven, in time. A turkey, a ham, many side dishes and pies and cakes, and 17 guests coming — what to do?

    Well, first we panicked, but then Anne did a bit of research and we decided to cook the turkey in the wood-fired outdoor oven — the wonky indoor one, with some supervision, would work well enough for the (pre-cooked) ham as well as those side dishes that couldn’t be repurposed to range-top cooking, and guests would bring plenty of their own food. The only real hit would be the extra work of starting and running the fire.

    This mostly fell to me, and was a two day process. We got up early Wednesday and I got a fire going, and fed it periodically during the day while doing other prep chores. This fire was just a pre-heat to get some temperature into the core thermal ballast; all we cooked on Wednesday was the pizza we had for dinner. A little measurement told us the whole turkey, with a pan big enough to cook it, wouldn’t fit in the oven, so Anne split the bird in half as part of the prep, while I got the fire going again Thursday morning. At around 900 degrees we put the bird in to brown, then we put it back in again when the temperature was down around 500. It cooked in just under two hours, and it came out fabulous. Everything else was awesome too, and the whole day was a fun success.


  • On The Bridge: One Step Forward, Two Steps Back

    Talk about hedonic adaptation! A week or so ago I was sure I was far from ever being able to route on my web map using pgRouting, then a triumphant breakthrough, and now here I am, annoyed that it’s not perfect…

    My first problem is a data issue, and a recurring one in my mapping and routing life: dealing with bridges. Once I got the routing to work, I started to customize it with a separate “get_cost” function, which deals with ascent and descent (other criteria are coming), and that worked fine. Then I noticed that the routes seemed a little off, like they were avoiding what I thought would be the optimal routes, the main one being that it would do a lot to avoid a certain section of Broad Street. That’s when I remembered: there are several bridges on that section, and rather than following the elevation of the bridge’s road surface, my elevation data followed the contours of the ground below it, leading to large ascents and descents along that section.

    To solve the immediate problem, I changed the ascent and descent to be zero for the section containing the bridge — close enough to the truth, for that short a span on a mostly flat road. That made the routes in the vicinity more sensible, but what to do about other bridges?

    I think I have three options: I can either find the actual elevation data for the top of the road surface (using a “digital surface model” rather than “digital elevation model” and probably using LIDAR rather than satellite radar data), or I can assume that the bridge has a mostly constant slope, and calculate the slope from elevations where it attaches to the ground, or I can save myself a lot of work and just say “they’re flat, or flat enough to make no difference,” and make all ascents and descents be zero for bridges. I am still thinking about this…

    The second problem is a little harder to figure out, since it involves the PostGIS routing function I got off the Internet: when the beginning and end points of the route are on the same segment of road (ie there are no intersections between them), the function fails. I don’t know enough about Postgres functions to be able to solve this, so I may have to either live with it for a while, and contact the person who wrote the function for some help.

  • Excelsior!

    I thought this would take so much longer… I have a web map designed and working, and I was able to add generic routing functionality to it (via the Leaflet Routing Machine plug-in). Here’s a screenshot:


    A screenshot from my web map showing a path around town, using the Leaflet Routing Machine and my PostGIS/pgRouting database.

    The actual routing engine on the server side is done with PostGIS/PgRouting, accessed through a PHP script — Leaflet Routing Machine does not normally work with PostGIS,  but I found another plug-in (called lrm-pgrouting), which included a PostGIS function and some Javascript, to add this functonality. Unfortunately, the unmodified lrm-pgrouting plug-in requires something called Geoserver (rather than just a plain old PHP script) to talk to PostGIS, but it was easy enough to modify the plug-in to use PHP, and writing the script itself was the easiest part of the whole thing.

    Once I got routing up and running, I came across a few problems, mostly with how the directions are displayed: the display only showed street names and distances — that is, no turn information — and, for wherever the route continues straight on the same road through an intersection, there is an unnecessary instruction to “go straight.” Both of these were due to errors in lrm-pgrouting which I managed to fix to my liking.

    So, success! But this is using a generic routing function, rather than the “climbing vs busy road vs recommended commuter routes” function I ultimately want to use, so there’s some room for improvement. I also think I may want to abandon the Leaflet Routing Machine and do the input and display on my own. Also also, all this stuff currently resides only on my laptop; I have to find a host that will let me run a PostGIS database (among other things). I still have some work to do.


  • Housecleaning

    The warm weather finally broke (again) with this recent rainy spell, let’s hope it lasts but for now it’s nice and cool…

    While we were in Pittsburgh I kind of got fed up with my phone sending me messages about memory use — it wasn’t really all that close to full, but it was getting closer every day, for no reason I could see, and the messages were getting more ominous. I had already moved as many apps as I could to the external drive, my photos, music etc, there should have been very little on the internal drive at all, much less enough to cause problems.  What gives?

    I started looking through the folders on the drive, Googling their names and trying to find what was going wrong. Turns out (among other things) that my photos were being stored several times on my phone, over a thousand photos, each a few megabytes, and while my regular photo storage is on the external drive, the backups were filling the problematic internal one. Several minor changes to the settings, turning off “cloudagent” or whatever, and I recovered a huge chunk of storage space. I was so happy I deleted a bunch of apps I don’t use, freeing up another chunk. The best part? They’re staying freed up.

    Meanwhile, back home on the laptop… my hard drive has two partitions: one large partition where my old system was, and another one that holds the root of my current system, which is smaller than the first partition but by no means small. Unfortunately, it was also constantly growing, and I was down to like 25 gigabytes — which sounds like a lot, but the disk is old and on the small side, and still it’s 350 gigs. I’m down to less than 10 percent usable space?

    Once again, it was the cloud. I’ve been putting it off for a while, but I knew that the problem was my Dropbox folder, which was huge and growing (photo backups from my phone), and things would be a lot better if that folder was on the bigger, emptier partition. This required some work — the Dropbox preferences program wasn’t working correctly, so I had to fix that first, then the move itself took a while — but in the end, the move was a success, and I freed about 25G on my main partition.

    That got me excited enough to look at what was on my auxiliary partition, maybe there was even more I could free up… I have my old system backed up on a network drive, but the old home folder (about 65G) was still there, and all the useful stuff had been moved over to the main partition already. I couldn’t bear to just erase it, so I moved it all over to the network drive (where there are now two full backups, but with two terabytes it’s a drop in the bucket) and now I have tons of free space — 190G of 350G total — on my drive.

    I know, the excitement is palpable…

  • Some Photos of Our Pittsburgh Trip

    We were out in Pittsburgh over the weekend; Anne and I went with Ben to her nephew Mike’s wedding. An awesome time (though exhausting), hanging and partying with Anne’s siblings and their kids. We drove up Friday morning, the wedding was Friday night, we went to Mattress Factory, an art museum, on Saturday, followed by a picnic at Anne’s brother Bill’s place, and drove home Sunday. Boom!

    So anyway, here are some of the pictures I took.

  • Getting Things Done

    I woke up — late — this morning, with a full day of tasks and errands I wanted to get to: I had a noontime ride planned with Ted B, a friend from back in Manalapan days, and I wanted to get to some computer stuff (mapping, among other things). I was thinking of mowing the lawn, I had some laundry to get done, and, since we’re heading to Pittsburgh for a wedding this weekend, I wanted to get my packing and other preparations for the trip done ahead of time. I have a simple to-do list I keep, a list of the day’s chores and errands, which I update whenever there’s enough to do that I might forget something, and it was all there.

    But life comes at you fast… When I came downstairs Anne said “I want to show you something,” and brought me around the back of the house, where the concrete wall near the basement was all moist — keep in mind, it hasn’t rained here in more than a week. Uh-oh, we’ve had an intermittent problem with the downstairs toilet running, leaky flapper or whatever, a DIY fix-up that would be addressed soon enough, but this now looked like we had a serious leak working its way though the floor and out. Anne called the plumber, and I canceled my ride — she had to take her Mom to the doctor, so I was now on “wait for the plumber” duty.

    I also looked over my to-do list. The planned yardwork was basically weed-whacking and mowing, but the grass hadn’t been doing much growing, and meantime something had been digging in the new mulch (covering the crocus bulbs) so I swapped mowing out for sprinkling cinnamon and cloves around the garden — the Internet says that keeps critters away, we’ll see. I did the same for other things on my list, paring down and swapping out a few things, and managed to get all the things done for today’s modified plan.

    (The plumber came, declared it was most likely condensation from the constantly-flowing toilet pipes, and replaced the flapper for us. We’ll have to keep an eye on it to see if the problem goes away, but I’m hoping.)

    One thing I had on that list was to look at some productivity software I downloaded, something called Task Coach. This is a task and time management tool, almost to the  point of being a lightweight project management tool, where tasks and subtasks can be added, organized, prioritized, assigned prerequisites, tracked, etc. It has a lot of really neat features, including a few I was looking for — for a friend — about a year ago: Lori M, another grade-school buddy, put out a request for recommendations for some to-do list software, which had to include (among other things) the ability to add notes at the project as well as the task level. The moment has long since come and gone to make any recommendations, but Task Coach has this and more. It’s really good, except…

    When I was a kid, allergies precluded me from doing the lawn work at home; my brothers got that job instead, and I was the one who took care of the swimming pool. This meant chlorinating the water daily, maintaining the pump and filter, cleaning out the skimmers and skimming leaves manually, and, most important but also most time consuming, vacuuming the pool. It was kind of a pain but really did not require that much effort.

    Then one year, my parents got a Polaris automatic pool vacuum, sort of like a Roomba for the pool. It would run constantly — it was almost hypnotic to watch, like a slow-moving aquatic creature foraging — and my vacuuming days were over. Sort of: I basically traded the vacuuming job for the one called “maintain the automatic vacuum,” cleaning it out, disentangling it when it got caught in things, and even vacuuming occasionally when the machine didn’t work well enough. In other words, my effort level did not actually get reduced with the coming of this “labor-saving device.”

    It’s something I always remembered, and often used as a good, and surprisingly apt, joke/parable over the years, especially at work. And now here I am, looking at the same situation: beneath a certain level of scale or effort, brute-force methods often work better than the more elegant solutions, and I find myself wondering if I’ll spend more time curating my task lists with Task Coach than I actually save by being organized — Task Coach looks like it would only really shine when it’s managing more complicated projects than my daily chores. We shall see.

    In the meantime, I still have my simple, ever-growing text file. Whenever I feel the need, I start my morning (or end my evening) by writing up, at the top of the file, a list of the tasks that need doing that day. Then I can run through the checklist, marking them done as I go. I can add or remove tasks as it occurs to me to, and if I don’t get everything done — no big deal, I can reschedule it, or not, as I see fit. The file sits there on my desktop where I can see it whenever I go to browse — my usual worst time waster. Just knowing that I have a list of things that I need to do and check off seems to drive me a bit; it gives the day a little bit of an unpleasant sense of urgency, and days that I don’t use it feel deliciously unstructured (even if I do manage to get things done), but “to-do list days” leave me with a real a feeling of accomplishment. It’s the little things…

  • Some Photos From Our New York Trip

    I took a bunch of pictures in Watkins Glen and the Corning Museum of Glass, thought I’d show some of them here.

    In Watkins Glen, we camped at the state park, and there was a trail basically from our campground to the trails along the Glen, and then into town. It was cool and convenient, but the park was very crowded, overcrowded actually, and so it wasn’t as much fun as if we had the place (relatively) to ourselves.

    The Museum of Glass was also very cool, but it really wasn’t a museum about glass per se, with history of glassmaking, chemistry and physics of glass, etc, which I would have liked — they had that, but it was more geared for kids, and in the “kid section” — but more an art museum, and museum art history, where the common medium is glass. Early artifacts, modern creations… the craziest part was the gift shop, where I saw several items in the $10,000-$50,000 price range and fragile as, well, glass, just sitting out on pedestals where kids were running around and anyone could bump into them…

    Anyway, here are my photos