• Category Archives tech talk

  • Computers and programs, maps and GPS, anything to do with data big or small, as well as my take on the pieces of equipment I use in other hobbies — think bike components, camping gear etc.

  • Getting Things Done

    Posted on by Don

    I woke up — late — this morning, with a full day of tasks and errands I wanted to get to: I had a noontime ride planned with Ted B, a friend from back in Manalapan days, and I wanted to get to some computer stuff (mapping, among other things). I was thinking of mowing the lawn, I had some laundry to get done, and, since we’re heading to Pittsburgh for a wedding this weekend, I wanted to get my packing and other preparations for the trip done ahead of time. I have a simple to-do list I keep, a list of the day’s chores and errands, which I update whenever there’s enough to do that I might forget something, and it was all there.

    But life comes at you fast… When I came downstairs Anne said “I want to show you something,” and brought me around the back of the house, where the concrete wall near the basement was all moist — keep in mind, it hasn’t rained here in more than a week. Uh-oh, we’ve had an intermittent problem with the downstairs toilet running, leaky flapper or whatever, a DIY fix-up that would be addressed soon enough, but this now looked like we had a serious leak working its way though the floor and out. Anne called the plumber, and I canceled my ride — she had to take her Mom to the doctor, so I was now on “wait for the plumber” duty.

    I also looked over my to-do list. The planned yardwork was basically weed-whacking and mowing, but the grass hadn’t been doing much growing, and meantime something had been digging in the new mulch (covering the crocus bulbs) so I swapped mowing out for sprinkling cinnamon and cloves around the garden — the Internet says that keeps critters away, we’ll see. I did the same for other things on my list, paring down and swapping out a few things, and managed to get all the things done for today’s modified plan.

    (The plumber came, declared it was most likely condensation from the constantly-flowing toilet pipes, and replaced the flapper for us. We’ll have to keep an eye on it to see if the problem goes away, but I’m hoping.)

    One thing I had on that list was to look at some productivity software I downloaded, something called Task Coach. This is a task and time management tool, almost to the  point of being a lightweight project management tool, where tasks and subtasks can be added, organized, prioritized, assigned prerequisites, tracked, etc. It has a lot of really neat features, including a few I was looking for — for a friend — about a year ago: Lori M, another grade-school buddy, put out a request for recommendations for some to-do list software, which had to include (among other things) the ability to add notes at the project as well as the task level. The moment has long since come and gone to make any recommendations, but Task Coach has this and more. It’s really good, except…

    When I was a kid, allergies precluded me from doing the lawn work at home; my brothers got that job instead, and I was the one who took care of the swimming pool. This meant chlorinating the water daily, maintaining the pump and filter, cleaning out the skimmers and skimming leaves manually, and, most important but also most time consuming, vacuuming the pool. It was kind of a pain but really did not require that much effort.

    Then one year, my parents got a Polaris automatic pool vacuum, sort of like a Roomba for the pool. It would run constantly — it was almost hypnotic to watch, like a slow-moving aquatic creature foraging — and my vacuuming days were over. Sort of: I basically traded the vacuuming job for the one called “maintain the automatic vacuum,” cleaning it out, disentangling it when it got caught in things, and even vacuuming occasionally when the machine didn’t work well enough. In other words, my effort level did not actually get reduced with the coming of this “labor-saving device.”

    It’s something I always remembered, and often used as a good, and surprisingly apt, joke/parable over the years, especially at work. And now here I am, looking at the same situation: beneath a certain level of scale or effort, brute-force methods often work better than the more elegant solutions, and I find myself wondering if I’ll spend more time curating my task lists with Task Coach than I actually save by being organized — Task Coach looks like it would only really shine when it’s managing more complicated projects than my daily chores. We shall see.

    In the meantime, I still have my simple, ever-growing text file. Whenever I feel the need, I start my morning (or end my evening) by writing up, at the top of the file, a list of the tasks that need doing that day. Then I can run through the checklist, marking them done as I go. I can add or remove tasks as it occurs to me to, and if I don’t get everything done — no big deal, I can reschedule it, or not, as I see fit. The file sits there on my desktop where I can see it whenever I go to browse — my usual worst time waster. Just knowing that I have a list of things that I need to do and check off seems to drive me a bit; it gives the day a little bit of an unpleasant sense of urgency, and days that I don’t use it feel deliciously unstructured (even if I do manage to get things done), but “to-do list days” leave me with a real a feeling of accomplishment. It’s the little things…


    day by day tech talk

  • Go, But At What Cost?

    Posted on by Don

    Listening: “Golden Brown” by the Stranglers

    Well, so much for my previous experiment — don’t compartmentalize, just write — the real task is to actually take the time to write something. I haven’t been motivated lately  to do any writing, but in my defense there have been a lot of things to write about, which might have caused a bit of vapor-lock. Among other things, I’ve still been volunteering at the Canal Museum and at CAT, we did another overnight trip up to Jim Thorpe, and I’ve been exploring mapping and routing again.

    Listening: “Colossal” by Wolfmother

    Listening: “Beautiful Red Dress” by Laurie Anderson

    ON THE BIKE:

    We did another trip up the towpath: Anne and me, Sarah A and Dianna H. We rode to Jim Thorpe, lunching in town and camping overnight at the lake. Breakfast in town, then Anne continued north on her own from there while Sarah and I did the return trip (Dianna met her husband in town and got a ride home). Anne’s destination was Watkins Glen, and I caught up with her by car there a few days later. I brought my road bike, but we didn’t do much cycling, just some hiking at the Glen, then a trip to the Museum of Glass in Corning on the way home.

    Listening: “Help Me Mary” by Liz Phair

    I kind of got the mountain biking itch again: I did a ride on some seriously “old school” trails, with Greg H up in the Poconos near his cabin, probably my favorite ride of the year so far (except for a persistent creaking out of my pivots). Got the pivots fixed, rode Nox on a weekday with Anne, and did Deer Path/Pine Tar in Jim Thorpe Sunday. I’ll be doing a towpath ride later today.

    In between these things, I also took the Cycling Savvy course. Really fun, and though it covered a lot of the same ground as Road 1/LCI, I came away with more than a few choice new insights. I also rode across the Hill-to-Hill Bridge with my sister-in-law, which should have been a crazy idea, but by the time we did it (as part of the course), it was easy enough to be a bit anticlimactic.

    Listening:“Drumming Song” by Florence + The Machine

    We also managed to do some riding as part of Lehigh’s Car-free Day.

    Listening: “Make You Mine” by Heather Nova

    All this music just lets you know how slowly I write, and also how many in my “Favorites” playlist are female vocalists…

    READING

    I re-read The Mathematician’s Shiva recently, as well as all the “Expanse” books (which didn’t hold up to a re-read as well as I would have liked), and China Mieville’s Iron Council (ditto), the most recent new book was Walkable City by city planner and walkability expert/advocate Jeff Speck. Interestingly, he once was commissioned to do a study of Bethlehem, and gave a talk at Lehigh about his findings. (The town skipped over a bunch of his advice, but they did incorporate at least some of it, some parts more slowly than others.) It was fun (at first) to see him name-drop Bethlehem, and CAT, likely referring back to his study, but it became annoying after a while since it was mostly examples of what we were doing wrong…

    Case in point: I had just finished the “Cycling Savvy” course when I got to Speck’s critique of “Vehicular Cycling,” where bicyclists are trained to bike (on the roads) as drivers of vehicles — in other words, “Road 1” and “Cycling Savvy.” His contention, and there is some merit to it, is that while this may help an individual graduate be safer, it makes cycling grim and scary, a turn-off, thus reducing the number of actual cyclists on the road — and since the biggest driver of cycling safety is not cyclist skill (or wearing a helmet or whatnot), but the number of cyclists on the road, the vehicular cycling approach actually reduces general cycling safety. Oh well, he has a point, but I still liked biking over that bridge.

    Listening: “Old World” by The Modern Lovers

    FUN WITH POSTGIS

    I’ve been playing with a new project recently: building a web map for cycle commuters in the Valley. In the end it will show the major Lehigh Valley towns, and the locations of the major employers, and recommended routes that a cycle commuter might use to get around; I used Leaflet to get these basics down, but then I thought that what the map really needs is routing, and I thought it would be best to build a custom routing engine using PostGIS and pgRouting.

    Listening: “Wildewoman” by Lucius

    So, I’m back to my routing kick; this will be part 2 but I’ll be abandoning my previous project in favor of the web map.

    Listening: “Twenty-first Century Schizoid Man” by King Crimson

    First step (of many) was to make sure I had pgRouting installed properly along with PostGIS, and they both were, no problem. Next up was to build my road network — for right now I’m working on a smaller area, a part of Bethlehem City. I got the road data from OpenStreetMap and used the osm2pgrouting utility to get the roads into the database. So far so good, and the whole process was surprisingly easy.

    Using the routing functions took me a while to work out, but in the end they were also pretty straightforward. PostGIS/pgRouting seem to be easier to use, and easier to do sophisticated things with, than the original QGIS networking utilities.

    Listening: “White Unicorn” by Wolfmother (oh no the same band again!)

    Two things in particular came more easily: dealing with one-way streets, which I ignored in the first project since it seemed more trouble than it was worth (you could always walk your bike) and the actual “cost” of cycling.

    The basic idea behind routing is to find a path through the network that minimizes some function, the total of the “costs” of moving from each individual point to point within the network. The default cost function for my first project was distance (the default, and by far the easiest thing to do), which is a pretty good cost function as far as it goes. But with bicycling, elevation changes could also play a major role, and with pgRouting it’s easy enough to define your own costs.

    Listening: “Funkytown” by Lipps Inc.

    So I decided to consider total ascent, every meter climbed, as part of the cost; I found some studies that cyclists generally take a meter of ascent as equivalent to eight meters of travel — that is, you  might go eight meters longer to avoid another meter of climbing. I also thought that grade would affect that eight meters, and found another online study that multiplied ascent by a factor proportional to some power of grade. I eventually opted to go with a geometric factor, doubling elevation cost every 5% change in slope.

    Listening: “Strangeness and Charm” by Florence + The Machine

    I’ll get into it some other time, but I got QGIS to break up my network for finding ascent, and SAGA to assign elevations from DEM data, then wrote a Python script to extract and calculate the elevation costs. Took some doing, mostly dealing with my own typos, but finally I got the whole thing to work, and it routes beautifully — in QGIS, on my machine.

    Listening: “There There” by Radiohead

    I learned a few interesting things about elevation along the way. A sanity check of my cost results showed some anomalies, especially on 8th Avenue — it turns out my original “elevation data” was raw Space Shuttle radar data, and it picked up the top of the old Martin tower which screwed up nearby elevations. (This difference between ground elevation and radar/lidar elevation readings, the realm of buildings and trees, is — according to the Internet — very important to telecommunications people, who call it “clutter.”)

    So for my second iteration I used actual DEM data  (“digital elevation model,” the elevations at the surface of the Earth, as if it were scraped clear of buildings and vegetation), and that fixed the Martin Tower problem but revealed another one: bridges, my nemesis… I’ll have to figure out how to adjust elevations on bridges so they don’t follow the depressions (creeks, rivers) they jump over.

    Listening: “Furr” by Blitzen Trapper

    Well, that’s it for now…

    “But I still dream of running careless through the snow,

    And through the howling winds that blow

    Across the ancient, distant flow

    To fill our bodies up like water till we know.”

    … now let’s ride!

     


    day by day tech talk the sporting life 📎and tagged

  • Huckleberry Hounds

    Posted on by Don

    We just got back from a trip to Brady’s Lake to pick blueberries, the high-bush kind that may or may not actually be huckleberries rather than blueberries — these are big berries, and huckleberries are supposedly smaller (and grow on evergreens), so I’d still go with blueberries. Two hours of picking, and I think we got more than three quarts — and it looks like there’s a lot more to be had as the season progresses. No bear sightings, though…

    Yessssssss! Successsssss! The keyboard crapped out on my laptop a few weeks ago, with the “S” key sticking more and more — very frustrating. I finally brought it in to a repair shop yesterday, where I had the keyboard replaced, as well as the insides cleaned (like, the fan/air vents) and the heat sink refurbished. I thought it would be days, but he called in about an hour saying it was done, and I picked up my cool-running, feels-like-new laptop that afternoon. Sweet!

    No Bearings On The Case We rode to Anne’s orchestras summer picnic yesterday, along with Shari, an orchestra-mate who lives in our neighborhood. It was a reasonable distance, maybe 14 miles one way, but we were going slow and I wanted to be wearing normal clothes/shows when we arrived. So, I took the Iguana, for its first big ride since I re-worked the headset last Tuesday at the CAT office. That was a bit of a disaster: I took off the stem, lock nut and spacers, then as I was taking the upper race off all the ball bearings fell out of the bottom bearing and scattered bouncing across the floor. Turns out the seal was gone, and the bearing cage was mostly gone, so once I loosened the fork there was nothing holding them in. I managed to retrieve most of them and replace the missing ones, put them into what was left of the cage with a whole lot of grease to act as “glue,” and put it all back together. This is a temporary fix until I can get a new bearing/cage assembly, but despite everything, the bike’s steering feels better now than it has in years.


    day by day nature journal tech talk the sporting life

  • Dissolution

    Posted on by Don

    Beautiful weather so far this week, but I haven’t been able to get out quite yet. I did get to do some trail work at Sals on Sunday, while Anne did some hiking with Deb, so there’s that. Tomorrow is my volunteer day at the Canal Museum, and I’ll be riding over.

    QGIS vs GRASS vs SAGA: I’ve been trying to work with some more geoprocessing tools, specifically Dissolve (where several regions that touch or overlap are combined into a single shape — their borders are “dissolved,” hence the name). I set up some small test shapes and tried dissolving them. It worked perfectly well with GRASS, but with QGIS the two shapes did not actually become one. I tried using SAGA, and got the same failure — WTF? I thought at first that both programs were broken, but then found out that the two shapes weren’t actually adjacent, there was an infinitesimal sliver of empty space between them; technically, they didn’t actually have a common boundary for these two programs to dissolve, while GRASS apparently has a built-in tolerance to handle slivers like this and just ran with it. The more you know…

    More Maps: Check out Walking Purchase Park on the MTB Project. There were a few trails already done, but I added a bunch more, and it should now hopefully be a useful guide to the trail system there.

     


    tech talk

  • Winterized

    Posted on by Don

    There is still snow on the ground, but what’s been falling, overnight and all day so far, is freezing rain. Things have quieted down since last week’s big move, so today we slept in and are doing indoor hobby things: Anne is sewing upstairs, and I spent the morning getting various QGIS add-ons up and running.

    This is really a story for another post, but I’d had trouble running SAGA through QGIS (when I was doing those online courses), even though I had SAGA running on my laptop. There had been some upgrades since that misadventure, so I gave it another try the last week, but it was still broken. Then I got sucked in: I sat down and read the error logs, realized that QGIS was calling the SAGA routines with outdated command line parameters, and managed to batch fix the QGIS end of it — 250 commands fixed with a simple, multifile search-and-replace, two minutes and I was done. And it worked!

    I got so excited that I SAGA’ed the shit out of a bunch of data, realized I had no current use for the tool, and decided to install and play with a bunch of other processing add-ons, like hydrological (flow and runoff) modeling with TauDEM, satellite image processing using Orfeo, and statistics using R, which I already had installed. TauDEM and Orfeo seemed to work fine, but R didn’t so I spent some time getting that figured out. I think I like the idea of these tools more than actually using them, because I get a lot of satisfaction out of getting them to work.

    Anyway, there will be no outdoor sports activities today, or even outdoors at all given the general nastiness in the air (and slipperiness on the ground). We’ll probably go out to the usual later with John and Donna — we went out to Easton last night with them, but found Two Rivers and Porters both completely  full up, and Black & Blue still closed, so in effect we did the “Beermuda Triangle” without any beer, drove back to Bethlehem, and went instead to the Beef Baron, which was a nice change of pace.


    day by day tech talk

  • Another Win Is Another Win

    Posted on by Don

    Wooo! Did the Superbowl of Chili yesterday with Eric and his neighbor George, and we had a beer-and-chili-fueled blast, if you’ll pardon the pun. No morning run, but that’s fine, Anne and I may do a hike in Jim Thorpe this afternoon.

    Meanwhile… My WordPress page (you’re looking at it) uses a theme called Raindrops, which I like a lot of course, but it seems to be a hobby project of some guy in Japan, and there is a persistent error in his code. Specifically, he’s missing a closing anchor tag in the function that creates “featured images,” so that on a page with a featured image, everything after the image is part of the link back to the image’s page. It’s easy enough to find and fix (I did it) but my attempts to contact the author about the error got me nowhere.

    Like I said, fixing this error is easy enough, but the Raindrops theme gets updated a lot, and when it does, the file with the corrected code gets overwritten (with a newer version of the buggy code). Every update would find me going in and manually re-doing the fix — it got old.

    WordPress themes are built so that one can refer to another, as in a parent theme and a child theme, and the child would inherit from the parent theme all its attributes (layout, behavior etc) except those specifically overridden by the child theme, which can also add functionality etc. This seemed to be ideal for my needs: I could create a child theme, which would only override that one bad function with my corrected version.

    My first look at this, several months ago, made me think it wouldn’t work: the original theme’s function coding has to be set up to allow itself to be overridden (each function is tested to see if it does not already exist before being defined), and I could have sworn that my parent theme was not set up like this. I looked again more recently though, and sure enough, everything was good. (Maybe I was mistaken in my first look at the code, or maybe this was added in one of the theme’s many upgrades.)

    So I’m good to go, and late last week I sat down and made my child theme, and that all worked out fine. The real moment of truth came this weekend: the parent theme had an upgrade, and the upgrade didn’t break my website. Success!


    day by day tech talk

  • Experiments in Routing, Part 2

    Posted on by Don

    This is my second post in a series, where I report back on my results from playing with various ways to use routing, in QGIS and related programs. My immediate task is to identify those cycling amenities that are nearest to access points along the Lehigh Towpath. You can read Part 1 (the introduction) here. In this post, I’ll be using the QGIS built-in Network Analysis Library. Follow along after the break…

    Continue reading → Post ID 795


    tech talk 📎and tagged

  • Experiments in Routing, Part 1

    Posted on by Don

    This is the introductory post for a hopefully four-part series about using QGIS to find the shortest path between two points, not shortest as the crow flies, but following a given network of roads. This is called routing, it’s what’s Google Maps and other mapping software uses, and it relies on graph theory and network analysis to do its job. I’ll talk about the what and the why of this little experiment here; the how (for three different versions of how) will be the subject of subsequent posts.

    UPDATE: Part 2 can be found here.

    The reason I’m looking at all this goes back to my interest in cycling tourism, and my attempts to identify cycling-accessible amenities — convenience stores, restaurants, hotels, that sort of thing — along the Lehigh Towpath. My first attempt (you can find it here) basically looked at a region, within a mile (as the crow flies) of one section of the Lehigh River, and searching within that region for the amenities I was interested in. That was an interesting project in its own right, but, as I said in my earlier post, it didn’t really solve the right problem:  there are many places within a mile, or even a quarter mile of the river, that are not anywhere near accessible from the towpath: they may be on the wrong side of the river, say, or not near a towpath access point. To be considered accessible, the points of interest would need to be within a mile (or whatever arbitrary distance I end up choosing), by road, of an access point on the towpath.

    map of roads near lehigh river
    Data gathered, and almost ready for routing.

    I didn’t really have a plan to make this happen yet, but with or without a specific plan, I figured my first order of business was to get the information I would use. That previous analysis used Google Maps, but I felt that their data was a bit encumbered (in terms of my rights to it), and it seemed that Google didn’t play as well as I’d like with QGIS anyway, so I decided to use the data available through Open Streetmap, for both the road network and the set of amenities. (I already had a collection of the towpath access point locations left over from a previous experiment.) I got those sets of data, and massaged them so that I only had the parts that fell within a mile of the bike paths in the Lehigh Valley. This gave me the data seen to the right, where the aqua lines are the road network, the red lines are bike trails (the towpath, plus the Palmer Bike Path), the red stars are trail access points, and the orange dots are the amenities (restaurants, fast food etc).

    (One note about the road network: You probably can’t see it at this resolution, but I made a point of excluding roads that are not practical/legal/safe for cycling, like US-22, I-78 and a few others. There are also a number of places, like the New Street and Hill-to-Hill Bridges, where roads or the trail are connected via stairways to the bridges above; after our own struggles, a few years ago, with stairs and fully loaded touring bikes at the Ben Franklin Bridge, I decided to also exclude stairways from my network.)

    So that gets us the data, what about the analysis? My first thoughts were to see if I could find all the points on the road network that were a mile away from an access point, then connect the dots to define a region, and then find all the amenities within that region. My second thoughts were that this approach would put me back in the same situation as my first attempt, since I could easily find roads that were not reachable within that region, such as bridges. (Bridges became my nemesis for a while.) I eventually decided that my best strategy would be to find the shortest route between each access point and each amenity, and select from the amenities based on the lengths of the routes I found.

    To perform the actual routing analysis, I have three options:

    In terms of a learning curve, I have some experience with networks in GRASS, and I feel at least a little comfortable with Python (and copy-paste, with scripts I find online), so pgRouting will probably be the most difficult for me to pick up. Meanwhile, the Network Analysis library can use the data I already have, but Open Streetmap deals with road networks in a way that’s not directly compatible with either GRASS or pgRouting — their topological models are different, but that’s an issue for a future post. I would have to either re-import the road network to get it to work with pgRouting, or further process the one I have for GRASS.

    Each one of these approaches will be the subject of its own post. Given that the Python approach is not the hardest, and my data is already in the form I’d need, I am going to try my hand with the Network Analysis library first. Stay tuned for Part 2, whenever…


    tech talk 📎and tagged

  • QR Code Update

    Posted on by Don

    I still have no idea what’s going wrong with scanning photographs of QR codes  (other than, say, generic image quality issues inherent in the process), but I’ve sort of abandoned the whole QR thing. The obsession ran its course, and there was also this:

    We went out last weekend with John and Donna, and also a friend of ours who is a programmer. She asked me about my recent projects and I said I was intrigued with QR codes, and she said something to the effect of “Oh, aren’t they a bit passé?”

    What?!?? I asked John, and he also felt that they were a technology that seemed promising maybe a few years ago, but eventually the buzz faded as they were seen to be superfluous — users could write information (or capture the info another way, like near field communication) as easily as they could use a phone to scan and capture it from a QR code.

    I went home and did a little Googling and — except in the marketroid world where it definitely seemed passé — the situation wasn’t nearly as dire as the picture my friends painted, but what I saw online did make me reevaluate their usefulness, to take stock as it were, and my interest, already waning, disappeared.

     


    tech talk