• Category Archives tech talk

  • Computers and programs, maps and GPS, anything to do with data big or small, as well as my take on the pieces of equipment I use in other hobbies — think bike components, camping gear etc.

  • Another Day’s Useless Energy Spent

    Posted on by Don

    I make online maps of Shawnee’s Road Scholar bike rides, to help the sag-wagon drivers find things like drop-off and pick-up locations, lunch spots, and other places where they’d meet up with the riders. (I did this after a driver searched for directions to the “D&L trailhead in White Haven,” but there were two trailheads in the town and he went to the wrong one. No biggie, the trailheads were less than a mile apart, but this kind of ambiguity looked like a problem that needed solving.) My first iteration was just a list of all locations in Google Maps’ “My Places,” but now I have a separate, full-blown map (again in Google Maps) for each ride, showing the route itself as well as marked points for the drop-off, pick-up, lunch, and all practical access locations, each with appropriate symbology.

    Here is an example for the Lehigh Towpath ride:

    (You can view the legend by clicking the button to the left of my name at the top of that map, or view the bigger map by clicking the square at top right.)

    I use Google Maps because that’s what the drivers use on their phones for navigation, and this makes it easy for them to navigate to any of my marked points without having to leave the Google Maps universe. To make the maps available, I generate QR codes for each map link; the driver can get the map and open it almost effortlessly with their phone’s barcode scanner. They don’t always use it — they weren’t as dazzled by my geeky brilliance as I was hoping — but hey I was impressed.

    The map data comes from a QGIS project I keep for all of the rides we do, or might do, as part of Shawnee’s Road Scholar bike program. My workflow is: select the route for the chosen ride and add it to a KML file, then select the relevant points of interest and append them as another layer in that file. (KML is “Keyhole Markup Language,” Google’s XML-based geographic data format.) I then import the KML file into Google Maps, and I’ve got my route and points on their own new map. A little bit of styling and I’m done — easy peasy! It takes about 5-10 minutes for each map. (The QR codes are generated separately using Jasper Reports.)

    And Here My Troubles Began

    I have these maps made for all the rides we’ve done, but I have a whole bunch of other rides, either unused or not fully developed, which don’t have their own maps yet, so there are more maps to do. Making each individual map is pretty easy, but I don’t always get the symbology (or the data associated with each feature) exactly the same each time, and I think that automating the task will help with consistency; also, if I start creating a lot more potential rides, the automated map-making could be a potential time saver.

    To automate my workflow, I decided to break it down into three basic tasks:

    1. extract the data I’d need from the database,
    2. use that data to generate a KML file, and
    3. import the KML file into Google Maps. (OK, this last step is still pretty manual…)

    This is the project I took with me to Colorado, working on it whenever we had an hour or two of downtime — something I can get away with when I vacation with a bunch of readers…

    The first step was pretty straightforward, though the SQL (especially for the points of interest) grew into a behemoth, with multiple CTE’s and a whole lot of joins to get things the way I wanted — setting it up just so took a bit of work, but like I said, the task was straightforward.

    The next task was more convoluted, and involved a learning curve — I decided to use Python to generate the KML, and this meant using several new packages: geopandas, which makes working with data a bit easier, and simplekml, which, well, does the KML part. Geopandas was pretty easy to use, but simplekml was quirky, maybe a bit too simple, and in the end I actually had to modify the source code to make it work right.

    But, in the end I got my KML file (technically, a KMZ file, which included the icons I’d need for styling), and I was able to import it to both Google Earth and Google Maps — the only problem was that Google Maps ignored all my styling! (It worked fine in Google Earth.)

    So, somewhere between steps 2 and 3 there was a glitch, a mismatch between what “styles ” meant for KML, and what Google Maps actually used. I spent a good part of the last few days looking at different versions of KML and KMZ files, seeing what worked, or didn’t work, with Google Maps, and I think I have an idea of what’s going on:

    Google Maps — or at least, the “My Maps” part of Google Maps — ignores most of the style information, like “use this icon and color,” and seems to code the style information, what they actually use, directly within the style ID’s: a style ID of “icon-1567-9C27B0-normal” would mean “use specific icon #1567, with color #9C27B0 for the normal (un-highlighted) icon.”

    So OK, first mystery solved, but what does that get me? I suppose I can use Google’s hard-coded style ID’s, but simplekml does not allow you to specify ID’s or change them, so it looks like that package isn’t as useful as it needs to be, and other Python KML packages seem… not simple.

    My next steps? My original plan was to code the entire task in SQL, and I may fall back on that, or I may do something using PHP. I might even take some text-based approach (like I did with geojson a few years ago), since KML is at bottom just a text file. But the truth is, I’m really just going to put this away for a while and let it marinate.


    tech talk 📎and tagged

  • Indian Paths Update

    Posted on by Don

    I’m still cruising along on this project: I’ve got just over 110 paths in the database (of maybe 150 total), about 130 towns or other path endpoints, and 92 motorway routes. I have added no actual paths yet, but the motor routes are starting to look like a real network.

    My current plan is to parse the book three times: once (this time around) to capture the paths, path endpoints, and motor routes; once (the final, and probably most difficult, round) to try and develop the original foot paths; and in between these rounds I will go through the paths/chapters and try to capture all the cross-references between them.

    I noticed early on that there were a lot of things like “this is an extension of that other path,” “so-and-so path also goes by this name,” “this path intersects with these others,” and such like throughout the text; the path descriptions are festooned with these kinds of cross-references.

    (I also finally picked up on the fact that paths without a path/chapter number are not actually part of the previous chapter, but are basically “chapterless,” just the next path name in alphabetical order. They act sort of as placeholders, the alternate names of other, more fully fleshed-out paths — that is, more cross-references.)

    I want to hold on to all this cross-reference information in my database, so I set up a bridge table to work something like a resource description framework, with the referring path as the subject, the referenced path as the object, and for the predicate I would use a description of the relationship type, such as “[subject path] is a continuation of [object path],” “[object path’s name] is an alternate name for [subject path],” “for more info see [object],” and so on. I now have all of this set up and ready to go, but before filling it in with information I want to have all the paths already in the database. Soon…

    Meanwhile, the details, of each path or town I add, have all been real eye-openers. I often do a little internet research on each town, or village, or Native name I come across, and each bit of info, each piece of the puzzle is another portal into that era.


    looking back tech talk 📎and tagged

  • New Bike!

    Posted on by Don

    My writer’s block continues, but I thought I’d pop in to announce that I finally got my new bike, a Kona Sutra SE. This is a bike seriously designed for touring: fenders, a triple ring, (mechanical) disk brakes, bullet-proof tires, bar-end shifters, a Brooks saddle, and a rear rack, with frame attachments for a front rack as well — and oh yeah, steel is real, baby!

    bicycle
    My New Kona Sutra SE

    I picked it up Thursday night from Cutters Bike Shop, where it had just arrived; they knew I wanted one, and they called a week or so ago to say one was on the way, and then it was there at the shop, and so was I… I gave it a test ride with panniers, then rode it home — it doesn’t fit on the roof rack. Yesterday was its real maiden voyage, a bakery ride up to Nazareth.


    tech talk the sporting life 📎and tagged

  • Native Paths Update

    Posted on by Don

    I kept at it, and am now about a quarter of the way through the trails — the motorway parts, at least — in my Indian Paths of Pennsylvania project. I have a pretty good idea of how the book is organized now, and came up with a pretty decent workflow that gets me through a single path in just under an hour. I do one or two a day. It’s pretty easy to get absorbed, trying to find the tiny old roads and landmarks based on their descriptions in the book, and I’ve been totally sucked into the history of that Colonial-Revolutionary era. (I picked up Mason & Dixon again, since it goes right through the middle of that time and place.)

    I also think there will be some epic rides this summer, based on these routes — I’ve been drooling over some of the scenes I see in Google Street View.


    day by day looking back tech talk 📎and tagged

  • N+1

    Posted on by Don

    I knew it would come to this sooner or later — I’m in the market for a new bike. There’s nothing wrong with any of my other bikes, but none of them are really touring bikes, and I’ll be joining Anne on a trip this summer. Fully self-supported, front and rear panniers, camping in the Rockies — the works. (I hated “touring” every time I’ve ever done it, but I suspect that that’s really an equipment issue — I do enjoy our towpath “bikepacking” trips.)

    Anyway, I’ve been doing some research, and what I think I need is:

    • a bike with a “touring” frame — low bottom bracket, longer wheelbase and chainstays,
    • a triple chainring,
    • mechanical disk brakes, and
    • a few add-ons that would be nice if they came stock, like racks and fenders.

    There are a few bikes that I think might fit the bill, ones I found in several “best touring bike” listicles, namely the Trek 520, the Kona Sutra SE, and the Surly Disk Trucker. The Trek looks to be impossible to find anywhere right now, and the Surly only seems available (sight unseen) via the Internet, but I found a Sutra SE at a local bike shop, and it looks like my size. I need to do a bit more search and research — I’m also looking among the world’s used bikes — but I think I can already see how this will shake out.


    tech talk the sporting life

  • Foiled Again!

    Posted on by Don

    I have a love-hate relationship with Paul A. W. Wallace’s Indian Paths of Pennsylvania. I love reading the individual chapters on each path — their descriptions, and the accounts of them in the letters and diary entries of early Colonial explorers, but any hard look at the specifics and the trails themselves become frustratingly vague. This is all the more frustrating because the information looks specific and authoritative enough, until you take that close look…

    Some of this is because the original information is vague — nobody was tracking their steps with a GPS back then — so the actual trail location is not perfectly known, and partly it’s because the trails themselves are long gone (though some are at least partly followed by modern roads), so it’s hard to search them out without trespassing, but there also just seems to be some missing ingredient needed to define a trail network.

    A few years ago I thought that this last part could be solved with a little bit of GIS detective work, so I started a QGIS project to define the trails and see about building a network, but I sort of ran out of steam — I basically foundered on the vagueness of the trail descriptions. I did one or two in the Lehigh Valley, and realized that the sleuthing needed was a lot more laborious than simple data entry, and the project languished after those first few paths.

    I was thinking about all this again recently, and realized that there is a critical first step I ignored: the book serves primarily as an automotive guide, with detailed instructions for driving in the vicinity of each path. I also thought that if I broke the task down to a set of database tables, I could link these auto routes to their various paths and book chapters . (Some trail chapters actually describe multiple trails and subtrails, while some motorway descriptions continue across multiple chapters, so many-to-many relationships abound but that’s what databases are for. Furthermore, most of the trail chapters have a start and an endpoint, yet more data I can use to cross reference.)

    This scheme fell apart within the first few trails. The very first trail, the “Allegheny Path,” has Philadelphia as the start point and “Pittsburg and Kittanning” as the endpoint — so which is the endpoint? Apparently neither, because the trail is only described as far as Harrisburg; the “Allegheny Path” chapter ends with references to several other trails (different chapters, in other words) heading West from Harrisburg as possible continuations. So OK, I can deal with this: my endpoints are really Philadelphia and Harrisburg, and I’ll stuff the rest of the info into my “description” column. (There is a second path listed in that first chapter, but it is little more than a historical aside and a reference to another path/chapter. This is going to get tricky.)

    Luckily the motorway for the Allegheny Path is easy to follow. I used an open routing plugin to follow along a bunch of control points, and voilá I had my linestring. This ain’t so bad!

    The very next chapter, I ran into motorway difficulties: the route description made no sense. Either the routes were not prepared with adequate ground-truthing (unlikely, though I was starting to feel uncharitable), or the roads (and their designations) had changed at some point in the 55 years since the book came out. This seems the more likely explanation, since I-80 goes right through the area in question, was only finished in 1970, and probably changed a lot of things in its wake. I actually found the Wikipedia article on the Bald Eagle Creek Path more useful.

    So I’m back to deciphering and making judgement calls rather than strictly converting the information from one format to another, even for these road descriptions. I didn’t expect this project to be done in an afternoon, or even a week or so, but “going to take forever because I’m not really working on it” is now closer to my expectation.

    (Note: I found that someone already took these paths and put them into a GIS, but it’s on PA-Share and that’s proved difficult to work with — and deliberately limited, unless you pay — so far. We’ll see…)


    looking back tech talk 📎and tagged

  • Oh No Not That Again (Part 2 of 2)

    Posted on by Don

    So I was playing with the commuter mapping program the other day after doing some simple maintenance, just finding routes from here to there, and it started bothering me again that I could not route onto the towpath from Sand Island — the network was incorrect, it had no intersection from Main Street onto the path.

    I get my road data from OpenStreetmap, and I know that, in OSM, the trail is properly connected at Sand Island — I fixed that myself years ago, but never went through the rigamarole of updating and rebuilding my network. It didn’t seem worth the work for such a small change. There is another way to make that change though: I could modify my existing network, but that always seemed like it would be even more complicated and difficult than rebuilding from scratch.

    But would it be? The task really boils down to two things: adding a node where I want the new intersection to be, and then splitting the newly intersecting roads in two at the intersection point. Adding a node is easy enough, but splitting a road has a lot of moving parts — each of the two new road segments has to be assigned about 30 attributes, some of which they can inherit directly from the original road, others basically pro-rated from the original road based on the new road segment lengths, and yet others related to connecting the new road segments to the new node. It’s straightforward, but there are a lot of small, tedious calculations to perform and keep track of. Sounds like a job for the computer…

    What I did was write a PostgreSQL function that takes the node and the road, and returns two new road segments. I also wrote a wrapper script to update the network by calling this function. (I decided to just add my new nodes “by hand.”) It mostly works, though in one test case it didn’t split the road exactly where I thought it should (no idea why), and the new network routes like a champ.

    This isn’t a substitute for rebuilding the network: this is a quick fix for a small problem, and the the pro-rated attributes especially are a hack, an approximation; I can easily see situations where pro-rating say, ascent/descent data would be inaccurate. But this is fine for now.


    tech talk 📎and tagged

  • Oh No Not That Again (Part 1 of 2)

    Posted on by Don

    I’ve been looking at my Lehigh Valley bike commuter routing project again.

    I decided to update the recommended routes with additions based on some of our recent CAT rides, and found that the line geometries representing the various routes were missing. It’s no biggie, some things didn’t survive those destructive “upgrades” I did a few years ago, and the actual recommendation info is stored in among the road network data anyway.

    But, I still had the old routes as GeoJSON files, and it’s easier to work with them as geometries in their own right than as attributes on the road network, so I added them back into the database. Then I added that new route (Cedar Street, which parallels Union Boulevard but is much quieter), and used it to update the network. Piece of cake!

    I also decided to tackle the problem of updating the network paths themselves, which is not so much a piece of cake. I get the roads from OpenStreetmap, and there are mostly automated tools to build a routing network from OSM road data, and that’s followed by a whole lot of additional data massage to put it in the form I use. But the underlying OSM data isn’t always accurate — roads don’t go where they are supposed to, intersections don’t actually connect, that sort of thing. I would find a lot of this out after building the network, but the task of editing the network, once it’s built, is so onerous that my preferred method has been to fix the issues in OpenStreetmap, then just download the roads and rebuild the whole network from scratch — also onerous, but slightly less so.

    Anyway, I planned to make this a part of the usual site maintenance if this ever went live: maybe once a month I would download the OSM roads, rebuild the network, and then install all my extra stuff, and in between these upgrades I would fix OpenStreetmap whenever I found a problem.

    The last time, and in fact the only time, I ever went through this updating process, was October 2018. I did some serious cleanup on OSM before that, so the map was in pretty good shape, but I got an embarrassing surprise when I demo’ed it to John R (an actual computer professional), who was thinking of commuting to Easton via the towpath. I’d just added offroad path options, and I was eager to show John my new toy, but the program refused to route onto the towpath at Sand Island — there was a missing intersection! A classic case of “broken demo.”

    The need for (and my interest in) the routing program faded not long after that, so, although I cleaned up the offending roads and paths on Sand Island within OpenStreetmap, I never did download any newer road versions. And that’s how it sat for three years, until this week…

    (to be continued)


    day by day tech talk 📎and tagged

  • Little Hacks

    Posted on by Don

    Just a few simple solutions to life’s little computer problems…

    WordPress

    When I post photos on my blog (in photo galleries, like the one here), if the photos have captions they show up when you hover over the photo thumbs. So far so good, but when the caption appeared it was at the top of the thumb, as opposed to appearing at the bottom, and the space used by the caption was much bigger than it needed to be, almost covering the thumbnail image. This has been driving me nuts for about a year; I assumed it had something to do with my offbeat WordPress theme, but yesterday I did some Googling and found people complaining about it, starting abut a year ago — apparently the offending caption styling was related to some WordPress update.

    I found several styling fixes posted online, but none seemed to work — it seems that a later WP update might have broken these fixes as well. The nature of these posted solutions gave me an idea though: I used Mozilla’s “inspect element” to get the CSS elements responsible for the captions, and used what I found to come up with a modification that would properly style my gallery figure captions. I added that to my theme’s custom CSS and it worked like a charm.

    Here is the new custom CSS that did the job, in case I need it again, or others need it and find themselves here:

    .wp-block-gallery.aligncenter .blocks-gallery-item figcaption, .wp-block-gallery.alignleft .blocks-gallery-item figcaption, .wp-block-gallery.alignright .blocks-gallery-item figcaption, .wp-block-gallery .blocks-gallery-item figcaption, .gallery figcaption {
     min-height: 5%;
     left: 0px;
     top: auto;
     height: auto;
     bottom: 0px;
     text-align: center;
     margin: auto;
}

    Fonts

    Fonts have always been a mysterious pain in the butt for me. It doesn’t help that I have hundreds of fonts installed by default, many with similar names and meant mainly for non-Latin alphabets. I was experimenting, searching for the right look for a document, and had to scroll through hundreds of useless fonts while trying to find something to suit me — very frustrating!

    I did some Googling, and I found I could just delete font files to get rid of them — but that’s a non-solution since I hate to throw anything away, you never know when you might need the perfect Linear B font…

    A little more research, and I eventually found and installed a simple utility called (surprise) “font manager,” which let me enable and disable fonts without deleting them. I spent some time shutting off my nuisance fonts, and now my word processing life is much simpler.


    tech talk

  • On Second Thought

    Posted on by Don

    Well, that didn’t take long…

    A few days of actually using the Input app, and I’m ready to throw it away and go back to QField, despite QField’s clunky data transfer method. The need to put my collected data into a PostGIS database (without too many hoops to jump through) is more important than I realized, and Input’s data entry UI had a few quirks that just became more unpleasant every time I used it — there was something just plain off about its text boxes and typing…

    It’s a shame too, because the Mergin update process was exactly what I wanted. (Input can also handle QR codes as data sources, something I have absolutely no use for — but hey, nerdgasm alert.) Well, QField is eventually supposed to get its own cloud service, maybe they’ll be able to upgrade their data transfer process once they have that in place. My luck, they’ll reproduce Mergin’s setup, and reproduce Mergin’s PostGIS problems along with it.


    tech talk 📎and tagged