• Category Archives tech talk

  • Computers and programs, maps and GPS, anything to do with data big or small, as well as my take on the pieces of equipment I use in other hobbies — think bike components, camping gear etc.

  • Foiled Again!

    Posted on by Don

    I have a love-hate relationship with Paul A. W. Wallace’s Indian Paths of Pennsylvania. I love reading the individual chapters on each path — their descriptions, and the accounts of them in the letters and diary entries of early Colonial explorers, but any hard look at the specifics and the trails themselves become frustratingly vague. This is all the more frustrating because the information looks specific and authoritative enough, until you take that close look…

    Some of this is because the original information is vague — nobody was tracking their steps with a GPS back then — so the actual trail location is not perfectly known, and partly it’s because the trails themselves are long gone (though some are at least partly followed by modern roads), so it’s hard to search them out without trespassing, but there also just seems to be some missing ingredient needed to define a trail network.

    A few years ago I thought that this last part could be solved with a little bit of GIS detective work, so I started a QGIS project to define the trails and see about building a network, but I sort of ran out of steam — I basically foundered on the vagueness of the trail descriptions. I did one or two in the Lehigh Valley, and realized that the sleuthing needed was a lot more laborious than simple data entry, and the project languished after those first few paths.

    I was thinking about all this again recently, and realized that there is a critical first step I ignored: the book serves primarily as an automotive guide, with detailed instructions for driving in the vicinity of each path. I also thought that if I broke the task down to a set of database tables, I could link these auto routes to their various paths and book chapters . (Some trail chapters actually describe multiple trails and subtrails, while some motorway descriptions continue across multiple chapters, so many-to-many relationships abound but that’s what databases are for. Furthermore, most of the trail chapters have a start and an endpoint, yet more data I can use to cross reference.)

    This scheme fell apart within the first few trails. The very first trail, the “Allegheny Path,” has Philadelphia as the start point and “Pittsburg and Kittanning” as the endpoint — so which is the endpoint? Apparently neither, because the trail is only described as far as Harrisburg; the “Allegheny Path” chapter ends with references to several other trails (different chapters, in other words) heading West from Harrisburg as possible continuations. So OK, I can deal with this: my endpoints are really Philadelphia and Harrisburg, and I’ll stuff the rest of the info into my “description” column. (There is a second path listed in that first chapter, but it is little more than a historical aside and a reference to another path/chapter. This is going to get tricky.)

    Luckily the motorway for the Allegheny Path is easy to follow. I used an open routing plugin to follow along a bunch of control points, and voilá I had my linestring. This ain’t so bad!

    The very next chapter, I ran into motorway difficulties: the route description made no sense. Either the routes were not prepared with adequate ground-truthing (unlikely, though I was starting to feel uncharitable), or the roads (and their designations) had changed at some point in the 55 years since the book came out. This seems the more likely explanation, since I-80 goes right through the area in question, was only finished in 1970, and probably changed a lot of things in its wake. I actually found the Wikipedia article on the Bald Eagle Creek Path more useful.

    So I’m back to deciphering and making judgement calls rather than strictly converting the information from one format to another, even for these road descriptions. I didn’t expect this project to be done in an afternoon, or even a week or so, but “going to take forever because I’m not really working on it” is now closer to my expectation.

    (Note: I found that someone already took these paths and put them into a GIS, but it’s on PA-Share and that’s proved difficult to work with — and deliberately limited, unless you pay — so far. We’ll see…)


    looking back tech talk 📎and tagged

  • Oh No Not That Again (Part 2 of 2)

    Posted on by Don

    So I was playing with the commuter mapping program the other day after doing some simple maintenance, just finding routes from here to there, and it started bothering me again that I could not route onto the towpath from Sand Island — the network was incorrect, it had no intersection from Main Street onto the path.

    I get my road data from OpenStreetmap, and I know that, in OSM, the trail is properly connected at Sand Island — I fixed that myself years ago, but never went through the rigamarole of updating and rebuilding my network. It didn’t seem worth the work for such a small change. There is another way to make that change though: I could modify my existing network, but that always seemed like it would be even more complicated and difficult than rebuilding from scratch.

    But would it be? The task really boils down to two things: adding a node where I want the new intersection to be, and then splitting the newly intersecting roads in two at the intersection point. Adding a node is easy enough, but splitting a road has a lot of moving parts — each of the two new road segments has to be assigned about 30 attributes, some of which they can inherit directly from the original road, others basically pro-rated from the original road based on the new road segment lengths, and yet others related to connecting the new road segments to the new node. It’s straightforward, but there are a lot of small, tedious calculations to perform and keep track of. Sounds like a job for the computer…

    What I did was write a PostgreSQL function that takes the node and the road, and returns two new road segments. I also wrote a wrapper script to update the network by calling this function. (I decided to just add my new nodes “by hand.”) It mostly works, though in one test case it didn’t split the road exactly where I thought it should (no idea why), and the new network routes like a champ.

    This isn’t a substitute for rebuilding the network: this is a quick fix for a small problem, and the the pro-rated attributes especially are a hack, an approximation; I can easily see situations where pro-rating say, ascent/descent data would be inaccurate. But this is fine for now.


    tech talk 📎and tagged

  • Oh No Not That Again (Part 1 of 2)

    Posted on by Don

    I’ve been looking at my Lehigh Valley bike commuter routing project again.

    I decided to update the recommended routes with additions based on some of our recent CAT rides, and found that the line geometries representing the various routes were missing. It’s no biggie, some things didn’t survive those destructive “upgrades” I did a few years ago, and the actual recommendation info is stored in among the road network data anyway.

    But, I still had the old routes as GeoJSON files, and it’s easier to work with them as geometries in their own right than as attributes on the road network, so I added them back into the database. Then I added that new route (Cedar Street, which parallels Union Boulevard but is much quieter), and used it to update the network. Piece of cake!

    I also decided to tackle the problem of updating the network paths themselves, which is not so much a piece of cake. I get the roads from OpenStreetmap, and there are mostly automated tools to build a routing network from OSM road data, and that’s followed by a whole lot of additional data massage to put it in the form I use. But the underlying OSM data isn’t always accurate — roads don’t go where they are supposed to, intersections don’t actually connect, that sort of thing. I would find a lot of this out after building the network, but the task of editing the network, once it’s built, is so onerous that my preferred method has been to fix the issues in OpenStreetmap, then just download the roads and rebuild the whole network from scratch — also onerous, but slightly less so.

    Anyway, I planned to make this a part of the usual site maintenance if this ever went live: maybe once a month I would download the OSM roads, rebuild the network, and then install all my extra stuff, and in between these upgrades I would fix OpenStreetmap whenever I found a problem.

    The last time, and in fact the only time, I ever went through this updating process, was October 2018. I did some serious cleanup on OSM before that, so the map was in pretty good shape, but I got an embarrassing surprise when I demo’ed it to John R (an actual computer professional), who was thinking of commuting to Easton via the towpath. I’d just added offroad path options, and I was eager to show John my new toy, but the program refused to route onto the towpath at Sand Island — there was a missing intersection! A classic case of “broken demo.”

    The need for (and my interest in) the routing program faded not long after that, so, although I cleaned up the offending roads and paths on Sand Island within OpenStreetmap, I never did download any newer road versions. And that’s how it sat for three years, until this week…

    (to be continued)


    day by day tech talk 📎and tagged

  • Little Hacks

    Posted on by Don

    Just a few simple solutions to life’s little computer problems…

    WordPress

    When I post photos on my blog (in photo galleries, like the one here), if the photos have captions they show up when you hover over the photo thumbs. So far so good, but when the caption appeared it was at the top of the thumb, as opposed to appearing at the bottom, and the space used by the caption was much bigger than it needed to be, almost covering the thumbnail image. This has been driving me nuts for about a year; I assumed it had something to do with my offbeat WordPress theme, but yesterday I did some Googling and found people complaining about it, starting abut a year ago — apparently the offending caption styling was related to some WordPress update.

    I found several styling fixes posted online, but none seemed to work — it seems that a later WP update might have broken these fixes as well. The nature of these posted solutions gave me an idea though: I used Mozilla’s “inspect element” to get the CSS elements responsible for the captions, and used what I found to come up with a modification that would properly style my gallery figure captions. I added that to my theme’s custom CSS and it worked like a charm.

    Here is the new custom CSS that did the job, in case I need it again, or others need it and find themselves here:

    .wp-block-gallery.aligncenter .blocks-gallery-item figcaption, .wp-block-gallery.alignleft .blocks-gallery-item figcaption, .wp-block-gallery.alignright .blocks-gallery-item figcaption, .wp-block-gallery .blocks-gallery-item figcaption, .gallery figcaption {
     min-height: 5%;
     left: 0px;
     top: auto;
     height: auto;
     bottom: 0px;
     text-align: center;
     margin: auto;
}

    Fonts

    Fonts have always been a mysterious pain in the butt for me. It doesn’t help that I have hundreds of fonts installed by default, many with similar names and meant mainly for non-Latin alphabets. I was experimenting, searching for the right look for a document, and had to scroll through hundreds of useless fonts while trying to find something to suit me — very frustrating!

    I did some Googling, and I found I could just delete font files to get rid of them — but that’s a non-solution since I hate to throw anything away, you never know when you might need the perfect Linear B font…

    A little more research, and I eventually found and installed a simple utility called (surprise) “font manager,” which let me enable and disable fonts without deleting them. I spent some time shutting off my nuisance fonts, and now my word processing life is much simpler.


    tech talk

  • On Second Thought

    Posted on by Don

    Well, that didn’t take long…

    A few days of actually using the Input app, and I’m ready to throw it away and go back to QField, despite QField’s clunky data transfer method. The need to put my collected data into a PostGIS database (without too many hoops to jump through) is more important than I realized, and Input’s data entry UI had a few quirks that just became more unpleasant every time I used it — there was something just plain off about its text boxes and typing…

    It’s a shame too, because the Mergin update process was exactly what I wanted. (Input can also handle QR codes as data sources, something I have absolutely no use for — but hey, nerdgasm alert.) Well, QField is eventually supposed to get its own cloud service, maybe they’ll be able to upgrade their data transfer process once they have that in place. My luck, they’ll reproduce Mergin’s setup, and reproduce Mergin’s PostGIS problems along with it.


    tech talk 📎and tagged

  • Data Collection II: Input

    Posted on by Don

    Just picking up where I left off here…

    I installed and started working with the other field geo-data collection app, called Input. It has a few drawbacks (so far) compared to QField, but it’s just as easy to use on the phone, and much easier to set up and transfer data back to the home computer.

    Input is basically a phone app front end for Mergin, a cloud-based data storage service built to integrate with QGIS. Mergin stores the data that then can be synch’ed with QGIS on the desktop, or with the app. Data collection does not need to be sent to the cloud in real time if you have no data connection, but can be done later, and is “just push the button” easy, as is synching between the cloud and QGIS. (This is the biggest advantage over QField, which has a pretty clunky update workflow.)

    The biggest problem I found so far has been that the data is not easily uploaded into a PostGIS table; the project relies on GeoPackages for data storage, even on the QGIS end. (Strictly speaking, there is a way to use PostGIS, but it seems involved, and needs to use another program that I have not yet tried.)

    There is also the issue of cloud storage: a free account is limited to 100 MB, and while the actual “location and description” data is usually pretty small, all the accompanying photos, at 5-10 MB each, will quickly bump up against that limit. If I could offload the data and photos, from the cloud-based project into something else (like my computer, and PostGIS), this limit wouldn’t be so pressing, but the workflow is starting to look unwieldy again…

    (Speaking of unwieldy, there is no direct way to store basemap tiles for offline use. Again, there is a way, but it’s not straightforward, and storing the tiles will also consume a significant portion of that 100MB limit.)

    None of these problems are deal-breakers (so far), and the single advantage of easy synchronization more than makes up for the lot of them. I’ll be playing with Input/Mergin some more, but I think it’s the one I’ll decide to keep.


    tech talk 📎and tagged

  • Golem Unchained

    Posted on by Don

    I’ve been using my handy, text-based “to-do” list for a while now, but my use of it was inconsistent, waxing and waning as it occurred (or didn’t occur) to me to use it, and I also didn’t like keeping a text editor open all the time whenever I did use it. So, the other day I added a “desklet” to my desktop, one that displays (the first lines of) a text file. I set it to display my to-do list, which I now can see any time, and open for editing with a click.

    Which is all well and good, but now I find that my to-do list is staring me in the face all the time, and I’ve been using it more, and more, and more… I guess I’m more productive now, as I write down all the things I need to do, then do them and mark them completed, but it seems that the tool I added to make my life easier is starting to get the upper hand, running my life instead of the other way around.

    I have to say that things get done, though.


    day by day tech talk

  • Data Collection

    Posted on by Don

    Part of what I do, as a member of the D&L trail patrol, is document issues along the trail — down trees, washouts — that may need to be addressed by the land managers. There is a specific report form for this kind of thing, where we enter a description, location (GPS coordinates) and maybe one or two photographs; you can fill out the form on the trail if you have a data connection — a big “if” on the trail sometimes, so I usually do it at home on the laptop, where all things computer are easier anyway.

    My typical workflow: I stop and take a picture, and later at home I use the photo’s EXIF data to get the location. This can be a bit of a pain, so I was thinking that maybe there is some app where I could create an entry, with location data, photos and maybe a timestamp, all added on the spot without need for data connectivity; I can then call the note up and refer to it later at home. (I have a “notes” app, where I can add photos, and paste my location, from say Google Maps or whatever, into a note, but I want the whole thing to be more integrated than that, with less human intervention.)

    There actually are some “geo-notes” apps, but I started overthinking things as usual, my wish list expanded, and then I discovered that here are two apps that actually integrate with QGIS: QField and Input.

    I am currently working with QField. You build a data-collection project in QGIS, then you run the “Qfield synch” plugin to export the project into a format that the app can use. Move the exported files to your phone, do your data collecting, move the updated files back to your computer, and import the updated version back into the original project. This process (export, move files, get data, move files back, import) is tiresome, especially since every data collection effort requires you to go through that entire process — I would much prefer something like “build project, export to phone, then: get data, upload data, get data, upload data…”

    Actually using the app, however, is easy. I set it up to record a timestamp with every point collected, then add a description and optional photos, and it works flawlessly; the only limitation I found so far is from the phone’s GPS, which is sometimes inaccurate. Once the data is back on my laptop, I can massage it (mostly automatically) into the form I need for the trail report.

    Next up is Input. This looks like the more promising app (based on their website), but the grass is always greener on the other trail…


    tech talk 📎and tagged

  • Working Man’s Blues

    Posted on by Don

    “Plans are worthless, but planning is everything.”

    Dwight D. Eisenhower

    I did another stint as a bicycle guide over this past week — this time, and rather unexpectedly, as the “ride leader”. I think I did well enough and liked it for the most part, and I think the riders got a pretty good experience out of it, but I also think I’ll not be doing that again anytime soon. Lesson learned: I like riding, and riding with other people, but I am not as comfortable schmoozing as I thought I’d be, and I absolutely hated feeling responsible for people and situations over which I didn’t have much control.

    What I did enjoy, strangely enough, is what I always enjoy: planning bike rides using map and database software. I started this particular ride-mapping project a few years ago. Rides, routes and alternate routes; points of interest and local trivia along the ride; ride logistics like bathroom locations and transportation times between the base and the start/finish — I stuffed it all into a database and managed the whole thing with QGIS. That was all well and good, but then it sat moribund over the COVID hiatus.

    As this new ride season approached I cleaned up and updated my project. Maybe this was really just the “fantasy football” or cosplay of the bike-guide biz, but it did come in handy after some local trails got wrecked by recent storms: I had backup rides already on deck, and enough information in my head to make plenty of other sudden changes “on the fly.”

    It seems a shame that I no longer have a use for my toy. We’ll see, maybe it can be the basis for some other project.


    tech talk the sporting life

  • Map Update

    Posted on by Don

    I finally got around to riding the southernmost part of the D&L about two weeks ago, riding from Yardley to Bristol and back, and ground-truthing the trail and access points. I can scratch that off my bucket list, and I don’t see any reason to ride south of Yardley again — this trail section, especially the Morrisville-Levittown portion, is nowhere near as nice as other areas — but I got what I needed to finish my trail amenities map. I may do a little exploring on the Black Diamond north of White Haven just for the sake of completeness, but I think I now have everything I was looking for.


    tech talk 📎and tagged