• Category Archives tech talk

  • Computers and programs, maps and GPS, anything to do with data big or small, as well as my take on the pieces of equipment I use in other hobbies — think bike components, camping gear etc.

  • Oh No Not That Again (Part 2 of 2)

    Posted on by Don

    So I was playing with the commuter mapping program the other day after doing some simple maintenance, just finding routes from here to there, and it started bothering me again that I could not route onto the towpath from Sand Island — the network was incorrect, it had no intersection from Main Street onto the path.

    I get my road data from OpenStreetmap, and I know that, in OSM, the trail is properly connected at Sand Island — I fixed that myself years ago, but never went through the rigamarole of updating and rebuilding my network. It didn’t seem worth the work for such a small change. There is another way to make that change though: I could modify my existing network, but that always seemed like it would be even more complicated and difficult than rebuilding from scratch.

    But would it be? The task really boils down to two things: adding a node where I want the new intersection to be, and then splitting the newly intersecting roads in two at the intersection point. Adding a node is easy enough, but splitting a road has a lot of moving parts — each of the two new road segments has to be assigned about 30 attributes, some of which they can inherit directly from the original road, others basically pro-rated from the original road based on the new road segment lengths, and yet others related to connecting the new road segments to the new node. It’s straightforward, but there are a lot of small, tedious calculations to perform and keep track of. Sounds like a job for the computer…

    What I did was write a PostgreSQL function that takes the node and the road, and returns two new road segments. I also wrote a wrapper script to update the network by calling this function. (I decided to just add my new nodes “by hand.”) It mostly works, though in one test case it didn’t split the road exactly where I thought it should (no idea why), and the new network routes like a champ.

    This isn’t a substitute for rebuilding the network: this is a quick fix for a small problem, and the the pro-rated attributes especially are a hack, an approximation; I can easily see situations where pro-rating say, ascent/descent data would be inaccurate. But this is fine for now.


    tech talk 📎and tagged

  • Oh No Not That Again (Part 1 of 2)

    Posted on by Don

    I’ve been looking at my Lehigh Valley bike commuter routing project again.

    I decided to update the recommended routes with additions based on some of our recent CAT rides, and found that the line geometries representing the various routes were missing. It’s no biggie, some things didn’t survive those destructive “upgrades” I did a few years ago, and the actual recommendation info is stored in among the road network data anyway.

    But, I still had the old routes as GeoJSON files, and it’s easier to work with them as geometries in their own right than as attributes on the road network, so I added them back into the database. Then I added that new route (Cedar Street, which parallels Union Boulevard but is much quieter), and used it to update the network. Piece of cake!

    I also decided to tackle the problem of updating the network paths themselves, which is not so much a piece of cake. I get the roads from OpenStreetmap, and there are mostly automated tools to build a routing network from OSM road data, and that’s followed by a whole lot of additional data massage to put it in the form I use. But the underlying OSM data isn’t always accurate — roads don’t go where they are supposed to, intersections don’t actually connect, that sort of thing. I would find a lot of this out after building the network, but the task of editing the network, once it’s built, is so onerous that my preferred method has been to fix the issues in OpenStreetmap, then just download the roads and rebuild the whole network from scratch — also onerous, but slightly less so.

    Anyway, I planned to make this a part of the usual site maintenance if this ever went live: maybe once a month I would download the OSM roads, rebuild the network, and then install all my extra stuff, and in between these upgrades I would fix OpenStreetmap whenever I found a problem.

    The last time, and in fact the only time, I ever went through this updating process, was October 2018. I did some serious cleanup on OSM before that, so the map was in pretty good shape, but I got an embarrassing surprise when I demo’ed it to John R (an actual computer professional), who was thinking of commuting to Easton via the towpath. I’d just added offroad path options, and I was eager to show John my new toy, but the program refused to route onto the towpath at Sand Island — there was a missing intersection! A classic case of “broken demo.”

    The need for (and my interest in) the routing program faded not long after that, so, although I cleaned up the offending roads and paths on Sand Island within OpenStreetmap, I never did download any newer road versions. And that’s how it sat for three years, until this week…

    (to be continued)


    day by day tech talk 📎and tagged

  • Little Hacks

    Posted on by Don

    Just a few simple solutions to life’s little computer problems…

    WordPress

    When I post photos on my blog (in photo galleries, like the one here), if the photos have captions they show up when you hover over the photo thumbs. So far so good, but when the caption appeared it was at the top of the thumb, as opposed to appearing at the bottom, and the space used by the caption was much bigger than it needed to be, almost covering the thumbnail image. This has been driving me nuts for about a year; I assumed it had something to do with my offbeat WordPress theme, but yesterday I did some Googling and found people complaining about it, starting abut a year ago — apparently the offending caption styling was related to some WordPress update.

    I found several styling fixes posted online, but none seemed to work — it seems that a later WP update might have broken these fixes as well. The nature of these posted solutions gave me an idea though: I used Mozilla’s “inspect element” to get the CSS elements responsible for the captions, and used what I found to come up with a modification that would properly style my gallery figure captions. I added that to my theme’s custom CSS and it worked like a charm.

    Here is the new custom CSS that did the job, in case I need it again, or others need it and find themselves here:

    .wp-block-gallery.aligncenter .blocks-gallery-item figcaption, .wp-block-gallery.alignleft .blocks-gallery-item figcaption, .wp-block-gallery.alignright .blocks-gallery-item figcaption, .wp-block-gallery .blocks-gallery-item figcaption, .gallery figcaption {
     min-height: 5%;
     left: 0px;
     top: auto;
     height: auto;
     bottom: 0px;
     text-align: center;
     margin: auto;
}

    Fonts

    Fonts have always been a mysterious pain in the butt for me. It doesn’t help that I have hundreds of fonts installed by default, many with similar names and meant mainly for non-Latin alphabets. I was experimenting, searching for the right look for a document, and had to scroll through hundreds of useless fonts while trying to find something to suit me — very frustrating!

    I did some Googling, and I found I could just delete font files to get rid of them — but that’s a non-solution since I hate to throw anything away, you never know when you might need the perfect Linear B font…

    A little more research, and I eventually found and installed a simple utility called (surprise) “font manager,” which let me enable and disable fonts without deleting them. I spent some time shutting off my nuisance fonts, and now my word processing life is much simpler.


    tech talk

  • On Second Thought

    Posted on by Don

    Well, that didn’t take long…

    A few days of actually using the Input app, and I’m ready to throw it away and go back to QField, despite QField’s clunky data transfer method. The need to put my collected data into a PostGIS database (without too many hoops to jump through) is more important than I realized, and Input’s data entry UI had a few quirks that just became more unpleasant every time I used it — there was something just plain off about its text boxes and typing…

    It’s a shame too, because the Mergin update process was exactly what I wanted. (Input can also handle QR codes as data sources, something I have absolutely no use for — but hey, nerdgasm alert.) Well, QField is eventually supposed to get its own cloud service, maybe they’ll be able to upgrade their data transfer process once they have that in place. My luck, they’ll reproduce Mergin’s setup, and reproduce Mergin’s PostGIS problems along with it.


    tech talk 📎and tagged

  • Data Collection II: Input

    Posted on by Don

    Just picking up where I left off here…

    I installed and started working with the other field geo-data collection app, called Input. It has a few drawbacks (so far) compared to QField, but it’s just as easy to use on the phone, and much easier to set up and transfer data back to the home computer.

    Input is basically a phone app front end for Mergin, a cloud-based data storage service built to integrate with QGIS. Mergin stores the data that then can be synch’ed with QGIS on the desktop, or with the app. Data collection does not need to be sent to the cloud in real time if you have no data connection, but can be done later, and is “just push the button” easy, as is synching between the cloud and QGIS. (This is the biggest advantage over QField, which has a pretty clunky update workflow.)

    The biggest problem I found so far has been that the data is not easily uploaded into a PostGIS table; the project relies on GeoPackages for data storage, even on the QGIS end. (Strictly speaking, there is a way to use PostGIS, but it seems involved, and needs to use another program that I have not yet tried.)

    There is also the issue of cloud storage: a free account is limited to 100 MB, and while the actual “location and description” data is usually pretty small, all the accompanying photos, at 5-10 MB each, will quickly bump up against that limit. If I could offload the data and photos, from the cloud-based project into something else (like my computer, and PostGIS), this limit wouldn’t be so pressing, but the workflow is starting to look unwieldy again…

    (Speaking of unwieldy, there is no direct way to store basemap tiles for offline use. Again, there is a way, but it’s not straightforward, and storing the tiles will also consume a significant portion of that 100MB limit.)

    None of these problems are deal-breakers (so far), and the single advantage of easy synchronization more than makes up for the lot of them. I’ll be playing with Input/Mergin some more, but I think it’s the one I’ll decide to keep.


    tech talk 📎and tagged

  • Golem Unchained

    Posted on by Don

    I’ve been using my handy, text-based “to-do” list for a while now, but my use of it was inconsistent, waxing and waning as it occurred (or didn’t occur) to me to use it, and I also didn’t like keeping a text editor open all the time whenever I did use it. So, the other day I added a “desklet” to my desktop, one that displays (the first lines of) a text file. I set it to display my to-do list, which I now can see any time, and open for editing with a click.

    Which is all well and good, but now I find that my to-do list is staring me in the face all the time, and I’ve been using it more, and more, and more… I guess I’m more productive now, as I write down all the things I need to do, then do them and mark them completed, but it seems that the tool I added to make my life easier is starting to get the upper hand, running my life instead of the other way around.

    I have to say that things get done, though.


    day by day tech talk

  • Data Collection

    Posted on by Don

    Part of what I do, as a member of the D&L trail patrol, is document issues along the trail — down trees, washouts — that may need to be addressed by the land managers. There is a specific report form for this kind of thing, where we enter a description, location (GPS coordinates) and maybe one or two photographs; you can fill out the form on the trail if you have a data connection — a big “if” on the trail sometimes, so I usually do it at home on the laptop, where all things computer are easier anyway.

    My typical workflow: I stop and take a picture, and later at home I use the photo’s EXIF data to get the location. This can be a bit of a pain, so I was thinking that maybe there is some app where I could create an entry, with location data, photos and maybe a timestamp, all added on the spot without need for data connectivity; I can then call the note up and refer to it later at home. (I have a “notes” app, where I can add photos, and paste my location, from say Google Maps or whatever, into a note, but I want the whole thing to be more integrated than that, with less human intervention.)

    There actually are some “geo-notes” apps, but I started overthinking things as usual, my wish list expanded, and then I discovered that here are two apps that actually integrate with QGIS: QField and Input.

    I am currently working with QField. You build a data-collection project in QGIS, then you run the “Qfield synch” plugin to export the project into a format that the app can use. Move the exported files to your phone, do your data collecting, move the updated files back to your computer, and import the updated version back into the original project. This process (export, move files, get data, move files back, import) is tiresome, especially since every data collection effort requires you to go through that entire process — I would much prefer something like “build project, export to phone, then: get data, upload data, get data, upload data…”

    Actually using the app, however, is easy. I set it up to record a timestamp with every point collected, then add a description and optional photos, and it works flawlessly; the only limitation I found so far is from the phone’s GPS, which is sometimes inaccurate. Once the data is back on my laptop, I can massage it (mostly automatically) into the form I need for the trail report.

    Next up is Input. This looks like the more promising app (based on their website), but the grass is always greener on the other trail…


    tech talk 📎and tagged

  • Working Man’s Blues

    Posted on by Don

    “Plans are worthless, but planning is everything.”

    Dwight D. Eisenhower

    I did another stint as a bicycle guide over this past week — this time, and rather unexpectedly, as the “ride leader”. I think I did well enough and liked it for the most part, and I think the riders got a pretty good experience out of it, but I also think I’ll not be doing that again anytime soon. Lesson learned: I like riding, and riding with other people, but I am not as comfortable schmoozing as I thought I’d be, and I absolutely hated feeling responsible for people and situations over which I didn’t have much control.

    What I did enjoy, strangely enough, is what I always enjoy: planning bike rides using map and database software. I started this particular ride-mapping project a few years ago. Rides, routes and alternate routes; points of interest and local trivia along the ride; ride logistics like bathroom locations and transportation times between the base and the start/finish — I stuffed it all into a database and managed the whole thing with QGIS. That was all well and good, but then it sat moribund over the COVID hiatus.

    As this new ride season approached I cleaned up and updated my project. Maybe this was really just the “fantasy football” or cosplay of the bike-guide biz, but it did come in handy after some local trails got wrecked by recent storms: I had backup rides already on deck, and enough information in my head to make plenty of other sudden changes “on the fly.”

    It seems a shame that I no longer have a use for my toy. We’ll see, maybe it can be the basis for some other project.


    tech talk the sporting life

  • Map Update

    Posted on by Don

    I finally got around to riding the southernmost part of the D&L about two weeks ago, riding from Yardley to Bristol and back, and ground-truthing the trail and access points. I can scratch that off my bucket list, and I don’t see any reason to ride south of Yardley again — this trail section, especially the Morrisville-Levittown portion, is nowhere near as nice as other areas — but I got what I needed to finish my trail amenities map. I may do a little exploring on the Black Diamond north of White Haven just for the sake of completeness, but I think I now have everything I was looking for.


    tech talk 📎and tagged

  • Every Week Is Infrastructure Week

    Posted on by Don

    Bike

    Speaking of Scott & Kellyn… I went looking for new parts for my road bike, but found instead that the component supply chain is still completely disrupted by COVID, and I may need to wait a long time to replace that freehub. So I went back to CAT, where Scott helped me find find a wheel compatible with my shifting system, and then guided me through rebuilding it: cleaning out and re-greasing the freehub, ditto the bearings, and putting it all together with my old sprockets. It looks a bit weird on my bike — even though it’s also Campagnolo it’s decades older than my other components — and I am still on borrowed time with my other worn drivetrain components, but the bike is rideable again. Thanks Scott!

    Computer

    My laptop started making a tiny creaking sound when I opened it, and the other day I noticed that the right hinge had become detached in some way. Fixing this, according to YouTube, is an easy enough home repair, and System76 says I can send the laptop back to the factory to get it fixed, but I think I’ll split the difference and bring it to some local repair place.

    Before I brought it somewhere though, I wanted to make sure I had my data backed up. (I used to do backups regularly but, ironically enough, my old backup drive crashed a while ago. So, step one was to get a new drive.) I picked up a 2.0 TB, USB hard drive at Staples, then spent a little time fixing it up the way I wanted it, re-formatting it with a more Linux-friendly filesystem and replacing the drive’s icon with one I like better. After that I just copied my home folder over to the new drive and called it a backup. (I also needed to back up a few other things, like global configuration files, but that was just more file copying.)

    My last remaining backups were the databases. These required using a few special programs, which, since I haven’t used them since my last major upgrade, I just discovered in the moment that they were not configured correctly, and in fact my whole database system was a misconfigured hodgepodge… I had to backtrack a bit and get my system in order, which meant I had to do a bit of learning first, but I eventually got the whole thing running and even managed to automate the process.

    The next step is to bring the laptop to a repair store.


    tech talk