• Category Archives tech talk

  • Computers and programs, maps and GPS, anything to do with data big or small, as well as my take on the pieces of equipment I use in other hobbies — think bike components, camping gear etc.

  • On Second Thought

    Posted on by Don

    Well, that didn’t take long…

    A few days of actually using the Input app, and I’m ready to throw it away and go back to QField, despite QField’s clunky data transfer method. The need to put my collected data into a PostGIS database (without too many hoops to jump through) is more important than I realized, and Input’s data entry UI had a few quirks that just became more unpleasant every time I used it — there was something just plain off about its text boxes and typing…

    It’s a shame too, because the Mergin update process was exactly what I wanted. (Input can also handle QR codes as data sources, something I have absolutely no use for — but hey, nerdgasm alert.) Well, QField is eventually supposed to get its own cloud service, maybe they’ll be able to upgrade their data transfer process once they have that in place. My luck, they’ll reproduce Mergin’s setup, and reproduce Mergin’s PostGIS problems along with it.


    tech talk 📎and tagged

  • Data Collection II: Input

    Posted on by Don

    Just picking up where I left off here…

    I installed and started working with the other field geo-data collection app, called Input. It has a few drawbacks (so far) compared to QField, but it’s just as easy to use on the phone, and much easier to set up and transfer data back to the home computer.

    Input is basically a phone app front end for Mergin, a cloud-based data storage service built to integrate with QGIS. Mergin stores the data that then can be synch’ed with QGIS on the desktop, or with the app. Data collection does not need to be sent to the cloud in real time if you have no data connection, but can be done later, and is “just push the button” easy, as is synching between the cloud and QGIS. (This is the biggest advantage over QField, which has a pretty clunky update workflow.)

    The biggest problem I found so far has been that the data is not easily uploaded into a PostGIS table; the project relies on GeoPackages for data storage, even on the QGIS end. (Strictly speaking, there is a way to use PostGIS, but it seems involved, and needs to use another program that I have not yet tried.)

    There is also the issue of cloud storage: a free account is limited to 100 MB, and while the actual “location and description” data is usually pretty small, all the accompanying photos, at 5-10 MB each, will quickly bump up against that limit. If I could offload the data and photos, from the cloud-based project into something else (like my computer, and PostGIS), this limit wouldn’t be so pressing, but the workflow is starting to look unwieldy again…

    (Speaking of unwieldy, there is no direct way to store basemap tiles for offline use. Again, there is a way, but it’s not straightforward, and storing the tiles will also consume a significant portion of that 100MB limit.)

    None of these problems are deal-breakers (so far), and the single advantage of easy synchronization more than makes up for the lot of them. I’ll be playing with Input/Mergin some more, but I think it’s the one I’ll decide to keep.


    tech talk 📎and tagged

  • Golem Unchained

    Posted on by Don

    I’ve been using my handy, text-based “to-do” list for a while now, but my use of it was inconsistent, waxing and waning as it occurred (or didn’t occur) to me to use it, and I also didn’t like keeping a text editor open all the time whenever I did use it. So, the other day I added a “desklet” to my desktop, one that displays (the first lines of) a text file. I set it to display my to-do list, which I now can see any time, and open for editing with a click.

    Which is all well and good, but now I find that my to-do list is staring me in the face all the time, and I’ve been using it more, and more, and more… I guess I’m more productive now, as I write down all the things I need to do, then do them and mark them completed, but it seems that the tool I added to make my life easier is starting to get the upper hand, running my life instead of the other way around.

    I have to say that things get done, though.


    day by day tech talk

  • Data Collection

    Posted on by Don

    Part of what I do, as a member of the D&L trail patrol, is document issues along the trail — down trees, washouts — that may need to be addressed by the land managers. There is a specific report form for this kind of thing, where we enter a description, location (GPS coordinates) and maybe one or two photographs; you can fill out the form on the trail if you have a data connection — a big “if” on the trail sometimes, so I usually do it at home on the laptop, where all things computer are easier anyway.

    My typical workflow: I stop and take a picture, and later at home I use the photo’s EXIF data to get the location. This can be a bit of a pain, so I was thinking that maybe there is some app where I could create an entry, with location data, photos and maybe a timestamp, all added on the spot without need for data connectivity; I can then call the note up and refer to it later at home. (I have a “notes” app, where I can add photos, and paste my location, from say Google Maps or whatever, into a note, but I want the whole thing to be more integrated than that, with less human intervention.)

    There actually are some “geo-notes” apps, but I started overthinking things as usual, my wish list expanded, and then I discovered that here are two apps that actually integrate with QGIS: QField and Input.

    I am currently working with QField. You build a data-collection project in QGIS, then you run the “Qfield synch” plugin to export the project into a format that the app can use. Move the exported files to your phone, do your data collecting, move the updated files back to your computer, and import the updated version back into the original project. This process (export, move files, get data, move files back, import) is tiresome, especially since every data collection effort requires you to go through that entire process — I would much prefer something like “build project, export to phone, then: get data, upload data, get data, upload data…”

    Actually using the app, however, is easy. I set it up to record a timestamp with every point collected, then add a description and optional photos, and it works flawlessly; the only limitation I found so far is from the phone’s GPS, which is sometimes inaccurate. Once the data is back on my laptop, I can massage it (mostly automatically) into the form I need for the trail report.

    Next up is Input. This looks like the more promising app (based on their website), but the grass is always greener on the other trail…


    tech talk 📎and tagged

  • Working Man’s Blues

    Posted on by Don

    “Plans are worthless, but planning is everything.”

    Dwight D. Eisenhower

    I did another stint as a bicycle guide over this past week — this time, and rather unexpectedly, as the “ride leader”. I think I did well enough and liked it for the most part, and I think the riders got a pretty good experience out of it, but I also think I’ll not be doing that again anytime soon. Lesson learned: I like riding, and riding with other people, but I am not as comfortable schmoozing as I thought I’d be, and I absolutely hated feeling responsible for people and situations over which I didn’t have much control.

    What I did enjoy, strangely enough, is what I always enjoy: planning bike rides using map and database software. I started this particular ride-mapping project a few years ago. Rides, routes and alternate routes; points of interest and local trivia along the ride; ride logistics like bathroom locations and transportation times between the base and the start/finish — I stuffed it all into a database and managed the whole thing with QGIS. That was all well and good, but then it sat moribund over the COVID hiatus.

    As this new ride season approached I cleaned up and updated my project. Maybe this was really just the “fantasy football” or cosplay of the bike-guide biz, but it did come in handy after some local trails got wrecked by recent storms: I had backup rides already on deck, and enough information in my head to make plenty of other sudden changes “on the fly.”

    It seems a shame that I no longer have a use for my toy. We’ll see, maybe it can be the basis for some other project.


    tech talk the sporting life

  • Map Update

    Posted on by Don

    I finally got around to riding the southernmost part of the D&L about two weeks ago, riding from Yardley to Bristol and back, and ground-truthing the trail and access points. I can scratch that off my bucket list, and I don’t see any reason to ride south of Yardley again — this trail section, especially the Morrisville-Levittown portion, is nowhere near as nice as other areas — but I got what I needed to finish my trail amenities map. I may do a little exploring on the Black Diamond north of White Haven just for the sake of completeness, but I think I now have everything I was looking for.


    tech talk 📎and tagged

  • Every Week Is Infrastructure Week

    Posted on by Don

    Bike

    Speaking of Scott & Kellyn… I went looking for new parts for my road bike, but found instead that the component supply chain is still completely disrupted by COVID, and I may need to wait a long time to replace that freehub. So I went back to CAT, where Scott helped me find find a wheel compatible with my shifting system, and then guided me through rebuilding it: cleaning out and re-greasing the freehub, ditto the bearings, and putting it all together with my old sprockets. It looks a bit weird on my bike — even though it’s also Campagnolo it’s decades older than my other components — and I am still on borrowed time with my other worn drivetrain components, but the bike is rideable again. Thanks Scott!

    Computer

    My laptop started making a tiny creaking sound when I opened it, and the other day I noticed that the right hinge had become detached in some way. Fixing this, according to YouTube, is an easy enough home repair, and System76 says I can send the laptop back to the factory to get it fixed, but I think I’ll split the difference and bring it to some local repair place.

    Before I brought it somewhere though, I wanted to make sure I had my data backed up. (I used to do backups regularly but, ironically enough, my old backup drive crashed a while ago. So, step one was to get a new drive.) I picked up a 2.0 TB, USB hard drive at Staples, then spent a little time fixing it up the way I wanted it, re-formatting it with a more Linux-friendly filesystem and replacing the drive’s icon with one I like better. After that I just copied my home folder over to the new drive and called it a backup. (I also needed to back up a few other things, like global configuration files, but that was just more file copying.)

    My last remaining backups were the databases. These required using a few special programs, which, since I haven’t used them since my last major upgrade, I just discovered in the moment that they were not configured correctly, and in fact my whole database system was a misconfigured hodgepodge… I had to backtrack a bit and get my system in order, which meant I had to do a bit of learning first, but I eventually got the whole thing running and even managed to automate the process.

    The next step is to bring the laptop to a repair store.


    tech talk

  • More Ball Bearing Woes

    Posted on by Don

    Morning weigh-in: 175.5#, 11.5% BF

    I took my road bike over to the CAT office yesterday and took apart my freehub. What a mess! There are a whole lot of very small ball bearings in there, and the bearing cage basically fell apart — sound familiar? The slop in the freehub was basically caused by missing BB’s but the entire inside was trashed anyway. Maybe it can be serviced, but I think it’ll be better to just get a new one, if I can find something compatible with that wheel. I will also be replacing the cassette, the chain and the front rings, so maybe I should just get a new wheel as well.

    One good thing: while I was at CAT, Scott was able to get my pedals off, so now I can replace them.

    Today we had to run a bunch of errands, and when we got home we were hungry and tired and crabby. So, we hopped on the mountain bikes and rode to Freemansburg, maybe four miles away on the towpath. We got takeout at a place called Cherry’s Caribbean Palace (curried goat, mac & cheese, rice and plantains) and ate at picnic tables by the restored Mule Barn. It totally changed the afternoon’s vibe. I’ve been meaning to check that place out (it’s on my amenities map), and I saw online that they just got recognition as one of the best Caribbean food places around, so this was a pretty good opportunity to do some food exploring. Two thumbs up.


    day by day tech talk

  • New Phones

    Posted on by Don

    Morning weigh-in: 175.5#, 15% BF

    Anne and I just replaced our phones, and in the nick of time too: they were ancient, and falling apart (literally my power button just fell off), and the network they worked on was about to be sunsetted…

    We both got the new Samsung Galaxy A52, which is not top of the line but it still blows our old phones away, thank you very much, and it costs about half of what the top models go for. We bought them online, unlocked, and got new SIM chips from our phone company, and then we transferred our numbers and data to the new phones. Easy enough process, and everything seems to be working OK, now all we have to do is get used to them.

    Rain Comes At You Fast: We had plans to head up to the Finger Lakes for the 4th of July, Ben and Candace (and their new dog) and us, camping and cycling from tomorrow through the 5th. But the forecast has been getting more and more dire as our trip approached, and so we got together in a Zoom chat last night and changed our plans: they’ll be coming to visit us for part of the weekend, and we’ll do a hike if the weather allows.

    I got in another sweltering towpath ride this afternoon, super hot even though I was moseying, but it was beautiful out there. Still, I had the trail mostly to myself, except for a few kids at the swimming holes.


    day by day tech talk

  • The Wheat From The Chaff

    Posted on by Don

    I’m not sure if this is going to rise to the level of “new GIS project,” but I have been playing around a lot lately with the local transportation authority’s GTFS feed — where GTFS stands for “General Transit Feed Specification,” a standard for publishing public transit information on the Internet.

    These feeds are like a cross between spreadsheets and database tables, and by a judicious massaging of the data you can extract bus stop and route information. Unfortunately, that massaging is a real necessity: the specification is built to convey a lot of information, and to cover a lot of different transit situations, so there’s no simple route-and-stop information — it’s buried in cross-references and spread across multiple tables. All this extraction and data crunching is fairly straightforward though, and there are even tools to automate the process (I use a QGIS plugin).

    Or the process would be straightforward, if we were not dealing with LANTA. These feeds are updated periodically, and about a year ago the new LANTA feeds sort of devolved into chaos, with extra routes showing up that had no real world connection, odd use of abbreviations for bus stop names (abbreviations are sort of frowned upon, for what ought to be obvious reasons), and their cross-referencing system becoming unnecessarily complex. It was hard to figure out what was going on — I thought at first that it was my analysis software mangling the data, but no it was them.

    Well, they’ve been working through a huge revamp of their entire bus route network, so maybe that was the source of some of the bogus data. The new routes and schedules went into effect on June 21, and an updated feed followed soon after; I downloaded the new one and crunched the data — and the garbage was all still there! But, I noticed that in among the old chaos was a new and much cleaner set of data, valid starting on the 21st, showing the new bus routes and the correctly-named bus stops. So now I do a double extraction, first massaging the feed into a useful form, then extracting from that the new, valid and cleaned-up route data. Voilá!

    I have some vague plan to add these bus routes to OpenStreetMap, but that’s a big undertaking, and I would prefer to rely on eyewitness ground-truthing (ie riding the bus) than a data set — which means even more work. For now I’m content with just having got the damn data.


    tech talk 📎and tagged