Banishing SmartQuotes from Allegedly "plaintext" Files

I don't consider a text file to be "plain text" if it's UTF-8 with those accursed "smart quotes". To get rid of them, use the iconv utility. Example:

$ iconv -f utf-8 -t ascii//translit <file>

Maybe not worth a full post, but it took me a while to find the answer online, so there it is.

-R.

Migrating From Apache2 To Caddy

 I have wanted to investigate the feasibility of moving to a more modern webserver setup on my home server, specifically Caddy or Nginx as they both sounded like they are much easier to set up and more streamlined than apache; but never seemed to get around to it. Well, finally I manged to over this past weekend.

The main case, serving out static content from one or more domains with HTTPS cert setup, was surprisingly easy and I was impressed by its ease of use. Caddy has a nice config syntax which is easy to read, reasonably intuitive and supports single- or multi-file configurations, with a fragment syntax and include mechanism to keep large, complex configs more manageable.

In order to fully replace Apache on my home server, I needed to ensure any replacement would support a few things over and above serving just static content from a single domain:

• Multiple domain hosting (virtual hosts)
• UNIX ~username webspaces
• Use traditional syslog format instead of JSON logs
• PHP (for Nextcloud)
• CGI (Common Gateway Interface)
• Golang vanity URLs to serve out go modules

Hosting Multiple Domains (virtual hosts) with Caddy

Caddy's docs are pretty good (especially the Common Patterns section) but it's a bit tricky sometimes to know just which directive to use, where.

• Setting up multiple domain serving was very straightforward -- basic Caddy syntax and documentation spells it out clearly, refer to the Common Patterns section of the Caddy docs.
  
  

UNIX ~username webspaces with Caddy

• Hosting UNIX ~username style webspaces was pretty simple, just a basic use of the handle_path and root directives:
  
  
handle_path /~joebloggs* {
    root * /home/joebloggs/public_html
}

Use Traditional syslog Format with Caddy instead of JSON Logs

• This one was simple, just add the following entry in the log {..} section of Caddyfile config:

format transform "{common_log}"

Hosting PHP and Nextcloud from Caddy

• Hosting a PHP app behind Caddy was trickier: I couldn't find examples that spelled out precisely the proper setup. I had to first discover what the fpm-php helper util/pkg was, and how to set it up, and that it has to run with the same user:group as Caddy, and the PHP files and data that are being served (by default fpm-php runs as nobody:nogroup); simply /etc/php/fpm-php[x.y]/fpm.d/www.conf to set user=caddy and group=caddy.
• Then, finally, the Caddyfile entry for one's domain must include the proper root and php_fastcgi proxy config:

blitter.com {

        # dir as root of the website

        root * /var/www/www.blitter.com/htdocs

        # enable serving static files from root dir

        encode

        file_server

        php_fastcgi localhost:9000

} 

Hosting CGI from ~user web spaces with Caddy

• CGI was not too hard though it isn't spelled out in the Caddy Common Patterns documentation:
  
  

handle_path /~russtopia* {

    root * /home/russtopia/public_html

handle_path /cgi-bin/* {

    reverse_proxy localhost:10000 {

        transport fastcgi {

            env SCRIPT_FILENAME /home/russtopia/public_html/cgi-bin/{path}

        }

    }

}

An entry such as the above must be created for each unique user or app's local cgi-bin/ area. If CGI files must write to the user/app's backend filesystem, the intended output folder should have proper permissions to allow Caddy, if not running as root (and you don't need to run Caddy as root, so avoid it!) to access them. I created a new distinct 'cgi' group and made sure my user and caddy itself were members, and set user/group read/write perms on the cgi script's output folder.

Serving Golang Vanity URLs with Caddy

• Golang 'vanity URLs' are a way to self-host one's own Go source package modules, or redirect domain assignments such that they are referred to just as modules on the big centralized source hosts like github.com, gitlab.io etc. For example, I have authored some Go programs and modules which I make available via the root name blitter.com/go/<module>. I want the Go tooling to be able to use go get, go install, and so on without issue.
  
  I also have a rather unique git setup: I run git-daemon with packages visible from the server's /var/git/ directory, of which many are symlinked to the repo's actual location within a Gogs installation which lives and runs under the git:git user/group. In this way I can do raw git checkouts via the git:// or ssh:// schemes, as well as via https:// to my Gogs instance.
  
  This setup in Caddy was not fully covered by any other online tutorial, at least not in my server's particular setup -- I use a /go/ URL endpoint within the overall blitter.com domain to host modules, rather than dedicating a go.blitter.com domain just for Go modules, so other tutorials didn't quite fit. So here's the entire setup that integrates the git-backend as well as golang vanity URL setup:

# Caddy tutorial on serving vanity go module URLs:

# https://abhijithota.com/posts/golang-vanity-urls-using-caddy/

####

(gomodhandler) {

        handle /go/{args[0]} {

                @from_go query go-get=1

                handle @from_go {

                        header Content-Type text/html

                        respond <<HTML

                        <!DOCTYPE html>

                        <html>

                        <head>

                        <meta name="go-import" content="blitter.com/go/{args[0]} git https://blitter.com/git/{args[0]}">

                        <!-- <meta http-equiv="refresh" content="0; url=https://blitter.com/git/{args[0]}" /> -->

                        </head>

                        </html>

                        HTML 200

                }

        }

}

####

www.blitter.com {

        redir https://blitter.com{uri}

}

blitter.com {

        ####> git-daemon ####

        # Caddy tutorial on git over HTTP(s) proxying git-daemon:

        # https://www.jamesatkins.com/posts/git-over-http-with-caddy/

        handle_path /git* {

                root * /var/git

        }

        handle_path /var/git* {

                root * /git

        }

        @git_cgi path_regexp "^.*/(HEAD|info/refs|objects/info/[^/]+|git-upload-pack)$"

        @git_static path_regexp "^.*/objects/([0-9a-f]{2}/[0-9a-f]{38}|pack/pack-[0-9a-f]{40}\.(pack|idx))$"

        handle @git_cgi {

                reverse_proxy unix//run/git-cgi.socket {

                        transport fastcgi {

                                #env SCRIPT_FILENAME ${pkgs.git}/libexec/git-core/git-http-backend

                                env SCRIPT_FILENAME /usr/libexec/git-core/git-http-backend

                                env GIT_HTTP_EXPORT_ALL 1

                                env GIT_PROJECT_ROOT /var/git

                        }

                }

        }

        handle @git_static {

                file_server {

                        root /var/git

                }

        }

        ####< git-daemon ####

        import gomodhandler bacillus

        import gomodhandler brevity

        import gomodhandler chacha20

        import gomodhandler cryptmt

        import gomodhandler go-frodokem

        import gomodhandler goutmp

        import gomodhandler groestl

        import gomodhandler herradurakex

        import gomodhandler hkexsh

        import gomodhandler hopscotch

        import gomodhandler kyber

        import gomodhandler lpasswd

        import gomodhandler moonphase

        import gomodhandler mtwist

        import gomodhandler newhope

        import gomodhandler xs

        import gomodhandler xsd

}

Footnote: If switching on-the-fly to Caddy from Apache and vice-versa while testing the overall setup while an active Nextcloud instance is running involves also changing ownership of: nextcloud install and config dirs in /var/www/...; the nextcloud /data dir; *and* any existing session files, /tmp/sess_*).

Footnote 2: Sometime recently (as of go v1.2x) the go get system seemed to start requiring any go module have at least a latest tag applied otherwise the module will not be found. I had a few modules that lacked tags, so that was a real head-scratcher. Also do a go clean -modcache from time to time to ensure go is really fetching things as opposed to using cached copies, so if dependencies have broken it will be revealed when attempting go mod init && go mod tidy.

An APL Translation of 'Square Joy: Trapped Rainwater' J Posting

 I saw an interesting post entitled "Square Joy: Trapped Rainwater" on the mmapped blog, describing how to approach the problem of computing water levels in a 2D Flatland cityscape.

The configuration of bars with heights 0, 1, 0, 2, 1, 0, 1, 3, 2, 1, 2, 1, and the water trapped by this configuration.

The author uses J, so I thought I'd take a quick look at converting the basic computation into its parent language, APL. Turns out it's quite simple (minus the graphic representation which I haven't yet looked at doing in GNU APL).

J has the  /\. adverb ('suffix'), here applied to the 'insert' (/) which it shares with APL. APL must simulate it via two applications of the reversal (⌽) function:

      H ← 0 1 0 2 1 0 1 3 2 1 2 1
      ⌈\H
0 1 1 2 2 2 2 3 3 3 3 3
      ⌽⌈\⌽H    ⍝ Note the use of 'reversal' here (⌽) with insert (/) to match J's 'suffix scan'
3 3 3 3 3 3 3 3 2 2 2 1
      (⌈\H) ⌊ (⌽⌈\⌽H)
0 1 1 2 2 2 2 3 2 2 2 1

Maybe later I'll try to create the visual renderings in GNU APL to match what the J solution does.

String Interpolation in APL -- Formatted string output ala printf()

This is not as sophisticated as C's printf() of course, but it's enough for many uses.

  ∇r ← s sInterp sv
⍝⍝ Interpolate items in sv into s (string field substitution)
⍝ s: string - format string, '∆' used for interpolation points
⍝ sv: vector - vector of items to interpolate into s
⍝ r: interpolated string
  s[('∆'=s)/⍳⍴s] ← ⊃¨(⍕¨sv)
  r ← ∊s
∇
      'Mary had a ∆ lamb, its fleece was ∆ as ∆.' sInterp 'little' 'black' 'night'
Mary had a little lamb, its fleece was black as night.
      'Mary had a ∆ lamb, its fleece was ∆ as ∆.' sInterp 'little' 'large' 42
Mary had a little lamb, its fleece was large as 42.

A Recursive Depth-First Maze Generator in GNU APL

Rosettacode.org is a great place to grab little ideas and apply them to learning a new language; especially if there isn't a solution there yet in the language you're learning! This past week I took some spare time in the evenings to implement the classic maze generator problem, in GNU APL.

Takeaways:

• Sometimes just manipulating the string representation of a maze is easier than trying to come up with a clever binary representation (I started with the idea that each maze cell's walls could be a 4-bit field, eg. nsew = [3210], but the shared walls between cells made it too messy);
• GNU APL's ? (shuffle) operator appears to have a static seed and the docs don't clearly state how to seed it: the ⎕RL system variable has the shuffle op's internal state so assigning to it will seed the PRNG, eg:

⎕RL ← +/ ⎕TS ⍝⍝ Seed ⎕RL (?) PRNG with sum of timestamp Y, M, D, H, M, S, ms

#!/usr/local/bin/apl --script --
 ⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝
⍝                                                                    ⍝
⍝ mazeGen.apl                          2022-01-07  19:47:35 (GMT-8)  ⍝
⍝                                                                    ⍝
 ⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝⍝

∇initPRNG
  ⍝⍝ Seed the internal PRNG used by APL ? operator
  ⎕RL ← +/ ⎕TS   ⍝⍝ Not great... but good enough
∇

∇offs ← cellTo dir
  ⍝⍝ Return the offset (row col) to cell which lies in compass (dir)
  offs ← ∊((¯1 0)(0 1)(1 0)(0 ¯1))[('nesw'⍳dir)]
∇

∇doMaze rc
  ⍝⍝ Main function
  0 0 mazeGen rc      ⍝⍝ Do the maze gen
  m                   ⍝⍝ output result
∇

∇b ← m isVisited coord;mr;mc
  →( ∨/ (coord[1] < 1) (coord[2] < 1) )/yes
  →( ∨/ (coord > ⌊(⍴m)÷2) )/yes
  b ← ' ' ∊ m[2×coord[1];2×coord[2]]
  →0
yes:
  b←1
∇

∇c mazeGen sz ;dirs;c;dir;cell;next
  →(c≠(0 0))/gen
init:
  c ← ?sz[1],?sz[2]
  m ← mazeInit sz
gen:
  cell ← c
  dirs ← 'nesw'[4?4]
  m[2×c[1];2×c[2]] ← ' '  ⍝ mark cell as visited
dir1:
  dir ← dirs[1]
  next ← cell + cellTo dir
  →(m isVisited next)/dir2
  m ← m openWall cell dir
  next mazeGen sz
dir2:
  dir ← dirs[2]
  next ← cell + cellTo dir
  →(m isVisited next)/dir3
  m ← m openWall cell dir
  next mazeGen sz
dir3:
  dir ← dirs[3]
  next ← cell + cellTo dir
  →(m isVisited next)/dir4
  m ← m openWall cell dir
  next mazeGen sz
dir4:
  dir ← dirs[4]
  next ← cell + cellTo dir
  →(m isVisited next)/done
  m ← m openWall cell dir
  next mazeGen sz
done:
∇

∇m ← mazeInit sz;rows;cols;r
  ⍝⍝ Init an ASCII grid of size (rows cols) which
  ⍝⍝ has all closed and unvisited cells:
  ⍝⍝
  ⍝⍝  +-+
  ⍝⍝  |.|
  ⍝⍝  +-+
  ⍝⍝
  ⍝⍝ @param sz - tuple (rows cols)
  ⍝⍝ @return m - a (rows × cols) ASCII matrix
  ⍝⍝⍝⍝

  initPRNG
  (rows cols) ← sz
  r ← ∊ (cols ⍴ ⊂"+-" ),"+"
  r ← r,∊ (cols ⍴ ⊂"|." ),"|"
  r ← (rows,(⍴r))⍴r
  r ← ((2×rows),(1+2×cols))⍴r
  r ← r⍪ (∊ (cols ⍴ ⊂"+-" ),"+")
  m ← r
∇

∇r ← m openWall cellAndDir ;ri;ci;rw;cw;row;col;dir
  (row col dir) ← ∊cellAndDir
  ri ← 2×row
  ci ← 2×col
  (rw cw) ← (ri ci) + cellTo dir
  m[rw;cw] ← ' '   ⍝ open wall in (dir)
  r ← m
∇

⎕IO←1

doMaze 9 9
)OFF

russtopia@rlm-devuan:~/GNUAPL$ workspaces/mazeGen.apl 
+-+-+-+-+-+-+-+-+-+
|     |       |   |
+-+ + +-+-+ + +-+ +
|   |       |   | |
+ +-+-+-+-+-+-+ + +
|     |     | | | |
+-+-+ +-+ + + + + +
|   |   | |   |   |
+ + +-+-+ +-+-+-+ +
| | |   |       | |
+ +-+ + + +-+ +-+ +
| |   |   |   |   |
+ + +-+-+-+ + + +-+
| |   |   | | | | |
+ +-+ + +-+ +-+ + +
|   | | |   |   | |
+ +-+ + + +-+ +-+ +
|       |         |
+-+-+-+-+-+-+-+-+-+

Parsing script arguments in GNU APL (⎕ARG)

In scripting languages one often wants to run the overall script like a standalone program, with a top 'main' function, yet also allow the script to be read into the REPL for editing, testing interactively and so on.

In Python a common way to do this is by checking the __main__ variable to determine execution scope.

GNU APL has the --script switch to enable invoking a .APL file and immediately executing it. In standard UNIX fashion the -- switch is also useful to divide GNU APL arguments (those for the interpreter itself) from those intended for the script. Using these two bits of information one can do the same thing in APL:

#!/usr/bin/env apl --script --

∇scriptFunction
    ⍝⍝ Whatever the script should do by default when run from shell
    ⎕←'This is an APL script'
∇


⍝ Parse args & run 'main' if not in REPL mode
⍝ in REPL mode, ⎕ARG[1] is the script path itself
⍝ (Recalling that by default, ⎕IO = 1)
∇checkMode;args
    args←(⎕ARG⍳⊂"--")↓⎕ARG ⍝ Drop all args up to and including "--"
    →(~(⊂'--script')∊⎕ARG)/0 ⍝ If "--script" not present, quit to allow REPL
    ⍝⍝ ⎕←"[",args[1],"]" ⍝ (just debugging to see args)
    scriptFunction ⍎∊args[2] ⍝ in --script mode, so run default func
∇

checkMode ⍝⍝ check the ⎕ARG vector for --script mode

A Hexadecimal Pronunciation Guide, by Robert A. Magnuson - Datamation Vol. 14, No. 1, Jan 1968

I recall discussing with a fellow CS student many years ago the topic of how speaking hexadecimal values seemed to be.. well, clunky. We both thought it strange no one had standardized a set of word stems like we have for the base ten numerals, 'teens', 'fortieth', and so on. All these years later I discovered, as it turns out, someone did. It just didn't catch on, for some reason.

An obscure article in Datamation, Vol. 14, No. 1 dated January, 1968 (alternate link, archive.org) contains an article by one Robert A. Magnuson, proposing an extension to English number pronunciation for hexadecimal:

Table 2. New Names for Hexadecimal Digits
A ann
B bet
C chris
D dot
E ernest
F frost
The pronunciation of the -teen's and -ty's for the new
digits is shown in Table 3. Note that the analog of the decimal pronunciation system has been used. The new name for each new digit has been chosen so that at least one of the -teen and -ty modifications is familiar sounding.

Table 3. -Teen and -Ty Pronunciation for the New Digits

1 A annteen
1 B betteen
1 C christeen
1 D dotteen
1 E ernesteen
1 F frosteen
A0 annty
B0 betty
C0 christy
D0 dotty
E0 ernesty
F0 frosty

Now 29's successor 2A can be pronounced "twenty-
ann" without the slightest tendency to confuse it with 28.
The pronunciation of C4 is "christy-four," and that of A4
is "annty-four." There is no problem in distinguishing
between 1A, "annteen" and 18, "eighteen." And 88, 8A,
A8, and AA are easily distinguished when pronounced
"eighty-eight," "eighty-ann," "annty-eight," and "annty-
ann."

Some two-digit hex numbers, each representing one byte, appear with their new pronunciations in Table 4.

Table 4. One-Byte Strings with Pronunciation
2F twenty-frost
F2 frosty-two
5B fifty-bet
3E thirty-ernest
AF annty-frost

Some four-digit hex numbers, each representing two
bytes, appear with their new pronunciations in Table 5.

Table 5. Two-Byte Strings with Pronunciation
A01C annty christeen
1ED0 ernesteen dotty
A007 annty oh-seven
DEAF dotty-ernest annty-frost
3A7D thirty·ann seventy-dot
47F0 forty-seven frosty

The problem of the values of the added digits being off
by one is now easily solved. Merely remember the "ages"
of these new-found friends. Learn that ann is 10, bet is
11, etc. without becoming confused with the fact that A
is associated with 1, B with 2, etc.

The problems of some of the hex digits being NUMERIC
and the others ALPHA on the model 29 keypunch is solved
in the following fashion. Select a particular finger of the
left hand, say, the little finger. No other finger of the left
hand is to be used. The home position of that left little
finger is on the NUMERIC key. The right hand is used for
typing 0-9 and the (numeric) comma while the left little
finger holds down the NUMERIC key. When· A - F arise,
they are typed with the left little finger-thus ensuring
that the NUMERIC key is not depressed. Return the left
finger to the home position on the NUMERIC key imme-
diately upon finishing A-F.

 

APL Keyboard Sticker Set

APL Keyboard Stickers for Laptop or Desktop

APL uses a special symbol set, which presents a barrier to entry for new users. Furthermore, there don't seem to be many options to get a keyboard with APL symbols, other than expensive specialty ones (here or here), and they're for PC desktops only; so what's a laptop user to do?

I figured I'd take matters into my own hands, and produce a run of APL keyboard stickers. These match the standard APL keyboard map used by historical APL implementations as well as GNU APL and Dyalog APL. I got the shipment at end of August, 2021, and they look pretty good on my laptop. Now available on my Tindie store. Check it out!

Black vinyl keycap stickers, suitable for any standard laptop or PC desktop keyboard.

To use these you'll need to set a custom keymap. For Linux, See my post here. For Windows, see here, here and finally here.

With some trivial setup, modern systems have no issue at all supporting APL symbols. I wonder if its successor, J, would have even come to be if the APL character set wasn't seen as an impediment prior to the 2000s. As a newcomer to APL and J, comparing the two, J just seems so much harder to parse as it uses ASCII exclusively; it looks a lot more like line noise to me whereas APL encourages seeing a unique notation.

Getting Started with APL: Keyboard Mapping and Using GNU APL on Android with TermUX

APL is an under-appreciated language, pioneering many concepts that have re-appeared in almost every language since. Iverson Ghosts! Boo.

A huge hurdle for new users is those funny symbols. How does one enter them on a modern keyboard?

Wonder no longer! It's not so tough, just not documented that well on the web today. APL has its own Unicode point so all the symbols are already inside your box waiting to be used. One can even use GNU APL on an Android phone or tablet with the Hacker's Keyboard (fork for APL layout, see bottom)

One would think APL enthusiasts would have keymaps with an easy setup script to run, to encourage newcomers, but I guess not since I didn't run across any... so here's my attempt to help out any other APL-curious people out there.

BONUS: Hard-to-find APL book links. These books are out of print and insanely expensive on the used market. I manually scanned in APL2 At A Glance to save it for posterity. It really is the best introductory text for APL2.

APL2 At a Glance (English)

APL2 Ein erster Einblick (APL2 At A Glance, German)

APL2 In Depth

Adding an APL Keymap to X terminals

From X (any windowed terminal, eg. Konsole, xfce4-terminal etc.)

--

My preferred setup uses CAPS LOCK as the APL key:

setxkbmap us,apl -option grp:caps_switch

Try the above interactively, then put it into your .bashrc and you're good to go. If you *really* need CAPS LOCK for some reason, ALT+CAPS works.

One might also be able to define it system-wide via an xorg.conf.d/ file such as the following snippet, but I haven't rebooted yet to test it:

Section "InputClass"

Identifier "system-keyboard"

MatchIsKeyboard "on"

Option "XkbLayout" "us,apl"

Option "XkbModel" "pc104"

#Option "XkbVariant" ",dvorak"

Option "XkbOptions" "grp:caps_switch"

EndSection

APL Keys For 'dfns' (lambdas)

Dyalog APL and GNU APL both use the ⍺ and ⍵ symbols for the unbound left- and right-hand variables of 'dfns' (basically lambda functions). These are standard for both APL dialects, but GNU APL also recognizes APL Functional Symbol alpha-underber, ⍶:U+2376, APL Functional Symbol omega-underbar, ⍹:U+2379 and lowercase Greek chi, χ:U+03C7.

On my Dell XPS 15 laptop, it seems using the CAPS key as the APL COMPOSE in combination with the SHIFT key does not work for the W/S/X key column. So I chose the U and C keys as alternates to yield omega-underbar and chi, respectively, instead, as Dyalog and GNU APL don't appear to have anything assigned to those with SHIFT. Modifications to my .Xmodmap are thus as follows:

keycode  25 = w W U2375 U2379

keycode  38 = a A U237A U2376

keycode  30 = u U U2193 U2379

keycode  53 = x X U2283 U03C7

keycode  54 = c C U2229 U03C7

.. and yet on my Acer Aspire, keys W and X work just fine. Weird.

To customize your X modmap (be sure you've already added the ,apl to your xkbmap above):

$ xmodmap -pke >~/.Xmodmap

$ vi .Xmodmap  ## editing the above

$ xmodmap ~/.Xmodmap   # To reload

Download my .Xmodmap keymap file here:

https://www.blitter.com/nextcloud/index.php/s/bPz2NxpKyAPosqN

APL Linux Console Key Map

Download my console keymap file here: 

https://www.blitter.com/nextcloud/index.php/s/9oY7pEbDZeiZNfq

As root,

# loadkeys APL.kmap

Almost the same as the X mapping -- CAPS LOCK and shift+CAPS LOCK are combo keys with any other key in the GNU APL mapping for APL symbols; right-ALT is APL-mode lock.

APL fonts require running within an 'fbterm' on virtual consoles, so install that first, and if you want to use APL from the console often, start 'fbterm' and 'loadkeys' from startup.

GNU APL on Android with TermUX

* open TermUX.

* Make sure you have lots of free space on your phone, as you're going to be building GNU APL from source using g++

$ apt-get install subversion

$ apt-get install g++

## might need autotools and other things as well (automake, autoconf, etc.)

$ svn checkout svn://savannah.gnu.org/apl $ cd apl/trunk

You may have issues with network timeouts checking out the repo. If so, retry after running an svn cleanup. if you keep having trouble, fetch it on a PC Linux machine, tar+gzip the whole svn/trunk dir and then use 'scp' to just copy it to your TermUX home dir (either by installing openssh in TermUX or using an Android ssh program... but you'll have to hunt around in your phone's filesystem to find your TermUX home dir in that case...)

[Again, within TermUX]:

$ apt-get install ncurses pcre pcre2
$ cd trunk/   # wherever gnu apl source from svn was fetched
$ ./configure
$ make
$ mkdir $HOME/bin
$ cp src/apl $HOME/bin
$ export PATH=$HOME/bin:$PATH ## or set up a .bashrc or .profile with this to make it permanent

.. I might have missed a few apt-get calls required for some libraries but otherwise it did build just fine right on the phone!

Oh, and remove the svn/trunk afterwards because you'll probably be low on space :)

Finally, install the fork of Hacker's Keyboard with APL layout/language available here!

Locking Linux X Sessions without XScreensaver

 I'm currenlty using Funtoo Linux on my main laptop, and XScreenSaver was displaying a big scary warning "This version of XScreenSaver is very old! Please upgrade!". It turns out the author of XScreenSaver is, shall we say, very opinionated about downstream distros modifying this warning in any way -- though the codebase itself is allegedly free for downstream to modify, if this particular warning is touched by downstream the XScreenSaver author has insisted the program be removed entirely, said author even protesting a commonsense modification to the warning to redirect users first to the specific distribution for bug reports.

Normally I don't care so much about OSS drama, but this warning timebomb just seems antisocial. I don't care much about screensaver functionality, I just want simple session locking and the warning was bugging me. So I went searching and surprisingly there are not many other options for X session locking that aren't tightly bound to specific widget families. There's Gnome-screensaver, some built-in lock with KDE, and then there's the most neutral setup I found so far: i3lock launched from xautolock.

Depending on your desktop it takes a bit of manual setup but it's not too hard. For my XFCE4 setup, what worked for me is:

1. Create a script (remember to chmod u+x) named autolock.sh in ~/.config/autostart-scripts/

#!/bin/bash
/usr/bin/xautolock -time 10 -locker "/usr/bin/i3lock -c 303030" &
sleep 1
pgrep xautolock && notify-send -u normal -t 4000 "xautolock active"

2. Now set that script to auto-run on login, by creating a new entry called in the XFCE4 Settings > Session and Startup, Application Autostart tab.

3. Set a keyboard shortcut eg., for Super-L, in XFCE4 Settings > Keyboard, Application Shortcuts tab, calling i3lock (use the -c option for a custom colour if you don't like the default white screen).

Log out and log in, and verify xautolock is running -- that's it!

Using HTTP Basic Auth (with Logout!) in a Go Application

HTTP Basic Auth (Wikipedia) is a thing that is actually still quite useful despite its neglect in modern web standards. By neglect, I mean that it hasn't been updated since its introduction in RFC7617 and as such the logout mechanism hasn't been improved to take into account modern browsers' tendency to aggressively cache session data within the HTTP headers, which is where the login state is stored. However, with some tricks it still can reliably be used in modern browsers. But some Javascript is required, sorry :(

First, a note: don't even consider using HTTP basic auth in your public-facing page unless you have it served behind an HTTPS reverse proxy! The username and password, sent to and fro from client to server in the HTTP headers, is in plaintext by default, and only HTTPS with TLS will guarantee that the credentials in those HTTP headers are encrypted.

Given that caveat, here is a complete minimal example of using HTTP basic auth to gate access to a Go web app.

Go Playground Example <-- this won't work in the Playground -- copy and build locally

FAQ

Q: Go's http lib supports TLS to serve out endpoints. Why didn't you just do that instead of serving out an HTTP app behind a reverse proxy?

A: HTTP basic auth seems to be mutually exclusive with direct use of the HTTPS protocol (see my comment at start of this post about the 'basic auth' mechanism being neglected...). Perhaps I missed something. Let me know if I'm wrong, and how to do it securely without an HTTPS reverse proxy! Thanks.

Q: How do I support multiple users/roles using the example you give?

A: No idea. I think it could be done, with auxiliary logic to track separate session users/passwords, but this is left as an exercise for the reader. [Meaning, like all my college profs ... I forget/I can't be arsed to work it out right now.]

-R.

bacillμs - a simple build automation server written in Go

Most non-trivial software projects that do rapid releases use a build automation server. One of the most popular solutions for this is Jenkins. There are many others.

While Jenkins is easy to install and use in my experience, I wanted to learn some others to broaden my expertise, like Concourse https://concourse-ci.org/ or buildbot https://github.com/buildbot. The former turned out to be hellish to install and I burned a few evenings hitting many head-scratching dead-ends in the start up config; forums were filled with users asking the dev team for updated installation instructions, met with brusk dismissals if one wanted to use it outside of the dev-blessed containers (ie., use our black box, never mind how it really works). The latter, while easier to get up and running in a 'hello world' configuration, seemed difficult to configure further into a real-world setup. It seemed to me that these things, in general, are overly complicated.

So, in a fit of insanity I wrote my own simple build automation server in Go. No containers, java VM, or dependencies.. Use whatever scripting language you want. Total line count is under 1k.

Of course, it's nowhere near production quality, and probably violates every go coding standard there is, but it does the essential things one might expect: responding to git triggers or webhooks from web-based systems like gogs.io or gitlab, a web dashboard, viewing of running and completed jobs and their artifacts. Jobs may be scheduled (externally via cron), or launched manually from the dashboard. Jobs can be parameterized, with a simple but powerful form notation to allow int, string, and boolean job parameters. Build artifacts are archived and browseable from the dashboard. There's even a simple way to display stages of a job's run in the live view, aka a simple 'pipeline' status.

https://gogs.blitter.com/Russtopia/bacillus/src/master/README.md

Suggestions welcome.

Obscure git issue: git push hangs silently on un-writeable repos

Making this blog post as a note mostly to myself; but since I couldn't find a posted solution elsewhere, this might also help someone else encountering a 'git push hangs' issue...

Situation: Recently installed Gogs (https://gogs.io), an awesome github-style self-hosted web service, written in Go. I have a bunch of repos already in /var/git/, and didn't want to move them, to preserve the ability to use go get (see this post about setting that up) and raw command-line git without changing the repo URIs everywhere they were already checked out on remote systems.

To put some of these repos under the purview of Gogs, yet keep them visible in /var/git/, I made soft symlinks in /var/git/, moving the actual repos in question to
/home/git/gogs-repositories/<user>/<repo>.git. By default, these will have permissions allowing git (and the Gogs web app) to manage the repo, eg:

drwxrwxr-x  6 git git 4096 Jun  8 20:48 go_login.git

However, since these were my legacy repos, for the above reasons the symlink at the location in /var/git/ is owned by another user and I didn't want the git user dedicated to Gogs owning things there, outside of the git user's home tree.

As it turns out, if the owner of the symlink in /var/git/ pointing to the moved repo within /home/git/gogs-repositories/ doesn't have write permissions to the repo, git push will just silently hang after one supplies ssh:// credentials.

Solution: Add the legacy user to group git, and add group write permissions to all repos linked to in this way in /home/git/gogs-repositories/ .

This was a head-scratcher, since git-daemon writes nothing to /var/log/daemon.log indicating an issue, at least on my setup -- perhaps git-daemon can be made to be more verbose?

-R.

Export Go Packages via 'go get' From Your Own Server

Self-Hosting Go Packages With Support For go get

[NOTE: Since originally posting I've clued in that what's documented here is only one way of achieving what's commonly referred to as 'vanity URLs' or 'vanity imports'. Adding this note here just to help anyone searching find this post more easily. -R.]

Go has a really neat package import tool, go get, to fetch packages from upstream sources into one's own local $GOPATH package tree. The 'big' sites like github.com, gitlab.io and others support use of go get from their project hosting spaces, which is cool, but they charge extra for hosting private code repos, or having more than a small fixed number of contributors, or other annoying limitations. Understandably these sites need some way to monetize their cloud offerings but for individuals or those with their own infrastructure there should be other ways that don't depend on the 'cloud' (ie., someone else's servers).

While the collaborative aspects of these sites and web-based features are their main draw (encouraging public pull requests for distributed development), perhaps you or your company want the convenience of using go get for your own repositories, but don't want to entrust your code repositories to one of these external entities.

Note: If you're considering moving off of github and self-hosting your repos, consider Gogs.io. It's really easy to set up and feels very familiar if you're used to github. Also, see my other post for notes on how to let Gogs.io refer to your legacy repos whilst preserving traditional access to your old repos in their original locations. 

The go get command and its import mechanism is described in the go command documentation, but to be frank, the docs for the go import mechanism aren't too clear on exactly how to set up one's own server to support it. One can't just go get a repo that is available via git clone without a lot of setup first.

Basic requirements:

• Proper DNS 'A' record info for your package server
• A common webserver (ie., apache v2 is used here but others are supported)
• HTTPS enabled (ie., a properly-configured, authority-signed server cert -- sorry, self-signed won't work)
• The git-http-backend helper tool (included with most git distributions)
• Properly configured web server rewrite rules for calling git-http-backend when requests from go get are seen by your server

All these bits need to be set up 'just so' for the go get command to work smoothly, and the go docs don't really spell out the full setup, probably due to the myriad platforms and web servers out there.

I'll show here my setup, which isn't the most common, but should with ease adapt to other systems: Funtoo Linux + Apache v2. With some path adjustments this should apply to Ubuntu and other popular Linux distros.

I pieced together this tutorial from the following sources:

https://askjong.com/howto/automatically-enable-https-on-your-website-with-effs-certbot-deploying-lets-encrypt-certificates
https://kasunh.wordpress.com/2011/01/15/git-over-https/
https://www.creang.com/howtoforge/howto_set_up_git_over_https_with_apache_on_ubuntu/

I also studied the verbose output of go get -d -v to see just what the command was assuming when it tried to fetch things.

Basic Theory of 'go get'

The go get command works over SSH, HTTP or HTTPS, though it refuses to use plain HTTP unless one specifies the -insecure flag. This means generally you'll want to get your server's HTTPS cert setup working to avoid having to specify this every time, and, in the case of private repositories, to protect your proprietary source code from travelling over the open internet whenever go get is run.

The tool looks for files with special <meta> tags, which specify where to redirect the partial URI given by the go get command to the git-http-backend tool. In this way, one can store the actual repositories nearly anywhere on the system and move them around, without breaking the package URI published to users.

go get can fetch packages contained in each <meta> tag via either the ssh:// or https:// protocols. The ssh:// protocol will require a shell account on the hosting server for each of your contributors -- they'll be prompted for their password before go get will proceed to pull anything. This is good for private groups wishing to share both read (pull) and commit (push) access. For public repos or projects where you want team members to submit patches via other means like email or an external review tool, the https:// method is appropriate -- however it will require a web server with valid authority-signed cert to allow HTTPS.

Proper DNS 'A' record setup

You'll need to ensure your domain allows proper HTTP/HTTPS access with the bare domain (ie., foo.com should redirect to www.foo.com). go get and package imports in go source code expect just a domain name, not a host.domain syntax, eg. the Go source import statement

import   "example.org/go/mylib"

... implies one has previously performed

$ go get example.org/go/mylib

... which expects the server at example.org to resolve web requests with no host prefix. If you serve regular web content from the same server, you'll probably already have an 'A' record for www.example.org, but go get will require an 'A' record also for plain example.org. While you're doing this you might as well add a permanent redirect from example.org to www.example.org if you don't already have it.

Check your DNS configuration (if you control it yourself) or ask your admin to ensure there's an 'A' record for example.org  which maps to the same IP address as www.example.org. Sometimes this is named the '@' entry.

Apache modules required: mod_rewrite, mod_cgi, mod_alias, mod_env

The web server needs to do some URL rewriting and CGI operations in order to send go get requests to git-http-backend (ie., fetching git repos with the http:// or https:// prefix). For this you'll need to ensure the following Apache modules are enabled: mod_rewrite, mod_cgi, mod_alias, mod_env.

Enable the above modules by adding LoadModule directives in whatever manner your server  expects, eg., /etc/apache2/httpd.conf;  then add the  following .htaccess    rule   to   your   web   root   (mine,    using    apache2,    is in /var/www/localhost/htdocs/.htaccess):

RewriteEngine on
RewriteBase /
RewriteCond %{HTTP_HOST} !^www\. [NC]
RewriteRule ^(.*)$ http://www.%{HTTP_HOST}/$1 [R=301,L]

This rule rewrites requests of the form

http://example.org/foo

to

http://www.example.org/foo

Configuring RewriteRule to allow proper <meta> tags per-repo

Now you need to somehow let Apache distinguish regular web traffic from 'go get' queries, which implicitly look for files served with a <meta> tag that is unique per package.

I experimented for a while without success, adding multiple  <meta>  tags, one for each repo, to my webroot's index.html <head> section, until I realized that 'go get' was only looking at the first <meta> tag it found. It turns out 'go get' expects there to be only one <meta> tag in a file, so each exported go package must have its own file with its own <meta> tag.

The solution is not to put <meta> tags into the webroot index.html at all, but rather to use another mod_rewrite rule to distinguish 'go get' requests by the repo name and point them to a unique URL for each. These URLs should reside within a subfolder of the webroot.

Add this line to the .htaccess file in your webroot (see 1. above, mine was /var/www/localhost/htdocs/.htaccess):

RewriteRule ^go/(.*)$ pkg/$1 [QSA]

The [QSA] means (I think) 'Query String Append', which keeps any CGI-style GET
params in the original URL and puts them back onto the end of the re-written
URL, which may be important for 'go get' as it sends a '?go-get=1' param for its own purposes.

Now, with the above rule, let's say you have a file structure like this in your webroot:

/var/www/localhost/htdocs/pkg/
/var/www/localhost/htdocs/pkg/foo
/var/www/localhost/htdocs/pkg/bar
/var/www/localhost/htdocs/pkg/private-baz

.. and git repositories served by your git-daemon in /var/git/foo.git, /var/git/bar.git, and /var/git/private-baz.git. You can set up files in the webroot that contain <meta> tags pointing to each:

[/var/www/localhost/htdocs/pkg/foo]
<meta name="go-import" content="example.com/go/foo git https://example.com/git/foo">

[/var/www/localhost/htdocs/pkg/bar]
<meta name="go-import" content="example.com/go/bar git https://example.com/git/bar">

[/var/www/localhost/htdocs/pkg/private-baz]
<meta name="go-import" content="example.com/go/private-baz git ssh://example.com/var/git/private-baz">

The files themselves don't need to be html files. They can be text files with just the <meta> tag.

NOTE 1: In the examples above, each go get exported repo is within a go/ subdirectory. This is required to give the apache2 server a pathname root to 'hook onto' for its RewriteRule, otherwise there's no way to tell other requests within your web server's URI space apart from ones specifically meant for go get. The sub-directory doesn't need to be named 'go', it could be anything; just as github places repos under your username, eg. github.com/ThisUser/that-repo.

NOTE 2: make sure your git-daemon has  --export-all, or  a file named git-daemon-export-ok in each public git repo. Test with regular git clone commands to verify each is fetchable before trying to use go get with <meta> tags. Repositories exported with ssh:// appear to use the git-daemon-export-ok file when determining whether a repo is available via go get, whilst ones exported in the <meta> tag via https:// listen to the Apache SetEnv statements (see below) which set the export permissions, since they're being served via the git-http-backend helper rather than via ssh.

More on Public vs. Private Package Repos

If you have some private packages that are not yet ready for the public eye, make note of the above example: the repo named 'private-baz' was exported in the <meta> tag via ssh://, not https://, so it will ask for authentication via ssh (password, phrase or host-key).

Exporting via <meta> tags, but using ssh:// in the git repo URI, doesn't require your webserver to have HTTPS set up, but will require the -insecure flag to 'go get' to convince it to even fetch the <meta> redirection info so it's still annoying and worth going full HTTPS on your webserver even if you're not publishing anonymous read (pull) go packages.

Finally, note the ssh:// URI for git repos usually has a slightly different path than git:// or https:// read-only URIs (note the /var/git/ path component in the third private-baz repo).

You can even serve out multiple users' repos via 'go get' this way, since using git  with  the  ssh:// (git+ssh) URI  syntax  lets  a  git-daemon  otherwise configured  to serve  public  repos from  /var/git or wherever, to also serve out individual users' private repos from their home dirs. For example I have
public repos in my /var/git/ and private repos in ~user/git/,  and both can be served to the 'go get' command via  appropriate  <meta> tags defined as above, with private ones doing authentication as expected.

git-http-backend Setup

In your main apache2 config (eg., httpd.conf or similar) add this:

SetEnv GIT_PROJECT_ROOT /var/git
SetEnv GIT_HTTP_EXPORT_ALL
ScriptAlias /git/ /usr/libexec/git-core/git-http-backend/

RewriteCond %{QUERY_STRING} service=git-receive-pack
#[OR]
#RewriteCond %{REQUEST_URI} /git-receive-pack$
RewriteRule ^/git/ - [E=AUTHREQUIRED:yes]
<LocationMatch "^/git/">
  #apache 1.x# Deny from env=AUTHREQUIRED

  AuthType Basic
  AuthName "Git Access"
  Require all granted
  #apache 1.x# Require group committers
  #apache 1.x# Satisfy Any
</LocationMatch>

'safe' directory setting (git v2.35.2 and newer, 2024)

git added a security feature to check user ownership before allowing git operations, which messes up Apache's use of git-http-backend. In order for go to export packages properly one must disable the new strict ownership checks at the git system level (doing it per-repo would be best, but to get things running just disable it for all):

git config --system --add safe.directory '*'

LetsEncrypt

Now, after all of the above, I discovered go get refuses to import packages with a self-signed
cert! What a pain.

If you don't already have HTTPS with a certificate-authority signed cert on your server, you'll need to get one. Either consult your business IT department for the server hosting all of this, or set up EFF's certbot utility. Thankfully the EFF has made it relatively easy for regular people to get a free certificate with valid signing for personal servers.

On Gentoo or Funtoo, the steps to install a LetsEncrypt cert are (as root):

# emerge app-crypt/certbot app-crypt/certbot-apache
#
# certbot certonly --webroot -w /var/www/localhost/htdocs/ -d example.com -w /var/www/localhost/htdocs/ -d www.example.com

Now, verify the Apache configuration from all previous steps and restart the web server:

# apache2ctl configtest
# rc-config restart apache2

Now test out your fancy go get-able package server!

[from some other host or account]
$ go get example.com/go/foo
$ ls $GOPATH/src/example.com/go/foo

This is the minimum setup just to get HTTPS working with Apache v2 for your primary domain, to make go get happy. If you have multiple 'vhost' domains or other complex requirements, you're on your own.. I'm still trying to get my server to server full HTTPS for all of the domains it hosts.

Conclusion

While the go get command is the preferred way for golang programmers to fetch external packages into their working $GOPATH tree, the documentation is not extremely helpful in setting up all of the server-side bits that are required to support it. Individuals or organizations may want a mixture of public (read-only) as well as private/group read/write (pull/push) repos exported via go get without the risks or costs associated with hosting via an external party.

A self-hosted golang package server supporting the standard go get command can be implemented by configuring a webserver with proper type 'A' domain records, HTTPS plus a valid authority-signed certificate, proper git-http-backend tool configuration, URL rewrite rules and package export <meta> tags placed within the webroot on a per-package basis.