Graham King

Time 2020-07-22 19:02:58

Web Name: Graham King

WebSite: http://www.darkcoding.net

ID:32028

Keywords:

Graham,King,

Description:

What do you mean? What do you demand of your captain? Are you then so easily turned from your design? Did you not call this a glorious expedition? And wherefore was it glorious? Not because the way was smooth and placid as a southern sea, but because it was full of dangers and terror; because, at every new incident, your fortitude was to be called forth, and your courage exhibited; because danger and death surrounded it, and these you were to brave and overcome. For this was it a glorious, for this was it an honourable undertaking. You were hereafter to be hailed as the benefactors of your species; your names adored, as belonging to brave men who encountered death for honour, and the benefit of mankind. And now, behold, with the first imagination of danger, or, if you will, the first mighty and terrific trial of your courage, you shrink away, and are content to be handed down as men who had not strength enough to endure cold and peril; and so, poor souls, they were chilly, and returned to their warm fire-sides. Why, that requires not this preparation; ye need not have come thus far, and dragged your captain to the shame of a defeat, merely to prove yourselves cowards. Oh! be men, or be more than men. Be steady to your purposes, and firm as a rock. This ice is not made of such stuff as your hearts may be; it is mutable, and cannot withstand you, if you say that it shall not. Do not return to your families with the stigma of disgrace marked on your brows. Return, as heroes who have fought and conquered, and who know not what it is to turn their backs on the foe. June 6, 2020 The simplest CLI that could possibly work Posted in Software at 19:12 by graham Most teams have many little command-line utilities that need to be kept organised. At Screenleap for example I have written tools in Go to manage build artifacts, releases, servers, and local dev environments, to run test clients in various roles, and many other things.What s the simplest possible way to put all your ad-hoc tools in one place? We have a single binary called sl that you run as sl [command] [command options].The sl main function is basically this:var commands = map[string]func(){ "thing1": func1, "thing2": pkgA.Main,func main() { commands[os.Args[1]]()}The commands define their own flags, print errors, everything a normal binary would do. In practice sl should check os.Args[1] and print what commands are available, so here s the real code:func main() { if len(os.Args) == 1 { fmt.Println(usage()) os.Exit(1) cmd, ok := commands[os.Args[1]] if !ok { fmt.Println(usage()) os.Exit(1) cmd()func usage() string { s := "Usage: sl [cmd] [options]\nCommands:\n" for k := range commands { s += " - " + k + "\n" return s}Sure you could use one of the Go command line frameworks out there, but then you re imposing a dependency on your colleagues. We ve been using the above map-of-functions approach since I joined five years ago because I couldn t think of a simpler way to start, and so far we re happy with it.Advantages: Single binary. Everyone knows where to look for a tool, there s only one file to `scp` around, and in Go it saves a lot of disk space. Our servers are in Go, and these command line tools use the same libraries as the servers. It makes experimenting with a scenario quite fast, you can script with the server domain objects. No requirements on the individual commands, a command is just a function. The commands print their own usage documentation. No dependencies. May 22, 2020 Collect and handle multiple errors in Go Posted in Software at 18:33 by graham In Go, how do you run several operations that might return an error, and return those errors at the end? For example you might be stopping several services:func stopAll() error { if err := stopIndexer(); err != nil { // save the error but continue if err := stopAuth(); err != nil { // save the error but continue if err := stopJobs(); err != nil { // save the error but continue [...] return allTheErrors}There are many ways to do it. Here s how I do it, maybe it will be useful to you:func stopAll() error { var errs util.Errs // code for Errs is below errs.Add(stopIndexer()) errs.Add(stopAuth()) errs.Add(stopJobs()) return errs.Ret()You treat errs.Ret() like any other error. It s nil if none of the operations returned an error, otherwise it contains the text of all the errors. You can use errors.Is and errors.As on it, it will report if any of the internal errors match.Why not use one of the many packages other people wrote, or publish this as a package? I try very hard to minimize dependencies. Each dependency imposes a cognitive cost on your colleagues, it is not something that should be done lightly, or alone.Read the rest of this entry Comments September 13, 2019 A developer goes to a DevOps conference Posted in Software at 18:54 by graham As a professional software engineer and enthusiastic pro-am systems administrator, I have long been curious about DevOps.I spent the last two days at PDX DevOpsDays. Across organised talks, Ignite s, Open Spaces, and as many hallway conversations as I could fit in, here is what I learned about what DevOps is and does.DevOps is Ops with new tools. I went to DevOpsDays thinking DevOps means developers and operations merged into one team. I was wrong.I did not meet or hear from any developers. It was a gathering of system administrators who use version control, and write a lot of YAML. Programming languages were not a topic; I heard Python mentioned once in passing, that s it.DevOps means the veteran admins had to check in their personal scripts, and everyone is expected to automate more things. They relate to the software their business runs ( app , asset ) the way a merchant-navy captain relates to what s in the containers on his ship.Read the rest of this entry 13 Comments July 27, 2018 Reading MediaRecorder s webm/opus output Posted in Software at 20:34 by graham Javascript can now record, manipulate and play sound thanks to a whole range of audio-related API s. One of the simpler and more useful parts of that is MediaRecorder which can record sound, typically from the users microphone.const stream = await navigator.mediaDevices .getUserMedia({audio: true, video: false}); // Media Capture and Streams APIconst mediaRecorder = new MediaRecorder(stream); // MediaStream Recording APImediaRecorder.ondataavailable = (ev) = { // ev is a BlobEvent // ev.data is a Blob. Send it to the server.mediaRecorder.start(50); // call ondataavailable every 50msMediaRecorder on Chrome produces output with mime type audio/webm;codecs=opus.Let me guide you through the WebM / Matroska / EBML maze.The WebM format is a container, which can contain many things. In our case it contains audio encoded with the Opus codec. We are only going to look at the container, not the encoded audio.Read the rest of this entry 1 Comment Javascript s async/await and Promise in a few words Posted in Software at 15:30 by graham This is a very common pattern in Javascript:myFunc(value, successCallback, errorCallback) If myFunc could block (such as on network access), we don t want to block Javascript s single thread, so we asked to be notified when the operation is complete.Javascript added syntactic sugar to make this pattern more pleasant, in two steps.The first step is adding a Promise object. myFunc would now returns a Promise, and the caller looks like this:myFunc(value).then(successCallback, errorCallback)Very similar. An improvement is you can now return this Promise up the call chain, instead of passing down your callbacks, and you can chain then. I sometimes think of Promise as CallbackManager . It wraps them and gives them a nicer interface.Read the rest of this entry Comments March 29, 2018 Growing software, in two tweets Posted in Software at 19:34 by graham Gardening is my preferred metaphor for software. When I grow a new piece of software, there is a very predictable path I go through as I attempt to manage it s complexity. It is, in fast-forward, the history of programming language design. That process is short enough to capture in about 400 characters, as two tweets.Growing software, in two tweets:1. Write `main`. Put your program in there.2. `main` too big? Extract functions. Use global vars.3. Too many functions? Group functions and their data (replacing global vars) into objects. grahamking (@grahamking) March 12, 20184. Too many objects? Group objects into packages. Make some methods private.5. Too many packages? Split out a new project, either library or network service.6. Iterate from 2 when other functions grow too big. grahamking (@grahamking) March 12, 2018Read the rest of this entry Comments November 11, 2017 What Made Maddy Run Posted in Behaviour, Society at 22:58 by graham What Made Maddy Run by Kate Fagan, is a book about the importance of doing what you love, of really listening to the people close to you. It is a case study of both the valedictorians from Lives of Promise and the dead from Why People Die by Suicide. It could have been a book about Overtraining Syndrome.It is the factual account of the suicide of Madison Holleran, reconstructed from digital media (email, text, Instagram, etc) with help from her family.In our safe and well-fed developed world the single biggest killer of teens and adults is suicide. Case studies matter.Read the rest of this entry 1 Comment May 22, 2017 Learn Better book notes Posted in Behaviour at 05:25 by graham Learn Better, by Ulrich Bosner is an interesting, valuable book, that is too long. The information would fit comfortably into 50 pages, but you can t sell a book that short. Amazon is fixing this, but I have a strong preference for paper books over e-books. Anyway, here are my notes.The core ideasThere are two big ideas in Learn Better:I. Learning is a skill which you can improve atCarol Dweck s Mindset claims that individuals who believe this (versus thinking the ability to learn is fixed and innate) live a more successful life . If you picked up a book called Learn Better , I m guessing you already think you can learn better, so hey, you re half way there already.In noticed the same viewpoint in a large study of high-school valedictorians: The top students readily identified themselves as school smart . Academic talent, to them, meant the ability to excel at academic learning and school tasks such as note taking, memorization, and testing. Many of them clearly attributed their success primarily to effort rather than ability. II. Learning is a generative activityYou don t learn by loading information from somewhere into your brain. You have to create it. To learn, you must do.Tangent: I found this interesting because of the similarities with memory. We often think of memory as a video camera, yet recalling a memory is a creative activity. I d highly recommend Elizabeth Loftus s The myth of repressed memory which explains this very well.What does generative mean here? What kind of activities must you do to learn?Summarize.Tell other people what you learned (even imaginary other people); this is what we do in programming with the rubber duck.Take practice quizzes.Read the rest of this entry April 9, 2016 systemd socket activation in Go Posted in Software at 20:01 by graham To start a server on a port below 1024 (i.e. 80, 443), you need root permissions or capability CAP_NET_BIND_SERVICE, but you also want most of your server to run unprivileged, reducing your attack surface. The traditional way to achieve this was to start as root, bind the socket, then drop privileges. It s a hassle, and if you get it wrong it s a big security hole, so often we just run on an unprivileged port and use something like nginx to proxy. But no longer. There s a much better way: systemd socket activation.Here is a Go program that will listen on port 8080 if started manually, or port 80 if run via systemd.package mainimport ( "log" "net" "net/http" "os" "strconv"func main() { http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { w.Write([]byte("Hello World!")) if os.Getenv("LISTEN_PID") == strconv.Itoa(os.Getpid()) { // systemd run f := os.NewFile(3, "from systemd") l, err := net.FileListener(f) if err != nil { log.Fatal(err) http.Serve(l, nil) } else { // manual run log.Fatal(http.ListenAndServe(":8080", nil))Build it (go build hello.go) and put the hello binary on your path (~/bin/ in my case).Create /etc/systemd/system/hello.service:[Unit]Requires=hello.socket[Service]ExecStart=/home/graham/bin/helloNonBlocking=trueCreate /etc/systemd/system/hello.socket:[Unit]Description=Hello socket[Socket]ListenStream=80NoDelay=trueWe use NoDelay (TCP_NODELAY) and NonBlocking (O_NONBLOCK) because Go sets these by default on sockets it opens, and we want similar behavior.Reload the config in systemd (sudo systemctl daemon-reload), and you re done. If you start it manually it will listen on 8080, if you start it via sudo systemctl start hello it will listen on port 80.To stop the service you also need to stop the socket. If you don t stop the socket, incoming connections will auto-start the service (inetd style). systemd will warn you about this.sudo systemctl stop hello.socket helloHow it worksWhen systemd starts a process that uses socket-based activation it sets the following environment variables. To check whether we are being started via socket activation we just need to check if one of those environment variables is set.LISTEN_PID: The process id of the process who gets the sockets. This prevents a child forked from your main process from thinking it is being given some sockets.LISTEN_FDS: The number of file descriptors (sockets) your process is being given. In our case this will be 1.Every process gets three standard file descriptors: STDIN=0, STDOUT=1, and STDERR=2. Descriptors given to us by systemd hence start at number 3.Here are all the .socket file options. You should also set protections in your .service file, see The joy of systemd.TLS and HTTP2Edit /etc/systemd/system/hello.socket to use port 443:ListenStream=443Wrap the listener with tls.NewListener, and set tls.Config.NextProtos to be []string{"h2", "http/1.1"} to maintain http2 support. Here s the full // systemd run section:config := tls.Config{ Certificates: make([]tls.Certificate, 1), NextProtos: []string{"h2", "http/1.1"}, PreferServerCipherSuites: true,var err errorconfig.Certificates[0], err = tls.LoadX509KeyPair( "my_cert.pem", "my_cert.key",if err != nil { log.Fatal(err)f := os.NewFile(3, "from systemd")l, err := net.FileListener(f)if err != nil { log.Fatal(err)tlsListener := tls.NewListener(l, config)http.Serve(tlsListener, nil)That s it!UPDATE 2018: Vincent Bernat has an excellent blog-post on zero-downtime upgrades in Go with systemd. 1 Comment

TAGS:Graham King 

<<< Thank you for your visit >>>

Hot Websites