L4 fine tuning

[ALERT] 330/181418 (1) : parsing [/usr/local/etc/haproxy/haproxy.cfg:17] : timer overflow in argument '8760h' to 'timeout tunnel' (maximum value is 2147483647 ms or ~24.8 days)

The Kubernetes patch

• kubectl exec -it
  

  ~ > kubectl exec -v8 -it kube-apiserver-kind-control-plane -n kube-system sh
  (output omitted)
  I1127 23:54:11.979201   28755 round_trippers.go:427] Request Headers:
  I1127 23:54:11.979209   28755 round_trippers.go:431]     X-Stream-Protocol-Version: v4.channel.k8s.io
  I1127 23:54:11.979216   28755 round_trippers.go:431]     X-Stream-Protocol-Version: v3.channel.k8s.io
  I1127 23:54:11.979221   28755 round_trippers.go:431]     X-Stream-Protocol-Version: v2.channel.k8s.io
  I1127 23:54:11.979228   28755 round_trippers.go:431]     X-Stream-Protocol-Version: channel.k8s.io
  I1127 23:54:11.979234   28755 round_trippers.go:431]     User-Agent: kubectl/v1.18.0 (linux/amd64) kubernetes/9905a33
  I1127 23:54:11.995835   28755 round_trippers.go:446] Response Status: 101 Switching Protocols in 16 milliseconds
  I1127 23:54:11.995852   28755 round_trippers.go:449] Response Headers:
  I1127 23:54:11.995860   28755 round_trippers.go:452]     Connection: Upgrade
  I1127 23:54:11.995867   28755 round_trippers.go:452]     Upgrade: SPDY/3.1
  I1127 23:54:11.995873   28755 round_trippers.go:452]     X-Stream-Protocol-Version: v4.channel.k8s.io
  I1127 23:54:11.995878   28755 round_trippers.go:452]     Date: Wed, 27 Nov 2019 22:54:11 GMT
  # ⏎
  ~ >
  

• kubectl attach
  

  ~ > kubectl attach -v8 kube-apiserver-kind-control-plane -n kube-system
  (output omitted)
  I1127 23:59:52.210600   31733 round_trippers.go:420] POST https://127.0.0.1:35411/api/v1/namespaces/kube-system/pods/kube-apiserver-kind-control-plane/attach?container=kube-apiserver&stderr=true&stdout=true
  I1127 23:59:52.210609   31733 round_trippers.go:427] Request Headers:
  I1127 23:59:52.210614   31733 round_trippers.go:431]     X-Stream-Protocol-Version: v4.channel.k8s.io
  I1127 23:59:52.210618   31733 round_trippers.go:431]     X-Stream-Protocol-Version: v3.channel.k8s.io
  I1127 23:59:52.210621   31733 round_trippers.go:431]     X-Stream-Protocol-Version: v2.channel.k8s.io
  I1127 23:59:52.210625   31733 round_trippers.go:431]     X-Stream-Protocol-Version: channel.k8s.io
  I1127 23:59:52.210631   31733 round_trippers.go:431]     User-Agent: kubectl/v1.18.0 (linux/amd64) kubernetes/9905a33
  I1127 23:59:52.298756   31733 round_trippers.go:446] Response Status: 101 Switching Protocols in 88 milliseconds
  I1127 23:59:52.298777   31733 round_trippers.go:449] Response Headers:
  I1127 23:59:52.298783   31733 round_trippers.go:452]     X-Stream-Protocol-Version: v4.channel.k8s.io
  I1127 23:59:52.298788   31733 round_trippers.go:452]     Date: Wed, 27 Nov 2019 22:59:52 GMT
  I1127 23:59:52.298793   31733 round_trippers.go:452]     Connection: Upgrade
  I1127 23:59:52.298798   31733 round_trippers.go:452]     Upgrade: SPDY/3.1
  ~ >
  

• kubectl logs -f
  

  ~ > kubectl logs -v8 -lk8s-app=kube-dns -f -n kube-system
  (output omitted)
  I1127 23:58:21.008411   31143 round_trippers.go:420] GET https://127.0.0.1:35411/api/v1/namespaces/kube-system/pods/coredns-5644d7b6d9-8vflb/log?follow=true&tailLines=10
  I1127 23:58:21.008411   31143 round_trippers.go:420] GET https://127.0.0.1:35411/api/v1/namespaces/kube-system/pods/coredns-5644d7b6d9-r2vkw/log?follow=true&tailLines=10
  I1127 23:58:21.008436   31143 round_trippers.go:427] Request Headers:
  I1127 23:58:21.008440   31143 round_trippers.go:427] Request Headers:
  I1127 23:58:21.008445   31143 round_trippers.go:431]     Accept: application/json, */*
  I1127 23:58:21.008447   31143 round_trippers.go:431]     Accept: application/json, */*
  I1127 23:58:21.008452   31143 round_trippers.go:431]     User-Agent: kubectl/v1.18.0 (linux/amd64) kubernetes/9905a33
  I1127 23:58:21.008456   31143 round_trippers.go:431]     User-Agent: kubectl/v1.18.0 (linux/amd64) kubernetes/9905a33
  I1127 23:58:21.021126   31143 round_trippers.go:446] Response Status: 200 OK in 12 milliseconds
  I1127 23:58:21.021151   31143 round_trippers.go:449] Response Headers:
  I1127 23:58:21.021162   31143 round_trippers.go:452]     Content-Type: text/plain
  I1127 23:58:21.021179   31143 round_trippers.go:452]     Date: Wed, 27 Nov 2019 22:58:21 GMT
  I1127 23:58:21.021187   31143 round_trippers.go:452]     Cache-Control: no-cache, private
  .:53
  2019-11-27T22:37:59.639Z [INFO] plugin/reload: Running configuration MD5 = f64cb9b977c7dfca58c4fab108535a76
  2019-11-27T22:37:59.639Z [INFO] CoreDNS-1.6.2
  2019-11-27T22:37:59.639Z [INFO] linux/amd64, go1.12.8, 795a3eb
  CoreDNS-1.6.2
  linux/amd64, go1.12.8, 795a3eb
  I1127 23:58:21.023865   31143 round_trippers.go:446] Response Status: 200 OK in 15 milliseconds
  I1127 23:58:21.023881   31143 round_trippers.go:449] Response Headers:
  I1127 23:58:21.023890   31143 round_trippers.go:452]     Cache-Control: no-cache, private
  I1127 23:58:21.023908   31143 round_trippers.go:452]     Content-Type: text/plain
  I1127 23:58:21.023915   31143 round_trippers.go:452]     Date: Wed, 27 Nov 2019 22:58:21 GMT
  .:53
  2019-11-27T22:37:59.608Z [INFO] plugin/reload: Running configuration MD5 = f64cb9b977c7dfca58c4fab108535a76
  2019-11-27T22:37:59.608Z [INFO] CoreDNS-1.6.2
  2019-11-27T22:37:59.608Z [INFO] linux/amd64, go1.12.8, 795a3eb
  CoreDNS-1.6.2
  linux/amd64, go1.12.8, 795a3eb
  F1127 23:58:22.025841   31143 helpers.go:114] error: unexpected EOF
  ~ >
  

• kubectl get -w
  

  ~ > kubectl get -v8 -w pods -n kube-system
  (output omitted)
  I1128 00:07:33.132398   36629 round_trippers.go:420] GET https://127.0.0.1:35411/api/v1/namespaces/kube-system/pods?resourceVersion=3023&watch=true
  I1128 00:07:33.132412   36629 round_trippers.go:427] Request Headers:
  I1128 00:07:33.132421   36629 round_trippers.go:431]     Accept: application/json;as=Table;v=v1beta1;g=meta.k8s.io, application/json
  I1128 00:07:33.132430   36629 round_trippers.go:431]     User-Agent: kubectl/v1.18.0 (linux/amd64) kubernetes/9905a33
  I1128 00:07:33.133569   36629 round_trippers.go:446] Response Status: 200 OK in 1 milliseconds
  I1128 00:07:33.133583   36629 round_trippers.go:449] Response Headers:
  I1128 00:07:33.133590   36629 round_trippers.go:452]     Date: Wed, 27 Nov 2019 23:07:33 GMT
  I1128 00:07:33.133594   36629 round_trippers.go:452]     Cache-Control: no-cache, private
  I1128 00:07:33.133599   36629 round_trippers.go:452]     Content-Type: application/json
  I1128 00:07:34.135522   36629 streamwatcher.go:114] Unexpected EOF during watch stream event decoding: unexpected EOF
  ~ >
  

• Observations
  

  At first sight, exec and attach are very similar. Both are upgrading the HTTP connection to SPDY/3.1.

  On the other hand, logs -f and get -w are HTTP/1.1 requests.

• The patch
  

  The pull request I opened can be seen here. Please, bear in mind that it was a heavy WIP and many things are not even considered. Let's go through it.

• The strategy
  

  Different commands had slightly different strategies.

  • exec -it and attach
    

    Both commands are relying on apimachinery's httpstream. The main idea was to add a Ping() method to the Connection interface. In this case, this method would rely directly on calling to the docker's spdystream backend (that looks like this.)

  • logs -f
    

    ResponseWrapper interface has a Stream() function, so the idea was to add a new StreamWithPing() function. So users of the client-go library could choose which one to use depending on their needs, while keeping the original behavior intact.

  • get -w
    

    get -w is another slightly different case. It will run in a loop watching for changes, so there's no need to strictly keep the connection alive. If the connection is closed on the server end, the client could just be instructed to reissue the same request.

  • client-go
    

    Then, a keepalive package would be included into client-go, so it could be used with different backends (e.g. spdystreamer, golang's net/http and net/http2…). This package would define a Pinger interface containing a single function: Ping(). The different implementations here is in part what I liked the least about this proposal. There were several pinger implementations in the PR:

    • spdy pinger
    • restclient pinger
    • http pinger

    While the spdy pinger was clearly the implementation I felt more comfortable with, it wasn't the case with the restclient and http pinger implementations.

    All my tests were successful with all backends, but the restclient and http solutions were working because of the inner implementation of the internal socket reuse of golang's net/http internally.

    Apart from that, it wasn't possible to have a dedicated net/http2 backend because if HTTP/2 is used through golang's net/http (that includes net/http2 in a bundle), it's impossible to access net/http2 internals from within net/http – in particular the ClientConn associated with the connection, to Ping() the other end. Had golang permitted this somehow it would have been possible to take advantage of HTTP/2's native ping frame, what would have been sufficient to generate some TCP traffic, knowing that the right socket would have been used.

    The main problem with the restclient and http pingers is that if golang's internal implementation changed and didn't reuse sockets under certain circumstances our "pings" wouldn't go through the expected TCP socket, what wouldn't count towards TCP traffic after all, and eventually the connection would have been closed nevertheless.

    Also, there is the problem of what ping means in an HTTP/1.1 context. In my very first proposal of the PR it was a mere HTTP request to scheme://hostPort/healthz. Whether this operation was allowed, whether we had permissions to perform this operation, or even if the endpoint didn't exist are things we didn't really care about. As long as the request was made, and some TCP traffic made it through, the goal was achieved.

    In any case, the solution was not optimal for restclient, neither for http and it wasn't possible to properly extract and create a dedicated http2 pinger because of how golang hides the internal implementation of net/http2 behind net/http when using HTTP/2 through net/http.

    And so, while I was thinking about all these problems, and all these things I didn't really like about this possible solution, another possibility was proposed.

The golang patch

Notes

Blog

Front page

RSS

Arbitrary static pages

org-publish project alist

• website-content
  

  Published using org-html-publish-to-html, will generate the HTML files out of org files.

  • Sitemap generation: content/index.org
    

    Taking advantage of the sitemap functionality, content/index.org and the content/blog/feed.org files will be automatically generated. The sitemap-function of the website-content project is set to ereslibre/sitemap. It looks like:

    (defun ereslibre/sitemap (title list)
      (progn (ereslibre/generate-org-rss-feed list)
             (format "#+options: title:nil\n
                      #+begin_export html\n
                      <div class=\"content container front-container\">
                        <div class=\"side-by-side\">
                          <h1 class=\"post-title\">Notes</h1><hr/>
                          %s
                        </div>
                        <div class=\"side-by-side\">
                          <h1 class=\"post-title\">Blog</h1><hr/>
                          %s
                        </div>
                      </div>\n
                      #+end_export"
                     (ereslibre/all-entries 'notes list)
                     (ereslibre/all-entries 'blog list))))
    

    Writing raw HTML here helps in getting the desired HTML layout easier. This will be automatically generated anyway, and I don't have to fiddle with this anymore – I hope!

    Take the first progn call to ereslibre/generate-org-rss-feed into account, we'll check what it does later.

    Each sitemap entry is formatted using ereslibre/sitemap-format-entry, creating the link to the given entry:

    (defun ereslibre/sitemap-format-entry (entry style project)
      (let ((date (ereslibre/org-publish-find-explicit-date entry project)))
        `(:content ,(format "<div class=\"post-preview\">
                                <h2 class=\"post-title\">%s</h2>
                                <span class=\"post-date\">%s</span>
                             </div>"
                            (org-export-string-as (format "[[file:%s][%s]]" entry (org-publish-find-title entry project)) 'html t)
                            (if date
                                (format-time-string "%Y-%m-%d" date)
                              "&nbsp;"))
          :entry ,entry)))
    

  • Feed RSS generation: content/blog/feed.org
    

    ox-rss expects a single file with all blog posts, but this is not how my set up works, so the feed.org file will be automatically generated – this has some caveats, though.

    The contents ox-rss expects are of the form:

    #+title: ereslibre.es
    
    * [[file:year/month/some-post.org][Some post]]
     :PROPERTIES:
     :RSS_PERMALINK: blog/year/month/some-post.html
     :PUBDATE:  2019-09-06
     :ID:       0b382fe7-f943-4997-8568-28179abe8f23
     :END:
    Blog post contents, or description.
    
    * [[file:year/month/some-other-post.org][Some other post]]
     :PROPERTIES:
     :RSS_PERMALINK: blog/year/month/some-other-post.html
     :PUBDATE:  2019-09-06
     :ID:       d0d46dcf-ae23-42e5-b279-b17956b3d82a
     :END:
    Blog post contents, or description.
    

    The call on ereslibre/sitemap to ereslibre/generate-org-rss-feed is what generates the RSS feed contents. content/blog/feed.org file will be created, listing the contents of all blog posts.

    (defun ereslibre/generate-org-rss-feed (list)
      (let ((blog-entries (seq-filter (apply-partially #'ereslibre/is-entry-of-type 'blog) (cdr list))))
        (let* ((rss-contents (mapconcat #'ereslibre/rss-entry blog-entries "\n\n"))
               (full-rss-contents (concat "#+title: ereslibre.es\n\n" rss-contents)))
          (write-region full-rss-contents nil "./content/blog/feed.org"))))
    

    The ereslibre/rss-entry function looks like:

    (defun ereslibre/rss-entry (entry)
      (let* ((entry (plist-get (car entry) :entry))
             (title (org-publish-find-title entry (ereslibre/website-project)))
             (date (org-publish-find-date entry (ereslibre/website-project)))
             (link (concat (file-name-sans-extension entry) ".html"))
             (source-file (concat (file-name-as-directory "content") entry))
             (source-file-dir (file-name-directory source-file))
             (home-url-prefix (plist-get (cdr (ereslibre/rss-project)) :html-link-home))
             (contents (with-temp-buffer
                         (org-mode)
                         (insert-file-contents source-file)
                         (beginning-of-buffer)
                         ;; demote all headlines
                         (save-excursion
                           (while (re-search-forward "^\\*" nil t)
                             (replace-match "**")))
                         ;; remove certain attributes from inserted org file
                         (save-excursion
                           (while (re-search-forward "^#\\+\\(title\\|date\\).*" nil t)
                             (replace-match "")))
                         ;; transcode embedded links to files -- e.g. expand relative paths
                         (save-excursion
                           (while (re-search-forward "\\[file:\\([^]]+\\)" nil t)
                             (let* ((match (match-string 1))
                                    (element (save-match-data (org-element-at-point))))
                               (when (not (or (eq (org-element-type element) 'example-block)
                                              (eq (org-element-type element) 'src-block)))
                                 (replace-match
                                  (concat "[" home-url-prefix
                                          (file-name-sans-extension
                                           (file-relative-name
                                            (expand-file-name match source-file-dir)
                                            "content"))
                                          ".html"))))))
                         (buffer-string))))
        (with-temp-buffer
          (insert (format "* [[file:%s][%s]]\n" (ereslibre/path-relative-from-to-relative-to entry "content" "content/blog") title))
          (org-set-property "RSS_PERMALINK" link)
          (org-set-property "PUBDATE" (format-time-string "%Y-%m-%d" date))
          (insert contents)
          (buffer-string))))
    

    It is worth dissecting what tasks it performs:

    • Insert the contents of the target org file inside a temporary buffer
    • Demote all headlines from the inserted content
    • Transcode embedded links to other relative files. Since we are copying and pasting the contents of a file that is in other subdirectory (content/blog/year/month), all its relative references to other files will be broken when writing the content/blog/feed.org file
      • ereslibre/path-relative-from-to-relative-to rewrites a relative path from the original directory, to a relative path from the target directory. This is used for the toplevel entries in feed.org and for correctness, since it's not really used when publishing – as far as I can tell
      • RSS readers won't know how to handle relative links like ../../../contact.html, so all [https://www.ereslibre.es/blog/2019/09/some-file.html] occurrences will be transcoded into a [https://html-link-home/some-path/some-file], only if they are not in src or example blocks
    • Write the entry itself
      • Insert the link to the blog post org file
      • Add RSS_PERMALINK and PUBDATE org properties
      • Insert the modified contents of the blog post
• website-assets
  

  Published using org-publish-attachment. This will copy all assets from assets inside public_html/assets. These are strictly template related assets.

• website-content-assets
  

  Published using org-publish-attachment. This will copy all assets from content to public_html. These are assets related to blog posts or pages themselves.

• website-rss
  

  RSS generation using the auto generated content/blog/feed.org file, that was created during the website-content publishing. It will only generate a target public_html/blog/feed.xml with a list of all the available blog posts.

RSS with broken <pre> in CDATA sections

(defun org-rss-final-function (contents backend info)
  "Prettify the RSS output."
  (with-temp-buffer
    (xml-mode)
    (insert contents)
    (indent-region (point-min) (point-max))
    (buffer-substring-no-properties (point-min) (point-max))))

<description><![CDATA[
<pre class="example">
require (
  k8s.io/kubernetes v1.16.0-beta.1
)
</pre>
]]></description>

<description><![CDATA[
<pre class="example">
require (
k8s.io/kubernetes v1.16.0-beta.1
)
</pre>
]]></description>

ox-publish insists in adding certain elements

• The global template case
  

  Even when setting certain configurations like :html-head-include-scripts or :html-head-include-default-style to nil, I was still getting some template related elements that I could not remove with configuration settings, so I wrote my really simple org-html-template.

• The <p> case
  

  When creating the index.org contents, I started with the approach of using @@html:some-html@@[[https://www.ereslibre.es/blog/2019/09/some-org-file.html][A link]]@@html:other-html@@, so I could deliberately use org's feature of linking other files, while having control of the HTML directly to create the expected structure.

  This didn't go well, as an extra <p> entity was printed at the beginning of the page, and moved the content a little. I could have fixed that with some CSS sorcery, but I didn't want extra output in my website either.

  Then, I took the path that is currently used, use #+begin_export html and generate the org links manually while still relying on org's linking:

  (org-export-string-as (format "[[file:%s][%s]]" entry (org-publish-find-title entry project)) 'html t)
  

• <title>'s inside <head> with non-optimal contents
  

  I didn't fix this issue, what I did instead was to change the title of this post. It was previously named:

  #+title: Building a personal website with ~org-mode~
  

  and I had to rename it to:

  #+title: Building a personal website with org-mode
  

  The first title output in the generated HTML was:

  <head>
    <title>Building a personal website with <code>org-mode</code></title>
  </head>
  

  I'm fairly sure this is a bug, but the question is then, what it should be:

  <head>
    <title>Building a personal website with ~org-mode~</title>
  </head>
  

  or

  <head>
    <title>Building a personal website with org-mode</title>
  </head>
  

  Since the solution was really easy, and I was not completely sure I want headlines with different formatting, I ignored this problem and removed the special formatting from the title of the article.