Skip to content
Snippets Groups Projects

Traefik needs a long time to server proper cert

    • Closed Issue created by Maarten de Waard @maarten

    Sometimes it happens that keycloak is coming up even faster than traefik. Then traefik will serve self-signed certs instead of the proper LE ones, even when the LE certs were successfully fetched ~10 mins ago.

    Next to fixing this we should also create a test to check the cert validity.

    Edited
    To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information

    Child items ...

    Activity

    • Maarten de Waard added Bug CI + 1 deleted label

      added Bug CI + 1 deleted label

    • Maarten de Waard changed milestone to %First user test

      changed milestone to %First user test

      By Varac on 2019-02-27T11:46:37 (imported from GitLab project)

    • Maarten de Waard
      Maarten de Waard @maarten ·
      Author Contributor

      found this bug again at https://code.greenhost.net/openappstack/bootstrap/-/jobs/20460

      By Varac on 2019-03-01T11:47:40 (imported from GitLab project)

    • Maarten de Waard
      Maarten de Waard @maarten ·
      Author Contributor

      Oh wait, shortly after reporting the job above, it recovered and now https://auth.ci-20460.ci.openappstack.net is serving a proper cert. It just needs a long time to do so. We must have a test for a proper cert.

      By Varac on 2019-03-01T11:49:49 (imported from GitLab project)

      Edited by Maarten de Waard
    • Maarten de Waard changed title from Traefik needs restart to serve proper cert to Traefik needs a long time to server proper cert

      changed title from Traefik needs restart to serve proper cert to Traefik needs a long time to server proper cert

      By Varac on 2019-03-01T12:17:09 (imported from GitLab project)

    • Maarten de Waard mentioned in commit 61951613

      mentioned in commit 61951613

      By Varac on 2019-03-01T12:17:30 (imported from GitLab project)

    • Maarten de Waard
      Maarten de Waard @maarten ·
      Author Contributor

      So the issue is that trafik recovers by itself, after some time it will server the proper LE cert, it just take a while.

      See i.e. https://code.greenhost.net/openappstack/bootstrap/-/jobs/20463:

      The deploment finished on 13:11:41, and shortly after the behave test fails (it should fail already when trying to open the admin console, I'll open an issue for not checking the valid cert). On my laptop I run this check:

      while ! curl -s https://auth.ci-20463.ci.openappstack.net/auth/>/dev/null ; do echo "$(date): Waiting for LE cert..."; sleep 5 ; done
…
Fr 1. Mär 13:14:27 CET 2019: Waiting for LE cert...
Fr 1. Mär 13:14:33 CET 2019: Waiting for LE cert...
Fr 1. Mär 13:14:38 CET 2019: Waiting for LE cert...

      So about 3 minutes after deployment the proper cert was served.

      I pushed the above check in .gitlab-ci.yml (61951613) so we wait until the LE cert is served.

      By Varac on 2019-03-01T12:23:00 (imported from GitLab project)

    • Maarten de Waard mentioned in commit f71d59f4

      mentioned in commit f71d59f4

      By Varac on 2019-03-01T12:24:19 (imported from GitLab project)

    • Maarten de Waard mentioned in commit bf7e9aaf

      mentioned in commit bf7e9aaf

      By Varac on 2019-03-01T12:28:16 (imported from GitLab project)

    • Maarten de Waard mentioned in commit 9062b848

      mentioned in commit 9062b848

      By Varac on 2019-03-01T12:29:18 (imported from GitLab project)

    • Maarten de Waard mentioned in commit 00536698

      mentioned in commit 00536698

      By Varac on 2019-03-01T13:55:03 (imported from GitLab project)

    • Maarten de Waard
      Maarten de Waard @maarten ·
      Author Contributor

      So here's the traefik log for https://code.greenhost.net/openappstack/bootstrap/-/jobs/20490, which run into a timeout of 20mins. See the ratelimit error ([auth.ci-20490.ci.openappstack.net] acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/finalize/52463812/336909117 :: urn:ietf:params:acme:error:rateLimited :: Error finalizing order :: too many certificates already issued for: openappstack.net: see https://letsencrypt.org/docs/rate-limits/, url: \n","time":"2019-03-01T16:20:58Z"}) in the last line:

      root@ci-20490:~# oas_control kubectl logs oas-test-proxy-traefik-76679b6bd5-prlxk
{"level":"info","msg":"Using TOML configuration file /config/traefik.toml","time":"2019-03-01T16:09:50Z"}
{"level":"info","msg":"No tls.defaultCertificate given for https: using the first item in tls.certificates as a fallback.","time":"2019-03-01T16:09:50Z"}
{"level":"info","msg":"Traefik version v1.7.9 built on 2019-02-11_11:36:32AM","time":"2019-03-01T16:09:50Z"}
{"level":"info","msg":"\nStats collection is disabled.\nHelp us improve Traefik by turning this feature on :)\nMore details on: https://docs.traefik.io/basics/#collected-data\n","time":"2019-03-01T16:09:51Z"}
{"level":"info","msg":"Preparing server traefik \u0026{Address::8080 TLS:\u003cnil\u003e Redirect:\u003cnil\u003e Auth:\u003cnil\u003e WhitelistSourceRange:[] WhiteList:\u003cnil\u003e Compress:false ProxyProtocol:\u003cnil\u003e ForwardedHeaders:0xc0004cdd20} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s","time":"2019-03-01T16:09:51Z"}
{"level":"info","msg":"Preparing server http \u0026{Address::80 TLS:\u003cnil\u003e Redirect:0xc00038a440 Auth:\u003cnil\u003e WhitelistSourceRange:[] WhiteList:\u003cnil\u003e Compress:true ProxyProtocol:\u003cnil\u003e ForwardedHeaders:0xc0004cdc80} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s","time":"2019-03-01T16:09:51Z"}
{"level":"info","msg":"Preparing server https \u0026{Address::443 TLS:0xc0002a70e0 Redirect:\u003cnil\u003e Auth:\u003cnil\u003e WhitelistSourceRange:[] WhiteList:\u003cnil\u003e Compress:true ProxyProtocol:\u003cnil\u003e ForwardedHeaders:0xc0004cdcc0} with readTimeout=0s writeTimeout=0s idleTimeout=3m0s","time":"2019-03-01T16:09:51Z"}
{"level":"info","msg":"Starting provider configuration.ProviderAggregator {}","time":"2019-03-01T16:09:51Z"}
{"level":"info","msg":"Starting server on :8080","time":"2019-03-01T16:09:51Z"}
{"level":"info","msg":"Starting server on :80","time":"2019-03-01T16:09:51Z"}
{"level":"info","msg":"Starting server on :443","time":"2019-03-01T16:09:51Z"}
{"level":"info","msg":"Starting provider *kubernetes.Provider {\"Watch\":true,\"Filename\":\"\",\"Constraints\":[],\"Trace\":false,\"TemplateVersion\":0,\"DebugLogGeneratedTemplate\":false,\"Endpoint\":\"\",\"Token\":\"\",\"CertAuthFilePath\":\"\",\"DisablePassHostHeaders\":false,\"EnablePassTLSCert\":false,\"Namespaces\":null,\"LabelSelector\":\"\",\"IngressClass\":\"\",\"IngressEndpoint\":null}","time":"2019-03-01T16:09:51Z"}
{"level":"info","msg":"Starting provider *acme.Provider {\"Email\":\"admin@ci-20490.ci.openappstack.net\",\"ACMELogging\":true,\"CAServer\":\"https://acme-v02.api.letsencrypt.org/directory\",\"Storage\":\"/acme/acme.json\",\"EntryPoint\":\"https\",\"KeyType\":\"\",\"OnHostRule\":true,\"OnDemand\":false,\"DNSChallenge\":null,\"HTTPChallenge\":{\"EntryPoint\":\"http\"},\"TLSChallenge\":null,\"Domains\":null,\"Store\":{}}","time":"2019-03-01T16:09:51Z"}
{"level":"info","msg":"Testing certificate renew...","time":"2019-03-01T16:09:51Z"}
{"level":"info","msg":"ingress label selector is: \"\"","time":"2019-03-01T16:09:51Z"}
{"level":"info","msg":"Creating in-cluster Provider client","time":"2019-03-01T16:09:51Z"}
{"level":"info","msg":"Server configuration reloaded on :8080","time":"2019-03-01T16:09:52Z"}
{"level":"info","msg":"Server configuration reloaded on :80","time":"2019-03-01T16:09:52Z"}
{"level":"info","msg":"Server configuration reloaded on :443","time":"2019-03-01T16:09:52Z"}
{"level":"info","msg":"Server configuration reloaded on :80","time":"2019-03-01T16:09:53Z"}
{"level":"info","msg":"Server configuration reloaded on :443","time":"2019-03-01T16:09:53Z"}
{"level":"info","msg":"Server configuration reloaded on :8080","time":"2019-03-01T16:09:53Z"}
{"level":"info","msg":"The key type is empty. Use default key type 4096.","time":"2019-03-01T16:09:53Z"}
{"level":"info","msg":"Server configuration reloaded on :80","time":"2019-03-01T16:09:56Z"}
{"level":"info","msg":"Server configuration reloaded on :443","time":"2019-03-01T16:09:56Z"}
{"level":"info","msg":"Server configuration reloaded on :8080","time":"2019-03-01T16:09:56Z"}
{"level":"info","msg":"Server configuration reloaded on :80","time":"2019-03-01T16:12:28Z"}
{"level":"info","msg":"Server configuration reloaded on :443","time":"2019-03-01T16:12:28Z"}
{"level":"info","msg":"Server configuration reloaded on :8080","time":"2019-03-01T16:12:28Z"}
{"level":"info","msg":"Register...","time":"2019-03-01T16:16:17Z"}
{"level":"info","msg":"legolog: [INFO] acme: Registering account for admin@ci-20490.ci.openappstack.net","time":"2019-03-01T16:16:17Z"}
{"level":"info","msg":"legolog: [INFO] [traefik.ci-20490.ci.openappstack.net] acme: Obtaining bundled SAN certificate","time":"2019-03-01T16:16:21Z"}
{"level":"info","msg":"legolog: [INFO] [auth.ci-20490.ci.openappstack.net] acme: Obtaining bundled SAN certificate","time":"2019-03-01T16:16:21Z"}
{"level":"info","msg":"legolog: [INFO] [traefik.ci-20490.ci.openappstack.net] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz/q3aIQCm9EVN3S-w31z0IsOzGfabThDnmH08-7_FouVo","time":"2019-03-01T16:16:30Z"}
{"level":"info","msg":"legolog: [INFO] [traefik.ci-20490.ci.openappstack.net] acme: Could not find solver for: tls-alpn-01","time":"2019-03-01T16:16:30Z"}
{"level":"info","msg":"legolog: [INFO] [traefik.ci-20490.ci.openappstack.net] acme: use http-01 solver","time":"2019-03-01T16:16:30Z"}
{"level":"info","msg":"legolog: [INFO] [traefik.ci-20490.ci.openappstack.net] acme: Trying to solve HTTP-01","time":"2019-03-01T16:16:30Z"}
{"level":"info","msg":"legolog: [INFO] [auth.ci-20490.ci.openappstack.net] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz/aM4vRVxDODQ3KZHruesKm2jgc0Aa-OvO6B3SQeZiRKQ","time":"2019-03-01T16:16:31Z"}
{"level":"info","msg":"legolog: [INFO] [auth.ci-20490.ci.openappstack.net] acme: Could not find solver for: tls-alpn-01","time":"2019-03-01T16:16:31Z"}
{"level":"info","msg":"legolog: [INFO] [auth.ci-20490.ci.openappstack.net] acme: use http-01 solver","time":"2019-03-01T16:16:31Z"}
{"level":"info","msg":"legolog: [INFO] [auth.ci-20490.ci.openappstack.net] acme: Trying to solve HTTP-01","time":"2019-03-01T16:16:31Z"}
{"level":"info","msg":"legolog: [INFO] [traefik.ci-20490.ci.openappstack.net] The server validated our request","time":"2019-03-01T16:16:37Z"}
{"level":"info","msg":"legolog: [INFO] [traefik.ci-20490.ci.openappstack.net] acme: Validations succeeded; requesting certificates","time":"2019-03-01T16:16:37Z"}
{"level":"info","msg":"legolog: [INFO] [auth.ci-20490.ci.openappstack.net] The server validated our request","time":"2019-03-01T16:16:39Z"}
{"level":"info","msg":"legolog: [INFO] [auth.ci-20490.ci.openappstack.net] acme: Validations succeeded; requesting certificates","time":"2019-03-01T16:16:39Z"}
{"level":"error","msg":"Unable to obtain ACME certificate for domains \"auth.ci-20490.ci.openappstack.net\" detected thanks to rule \"Host:auth.ci-20490.ci.openappstack.net\" : unable to generate a certificate for the domains [auth.ci-20490.ci.openappstack.net]: acme: Error -\u003e One or more domains had a problem:\n[auth.ci-20490.ci.openappstack.net] acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/finalize/52463812/336909117 :: urn:ietf:params:acme:error:rateLimited :: Error finalizing order :: too many certificates already issued for: openappstack.net: see https://letsencrypt.org/docs/rate-limits/, url: \n","time":"2019-03-01T16:20:58Z"}

      By Varac on 2019-03-01T16:46:32 (imported from GitLab project)

      Edited by Maarten de Waard
    • Maarten de Waard mentioned in issue #60 (closed)

      mentioned in issue #60 (closed)

      By Varac on 2019-03-01T16:49:45 (imported from GitLab project)

    • Maarten de Waard mentioned in commit f3fda89d

      mentioned in commit f3fda89d

      By Varac on 2019-03-04T12:29:37 (imported from GitLab project)

    • Maarten de Waard
      Maarten de Waard @maarten ·
      Author Contributor

      Another traefik eror: No valid IP addresses found for auth.ci-20672.ci.openappstack.net:

      {"level":"info","msg":"legolog: [INFO] [traefik.ci-20672.ci.openappstack.net] acme: Obtaining bundled SAN certificate","time":"2019-03-05T18:55:43Z"}
{"level":"info","msg":"legolog: [INFO] [auth.ci-20672.ci.openappstack.net] acme: Obtaining bundled SAN certificate","time":"2019-03-05T18:55:43Z"}
{"level":"info","msg":"legolog: [INFO] [auth.ci-20672.ci.openappstack.net] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/ezttX3rPZtxiMFsI0gO3FVal1GjjDUPtUx2Skef9XnU","time":"2019-03-05T18:55:52Z"}
{"level":"info","msg":"legolog: [INFO] [auth.ci-20672.ci.openappstack.net] acme: Could not find solver for: tls-alpn-01","time":"2019-03-05T18:55:52Z"}
{"level":"info","msg":"legolog: [INFO] [auth.ci-20672.ci.openappstack.net] acme: use http-01 solver","time":"2019-03-05T18:55:52Z"}
{"level":"info","msg":"legolog: [INFO] [auth.ci-20672.ci.openappstack.net] acme: Trying to solve HTTP-01","time":"2019-03-05T18:55:52Z"}
{"level":"info","msg":"legolog: [INFO] [traefik.ci-20672.ci.openappstack.net] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/3hteev73BE7StAsJP_XEtfFcz4WPD4MlM1e1-VSMuX0","time":"2019-03-05T18:55:56Z"}
{"level":"info","msg":"legolog: [INFO] [traefik.ci-20672.ci.openappstack.net] acme: Could not find solver for: tls-alpn-01","time":"2019-03-05T18:55:56Z"}
{"level":"info","msg":"legolog: [INFO] [traefik.ci-20672.ci.openappstack.net] acme: use http-01 solver","time":"2019-03-05T18:55:56Z"}
{"level":"info","msg":"legolog: [INFO] [traefik.ci-20672.ci.openappstack.net] acme: Trying to solve HTTP-01","time":"2019-03-05T18:55:56Z"}
{"level":"error","msg":"Unable to obtain ACME certificate for domains \"auth.ci-20672.ci.openappstack.net\" detected thanks to rule \"Host:auth.ci-20672.ci.openappstack.net\" : unable to generate a certificate for the domains [auth.ci-20672.ci.openappstack.net]: acme: Error -\u003e One or more domains had a problem:\n[auth.ci-20672.ci.openappstack.net] acme: error: 400 :: urn:ietf:params:acme:error:connection :: unknownHost :: No valid IP addresses found for auth.ci-20672.ci.openappstack.net, url: \n","time":"2019-03-05T18:56:02Z"}
{"level":"error","msg":"Unable to obtain ACME certificate for domains \"traefik.ci-20672.ci.openappstack.net\" detected thanks to rule \"Host:traefik.ci-20672.ci.openappstack.net\" : unable to generate a certificate for the domains [traefik.ci-20672.ci.openappstack.net]: acme: Error -\u003e One or more domains had a problem:\n[traefik.ci-20672.ci.openappstack.net] acme: error: 400 :: urn:ietf:params:acme:error:connection :: unknownHost :: No valid IP addresses found for traefik.ci-20672.ci.openappstack.net, url: \n","time":"2019-03-05T18:56:04Z"}

      By Varac on 2019-03-05T18:59:02 (imported from GitLab project)

    • Maarten de Waard
      Maarten de Waard @maarten ·
      Author Contributor

      I think there's nothing we can do about it other than increasing our timeout for waiting for the cert, since the problem is not our DNS setup, but that of letsencrypt.

      By Varac on 2019-03-12T14:50:03 (imported from GitLab project)

    • Maarten de Waard mentioned in commit 1afee3ef

      mentioned in commit 1afee3ef

      By Varac on 2019-03-12T14:57:29 (imported from GitLab project)

    • Maarten de Waard added Doing label and removed 1 deleted label

      added Doing label and removed 1 deleted label

    • Maarten de Waard assigned to @varac

      assigned to @varac

      By Varac on 2019-03-12T14:57:49 (imported from GitLab project)

    • Maarten de Waard added QA label and removed Doing label

      added QA label and removed Doing label

    • Maarten de Waard mentioned in commit 8f2ebfa3

      mentioned in commit 8f2ebfa3

      By Varac on 2019-03-12T18:00:21 (imported from GitLab project)

    • Maarten de Waard mentioned in commit 5a5d9376

      mentioned in commit 5a5d9376

      By Varac on 2019-03-12T20:27:26 (imported from GitLab project)

    • Maarten de Waard
      Maarten de Waard @maarten ·
      Author Contributor

      here is another example of a big gap between acme: Validations succeeded; requesting certificates and Server responded with a certificate. after 8 (!) mins:

      {"level":"info","msg":"legolog: [INFO] acme: Registering account for admin@ci-6690.ci.openappstack.net","time":"2019-03-12T20:08:26Z"}
{"level":"info","msg":"legolog: [INFO] [auth.ci-6690.ci.openappstack.net] acme: Obtaining bundled SAN certificate","time":"2019-03-12T20:08:27Z"}
{"level":"info","msg":"legolog: [INFO] [auth.ci-6690.ci.openappstack.net] AuthURL: https://acme-staging-v02.api.letsencrypt.org/acme/authz/Su_Hj3jJ8NE_ApE_1nVYp7E7kZEh3NvFzwfqCdG_wX0","time":"2019-03-12T20:08:31Z"}
{"level":"info","msg":"legolog: [INFO] [auth.ci-6690.ci.openappstack.net] acme: Could not find solver for: tls-alpn-01","time":"2019-03-12T20:08:31Z"}
{"level":"info","msg":"legolog: [INFO] [auth.ci-6690.ci.openappstack.net] acme: use http-01 solver","time":"2019-03-12T20:08:31Z"}
{"level":"info","msg":"legolog: [INFO] [auth.ci-6690.ci.openappstack.net] acme: Trying to solve HTTP-01","time":"2019-03-12T20:08:31Z"}
{"level":"info","msg":"legolog: [INFO] [auth.ci-6690.ci.openappstack.net] The server validated our request","time":"2019-03-12T20:08:34Z"}

{"level":"info","msg":"legolog: [INFO] [auth.ci-6690.ci.openappstack.net] acme: Validations succeeded; requesting certificates","time":"2019-03-12T20:08:34Z"}



{"level":"info","msg":"legolog: [INFO] [auth.ci-6690.ci.openappstack.net] Server responded with a certificate.","time":"2019-03-12T20:16:00Z"}

{"level":"info","msg":"Server configuration reloaded on :80","time":"2019-03-12T20:16:00Z"}
{"level":"info","msg":"Server configuration reloaded on :443","time":"2019-03-12T20:16:00Z"}
{"level":"info","msg":"Server configuration reloaded on :8080","time":"2019-03-12T20:16:00Z"}

      By Varac on 2019-03-12T20:30:18 (imported from GitLab project)

    • Maarten de Waard closed via merge request !36

      closed via merge request !36

      By Maarten de Waard on 2019-03-13T10:29:53 (imported from GitLab project)

    • Maarten de Waard mentioned in commit d885300a

      mentioned in commit d885300a

      By Maarten de Waard on 2019-03-13T10:29:53 (imported from GitLab project)

    • Maarten de Waard closed via commit 5a5d9376

      closed via commit 5a5d9376

      By Varac on 2019-03-13T10:29:53 (imported from GitLab project)

    • Maarten de Waard mentioned in issue #106 (closed)

      mentioned in issue #106 (closed)

      By Varac on 2019-03-21T11:34:58 (imported from GitLab project)

    Please register or sign in to reply