[{"data":1,"prerenderedAt":707},["ShallowReactive",2],{"/en-us/blog/this-sre-attempted-to-roll-out-an-haproxy-change/":3,"navigation-en-us":36,"banner-en-us":453,"footer-en-us":468,"Igor Wiedler":679,"next-steps-en-us":692},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"seo":8,"content":16,"config":26,"_id":29,"_type":30,"title":31,"_source":32,"_file":33,"_stem":34,"_extension":35},"/en-us/blog/this-sre-attempted-to-roll-out-an-haproxy-change","blog",false,"",{"title":9,"description":10,"ogTitle":9,"ogDescription":10,"noIndex":6,"ogImage":11,"ogUrl":12,"ogSiteName":13,"ogType":14,"canonicalUrls":12,"schema":15},"This SRE's HAProxy Config Change: An Unexpected Journey","This post is about a wild discovery made while investigating strange behavior from HAProxy. We dive into the pathology, describe how we found it, and share some investigative techniques used along the way.","https://res.cloudinary.com/about-gitlab-com/image/upload/v1749681844/Blog/Hero%20Images/infra-proxy-protocol-wireshark-header.png","https://about.gitlab.com/blog/this-sre-attempted-to-roll-out-an-haproxy-change","https://about.gitlab.com","article","\n                        {\n        \"@context\": \"https://schema.org\",\n        \"@type\": \"Article\",\n        \"headline\": \"This SRE attempted to roll out an HAProxy config change. You won't believe what happened next... \",\n        \"author\": [{\"@type\":\"Person\",\"name\":\"Igor Wiedler\"}],\n        \"datePublished\": \"2021-01-14\",\n      }",{"title":17,"description":10,"authors":18,"heroImage":11,"date":20,"body":21,"category":22,"tags":23},"This SRE attempted to roll out an HAProxy config change. You won't believe what happened next...",[19],"Igor Wiedler","2021-01-14","\n\nThis blog post was originally published on the GitLab Unfiltered blog. It was reviewed and republished on 2021-02-12.\n{: .note .alert-info .text-center}\n\n## TL;DR\n\n- HAProxy has a `server-state-file` directive that persists some of its state across restarts.\n- This state file contains the port of each backend server.\n- If an `haproxy.cfg` change modifies the port, the new port will be overwritten with the previous one from the state file.\n- A workaround is to change the backend server name, so that it is considered to be a separate server that does not match what is in the state file.\n- This has implications for the rollout procedure we use on HAProxy.\n\n## Background\n\nAll of this occurred in the context of [the gitlab-pages PROXYv2\nproject](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11902).\n\nThe rollout to staging involves changing the request flow from TCP proxying...\n```\n                   443                   443                        1443\n[ client ] -> [ google lb ] -> [ fe-pages-01-lb-gstg ] -> [ web-pages-01-sv-gstg ]\n      tcp,tls,http         tcp                        tcp            tcp,tls,http\n```\n\n... to using the [PROXY protocol](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt):\n```\n                   443                   443                        2443\n[ client ] -> [ google lb ] -> [ fe-pages-01-lb-gstg ] -> [ web-pages-01-sv-gstg ]\n      tcp,tls,http         tcp                     proxyv2,tcp       proxyv2,tcp,tls,http\n```\n\nThis is done through this change to `/etc/haproxy/haproxy.cfg` on\n`fe-pages-01-lb-gstg` (note the port change):\n```diff\n-    server web-pages-01-sv-gstg web-pages-01-sv-gstg.c.gitlab-staging-1.internal:1443 check inter 3s fastinter 1s downinter 5s fall 3 port 1080\n-    server web-pages-02-sv-gstg web-pages-02-sv-gstg.c.gitlab-staging-1.internal:1443 check inter 3s fastinter 1s downinter 5s fall 3 port 1080\n+    server web-pages-01-sv-gstg web-pages-01-sv-gstg.c.gitlab-staging-1.internal:2443 check inter 3s fastinter 1s downinter 5s fall 3 port 1080 send-proxy-v2\n+    server web-pages-02-sv-gstg web-pages-02-sv-gstg.c.gitlab-staging-1.internal:2443 check inter 3s fastinter 1s downinter 5s fall 3 port 1080 send-proxy-v2\n```\n\nSeems straightforward enough, let's go ahead and apply that change.\n\n## The brokenness\n\nAfter applying this change on one of the two `fe-pages` nodes, the requests to\nthat node start failing.\n\nBy retrying a few times via `curl` on the command line, we see this error:\n```\n➜  ~ curl -vvv https://jarv.staging.gitlab.io/pages-test/\n*   Trying 35.229.69.78...\n* TCP_NODELAY set\n* Connected to jarv.staging.gitlab.io (35.229.69.78) port 443 (#0)\n* ALPN, offering h2\n* ALPN, offering http/1.1\n* successfully set certificate verify locations:\n*   CAfile: /etc/ssl/cert.pem\n  CApath: none\n* TLSv1.2 (OUT), TLS handshake, Client hello (1):\n* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to jarv.staging.gitlab.io:443\n* Closing connection 0\ncurl: (35) LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to jarv.staging.gitlab.io:443\n```\n\nThis looks like some issue in the TLS stack, or possibly with the underlying\nconnection. It turns out that `LibreSSL` does not give us much insight into the\nunderlying issue here.\n\nSo to get a better idea, let's capture a traffic dump on the HAProxy node:\n```\nsudo tcpdump -v -w \"$(pwd)/$(hostname).$(date +%Y%m%d_%H%M%S).pcap\"\n```\n\nWhile `tcpdump` is running, we can generate some traffic, then ctrl+c and pull\nthe dump down for further analysis. That `pcap` file can be opened in Wireshark, and this allows the data to be\nexplored and filtered interactively. Here, the first really surprising thing happens:\n\n**We do not see any traffic on port 2443.**\n\nAt the same time, we _do_ see some traffic on port 1443. But we came here to look at what underlies the LibreSSL error, and what we find\nis the following (by filtering for `ip.addr == \u003Cmy external ip>`). We have a TCP SYN/ACK, establishing the connection. Followed by the client\nsending a TLS \"hello\". After which the server closes the connection with a FIN.\n\nIn other words, the server is closing the connection on the client.\n\n## The early hypotheses\n\nSo here come the usual suspects:\n\n* Did we modify the correct place in the config file?\n* Did we catch all places we need to update in the config?\n* Did the HAProxy process parse th econfig successfully?\n* Did HAProxy actually reload?\n* Is there a difference between reload and restart?\n* Did we modify the correct config file?\n* Are there old lingering HAProxy processes on the box?\n* Are we actually sending traffic to this node?\n* Are backend health checks failing?\n* Is there anything in the HAProxy logs?\n\nNone of these gave any insights whatsoever.\n\nIn an effort to reproduce the issue, I ran HAProxy on my local machine with a\nsimilar config, proxying traffic to `web-pages-01-sv-gstg`. To my surprise, this\nworked correctly. I tested with different HAProxy versions. It worked locally, but not on\n`fe-pages-01`.\n\nAt this point I'm stumped. The local config is not identical to gstg, but quite\nsimilar. What could possibly be the difference?\n\n## Digging deeper\n\nThis is when I reached out to [Matt Smiley](/company/team#/msmiley) to help with the investigation.\n\nWe started off by repeating the experiment. We saw the same results:\n\n* Server closes connection after client sends TLS hello\n* No traffic from fe-pages to web-pages on port 2443\n* Traffic from fe-pages to web-pages on port 1443\n\nThe first lead was to look at the packets going to port 1443. What do they\ncontain? We see this:\n\n![Traffic capture in wireshark showing a TCP FIN and the string QUIT in the stream](https://about.gitlab.com/images/blogimages/infra-proxy-protocol-wireshark.png){: .shadow.center}\nTraffic capture in Wireshark showing a TCP FIN and the string QUIT in the stream\n{: .note.text-center}\n\nThere is mention of `jarv.staging.gitlab.io` which does match what the client sent. And before that there is some really weird preamble:\n\n```\n\"\\r\\n\\r\\n\\0\\r\\nQUIT\\n\"\n```\n\nWhat on earth is this? Is it from the PROXY protocol? Let's search [the\nspec](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt) for the word\n\"QUIT.\" Nothing.\n\nIs this something in the HAProxy source? Searching for \"QUIT\" in the code\nreveals some hits, but none that explain this.\n\nSo this is a mystery. We leave it for now, and probe in a different direction.\n\n## Honing in\n\nHow come we are sending traffic to port 1443, when that port is not mentioned in\n`haproxy.cfg`? Where on earth is HAProxy getting that information from?\n\nI suggested running `strace` on HAProxy startup, so that we can see which files\nare being `open`ed. This is a bit tricky to do though, because the process is\nsystemd-managed.\n\nIt turns out that thanks to BPF and [BCC](https://github.com/iovisor/bcc), we\ncan actually listen on open events system-wide using the wonderful\n[opensnoop](https://github.com/iovisor/bcc/blob/master/tools/opensnoop.py). So we run `opensnoop` and restart `haproxy`, and this is what we see, highlighting the relevant bit:\n```\niwiedler@fe-pages-01-lb-gstg.c.gitlab-staging-1.internal:~$ sudo /usr/share/bcc/tools/opensnoop  -T --name haproxy\n\n...\n\n24.117171000  16702  haproxy             3   0 /etc/haproxy/haproxy.cfg\n...\n24.118099000  16702  haproxy             4   0 /etc/haproxy/errors/400.http\n...\n24.118333000  16702  haproxy             4   0 /etc/haproxy/cloudflare_ips_v4.lst\n...\n24.119109000  16702  haproxy             3   0 /etc/haproxy/state/global\n```\n\nWhat do we have here? `/etc/haproxy/state/global`, this seems oddly suspicious.\nWhat could it possibly be? Let's see what this file contains.\n```\niwiedler@fe-pages-01-lb-gstg.c.gitlab-staging-1.internal:~$ sudo cat /etc/haproxy/state/global\n\n1\n# be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state srv_uweight srv_iweight srv_time_since_last_change srv_check_status srv_check_result srv_check_health srv_check_state srv_agent_state bk_f_forced_id srv_f_forced_id srv_fqdn srv_port srvrecord\n5 pages_http 1 web-pages-01-sv-gstg 10.224.26.2 2 0 1 1 21134 15 3 4 6 0 0 0 web-pages-01-sv-gstg.c.gitlab-staging-1.internal 1080 -\n5 pages_http 2 web-pages-02-sv-gstg 10.224.26.3 2 0 1 1 20994 15 3 4 6 0 0 0 web-pages-02-sv-gstg.c.gitlab-staging-1.internal 1080 -\n6 pages_https 1 web-pages-01-sv-gstg 10.224.26.2 2 0 1 1 21134 15 3 4 6 0 0 0 web-pages-01-sv-gstg.c.gitlab-staging-1.internal 1443 -\n6 pages_https 2 web-pages-02-sv-gstg 10.224.26.3 2 0 1 1 20994 15 3 4 6 0 0 0 web-pages-02-sv-gstg.c.gitlab-staging-1.internal 1443 -\n```\n\nIt appears we are storing some metadata for each backend server, including its old port number!\n\nNow, looking again in `haproxy.cfg`, we see:\n```\nglobal\n    ...\n    server-state-file /etc/haproxy/state/global\n```\n\nSo we are using the\n[`server-state-file`](https://cbonte.github.io/haproxy-dconv/1.8/configuration.html#server-state-file)\ndirective. This will persist server state across HAProxy restarts. That is\nuseful to keep metadata consistent, such as whether a server was marked as\nMAINT.\n\n**However, it appears to be clobbering the port from `haproxy.cfg`!**\n\nThe suspected behavior is:\n\n* HAProxy is running with the old config: `web-pages-01-sv-gstg`, `1443`\n* `haproxy.cfg` is updated with the new config: `web-pages-01-sv-gstg`, `2443`, `send-proxy-v2`\n* HAProxy reload is initiated\n* HAProxy writes out the state to `/etc/haproxy/state/global` (including the old port of each backend server)\n* HAProxy starts up, reads `haproxy.cfg`, initializes itself with the new config: `web-pages-01-sv-gstg`, `2443`, `send-proxy-v2`\n* HAProxy reads the state from `/etc/haproxy/state/global`, matches on the backend server `web-pages-01-sv-gstg`, and overrides all values, including the port!\n\nThe result is that we are now attempting to send PROXYv2 traffic to the TLS port.\n\n## The workaround\n\nTo validate the theory and develop a potential workaround, we modify\n`haproxy.cfg` to use a different backend server name.\n\nThe new diff is:\n```diff\n-    server web-pages-01-sv-gstg         web-pages-01-sv-gstg.c.gitlab-staging-1.internal:1443 check inter 3s fastinter 1s downinter 5s fall 3 port 1080\n-    server web-pages-02-sv-gstg         web-pages-02-sv-gstg.c.gitlab-staging-1.internal:1443 check inter 3s fastinter 1s downinter 5s fall 3 port 1080\n+    server web-pages-01-sv-gstg-proxyv2 web-pages-01-sv-gstg.c.gitlab-staging-1.internal:2443 check inter 3s fastinter 1s downinter 5s fall 3 port 1080 send-proxy-v2\n+    server web-pages-02-sv-gstg-proxyv2 web-pages-02-sv-gstg.c.gitlab-staging-1.internal:2443 check inter 3s fastinter 1s downinter 5s fall 3 port 1080 send-proxy-v2\n```\n\nWith this config change in place, we reload HAProxy and indeed, it is now\nserving traffic correctly. See [the merge request fixing it](https://gitlab.com/gitlab-cookbooks/gitlab-haproxy/-/merge_requests/261).\n\n## A follow-up on those `QUIT` bytes\n\nNow, what is up with that `QUIT` message? Is it part of the PROXY protocol? Remember, searching [the\nspec](https://www.haproxy.org/download/1.8/doc/proxy-protocol.txt) for that\nstring did not find any matches. However, Matt actually read the spec, and found this section on version 2 of\nthe protocol:\n```\nThe binary header format starts with a constant 12 bytes block containing the\nprotocol signature :\n\n   \\x0D \\x0A \\x0D \\x0A \\x00 \\x0D \\x0A \\x51 \\x55 \\x49 \\x54 \\x0A\n```\n\nThose are indeed the bytes that make up \"\\r\\n\\r\\n\\0\\r\\nQUIT\\n\". Slightly less mnemonic than the header from text-based version 1 of the protocol:\n```\n- a string identifying the protocol : \"PROXY\" ( \\x50 \\x52 \\x4F \\x58 \\x59 )\n  Seeing this string indicates that this is version 1 of the protocol.\n```\n\nWell, I suppose that explains it.\n\nI believe our work here is done. Don't forget to like and subscribe!\n","engineering",[24,25],"production","inside GitLab",{"slug":27,"featured":6,"template":28},"this-sre-attempted-to-roll-out-an-haproxy-change","BlogPost","content:en-us:blog:this-sre-attempted-to-roll-out-an-haproxy-change.yml","yaml","This Sre Attempted To Roll Out An Haproxy Change","content","en-us/blog/this-sre-attempted-to-roll-out-an-haproxy-change.yml","en-us/blog/this-sre-attempted-to-roll-out-an-haproxy-change","yml",{"_path":37,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"data":39,"_id":449,"_type":30,"title":450,"_source":32,"_file":451,"_stem":452,"_extension":35},"/shared/en-us/main-navigation","en-us",{"logo":40,"freeTrial":45,"sales":50,"login":55,"items":60,"search":390,"minimal":421,"duo":440},{"config":41},{"href":42,"dataGaName":43,"dataGaLocation":44},"/","gitlab logo","header",{"text":46,"config":47},"Get free trial",{"href":48,"dataGaName":49,"dataGaLocation":44},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com&glm_content=default-saas-trial/","free trial",{"text":51,"config":52},"Talk to sales",{"href":53,"dataGaName":54,"dataGaLocation":44},"/sales/","sales",{"text":56,"config":57},"Sign in",{"href":58,"dataGaName":59,"dataGaLocation":44},"https://gitlab.com/users/sign_in/","sign in",[61,105,201,206,311,371],{"text":62,"config":63,"cards":65,"footer":88},"Platform",{"dataNavLevelOne":64},"platform",[66,72,80],{"title":62,"description":67,"link":68},"The most comprehensive AI-powered DevSecOps Platform",{"text":69,"config":70},"Explore our Platform",{"href":71,"dataGaName":64,"dataGaLocation":44},"/platform/",{"title":73,"description":74,"link":75},"GitLab Duo (AI)","Build software faster with AI at every stage of development",{"text":76,"config":77},"Meet GitLab Duo",{"href":78,"dataGaName":79,"dataGaLocation":44},"/gitlab-duo/","gitlab duo ai",{"title":81,"description":82,"link":83},"Why GitLab","10 reasons why Enterprises choose GitLab",{"text":84,"config":85},"Learn more",{"href":86,"dataGaName":87,"dataGaLocation":44},"/why-gitlab/","why gitlab",{"title":89,"items":90},"Get started with",[91,96,101],{"text":92,"config":93},"Platform Engineering",{"href":94,"dataGaName":95,"dataGaLocation":44},"/solutions/platform-engineering/","platform engineering",{"text":97,"config":98},"Developer Experience",{"href":99,"dataGaName":100,"dataGaLocation":44},"/developer-experience/","Developer experience",{"text":102,"config":103},"MLOps",{"href":104,"dataGaName":102,"dataGaLocation":44},"/topics/devops/the-role-of-ai-in-devops/",{"text":106,"left":107,"config":108,"link":110,"lists":114,"footer":183},"Product",true,{"dataNavLevelOne":109},"solutions",{"text":111,"config":112},"View all Solutions",{"href":113,"dataGaName":109,"dataGaLocation":44},"/solutions/",[115,140,162],{"title":116,"description":117,"link":118,"items":123},"Automation","CI/CD and automation to accelerate deployment",{"config":119},{"icon":120,"href":121,"dataGaName":122,"dataGaLocation":44},"AutomatedCodeAlt","/solutions/delivery-automation/","automated software delivery",[124,128,132,136],{"text":125,"config":126},"CI/CD",{"href":127,"dataGaLocation":44,"dataGaName":125},"/solutions/continuous-integration/",{"text":129,"config":130},"AI-Assisted Development",{"href":78,"dataGaLocation":44,"dataGaName":131},"AI assisted development",{"text":133,"config":134},"Source Code Management",{"href":135,"dataGaLocation":44,"dataGaName":133},"/solutions/source-code-management/",{"text":137,"config":138},"Automated Software Delivery",{"href":121,"dataGaLocation":44,"dataGaName":139},"Automated software delivery",{"title":141,"description":142,"link":143,"items":148},"Security","Deliver code faster without compromising security",{"config":144},{"href":145,"dataGaName":146,"dataGaLocation":44,"icon":147},"/solutions/security-compliance/","security and compliance","ShieldCheckLight",[149,152,157],{"text":150,"config":151},"Security & Compliance",{"href":145,"dataGaLocation":44,"dataGaName":150},{"text":153,"config":154},"Software Supply Chain Security",{"href":155,"dataGaLocation":44,"dataGaName":156},"/solutions/supply-chain/","Software supply chain security",{"text":158,"config":159},"Compliance & Governance",{"href":160,"dataGaLocation":44,"dataGaName":161},"/solutions/continuous-software-compliance/","Compliance and governance",{"title":163,"link":164,"items":169},"Measurement",{"config":165},{"icon":166,"href":167,"dataGaName":168,"dataGaLocation":44},"DigitalTransformation","/solutions/visibility-measurement/","visibility and measurement",[170,174,178],{"text":171,"config":172},"Visibility & Measurement",{"href":167,"dataGaLocation":44,"dataGaName":173},"Visibility and Measurement",{"text":175,"config":176},"Value Stream Management",{"href":177,"dataGaLocation":44,"dataGaName":175},"/solutions/value-stream-management/",{"text":179,"config":180},"Analytics & Insights",{"href":181,"dataGaLocation":44,"dataGaName":182},"/solutions/analytics-and-insights/","Analytics and insights",{"title":184,"items":185},"GitLab for",[186,191,196],{"text":187,"config":188},"Enterprise",{"href":189,"dataGaLocation":44,"dataGaName":190},"/enterprise/","enterprise",{"text":192,"config":193},"Small Business",{"href":194,"dataGaLocation":44,"dataGaName":195},"/small-business/","small business",{"text":197,"config":198},"Public Sector",{"href":199,"dataGaLocation":44,"dataGaName":200},"/solutions/public-sector/","public sector",{"text":202,"config":203},"Pricing",{"href":204,"dataGaName":205,"dataGaLocation":44,"dataNavLevelOne":205},"/pricing/","pricing",{"text":207,"config":208,"link":210,"lists":214,"feature":298},"Resources",{"dataNavLevelOne":209},"resources",{"text":211,"config":212},"View all resources",{"href":213,"dataGaName":209,"dataGaLocation":44},"/resources/",[215,248,270],{"title":216,"items":217},"Getting started",[218,223,228,233,238,243],{"text":219,"config":220},"Install",{"href":221,"dataGaName":222,"dataGaLocation":44},"/install/","install",{"text":224,"config":225},"Quick start guides",{"href":226,"dataGaName":227,"dataGaLocation":44},"/get-started/","quick setup checklists",{"text":229,"config":230},"Learn",{"href":231,"dataGaLocation":44,"dataGaName":232},"https://university.gitlab.com/","learn",{"text":234,"config":235},"Product documentation",{"href":236,"dataGaName":237,"dataGaLocation":44},"https://docs.gitlab.com/","product documentation",{"text":239,"config":240},"Best practice videos",{"href":241,"dataGaName":242,"dataGaLocation":44},"/getting-started-videos/","best practice videos",{"text":244,"config":245},"Integrations",{"href":246,"dataGaName":247,"dataGaLocation":44},"/integrations/","integrations",{"title":249,"items":250},"Discover",[251,256,260,265],{"text":252,"config":253},"Customer success stories",{"href":254,"dataGaName":255,"dataGaLocation":44},"/customers/","customer success stories",{"text":257,"config":258},"Blog",{"href":259,"dataGaName":5,"dataGaLocation":44},"/blog/",{"text":261,"config":262},"Remote",{"href":263,"dataGaName":264,"dataGaLocation":44},"https://handbook.gitlab.com/handbook/company/culture/all-remote/","remote",{"text":266,"config":267},"TeamOps",{"href":268,"dataGaName":269,"dataGaLocation":44},"/teamops/","teamops",{"title":271,"items":272},"Connect",[273,278,283,288,293],{"text":274,"config":275},"GitLab Services",{"href":276,"dataGaName":277,"dataGaLocation":44},"/services/","services",{"text":279,"config":280},"Community",{"href":281,"dataGaName":282,"dataGaLocation":44},"/community/","community",{"text":284,"config":285},"Forum",{"href":286,"dataGaName":287,"dataGaLocation":44},"https://forum.gitlab.com/","forum",{"text":289,"config":290},"Events",{"href":291,"dataGaName":292,"dataGaLocation":44},"/events/","events",{"text":294,"config":295},"Partners",{"href":296,"dataGaName":297,"dataGaLocation":44},"/partners/","partners",{"backgroundColor":299,"textColor":300,"text":301,"image":302,"link":306},"#2f2a6b","#fff","Insights for the future of software development",{"altText":303,"config":304},"the source promo card",{"src":305},"/images/navigation/the-source-promo-card.svg",{"text":307,"config":308},"Read the latest",{"href":309,"dataGaName":310,"dataGaLocation":44},"/the-source/","the source",{"text":312,"config":313,"lists":315},"Company",{"dataNavLevelOne":314},"company",[316],{"items":317},[318,323,329,331,336,341,346,351,356,361,366],{"text":319,"config":320},"About",{"href":321,"dataGaName":322,"dataGaLocation":44},"/company/","about",{"text":324,"config":325,"footerGa":328},"Jobs",{"href":326,"dataGaName":327,"dataGaLocation":44},"/jobs/","jobs",{"dataGaName":327},{"text":289,"config":330},{"href":291,"dataGaName":292,"dataGaLocation":44},{"text":332,"config":333},"Leadership",{"href":334,"dataGaName":335,"dataGaLocation":44},"/company/team/e-group/","leadership",{"text":337,"config":338},"Team",{"href":339,"dataGaName":340,"dataGaLocation":44},"/company/team/","team",{"text":342,"config":343},"Handbook",{"href":344,"dataGaName":345,"dataGaLocation":44},"https://handbook.gitlab.com/","handbook",{"text":347,"config":348},"Investor relations",{"href":349,"dataGaName":350,"dataGaLocation":44},"https://ir.gitlab.com/","investor relations",{"text":352,"config":353},"Trust Center",{"href":354,"dataGaName":355,"dataGaLocation":44},"/security/","trust center",{"text":357,"config":358},"AI Transparency Center",{"href":359,"dataGaName":360,"dataGaLocation":44},"/ai-transparency-center/","ai transparency center",{"text":362,"config":363},"Newsletter",{"href":364,"dataGaName":365,"dataGaLocation":44},"/company/contact/","newsletter",{"text":367,"config":368},"Press",{"href":369,"dataGaName":370,"dataGaLocation":44},"/press/","press",{"text":372,"config":373,"lists":374},"Contact us",{"dataNavLevelOne":314},[375],{"items":376},[377,380,385],{"text":51,"config":378},{"href":53,"dataGaName":379,"dataGaLocation":44},"talk to sales",{"text":381,"config":382},"Get help",{"href":383,"dataGaName":384,"dataGaLocation":44},"/support/","get help",{"text":386,"config":387},"Customer portal",{"href":388,"dataGaName":389,"dataGaLocation":44},"https://customers.gitlab.com/customers/sign_in/","customer portal",{"close":391,"login":392,"suggestions":399},"Close",{"text":393,"link":394},"To search repositories and projects, login to",{"text":395,"config":396},"gitlab.com",{"href":58,"dataGaName":397,"dataGaLocation":398},"search login","search",{"text":400,"default":401},"Suggestions",[402,404,408,410,414,418],{"text":73,"config":403},{"href":78,"dataGaName":73,"dataGaLocation":398},{"text":405,"config":406},"Code Suggestions (AI)",{"href":407,"dataGaName":405,"dataGaLocation":398},"/solutions/code-suggestions/",{"text":125,"config":409},{"href":127,"dataGaName":125,"dataGaLocation":398},{"text":411,"config":412},"GitLab on AWS",{"href":413,"dataGaName":411,"dataGaLocation":398},"/partners/technology-partners/aws/",{"text":415,"config":416},"GitLab on Google Cloud",{"href":417,"dataGaName":415,"dataGaLocation":398},"/partners/technology-partners/google-cloud-platform/",{"text":419,"config":420},"Why GitLab?",{"href":86,"dataGaName":419,"dataGaLocation":398},{"freeTrial":422,"mobileIcon":427,"desktopIcon":432,"secondaryButton":435},{"text":423,"config":424},"Start free trial",{"href":425,"dataGaName":49,"dataGaLocation":426},"https://gitlab.com/-/trials/new/","nav",{"altText":428,"config":429},"Gitlab Icon",{"src":430,"dataGaName":431,"dataGaLocation":426},"/images/brand/gitlab-logo-tanuki.svg","gitlab icon",{"altText":428,"config":433},{"src":434,"dataGaName":431,"dataGaLocation":426},"/images/brand/gitlab-logo-type.svg",{"text":436,"config":437},"Get Started",{"href":438,"dataGaName":439,"dataGaLocation":426},"https://gitlab.com/-/trial_registrations/new?glm_source=about.gitlab.com/compare/gitlab-vs-github/","get started",{"freeTrial":441,"mobileIcon":445,"desktopIcon":447},{"text":442,"config":443},"Learn more about GitLab Duo",{"href":78,"dataGaName":444,"dataGaLocation":426},"gitlab duo",{"altText":428,"config":446},{"src":430,"dataGaName":431,"dataGaLocation":426},{"altText":428,"config":448},{"src":434,"dataGaName":431,"dataGaLocation":426},"content:shared:en-us:main-navigation.yml","Main Navigation","shared/en-us/main-navigation.yml","shared/en-us/main-navigation",{"_path":454,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"title":455,"button":456,"image":460,"config":463,"_id":465,"_type":30,"_source":32,"_file":466,"_stem":467,"_extension":35},"/shared/en-us/banner","is now in public beta!",{"text":84,"config":457},{"href":458,"dataGaName":459,"dataGaLocation":44},"/gitlab-duo/agent-platform/","duo banner",{"config":461},{"src":462},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1753720689/somrf9zaunk0xlt7ne4x.svg",{"layout":464},"release","content:shared:en-us:banner.yml","shared/en-us/banner.yml","shared/en-us/banner",{"_path":469,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"data":470,"_id":675,"_type":30,"title":676,"_source":32,"_file":677,"_stem":678,"_extension":35},"/shared/en-us/main-footer",{"text":471,"source":472,"edit":478,"contribute":483,"config":488,"items":493,"minimal":667},"Git is a trademark of Software Freedom Conservancy and our use of 'GitLab' is under license",{"text":473,"config":474},"View page source",{"href":475,"dataGaName":476,"dataGaLocation":477},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/","page source","footer",{"text":479,"config":480},"Edit this page",{"href":481,"dataGaName":482,"dataGaLocation":477},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/content/","web ide",{"text":484,"config":485},"Please contribute",{"href":486,"dataGaName":487,"dataGaLocation":477},"https://gitlab.com/gitlab-com/marketing/digital-experience/about-gitlab-com/-/blob/main/CONTRIBUTING.md/","please contribute",{"twitter":489,"facebook":490,"youtube":491,"linkedin":492},"https://twitter.com/gitlab","https://www.facebook.com/gitlab","https://www.youtube.com/channel/UCnMGQ8QHMAnVIsI3xJrihhg","https://www.linkedin.com/company/gitlab-com",[494,517,574,603,637],{"title":62,"links":495,"subMenu":500},[496],{"text":497,"config":498},"DevSecOps platform",{"href":71,"dataGaName":499,"dataGaLocation":477},"devsecops platform",[501],{"title":202,"links":502},[503,507,512],{"text":504,"config":505},"View plans",{"href":204,"dataGaName":506,"dataGaLocation":477},"view plans",{"text":508,"config":509},"Why Premium?",{"href":510,"dataGaName":511,"dataGaLocation":477},"/pricing/premium/","why premium",{"text":513,"config":514},"Why Ultimate?",{"href":515,"dataGaName":516,"dataGaLocation":477},"/pricing/ultimate/","why ultimate",{"title":518,"links":519},"Solutions",[520,525,528,530,535,540,544,547,551,556,558,561,564,569],{"text":521,"config":522},"Digital transformation",{"href":523,"dataGaName":524,"dataGaLocation":477},"/topics/digital-transformation/","digital transformation",{"text":150,"config":526},{"href":145,"dataGaName":527,"dataGaLocation":477},"security & compliance",{"text":139,"config":529},{"href":121,"dataGaName":122,"dataGaLocation":477},{"text":531,"config":532},"Agile development",{"href":533,"dataGaName":534,"dataGaLocation":477},"/solutions/agile-delivery/","agile delivery",{"text":536,"config":537},"Cloud transformation",{"href":538,"dataGaName":539,"dataGaLocation":477},"/topics/cloud-native/","cloud transformation",{"text":541,"config":542},"SCM",{"href":135,"dataGaName":543,"dataGaLocation":477},"source code management",{"text":125,"config":545},{"href":127,"dataGaName":546,"dataGaLocation":477},"continuous integration & delivery",{"text":548,"config":549},"Value stream management",{"href":177,"dataGaName":550,"dataGaLocation":477},"value stream management",{"text":552,"config":553},"GitOps",{"href":554,"dataGaName":555,"dataGaLocation":477},"/solutions/gitops/","gitops",{"text":187,"config":557},{"href":189,"dataGaName":190,"dataGaLocation":477},{"text":559,"config":560},"Small business",{"href":194,"dataGaName":195,"dataGaLocation":477},{"text":562,"config":563},"Public sector",{"href":199,"dataGaName":200,"dataGaLocation":477},{"text":565,"config":566},"Education",{"href":567,"dataGaName":568,"dataGaLocation":477},"/solutions/education/","education",{"text":570,"config":571},"Financial services",{"href":572,"dataGaName":573,"dataGaLocation":477},"/solutions/finance/","financial services",{"title":207,"links":575},[576,578,580,582,585,587,589,591,593,595,597,599,601],{"text":219,"config":577},{"href":221,"dataGaName":222,"dataGaLocation":477},{"text":224,"config":579},{"href":226,"dataGaName":227,"dataGaLocation":477},{"text":229,"config":581},{"href":231,"dataGaName":232,"dataGaLocation":477},{"text":234,"config":583},{"href":236,"dataGaName":584,"dataGaLocation":477},"docs",{"text":257,"config":586},{"href":259,"dataGaName":5,"dataGaLocation":477},{"text":252,"config":588},{"href":254,"dataGaName":255,"dataGaLocation":477},{"text":261,"config":590},{"href":263,"dataGaName":264,"dataGaLocation":477},{"text":274,"config":592},{"href":276,"dataGaName":277,"dataGaLocation":477},{"text":266,"config":594},{"href":268,"dataGaName":269,"dataGaLocation":477},{"text":279,"config":596},{"href":281,"dataGaName":282,"dataGaLocation":477},{"text":284,"config":598},{"href":286,"dataGaName":287,"dataGaLocation":477},{"text":289,"config":600},{"href":291,"dataGaName":292,"dataGaLocation":477},{"text":294,"config":602},{"href":296,"dataGaName":297,"dataGaLocation":477},{"title":312,"links":604},[605,607,609,611,613,615,617,621,626,628,630,632],{"text":319,"config":606},{"href":321,"dataGaName":314,"dataGaLocation":477},{"text":324,"config":608},{"href":326,"dataGaName":327,"dataGaLocation":477},{"text":332,"config":610},{"href":334,"dataGaName":335,"dataGaLocation":477},{"text":337,"config":612},{"href":339,"dataGaName":340,"dataGaLocation":477},{"text":342,"config":614},{"href":344,"dataGaName":345,"dataGaLocation":477},{"text":347,"config":616},{"href":349,"dataGaName":350,"dataGaLocation":477},{"text":618,"config":619},"Sustainability",{"href":620,"dataGaName":618,"dataGaLocation":477},"/sustainability/",{"text":622,"config":623},"Diversity, inclusion and belonging (DIB)",{"href":624,"dataGaName":625,"dataGaLocation":477},"/diversity-inclusion-belonging/","Diversity, inclusion and belonging",{"text":352,"config":627},{"href":354,"dataGaName":355,"dataGaLocation":477},{"text":362,"config":629},{"href":364,"dataGaName":365,"dataGaLocation":477},{"text":367,"config":631},{"href":369,"dataGaName":370,"dataGaLocation":477},{"text":633,"config":634},"Modern Slavery Transparency Statement",{"href":635,"dataGaName":636,"dataGaLocation":477},"https://handbook.gitlab.com/handbook/legal/modern-slavery-act-transparency-statement/","modern slavery transparency statement",{"title":638,"links":639},"Contact Us",[640,643,645,647,652,657,662],{"text":641,"config":642},"Contact an expert",{"href":53,"dataGaName":54,"dataGaLocation":477},{"text":381,"config":644},{"href":383,"dataGaName":384,"dataGaLocation":477},{"text":386,"config":646},{"href":388,"dataGaName":389,"dataGaLocation":477},{"text":648,"config":649},"Status",{"href":650,"dataGaName":651,"dataGaLocation":477},"https://status.gitlab.com/","status",{"text":653,"config":654},"Terms of use",{"href":655,"dataGaName":656,"dataGaLocation":477},"/terms/","terms of use",{"text":658,"config":659},"Privacy statement",{"href":660,"dataGaName":661,"dataGaLocation":477},"/privacy/","privacy statement",{"text":663,"config":664},"Cookie preferences",{"dataGaName":665,"dataGaLocation":477,"id":666,"isOneTrustButton":107},"cookie preferences","ot-sdk-btn",{"items":668},[669,671,673],{"text":653,"config":670},{"href":655,"dataGaName":656,"dataGaLocation":477},{"text":658,"config":672},{"href":660,"dataGaName":661,"dataGaLocation":477},{"text":663,"config":674},{"dataGaName":665,"dataGaLocation":477,"id":666,"isOneTrustButton":107},"content:shared:en-us:main-footer.yml","Main Footer","shared/en-us/main-footer.yml","shared/en-us/main-footer",[680],{"_path":681,"_dir":682,"_draft":6,"_partial":6,"_locale":7,"content":683,"config":687,"_id":689,"_type":30,"title":19,"_source":32,"_file":690,"_stem":691,"_extension":35},"/en-us/blog/authors/igor-wiedler","authors",{"name":19,"config":684},{"headshot":685,"ctfId":686},"https://res.cloudinary.com/about-gitlab-com/image/upload/v1749681841/Blog/Author%20Headshots/igorwwwwwwwwwwwwwwwwwwww-headshot.png","igorwwwwwwwwwwwwwwwwwwww",{"template":688},"BlogAuthor","content:en-us:blog:authors:igor-wiedler.yml","en-us/blog/authors/igor-wiedler.yml","en-us/blog/authors/igor-wiedler",{"_path":693,"_dir":38,"_draft":6,"_partial":6,"_locale":7,"header":694,"eyebrow":695,"blurb":696,"button":697,"secondaryButton":701,"_id":703,"_type":30,"title":704,"_source":32,"_file":705,"_stem":706,"_extension":35},"/shared/en-us/next-steps","Start shipping better software faster","50%+ of the Fortune 100 trust GitLab","See what your team can do with the intelligent\n\n\nDevSecOps platform.\n",{"text":46,"config":698},{"href":699,"dataGaName":49,"dataGaLocation":700},"https://gitlab.com/-/trial_registrations/new?glm_content=default-saas-trial&glm_source=about.gitlab.com/","feature",{"text":51,"config":702},{"href":53,"dataGaName":54,"dataGaLocation":700},"content:shared:en-us:next-steps.yml","Next Steps","shared/en-us/next-steps.yml","shared/en-us/next-steps",1753981633895]