422 The change you wanted was rejected

[hermann-san]

Hi all,
I've setup Consul on a subdomain.domain.com with a reverse proxy using a private IP range internally. Consul is using http because I use https on the reverse proxy already.
Things are working fine when I access the server internally with the private IP address. I can register as a new user and login to the server.
However, when I connect to the server externally via subdomain.domain.com then I have an issue when I try to log-in, register a new user or click "forgot password")
The error message comes in all 3 above mentioned cases:
422 The change you wanted was rejected
I can browse my consul website without log-in, but as soon as I try to log-in or register , there is this error message.

From an internet search I've found that the 422 error may have to do with CSRF Token error and may have to do with my reverse proxy setup
https://stackoverflow.com/questions/41027048/rails-devise-sign-in-doesnt-work-on-safari-422-csrf-error

it was said to change the following line in
deploy/consul/current/config/initializers/session_store.rb
line
Rails.application.config.session_store :cookie_store, key: "_consul_session"

I don't know if that will help. I've removed this part "_consul_session"` , but it didn't help.

Do you have any idea what I could try to make it work?
Thanks.

[Senen]

Hi,

We never encountered this error because, in our experience, each institution's networking or system administration usually configures the proxies. We know of some Consul Democracy installations using reverse proxies without issues.

Can you paste the application log of the 422 error? You can find the logs in /home/deploy/shared/logs

Can you pass us more instructions so we can reproduce the error you experience (proxies software and its version, the proxy configuration you are using, and any other detail we can use to reproduce the same behaviour)? If so, we can take a look and try to figure out a solution.

[Senen]

cc @hermann-san

[hermann-san]

Hi @Senen ,

thanks for looking into this.
Proxy software is HAProxy 4.1_1 on OpnSense Firewall 23.7.6 which run's on Proxmox.
Below is the info from the production.log and the proxy config. I hope I didn't miss something cause I took it from a larger config.

https://pastebin.grey.pw/?cf957178dc0906f7#62MGoME5TEix4MCW855xbXLj74Wu7y5m8Bfh7RSfgFK

W, [2023-11-26T22:39:56.984794 #568]  WARN -- : [655806da-a8fc-4835-9570-a2dfdfc491ba] HTTP Origin header (https://sub-domain.domain.org/) didn't match request.base_url (http://sub-domain.domain.org/)
F, [2023-11-26T22:39:56.986825 #568] FATAL -- : [655806da-a8fc-4835-9570-a2dfdfc491ba]

I could not find anything blocking on the firewall. Other apps do work well with the reverse proxy config, but there are also 2 other apps which also appear to have issues with the reverse proxy config. But those apps have specific instructions for reverse proxy config. E.g. Taiga. I'm not done with troubleshooting those issues yet. Consul doesn't seem to have any documentation on reverse proxy scenario.
I'm using opnsense acme client to generate the certificate for Consul subdomain and the status is OK.

If you need something else , please let me know.

[Senen]

Hi @hermann-san,

Thanks for sharing more information.

I can confirm that we had the same error with a customer using reverse proxies, and finally, the customer's networking department solved it by changing their proxy configuration. It did not require any change from the Consul Democracy side to fix it.

We cannot access the proxy configuration files, but we can ask them if they can share some information about how they solved this problem so you can try it or at least search for a similar solution for the software you are using. It could take a long time to get an answer. We'll let you know once we know more.

Consul doesn't seem to have any documentation on reverse proxy scenario.

We recently started working with institutions that are already using Consul to build documentation or guidance for typical proxy scenarios. Hopefully, at some point, we will add some information about this to the documentation site. Thanks for pointing this out!

[hermann-san]

Hi @Senen ,
I've found out that the problem doesn't have to do with the external HAProxy, but with the nginx web server on the consul server itself.
I was able to get it working with the following modifications of the NGINX reverse proxy config.

The steps on Ubunutu Linux are:

Open the config file
sudo nano /etc/nginx/sites-enabled/default

The lower part needs to looks like this

  try_files $uri/index.html $uri @app;
  location @app {
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-Proto https;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $http_host;
    proxy_redirect off;
    proxy_pass http://app;/
  }
}

Restart nginx

systemctl restart nginx

That was already it. There is one line I've added and I've changed another line (to https).

    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-Proto https;