Troubleshooting Guide
This guide documents common configuration and connectivity issues. For any problems that arise, a good first bet is to check the Firezone logs.
Each container stores logs as a JSON file on the host machine. These can be
shown with the docker logs {CONTAINER}
command. Log files are found at
var/lib/docker/containers/{CONTAINER_ID}/{CONTAINER_ID}-json.log
by default.
See debug logs for additional details.
Firezone logs are stored in /var/log/firezone
and can be viewed with
sudo firezone-ctl tail
.
Application crash loop preventing config changes
In cases where the application is crash looping because of corrupt, inaccessible, or invalid OIDC or SAML configuration in the DB, you can try clearing the affected fields.
For example, to clear OIDC configs:
psql -d firezone -h 127.0.0.1 -U postgres -c "UPDATE configurations SET openid_connect_providers = '[]'"
Similarly, to clear SAML configs:
psql -d firezone -h 127.0.0.1 -U postgres -c "UPDATE configurations SET saml_identity_providers = '[]'"
/opt/firezone/embedded/bin/psql -d $DATABASE_NAME -h $DATABASE_HOST -U $DATABASE_USER -c "UPDATE configurations SET openid_connect_providers = '[]'"
Similarly, to clear SAML configs:
/opt/firezone/embedded/bin/psql -d $DATABASE_NAME -h $DATABASE_HOST -U $DATABASE_USER -c "UPDATE configurations SET saml_identity_providers = '[]'"
Debugging WebSocket connectivity issues
The Web UI requires a secure websocket connection to function. If a secure websocket connection can't be established, you'll see a red dot indicator in the upper-right portion of the Firezone web UI and a corresponding message when you hover over it:
Secure websocket not connected! ...
If you're accessing Firezone using the same URL defined in your EXTERNAL_URL
variable from above, the issue is likely to be in your reverse proxy
configuration. Ensure your reverse proxy has WebSocket support enabled for
Firezone. If you're using the default Caddy reverse proxy, WebSocket is enabled
and configured automatically.
In most cases, you'll find clues in one or more of the following locations:
firezone
service logs:docker compose logs firezone
caddy
service logs:docker compose logs caddy
- Your browser's developer tool logs, specifically the
Network
tab. sudo firezone-ctl tail nginx
sudo firezone-ctl tail phoenix
If the websocket connection is successful, you should see output in the
phoenix
service logs similar the following:
2022-09-23_15:05:47.29158 15:05:47.291 [info] CONNECTED TO Phoenix.LiveView.Socket in 24µs
2022-09-23_15:05:47.29160 Transport: :websocket
2022-09-23_15:05:47.29160 Serializer: Phoenix.Socket.V2.JSONSerializer
2022-09-23_15:05:47.29161 Parameters: %{"_csrf_token" => "XFEFCHQ2fRQABQwtKQdCJDlFAzEcCCJvn7LqDsMXE4eGZtsBzuwVRCJ6", "_mounts" => "0", "_track_static" => %{"0" => "https://demo.firez.one/dist/admin-02fabe0f543c64122dbf5bc5b3451e51.css?vsn=d", "1" => "https://demo.firez.one/dist/admin-04e75c78295062c2c07af61be50248b0.js?vsn=d"}, "vsn" => "2.0.0"}
2022-09-23_15:05:47.33655 15:05:47.336 [info] CONNECTED TO FzHttpWeb.UserSocket in 430µs
2022-09-23_15:05:47.33657 Transport: :websocket
2022-09-23_15:05:47.33658 Serializer: Phoenix.Socket.V2.JSONSerializer
2022-09-23_15:05:47.33658 Parameters: %{"token" => "SFMyNTY.g2gDYQFuBgB6HeJqgwFiAAFRgA.qKoC2bi9DubPkE0tfaRSPERWUFyQQPQV5n4nFKVppxc", "vsn" => "2.0.0"}
2022-09-23_15:05:47.35063 15:05:47.350 [info] JOINED notification:session in 636µs
2022-09-23_15:05:47.35065 Parameters: %{"token" => "SFMyNTY.g2gDYQFuBgB6HeJqgwFiAAFRgA.zSG7pefm-Vgf_zvduxa5E9VK4s8PKqzc0xBDGNx5FQE", "user_agent" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:106.0) Gecko/20100101 Firefox/106.0"}
Debugging WireGuard connectivity issues
Most connectivity issues with Firezone are caused by other iptables
or
nftables
rules which interfere with Firezone's operation. If you have rules
active, you'll need to ensure these don't conflict with the Firezone rules.
Internet connectivity drops when tunnel is active
If your Internet connectivity drops whenever you activate your WireGuard tunnel,
you should make sure that the FORWARD
chain allows packets from your WireGuard
clients to the destinations you want to allow through Firezone.
If you're using ufw
, this can be done by making sure the default routing
policy is allow
:
ubuntu@fz:~$ sudo ufw default allow routed
Default routed policy changed to 'allow'
(be sure to update your rules accordingly)
A ufw
status for a typical Firezone server might look like this:
ubuntu@fz:~$ sudo ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), allow (routed)
New profiles: skip
To Action From
-- ------ ----
22/tcp ALLOW IN Anywhere
80/tcp ALLOW IN Anywhere
443/tcp ALLOW IN Anywhere
51820/udp ALLOW IN Anywhere
22/tcp (v6) ALLOW IN Anywhere (v6)
80/tcp (v6) ALLOW IN Anywhere (v6)
443/tcp (v6) ALLOW IN Anywhere (v6)
51820/udp (v6) ALLOW IN Anywhere (v6)
Admin login isn't working
If the password for the account with email DEFAULT_ADMIN_EMAIL
isn't working,
you can reset it using the process below.
First change directory to your Firezone installation directory
($HOME/.firezone
by default), then run the bin/create-or-reset-admin
script
to reset the admin user's password. The password for the user specified by
DEFAULT_ADMIN_EMAIL
in $HOME/.firezone/.env
will be reset to the
DEFAULT_ADMIN_PASSWORD
variable.
cd $HOME/.firezone
docker compose exec firezone bin/create-or-reset-admin
Note: If local authentication is disabled, resetting the admin user's password will not re-enable it.
Run the following command to reset the password on the default admin user.
sudo firezone-ctl create-or-reset-admin
Re-enable local authentication via CLI
If you've configured an OIDC or SAML provider, you can consider disabling local authentication for additional security.
If, however, issues arise with your identity provider integration, it's possible you could be locked out of the admin portal. To re-enable local authentication so you can log in and resolve the issue, you can temporarily re-enable local authentication via the REST API.
If that's not an option, you can re-enable local authentication by running the following commands on the host of your Firezone instance:
cd $HOME/.firezone
docker compose exec postgres psql -U postgres -h 127.0.0.1 -d firezone -c "UPDATE configurations SET local_auth_enabled = 't'"
Need additional help?
Try asking on one of our community-powered support channels:
- Discussion forums: Ask questions, report bugs, and suggest features.
- Public Slack group: join discussions, meet other users, and meet the contributors