Fixing the Firefox Sync Server

April 12, 2020 on captainepoch's log

UPDATE 2020-05-09: added at the end a perfect deploy guide and a bit of config tunning for the deploy. I have the guide archived just in case it is lost. The small-sized guide can be found at the end of this post.

I changed from Chrome to Firefox a long time ago, and I mostly do not regret it. I prefer to have all my personal data under my control, and that is the reason I host my own Firefox Sync Server.

Chrome, of course, has your data within Google’s servers (the botnet). Firefox, too. But, at least, Mozilla has a 2018-something version of the synchronization server open in GitHub (Firefox Sync Server).

I upgraded the software a month or so ago. I though it was easy, just following the steps in the README file and it is done. And, of course, it works. The software is upgraded after doing those steps. The problems are the changes made and not described in any part but a lost issue in the repository issue tracker.

This week I noticed that my changes were not synchronizing between my two computers. I though it was a problem of the Nginx configuration, so I started digging into it. It was not. Logs did not show anything special but one thing (paste):

1586625940767	Services.Common.RESTRequest	DEBUG	GET https://DOMAIN/token/1.0/sync/1.5 500
1586625940767	Services.Common.TokenServerClient	DEBUG	Got token response: 500
1586625940767	Services.Common.TokenServerClient	WARN	Did not receive JSON response. Misconfigured server?
1586625940767	Services.Common.TokenServerClient	DEBUG	Content-Type: text/html; charset=utf-8
1586625940767	Services.Common.TokenServerClient	DEBUG	Body: <html>
  <head>
    <title>Internal Server Error</title>
  </head>
  <body>
    <h1><p>Internal Server Error</p></h1>

  </body>
</html>

1586625940767	Sync.BrowserIDManager	ERROR	Non-authentication error in _fetchTokenForUser: TokenServerClientServerError(...)(resource://services-common/tokenserverclient.js:39:36) JS Stack trace: TokenServerClientServerError@tokenserverclient.js:105:16
_processTokenResponse@tokenserverclient.js:279:19

What the fuck?!

I started diggin under the skin during 2 days until, for some reason, I left the watch command open with the status of the systemd service and found this (paste):

ProgrammingError: (psycopg2.errors.UndefinedColumn) column "keys_changed_at" does not exist
LINE 3:     keys_changed_at, node
            ^

[SQL: select
    uid, generation, client_state, created_at, replaced_at,
    keys_changed_at, node
from
    users
where
    email = %(email)s
and
    service = %(service)s
order by
    created_at desc, uid desc
limit
    20
]
[parameters: {'email': u'REDACTED@api.accounts.firefox.com', 'service': u'sync-1.5'}]

Now, you might have guessed I use PostgreSQL to store the data. From my old setup to the new one, even doing the steps of the docs, the database was not upgraded at all. For this, I spent 2 days looking into the wrong place. The fix is in an issue (you can go to the issue), but if you do not want to get into it, here is the fix:

ALTER TABLE users ADD keys_changed_at BIGINT;
ALTER TABLE users ADD node VARCHAR(255);

Where do you have to apply it? Of course, in the database you put into the configuration. And, with such simple thing, it works. It could be pushed as a migration to be done in the make build step or something, but nope.

I faced another problem related to the Nginx configuration I have in place, but that is not relevant for this post.

And that is a writeup made by spending two days fixing something the upgrade process should consider in the documentation. I understand that hosting yourself the software has its problems, but damn…

There is a lot of things to improve with the Firefox Sync Server, for example, not using Python 2.7 or a better documentation, because it is not clear at all.

–

If you want to deploy a Firefox Sync Server in your own server, or VPS, or whatever server you own, a guide to deploy the Firefox Sync Server can be found at Firefox Bookmark Syncing (Wayback Machine backup), writen by Ansgar Kellner. Credits to him, it works like a charm.

The guide above this paragraph does not take into account using PostreSQL as the backend database, so you must do this in order to install witk before executing the make build command:

$ echo "psycopg2" >> requirements.txt

If it fails, try with psycopg2-binary instead.

Then, I recommend to you to setup the server for using gevent workers. To install them before installing the sync server:

$ echo "gevent" >> requirements.txt

If you have deployed your sync server already, you can install it and setup in the config file:

$ pip install gevent

In the file syncserver.ini:

[server:main]
...
worker_class = gevent

The three dots represents the rest of the configuration, put the last line at the end of that category.

And, with all of this, you are going to have a smooth and fast Firefox Sync Server at your disposal.