Flake in syncstorage test_storage test_users_with_the_same_batch_id_get_separate_data

[data-sync-user]

There’s occasional flake in this test on CI:

mysql-e2e-tests    | =================================== FAILURES ===================================
mysql-e2e-tests    | _______ TestStorage.test_users_with_the_same_batch_id_get_separate_data ________
mysql-e2e-tests    | 
mysql-e2e-tests    | self = <integration_tests.test_storage.TestStorage testMethod=test_users_with_the_same_batch_id_get_separate_data>
mysql-e2e-tests    | 
mysql-e2e-tests    |     def test_users_with_the_same_batch_id_get_separate_data(self):
mysql-e2e-tests    |         # Try to generate two users with the same batch-id.
mysql-e2e-tests    |         # It might take a couple of attempts...
mysql-e2e-tests    |         for _ in range(100):
mysql-e2e-tests    |             bsos = [{"id": "a", "payload": "aih"}]
mysql-e2e-tests    |             req = "/storage/xxx_col1?batch=true"
mysql-e2e-tests    |             resp = self.retry_post_json(self.root + req, bsos)
mysql-e2e-tests    |             batch1 = resp.json["batch"]
mysql-e2e-tests    |             with self._switch_user():
mysql-e2e-tests    |                 bsos = [{"id": "b", "payload": "bee"}]
mysql-e2e-tests    |                 req = "/storage/xxx_col1?batch=true"
mysql-e2e-tests    |                 resp = self.retry_post_json(self.root + req, bsos)
mysql-e2e-tests    |                 batch2 = resp.json["batch"]
mysql-e2e-tests    |                 # Let the second user commit their batch.
mysql-e2e-tests    |                 req = "/storage/xxx_col1?batch={0}&commit=true".format(batch2)
mysql-e2e-tests    |                 self.retry_post_json(self.root + req, [])
mysql-e2e-tests    |                 # It should only have a single item.
mysql-e2e-tests    |                 resp = self.app.get(self.root + "/storage/xxx_col1")
mysql-e2e-tests    | >               self.assertEqual(resp.json, ["b"])
mysql-e2e-tests    | E               AssertionError: Lists differ: ['a', 'b', 'c', 'd'] != ['b']
mysql-e2e-tests    | E               
mysql-e2e-tests    | E               First differing element 0:
mysql-e2e-tests    | E               'a'
mysql-e2e-tests    | E               'b'
mysql-e2e-tests    | E               
mysql-e2e-tests    | E               First list contains 3 additional elements.
mysql-e2e-tests    | E               First extra element 1:
mysql-e2e-tests    | E               'b'
mysql-e2e-tests    | E               
mysql-e2e-tests    | E               - ['a', 'b', 'c', 'd']
mysql-e2e-tests    | E               + ['b']
mysql-e2e-tests    | 
mysql-e2e-tests    | tools/integration_tests/test_storage.py:2098: AssertionError

The intent of this test is to generate conflicting batch ids with different users, which is still a possibility on the mysql backend as it uses the current “sync” timestamp, adding userid % 10. Despite conflicting batch ids, the primary key includes the userid so the data won’t clash.

However the test seems to be encountering data produced from other tests (there’s a few other tests that write bsos with ids of a/b/c/d to a batch).

It’s not too surprising that the userids generated by this test via _switch_user could be the same as userids generated in previous tests as the userid is only a 5 digit number, see:

https://github.com/mozilla-services/syncstorage-rs/blob/4d6fe26a3553c4ea11fe48b6bcf57b6611d81f7f/tools/integration_tests/test_support.py#L422

https://github.com/mozilla-services/syncstorage-rs/blob/4d6fe26a3553c4ea11fe48b6bcf57b6611d81f7f/tools/integration_tests/test_support.py#L394

The TestCase setUp is responsible for deleting all of the existing user before running the test, so I believe the flake is due to:

• a previous test generates userid A (every test generates a new userid), clearing out the data on setUp
• that previous test populates this xxx_col1 collection with a/b/c/d bsos, which are never deleted (TestCase tearDown doesn’t do it)
• this test begins with userid B, but ends up yielding a _switch_user back to userid A
• this test encounters the previous test’s data

To fix: we should have either _switch_user (or this specific test should do it on a successful _switch_user call) or tearDown delete the user data.

┆Issue is synchronized with this Jira Task