We recently implemented Google’s Account Password Synchronization (GAPS) 1.1 into our environment. If you are not familiar with the product, it synchronizes passwords changed within Active Directory and a particular domain controller, over to the Google Apps directory on the cloud. This allows our users to use their AD passwords to access their Google Mail with mobile devices leveraging one single password.
As per Google’s recommendation and online guide found under the Google support page for GAPS, we had followed all their recommendations. However, a month after we implemented GAPS we expanded our domain controllers from 13 over to 25 DC’s and started to experience issues left and right.
Both of our internal Google and AD admins looked over everything including logs and configuration settings over and over. The initial issue we had thought was firewall/proxy related still re-appeared even after excluding a DC from various restrictions whatsoever. Looking over the recommended logs we kept seeing items such as:
2014-07-24T07:28:54.694-04:00 684 A:PasswordSync password_sync_service!PasswordSyncTask::RetriveUser @ 210 (User@domain.org)> retrieved user……
2014-07-24T07:28:54.694-04:00 684 A:PasswordSync password_sync_service!PasswordSyncTask::RetriveUser @ 212 (User@domain.org)> Get user failed, retrying..
2014-07-24T07:28:54.694-04:00 684 E:Parser password_sync_service!DateTime::SetUtcNow @ 898 (User@domain.org)> Failed with 0x80070057, last successful line = 894.
However, after looking over our logs many times we started to notice a pattern in which one DC would fail and would need to get re-configured shortly another DC would fail the following error:
2014-07-24T07:28:54.725-04:00 684 E:Network password_sync_service!WinHttp::ExecuteHttpRequestIStreamResponse @ 775 (User@domain.org)> HttpRequest output.
HTTP/1.1 407 Proxy Authentication Required
Content-Type: text/html; charset=utf-8
Not having any luck figuring out the issue, we contacted Google support and received a fundamental piece of information. The Google service account that connects to the Google API seems to max out every 10 domain controllers. With 25 DCs we would need to have 3 separate Google service accounts configured and delegated the appropriate permissions.
We were stumped, for some reason Googles OAP api seems to crap out after so many concurrent connections, making this version of GAPS a pain to monitor and manage. Worst of all, there is no documentation we could find at the time that talks about this artificial limit.