Gateway supports multiple authentication methods to enable Single Sign-On: Basic Authentication, SAML 2.0, X.509 Certificates, SAP Logon Tickets, OAuth. Correct operation of SSO between a Gateway services consumer (e.g. SAP Fiori, SharePoint App, …) and Gateway requires that the consumer and the Gateway system have established an identity trust relationship. This typically (except for basic authentication, but I do not consider that as a viable enterprise-ready SSO option) requires configuration on consumer and Gateway side.
What to do in case the service consumer does not succeed in successfully sign-in on Gateway? How to find out what is the root cause, when you have configuration settings on both the consumer and on the Gateway side? Well, it appears that the NetWeaver stack provides a convenient diagnose tool for this:
Make sure to activate this service in SICF, open the service in a browser, start a recording session, and repeat from consumer side the attempt to single sign-on. Next stop the recording, and inspect the trace file. In case of security related exception, you're likely to find useful information logged in that trace file.