We recently noticed our SRM was only connecting to 1 of 2 Isilons. One was green, one grey. Both clusters are configured identically, same code @ 8.0.0.6. We could reconfigure the disconnected Isi using SNMPv2 and admin creds and SRM would test the connection, all green. Go to discover and it would break saying "no array found at destination". We changed access accounts to one specifically given privs that are laid out in Thwack (Unable to add EMC Isilon array - SolarWinds Worldwide, LLC. Help and Support ). This time the test would work and discovery would just say "Unauthorized".
We turned up Solarwinds logging to debug levels and captured the failures and verified they were getting HTTP - 401 Unauthorized. We ran 'tcpdump' on Isi and confirmed that an HTTPS handshake was being created. So no firewalls or networking issues getting in our way.
We also have Commvault as our backup and it shows us something very similar when we try to enable NDMP style Intellisnap backups of the Isilons. The green cluster in SW responds to CV API calls and I can crawl the file system to setup backups. The grey cluster says API auth failed, not authorized. We called EMC and they simply logged into each cluster and used curl commands using Basic Auth to show they were both responding to API.
I'm not much of a scripter but I installed Postman and was able to recreate EMC supports success using Basic Auth. But Isilon also supports Session Authentication, which is what most machines use. They verify credentials and are given a cookie that they then send with all subsequent commands while the session is active. On the Isilon that's about 900-1400 seconds.
More Postman, this time using the https://www.emc.com/collateral/TechnicalDocument/docu66301.pdf EMC API reference to create a session, page 17. Using the 'Set-Cookie: isisessid" value sent back in another GET request I received a HTTP 401 Unauthorized error. VERY similar to the issue SRM was having. Test just verified connection and got the cookie, discovery gets 401'd. So I tested this procedure on our green cluster to make sure I knew what was going on. Boom! Got in using the isisessid value sent back to me. It's then that I noticed the green cluster only sent me 1 cookie. The grey cluster sent me 2. The 2nd Cookie was 'Set-Cookie: isicsrf"
After looking up CSRF in EMC support I found that they have changed how their API does authentication. This article explains this new CSRF handling and how Isilon now requires a special header in API requests when this feature is enabled. https://emcservice.force.com/CustomersPartners/kA5f10000004Jn3CAE The relevent example is towards the bottom:
To make authenticated requests to OneFS platform API the client sends
their OneFS session cookie (isisessid) as always. Additionally, to pass
CSRF request validation checks, the client must send the CSRF token cookie
(isicsrf) obtained at initial authentication in a special header
(X-CSRF-Token) as well as a populated Referer header matching the
connecting host.
* Example curl request
$ curl -vk 10.0.0.1:8080/.../id \
-b 'isisessid=924bb64a-cffd-4d98-9ccc-6703fabc3210' \
-H 'X-CSRF-Token: 8c5da1e4-5508-4609-9978-4a6d283e4c3a' \
--referer https://10.0.0.1:8080
The bottom 2 lines are now required, and not being sent by SRM, or Commvault most likely. I added this fields to my Postman request and it worked. Verified on EMC boards here EMC Community Network - DECN: API Auth Changes with Recent Security Patches (anti-CSRF token)
I HAVE NO IDEA HOW THIS GOT TURNED ON IN ONLY 1 CLUSTER
HERE'S THE KICKER. I tried to comment out the flags in the .conf file explained in the emcservice article, but nothing changed. I did another webex with EMC and laid all this out ahead of time and they came back and said that once anti-CSRF is turned on it can't be turned off due to a bug that isn't fixed until 8.0.0.7 or 8.1.0.5 which are both unreleased! SRM needs to be fixed to use this new anti CSRF authentication method.
If you want to verify if your Isilon has this anti-CSRF enabled open a bash somewhere and use the script given here: https://emcservice.force.com/CustomersPartners/kA2f1000000K79iCAC If the output is True, then anti-CSRF is on and refusing SRM API calls!!
echo "Please enter cluster IP/FQDN" ; read cluster && echo "Please enter root password" ; read -s pw && OUTPUT=$(curl --fail -v -k -d "{\"username\": \"root\", \"services\": [\"platform\"], \"password\": \"${pw}\"}" -H "Content-Type: application/json" -XPOST https://${cluster}:8080/session/1/session 2>&1); CALL_OK=$(echo ${OUTPUT} | tail -c 72 | grep 'timeout_absolute' > /dev/null && echo 'ok'); CSRF=$(echo ${OUTPUT} | grep 'isicsrf' > /dev/null && echo 'True' || echo 'False'); if [ ${CALL_OK} = 'ok' ]; then echo ${CSRF} ; else echo ${OUTPUT} ; fi
TL;DR Isilon uses new anti-CSRF authentication for API calls and Solarwinds needs to update how they communicate with Isilons on newer code.
