18 Replies Latest reply on May 21, 2014 4:20 AM by olha.zelena

    Alert Central fails to send alert emails, "Error attempting to notify..."

    charliemsmith

      Alert Central will not consistently push alert emails to assigned mail-boxes. After
      several successful alert distributions, AC will encounter “Error attempting to
      notify someone@somewhere”.
      If I delete the user and re-create the user AC will
      successfully deliver future alerts. Although, after recreating the user in AC, another
      user will error with same error.  Please
      help.

        • Re: Alert Central fails to send alert emails, "Error attempting to notify..."
          nazar_hryt

          Hello, could you reproduce it one more and attach logs here(Instructions for collecting Alert Central Logs).

           

          Thanks,

          Nazar

            • Re: Alert Central fails to send alert emails, "Error attempting to notify..."
              jeff.cox

              I'm getting the same error.

               

              I have about 85k recurrences of this: 15:19:35.625 [EmailPoller-1] ERROR c.solarwinds.oncall.common.crypt.AES - Decryption Error... To Solve: Reset username/password fields in config files then restart the application.

               

              Which username and password is it referencing?  I tried the email send/receive accounts, no change.

                • Re: Alert Central fails to send alert emails, "Error attempting to notify..."
                  oleg.zastavnyi

                  Hi, usually it happens when you move database to the new appliance (or use database recovery mechanism). So, the our security is working here. All that you need is just to reset password of your integration email source.

                    • Re: Alert Central fails to send alert emails, "Error attempting to notify..."
                      jeff.cox

                      I reset the passwords for the send/receive email accounts.  I'm still seeing the following quite regularly in the logs:

                       

                      09:44:51.710 [EmailPoller-1] ERROR c.solarwinds.oncall.common.crypt.AES - Decryption Error... To Solve: Reset username/password fields in config files then restart the application.

                      java.lang.NullPointerException: null

                        at java.nio.ByteBuffer.wrap(ByteBuffer.java:392) ~[na:1.7.0_04]

                        at com.solarwinds.oncall.common.crypt.AES.decrypt(AES.java:117) ~[oncall-common-1.1.4-SNAPSHOT.jar:na]

                        at com.solarwinds.oncall.logic.SettingsLogic.getValue(SettingsLogic.java:101) [oncall-logic-1.1.4-SNAPSHOT.jar:na]

                        at com.solarwinds.oncall.logic.SettingsLogic.getStringValue(SettingsLogic.java:113) [oncall-logic-1.1.4-SNAPSHOT.jar:na]

                        at com.solarwinds.oncall.logic.SettingsLogic$$FastClassByCGLIB$$a46c911.invoke(<generated>) [cglib-2.2.2.jar:na]

                        at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) [cglib-2.2.2.jar:na]

                        at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:618) [spring-aop-3.1.1.RELEASE.jar:3.1.1.RELEASE]

                        at com.solarwinds.oncall.logic.SettingsLogic$$EnhancerByCGLIB$$a42dc676.getStringValue(<generated>) [cglib-2.2.2.jar:na]

                        at com.solarwinds.oncall.email.EmailPollerImpl.getEmailSettings(EmailPollerImpl.java:167) [oncall-email-1.1.4-SNAPSHOT.jar:na]

                        at com.solarwinds.oncall.email.EmailPollerImpl.access$400(EmailPollerImpl.java:71) [oncall-email-1.1.4-SNAPSHOT.jar:na]

                        at com.solarwinds.oncall.email.EmailPollerImpl$FetchEmail.run(EmailPollerImpl.java:273) [oncall-email-1.1.4-SNAPSHOT.jar:na]

                        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_04]

                        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) [na:1.7.0_04]

                        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) [na:1.7.0_04]

                        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_04]

                        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_04]

                        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_04]

                        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_04]

                        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_04]

                        • Re: Alert Central fails to send alert emails, "Error attempting to notify..."
                          olha.zelena

                          Could you please open Getting Started Wizard - > Navigate to 'Tell Us About the Email Account You Created' step and make sure that after 'Next' clicking, all the steps on 'Email Settings Test' page are pass.

                            • Re: Alert Central fails to send alert emails, "Error attempting to notify..."
                              jeff.cox

                              It fails with Invalid username or password, despite my setting the password moments ago, and copy and pasting it exactly.

                               

                              It was working fine before the version upgrade.

                                • Re: Alert Central fails to send alert emails, "Error attempting to notify..."
                                  olha.zelena

                                  Could you please do the next:

                                  1. Make sure that you can log in with your new credentials for integration account through web

                                  2. You said "I reset the passwords for the send/receive email accounts". Why passwords and accounts? There could be only one integration account. Have I missed anything?

                                  3. Provide an information about your email server

                                  4. Post the exact error you get on 'Email Settings Test' page

                                   

                                  Thanks!

                                    • Re: Alert Central fails to send alert emails, "Error attempting to notify..."
                                      jeff.cox

                                      1. The Orion account is fine

                                      2. The send and receive servers are different

                                      3. We're using Exchange as the mailbox for Alert Central.

                                      4.

                                      DEBUG: setDebug: JavaMail version 1.4.5

                                      DEBUG: getProvider() returning javax.mail.Provider[STORE,pop3,com.sun.mail.pop3.POP3Store,Sun Microsystems, Inc]

                                      DEBUG POP3: mail.pop3.rsetbeforequit: false

                                      DEBUG POP3: mail.pop3.disabletop: true

                                      DEBUG POP3: mail.pop3.forgettopheaders: false

                                      DEBUG POP3: mail.pop3.cachewriteto: false

                                      DEBUG POP3: mail.pop3.filecache.enable: false

                                      DEBUG POP3: mail.pop3.keepmessagecontent: false

                                      DEBUG POP3: mail.pop3.starttls.enable: false

                                      DEBUG POP3: mail.pop3.starttls.required: false

                                        • Re: Alert Central fails to send alert emails, "Error attempting to notify..."
                                          ScottRich

                                          Jeff, what version of Exchange are you using? We are on 2013 and I was getting the similar errors on the POP3 side as well as problems sending on port 25. To fix the POP3 I had to reset the password to only 8 characters and it started working fine (When the password was longer than 8 chars. one of the errors I was getting was 'password cannot be null' so maybe there is a problem with how the password is saved). I also switched to incoming server type Microsoft EWS which seems to work much better with Exchange 2013. For outgoing, I selected 'Use different credentials for SMTP' under Advanced Options, then left the username and password blank.

                                            • Re: Alert Central fails to send alert emails, "Error attempting to notify..."
                                              jeff.cox

                                              I tried changing the password to 8 characters, as well as using EWS instead of POP - no joy.  We're using Exchange 2010.

                                                • Re: Alert Central fails to send alert emails, "Error attempting to notify..."
                                                  olha.zelena

                                                  Jeff, could you please make sure that when you set up your integration account you're using SSL/TLS connection.

                                                   

                                                  SSL&amp;TLS.png

                                                  If it won't help, try to use IMAP or EWS also with SSL/TLS connection types

                                                    • Re: Alert Central fails to send alert emails, "Error attempting to notify..."
                                                      jeff.cox

                                                      I switched to SSL, now I see this when stepping through the configuration wizard:

                                                      Screen Shot 2014-04-08 at 10.40.29 AM.png

                                                       

                                                      I see this error in the logs when trying to send a test:

                                                      11:20:35.286 [NotificationQueue Listener-1] ERROR c.s.o.n.p.EmailNotificationPlugin - email sender failed to send: [AC-Assign] Test alert - NID[2726] to test@test.com

                                                      java.lang.NullPointerException: null

                                                        at java.util.Hashtable.put(Hashtable.java:432) ~[na:1.7.0_04]

                                                        at com.solarwinds.oncall.notification.EmailSender.getNewSession(EmailSender.java:86) ~[oncall-notification-1.1.4-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.notification.EmailSender.send(EmailSender.java:139) ~[oncall-notification-1.1.4-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.notification.plugins.EmailNotificationPlugin.notify(EmailNotificationPlugin.java:169) ~[oncall-notification-1.1.4-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.notification.NotificationMessageProcessor.processNotification(NotificationMessageProcessor.java:52) [oncall-notification-1.1.4-SNAPSHOT.jar:na]

                                                        at com.solarwinds.oncall.notification.NotificationMessageListener.onMessage(NotificationMessageListener.java:29) [oncall-notification-1.1.4-SNAPSHOT.jar:na]

                                                        at org.springframework.jms.listener.adapter.MessageListenerAdapter.onMessage(MessageListenerAdapter.java:339) [spring-jms-3.1.1.RELEASE.jar:3.1.1.RELEASE]

                                                        at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:537) [spring-jms-3.1.1.RELEASE.jar:3.1.1.RELEASE]

                                                        at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:497) [spring-jms-3.1.1.RELEASE.jar:3.1.1.RELEASE]

                                                        at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:468) [spring-jms-3.1.1.RELEASE.jar:3.1.1.RELEASE]

                                                        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:326) [spring-jms-3.1.1.RELEASE.jar:3.1.1.RELEASE]

                                                        at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:264) [spring-jms-3.1.1.RELEASE.jar:3.1.1.RELEASE]

                                                        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1071) [spring-jms-3.1.1.RELEASE.jar:3.1.1.RELEASE]

                                                        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1063) [spring-jms-3.1.1.RELEASE.jar:3.1.1.RELEASE]

                                                        at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:960) [spring-jms-3.1.1.RELEASE.jar:3.1.1.RELEASE]

                                                        at java.lang.Thread.run(Thread.java:722) [na:1.7.0_04]

                                                      11:20:35.431 [WorkflowQueue Listener-1]  WARN c.s.o.w.WorkflowControllerImpl - Processing notification failure: notifId=2726, alertId=0, error=null