PagerDuty

Introduction

Cloudaware supports bi-directional integration with PagerDuty: 

• PagerDuty acts as a source of data for CMDB, whereby Cloudaware can discover incidents, services, users and other PagerDuty objects (PagerDuty Accounts) 

• PagerDuty serves to be a system of actions where Cloudaware can create and update incidents based on certain criteria met (PagerDuty Notification)

Use cases

Security and compliance

• Security violations to be escalated to the Security team

• Encryption related violation to be escalated to Security team

• CIS Compliance violations to be escalated to networking or operations team

• Available Critical patch vulnerabilities to be escalated to networking team

• HIDS are not active on Production instances

• Security related upgradable patches older than 30 days

• Critical vulnerability scans on production instances

• New publicly accessible S3 bucket

• User without MFA

Billing

Notifications to Application owners with copy to finance team:

• Billing anomalies to be escalated 

• MTD Spend reached 75% of Last Month Spend

• Unused instance Reservation 

• Idling and underutilized resources alerts

• Cost per transaction for S3 buckets higher than defined threshold 

Operations

• Overutilized resources 

• Scheduled AWS Instance and volume maintenance notifications

• Missing backups or monitoring on production environment

• Cloud accounts removal request 

• New resource spun up without required tag

PagerDuty Accounts Integration

The integration allows you to track PagerDuty Incidents, along with overviewing such objects as users, teams, services, etc. in Cloudaware CMDB. The object PagerDuty Incident now has lookups to AWS Account, AWS Region, AWS EC2 Instance, Azure Virtual Machine, Google GCE Instance.

PagerDuty Setup

1. Log in to your PagerDuty account. Click User icon → My Profile. Configure your User Profile adding Contact Information, Notification Rules, User Settings in 'Configuration':

Invite your team members. NOTE: each user will need to set up their Profile settings. 

2. Generate an API key*. In your PagerDuty account go to Configuration → API Access → Create New API Key. In the section ‘API version’:

a. Check the box 'v2(Current)' 

b. Select 'Read-Only API Key'

Click Create Key. Copy it to a safe place and click Close.

3. Configure the Incident Notification Pipeline:

3.1. Add users to On-Call Schedules to distribute incident response responsibility across your teams. Go to Configuration → Schedules → +New On-Call Schedule to add users to a schedule, select a daily or weekly rotation and pick a start time for the schedule:

Click Create Schedule. Learn more

3.2. Add On-Call Schedules to an Escalation Policy. Go to Configuration → Escalation Policies → +New Escalation Policy to create an escalation policy specifying who will be assigned responsibility for resolution when a service is triggered:

Click Save. Learn more

3.3. Create Teams* to customize the UI for specific users in your PagerDuty account (optional). Learn more
3.4. Set up Services and Integrations. Go to Configuration → Services → +New Service to add a service (an application, component, or team) you wish to open incidents against. Manage settings of your Service selecting a tool and the escalation policy you created before:

Click Add Service. Learn more

4. Test your PagerDuty configuration by creating a test incident. Click +New Incident → give your incident a name and select the appropriate policy → Create Incident.

Resolve your test incident. Learn more

Cloudaware Setup

1. Log in to your Cloudaware account. Go to Admin → PagerDuty Accounts. Click +Add.

2. Insert the integration details. Click Save.

3. The green light in 'Status' means that your PagerDuty account has been added successfully. If there is a red light, please contact support@cloudaware.com.

With PagerDuty Accounts integration added, Cloudaware allows you to track activity of PagerDuty users and services:

Cloudaware parses PagerDuty incidents allowing you to gain visibility of events. Sample PagerDuty Incident in Cloudaware:

PagerDuty Notification Integration

The integration allows Cloudaware to create and update incidents in PagerDuty using Outbound Webhooks functionality:

Sample outbound use cases:

• Create a PagerDuty incident in a specific service when Cloudaware detects a spending or a CloudTrail anomaly.

• Update a PagerDuty incident when a critical vulnerability has been remediated.

Event Passthrough:

Cloudaware users can decorate events from third party monitoring systems such as Zabbix, New Relic, Splunk, etc. by sending them to Cloudaware first. Cloudaware may enrich events with details like AWS Account ID or Azure Resource Group and tags, so that further downstream PagerDuty can make more intelligent routing, suppression and escalation decisions.

PagerDuty Setup

1. Generate an API key*. In your PagerDuty account go to Developer Tools → API Access and click Create New API Key. In the section 'API version':

a. Check the box 'v2(Current)' 

b. Select 'Read-Only API Key'

Click Create Key. Copy it to a safe place and click Close.

2. Add and configure a service in PagerDuty that Cloudaware will send notifications to:

2.1. Go to People → Escalation Policies → New Escalation Policy to create an escalation policy specifying who will be assigned responsibility for resolution when a service is triggered.

2.2. Go to Services → Services Directory → +New Service to add a service (an application, component, or team you wish to open incidents against) which will use this policy. Manage the settings:

• 'General Settings': give your service a meaningful name

• 'Integrations Settings': 

a. Select a tool

b. Check the box 'Use our API directly'

c. Select Integration Type: 'Events API v2' 

• 'Incident Settings': select the policy you created before 

Click Add Service.

2.3. Review your service details:

• 'Integrations' creates incidents by connecting to your monitoring tools. You can add multiple integrations to a service.  

• 'Extensions' provides additional functionality to PagerDuty services. Tied to services of your choice, service extensions can respond to actions taking place on services' incidents.

Cloudaware Setup

1. Log in to your Cloudaware account. Go to Admin → Other Integrations → PagerDuty Notification. Click +Add.

2. Insert the integration details:

2.1. You must type API names of fields in 'Incident Key Field', 'Resolve Time Field' and 'Fields' exactly as they are on the object (e.g. CA10__caUuid__c, CA10__disappearanceTime__c, etc).

2.2. The field 'Incident Key Field' acts like dedup_key (see PagerDuty documentation) and will be used by PagerDuty for searching an incident when it is being edited. If this field is not defined, CA10__caUuid__c will be used.

2.3. The field 'Resolve Time Field' will be used to indicate 'Resolve' for an incident in PagerDuty. If it is not mentioned, an incident won't be closed automatically in PagerDuty.

2.4. In the field 'Fields' a list of fields separated by line break sign (\n). This is the list of API Name fields that will be transmitted into a description of the incident in PagerDuty. If you add “-” in this field, it won't be added into a description of the incident (e.g. - Id - will exclude 'Record ID').

Click Save.

3. Set up the workflow rule with outbound messages for creating incidents in PagerDuty:

3.1. Click Setup in the main menu under your username.

3.2. In the Quick Find box start typing workflows to select Workflows & Approvals → Workflow Rules → New Rule:

3.3. Select the object for the rule to be applied to and click Next.
3.4. Add Rule Name (1), set Evaluation Criteria(2) and Rule Criteria(3). Add Filter Logic if necessary. Click Save & Next.

In this example, we set up incidents creation for the object 'AWS EC2 Instance Status Event'. Every time any AWS EC2 Instance's status is changed, Cloudaware triggers a workflow rule to take into action creating a new incident in PagerDuty.

3.5. Add Workflow Action → New Outbound Message:

3.6. Fill out your Outbound Message details. Endpoint URL can be copied from PagerDuty Notification integration you set up in Cloudaware before (e.g. https://inbound-dot-cloudaware-vm.appspot.com/inbound-message/pager-duty/{key}). Select Available Fields to be displayed in a PagerDuty incident. Click Save.

3.7. Review the workflow and click Done.

3.8. Activate the workflow.

4. Once outbound messages are set up, check your PagerDuty Notification integration in Cloudaware. The green light in ‘Status' means that the integration has been enabled successfully. If there is a red light, please contact support@cloudaware.com.