Avocode Security

Infrastructure Security

Architecture

Communication Security

• All communication between users and Avocode is encrypted using TLS (up to version 1.3).
• When a user connects to avocode.com, their request is first routed to Cloudflare. The TLS version for this connection is negotiated between the user and Cloudflare. The most secure version will always be used. More detailed information is available at https://cloudflare.com/ssl.
• TLS is terminated inside of Cloudflare's infrastructure so that some content can be cached. When a response from our origin servers is required, the request is sent using another TLS 1.3 connection to our infrastructure.
• We operate a load-balanced group of NGINX servers that terminate the Cloudflare TLS connections and pass the request onto their final destination.

Data Security

• All data that is stored in our infrastructure is encrypted at rest. This includes (but isn't limited to) design files, generated assets (like thumbnails), database storage and database backups.
• We use the identity and access management (IAM) tools in both GCP and AWS to only give our staff access to the services and data that they need to do their job.
• We maintain audit logs for all actions performed within GCP and AWS. Each staff member uses their own account to log into the infrastructure.

Network Segmentation

• Avocode infrastructure in GCP and AWS are both in isolated VPCs.
• All staff access to the services running inside the infrastructure is authenticated and authorized using corporate Google accounts (which all have multi-factor authentication enabled). We do not allow any privileged access based on IP address.

Public Footprint

• The servers running our Kubernetes clusters are exposed publicly but they run the hardened Container-Optimized OS from Google and all administrative access is disabled.
• The only application servers exposed to the public Internet are NGINX reverse proxies.

DDOS Prevention

• All requests to our infrastructure are routed through Cloudflare's CDN network, which is built to withstand complex and large DDOS attacks.

API Rate Limiting

• We have rate limits in place both in Cloudflare and in our infrastructure to reduce the risk of brute-force attacks.

Monitoring and Alerting

• We collect detailed application-level and infrastructure-level metrics. For certain key metrics, we set an alerting threshold so that our on-call team can investigate.

Automation

• All configuration that we need to deploy the entire infrastructure is stored in a central code repository.
• All deployments are scripted and reproducible. We don't operate any so-called "pet servers" that are managed by hand.
• As a result of this, entire environments can be recreated from scratch if needed.

Password Management

• Every employee has individual logins to services (via our corporate GSuite accounts if the service supports it) and they are responsible for creating strong passwords and storing them securely.

Separation of Roles

• Employees are only granted access to production infrastructure or tooling if their job description requires it.
• All access to production infrastructure or customer data is monitored.
• Inside of our Kubernetes clusters, each deployment uses its own service account credentials which only grants it access to certain allowed functions. Most of these service accounts don't allow any direct access to the Kubernetes API and of those that do, most of that access is read-only.
• Our services use dedicated service accounts (in GCP) or IAM accounts (in AWS) to communicate with these cloud services. Services do not share access keys.

Protecting Secrets

• For use by the deployments themselves, secret data is stored in Kubernetes secrets which in turn is stored by the Kubernetes API in etcd. The etc database is encrypted at rest and access is granted using RBAC (protections described above).

Network Security

• All traffic from users is routed to us via Cloudflare, our CDN partner. Cloudflare secures the network edge by:
  • Negotiating the highest TLS version that it can with the client before starting communication. This is TLS 1.3 with 128-bit encryption unless the client browser doesn't support it.
  • Reducing the impact of a DDOS attack by intelligently challenging suspicious users before they are able to continue to our site.
  • Filter out suspicious requests using a web application firewall (WAF).
  • Securing the connection between the CDN and our infrastructure using a TLS 1.3 link.
• All traffic into our backend infrastructure flows through a fleet of hardened NGINX proxies and then onto its final destination. These proxies ensure that external users cannot access open ports or endpoints that aren't explicitly whitelisted for public access.
• All network traffic is logged for later analysis.
• All network traffic to internal tools is logged and tied to the user for later analysis.

Vulnerability Disclosure

• We operate a vulnerability disclosure and bug bounty program through HackerOne.
• Since the program's inception, we've resolved over 60 issues and awarded more than $7000 to the security researchers that reported them.

Database Integrity and Security

• Access to any of the databases that we use in production is only given as necessary. See "Separation of Roles."
• Every database is backed up daily.
• Programmatic access to the database is only allowed with a valid service account (which is a RSA private key) and database password.

Data Location

• All compute, database and some design assets are stored in GCP Cloud Storage in the us-east1 region.
• Most design files and related assets are stored in AWS S3 in the us-east-1 region.

Audit Logging

• GCP and AWS maintain audit logs for all administrative activity related to our clusters, databases, compute nodes, object storage, etc.

Infrastructure Security

Architecture

Communication Security

• All communication between users and Avocode is encrypted using TLS (up to version 1.3).
• When a user connects to avocode.com, their request is first routed to Cloudflare. The TLS version for this connection is negotiated between the user and Cloudflare. The most secure version will always be used. More detailed information is available at https://cloudflare.com/ssl.
• TLS is terminated inside of Cloudflare's infrastructure so that some content can be cached. When a response from our origin servers is required, the request is sent using another TLS 1.3 connection to our infrastructure.
• We operate a load-balanced group of NGINX servers that terminate the Cloudflare TLS connections and pass the request onto their final destination.

Data Security

• All data that is stored in our infrastructure is encrypted at rest. This includes (but isn't limited to) design files, generated assets (like thumbnails), database storage and database backups.
• We use the identity and access management (IAM) tools in both GCP and AWS to only give our staff access to the services and data that they need to do their job.
• We maintain audit logs for all actions performed within GCP and AWS. Each staff member uses their own account to log into the infrastructure.

Network Segmentation

• Avocode infrastructure in GCP and AWS are both in isolated VPCs.
• All staff access to the services running inside the infrastructure is authenticated and authorized using corporate Google accounts (which all have multi-factor authentication enabled). We do not allow any privileged access based on IP address.

Public Footprint

• The servers running our Kubernetes clusters are exposed publicly but they run the hardened Container-Optimized OS from Google and all administrative access is disabled.
• The only application servers exposed to the public Internet are NGINX reverse proxies.

DDOS Prevention

• All requests to our infrastructure are routed through Cloudflare's CDN network, which is built to withstand complex and large DDOS attacks.

API Rate Limiting

• We have rate limits in place both in Cloudflare and in our infrastructure to reduce the risk of brute-force attacks.

Monitoring and Alerting

• We collect detailed application-level and infrastructure-level metrics. For certain key metrics, we set an alerting threshold so that our on-call team can investigate.

Automation

• All configuration that we need to deploy the entire infrastructure is stored in a central code repository.
• All deployments are scripted and reproducible. We don't operate any so-called "pet servers" that are managed by hand.
• As a result of this, entire environments can be recreated from scratch if needed.

Password Management

• Every employee has individual logins to services (via our corporate GSuite accounts if the service supports it) and they are responsible for creating strong passwords and storing them securely.

Separation of Roles

• Employees are only granted access to production infrastructure or tooling if their job description requires it.
• All access to production infrastructure or customer data is monitored.
• Inside of our Kubernetes clusters, each deployment uses its own service account credentials which only grants it access to certain allowed functions. Most of these service accounts don't allow any direct access to the Kubernetes API and of those that do, most of that access is read-only.
• Our services use dedicated service accounts (in GCP) or IAM accounts (in AWS) to communicate with these cloud services. Services do not share access keys.

Protecting Secrets

• For use by the deployments themselves, secret data is stored in Kubernetes secrets which in turn is stored by the Kubernetes API in etcd. The etc database is encrypted at rest and access is granted using RBAC (protections described above).

Network Security

• All traffic from users is routed to us via Cloudflare, our CDN partner. Cloudflare secures the network edge by:
  • Negotiating the highest TLS version that it can with the client before starting communication. This is TLS 1.3 with 128-bit encryption unless the client browser doesn't support it.
  • Reducing the impact of a DDOS attack by intelligently challenging suspicious users before they are able to continue to our site.
  • Filter out suspicious requests using a web application firewall (WAF).
  • Securing the connection between the CDN and our infrastructure using a TLS 1.3 link.
• All traffic into our backend infrastructure flows through a fleet of hardened NGINX proxies and then onto its final destination. These proxies ensure that external users cannot access open ports or endpoints that aren't explicitly whitelisted for public access.
• All network traffic is logged for later analysis.
• All network traffic to internal tools is logged and tied to the user for later analysis.

Vulnerability Disclosure

• We operate a vulnerability disclosure and bug bounty program through HackerOne.
• Since the program's inception, we've resolved over 60 issues and awarded more than $7000 to the security researchers that reported them.

Database Integrity and Security

• Access to any of the databases that we use in production is only given as necessary. See "Separation of Roles."
• Every database is backed up daily.
• Programmatic access to the database is only allowed with a valid service account (which is a RSA private key) and database password.

Data Location

• All compute, database and some design assets are stored in GCP Cloud Storage in the us-east1 region.
• Most design files and related assets are stored in AWS S3 in the us-east-1 region.

Audit Logging

• GCP and AWS maintain audit logs for all administrative activity related to our clusters, databases, compute nodes, object storage, etc.

Product Security

Password Security

• Passwords are stored using pbkdf2_sha256 encryption with 36,000 iterations.
• We enforce a minimum password strength of 6 alphanumeric characters that are not similar to either the email address or name of the user.
• Users can reset their passwords using a self-service email form.
• Users cannot reuse passwords when changing a password.

API Tokens

• When user account is deleted or has its password changed, all existing sessions are deleted.

Product Security

Password Security

• Passwords are stored using pbkdf2_sha256 encryption with 36,000 iterations.
• We enforce a minimum password strength of 6 alphanumeric characters that are not similar to either the email address or name of the user.
• Users can reset their passwords using a self-service email form.
• Users cannot reuse passwords when changing a password.

API Tokens

• When user account is deleted or has its password changed, all existing sessions are deleted.

Business Continuity Plan

This business continuity plan analyzes our risk and response for a number of different issues. They are grouped by severity.

Also relevant is the disaster recovery plan, which provides a plan of action for some of these scenarios:

SEV-1 Threats

Complete outage of service in the GCP and AWS regions that Avocode operates in

Description
Occasionally, our hosting providers experience outages that adversely affect our services. Even though these hosting providers build their infrastructure with reliability and resiliency in mind, occasionally some mistakes happen.

Effect
The Avocode service is not operational. New designs cannot be uploaded or processed and accessing existing designs is not possible.

Mitigation
We chose GCP for its excellent reputation in reliability and its track record in resolving issues quickly.

Additionally, we are planning to migrate some services to other GCP regions to increase our resiliency to region-specific outages.

Path to resolution
We rely on GCP technical staff to restore operations as quickly as they can. If core GCP services (like compute and network) are affected, we don't have the ability to work around those issues.

Disruption of service of Cloudflare

Description
We rely on our CDN partner, Cloudflare, to ensure fast global HTTP access to our backend services. Cloudflare operates a highly redundant infrastructure but occasionally there are global issues that could adversely affect Avocode's availability.

Effect
The Avocode service is not operational. New designs cannot be uploaded or processed and accessing existing designs is not possible.

Mitigation
Our ability to communicate an outage to our customers (via email, Intercom chat and our status page) do not depend on Cloudflare.

Path to resolution
We rely on Cloudflare to restore service in a timely fashion. We currently do not have any plan to migrate away from Cloudflare in the event of an outage.

Kubernetes control plane issues

Description
We use Kubernetes as a container orchestrator for all of our services. Kubernetes is complex and we rely on Google's managed service to handle most of the maintenance and operations work of keeping the clusters up and running.

Occasionally, there are issues that occur that prevent deployments from scaling or communicating on the network correctly. These could range anywhere from SEV-1 to SEV-5 - it just depends on the issue and how much of the cluster it affects.

Effect
In a SEV-1 instance, the Avocode service is not operational. New designs cannot be uploaded or processed and accessing existing designs is not possible.

Mitigation
We have monitoring in place to help us identify some of these types of issues. We monitor at the application layer (e.g. number of designs that failed to process) and at the infrastructure layer (e.g. number of 500 errors returned).

This monitoring alerts our on-call team if there is a critical problem.

Path to resolution
Once our on-call engineers are alerted to the issue, they investigate the issue and figure out if it is something that we can fix or something that we need to escalate to Google Cloud Platform support.

Malicious software or other type of security breach

Description
As a internet-based software company, there are many different ways that an attacker could affect our data integrity or operations.

Effect
It depends on the attack.

Mitigation
We have strong security protections in place, as described in Product Security.

Path to resolution
Again, it depends on the attack and the resolution time depends on the complexity of that attack and if we have the skills in-house to handle the resolution.

Database issues

Description
We use Postgres as our primary database for all of our services. We rely on Google's managed database service to handle maintenance and uptime monitoring.

Effect
If database operations were interrupted, the Avocode service would not be operational. New designs cannot be uploaded or processed and accessing existing designs is not possible.

Mitigation
We have monitoring and alerting in place to help us identify if there are any problems in the database. This monitoring alerts our on-call team if there is a critical problem.

Path to resolution
Once our on-call engineers are alerted to the issue, they investigate the issue and figure out if it is something that we can fix or something that we need to escalate to Google Cloud Platform support.

SEV-2 Threats

Disruption of service of Intercom

Description
We use Intercom to provide chat-based and email-based customer support for our users.

Effect
If Intercom were down, customers would have difficulty contacting Avocode support.

Mitigation
For the duration of the outage, we provide our support staff delegated access to the primary support email account so that they can communicate with customers via email. We can also provide a notice on avocode.com to point users to this email address instead of the in-app chat.

Path to resolution
We rely on Intercom to restore service as quickly as they can.

Disruption of normal communication channels

Description
At Avocode, we rely on Slack and email to communicate. We use these channels for normal business communication and also when responding to production incidents.

Effect
A service outage in one or both of these services during a production incident would slow down the identification and resolution of issues.

Mitigation
In addition to Slack and email (hosted by Google GSuite), every on-call team member has the phone number of every other member.

The communication fallback plan is this: Slack → in-person (if in the office) → email → phone

Path to resolution
We believe that the established fallback plan has sufficient redundancy to allow our team to communicate in almost every scenario.

SEV-3 Threats

Disruption of access to physical Avocode offices

Description
Avocode offices could be inaccessible or unusable due to natural disasters, political unrest, utility failures or a number of other reasons.

Effect
If we are unable to use our office for an extended period of time, some business processes that currently rely on face-to-face interaction may take longer.

Mitigation
Our teams can work remotely with the same level of access to our production and administrative infrastructure as they would have in our offices. We authenticate users based on their device and identity, not based on their access to a particular network.

Path to resolution
Our office management team and third parties as necessary work to restore access to the office as soon as possible.

Business Continuity Plan

This business continuity plan analyzes our risk and response for a number of different issues. They are grouped by severity.

Also relevant is the disaster recovery plan, which provides a plan of action for some of these scenarios:

SEV-1 Threats

Complete outage of service in the GCP and AWS regions that Avocode operates in

Description
Occasionally, our hosting providers experience outages that adversely affect our services. Even though these hosting providers build their infrastructure with reliability and resiliency in mind, occasionally some mistakes happen.

Effect
The Avocode service is not operational. New designs cannot be uploaded or processed and accessing existing designs is not possible.

Mitigation
We chose GCP for its excellent reputation in reliability and its track record in resolving issues quickly.

Additionally, we are planning to migrate some services to other GCP regions to increase our resiliency to region-specific outages.

Path to resolution
We rely on GCP technical staff to restore operations as quickly as they can. If core GCP services (like compute and network) are affected, we don't have the ability to work around those issues.

Disruption of service of Cloudflare

Description
We rely on our CDN partner, Cloudflare, to ensure fast global HTTP access to our backend services. Cloudflare operates a highly redundant infrastructure but occasionally there are global issues that could adversely affect Avocode's availability.

Effect
The Avocode service is not operational. New designs cannot be uploaded or processed and accessing existing designs is not possible.

Mitigation
Our ability to communicate an outage to our customers (via email, Intercom chat and our status page) do not depend on Cloudflare.

Path to resolution
We rely on Cloudflare to restore service in a timely fashion. We currently do not have any plan to migrate away from Cloudflare in the event of an outage.

Kubernetes control plane issues

Description
We use Kubernetes as a container orchestrator for all of our services. Kubernetes is complex and we rely on Google's managed service to handle most of the maintenance and operations work of keeping the clusters up and running.

Occasionally, there are issues that occur that prevent deployments from scaling or communicating on the network correctly. These could range anywhere from SEV-1 to SEV-5 - it just depends on the issue and how much of the cluster it affects.

Effect
In a SEV-1 instance, the Avocode service is not operational. New designs cannot be uploaded or processed and accessing existing designs is not possible.

Mitigation
We have monitoring in place to help us identify some of these types of issues. We monitor at the application layer (e.g. number of designs that failed to process) and at the infrastructure layer (e.g. number of 500 errors returned).

This monitoring alerts our on-call team if there is a critical problem.

Path to resolution
Once our on-call engineers are alerted to the issue, they investigate the issue and figure out if it is something that we can fix or something that we need to escalate to Google Cloud Platform support.

Malicious software or other type of security breach

Description
As a internet-based software company, there are many different ways that an attacker could affect our data integrity or operations.

Effect
It depends on the attack.

Mitigation
We have strong security protections in place, as described in Product Security.

Path to resolution
Again, it depends on the attack and the resolution time depends on the complexity of that attack and if we have the skills in-house to handle the resolution.

Database issues

Description
We use Postgres as our primary database for all of our services. We rely on Google's managed database service to handle maintenance and uptime monitoring.

Effect
If database operations were interrupted, the Avocode service would not be operational. New designs cannot be uploaded or processed and accessing existing designs is not possible.

Mitigation
We have monitoring and alerting in place to help us identify if there are any problems in the database. This monitoring alerts our on-call team if there is a critical problem.

Path to resolution
Once our on-call engineers are alerted to the issue, they investigate the issue and figure out if it is something that we can fix or something that we need to escalate to Google Cloud Platform support.

SEV-2 Threats

Disruption of service of Intercom

Description
We use Intercom to provide chat-based and email-based customer support for our users.

Effect
If Intercom were down, customers would have difficulty contacting Avocode support.

Mitigation
For the duration of the outage, we provide our support staff delegated access to the primary support email account so that they can communicate with customers via email. We can also provide a notice on avocode.com to point users to this email address instead of the in-app chat.

Path to resolution
We rely on Intercom to restore service as quickly as they can.

Disruption of normal communication channels

Description
At Avocode, we rely on Slack and email to communicate. We use these channels for normal business communication and also when responding to production incidents.

Effect
A service outage in one or both of these services during a production incident would slow down the identification and resolution of issues.

Mitigation
In addition to Slack and email (hosted by Google GSuite), every on-call team member has the phone number of every other member.

The communication fallback plan is this: Slack → in-person (if in the office) → email → phone

Path to resolution
We believe that the established fallback plan has sufficient redundancy to allow our team to communicate in almost every scenario.

SEV-3 Threats

Disruption of access to physical Avocode offices

Description
Avocode offices could be inaccessible or unusable due to natural disasters, political unrest, utility failures or a number of other reasons.

Effect
If we are unable to use our office for an extended period of time, some business processes that currently rely on face-to-face interaction may take longer.

Mitigation
Our teams can work remotely with the same level of access to our production and administrative infrastructure as they would have in our offices. We authenticate users based on their device and identity, not based on their access to a particular network.

Path to resolution
Our office management team and third parties as necessary work to restore access to the office as soon as possible.

Disaster Recovery Plan

Introduction

Events that disrupt the normal service of Avocode can happen at any time and can come from anywhere. This can range in severity from a bad application deployment to a deleted database.

For incidents that can be corrected without significant service downtime, we maintain a collection of "runbooks" for our on-call team to use during a production incident. There is a runbook for every major alert that details how to identify what the root cause it and the path to restoring service availability.

This document serves as documentation for "disasters", which we define as a catastrophic event such as deleted resources in our cloud environments or disruption of service in our cloud provider.

A quick word about availability zones

Every aspect of our production-facing infrastructure is highly available (HA) across multiple zones in the GCP region. This allows us to tolerate unexpected node failures and other issues that affect availability zones.

Therefore, we don't have any disaster recovery plan for a GCP availability zone outage (and typically don't even notice).

A quick word about storage

We run all of our compute workloads and databases in GCP, but have most of our object storage in Amazon Web Services (AWS) S3 for historical reasons. All three of these components are critical to the full operation of the Avocode service and thus the health of our service is dependent on both cloud providers.

Global Cloud Outage

It is extremely unlikely that a cloud provider would have a full global outage. However, it is possible that a specific service could be impaired globally. For example, we've seen a GCP networking plane issue that prevents all new instances from joining the network correctly. In this event:

Support Team

1. Follow our Incident Response Guidelines to communicate to our customers that we will have an extended outage.

Infrastructure Team

1. Assess the situation based on information provided by Google (status page) or Amazon (status page) and work around the issue if possible.
2. In the event of a compute-based outage (inability to start up new compute nodes, networking failure), engage the plan to spin up a DR compute environment in the other cloud provider. Once the environment is verified, start switching DNS to route traffic to the new environment until service is restored.

Regional Cloud Outage

In the event that our primary GCP region (currently us-east1) or AWS region (currently us-east-1) experiences an outage:

Support Team

1. Follow our Incident Response Guidelines to communicate to our customers that we will have an extended outage.

Infrastructure Team

1. Assess the situation based on information provided by Google (status page) or Amazon (status page) and work around the issue if possible. For example, if it is a partial networking issue that only affects some availability zones, drain nodes in that AZ so that the pods get rescheduled on nodes in other AZs.

Kubernetes Cluster Outage

In the event that our Kubernetes cluster is deleted or otherwise needs to be restored:

Support Team

1. Follow our Incident Response Guidelines to communicate to our customers that we will have an extended outage.

Infrastructure Team

1. Create a new Kubernetes Engine cluster with the same configuration as the primary cluster.
2. Once the cluster is provisioned, deploy all services to the new cluster. Everything deployed within our Kubernetes clusters is defined in code, so deploying is straightforward. The First Run document details how to do this.
3. Open a new shell to the new environment and obtain the public IP addresses for each ingress service. kubectl get svc | grep ingress
4. Go to Cloudflare and open the DNS section for avocode.com. Search for "ingress" and replace each A record with the IP address of the new cluster.

CDN Outage

We use Cloudflare as our primary CDN provider. All HTTP traffic to Avocode services is routed first through their CDN. Cloudflare has over 150 points of presence around the world and operates a highly redundant infrastructure. However, if they suffer a global outage:

Support Team

1. Follow our Incident Response Guidelines to communicate to our customers that we will have an extended outage.

Infrastructure Team

1. Assess the situation based on information provided by Cloudflare (status page). It is likely that the service will be back to normal operation quickly. We currently do not have any plan to migrate away from Cloudflare in the event of an outage.

Intercom Outage

We use Intercom to communicate with our users, using both email and in-app chat. In the event that Intercom has downtime:

Support Team

1. Create a StatusPage incident based on our Incident Response Guidelines and make sure to publish the information on Twitter.
2. Assess the situation based on information provided by Intercom (status page). If the outage lasts more than a few minutes, the Infrastructure Team can give support team members delegated access to the support inbox that is normally managed by Intercom.

Significant Dependency Outage

In the event that one of our third-party dependencies has an outage, Avocode itself could have a partial outage. Here is the severity of each service that we depend on:

Disaster Recovery Plan

Introduction

Events that disrupt the normal service of Avocode can happen at any time and can come from anywhere. This can range in severity from a bad application deployment to a deleted database.

For incidents that can be corrected without significant service downtime, we maintain a collection of "runbooks" for our on-call team to use during a production incident. There is a runbook for every major alert that details how to identify what the root cause it and the path to restoring service availability.

This document serves as documentation for "disasters", which we define as a catastrophic event such as deleted resources in our cloud environments or disruption of service in our cloud provider.

A quick word about availability zones

Every aspect of our production-facing infrastructure is highly available (HA) across multiple zones in the GCP region. This allows us to tolerate unexpected node failures and other issues that affect availability zones.

Therefore, we don't have any disaster recovery plan for a GCP availability zone outage (and typically don't even notice).

A quick word about storage

We run all of our compute workloads and databases in GCP, but have most of our object storage in Amazon Web Services (AWS) S3 for historical reasons. All three of these components are critical to the full operation of the Avocode service and thus the health of our service is dependent on both cloud providers.

Global Cloud Outage

It is extremely unlikely that a cloud provider would have a full global outage. However, it is possible that a specific service could be impaired globally. For example, we've seen a GCP networking plane issue that prevents all new instances from joining the network correctly. In this event:

Support Team

1. Follow our Incident Response Guidelines to communicate to our customers that we will have an extended outage.

Infrastructure Team

1. Assess the situation based on information provided by Google (status page) or Amazon (status page) and work around the issue if possible.
2. In the event of a compute-based outage (inability to start up new compute nodes, networking failure), engage the plan to spin up a DR compute environment in the other cloud provider. Once the environment is verified, start switching DNS to route traffic to the new environment until service is restored.

Regional Cloud Outage

In the event that our primary GCP region (currently us-east1) or AWS region (currently us-east-1) experiences an outage:

Support Team

1. Follow our Incident Response Guidelines to communicate to our customers that we will have an extended outage.

Infrastructure Team

1. Assess the situation based on information provided by Google (status page) or Amazon (status page) and work around the issue if possible. For example, if it is a partial networking issue that only affects some availability zones, drain nodes in that AZ so that the pods get rescheduled on nodes in other AZs.

Kubernetes Cluster Outage

In the event that our Kubernetes cluster is deleted or otherwise needs to be restored:

Support Team

1. Follow our Incident Response Guidelines to communicate to our customers that we will have an extended outage.

Infrastructure Team

1. Create a new Kubernetes Engine cluster with the same configuration as the primary cluster.
2. Once the cluster is provisioned, deploy all services to the new cluster. Everything deployed within our Kubernetes clusters is defined in code, so deploying is straightforward. The First Run document details how to do this.
3. Open a new shell to the new environment and obtain the public IP addresses for each ingress service. kubectl get svc | grep ingress
4. Go to Cloudflare and open the DNS section for avocode.com. Search for "ingress" and replace each A record with the IP address of the new cluster.

CDN Outage

We use Cloudflare as our primary CDN provider. All HTTP traffic to Avocode services is routed first through their CDN. Cloudflare has over 150 points of presence around the world and operates a highly redundant infrastructure. However, if they suffer a global outage:

Support Team

1. Follow our Incident Response Guidelines to communicate to our customers that we will have an extended outage.

Infrastructure Team

1. Assess the situation based on information provided by Cloudflare (status page). It is likely that the service will be back to normal operation quickly. We currently do not have any plan to migrate away from Cloudflare in the event of an outage.

Intercom Outage

We use Intercom to communicate with our users, using both email and in-app chat. In the event that Intercom has downtime:

Support Team

1. Create a StatusPage incident based on our Incident Response Guidelines and make sure to publish the information on Twitter.
2. Assess the situation based on information provided by Intercom (status page). If the outage lasts more than a few minutes, the Infrastructure Team can give support team members delegated access to the support inbox that is normally managed by Intercom.

Significant Dependency Outage

In the event that one of our third-party dependencies has an outage, Avocode itself could have a partial outage. Here is the severity of each service that we depend on:

Data Breach Notification Policy

This policy defines what constitutes a data breach and the actions that will be taken by Avocode to manage the situation and notify users.

What qualifies as "sensitive data"?

This policy covers the following sensitive data:

• Names and private email addresses
• Any information in the user profile (such as company name)
• Encrypted passwords
• Design files or assets generated from them (such as bitmaps generated for thumbnails)
• User interactions with designs (such as comments or version comments)

What qualifies as a data breach?

A data breach is any exposure of sensitive data. Breaches can be unintentional, due to things such as human errors or misconfigurations. Breaches can also be caused by malicious actors, both from internal team members and outside attackers.

Avocode considers an event to be a data breach if there is evidence that sensitive data has been exposed to an untrusted third party.

What doesn't qualify as a data breach?

Examples of events that are not considered to be data breaches:

• Transmission of sensitive data to a trusted third-party partners. These partners have signed non-disclosure agreements and are authorized to process and store sensitive data.
• Theft of a team member's laptop which doesn't contain sensitive data.
• Malware infection on server or endpoint that doesn't contain sensitive data.
• Compromise of access keys or other credentials if there is no evidence that they were used to access sensitive data.
• Discovery of a security vulnerability that could be used by a third-party to access sensitive data if there is no evidence that it was used to access sensitive data.
• Exposure of non-sensitive data such as server logs that do not have any design metadata.

Who will be notified after a data breach?

If Avocode considers an event to be a data breach (per the guidelines stated above), all affected users will be notified via their registered email address and, if possible, the in-app Intercom chat.

In the event of a data breach that affects a large number of users, Avocode may choose to publish information about the breach via a blog post or other means of communication.

This section was adapted from https://about.gitlab.com/security, which is licensed under the Attribution-ShareAlike 4.0 license. As such, this document is also released under that license.

Data Breach Notification Policy

This policy defines what constitutes a data breach and the actions that will be taken by Avocode to manage the situation and notify users.

What qualifies as "sensitive data"?

This policy covers the following sensitive data:

• Names and private email addresses
• Any information in the user profile (such as company name)
• Encrypted passwords
• Design files or assets generated from them (such as bitmaps generated for thumbnails)
• User interactions with designs (such as comments or version comments)

What qualifies as a data breach?

A data breach is any exposure of sensitive data. Breaches can be unintentional, due to things such as human errors or misconfigurations. Breaches can also be caused by malicious actors, both from internal team members and outside attackers.

Avocode considers an event to be a data breach if there is evidence that sensitive data has been exposed to an untrusted third party.

What doesn't qualify as a data breach?

Examples of events that are not considered to be data breaches:

• Transmission of sensitive data to a trusted third-party partners. These partners have signed non-disclosure agreements and are authorized to process and store sensitive data.
• Theft of a team member's laptop which doesn't contain sensitive data.
• Malware infection on server or endpoint that doesn't contain sensitive data.
• Compromise of access keys or other credentials if there is no evidence that they were used to access sensitive data.
• Discovery of a security vulnerability that could be used by a third-party to access sensitive data if there is no evidence that it was used to access sensitive data.
• Exposure of non-sensitive data such as server logs that do not have any design metadata.

Who will be notified after a data breach?

If Avocode considers an event to be a data breach (per the guidelines stated above), all affected users will be notified via their registered email address and, if possible, the in-app Intercom chat.

In the event of a data breach that affects a large number of users, Avocode may choose to publish information about the breach via a blog post or other means of communication.

This section was adapted from https://about.gitlab.com/security, which is licensed under the Attribution-ShareAlike 4.0 license. As such, this document is also released under that license.

Human Resources

During Employment

Non-disclosure Agreements

All staff members must sign a standard non-disclosure agreement (NDA) that legally prevents them from sharing internal or customer data.

Disciplinary Process

If a staff member fails to comply with the stated Avocode information security requirements, there will be disciplinary action taken against said individual.

After Employment

Access Revocation

When a staff member ends their employment with Avocode, access to internal tools and services is revoked. The HR and IT teams have access to an updated list of services.

Human Resources

During Employment

Non-disclosure Agreements

All staff members must sign a standard non-disclosure agreement (NDA) that legally prevents them from sharing internal or customer data.

Disciplinary Process

If a staff member fails to comply with the stated Avocode information security requirements, there will be disciplinary action taken against said individual.

After Employment

Access Revocation

When a staff member ends their employment with Avocode, access to internal tools and services is revoked. The HR and IT teams have access to an updated list of services.

Information Security Levels

Avocode staff members deal with lots of different kinds of information on a daily basis. We comply with the ISO 27001 standard of information classification to determine how we should protect this data (from least to most sensitive):

Level 1: Public

This is generally available information that can be reproduced internally or externally without any restrictions.

An example of public information is marketing materials that have already been released on avocode.com.

Level 2: Internal Use Only

This is information that can be distributed broadly to all staff members and, with authorization, trusted third parties. Unauthorized or unintended distribution would be inappropriate and inconvenient, but wouldn't be especially damaging.

An example of internal use information is internal memos sent to all staff members.

Level 3: Restricted

This is information that is restricted within Avocode and shared only on a need-to-know basis. Unauthorized or unintended distribution could damage the company in a number of different ways. For example, it could cause a drop in customer confidence around our internal security or it could help a competitor gain an edge.

An example of restricted information is accounting data detailing our internal costs and expenses.

Level 4: Confidential

This is information that is highly restricted within Avocode and shared only on a need-to-know basis. Unauthorized or unintended distribution would cause significant damage to the company. A security incident involving confidential data related to customers would need to be communicated to all of those that are affected.

An example of confidential information is customer design assets.

Information Security Levels

Avocode staff members deal with lots of different kinds of information on a daily basis. We comply with the ISO 27001 standard of information classification to determine how we should protect this data (from least to most sensitive):

Level 1: Public

This is generally available information that can be reproduced internally or externally without any restrictions.

An example of public information is marketing materials that have already been released on avocode.com.

Level 2: Internal Use Only

This is information that can be distributed broadly to all staff members and, with authorization, trusted third parties. Unauthorized or unintended distribution would be inappropriate and inconvenient, but wouldn't be especially damaging.

An example of internal use information is internal memos sent to all staff members.

Level 3: Restricted

This is information that is restricted within Avocode and shared only on a need-to-know basis. Unauthorized or unintended distribution could damage the company in a number of different ways. For example, it could cause a drop in customer confidence around our internal security or it could help a competitor gain an edge.

An example of restricted information is accounting data detailing our internal costs and expenses.

Level 4: Confidential

This is information that is highly restricted within Avocode and shared only on a need-to-know basis. Unauthorized or unintended distribution would cause significant damage to the company. A security incident involving confidential data related to customers would need to be communicated to all of those that are affected.

An example of confidential information is customer design assets.

Corporate Security

Physical Network Security

Our physical offices do not have privileged network access to any Avocode internal services or tooling. All access to these services is authenticated and authorized using Avocode corporate SSO accounts or another secure method.

Two-Factor Authorization

All employee Google accounts use multi-factor authentication. We use these Google accounts with additional security protections to gate access to our internal infrastructure and tooling.

Corporate Security

Physical Network Security

Our physical offices do not have privileged network access to any Avocode internal services or tooling. All access to these services is authenticated and authorized using Avocode corporate SSO accounts or another secure method.

Two-Factor Authorization

All employee Google accounts use multi-factor authentication. We use these Google accounts with additional security protections to gate access to our internal infrastructure and tooling.

Production Operations

Change Management

Changes to the production environment can happen in two ways:

1. Authorized developers can run a job in the CI pipeline and deploy new versions of their application to production. The job, along with the staff member that started it, are recorded for later analysis. Importantly, these CI jobs only have permission to change the application version in production. They cannot create, read or delete any data.
2. Production engineers have full access to the production environment in order to make infrastructure changes or debug issues. These engineers can deploy new applications or apply new configuration to the underlying infrastructure. All access is logged by Google Cloud Platform audit logs.

Team leaders and/or release managers generally have the autonomy to review and deploy code written by their team. During large application deployments, we have an internal Slack channel to communicate about the release so that every team can perform a coordinated release.

After deployment, we have an internal metrics dashboard with both application-level metrics (like number of 500 errors) and business-level metrics (like how many designs are processed). If there is an issue, the deployer can roll back the change or request that the production engineer on-call provide assistance.

Patch Management

Avocode maintains a list of all third-party infrastructure components in use in our production environment and third-party software libraries that are used in our applications. The team leader or release manager have the responsibility to either use automated security scanning (where available, like with Node.js dependencies) or manually review new software releases.

We deploy these releases at different times depending on the content and types issues fixed:

• Critical security releases (classified as those that fix high-severity CVE issues) are prioritized in our development roadmap and pushed out as soon as possible.
• Releases that contain important bug fixes or less-severe security fixes are pushed out with our next scheduled release. If there is no release scheduled, a point version will be made and be pushed out.
• Otherwise, releases are applied as needed.

Production Operations

Change Management

Changes to the production environment can happen in two ways:

1. Authorized developers can run a job in the CI pipeline and deploy new versions of their application to production. The job, along with the staff member that started it, are recorded for later analysis. Importantly, these CI jobs only have permission to change the application version in production. They cannot create, read or delete any data.
2. Production engineers have full access to the production environment in order to make infrastructure changes or debug issues. These engineers can deploy new applications or apply new configuration to the underlying infrastructure. All access is logged by Google Cloud Platform audit logs.

Team leaders and/or release managers generally have the autonomy to review and deploy code written by their team. During large application deployments, we have an internal Slack channel to communicate about the release so that every team can perform a coordinated release.

After deployment, we have an internal metrics dashboard with both application-level metrics (like number of 500 errors) and business-level metrics (like how many designs are processed). If there is an issue, the deployer can roll back the change or request that the production engineer on-call provide assistance.

Patch Management

Avocode maintains a list of all third-party infrastructure components in use in our production environment and third-party software libraries that are used in our applications. The team leader or release manager have the responsibility to either use automated security scanning (where available, like with Node.js dependencies) or manually review new software releases.

We deploy these releases at different times depending on the content and types issues fixed:

• Critical security releases (classified as those that fix high-severity CVE issues) are prioritized in our development roadmap and pushed out as soon as possible.
• Releases that contain important bug fixes or less-severe security fixes are pushed out with our next scheduled release. If there is no release scheduled, a point version will be made and be pushed out.
• Otherwise, releases are applied as needed.

Third-Party Data Processors

Avocode uses some third-party services to process data. We trust different services with different levels of information. For more information about how we classify information, see Information Security Levels.

Third-Party Data Processors

Avocode uses some third-party services to process data. We trust different services with different levels of information. For more information about how we classify information, see Information Security Levels.

User permissions

Your designs are completely secured within your team. It's only up to the Owner and Admin or the team to decide who else should have access to particular design projects. When you invite a new person to your team, you can edit their team role and project permissions either upon invite or later when they accept the invitation:

• Owner
  The person who has registered the Avocode team. This user is allowed to access to all projects and designs, invite/delete other users from the team, change roles of other team members and edit their permissions, and access Billing & Payment settings, upgrade, downgrade or cancel the subscription. The Owner has by default the same rights as an Admin and a Full member.
• Admin
  Admin has the same user rights as a Full member and the Owner, but they cannot delete the designated Owner from the team.
• Full member
  Member who has access to all projects but will not see Billing & Payment information and cannot edit the team members (adding, deleting, changing the roles) in any way. The Full member can open any design, switch between versions, add new designs, inspect and comment on designs.
• Contractor
  Contractors are similar to Full members but can only access specific projects defined by the Admin or the Owner of your team. In these projects, the Contractor can add new designs, inspect and comment on designs.
• Guest user
  This user can access only specific projects defined by the Admin or the Owner of your team. In these projects, the Contractor can open any design, switch between versions and comment, but cannot add new designs, delete designs or inspect designs. This team role is available only in the Company and Enterprise plan.

Learn more

Sharing permissions

Avocode allows you to easily share designs via private or public links. There are two levels of link settings:

1. Set link target: Send people either to Preview, Comment or Inspect mode.
2. Set link privacy: Enable anyone with the link to access the design or make it private only for your team. People who don't have a valid login to your Avocode team cannot access the private links. If you send someone outside your team a public link and then change it to private, they won't be able to access it anymore. This way you're in complete control of who can access your designs.

Learn more

User permissions

Your designs are completely secured within your team. It's only up to the Owner and Admin or the team to decide who else should have access to particular design projects. When you invite a new person to your team, you can edit their team role and project permissions either upon invite or later when they accept the invitation:

• Owner
  The person who has registered the Avocode team. This user is allowed to access to all projects and designs, invite/delete other users from the team, change roles of other team members and edit their permissions, and access Billing & Payment settings, upgrade, downgrade or cancel the subscription. The Owner has by default the same rights as an Admin and a Full member.
• Admin
  Admin has the same user rights as a Full member and the Owner, but they cannot delete the designated Owner from the team.
• Full member
  Member who has access to all projects but will not see Billing & Payment information and cannot edit the team members (adding, deleting, changing the roles) in any way. The Full member can open any design, switch between versions, add new designs, inspect and comment on designs.
• Contractor
  Contractors are similar to Full members but can only access specific projects defined by the Admin or the Owner of your team. In these projects, the Contractor can add new designs, inspect and comment on designs.
• Guest user
  This user can access only specific projects defined by the Admin or the Owner of your team. In these projects, the Contractor can open any design, switch between versions and comment, but cannot add new designs, delete designs or inspect designs. This team role is available only in the Company and Enterprise plan.

Learn more

Sharing permissions

Avocode allows you to easily share designs via private or public links. There are two levels of link settings:

1. Set link target: Send people either to Preview, Comment or Inspect mode.
2. Set link privacy: Enable anyone with the link to access the design or make it private only for your team. People who don't have a valid login to your Avocode team cannot access the private links. If you send someone outside your team a public link and then change it to private, they won't be able to access it anymore. This way you're in complete control of who can access your designs.

Learn more

SSO & 2-Factor Authentication

Use SSO to log into Avocode

Avocode allows Enterprise customers to further secure their teams by requiring users to authenticate with Single Sign-On (SSO). We support SAML (Security Assertion Markup Language), which is an industry-standard way for identity providers like Okta and OneLogin to securely pass authorization credentials to Avocode. Enterprise admins can rest easy knowing that their users' Avocode accounts are secured by the same identity provider that they already trust.

2-Factor Authentication

You can protect your data with additional layer of security by enforcing your team users to verify login with 2-factor authentication if your identity provider (IDP), like Okta or OneLogin, supports it.

SSO & 2-Factor Authentication

Use SSO to log into Avocode

Avocode allows Enterprise customers to further secure their teams by requiring users to authenticate with Single Sign-On (SSO). We support SAML (Security Assertion Markup Language), which is an industry-standard way for identity providers like Okta and OneLogin to securely pass authorization credentials to Avocode. Enterprise admins can rest easy knowing that their users' Avocode accounts are secured by the same identity provider that they already trust.

2-Factor Authentication

You can protect your data with additional layer of security by enforcing your team users to verify login with 2-factor authentication if your identity provider (IDP), like Okta or OneLogin, supports it.

Glossary

Glossary