Auto-scale Drupal site: AWS or Azure

Ambarishrh

As the title says, I am trying to deploy a drupal site on either Azure or AWS, this site is expected to have huge traffic for a campaign next month end. Currently on a vps hosting, single server (LAMP), as that handles the normal traffic without issues. I still need to look at both solutions, but i guess Azure higher VMs has autoscaling in built i guess based on the usage, it scales up and down and for AWS you need to set it up with Amazon CLI to set threshold.

stacksofplates

@Ambarishrh said in Auto-scale Drupal site: AWS or Azure:

As the title says, I am trying to deploy a drupal site on either Azure or AWS, this site is expected to have huge traffic for a campaign next month end. Currently on a vps hosting, single server (LAMP), as that handles the normal traffic without issues. I still need to look at both solutions, but i guess Azure higher VMs has autoscaling in built i guess based on the usage, it scales up and down and for AWS you need to set it up with Amazon CLI to set threshold.

Just curious, Drupal 7 or 8?

Ambarishrh

Drupal 7

Ambarishrh

Interesting to see that this page is already indexed on google

https://i.imgur.com/y5utULY.png

scottalanmiller

AWS for sure. Why would Azure even be on the list for consideration?

Ambarishrh

Wondering why Azure is not an option!

Ambarishrh

For AWS: http://docs.aws.amazon.com/getting-started/latest/drupal/build-a-drupal-website-on-aws.html

scottalanmiller

@Ambarishrh said in Auto-scale Drupal site: AWS or Azure:

Wondering why Azure is not an option!

Why would it be an option? Just because a service exists doesn't put it on the table. There are hundreds of cloud vendors out there, why would Azure come up and not any of them? You need to know the "why I chose these two" not, "why not this one totally random, valueless service?"

Azure is expensive, fragile, complex and unreliable. What you are looking to do does not check any of the boxes that would normally make Azure a viable contender. So the reason why it is not an option is because it is "inferior to other options in every aspect and has no reason to have been considered."

Ambarishrh

Azure might be expensive (I need to take some time to actually compare the pricing) but i am interested to know why you consider Azure as unreliable. Would be great if you can give some more insights which makes you end up in the conclusion that Azure is not a great choice.

I read something here https://news.ycombinator.com/item?id=12626237

Mainly:

AWS documentation is excellent, Azure docs are weird and inconsistent and for some bits nonexistent.
Azure API's are inconsistent and weird, but once you figure out they work relatively well. But the lack of documentation compounds confusion.
Azure has a lot of very weird limitations that don't make any sense:

-- Default Centos images are 30GB osDisk and you can't resize them, you have to create your own images if you do want to.

-- You can have SSD's in 128/512/1024GB sizes and you pay for them in full, Spinning disks are billed per actual usage.

-- You have to store your osDisk image in the same storage account as your machine you are running (so you have to pay for your image the full SSD monthly price)

-- You have a VMSS (=Auto Scaling Group) and have a Load Balancer in front of it, your microservice connections fail if the load balancer routes the connection back to the same VM ... you now have to have another VMSS just for load balancing/service discovery.

Their services labelled Beta are really more like Alpha quality.
On the plus side, their ARM templates are richer and nicer to use than CloudFormation, however the lack of documentation for them kills all the advantages.
When you jump all the hoops and get past the issues, the things work relatively well.

scottalanmiller

@Ambarishrh said in Auto-scale Drupal site: AWS or Azure:

Azure might be expensive (I need to take some time to actually compare the pricing) but i am interested to know why you consider Azure as unreliable.

First hand experience and direct references from hundreds of Azure customers that all had the same issues. Just attend any Microsoft event and ask the people if they've had Azure outages that were extensive and poorly supported. It's a well known issue that Azure is unstable, especially in the service provider community where we see it more broadly. MS claims that outages are rare, but in conferences their complaints of major stability problems come from the majority of customers, not a rare few.

scottalanmiller

@Ambarishrh said in Auto-scale Drupal site: AWS or Azure:

Would be great if you can give some more insights which makes you end up in the conclusion that Azure is not a great choice.

The lack of selling points is a big one. In a pros/cons comparison, it has no pros, only cons.

stacksofplates

I had a couple running on Vultr and one still running on MNX.io. Never had any issues with either.

stacksofplates

I also recommend building the site on pantheon. It's free for development and you get dev, test, and live stages. Plus they push out the updates in your dev and you literally just commit them with the push of a button. Its really nice.

scottalanmiller

@stacksofplates said in Auto-scale Drupal site: AWS or Azure:

I also recommend building the site on pantheon. It's free for development and you get dev, test, and live stages. Plus they push out the updates in your dev and you literally just commit them with the push of a button. Its really nice.

That's one that I have not seen, yet.

stacksofplates

@scottalanmiller said in Auto-scale Drupal site: AWS or Azure:

@stacksofplates said in Auto-scale Drupal site: AWS or Azure:

I also recommend building the site on pantheon. It's free for development and you get dev, test, and live stages. Plus they push out the updates in your dev and you literally just commit them with the push of a button. Its really nice.

That's one that I have not seen, yet.

I use it all the time. I think I did a write up on here a long time ago.

marcinozga

@Ambarishrh said in Auto-scale Drupal site: AWS or Azure:

As the title says, I am trying to deploy a drupal site on either Azure or AWS, this site is expected to have huge traffic for a campaign next month end. Currently on a vps hosting, single server (LAMP), as that handles the normal traffic without issues. I still need to look at both solutions, but i guess Azure higher VMs has autoscaling in built i guess based on the usage, it scales up and down and for AWS you need to set it up with Amazon CLI to set threshold.

Define huge traffic. Anonymous or authenticated? Perhaps you don't need to throw money on cloud hosting if traffic is mostly anonymous. Varnish in front of your web server will let you handle much more than moving to scalable host. Or even Boost module and cache directory mounted in ramdisk - much simpler than Varnish - might do the trick.

Ambarishrh

@marcinozga Dont have the exact statistics, which is why i want to setup an autoscaling solution to make sure that the site doesnt go down. After the campaign is over, i can move back to the regular hosting. I saw different options to use AWS with autoscaling but still confused on what is the best approach to use the site solution added to new instances. There are several ways mentioned, create a custom AMI with all installed and configured and use that for auto scaling, use AWS code deploy etc. Still checking out options and testing.

Ambarishrh

Finally, i was able to complete this!

Using Below services from Amazon:

EC2- Compute instances: Used Amazon AMI, installed Apache, PHP and checked out drupal core files, converted to an image.
RDS- MariaDB dedicated instance
Autoscaling Group: Launch config & Auto Scaling Group, using the custom image created from EC2
ELB: Frontend load balancer for web servers
Code Deploy: To deploy the update code files to the web servers. Code deploy takes care of deploying the code to all new servers as it boot us on the autoscale group. (It starts from our internal gitlab server, once all files are committed, we will zip that solution and a gitlab runner will initiate a docker instance which then uses aws cli to push this zip file to S3, code deploy picks it up from there and deploy to the selected auto scale group)

Set threshold alarms: start with 1 server, when CPU reaches 80% and stays for a minute, boot up another instance. When CPU goes down for around 2 minutes, it will terminate 1 instance, to keep the minimal back to 1.

Testing: Using the Linux command stress.
Youtube Video

I will soon be writing a blog post on this, but just wanted to update all here

stacksofplates

@Ambarishrh said in Auto-scale Drupal site: AWS or Azure:

Finally, i was able to complete this!

Using Below services from Amazon:

EC2- Compute instances: Used Amazon AMI, installed Apache, PHP and checked out drupal core files, converted to an image.
RDS- MariaDB dedicated instance
Autoscaling Group: Launch config & Auto Scaling Group, using the custom image created from EC2
ELB: Frontend load balancer for web servers
Code Deploy: To deploy the update code files to the web servers. Code deploy takes care of deploying the code to all new servers as it boot us on the autoscale group. (It starts from our internal gitlab server, once all files are committed, we will zip that solution and a gitlab runner will initiate a docker instance which then uses aws cli to push this zip file to S3, code deploy picks it up from there and deploy to the selected auto scale group)

Set threshold alarms: start with 1 server, when CPU reaches 80% and stays for a minute, boot up another instance. When CPU goes down for around 2 minutes, it will terminate 1 instance, to keep the minimal back to 1.

Testing: Using the Linux command stress.
Youtube Video

I will soon be writing a blog post on this, but just wanted to update all here

Thats awesome!

Ambarishrh

So hit a roadblock about code deploy. This was taken considering we update the files. Later realized that we might have user generated content/content added by admin via the Drupal CMS backened. In that case those files for example images will be saved on the one particular EC2 and during a scale down the instance could go down, or may not be available in the newly booted up instance.

Luckily Amazon have another service called Elastic File System used for such cases! I now moved the sites folder of drupal to EFS.

So basic Drupal core files on the EC2 AMI image i created, plus sites folder on EFS, with auto enabling that on each loaded instance by adding the below script:

#cloud-config
package_upgrade: true
packages:
- nfs-utils
runcmd:
- mkdir -p /var/www/html/efs-mount-point/
- chown ec2-user:ec2-user /var/www/html/efs-mount-point/
- echo "$(curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone).file-system-id.efs.aws-region.amazonaws.com:/ /var/www/html/efs-mount-point nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2 0 0" >> /etc/fstab
- mount -a -t nfs4

I am hoping to do a video tutorial this weekend about the whole setup