During the past few months, I've been conducting technical interviews for DevOps/SRE roles in my team. It's been fascinating and insightful to talk to various people - from fresh graduates to experienced engineers - who regularly use containerization tools.

However, one thing struck me repeatedly during these conversations:

Many people believe Docker itself creates containers, and that without Docker, containers cannot exist.

Many candidates expressed surprise when I told them: "Docker itself doesn't actually create containers." They often responded with disbelief: "Wait, how? Isn't Docker all about containers? Isn't Docker responsible for creating containers? How else would you create containers if Docker doesn't do it?"

This got me thinking - I should write a detailed blog to clarify this common misconception and share some insights into what actually creates containers. So today, let’s dive deep into the world of containers, Docker, container runtimes, and see a practical example with a simple yet powerful "ContainerHub" project I created for managing container lifecycles.

🚩 The Docker Misconception: Clearing the Air

First things first - let's address the elephant in the room:

Docker is not directly responsible for creating containers.

This might seem confusing, given Docker’s immense popularity in the containerization ecosystem. Docker made container technology accessible and developer-friendly, leading to widespread adoption. But under the hood, Docker itself relies on specialized components known as container runtimes to actually create and manage containers.

To put it clearly:

• Docker is a user-friendly tool providing an intuitive interface and a set of commands that help you build, manage images, and interact with containers.

• Container runtime (like containerd and runc) is the one actually creating and running containers at the OS/kernel level.

Confused? Let's break down these concepts further.

🎯 What Exactly is Creating Containers, then?

Behind Docker lies a core component called a container runtime, responsible for container lifecycle management.

Creating containers involves packaging an application and its dependencies into a standardized unit that can be run consistently across different environments. Containers fundamentally rely on two types of runtimes:

High-Level Container Runtime

High-level container runtimes manage the overall lifecycle of containers, including image creation, transport, and orchestration. Here are their key functions:

1. Image Management: High-level runtimes handle the creation, storage, and distribution of container images. They allow developers to build images from Dockerfiles or other configuration files.

2. Transport: They manage the transport of container images from registries to the host system where containers will be run.

3. API Access: High-level runtimes provide APIs for interacting with containers, enabling operations like starting, stopping, and monitoring containers.

4. Networking: They set up networking between containers, allowing them to communicate with each other and the outside world.

5. Volume Management: High-level runtimes prepare and manage volumes that containers use for persistent storage.

6. Delegation: They delegate the actual running of containers to low-level runtimes.

   

Low-Level Container Runtime

Low-level container runtimes focus on the technical aspects of running containers. Their main responsibilities include:

1. Namespace Setup: They set up Linux namespaces to isolate system resources like the file system, network, and process IDs for each container.

2. Cgroups Management: Low-level runtimes manage cgroups to limit and allocate resources such as CPU and memory to containers.

3. Command Execution: They run commands inside the namespaces and cgroups, ensuring the container operates within its isolated environment.

4. Filesystem Setup: Low-level runtimes set up the root filesystem for containers, often using chroot to change the root directory.

5. Resource Limits: They enforce resource limits on containers, ensuring they do not exceed allocated CPU, memory, and other resources.

   

🔧 Why the Misunderstanding?

Docker's simplicity is the main reason for this confusion. When developers use Docker, everything seems seamless. You type `docker run`, and the container magically appears.

But behind the scenes, Docker delegates this task to runtimes like containerd and runc.

Docker → calls → containerd → calls → runc → Linux kernel → creates containers

💡 Practical Illustration: Introducing ContainerHub

To illustrate how containers can be managed independently of Docker, I've developed ContainerHub, a practical, lightweight project that directly interacts with container runtimes.

🚀 How Does ContainerHub Work (Without Docker!)

ContainerHub manages container lifecycles directly, demonstrating a clear separation of concerns:

• Root Filesystem Preparation: Unlike Docker, ContainerHub requires manual preparation of a root filesystem since no high-level runtime is involved.

• Daemon (containerhubd): A gRPC server that manages container states and interacts directly with runc.

• Client CLI (containerhub): Communicates through sockets to the daemon, allowing direct management of containers.

• Storage: Container states and OCI-compliant configurations are stored systematically at /var/lib/containerhub/<id>.

This setup clearly shows the fundamental responsibilities of container runtimes independently of Docker.

✅ Benefits of Understanding this Clearly

Knowing that Docker itself doesn’t create containers directly brings several benefits:

• Enhanced Debugging: Easier to troubleshoot issues by directly interacting with container runtimes.

• Flexibility: You're free to use alternative container runtimes like Podman, containerd, or CRI-O.

• Better Architecture Decisions: Helps you make informed decisions around security, resource management, and rootless containers.

🎖️ Conclusion and Takeaways

I hope this blog clears up the confusion:

Docker is a powerful tool, but containers are actually created and managed by underlying runtimes like containerd and runc.

Using the ContainerHub example, I've practically demonstrated this concept clearly. Next time someone says "Docker creates containers," you can confidently explain the reality:

“Docker doesn’t create containers itself - it relies on OCI-compliant container runtimes like containerd and runc to do that!”

I'd love to hear your thoughts, experiences, or questions in the comments below!

Happy Containerizing! 🚀🚀🚀

Imagine you’re the Engineering Manager of a fast-growing tech company. Your team is responsible for running a suite of applications on Kubernetes, and everything seems to be going well - until it isn’t. Suddenly, a critical service goes down because it couldn’t get the resources it needed, while lower-priority workloads hogged valuable compute power. Now, you’re facing a production incident and a frustrated product team asking why their mission-critical workload wasn’t prioritized. Sound familiar?

In situations like this, Kubernetes’ default scheduling can feel like a black box - effective for standard cases but not always aligned with your unique business priorities. That’s where custom schedulers come into play. Today, let’s explore a story of how custom schedulers helped an organization like yours take control of their workloads, save money, and ensure critical services always got the attention they deserved.

The Default Scheduler Dilemma

Our story starts with a company named Tecknobody (Fictional), a mid-sized enterprise specializing in financial technology. Tecknobody had adopted Kubernetes to scale their services and reduce manual overhead. At first, everything seemed great - Kubernetes made it easy to deploy applications, and the default scheduler worked well enough for most use cases. But as the business grew, so did the complexity of their infrastructure.

One day, during a high-stakes product demo, their flagship application suffered a significant slowdown. Investigations revealed that a batch of non-critical analytics jobs was taking up the majority of the cluster’s resources. The default Kubernetes scheduler had assigned these batch jobs to nodes with the most available resources, inadvertently starving the high-priority application of the compute it needed to run smoothly. This left Tecknobody’s leadership questioning whether Kubernetes was the right choice for their critical infrastructure.

The Custom Scheduler Solution

Enter Enna, Tecknobody’s Engineering Manager. Enna realized that while Kubernetes was a powerful tool, its default scheduling wasn’t quite tailored to their business needs. She knew that they needed a better way to ensure critical applications were prioritized, especially during peak times. After some research, Enna and her team decided to build a custom scheduler.

The custom scheduler allowed them to define specific rules for workload placement. By implementing a custom scheduling algorithm, they could ensure that mission-critical workloads always had the resources they needed, while less critical batch processes were scheduled with a lower priority. This not only resolved their immediate problem but also brought a sense of control back to the engineering team.

The Journey to Building a Custom Scheduler

Enna’s team started by understanding the scheduler framework that Kubernetes offers. They learned that Kubernetes allows multiple schedulers to run simultaneously in a cluster, meaning they could deploy their own custom scheduler alongside the default one. By specifying a different `schedulerName` in their pod definitions, they could dictate which pods would use the new custom logic.

Tecknobody’s custom scheduler was written in Go using the Kubernetes scheduler framework. It wasn’t an easy journey - Enna had to ensure her team understood the intricacies of scheduling cycles, including filtering nodes, scoring potential candidates, and ultimately binding pods to nodes. They created a set of custom plugins that allowed them to filter and score nodes based on more granular criteria: prioritizing critical workloads, minimizing cost by consolidating non-critical jobs during off-hours, and maintaining compliance by ensuring data-sensitive applications only ran in approved regions.

The Payoff: Control, Efficiency, and Peace of Mind

The impact was immediate. With the custom scheduler in place, Tecknobody could ensure that their flagship application always had priority access to the resources it needed. This translated into a smoother product demo, happier stakeholders, and a more reliable customer experience. Enna’s team also managed to reduce cloud costs by consolidating workloads more efficiently and strategically scaling down non-essential processes during low-demand periods.

For Tecknobody, the custom scheduler became more than just a tool - it was a strategic differentiator. The ability to exert fine-grained control over resource allocation helped align the engineering strategy with business goals. Instead of struggling to work within the limitations of the default Kubernetes scheduler, Enna’s team had created a solution that allowed them to focus on delivering value rather than firefighting infrastructure issues.

Why This Matters for You as an Engineering Leader

As an Engineering Manager or above, you’re likely balancing the needs of multiple teams and managing conflicting priorities. Kubernetes is an amazing tool, but its default settings are designed to fit general scenarios -not the specific needs of your business. Custom schedulers give you the power to implement your own business rules at the infrastructure level.

• Mission-Critical Focus: Ensure that your high-priority applications always get the resources they need, reducing the risk of downtime or degraded performance during crucial moments.

• Cost Savings: Implement logic that takes advantage of low-cost nodes for non-critical workloads, ultimately optimizing your cloud spend.

• Compliance and Control: Enforce data sovereignty requirements by using custom scheduling rules that are impossible to implement with the default scheduler.

The challenges you face aren’t just technical — they’re deeply tied to the outcomes your business wants to achieve. By adopting custom schedulers, you can create a tailored infrastructure that empowers your team, minimizes surprises, and directly contributes to the goals of your organization.

Wrapping Up: The Power of Customization

For Tecknobody, investing in a custom scheduler meant they could move from reacting to infrastructure problems to proactively managing them. For Enna, it meant peace of mind - knowing her team had built a system that aligned perfectly with the business’s unique needs.

If your organization has found itself limited by Kubernetes’ default behavior, it might be time to consider whether a custom scheduler could give you the competitive edge you need. As a leader, embracing this level of customization can be the key to unlocking better efficiency, more predictable performance, and ultimately, happier customers.

Key Learnings: How to Approach and Solve Problems Like an Engineering Leader

1. Understand the Problem Before Blaming the Tool
   Tools like Kubernetes are designed to be general-purpose, but your specific use case might reveal limitations. Take time to investigate the root cause of issues instead of assuming the tool is at fault.

2. Align Infrastructure Decisions with Business Goals
   Engineering challenges are rarely just technical. Consider the broader impact on your business priorities, like cost efficiency, reliability, and compliance, when designing solutions.

3. Research and Explore Alternatives
   Before dismissing the tool, explore its advanced features or alternative approaches. Kubernetes’ custom scheduler framework, for instance, provides a robust way to tailor resource management to your needs.

4. Collaborate and Upskill Your Team
   Complex challenges often require stepping into new territory, like building custom schedulers. Invest in your team’s growth by encouraging learning and experimentation.

5. Think Proactively, Not Reactively
   Instead of waiting for incidents to dictate your next steps, anticipate potential bottlenecks and design systems to mitigate them. This mindset turns firefighting into strategic planning.

6. Leverage the Power of Customization
   One size rarely fits all in engineering. Customization can bridge the gap between default behavior and the unique requirements of your business. Don’t hesitate to tailor solutions to fit your needs.

7. Iterate and Improve Continuously
   The first implementation may not be perfect, but it will serve as a foundation for future improvements. Focus on delivering incremental value while refining your approach over time.

8. Share Knowledge and Learnings
   Challenges and their solutions are opportunities for growth - not just for your team, but for the broader engineering community. Sharing your story helps others navigate similar hurdles and fosters collective innovation.

   Leave a comment

   Thanks for reading GrokTechie! Subscribe for free to receive new posts and support my work.

Why do we need Application Monitoring?

• Information gathering

Large application produces a humongous number of data that should be monitored and analyzed for the performance and improvement of product and thereby better business.

• Around the clock monitoring

Humans will never be able to monitor each server 24 X 7. But it is very crucial that any unexpected incident on the servers should be notified and act on it immediately.

• Pre-emptive Alerting

Monitoring tool is an invisible team member who will alert you about unexpected things in the system.

What is Zabbix?

Zabbix is an open source, real-time application, and network monitoring tool. It offers monitoring of thousands of metrics collected from physical machines or virtual machines. Zabbix has a web-based management interface which is centralized through a database. With Zabbix, visualization of your data is available in the form of graphs, screens, maps and overviews.

Why Choose Zabbix?

• Open Source – Zabbix is purely open source and comes at no cost.

• Active Monitoring – You can easily monitor servers, applications and network devices, gathering accurate statistics and performance data

• Enterprise ready – Zabbix has been designed to scale from small environments to large environments

• Capacity Planning – With the data collected by Zabbix, you will be easy to analyze your infrastructure and plan the capacity accordingly.

Zabbix Architecture

Zabbix is distributed monitoring tool with a central web interface. The simple architecture of Zabbix can be composed of three servers.

• Web Server

• RDBMS Server

• Zabbix server

This setup is good enough to handle large environment successfully. It’s better to use the dedicated server for each one of these components.

Zabbix uses client-server architecture and uses an agent in the servers that should be monitored. This agent gathers all needed information and status from the system and sends to Zabbix server. Zabbix version 3 and above support encrypted communication between server and clients.

The Zabbix server can take input data also from a Zabbix sender or Zabbix proxy servers.  Zabbix sender is a command line utility to send application performance and availability data to Zabbix servers for processing. Zabbix proxies are used to monitor remote servers. It can collect performance and availability data behalf of Zabbix server and save it in a buffer locally and then send it to Zabbix server. This way the proxy servers can take on itself some of the load and offload the Zabbix servers.

Let us look at some of the terminologies of Zabbix.

• Frontend – Web interface provided with Zabbix

• Zabbix Server – Central server to collect the data and process it.

• Zabbix agent – A process deployed on client servers to monitor locally

• Host – A networked device that should be monitored

• Host group – A logical group of hosts. It is used when assigning access rights.

• Template – A set of entities (items, triggers, applications, low-level discovery rules, graphs, screens, web scenarios) ready to be applied to one or several hosts

• Item – A data that you want to receive from a host

• Trigger – It is a logical expression which defines a threshold

• Event – Occurrence of something that deserves attention

• Action -  Predefined way of reacting to an event.

Install and configure Zabbix

Now let’s look at how to install and configure Zabbix 3.0. I’m going to use one Centos 6 machine for all the three components. That are Zabbix Web, Zabbix Server and Database.

Step 1: Disable firewall:

# service iptables stop
# chkconfig iptables off
# service ip6tables stop
# chkconfig ip6tables off
# vim /etc/selinux/config
 SELINUX=disabled
# echo 0 >/selinux/enforce
# reboot

Step 2: Install Mysql on server:

# yum list installed | grep mysql
# rpm -ivh http://dev.mysql.com/get/mysql-community-release-el6-5.noarch.rpm
# yum install mysql-server -y
# yum install vim -y

Step 3: Start Mysql on server:

# service mysqld start
# mysql_secure_installation

"Enter current password for root    (enter for none):   (Press enter)
 Set root password?             [Y/n]:   Y
 New Password          :   Password (Enter your root Password)
 Confirm Password               :   Password (Verify your root Password)
 Remove anonymous users?                [Y/n]:   Y
 Disallow root login remotely?          [Y/n]:   Y
 Remove test database and access to it? [Y/n]:   Y
 Reload privilege tables now?        [Y/n]:   Y

Step 4: Login to mysql:

# mysql -u root -p
# password : Password (Enter your password)
>> CREATE DATABASE zabbix CHARACTER SET utf8 COLLATE utf8_bin;
>> GRANT ALL PRIVILEGES ON zabbix.* TO zabbix@localhost IDENTIFIED BY 'Password';

Step 5: Install PHP on server:

# rpm -ivh http://repo.webtatic.com/yum/el6/latest.rpm
# yum install httpd php56w php56w-gd php56w-mysql php56w-bcmath php56w-mbstring php56w-xml php56w-ldap
# vim /etc/php.ini

 post_max_size=16M
 max_execution_time=300
 max_input_time=300
 date.timezone=Asia/Kolkata
 always_populate_raw_post_data=-1

Step 6: A zabbix user in a group:

# groupadd zabbix 
# useradd -g zabbix zabbix

Step 6: Install zabbix on server:

# mkdir /home/zabbix/downloads
# cd /home/zabbix/downloads
# wget https://sourceforge.net/projects/zabbix/files/ZABBIX%20Latest%20Stable/3.2.4/zabbix-3.2.4.tar.gz
# tar xvf zabbix-3.2.4.tar.gz

Step 7: install zabbix database in Mysql:

# cd /home/zabbix/downloads/zabbix-3.2.4/database/mysql/
# mysql -u zabbix -p zabbix < schema.sql
# mysql -u zabbix -p zabbix < images.sql
# mysql -u zabbix -p zabbix < data.sql

Step 8: Configure the source code:

# yum install gcc mysql-community-devel libxml2-devel  unixODBC-devel net-snmp-devel libcurl-devel libssh2-devel OpenIPMI-devel openssl-devel openldap-devel -y
# cd ../..
# ./configure --enable-server --enable-agent --with-mysql --enable-ipv6 --with-net-snmp --with-libcurl --with-libxml2 --with-unixodbc --with-ssh2 --with-openipmi --with-openssl
# make
# make install

Step 9: Edit the Zabbix config file:

# vim /usr/local/etc/zabbix_server.conf

 DBName=zabbix
 DBUser=zabbix
 DBPassword=your_password

Step 10: Configure zabbix:

# mkdir /var/www/html/zabbix
# cd /home/zabbix/downloads/zabbix-3.2.4/frontends/php/
# cp -a . /var/www/html/zabbix/
# chcon -Rv --type=httpd_sys_content_t /var/www/html
# setsebool -P httpd_can_network_connect=1
# setsebool -P zabbix_can_network=1
# chown -R apache:apache /var/www/html/zabbix
# chmod +x /var/www/html/zabbix/conf/
# cp /home/zabbix/downloads/zabbix-3.2.4/misc/init.d/fedora/core/zabbix_server /etc/init.d/zabbix-server
# cp /home/zabbix/downloads/zabbix-3.2.4/misc/init.d/fedora/core/zabbix_agentd /etc/init.d/zabbix-agentd

Step 11: Start services:

# service zabbix-server start
# service zabbix-agentd start
# service httpd start

Step 12: On services on boot time:

# chkconfig httpd on
# chkconfig mysqld on
# chkconfig zabbix-server on
# chkconfig zabbix-agentd on

1. Use the same address you have for your server now and add /zabbix to the URL.
URL : Server-IP/zabbix
>>Next

2. Check for pre-requisites and fix errors if you have any.
>>Next

3. Enter settings to connect to database.
password : Password (Enter the password of Mysql)
>>Next

4.  Enter details to connect to web interface.
Name : 3.0.3
>>Next
>>Finish

5. Connect to the Zabbix web interface with default credentials.
Username: Admin
Password: zabbix

Hope this will help you!

Please Remember me in your prayers!

Enjoy 

To achieve the desired results we create a POC (proof of concept environment) that is composed of the following pieces:

`

In this post I will show how created that site to site IPsec tunnel to connect to a Virtual Private Cloud (VPC) hosted in Amazon Web Services (AWS) to a Virtual Network in Azure. Using this setup you can have workloads in one or both clouds with full VM to VM connectivity over a secure IPsec tunnel. This easily can provide connectivity for high availability operations.

In our case we selected a Windows 2012 R2 server as the end point of the tunnel on the AWS side mostly because it is a supported platform as a Azure VPN device for route-based VPN that did not add costs to our POC unlike other supported virtual appliances from the AWS marketplace that are supported as validated VPN devices on Azure.

let’s get started…

Create an Elastic IP in AWS

First we need to an allocate address in AWS.  It will be assigned in later steps to the AWS VPN gateway. However we will need that address before we start to configuring the Azure infrastructure.

1. In the VPC dashboard, click Elastic IPs , allocate New Address and click Yes, Allocate

This will create the address but it will not assign it anywhere. We're doing this now since we need the IP address to setup the Azure side of the VPN

Prepare the Azure side of the VPN

1. We used the Resource Group deployment model.  So first, we need to create a Resource group for our POC.  We created AZURE2AWS

2. In the AZURE2AWS Resource Group Add a virtual Network.

3. In the Next blade ensure to select Resource Manager as the deployment model and click Create

4. Fill in creation details and click Create.

We used one Virtual Network called Azure2Aws with an address space of 172.16.0.0/16.  In it we created a subnet called FrontEnd with the address range 172.16.1.0/24

5. In the Virtual Network I created a GTW subnet that will be used when we create the Gateway.

When providing the subnet name ensure that you use “GatewaySubnet” (This name is mandatory for Gateway Subnets).  We used the subnet address space – 172.16.254.0/24

6. Once our virtual network was created. We deployed a Windows 2012 R2 server in the virtual network.  It will act as a DC and DNS server for the lab. ( we deployed it at this point so it will be ready by the time we need it).  We ensured that the NIC for the server was set to static since we will use it as a DC and as a DNS server.

7. We navigated back to the Azure2Aws resource group and added a Virtual Network Gateway.

Clicked Create in the Virtual network Gateway blade, provided the info needed for the creation (Gateway name, IP address Name, route-based) and click create. (Be patient it will take some time to generate the gateway).

After the gateway has been created you will need to note the IP Address of the Virtual network Gateway you will need it later.

8. Then, navigate back to the Resource Group and Add a Local Network Gateway.

I once you selected Local Network Gateway above, click create in the next blade, fill the information of the AWS VPC (IP address is the Elastic IP assigned to the GTW – We created that in the first step – in the AWS VPC, and address space is the VPC CIDR in AWS) ensure that you are always selecting the right Resource Group in our case Azure2Aws and click create

9. Wait for the Local network Gateway to be created. Once it is created, in the settings of the local network gateway, in the connections setting, add a new connection.

In the connection settings give it a name select the Azure Virtual network gateway we created earlier and supply a shared key note the shared key , you will need it later.  I recommend you DO NOT use a1b2c3d4e5 has we did.  This is a test environment.  please generate a strong shared key.

10. I set the Virtual Network DNS to be the DNS on my DC1 machine (the one we created earlier). Navigate to the Resource Group, select the Virtual Network, all settings, set the DNS to custom and provide the address of the DC1 NIC.

Prepare AWS side

1. Now that the Azure side of the equation is ready Let’s move to the AWS side of things.  First we created a new VPC.

• Name Aws2Azure

• Address block of 10.0.0.0/16

• I used the following name: awssubnet

• CIDR: 10.0.1.0/24

2. Insure your VPC is connected to and internet Gateway.

3. At this point having a VPC up and running we created an EC2 instance in the VPC that will act as our edge VPN device.

• A Windows Server 2012 R2 instance

• I used a t2.small instance but this could change depending on network performance needed.

• Ensure that Auto-assign Public IP is enabled.

Please ensure you pick the right VPC and Subnet then click review and launch

4. Once the instance is running associate the Elastic IP we created earlier to the Network interface of the instance. By clicking Elastic IPs in the VPC dashboard, selecting Associate Address in the actions menu, selecting the Network interface used by the Windows Server 2012 R2 instance we created earlier and click associate

5. Next, disable source/destination checking on the server.

6. Once the Elastic IP is associated connect to your instance over RDP. We are now ready to configure the instance to connect to Azure.

Configure The Windows 2012 R2 Server RRAS server

1. Copy the content of the script located here https://raw.githubusercontent.com/Azure/Azure-vpn-config-samples/master/Microsoft/microsoft-rras-windows-server-2012-r2.ps1.xslt we will use that 

PowerShell script to install RRAS on our server and configure it.  

On your AWS windows 2012 R2 instance RDP session. Login as administrator, Open the PowerShell ISE and paste the content of the script you copied above in the script pane of the ISE.

Modify as shown below. (red = delete, Green = add\modify) and execute the script.

<?xml version="1.0" encoding="utf-16"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:output method="text"/>
  <xsl:template match="/"># Microsoft Corporation
# Windows Azure Virtual Network

# This configuration template applies to Microsoft RRAS running on Windows Server 2012 R2.
# It configures an IPSec VPN tunnel connecting your on-premise VPN device with the Azure gateway.

# !!! Please notice that we have the following restrictions in our support for RRAS:
# !!! 1. Only IKEv2 is currently supported
# !!! 2. Only route-based VPN configuration is supported.
# !!! 3. Admin priveleges are required in order to run this script

Function Invoke-WindowsApi( 
    [string] $dllName,  
    [Type] $returnType,  
    [string] $methodName, 
    [Type[]] $parameterTypes, 
    [Object[]] $parameters 
    )
{
  ## Begin to build the dynamic assembly 
  $domain = [AppDomain]::CurrentDomain 
  $name = New-Object Reflection.AssemblyName 'PInvokeAssembly' 
  $assembly = $domain.DefineDynamicAssembly($name, 'Run') 
  $module = $assembly.DefineDynamicModule('PInvokeModule') 
  $type = $module.DefineType('PInvokeType', "Public,BeforeFieldInit") 

  $inputParameters = @() 

  for($counter = 1; $counter -le $parameterTypes.Length; $counter++) 
  { 
     $inputParameters += $parameters[$counter - 1] 
  } 

  $method = $type.DefineMethod($methodName, 'Public,HideBySig,Static,PinvokeImpl',$returnType, $parameterTypes) 

  ## Apply the P/Invoke constructor 
  $ctor = [Runtime.InteropServices.DllImportAttribute].GetConstructor([string]) 
  $attr = New-Object Reflection.Emit.CustomAttributeBuilder $ctor, $dllName 
  $method.SetCustomAttribute($attr) 

  ## Create the temporary type, and invoke the method. 
  $realType = $type.CreateType() 

  $ret = $realType.InvokeMember($methodName, 'Public,Static,InvokeMethod', $null, $null, $inputParameters) 

  return $ret
}

Function Set-PrivateProfileString( 
    $file, 
    $category, 
    $key, 
    $value) 
{
  ## Prepare the parameter types and parameter values for the Invoke-WindowsApi script 
  $parameterTypes = [string], [string], [string], [string] 
  $parameters = [string] $category, [string] $key, [string] $value, [string] $file 

  ## Invoke the API 
  [void] (Invoke-WindowsApi "kernel32.dll" ([UInt32]) "WritePrivateProfileString" $parameterTypes $parameters)
}

# Install RRAS role
Import-Module ServerManager
Install-WindowsFeature RemoteAccess -IncludeManagementTools
Add-WindowsFeature -name Routing -IncludeManagementTools

# !!! NOTE: A reboot of the machine might be required here after which the script can be executed again.

# Install S2S VPN
Import-Module RemoteAccess
if ((Get-RemoteAccess).VpnS2SStatus -ne "Installed")
{
  Install-RemoteAccess -VpnType VpnS2S
}

# Add and configure S2S VPN interface

Add-VpnS2SInterface -Protocol IKEv2 -AuthenticationMethod PSKOnly -NumberOfTries 3 -ResponderAuthenticationMethod PSKOnly
 -Name <IP address of your Azure gateway> -Destination <IP address of your Azure gateway> -IPv4Subnet @("<IP range of your subnet in Azure>:100")
 -SharedSecret <shared key>

Set-VpnServerIPsecConfiguration -EncryptionType MaximumEncryption

Set-VpnS2Sinterface -Name <IP address of -InitiateConfigPayload $false -Force

# Set S2S VPN connection to be persistent by editing the router.pbk file (required admin priveleges)
Set-PrivateProfileString $env:windir\System32\ras\router.pbk “<IP address of your Azure gateway>” "IdleDisconnectSeconds" "0"
Set-PrivateProfileString $env:windir\System32\ras\router.pbk "<IP address of your Azure gateway>" "RedialOnLinkFailure" "1"

# Restart the RRAS service
Restart-Service RemoteAccess

# Dial-in to Azure gateway
Connect-VpnS2SInterface -Name <IP address of your Azure gateway>

</xsl:template>
</xsl:stylesheet>

2. Once the script finishes. You may have to run the following command in the powershell console

Restart-Service RemoteAccess

3. And then this one

Connect-VpnS2SInterface -Name <IP address of your Azure gateway>

4. Close PowerShell and open the Routing and Remote Access console and verify that the connection is complete

5. Go back to your AWS Management Console to Configure Routing.

In the Amazon management console switch to the VPC view and select Route tables. Select the route table associated with your VPC and add a new route to the 172.16.0.0/16 (Windows Azure Network) and that routes traffic through the instance ID of the Server.

6. And finally, create a DHCP Option set that will point the machines in the AWS network to the DNS on the Azure network. That way you will be able to resolve machines.

now that the network and the VPN are complete, we created some other VMs on each side (Ubuntu servers in this case) to validate routing, and connectivity.

That’s it, we now have a site to site VPN between our AWS and Azure environment.  and we’re ready to start moving our workloads over.

Cheers!

Hope this will help you!

Please Remember me in your prayers!

Enjoy 

Subnetting your VPC

To understand the main difference between SG and ACL one must first understand the concept of a subnet, and how a VPC is divided into subnets. A subnet is a logical division of the IP address range into smaller logical networks. When you are creating a new VPC, you will be asked to specify a CIDR block. To create a new VPC, just go to the VPC Dashboard on the AWS console, select Your VPCs (1) and press on Create VPC (2):

In the popup window, I have selected a CIDR block of 10.0.0.0/16 (1) for a VPC named VPC Test (2), which we will use during this post. Press on Yes, Create (3) to create the VPC:

To create a new subnet select Subnets (1) and press on Create Subnet (2):

When creating a new subnet, the CIDR block can be further divided into smaller chunks. In the following example, I have created a subnet named Test Subnet with a CIDR block of 10.0.1.0/24 (1) which supports 256 IP addresses and resides in the Test VPC (2). You can also select a preferred
Availability Zone (3):

Each subnet inside a VPC is connected to a router which routes the traffic between the different subnets in the VPC – this is how you get connectivity between different subnets inside the VPC. For EC2 instances on the same subnet, no routing is needed. By now, you are probably asking yourself – but how do I connect to the Internet? This is done by connecting the router to an Internet Gateway. To create an Internet Gateway select Internet Gateways (1) from your VPC Dashboard and press on Create Internet Gateway (2):

In the popup window give a Name tag (1) to the internet gateway and press on Yes, Create (2):

The last thing to do will be to attach the new Internet Gateway to our new VPC. On the VPC Dashboard select Internet Gateways (1) and search for the Internet Gateway you would like to attach (2). Select the Internet Gateway (3) and press on Attach to VPC (4):

In the popup window select the VPC (1) and press Yes, Attach (2):

Whenever you create a new VPC, AWS will automatically connect a router to the VPC and create a default route table. Each subnet that you add to that VPC will be linked to this route table by default. Let’s take a look at the routes of the Route Table (1) of our subnet:

The route table has only one route, which sends all traffic with destination 10.0.0.0/16 to the local target. This rule will enable the different subnets of the VPC to communicate with each other. Our subnet and any other subnets added to the VPC with the current default route table will have no access to or from the Internet. This kind of subnets is called Private Subnets.
We will need to modify the route table in order to grant our subnet access to the Internet. On the VPC Dashboard select Route Tables (1), find the Route Table (2) and select it (3). Select the Routes tab (4) and press on Edit (5):

Press on the Add another route Button (1) which had appeared. Add 0.0.0.0/0 (which means any destination) to the Destination and select the Internet Gateway that we have created (2). To finish, press on Save (3):

With the Internet Gateway attached our subnet can now access the Internet. This kind of subnets is called Public Subnets. The following diagram illustrates the network setup that we have created:

Routes, subnets, and CIDR blocks can be a bit tricky. If you want to know more, just search it, as there are a lot of websites devoted to the subjects. For you book lovers, I can suggest this book, where you can find a good explanation of the subjects.

Secure Those Instances!

Some of the parameters we are asked to specify when launching a new EC2 instance is the VPC, the subnet and the Security Groups (SG) for the instance. We have already created a VPC and a subnet, so now it’s time to create an SG. On the EC2 Dashboard select Security Groups (1) and press on Create Security Group (2):

In the popup window give the SG a name (1), a description (2) and select the VPC (3). Below you will notice two tabs for the SG rules (4), Inbound and Outbound. On the Inbound tab, you will notice that there are no inbound rules. Press on the Add Role for the Inbound tab (5):

Inbound traffic means traffic coming into the instance. The origin of the traffic can be an instance on the same subnet, on a different subnet on the same VPC, or a server on the other side of the world.
By default, a new SG is created with no inbound traffic allowed. When setting a new rule, we must specify the Type (1), Protocol (2), Port Range (3) and Source (4). In the following example I have permitted inbound access for the SSH protocol only from My IP address (the public IP address I currently use to connect to the AWS console, not shown), and to the HTTP and HTTPS protocols from Anywhere (that is what the 0.0.0.0/0 means):

Outbound traffic means traffic going out from the instance to a specific destination. Let’s take a look at the Outbound tab. By default, a new SG will permit any traffic going outbound from the instance (1). If you are in a need to prevent outbound traffic, except for specific ports or protocols, delete the “any” rule and add your own rules. In order to finish and create the SG just press on Create (2):

The most important point to note here is that SG is instance specific. You associate an SG to an instance or to a group of instances, but not to a subnet or a VPC altogether.
If you ever worked with ACL on a firewall before, then you may be asking yourself a few questions – What is the order at which the SG inbound and outbound rules are evaluated, does the SG uses stateful or stateless filtering and where are my deny rules?
In an ACL (and, as we shell see, with AWS ACL also) each rule is numbered. When a packet arrives at the firewall, it gets evaluated against the rules of the ACL starting with the rule with the lowest number. When a match is found, the rule is enforced and the packet is permitted or dropped. The rules with higher numbering than the matched rule are not evaluated. If the packet doesn’t match a rule, the next rule with a higher number is evaluated and so forth.
Security Groups operate in a different way. As we saw, there are no numbers to the SG rules. The rules are evaluated as a whole, no rule has precedence over other rules.
When an EC2 instance initiates a connection to some server, its outbound traffic will be evaluated against the outbound SG rules. If the rules permit it, the traffic will flow to the destination. But what happens when the remote server response?  The inbound traffic should be evaluated against the inbound SG rules, but the remote server IP address and port are not always known beforehand. If we don’t know the remote server IP address and port, how can we create the proper inbound rules?  The answer is that we don’t need to.
Security Groups implement stateful filtering. With stateful filtering, the inbound traffic originating from the remote server, as a result of the new connection initiated by our EC2 instance, will be automatically permitted even though there are no specific inbound rules.  In the same manner, for a connection initiated from a remote server, if the inbound traffic is permitted to pass to our EC2 instance, the outbound traffic from our EC2 instance will be automatically permitted even though there are no specific outbound rules. As we shall see, ACL are using stateless filtering.
And no, there are no deny rules when using SG, only permit ones. All traffic flowing inbound or outbound should much to the evaluated SG rules or it will be dropped.  With most cases, it is enough, though it can be painful at times. For example, imagine you want to permit all traffic from a specific location except from a specific protocol. In contrast, ACL have allowed and deny rules.
Any EC2 instance that is launched must have at least one SG associated to it. For each VPC you create, a default SG will be created for you. Let’s take a look at the default SG created for our VPC:

The Group ID of the SG is sg-dd3900b8 (1) and the Description states “default VPC security group” (2). Take a look at the Inbound Rules (3). The only rule that is specified permits all traffic from sg-dd3900b8. But wait, this is the ID of this SG. This is an example of a very handy feature. This specific rule will permit any traffic between any two EC2 instances which are associated with this SG. Therefore, the VPC’s default SG permits any traffic between any instances launched into the VPC that were associated to the default SG.

Secure Those Subnets!

Let’s take a look at the Network ACL (1) of our Test subnet:
This ACL has Inbound (2) and Outbound rules (3). Both the inbound and outbound rules are the same, they both permit any traffic from any source (rule number 100) and have an implicit “deny all” at the end (the rule marked with an asterisk, will get back to it later). I don’t remember creating this ACL when creating the subnet, so where did it come from?
When you create a new Subnet, a default ACL will be associated with the subnet. The ACL that we saw in the previous image is an example of a default ACL that was associated with our Test subnet.
Default ACL have no restriction on the inbound and outbound traffic so you can keep using SG without even knowing of ACL existence. On the other hand, a default SG will permit inbound traffic only from within the VPC and drop any other traffic. Both default SG and ACL permit any traffic in the outbound direction.
This brings us to an important difference between ACL and SG, the scope of there influence. As we saw, an SG is associated to an EC2 instance and defines the allowed inbound and outbound traffic for that instance. An ACL, on the other hand, is associated with a subnet. A subnet can host one or more EC2 instances, and all of them will be affected by the inbound and outbound rules defined in the ACL of that subnet. For example, if we should create an ACL inbound rule that blocks SSH traffic from a specific IP address. All the EC2 instances on the subnet associated with that ACL will be blocked for SSH access from that specific IP address.
To create a new ACL, go to the VPC Dashboard, press on Network ACLs (1) and then on Create Network ACL (2):

In the popup window, set a Name tag (1) for the ACL, select the relevant VPC (2) and press on Yes, Create (3):

If we will look at the Inbound Rules (1) of the new ACL, we will see that all inbound traffic is blocked (2):

The same goes with the Outbound Rules (1) where all the outbound traffic is blocked as well (2):

As with a new SG, a new ACL will also block inbound traffic by default. However, while a new SG permits any outbound traffic, a new ACL blocks it by default.
Now, let’s add an inbound ACL rule which will permit SSH access from the 1.2.3.4 IP address. On the VPC Dashboard select Network ACLs (1) find the relevant ACL (2), select the Inbound Rules tab (3) and press on Edit (4):

I have set the rule number to 100 (1), the protocol to SSH (2), which automatically have set the protocol to TCP and the port range to 22 (3). I have set the IP address to 1.2.3.4/32 (4, the “/32” notation means that it is a host IP address) and allowed the access (5). To finish press on Save (6):

The updated ACL now has two rules. The first rule numbered 100 is the rule we just created (1). The second rule, marked with an asterisk, is the implicit “deny all” rule (2):

The implicit “deny all” rule will always be evaluated last. As was stated before, any packet arriving inbound or outbound will be evaluated against the relevant ACL rules. In the end, if the packet does not match any of the rules, it will hit the implicit “deny all” rule and will be dropped. This ensures that only packets with a perfect match to an ACL rule (like a specific source IP address) will be permitted to pass.
Presume that we have launched a new EC2 instance to a subnet associated with our new ACL, gave it a public IP address, and configure an SG which would allow access to the instance on the SSH protocol from the 1.2.3.4 IP address. Now, will a client with an IP address of 1.2.3.4 would be able to connect to the SSH port? The answer is no. To understand why we need to look at the Outbound rules (1) of our Test ACL. The only rule there is the implicit “deny all” rule (2):

The SSH connection will not work because ACL use stateless filtering. The inbound traffic from the remote server, accessing the EC2 instance on port 22/TCP, will get through because both the subnet ACL and the instance SG will permit it to pass. However, when the instance will try to respond, the SG will let the traffic pass, because it uses stateful filtering, but the ACL will drop the traffic because it uses stateless filtering and has no outbound rules for the remote server. To mitigate this problem we can add an outbound rule. I have added a custom outbound TCP rule number 100 (1) which allows a range of TCP ports (2) to the 1.2.3.4 IP address:

So, why allow outbound traffic to a range of TCP ports and not to a specific port like we did with the inbound rule? To answer this question we need to understand the concept of ephemeral and well-known ports.
When a client initiates a connection to a server he needs to know the destination server IP address and port number (let’s use HTTP port 80 as an example). With these two parameters, the client will create a pair of TCP sockets. One TCP socket will be Server-IP-Address: TCP-Port, or in our example, Server-IP-Address:80. The second TCP socket will be the Client-IP-Address: Ephemeral-Port. The client shares these two sockets with the server (via the IP and TCP headers).
The Server’s TCP port (80 in our example) is the well-known port for the HTTP protocol. The well-known ports range from 0 to 1023 and are associated with well-known protocols (22 for SSH, 443 for HTTPS). When the client knows that it need to access the HTTP protocol on the remote server he will use port 80/TCP by default. For that reason, we can always open specific port numbers inbound to a server when we know exactly which services are running on that server (this is what we did with port 22/TCP when we configured the inbound rules for the Test ACL).
But why does the client needs a port number? The traffic from the server reaches the client because the client had given the server its IP address. The packets arriving at the client must be directed to the correct application that initiated the connection. This is done by associated the port number to that application (just like associating port 22 to the SSH service on the server). Moreover, this enables both the client and the server to have multiple connections with the same well-known port.
When the client initiates the connection, he chooses a random port number from a range of ports named ephemeral ports and creates the socket. They are called ephemeral or dynamic ports because whenever the connection is terminated they are released, and can be reused for other connections afterwords. Linux kernels usually use the port range 32768-61000. This exact port range was used for the outbound rule configured for our Test ACL.
In situations where we do not know the client IP address beforehand (like for a web server on the Internet), we will need to create an ACL outbound rule that will permit the ephemeral port range to any IP address (1):

One last thing left to do is to associate our Test ACL with the Test Subnet. Open the VPC Dashboard, select Subnets (1) and search for the subnet (2). Select the subnet and press on Edit (3):

In the Change to a drop-down box (1) select the Test ACL and press on Save (2):

What is more secured?

Basically, ACL and SG do the same thing, they filter traffic. ACL adds a level of security by ensuring that each EC2 instance launched into the associated subnet is enforced by the rules specified in the ACL, even if the right SG where not configured. On the other hand, because of their stateless nature, there are more difficult to configure at times.

I usually use SG for the less secure, mostly internet facing instances. ACL are used for a very secure instance, like databases, where I want another layer of security and the clients accessing these instances are well known.

 How can I resolve Amazon Route 53 private hosted zones from an on-premises network via an  Ubuntu instance?

 Resolution

You can resolve domain names in private hosted zones from your on-premises network by configuring a DNS forwarder. The following instructions assume that your on-premises network is configured with a VPN or AWS Direct Connect to an AWS VPC, and a Route 53 private hosted zone is associated with that VPC.

Configure a DNS forwarder by completing the following steps:

1.    Ensure that DNS resolution and DNS hostnames are enabled on the target VPC.
Note: DNS hostnames are enabled for default VPCs and VPCs that you create using the VPC wizard in the Amazon VPC console.

2.    Install BIND DNS server on your instance by using the following command:

      sudo apt-get install bind9 bind9utils bind9-doc

3.    Configure the BIND server in a forward-only configuration by modifying the named.conf.options file. By default, BIND configuration files are kept at /etc/bind.

a.    Create an access control list (ACL) for the BIND server, replacing the IP address in the following example with a list of IP addresses you trust:

    acl "trusted" {

              172.16.0.0/16

              localhost;

              localnets;

  };

b.    Configure BIND to forward all DNS requests to the Amazon VPC name server. The VPC name server is always the second available address in the VPC. For example, if the VPC CIDR is 10.10.0.0/16, then the VPC name server has the IP address of 10.10.0.2, as in the following example:

options {

                directory "/var/cache/bind";

                recursion yes;

                allow-query { trusted; };

                 forwarders {

                            10.10.0.2;

                };



                forward only;

                dnssec-enable no;

                dnssec-validation no;

                dnssec-lookaside auto;

                auth-nxdomain no;

                listen-on-v6 { any; };

};

Note: In the above example, DNSSEC is disabled, because Route 53 does not support DNSSEC at this time.

4.    Test the syntax and restart the service by using the following commands:

    sudo named-checkconf 

    sudo service bind9 restart

Note: Ensure you have port 53 TCP/UDP open to the on-premises network in your DNS server’s security group.
Configure your clients to use the BIND DNS server to resolve DNS. For instructions, check the documentation for your client’s operating system.

Keywords

private hosted zone, name resolution, BIND, DNS forwarder

Related Information

Using DNS with Your VPC
Creating a Private Hosted Zone

This article will help you to enable swap filesystem in your running instance. There are two methods to add swap in Amazon ec2 linux instances. This article has been tested with CentOS 6.5 only, but it can work on most of Linux distributions

Method 1: By Creating File in Current File System

This option is helpful if we don’t want to add extra disks in our systems, In this we simply create a file in our current file system and make it type swap, which can be used as swap in our system. Use following commands to create and enable swap on our system.

# dd if=/dev/zero of=/var/myswap bs=1M count=2048
# mkswap /var/myswap
# swapon /var/myswap

bs=1M count=2048 means it will create 2GB of swap file, You may change as per you need. After enabling swap we can see that our system has swap enabled by running “free -m” command.

To make it enable on system boot, simply edit /etc/fstab file and add following entry at end of file.

/var/myswap   swap   swap   defaults  0 0

Method 2: By Adding new Disk in System

This option is helpful if you do not have enough space in our current drives mounted in system. In this option, first we need to add extra disk in our system first. In my case new disk mounted as /dev/xvdd (It may change in your case)

# mkswap -f /dev/xvdd
# swapon /dev/xvdd

To make it enable on system boot, simply edit /etc/fstab file and add following entry at end of file.

/dev/xvdd   swap   swap   defaults  0 0

Hope this will help you!

Please Remember me in your prayers!

Enjoy 

1. Install EPEL6 Repo
~]# wget http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
    or 

~]# wget  http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

~]# rpm -ivh epel-release-6-8.noarch.rpm

2. Install the dracut & cloud-utils growpart modules-growroot
~]# yum install -y dracut-modules-growroot cloud-utils-growpart

  or 

~]# yum -y --quiet install dracut-modules-growroot
~]# dracut -v --force --add growroot /boot/initramfs-$(uname -r).img
~]# reboot
3. The initrd needs to be rebuilt after installing the dracut-modules-growroot.
Create a backup copy of the current initramfs:

~]# cp -p /boot/initramfs-$(uname -r).img /boot/initramfs-$(uname -r).img.bak

4. Create the initramfs for the current kernel:
~]# dracut -f

5. Result after reboot
~]# reboot

extract the S3Sync.zip folder to C drive.

Location of S3sync folder = C:\S3Sync

4: Write a backup script

Create a batch file and paste this code into it:

cd C:\S3Sync
S3Sync.exe -AWSAccessKeyId xxxxxxx -AWSSecretAccessKey xxxxxxx -SyncDirection upload -LocalFolderPath "C:\inetpub\wwwroot" -BucketName YOURBUCKETNAME


The code above is pretty self-explanatory. Just replace the "xxxxxx" with your access codes from #2, "YOURBUCKETNAME" with the name of your S3 bucket, and "C:\inetpub\wwwroot" - with the folder you want to backup. Then create a scheduled task that runs the batch file every 24 hours, and you're all set.

Hope this will help you!

Please Remember me in your prayers!

Enjoy 

Consider a use case where you have 20 instances and you want to deploy your code or change the configuration file of these instances. The only solution would be to login into each particular instance and then changing the configuration file. AWS CodeDeploy lets you do this in just few steps . You just create a deploy application and your code will be deployed in all these 20 instances.

Deploying code without using AWS CodeDeploy

Deploying code using AWS CodeDeploy

There are two ways to deploy code in Amazon Web Services:-

• Using GIT 

• Using AWS S3 (Simple Storage Service)

Here, we will deploy the code using Amazon S3 service. Let us also understand few useful terms which will be used in the deployment process:

• AppSpec file:- It is an Application Specification file. It is a unique file that defines a series of deployment actions that you want CodeDeploy to execute.

• Deployment Application:- The unique name which will be given to your Deployment Application.

• Revision:- It is a combination of AppSpec file and other files such as scripts, images, index files, media etc.

• Deployment Group:- It is defined as a group of individual instances and auto-scaled instances.

• Deployment Configuration:- It lets you side that how you want your code to be deployed:- one at a time/ half at a time/ all at once.

Deploying Code Using AWS S3

We’ll take a simple example to deploy the code using S3. We are deploying the code in a single instance and are launching a single t2.micro instance. Launch the instance and install nginx in it as we are going to change the front page or index.html of the nginx default configuration. You can install nginx by logging into the instance and typing the following commands: 

sudo apt-get update 

sudo apt-get install nginx -y

Now lets move towards Code Deploy into an instance

Before starting with CodeDeploy, we need to have :-

• Two IAM ROLES : one role will be given to EC2-instances to access s3 buckets and the other role is given to CodeDeploy service to choose Ec2-instances based on their tags.

• One S3 bucket containing the appspec file, scripts and other files into a tar,gz or bz2 file (compressed format file). You need to store the compressed file into the S3 bucket. The files will automatically be uncompressed at the time of Deployment.

IAM Role Given to AWS CodeDeploy to access your EC2-instance:

=======================================================================

{

"Version": "2012-10-17",

 "Statement": [

    {

      "Action": [

         "autoscaling:PutLifecycleHook",

         "autoscaling:DeleteLifecycleHook",

         "autoscaling:RecordLifecycleActionHeartbeat",

         "autoscaling:CompleteLifecycleAction",

         "autoscaling:DescribeAutoscalingGroups",

         "autoscaling:PutInstanceInStandby",

         "autoscaling:PutInstanceInService",

         "ec2:Describe*"

                 ],

  "Effect": "Allow",

  "Resource": "*"

    }

               ]

 }

=======================================================================

IAM Role Given to EC2-instances to access S3 Buckets

=======================================================================

{

"Version": "2012-10-17",

"Statement":

     [

         {

             "Action":

                   [

                     "s3:Get*",

                     "s3:List*"

                   ],

             "Effect": "Allow",

             "Resource": "*"

         }

     ]

}

=======================================================================

Trusted Relationship With AWS CodeDeploy IAM Role

{

“Version”: “2012-10-17″,

“Statement”: [

    {

    “Sid”: “”,

    “Effect”: “Allow”,

    “Principal”: {

    “Service”:

           [

        "codedeploy.us-east-1.amazonaws.com",

        "codedeploy.us-west-2.amazonaws.com"

           ]

                 },

    “Action”: “sts:AssumeRole”

    }

             ]

}

We also need to install AWS CodeDeploy Client to our instance. It will allow the code to be deployed into the instance. You can install the code-deploy client onto your instance by the following process:

Installing AWS CLI and AWS CodeDeploy Agent on Ubuntu 14.04 LTS :

sudo apt-get update

sudo apt-get install awscli

sudo apt-get install ruby2.0

cd /home/ubuntu

sudo aws s3 cp s3://aws-codedeploy-us-east-1/latest/install . --region us-east-1

sudo chmod +x ./install

sudo ./install auto

Understanding APPSPEC FILE

AppSpec is the heart of CodeDeploy and is written in YAML. AppSpec defines how the application code will be deployed on deployment targets and which deployment lifecycle event hooks to run in response to various deployment lifecycle events. It should be in the root of an application source code’s directory structure.

High Level Structure of AppSpec File:

1                              version: 0.0

2                   os: operating-system-name

3          files: source-destination-files-mappings

4          permissions: permissions-specifications

5      hooks: deployment-lifecycle-event-mappings

Hooks: scripts to run at specific deployment lifecycle events during the deployment. The available event hooks are:

ApplicationStop: events to be performed when application is stopped

DownloadBundle: occurs when CodeDeploy agent downloads bundle from S3 bucket

BeforeInstall: occurs before AWSCodeDeploy starts deployment of application code to deployment target

Install: AWSCodeDeploy copies files to deployment targets

AfterInstall: occurs once files are copied and installed to deployment targets

ApplicationStart: occurs just before your application revision is started on the deployment target

ValidateService: occurs after the service has been validated

The sample AppSpec file used is as shown below:

version: 0.0

os: linux

files:

- source: /

destination: /usr/share/nginx/html

hooks:

BeforeInstall:

- location: scripts/install_dependencies.sh

timeout: 300

runas: root

AfterInstall:

- location: scripts/afterinstall

timeout: 300

runas: root

ApplicationStart:

- location: scripts/start_server

timeout: 300

runas: root

ApplicationStop:

- location: scripts/stop_server

timeout: 300

runas: root

While creating an instance you need to attach the s3 bucket role with your instance and after that you need to install AWS CLI and AWS Code Deploy Agent using the above procedure. Now you are ready to create the CodeDeploy Application.

Creating AWS CodeDeploy Application

Sign in to the AWS Console . Go to the services and click on “Code Deploy”  as shown below.

Now a new window will open as shown below. Click on “Create New Application” button. It will open up the prompt to create a new application.

A new window will appear which ask about the details for creating an application.  Enter Application Name, Application Group Name and choose instances to which you want to deploy the code using the Key and Value. Choose your Deploy Configuration :-  One at a time /Half at a time /All at a time . This configuration lets you choose how you want to deploy your code.

Enter the application name and Application Group Name.

Choose instances based om their Key and Value

Then Click on “CREATE APPLICATION ” button. Your application will be created and new window will appear as shown below.  

You have to create a new revision. Click on Deploy New Revision button to create a new revision.

Now enter the  Application Name, Deployment Group Name. Choose Revision type :- “My application is stored in Amazon S3.”. Give the Revision Location i.e. location of Bucket and the file name. (You can also copy the full path of file from AWS S3 and paste it here). After entering all the details, click on Deploy Now. Now your application and code is being deployed. Wait for few seconds and then refresh.

The status will appear as Succeeded. You can now hit the IP of your instance and you will get the index page that you deployed.

Hope this will help you!

Please Remember me in your prayers!

Enjoy 

• Setup your new User and Permissions (and Roles if needed).

• From within Route 53 copy the Hosted Zone ID for the domain you want to allow access.

• From the IAM dashboard Create a new policy:

• Change the Hosted zone ID with your hosted zone ID which you want to restrict.

{  

   "Version": "2012-10-17",

   "Statement":[

      {

         "Action":[

            "route53:ChangeResourceRecordSets",

            "route53:GetHostedZone",

            "route53:ListResourceRecordSets"

         ],

         "Effect":"Allow",

         "Resource":[

            "arn:aws:route53:::hostedzone/<Your zone ID>"

         ]

      },

      {

         "Action":[

            "route53:ListHostedZones"

         ],

         "Effect":"Allow",

         "Resource":[

            "*"

         ]

      }

   ]

}

Hope this will help you!

Please Remember me in your prayers!

Enjoy 

Codedeploy is one of the deployment service by AWS. The application can be deployed using either a s3 bucket or a git repository which contains the deployable content like code, scripts, configurations files, executables etc.

In this blog post, we are going to deploy a wordpress application in an elastic, highly available and scalable environment using codedeploy.

Get things ready

Get a copy of the WordPress source code in the local system using git command:

git clone https://github.com/WordPress/WordPress.git /tmp/WordPress  

Create Scripts to run your Application. Make a directory .scripts in the WordPress folder:

mkdir -p /tmp/WordPress/.scripts 

Create the following shell scripts in the .scripts folder: sudo vim install_dependencies.sh:

#!/bin/bash
yum groupinstall -y "PHP Support"  
yum install -y php-mysql  
yum install -y nginx  
yum install -y php-fpm  

Next sudo vim stop_server.sh:

#!/bin/bash
isExistApp=`pgrep nginx`  
if [[ -n  \$isExistApp ]]; then  
   service nginx stop
fi  
isExistApp=`pgrep php-fpm`  
if [[ -n  \$isExistApp ]]; then  
    service php-fpm stop
fi  

one more, sudo vim start_server.sh:

#!/bin/bash
service nginx start  
service php-fpm start  

and finally, sudo vim change_permissions.sh:

#!/bin/bash
chmod -R 755 /var/www/WordPress  

Make these scripts executable with this command:

chmod +x /tmp/WordPress/.scripts/*  

CodeDeploy uses an AppSpec file which is a unique file that defines the deployment actions you want CodeDeploy to execute. So along with the above scripts, create a appspec.yml file
sudo vim appspec.yml

version: 0.0  
os: linux  
files:  
  - source: /
    destination: /var/www/WordPress
hooks:  
  BeforeInstall:
    - location: .scripts/install_dependencies.sh
      timeout: 300
      runas: root
  AfterInstall:
    - location: .scripts/change_permissions.sh
      timeout: 300
      runas: root
  ApplicationStart:
    - location: .scripts/start_server.sh
      timeout: 300
      runas: root
  ApplicationStop:
    - location: .scripts/stop_server.sh
      timeout: 300
      runas: root

Now zip the WordPress folder and push it to your git repository.

Creating IAM Roles

Create an iam instance profile and attach AmazonEC2FullAccess policy and also attach the following inline policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:Get*",
                "s3:List*"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

Create a service role CodeDeployServiceRole. Select Role type AWS CodeDeploy. Attach the Policy AWSCodeDeployRole as shown in the below screenshots:

How about Scale?

Create an autoscaling group for a scalable environment. Steps below:

Choose an ami and select an instance type for it:

Attach the iam instance profile which we created in the earlier step:

Now go to Advanced Settings and type the following commands in “User Data” field to install codedeploy agent on your machine (if it’s not already installed on your ami):

#!/bin/bash
yum -y update  
yum install -y ruby  
yum install -y aws-cli  
sudo su -  
aws s3 cp s3://bucket-name/latest/install . --region region-name  
chmod +x ./install  
./install auto

where, bucket-name represents one of the following based on the instances in the specified region:

• aws-codedeploy-us-east-1

• aws-codedeploy-us-west-2

• aws-codedeploy-us-west-1

• aws-codedeploy-eu-west-1

• aws-codedeploy-eu-central-1

• aws-codedeploy-ap-southeast-1

• aws-codedeploy-ap-southeast-2

• aws-codedeploy-ap-northeast-1

• aws-codedeploy-ap-south-1

• aws-codedeploy-eu-west-2

• aws-codedeploy-ca-central-1

• aws-codedeploy-us-east-2

• aws-codedeploy-ap-northeast-2

• aws-codedeploy-sa-east-1

and region-name will be one of the following:

• us-east-1

• us-west-2

• us-west-1

• eu-west-1

• eu-central-1

• ap-southeast-1

• ap-southeast-2

• ap-northeast-1

• ap-south-1

• eu-west-2

• ca-central-1

• us-east-2

• ap-northeast-2

• sa-east-1

Select Security Group in the next step and create the launch configuration for the autoscaling group. Now using the launch configuration created in the above step, create an Autoscaling group.

Select the launch configuration from the given options:

Give the name of the group in the next screen and select a subnet for it.

Keep the remaining settings at its default and create the autoscaling group.

Time to Deploy

Choose Create New Application. Give some name for the application and a name for the deployment group as well.

Select Autoscaling Group in Search By Tags field to deploy the application on the group and select CodeDeployDefault.OneAtATime in the Deployment Config field.

Select ServiceRoleARN for the service role which we created in the “Creating IAM Roles” section of this post. Go to Deployments and choose Create New Deployment. Select Application and Deployment Group and select the revision type for your source code (i.e. an S3 bucket or a GitHub repository).

On the successful deployment of the application, something like this will appear on the screen:

The WordPress is now deployed on the AutoScaling Group. So when you hit the public IP of the instance which belongs to the autoscaling group, nginx test page will load.

Configurring WordPress 

Since nginx needs php-fpm to work with php pages, we need to configure php-fpm. Also we need to configure WordPress script as well. For this we need to do certain changes in the files as shown below:

sudo vim /etc/php.ini  

Uncomment cgi.fix_pathinfo=0 and change the value from 1 to 0.

sudo vim /etc/php-fpm.d/www.conf

Change user=nginx and group=nginx and also make sure the following values are uncommented:

  pm.min_spare_servers = 5
  pm.max_spare_servers = 35

Add this following script to the configuration file sudo vim /etc/nginx/conf.d/virtual.conf

server {  
listen 80;  
server_name example.com;  
location / {  
    root /var/www/WordPress;
    index index.php index.html index.htm; 
    if (-f $request_filename) {
    expires 30d;
    break;
    }
    if (!-e $request_filename) {
    rewrite ^(.+)$ /index.php?q=$1 last;
    }
    } 
location ~ .php$ {  
    fastcgi_pass   localhost:9000;  #port where FastCGI processes were spawned
    fastcgi_index  index.php;
    fastcgi_param SCRIPT_FILENAME   
    /var/www/WordPress$fastcgi_script_name; #same path as above
fastcgi_param PATH_INFO $fastcgi_script_name;  
include /etc/nginx/fastcgi_params;  
}
}

Hit the server name on the browser and It will load the WordPress Application.To avoid this manual work of configuring the application for other instances in the AutoScaling Group, you can create an image of the instance in which you have done these changes and provide the ami of the created image to the Launch Configuration and update the Launch Configuration in the AutoScaling Group. Hence, the new instances will be created with the updated image.

After the successful installation, the wordpress dashboard will appear as shown in the below screenshot:

Make It Stateless 

If you would like to scale at will and deploy at will, you need to make sure that the web/app is stateless. Make sure that you manage plugins in github repo and static content is stored outside the server, on S3.

To store the static media content of your WordPress Application in an S3 Bucket, we will need a plugin named WP Offload S3.
This plugin automatically copies the media files uploaded by WordPress into an S3 bucket. But this plugin has a dependency on another plugin, Amazon Web Services

So, after downloading the both plugins, we got the two zip files of these plugins now. Unzip these files to WordPress/wp-content/plugins path. If not already done, zip the WordPress folder again, push it to the git repository and redeploy the application through CodeDeploy using the CommitID of the latest commit.

Go to plugins, the two plugins(Amazon Web Services and WP Offload S3) will be shown. Activate these two plugins. Also, after activating the Amazon Web Services plugin, AWS console will be added to the left bar. Go to AWS and define your Access keys and Secret keys in the wp-config.php.

After activating the WP Offload S3, go to its Settings and enter the name of the bucket in which you want to store the media contents of your blog posts. Save the settings.

Now try posting some media content in your blog post.

A folder wp-content will be created in the S3 bucket and the content will get stored in the same folder.

Let there be a loadbalancer 

We are now almost done. In order to achieve the 'highly available' part of our initial goal, lets create a loadbalancer :)

Create an Elastic Load Balancer for high availability of your application. Give it a name.

Select a security group for it in the next screen and configure the health checks:

Review and Create.
Now, Attach this ELB with the autoscaling group:

Also, to access the application through the ELB endpoint, add the public DNS of the ELB to the server_name in /etc/nginx/conf.d/virtual.conf.

Happy CodeDeploy-ing! :)

Hope this will help you!

Please Remember me in your prayers!

Enjoy 



Apache configuration has a major affect on your Linode’s performance. The easiest way to increase server performance is to turn off unneeded modules. This guide covers Apache modules, information on which modules to turn off, and other Apache performance tuning options.

The steps in this guide require root privileges. Be sure to run the steps below as root or with the sudo prefix.

Tools

There are a variety of tools that can assist in determining if you need to alter resource settings, including the top command and the load-testing program Siege. Siege’s own Longview service can also help in server monitoring. At minimum, familiarize yourself with the RAM and CPU usage of your server. Discover usage statistics with these commands:

echo [PID]  [MEM]  [PATH] &&  ps aux | awk '{print $2, $4, $11}' | sort -k2rn | head -n 20

ps -eo pcpu,pid,user,args | sort -k 1 -r | head -20

More specific resources for resource tuning Apache includes Apache mod_status and Apache2Buddy.

Apache mod_status

Apache mod_status diplays information related to incoming server connections by generating a detailed status page. View an example of this page at Apache’s own website.

Open your website’s configuration file. This file is located at /etc/apache2/sites-available/example.com.conf on Debian/Ubuntu systems or /etc/httpd/conf.d/vhost.conf on CentOS/Fedora systems.

Add the following to the <virtual_hosts> block:

/etc/apache2/sites-available/example.com.conf (Debian/Ubuntu)

/etc/httpd/conf.d/vhost.conf (CentOS/Fedora)

<Location /server-status>  SetHandler server-status  Order Deny,Allow  Deny from all  Allow from localhost</Location>

Apache mod_status also offers an option called ExtendedStatus, which provides additional information about each request made to Apache. To enable ExtendedStatus edit your Apache configuration file:

/etc/apache2/apache2.conf(Debian/Ubuntu)

/etc/httpd/confd/httpd.conf(CentOS/Fedora)

ExtendedStatus On

NOTE: Enabling ExtendedStatus consumes additional system resources.

Restart Apache:

Debian/Ubuntu:

service apache2 restart

CentOS/Fedora:

/bin/systemctl reload httpd.service

To view the file generated, download Lynx:

Debian/Ubuntu:

apt-get install lynx

Fedora/CentOS:

yum install lynx

Open the file:

lynx http://localhost/server-status

Apache2Buddy

The Apache2Buddy script, similar to MySQLTuner, reviews your Apache setup, and makes suggestions based on your Apache process memory and overall RAM. Although it is a fairly basic program, focusing on the MaxClients directive, Apache2Buddy is useful, and can be run through a single command:

curl -L http://apache2buddy.pl/ | perl

Multi Processing Modules

Before making any changes to your Apache configuration, be sure to back up the configuration file:

On Debian/Ubuntu:

cp /etc/apache2/apache2.config ~/apache2.conf.backup

On CentOS/Fedora:

cp /etc/httpd/conf/httpd.config ~/httpd.conf.backup

Apache offers two Multi-Processing Modules, three if on Apache 2.4, for managing your settings.

Each module creates child processes, differing primarily in how they handle threads.

Prefork

The prefork module creates a number of child processes at launch, each child handles only one thread. Since these processes deal solely with one thread at a time, making request speed suffer should there be too many concurrent requests. Should this occur, some requests essentially have to wait in line to be acted upon. To handle this, increase the number of child processes spawned, but be aware that this increases the amount of RAM being used. Prefork is the safest module and should be used when using mods that are not thread safe.

Worker

The worker module’s child processes spawn many threads per process, each thread ready to take on new requests. This allows for a greater number of concurrent requests to come in, and in turn is easier on the server’s RAM usage. Overall, the worker module offers higher performance, but is less secure than prefork and cannot be used with modules that are not thread safe.

Event

The event module is only available on Apache 2.4 and is based off the worker MPM. Like the worker, it creates multiple threads per child process, with a thread dedicated to KeepAlive connections that are handed down to child threads once the request has been made. This is good for multiple concurrent connections, especially those that are not all active at the same time but make the occasional request. The event MPM functions the same as worker in the event of SSL connections.

Module Values

Once you select your MPM, you will need to change the values inside the configuration. These settings are located in the /etc/apache2/apache2.conf file on Debian/Ubuntu, and the /etc/httpd/conf/httpd.conf file on CentOS/Fedora. The MPM looks like this:

/etc/apache2/apache2.conf (Debian/Ubuntu)

/etc/httpd/conf/httpd.conf (CentOS/Fedora)

<IfModule mpm_prefork_module>

StartServers                  4

MinSpareServers          20

MaxSpareServers         40

MaxClients                   200

MaxRequestsPerChild  4500

</IfModule>

For other MPMs replace <IfModule mpm_prefork_module>  with <IfModule mpm_worker_module>  or <IfModule mpm_event_module> for worker and event, respectively.

The next step to reconfiguring your Apache server is altering the above settings. To do this, you need to be aware of what each value does, and how best to change it.

Again, the best way to make configuration changes is to make incremental changes and then monitor the effects.

After making alterations to the Apache configuration, restart the service using service apache restart on Debian/Ubuntu or /bin/systemctl reload httpd.service on CentOS/Fedora.

StartServers

The StartServers value indicates the number of child processes created at startup, and is dynamically controlled depending on load. There is often little reason to alter this number, unless your server is restarted frequently and contains a large number of requests upon reboot.

MinSpareServers

Sets the minimum number of idle child processes. If there are fewer processes than the MinSpareServer number, more processes are created at the rate of one per second on Apache 2.2 or lower. With Apache 2.4, this rate increases exponentially starting with 1 and ending with 32 children spawned per second. The benefit of this value is that when a request comes in it can take an idle thread; should a thread not be available, Apache would have to spawn a new child, taking up resources and extending the time it takes for the request to go through. Note, too many idle processes would also have an adverse effect on the server.

MaxSpareServers

Sets the maximum number of idle child processes. If there are more idle processes than this number, then they are terminated. Unless your website is extremely busy, this number should not be set too high, since even idle processes consume resources.

MaxClients

The maximum amount of requests that can be served simultaneously, with any number going past the limit being queued. If this is set too low, connections sent to queue eventually time-out; however, if set too high, it causes the memory to start swapping. If this value is increased past 256, the ServerLimit value must also be increased.

One way to calculate the best value for this is to divide the amount of RAM each Apache process uses by the amount of RAM available, leaving some room for other processes. Use ApacheBuddy to help determine these values, or the commands below.

To determine the RAM each Apache process uses, replace httpd with  apache2 on Debian or Ubuntu systems:

ps -ylC httpd --sort:rss

Divide the number by 2048 for megabytes.

To get information on memory usage:

free -m

To receive a fuller view of the resources Apache is using, use the top command.

MaxRequestsPerChild

This limits the number of requests a child server handles during its life. Once the limit has been hit, the child server dies. If set to 0, the child servers are set to never expire. The suggested value for this is a few thousand, to prevent memory leakage. Be aware that setting this too low can slow down the system, since creating new processes does take up resources.

ServerLimit

If you need to increase the MaxClients above 256, then increase your ServerLimit to match. To do this, add the ServerLimit line to your MPM code and alter the value:

ServerLimit        256

KeepAlive

The KeepAlive directive, when set to on allows for multiple requests to come from the same TCP connection. When a KeepAlive connection is used, it counts as only one request against the MaxRequestsPerChild directive. This value is kept outside of your MPM, but can tie in closely to your MPM choices.

More Information

You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.

• Apache Performance Tuning

• Apache MPM Common Directives

• Apache 2.4 Documentation




This guide is for Debian or Ubuntu systems.

Download and Configure Siege

1. Prior to installing any new programs, update your system:

sudo apt-get update && sudo apt-get upgrade --show-upgraded

2. Download the latest version of Siege (3.0.9 at the time of this publication), which is always available at Siege’s website:

wget http://download.joedog.org/siege/siege-latest.tar.gz

3. Extract the program:

tar -zxvf siege-latest.tar.gz

4. Navigate to the Siege directory:

cd siege-*/

5. If the GNU Compiler Collection (gcc) is not installed, install now:

sudo apt-get install build-essential

6. Configure and complete the installation:

./configure

make

sudo make install

7. Generate a configuration file:

siege.config

8. Open the .siegerc file located in your home directory.

9. The suggested Siege configuration is for 25 concurrent users over a period of 1 minute. Set a location for your log file. Be sure to uncomment the variables shown below, and any other commented settings you want to use by removing the pound sign (#):

File Excerpt:- ~/.siegerc

...

     

#

# Variable declarations. You can set variables here

# for use in the directives below. Example:

# PROXY = proxy.joedog.org

# Reference variables inside ${} or $(), example:

# proxy-host = ${PROXY}

# You can also reference ENVIRONMENT variables without

# actually declaring them, example:

logfile = $(HOME)/siege.log


...


#

# Default number of simulated  concurrent users

# ex: concurrent = 25

#

concurrent = 25


#

# Default duration of the siege.  The right hand argument has

# a modifier which specifies the time units, H=hours, M=minutes,

# and S=seconds. If a modifier is not specified, then minutes

# are assumed.

# ex: time = 50M

#

time = 1M

You are now ready to run Siege!

Run Siege

To run Siege with the default settings, input the following command, replacingwww.example.com with your domain name or IP address:

siege www.example.com

Siege outputs the results:

** SIEGE 2.70

** Preparing 25 concurrent users for battle.

The server is now under siege...

Lifting the server siege...       done.

Transactions:                          2913 hits

Availability:                            100.00 %

Elapsed time:                          59.51 secs

Data transferred:                     0.41 MB

Response time:                       0.00 secs

Transaction rate:                     48.95 trans/sec

Throughput:                            0.01 MB/sec

Concurrency:                          0.04

Successful transactions:         2913

Failed transactions:                0

Longest transaction:               0.01

Shortest transaction:               0.00


FILE: /var/log/siege.log

You can disable this annoying message by editing

the .siegerc file in your home directory; change

the directive 'show-logfile' to false.

If there are no failed connections and the availability remains at 100%, there are no problems.

Further Configuring and Commands

Creating a URL File

If you want Siege to hit a number of pages on your website at random, configure the program to read from a urls.txt file that lists the selected pages.

1. Open the urls.txt file generally created at /usr/local/etc/urls.txt. Add a list of URLs or IP addresses to that file:

File:- /usr/local/etc/urls.txt

# URLS file for siege

# --

# Format the url entries in any of the following formats:

# http://www.whoohoo.com/index.html

# http://www/index.html

# www/index.html

# http://www.whoohoo.com/cgi-bin/howto/display.cgi?1013

# Use the POST directive for pages that require it:

# http://www.whoohoo.com/cgi-bin/haha.cgi POST ha=1&ho=2

#      or POST content from a file:

# http://www.whoohoo.com/melvin.jsp POST </home/jeff/haha

# http://www.whoohoo.com/melvin.jsp POST <./haha

# You may also set and reference variables inside this file,

# for more information, man urls_txt

# -------------------------------------------------------


www.example.com

www.example.org

123.45.67.89

2. To run Siege with this file use the siege command:

siege

If using a separate file, run:

siege -f your/file/path.txt

Commands

Siege features a number of command line options to use when you want to deviate from the default configuration but do not wish to edit the file.

• -c [num]: Set the number of concurrent users. Most web servers have less than a couple hundred users trying to access their website at the same time, so setting this to more than a few hundred is often not needed.

• -t [num]: Set a time limit for which Siege runs. Siege can run with the modifiers s for seconds, m for minutes, or h for hours. There should be no space between the number and the modifier (-t10s not -t10 s).

• -d [num]: Set the delay for each Siege user. Each user is then delayed for a random amount of seconds in between 1 and the set number. The default value is 3.

• -i: Used in conjunction with a URLs file, this causes each user to randomly hit one of the URLs, with no predetermined pattern. Similar to real life (the ‘i’ stands for “internet”), where you will not know where site visitors go, not all pages may be hit.

• -v: Verbose output. This outputs the results Siege gets in real time before printing the final results.

• -f [file]: Run Siege with a file containing a list of URLs that is not the default urls.txt file.

• -g [url]: Pull down the HTTP headers.

• -l: Generates a log file.

• -m "[message]": Include a message in the log file.

• -C: Outputs Siege’s current configuration profile.

• -V: Outputs Siege’s version information.

• -h: Outputs help information.

More Information

You may wish to consult the following resources for additional information on this topic. While these are provided in the hope that they will be useful, please note that we cannot vouch for the accuracy or timeliness of externally hosted materials.
Siege Home

Hope this will help you!

Please Remember me in your prayers!

Enjoy 



These are Zombies. On this article I’m explaining the ways to find and kill all Zombies on the server.

A process is called a zombie process if the process has been completed, but its PID and process entry remains in the Linux process table. A process is removed from the process table when the process is completed, and its parent process reads the completed process' exit status by using the wait() system call. If a parent process fails to call wait() for whatever reason, its child process will be left in the process table, becoming a zombie. It almost always means that the parent is still around. If the parent exited, the child would be orphaned and re-parented to init, which would immediately perform the wait. In other words, they should go away once the parent process is done.

A zombie process doesn’t react to signals.

1. How can I get the Zombies from process list?

Its very simple! You can find out the Zombie processes in different ways:

# ps aux |grep "defunct"

Shashank  3366  0.0  0.0    0     0 ?        Z    07:34   0:00 [chrome] defunct 

Shashank  3435  0.0  0.0    0     0 ?        Z    07:44   0:19 [chrome] defunct 

Shashank  3722  0.0  0.0    0     0 ?        Z    08:21   0:00 [pidgin] defunct 

Shashank  4287  0.1  0.0    0     0 ?        Z    09:26   0:38 [chrome] defunct 

Shashank  5378  0.1  0.0    0     0 ?        Z    11:24   0:15 [chrome] defunct

# ps aux |grep Z

USER         PID  %CPU  %MEM    VSZ   RSS  TTY    STAT     START   TIME    COMMAND

Shashank   3366    0.0         0.0           0        0       ?           Z          07:34     0:00      [chrome] 

Shashank   3435    0.0         0.0           0        0       ?           Z          07:44     0:19      [chrome] 

Shashank   3722    0.0         0.0           0        0       ?           Z          08:21     0:00      [pidgin] 

Shashank   4287    0.1         0.0           0        0       ?           Z          09:26     0:38      [chrome] 

Shashank   5378    0.1         0.0           0        0       ?           Z          11:24     0:15      [chrome]

2. How many Zombie processes running on your server? Juz count it out!

This is just to make a count of Zombie processes on the server. It can be done in different ways.

Please see some examples.

# ps aux | awk {'print $8'} | grep -c Z

5

# ps aux | awk '{ print $8 " " $2 }' | grep -wc Z

5

# ps aux | awk {'print $8'} | grep Z | wc -l

5

3. List the PID of Zombie?

# ps aux | awk '{ print $8 " " $2 }' | grep -w Z

Z 3366

Z 3435

Z 3722

Z 4287

Z 5378

In order to kill these processes, you need to find the parent process first.

# pstree -paul

See the sample output:

[root@vps ~]# pstree -paul

init,1

  |-crond,542

  |-dovecot,6576

  |   |-anvil,6577,dovecot

  |   |-config,25099

  |   `-log,6578

  |-httpd,5047

  |   |-httpd,1900,apache

  |   |-httpd,9428,apache

  |   |   |-php-cgi,1904,ctalk

  |   |   |-php-cgi,11989,ctalk

  |   |   `-php-cgi,11994,ctalk

  |   |-httpd,19203,apache

  |   |-httpd,22975,apache

  |   |-httpd,25197,apache

  |   `-httpd,30417,apache

  |-(kthreadd/3929,2)

  |   `-(khelper/3929,3)

  |-master,5227

........

This will show the pid of the of the parent of the zombie process. Now you need to kill the parent process or restart the service.

4. To find zombie processes with PPID(Parent PID) on Linux:

# ps axo stat,ppid,pid,comm | grep -w defunct 

Z    27698   3366 chrome <defunct>

Z    27698   3435 chrome <defunct>

Z    27758   3722 pidgin <defunct>

Z    27698   4287 chrome <defunct>

Z    27698   5378 chrome <defunct> 

The above command searches for processes with zombie (defunct) state, and displays them in (state, PPID, PID, command-name) format. The sample output shows that there is two zombie processes associated with "chrome & pidgin", and these were spawned by parent process with PIDs 27698 & 27758 respectively.

Killing zombie processes is not obvious since zombie processes are already dead. You can try two options to kill a zombie process on Linux as follows.

First, you can try sending SIGCHLD signal to the zombie's parent process using the kill command. Note that the above command gives you PPID (PID of parent process) of each zombie. In our example, PPID of the zombie are 27698 & 27758.

# sudo kill -s SIGCHLD 27698

# sudo kill -s SIGCHLD 27758

If a zombie process still does not go away, you can kill the parent process (e.g., 27698 or 27758) of the zombie.

# sudo kill -9 27698

# sudo kill -9 27758

Once its parent process gets killed, the zombie will be adopted by the init process, which is a parent of all processes in Linux. The init process periodically calls wait() to reap any zombie process.

Hope this will help you!

Please Remember me in your prayers!

Enjoy 

Even as we know in this kind of technical World everyone desires to be Online along with Run as quick as the rabbit in these Daily growing Technology. But daily Tons of Articles Publish on Web Via Blogger, Wordpress, Facebook, Twitter and much more Social Media Sites and Blogging platforms. But Do you think the Content you Published with your Online Portal is Safe or not. There are a lot of visitors who just visit for Verifying and Comparing your Contents. These are so many peoples who wish to be a Finest Blogger in a short time span. So, In this Tutorial My goal is to share a Finest Tip Ever to Disable or Banned the Copy Paste Option Through your Site to protect them from Copycats who copy your whole contents.

How to Implement?

Just follow the below simple steps to disable copying option on your blogger's blog. Go to Blogger Account >>> Blogger Dashboard After Selecting "Layout" Tab will Open. Click On "Add a Gadget". Now again a New Window will Open. Here select "HTML/JavaScript" Give Title as "Do not Copy Paste" and Paste Below Code in Content Area.

<!-- start disable copy paste --><script src='demo-to-prevent-copy-paste-on-blogger_files/googleapis.js'></script><script type='text/javascript'> if(typeof document.onselectstart!="undefined" ) {document.onselectstart=new Function ("return false" ); } else{document.onmousedown=new Function ("return false" );document.onmouseup=new Function ("return false"); } </script><!-- End disable copy paste -->

Now Save the open window you are almost done!

Hope this will help you!

Please Remember me in your prayers!

Enjoy 

First of all, we will get to know the mode of thieves/plagiarism.

Usually, the thieves do these things to steal:

If you would steal them open blog template is usually the victim and press CTRL-U That will show your source code templates. Now living in a little edit the template will be his victim.

If you would steal the contents of the article, they will make a copy of our article and would copy-paste in his blog. If he did not paste the source of what to include. but if you are naughty, they will not include the source, even above named he wrote. Usually, they see themselves using Right Click and CTRL-C, also can use CTRL-A

Reproduction illustrations RIGHT CLICK

Immediately. Here's how:

1. As usual, you Log In first Blogger. 2. Choose a template / Edit HTML on the Draft TAB 3. Then you find the code (For ease of reference, press CTRL+ F on your keyboard) 4. Now replace that code with the code below 5. Click the preview first if you want to make sure the script above works what does not. 6. If you've Save Template

For additional course:

~ Oncontextmenu = 'return false;' (Function to disable right click / right click on the mouse) ~ OnKeyDown = 'return false;' (Function to disable CTRL + A and CTRL + U on the keyboard) ~ Onmousedown = "return false;" (Serves to turn off the block select the text/article) ~ 'return false;' (To enable scripts used)

For example, we just want to disable CTRL-U on the blog, then the script is inserted just this:

<body onkeydown='return false;'>

Whatever you wear that anywhere, just select. The most important thing when you edit the script you must be observant

From time to time I want to learn more about stability and high availability web servers. I usually get a sandbox site where I start to play with configurations.

This time all started because I also like testing service providers, and this week I have been moving my site between Site44 and NearlyFreeSpeech to test its features for static sites.

Finally, in order to test how Apache may work when serving static files (I usually use Nginx for that matter), I installed Apache 2.2 on an Ubuntu 12.10 server (Linode). After doing that I headed to blitz.io and tested my Apache 2.2 serving static files. To my surprise, it served without a sweat for one minute a page with user concurrency of 250 users per second.

The next day I tried to install a Wordpress site on the same server, as soon as I tried to install PHP, aptly told me that it was going to uninstall apache2-mpm-worker and install apache2-mpm-prefork instead.

I instructed apt to do nothing and started to read about the differences. It turned out that Apache may work in two different ways which are:

Apache MPM Worker

In this mode, Apache works more or less like Nginx.

This Multi-Processing Module (MPM) implements a hybrid multi-process multi-threaded server. By using threads to serve requests, it is able to serve a large number of requests with fewer system resources than a process-based server. However, it retains much of the stability of a process-based server by keeping multiple processes available, each with many threads.

A single control process (the parent) is responsible for launching child processes. Each child process creates a fixed number of server threads as specified in the ThreadsPerChild directive, as well as a listener thread which listens for connections and passes them to a server thread for processing when they arrive.

There is the reason why my tests with blitz.io and Apache2 serving static files went so well, I was using a Nginx-like configured Apache.

Apache MPM Prefork

This Multi-Processing Module (MPM) implements a non-threaded, pre-forking web server that handles requests in a manner similar to Apache 1.3. It is appropriate for sites that need to avoid threading for compatibility with non-thread-safe libraries. It is also the best MPM for isolating each request, so that a problem with a single request will not affect any other.

Differences are notable, and because we usually install Apache in a LAMP stack, we install something like this

 sudo apt-get install apache2 php5 mysql-server mysql-client

That way we end up with apache2-mpm-prefork package. Because I have installed my Apache2 copy as a stand alone server Ubuntu installed the apache2-mpm-worker pre-build package instead.

Why prefork

Despite MPM worker being more efficient than MPM prefork when the server is under heavy load, it is usually not installed that way because mod_php can't work with MPM worker.

We need to know a little bit about how you can execute PHP code when working with Apache.

The most common way is with mod_php which is an Apache module capable of understanding and executing PHP code inside Apache itself. This is the most common way to run PHP under Apache.

But there is another way, which is similar to what Nginx does, and that is having PHP interpreted as CGI, which means that Apache is going to forward PHP to an external interpreter and get the result back. Using this method allows you to keep Apache configured to run in worker MPM mode, and still be able to "understand" PHP.

Hope this will help you!

Please Remember me in your prayers!

Enjoy 

Are you worried about your Apache server performance? Okay, let’s talk about Apache Multi-Processing Modules (MPMs). There is a documentation for Apache MPM but who has got time to read the documentations. Let’s talk in simple plain English about Apache Multi-Processing Modules (MPMs). All you need is 15 mins to learn Apache MPMs(happy-face).

What

The MPMs are used to change the basic functionality of the web server. It’s possible due to Apache’s modular design. The MPM, or Multi-Processing Module, you use is responsible for just about the entire HTTP session. Starting from listening on the network, taking requests in and most importantly, how to handle those requests. With the MPM you use Apache’s behavior will change. Apache offers three MPMs to choose from; Prefork, Worker, and Event. You might be wondering which MPM module you should choose. The answer is right below.

How do I select which Apache MPM to use?

Above links explains about the three MPM modules and when to use them. If you don’t know about the Apache MPMs or which one to use, time to start reading.

How

1. Check what your Apache server has got

Most of the Apache server comes with Prefork module. To make sure whether your server has got Prefork, type the below command and see.

1

apache2ctl ­-l

See the output below

1

2

3

4

5

6

7

8

9

10

11

12

dasunhegoda@dasun:~$ apache2ctl ­l

Compiled in modules:

  core.c

  mod_so.c

  mod_watchdog.c

  http_core.c

  mod_log_config.c

  prefork.c

  mod_logio.c

  mod_version.c

  mod_unixd.c

  mod_unixd.c

If the Prefork Module is installed it should be shown under compiled in modules. prefork.c is shown on the list.

2. Let’s install Apache Worker MPM

Let’s configure Worker MPM. So time to install Worker.

1

apt-get install apache-MPM-worker

If you are willing to install Prefork or Event MPMs it should be as below

1

apt-get install apache-MPM-prefork

1

apt-get install apache-MPM-event

When the installation is completed type apache2ctl -l and see whether the prefork.c/worker.c shows up under “Compiled in modules”.

Note in Ubuntu you can only have one MPM module at a time. It means if you install Worker while the server has got Prefork, Prefork will be automatically removed. When you are switching MPMs it’s a good idea to backup your .conf files.

3. Let’s understand Apache MPM directives

Please read the comments below to understand about the each directive and what they do. Below configurations are extracted from apache2.conf

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

# prefork MPM

# StartServers: number of server processes to start

# MinSpareServers: minimum number of server processes which are kept spare

# MaxSpareServers: maximum number of server processes which are kept spare

# MaxClients: maximum number of server processes allowed to start

# MaxRequestsPerChild: maximum number of requests a server process serves

#

#

# worker MPM

# StartServers: initial number of server processes to start

# MinSpareThreads: minimum number of worker threads which are kept spare

# MaxSpareThreads: maximum number of worker threads which are kept spare

# ThreadLimit: ThreadsPerChild can be changed to this maximum value during a

#              graceful restart. ThreadLimit can only be changed by stopping

#              and starting Apache.

# ThreadsPerChild: constant number of worker threads in each server process

# MaxClients: maximum number of simultaneous client connections

# MaxRequestsPerChild: maximum number of requests a server process serves

If you need more information on directives checks the documentation.

4. Time to customize the Apache Worker directives as we need

1

2

3

4

5

6

ServerLimit 10

StartServers 2

MaxClients 100

MinSpareThreads 25

MaxSpareThreads 75

ThreadsPerChild 20

Before customizing the directives you need to understand how the directives work. Let me explain in plain English. The server will start 2 child processes which are determined by StartServers directive. Each process will start 20 threads which are determined by ThreadsPerChild directive so this means 2 processes can service only 40 concurrent connections/clients(i.e. 20×2=40). So what if more requests come in.

Now if more concurrent users come, then another child process will start, that can serve another 20 users. But how many child processes can be started is controlled by ServerLimit parameter, this means that in the configuration above, I can have 10 child processes in total, with each child process can handle 20 thread, in total handling 10×20=200 concurrent users.

But there is a problem, number defined in MaxClients is 100 here, this means that after 5 child processes, no extra process will start since we have defined an upper cap of MaxClients. This also means that if I set MaxClients to 500, after 10 child processes and 200 connections, no extra process will start and we cannot service more than 200 concurrent clients even if we have increased the MaxClient parameter. In this case, we need to also increase ServerLimit to 500/20 i.e. MaxClients/ThreadsPerChild=25

Okay now you know the directives and how they work, the problem is how to calculate the directives. Let’s jump into calculating directive values.

ps -ylC apache2 | awk '{x += $8;y += 1} END {print "Apache Memory Usage (MB): "x/1024; print "Average Proccess Size (MB): "x/((y-1)*1024)}'

You can use above shell script to determine an average amount of memory consumed by one Apache process. In addition to that, it’ll show the total amount of memory consumed by all Apache processes. Just unzip and execute with sh command. Accurate results will be shown when server is under heavy load.

The output

1

2

Apache Memory Usage (MB): 57.586

Average Process Size (MB): 10.2

if in average, let’s assume that one Apache process consumes 50MB RAM and server has got RAM is 2048MB, and you want to leave 512MB for the rest of the processes, then:

1

MaxClients = (2048MB – 512MB)/10MB = 153.6 ~ 153

So that’s how you configure Apache Multi-Processing Modules. If you have any questions let me know in the comments below. Your feedback is highly appreciated(happy-face).

Hope this will help you!

Please Remember me in your prayers!

Enjoy 

My EC2 instance in Amazon ECS has run out of space with an error message similar to the following in the /var/log/messages log file:

Mar 18 16:29:36 ip-10-10-10-10 lvm[2016]: Insufficient free space: 1120 extents needed, but only 19 available 

Short Description

By default, the Amazon ECS Agent on the container instance cleans up stopped containers in a task.

The default interval for this cleanup is 3 hours, but this can be controlled using the following agent parameter in the /etc/ecs/ecs.config file:

ECS_ENGINE_TASK_CLEANUP_WAIT_DURATION

Note: Any value shorter than 1 minute is ignored. For more information, see Available Parameters.

Resolution

With the release of ECS container agent version 1.13.0, Docker images and stopped tasks are cleaned up automatically with the following variable:

ECS_DISABLE_IMAGE_CLEANUP

Note: If you set this variable to true, automated image cleanup is disabled on your container instance, and no images are automatically removed.

If you are not on ECS agent version 1.13.0 or greater, you must upgrade to use the automatic image removal feature; see Updating the Amazon ECS Container Agent. The ECS container agent configuration variables can be configured to tune your automated task and image cleanup experience; see Tunable Parameters.

Hope this will help you!

Please Remember me in your prayers!

Enjoy