I am deploying a full scale production application using Symfony2 and which will need to support high load.

I’ve spent a lot of time getting this right and I didn’t immediately find much supporting evidence that this was being done – so I wanted to offer some notes/guidance on how I set it up. There are obviously lots of ways to skin the cat but a lot of the solutions I see out there are single environment, single box, LAMP on one server kind of solutions which won’t scale well.

Caveat: This is also a work in progress and there are lots of things I still need to improve. But it’s working well as it stands.

First the overall objectives:

• Database should use RDS in a multi-zone configuration so that it is robust to failure. Also benefit from AWS taking care of backups/failover/redundancy.
• Webserver group with auto-scaling capacity (so it can grow and shrink based on traffic demands)
• ElasticSearch group using it’s own clustering, again for scale.
• Use of AWS SQS (SImple Queue Service) for back-end messaging
• Support for three environments (test, staging, production) on the same set of hardware (to save costs). The production environment will be the only one to take significant load.
• Ability to easily deploy to any and all of those environments using Capifony
• Code all hosted in git
• Logs rotated and searchable centrally for troubleshooting

And here are the major steps:

• First start by registering with AWS (not going to document that here, there are plenty of tutorials)
• Create an Elastic Load Balancer in AWS
• Create three CNAME records with your DNS provider pointing to the public DNS of the Load Balancer (we do this first because it can take 24 hours to propagate). Mine were like production.example.com
• Now create a RDS instance in AWS. This was suprisingly easy. And once it’s created you can add your own IP address to the DB Security group for port 3306 and then use a local client like Sequel to connect to it.
• I restored a DB backup from the existing application and created three databases (mm_test, mm_staging, mm_production) for the three environments. They all sit within a single RDS instance. (Compared to administering via phpMyAdmin which I’ve done previously, using Sequel with RDS is like lightening).
• My RDS instance was a m1.small, that should do for now, it can grow later
• I started with the Auto scaling group for the Elastic Search servers. So I needed to install the ASG Command line tools, it seems this is the only bit which isn’t visible in the AWS web console yet (shame). It was a bit of a pain to get these tools working on my machine (Mac OS X) – various stupid mistakes which I worked through.
• I started with a fresh Amazon Linux instance and set to work installing ElasticSearch. I followed the useful guide http://www.elasticsearch.org/tutorials/2011/08/22/elasticsearch-on-ec2.html to do this. Couple of key gotchas, I needed to specify the region: cloud.aws.region: eu-west-1 (because I was in EU)
  I also set the memory:

  export ES_MIN_MEM=800M
  
  export ES_MAX_MEM=800M

• Once I’d got ElasticSearch working on a cluster and coming up green when I queried the cluster, I took an AMI (Image) of the working server.
• Then practiced creating a cluster (with ASG) of 10 machines. Amazing!
• They sat in a security group (mm-SearchGroup) I created.
• Now I want the web servers to be an auto-scaling group sat behind an Elastic Load Balancer.
  Because of the way the ElasticSearch clustering works, they each need to have ElasticSearch installed on them (in client mode) to talk to the cluster. So it made sense to start with the same AMI I just created.
• I created a new security group (mm-webGroup) to hold the webservers.
• SSHing into the first box, I went through the following steps to set up the server for Symfony2:
• Need to install Apache and PHP

   sudo yum install http24 php54
   sudo yum install git
   sudo yum install php54-pdo
   sudo yum install php54-mysql
   sudo yum install php54-mcrypt
   sudo yum install php54-xml
   sudo yum install php54-pecl-apc
   sudo yum install php54-mbstring
   sudo yum install php54-intl
   sudo yum install php54-process
    sudo yum install rubygems
   gem install sass
   gem install compass

  Need node:

  sudo yum install gcc-c++ make git
  cd /usr/local/src/
  sudo git clone git://github.com/joyent/node.git
  cd node
  sudo ./configure
  sudo make
  sudo make install
  

  And now less:

  cd ~
  sudo su -
  npm install -g less@1.3.2
  

  check version with:

  lessc -version

• Start Apache:

  sudo service httpd start
  

  In order for the production (EC2) machine to connect to the git hub repository on the VPS server, it must have once connected via SSH (otherwise it stalls at the RSA prompt)

  Created a ~/.ssh/config file for the ec2-user user on the EC2 box which contains the definition for how to access my gitrepo
  chmod g-w ~/.ssh/config (to get permissions right
  ssh gitrepo (run on EC2 box). Once this is done, it should work

  Now I set about trying the deployment and fixed each missing item I went:

  cap -d demo deploy:setup
  cap -d demo deploy:cold
  

Along the way I hit a Github API limit (which is now just 50 requests per hour – if you’re building every few minutes to test a new problem you’ll hit that quickly). I ggot around Github rate limit by getting my auth token from github:

curl -u 'githubuser' -d '{"note":"Composer 2"}' https://api.github.com/authorizations

Then adding this token to composer.json:

   "config": {
     "bin-dir": "bin",
       "github-oauth": {
          "github.com": "oauthtoken1234567"
        }
   }

So that when the build happens, Composer uses the token to communicate with Github.

Also had to set php.ini:

date.timezone = "Europe/London"

Eventually (after lots of attempted deployments), I got my symfony2 app deployed and the symfony check.php to come back green and OK.

None of the webservers needs to deal with SSL directly (it’s handled by the ELB). But they do have to support a health check on port 80. I set up a default virtual host on Apache to do this.

Now I was happily deploying to either of my three environments with capifony, but only on one machine – now to deploy on multiple machines:

I now took an AMI image of my built machine.

Then set up my new auto scaling group:

Create a launch config for the type of EC2 box I want in the webGroup (security)

as-create-launch-config mm-web-lc --image-id ami-xxxxxxxx --instance-type m1.medium --group mm-webGroup --key mm-key-pair

Create a group to specify what and how many and within which loadbalancers

as-create-auto-scaling-group mm-web-asg --launch-configuration mm-web-lc --availability-zones eu-west-1a --min-size 1 --max-size 3 --desired-capacity 3 --load-balancers mm-web-load-balancer

Now add to the capifony deploy.rb script to get it to deploy to all the servers in the mm-webGroup security group:

On my local machine:

sudo gem install capistrano-ec2group

then add to the top of deploy.rb

require 'capistrano/ec2group'

set :aws_access_key_id, '???'
set :aws_secret_access_key, '???'
set :aws_params, :region => 'eu-west-1'

set :aws_pvt_dns, false

group :"mm-webGroup", :web, :app

Also removed the set :domain and role xx lines from deploy.rb

Now

cap production deploy

and the build you lovingly put together, gets run three times across all three servers in the group. And a minute later, the new code is automatically deployed across all of them.

Few things still to do:

• What happens when the ASG grows, it will deploy images with an old build on – how will they get updated
• Log files – at the moment they sit on each server, need to rotate and send to S3 or ElasticSearch so we can review them
• I’ve got to configure the SSL certificate on the ELB properly

There doesn’t seem to be much written about this topic and so I thought I’d record my experience building a Symfony 2.1 form which offers the ability to choose (from a drop down) one of a group of existing entities AND the choice to add a new one of those entities via an embedded form.

A similar feature is available for Collections already within the framework but what if your entity is not a collection?
The secret to doing this is the DataTransformer, specifically a ViewTransformer. Here’s some notes on how it works:

What you will need:
* Underlying entity for your form (in my case OfferDetails containing a ManyToOne relationship to a nested entity, in my case Reward)
* A form type for the form (in my case OfferDetailsType)
* A form type for the new entity (in my case RewardType)
* An object RewardSelector to store the “view” data – in this case a RewardSelected, a checkbox to decide if you want to add a new Reward and a RewardNew
* A form to handle this view – (in my case RewardSelectOrAddType)
* A DataTransformer (RewardTransformer) which turns the view into a single Reward and back again during form rendering
* Some Javascript (JQuery in my case) which hides the add new form

Note that in my example I also need to make sure the newly added Reward entity is constructed by passing a Merchant entity to it – and so the code shows how this can be passed through the constructors of the form type.

I also pass the EntityManager through via the constructors of the FormType rather than by Dependency Injection as per the DataTransformer tutorial on the Symfony2.1 site. I found it easier/less confusing to do this.

The errors you can get whilst getting this to work vary from supremely helpful to very confusing, so take care and try to build it up piece by piece. I have found this solution to work very well even if it is a bit more involved than it should be. Perhaps a feature request for the entity form type should be the “add_new” flag?

Here’s some example code:

//Nora/AcmeBundle/Form/DataTransformer/RewardTransformer.php

namespace Nora\AcmeBundle\Form\DataTransformer;

use Symfony\Component\Form\DataTransformerInterface;
use Symfony\Component\Form\Exception\TransformationFailedException;
use Doctrine\Common\Persistence\ObjectManager;

use Nora\ReferralBundle\Entity\Reward;
use Nora\AcmeBundle\Form\DataTransformer\RewardSelector;

class RewardTransformer implements DataTransformerInterface
{
    /**
     * @var ObjectManager
     */
    private $om;
    private $merchantid;

    /**
     * @param ObjectManager $om
     */
    public function __construct(ObjectManager $om,  $merchantid)
    {
        $this->om = $om;
        $this->merchantid = $merchantid;
    }

    /**
     * Transforms a Reward object to a choice in a RewardSelector object
     *
     * @param  Reward $reward
     * @return RewardSelector|null
     * @throws TransformationFailedException if object (issue) is not found.
     */
    public function transform($reward)
    {
        if (!$reward) {
            return null;
        }

        $merchant = $this->om->getRepository('NoraAcmeBundle:Merchant')->find($this->merchantid);

        $rewardSelector = new RewardSelector($reward, $merchant);

        return $rewardSelector;
    }

    /**
     * Transforms a RewardSelector object to the selected Reward object
     *
     * @param  RewardSelector|null $selector
     * @return string
     */
    public function  reverseTransform($selector)
    {
        if (null === $selector) {
            return "";
        }

        return $selector->getReward();
    }


}

///Nora/AcmeBundle/Form/DataTransformer/RewardSelector.php

namespace Nora\AcmeBundle\Form\DataTransformer;

use Nora\ReferralBundle\Entity\Reward;
use Nora\ReferralBundle\Entity\Merchant;

/**
 * Used to handle objects in the reward selector type
 */
class RewardSelector
{

    private $AddNewReward=false;
    private $RewardNew;
    private $RewardSelected;


    public function __construct(Reward $reward, Merchant $merchant) {
        $this->RewardSelected = $reward;
        $class = get_class($reward);
        $this->RewardNew= new $class($merchant);
    }

    public function getReward() {
        if ($this->AddNewReward) {
            return $this->RewardNew;
        }
        return $this->RewardSelected;
    }

    public function setRewardNew(Reward $RewardNew)
    {
        $this->RewardNew = $RewardNew;
    }

    public function getRewardNew()
    {
        return $this->RewardNew;
    }

    public function setRewardSelected($RewardSelected)
    {
        $this->RewardSelected = $RewardSelected;
    }

    public function getRewardSelected()
    {
        return $this->RewardSelected;
    }

    public function setAddNewReward($AddNewReward)
    {
        $this->AddNewReward = $AddNewReward;
    }

    public function getAddNewReward()
    {
        return $this->AddNewReward;
    }


}

////Nora/AcmeBundle/Form/RewardSelectOrAddType.php
namespace Nora\AcmeBundle\Form;

use Symfony\Component\Form\AbstractType;
use Symfony\Component\Form\FormBuilderInterface;
use Symfony\Component\OptionsResolver\OptionsResolverInterface;
use Doctrine\ORM\EntityRepository;

class RefereeRewardSelectOrAddType extends AbstractType
{
    private $merchantid;

    public function __construct ($merchantid)
    {
        $this->merchantid = $merchantid;
    }

    public function buildForm(FormBuilderInterface $builder, array $options)
    {
        $merchantid = $this->merchantid;
        $builder


            //reward selector (for selecting an existing)
            ->add('RewardSelected','entity',array( 'label'  => 'Referee reward',
                                                            'class' => 'NoraAcmeBundle:Reward',
                                                            'query_builder' => function (EntityRepository $er) use ($merchantid)
                                                            {
                                                                return $er->getValidForMerchant($merchantid, 'Reward');
                                                            },
                                                        'property' => 'ShortName'
                                                    ))

            //reward editor (for new option)
            ->add('AddNewReward','checkbox',array( 'label_render'  => false, 'help_inline' => 'Add a new reward'))

            //reward editor (for new option)
            ->add('RewardNew',new RewardType($merchantid),array( 'label_render'  => false))



        ;
    }

    public function getName()
    {
        return 'RewardSelector';
    }

    public function setDefaultOptions(OptionsResolverInterface $resolver)
    {
        $resolver->setDefaults(array(
            'data_class'      => 'Nora\ClientBundle\Form\DataTransformer\RewardSelector',
            'validation_groups' =>  array('Creation'),
        ));
    }

}

///Nora/AcmeBundle/Form/RewardType.php
namespace Nora\AcmeBundle\Form;

use Symfony\Component\Form\AbstractType;
use Symfony\Component\Form\FormBuilderInterface;
use Symfony\Component\OptionsResolver\OptionsResolverInterface;
use Doctrine\ORM\EntityRepository;

class RewardType extends AbstractType
{
    public function buildForm(FormBuilderInterface $builder, array $options)
    {

        $builder

            ->add('ShortName','text',array( 'required' => true,'label'  => 'Description'))
            
            ->add('AmountType','choice',array( 'required' => true,'label'  => 'Definition is a ','expanded'=>true,
                                            'choices' => array("PERCENT" => "Percentage",
                                                               "AMOUNT" => "Amount",
                                                               "NUMBER" => "Number",
                                                               "DESCRIPTION" => "Text description")))
            ;
    }

    public function getName()
    {
        return 'Reward';
    }

    public function setDefaultOptions(OptionsResolverInterface $resolver)
    {
        $resolver->setDefaults(array(
            'data_class'      => 'Nora\AcmeBundle\Entity\Reward',
        ));
    }


}

And a snippet from inside OfferDetailsType where the form is rendered:

///Nora/AcmeBundle/Form/OfferDetailsType.php
     public function __construct ( $merchant, $em)
    {
        $this->merchant = $merchant;
        $this->em = $em;
    }

    public function buildForm(FormBuilderInterface $builder, array $options)
    {

        $merchantid = $this->merchant->getId();
        $transformer = new RewardTransformer($this->em, $merchantid);

        $builder
         ->add(
                $builder->create('Reward',new RewardSelectOrAddType($merchantid),
                    array('label_render'=>false, 'widget_control_group' => false,'widget_controls' => false,))
                    ->addViewTransformer($transformer))

Snippet from the view showing the JS to show and hide the sections and the creation of the sections

{#/Nora/AcmeBundle/Resources/views/edit.html.twig#}



{{ form_row(attribute(attribute(attribute(form,"OfferDetails"),"Reward"),"RewardSelected" )) }}
{{ form_row(attribute(attribute(attribute(form,"OfferDetails"),"Reward"),"AddNewReward" )) }}
 

   {{ form_row(attribute(attribute(attribute(form,"OfferDetails"),"Reward"),"RewardNew" )) }}
 

Sometimes you need to determine in advance of rendering whether a form has any errors and do something (eg for a form which is normally hidden, make it visible if it has errors).

This is trickier than it should be since Twig has no way to query whether a form has errors from the FormView object it receives. And determining if a Form has any errors in the controller is also tricky.

I wrote this function – which I include in the Controller – which can tell whether there is any errors or not by cycling recursively through the child fields.

Note that some solutions to this problem require error_bubbling=true to be set on every field (in order to bubble the errors from the fields up to the Form object before hand) – in my opinion this defeats the purpose of the nice field-specific error messages.

private function hasAnyErrors(\Symfony\Component\Form\Form $form) {

        $hasErrors = false;
        foreach ($form->getErrors() as $key => $error) {
            $hasErrors = true;
        }
        if ($form->hasChildren()) {
            foreach ($form->getChildren() as $child) {
                if (!$child->isValid()) {
                    $hasErrors = $this->hasAnyErrors($child);
                }
            }
        }

        return $hasErrors;
    }

I wanted to track users by capturing their http referrer when they arrive (wherever they arrive), drop that into a session cookie and then attach it to my database item when (if) they actually convert on the site.

That way, I can tell where the converting users came from.

To do this I needed to use two kernel event listeners – one on each request to pick up the referrer if not already known – and the other on the response to set the cookie if necessary.

If anyone knows how to make it a permanent cookie or whether there’s a way to do this in one listener instead of two, I’d be keen to hear!

Here’s the Request listener:

 
namespace Nora\ProjectBundle\Listener;

use Symfony\Component\DependencyInjection\ContainerInterface;
use Symfony\Component\HttpKernel\Event\GetResponseEvent;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\HttpFoundation\Cookie;

class NoraRequestListener
{
    /**
     * @var \Symfony\Component\DependencyInjection\ContainerInterface
     */
    private $router;

    public function __construct(\Symfony\Component\Routing\Router $router) {
        $this->router = $router;
    }

    public function onKernelRequest(GetResponseEvent $event)
    {

        if ($event->getRequestType() !== \Symfony\Component\HttpKernel\HttpKernel::MASTER_REQUEST) {
            return;
        }

        $request = $event->getRequest();
        $session = $request->getSession();

        //if not already set, set the referrer into the session, later we'll put this in a cookie
        if ($session->has('pwd_arriving_referrer')==false) {
            $session->set('pwd_arriving_referrer', $request->headers->get('referer'));
        }
    }
}

Here’s the REsponse listener:

namespace Nora\ProjectBundle\Listener;

use Symfony\Component\DependencyInjection\ContainerInterface;
use Symfony\Component\HttpKernel\Event\FilterResponseEvent;
use Symfony\Component\HttpFoundation\Response;
use Symfony\Component\HttpFoundation\Cookie;

class NoraResponseListener
{

    public function onKernelResponse(FilterResponseEvent $event)
    {

        if ($event->getRequestType() !== \Symfony\Component\HttpKernel\HttpKernel::MASTER_REQUEST) {
            return;
        }

        $request = $event->getRequest();
        $session = $request->getSession();


        //if the session is set and a cookie is not already set, set the referrer into a cookie
      buy cheap cialis   if ($session->has('pwd_arriving_referrer')) {
            $response = $event->getResponse();
            $cookies =  array_merge($request->cookies->all(),$response->headers->getCookies());

            if (array_key_exists('pwd_arriving_referrer',$cookies )==false) {
                $cookie = new Cookie('pwd_arriving_referrer',  $session->get('pwd_arriving_referrer'),0, '/', null, false, false);

                $response->headers->setCookie($cookie);
            }
         }
    }
}

And heres’ the config.yml where those listeners are configured (in yml):

services:
    kernel.listener.nora_request_listener:
            class: Nora\ProjectBundle\Listener\NoraRequestListener
            tags:
                - { name: kernel.event_listener, event: kernel.request, method: onKernelRequest }
            arguments:
                service: "@router"
    kernel.listener.nora_response_listener:
            class: Nora\ProjectBundle\Listener\NoraResponseListener
            tags:
                - { name: kernel.event_listener, event: kernel.response, method: onKernelResponse }

This was a bit of a pain until I figured it out.

Say you have a form type FormTypeA and it has a nested buy cialis form of FormTypeB Let’s say FormTypeA has a field Username and a nested form with field name User.Address In the nested form (FormTypeB) there is an Address1 field.

Then you want to manage the control of rendering of the form elements (maybe because you want to group fields together or have some neat JQuery stuff hiding elements etc). You need to be able to refer to the fields in the form in your view in both the parent and child forms.

You can refer to the Username field in the parent form easily using:

{{ form_row(form. Username) }}

But to access ALL the fields in the child form you need a more complex syntax:

{{ form_row(attribute(form, “User.Address”)) }}

(this will dump the subform)

And finally, if you need individual fields in the sub-form, you can nest the call like this:

{{ form_row(attribute(attribute(form, “User. Address”),”Address1″)) }}

It’s worth noting that when setting up forms, using the “required” => true flag in a form type is nice, but it ONLY tells the browser to do HTML5 requirement checking. It does no server-side checking. And is no replacement for validation. So if you have users with browsers not doing HTML5 then they will be able to submit forms without those fields.

You MUST also add validation (NotBlank or some other validation) on the fields of the entity in order to make sure they are checked server-side too. It’s mentioned in the docs but is quite an easy trap to fall into – especially if you are developing with a HTML5 browser.

You can buy cialis online add the novalidate=”true” attribute to the

If you have a form which has various sub-forms (relating to sub-entities) but you wish to control validation using a Validation Group, then you need to ensure that the sub-entities are validated within those validation groups.

Eg if you have a Project entity which has a sub-entity User, you might want to make only some of the Project fields validated via adding a buy viagra online validation group “Project_Setup”. if you do this alone, it’s possible the sub-entity User won’t be validated at all.

Instead, if you assert the following on the User entity:

* @Assert\Valid(groups={“project_setup”})

then the sub-entity will be validated. This stops you having to go crazy with deep Validation Group associations…

Oh and if you’re doing this via a form and wonder why your validation is actually working, make sure you add:

$options['validation_groups'] = array(‘project_setup’);

to your FormType default options

https://github.com/symfony/symfony/issues/1946

Use {% debug form %} in a Twig template to see all the variables and how to reference them (in a form)

And this is how you normally refer to a form name:

{{ form_widget(form.Name) }}

But if the name contains deeper methods, eg Name.Surname then you would need to reference this like:

{{ form_widget(attribute(form, “Name.Surname”)) }}

Just some useful reference links I found during Symfony2 development:

Doctrine DQL reference: http://docs.doctrine-project.org/projects/doctrine-orm/en/latest/reference/dql-doctrine-query-language.html

Mopa Bootstrap forms: http://bootstrap.mohrenweiserpartner.de/mopa/bootstrap/forms/extended

Dynamically changing form contents based on entity data: http://symfony.com/doc/current/cookbook/form/dynamic_form_generation.html

How to build a drop down list form containing items selected via a DQL query: http://stackoverflow.com/questions/7807388/passing-data-from-controller-to-type-symfony2

Working around a problem of selecting a list of objects (for a drop down list) from the inverse side of a unidirectional Many to Many relationship: http://stackoverflow.com/questions/10204700/doctrine-query-distinct-related-entity

Nice JQuery example to show and hide divs based on checkboxes: http://jsfiddle.net/wzMM4/26/

Kint: http://code.google.com/p/kint/ (not yet integrated with Symfony2 but looks promising)