EllisLab text mark
Advanced Search
     
Making Sites Fly with Varnish
Posted: 16 May 2012 05:16 PM   [ Ignore ]
Avatar
Joined: 2009-06-17
227 posts

We posted a guide to getting started with Varnish for your ExpressionEngine site. Post here if you have any comments or questions.

Thanks!

http://ellislab.com/blog/entry/making_sites_fly_with_varnish

 Signature 
 
Posted: 16 May 2012 05:36 PM   [ Ignore ]   [ # 1 ]   [ Rating: 0 ]
Avatar
Joined: 2009-02-16
155 posts

We’ve had some really great success with Varnish on a few projects. It’s an excellent package, but suffers a pretty serious flaw in that it requires fairly advanced systems knowledge in order to install, configure, and maintain properly. This is a great article on the complexity that goes into doing that stuff!

I’d also like to point out that there are other options as well - such as my own CacheCracker as well as the excellent CE Cache which I feel are a lot easier to setup for the typical ExpressionEngine dev. Just thought I’d throw it out there in case anyone wasn’t aware and is scared off from taking the plunge into Varnish.

 Signature 

@airways ::: MetaSushi, LLC
ProForm - EE’s first drag & drop form builder ::: CacheCracker - steroids for EE performance ::: Will Hunting - advanced math
Snippet Sync - edit snippets as files ::: Spyglass - advanced text search

 
Posted: 16 May 2012 10:52 PM   [ Ignore ]   [ # 2 ]   [ Rating: 0 ]
Avatar
Joined: 2006-06-03
257 posts

We’ve had excellent (shall I say amazing?) results using CE Cache and Template Morsels, in some cases in a single site.

We are able to get the perceived page speed that beats dedicated/virtual hosting solutions costing $2-300/mo., even on a shared hosting account.

I’d highly recommend both add-ons for any site, not just with high traffic concerns.

 Signature 

ejaedesign.com | creat-ee.com

 
Posted: 17 May 2012 03:55 AM   [ Ignore ]   [ # 3 ]   [ Rating: 0 ]
Avatar
Joined: 2010-01-30
19 posts

For one of our lasts projects (http://www.torfs.be/) we used Varnish. It was the first time we used Varnish and we ran into several issues; especially for the ‘dynamic’ part of that website. As it is a webshop including commenting, ratings and a continuously changing stock we realized that Varnish would not be the easiest part of our projects.

We ran the first few weeks with CE Cache enabled, but it couldn’t handle the massive content (more than 4000 unique pages) and all http requests were still handled by the webservers. With handling 100.000 to 300.000 pageviews a day we needed a better solution as our servers (5 virtual web nodes) were melting. Running with Varnish, the server load was seriously decreased and it could handle more than 500 concurrent users at lightning speed.

Things we learned/did;

- We used some AJAX calls for some ‘dynamic’ parts of our website (eg; comments) but they slow down (and you loose the benefits of your Varnish cache) and with a lot of visitors those AJAX calls still take your server down.
- We receive data from our clients ERP system, when that data is indexed in Solr we automatically run our purge scripts that remove all caches that are related to that data.
- Test a lot! Especially on user-driven content websites.
- Avoid excluding too many pages from the cache.
- Try using ESI snippets where possible but use them wisely.

 Signature 

Zoo keeper at EE-zoo.com
@ee_zoo
@filipvds

 
Posted: 28 June 2012 10:51 PM   [ Ignore ]   [ # 4 ]   [ Rating: 0 ]
Avatar
Joined: 2012-04-15
35 posts

Hi Guys,
Totally new to varnish but gotta start somewhere. As per http://ellislab.com/blog/entry/making_sites_fly_with_varnish I went thru all the steps I believe. I’m running Ubuntu 10.4 and installed Varnish 3.0 using https://www.varnish-cache.org/installation/ubuntu

I created a file main.vcl at /etc/varnish/main.vcl and put just

backend default {
    
.host '127.0.0.1';  # IP address of your backend (Apache, nginx, etc.)
    
.port '8080';       # Port your backend is listening on
}

sub vcl_recv {

    
# Set the URI of your system directory
    
if (req.url '^/system/' ||
        
req.url 'ACT=' ||
        
req.request == 'POST')
    
{
        
return (pass);
    
}

    
unset req.http.Cookie;

    return(
lookup);
}

sub vcl_fetch {

    
# Our cache TTL
    
set beresp.ttl 1m;

    return(
deliver);

..into it and restarted apache. Do I change the 127.0.0.1 to my actual hostname IP of eg 150.151.152.153?

My system folder is the default “system”.
The full path to my system folder is /srv/www/mysite.com/public_html/system
Should I change /system/ in the vcl file to something else? Changing /system/ to /srv/www/mysite.com/public_html/system/ didnt work :(
/etc/varnish/main.vcl is correct and exists at that location on my server.

Using the recommended settings from Kevin I get the error:

root@li482-221:~# root@li482-221:~# varnishd -f /etc/varnish/main.vcl -s malloc,200M
-bashroot@li482-221:~#: command not found
root@li482-221:~# Message from VCC-compiler:
-bashMessagecommand not found
root
@li482-221:~# Syntax error at
-bashSyntaxcommand not found
root
@li482-221:~# ('input' Line 9 Pos 19)
-bashinputcommand not found
root
@li482-221:~#     if (req.url ~ '^/system/' ||
>
> ------------------
#--------------
>
Running VCC-compiler failed, exit 1
>
VCL compilation failed 

Line 9 being

if (req.url '^/system/' || 

..so could someone identify what I am doing wrong please. I also tried ” instead of ’ quotes which didnt throw an error in SSH but instead i get

root@li482-221:~# varnishd -f /etc/varnish/main.vcl -s malloc,200M
WARNINGTaking over SHMFILE marked as owned by running process (pid=20740)
root@li482-221:~# sudo service apache2 restart
 
Restarting web server apache2                                                 ... waiting                                                             [ OK ]
root
@li482-221:~# sudo service varnish start
 
Starting HTTP accelerator varnishd                                    [fail] 

I also changed the apache port to 8080 in ports.conf using http://www.umarani.com/varnish as a loose guide.

Using Putty for SSH.
Thanks!!!

 
Posted: 29 June 2012 12:27 PM   [ Ignore ]   [ # 5 ]   [ Rating: 0 ]
Avatar
Joined: 2009-06-17
227 posts

Hi Ralph,

Ralph123 - 28 June 2012 10:51 PM

Do I change the 127.0.0.1 to my actual hostname IP of eg 150.151.152.153?

If Apache is listening on the same machine you have Varnish, which I think it is in your case, then you can leave it as 127.0.0.1.

Ralph123 - 28 June 2012 10:51 PM

Should I change /system/ in the vcl file to something else? Changing /system/ to /srv/www/mysite.com/public_html/system/ didnt work :(

Varnish doesn’t need to know the server path, just the path it sees in the URL. In this case, you should be fine leaving it as /system/.

It looks like double quotes may have worked for you, there wasn’t a compilation error at least. Running `sudo service varnish start` may not have worked because I think it was already running even though you got that warning. Always try to kill the existing process before starting another one.

 Signature 
 
Posted: 01 July 2012 02:07 PM   [ Ignore ]   [ # 6 ]   [ Rating: 0 ]
Avatar
Joined: 2012-04-15
35 posts

Great thanks Kevin. Thats all working now. Just one issue though and thats with PHPMYADMIN. After installing varnish I cant log in to PHPMYADMIN.
I even added

req.url "^/phpmyadmin/" || 

to

sub vcl_recv {

  
    remove req
.http.X-Forwarded-For;
    
set req.http.X-Forwarded-For = client.ip;

    
    if (
req.url "^/system/" ||
 
req.url "^/phpmyadmin/" ||
        
req.url "ACT=" ||
        
req.request == "POST" ||
        (
req.url "member_box" && req.http.Cookie "exp_sessionid"))
    
{
        
return (pass);
    
}

    
unset req.http.Cookie;

    
set req.grace 1h;

    return(
lookup);

Log in worked fine without Varnish. Is there anyway to fix this?
Thanks

 
Posted: 02 July 2012 10:52 AM   [ Ignore ]   [ # 7 ]   [ Rating: 0 ]
Avatar
Joined: 2009-06-17
227 posts

Strange, adding the check for /phpmyadmin/ should have done the trick. I assume you restarted Varnish after adding this?

If it still doesn’t work after a restart, it’s hard to diagnose what phpMyAdmin is doing from here that Varnish isn’t allowing to hit the backend.

As a last resort, you can access it straight through Apache by going to http://yoursite.com:8080/phpmyadmin/

 Signature 
 
Posted: 02 July 2012 06:09 PM   [ Ignore ]   [ # 8 ]   [ Rating: 0 ]
Avatar
Joined: 2012-04-15
35 posts

thanks it a worked after a few minutes. I have a symlink in the root called myadmin that redirects to phpmyadmin also to overcome a network firewall issue. I was hoping to apply the check to “myadmin” but that didnt work. If anyone knows how to get the symlink working it would be great. Great article by the way!

 
Posted: 05 August 2012 01:52 PM   [ Ignore ]   [ # 9 ]   [ Rating: 0 ]
Avatar
Joined: 2012-04-15
35 posts

Kevin how do you stop

varnishd -/etc/varnish/main.vcl -s malloc,200M 

using the ssh terminal. Whats the command

I tried service varnish stop…but no joy. Kill varnish also didnt work.

I just want to switch off varnish to see if it is causing a php sessions issue on my AWS hosting

Thanks