quick search:
 

Optimize your Zope site with Squid

Submitted by: bobvin
Last Edited: 2004-10-14

Category:

Average rating is: 5.0 out of 5 (1 ratings)

Description:
Zope includes an AcceleratedHTTPCacheManager product that allows it
to cooperate with Squid, notifying Squid automatically when a cached object
changes.

However, the default mechanism does not work with virtual hosts.

The attached patch solves that problem, and the accompanying text
explains my particular Zope/Squid setup.


Source (Text):
--- lib/python/Products/StandardCacheManagers/AcceleratedHTTPCacheManager.py    2002-08-14 18:25:12.000000000 -0400
+++ lib/python/Products/StandardCacheManagers/AcceleratedHTTPCacheManager.py    2003-02-27 16:26:02.000000000 -0500
@@ -45,7 +45,7 @@
         phys_path = ob.getPhysicalPath()
         if self.hit_counts.has_key(phys_path):
             del self.hit_counts[phys_path]
-        ob_path = quote('/'.join(phys_path))
+        ob_path = '/'+ob.absolute_url(relative=1)
         results = []
         for url in self.notify_urls:
             if not url:
@@ -57,6 +57,8 @@
                 u = 'http://' + url
             (scheme, host, path, params, query, fragment
              ) = urlparse.urlparse(u)
+            if path[:8].lower() == '/http://':
+                path = path[1:]
             if path[-1:] == '/':
                 p = path[:-1] + ob_path
             else:
@@ -67,6 +69,12 @@
             errcode, errmsg, headers = h.getreply()
             h.getfile().read()  # Mandatory for httplib?
             results.append('%s %s' % (errcode, errmsg))
+            if p[-11:]=='/index_html':
+                h.putrequest('PURGE',p[:-10])
+                h.endheaders()
+                errcode, errmsg, headers = h.getreply()
+                h.getfile().read()  # Mandatory for httplib?
+                results.append('%s %s' % (errcode, errmsg))
         return 'Server response(s): ' + ';'.join(results)

     def ZCache_get(self, ob, view_name, keywords, mtime_func, default):

Explanation:
Apply the above diff to your base Zope tree. It was produced with Zope 2.6b1
but the changes are minimal and should apply with few problems to earlier
versions.

Restart Zope. The remainder of this recipe explains how to configure Squid
and Zope to work together. For illustrative purposes, I'm assuming that:

* You're using the standard iptables firewall for Linux.
(You could achieve the same effect with URL rewriting, but I chose
to use iptables because it's quicker and simpler to set up.)
* Your Zope is running on address 1.2.3.4, port 80
(Replace "1.2.3.4" as appropriate for your installation.)
* Your Squid is running on ip address 1.2.3.5, port 3128
(Replace "1.2.3.5" and "3128" as appropriate. Note that with
this setup, you COULD run Squid and Zope on the same IP address.)
* The URLs http://example.org/ and http://www.example.org/ are served by
your Zope root "/" folder.
(Adjust as appropriate for your local setup)
* The URLs http;//subdomain.example.org/ and http://www.subdomain.example.org/
are served by the a subfolder "/subdomain_folder" of your Zope root.
(Again, adjust as appropriate)

1. Edit your /etc/squid/squid.conf file, using the following as an example:

#(start of squid.conf file)----------------------------------------------------------------------------#
# /etc/squid.conf
# Squid configuration file
#
# Optimized for serving as a reverse-proxy front-end for Zope.
#
# This example assumes that Squid is running on IP 1.2.3.4, port 3128,
# and Zope is running on IP 1.2.3.5, port 80
#
# If the interface for 1.2.3.4 and 1.2.3.5 is on eth1, the corresponding local
# firewall rule should read as follows:
#
# iptables -t nat -A PREROUTING \
# -p tcp -i eth1 -s ! 1.2.3.4 --dport 80 \
# -j DNAT --to-destination 1.2.3.5:3128
#
# ----------------------------------------------------
#
# Let squid listen on IP address 1.2.3.5 and port 3128
#
http_port 1.2.3.5:3128
#
# ----------------------------------------------------
#
# Allow unlimited-size file uploads
#
request_body_max_size 0
#
# ----------------------------------------------------
#
# Turn on the options that allow Zope to be a front-end proxy
#
httpd_accel_host virtual
httpd_accel_with_proxy on
httpd_accel_uses_host_header on
#
# ----------------------------------------------------
#
# Define our local IP addresses
#
acl localsrc src 127.0.0.0/255.0.0.0
acl localsrc src 192.168.0.0/255.255.0.0
acl localsrc src 10.0.0.0/255.0.0.0
acl localsrc src 1.2.3.4
acl localsrc src 1.2.3.5
#
# ----------------------------------------------------
#
# Let locally-hosted webservers PURGE cached url's
acl purge method PURGE
http_access allow localsrc purge
#
# ----------------------------------------------------
#
# Define our locally hosted Zope server
#
cache_peer 1.2.3.4 parent 80 0 no-digest no-netdb-exchange no-delay weight=100
#
# ----------------------------------------------------
#
# Define the hostnames served by the Zope server
#
acl zope dstdomain example.org
acl zope dstdomain www.example.org
acl zope dstdomain portal.example.org
acl zope dstdomain www.portal.example.org
#
# ----------------------------------------------------
#
# route requests for zope-hosted domains to the zope server
#
cache_peer_access 1.2.3.4 allow zope
cache_peer_access 1.2.3.4 deny all
#
# ----------------------------------------------------
#
# Force the listed domains to be served by the zope server,
# rather than looking up the hostnames in DNS.
#
always_direct deny zope
never_direct allow zope
#(end of squid.conf file)----------------------------------------------------------------------------#


2. Restart squid.

3. As root, run the following command:

iptables -t nat -A PREROUTING -p tcp -i eth1 -s ! 1.2.3.5 --dport 80 -j DNAT --to-destination 1.2.3.5:3128

Make sure the command is all on one line, and
also make sure you replace "1.2.3.5" with the
external IP address of your Squid server.

You should also add this command to your
startup scripts. so that it runs again when
you reboot.

4. Add a "Virtual Host Monster" object to your main Zope
folder, if you don't have one already. Click on the
[Mappings] tab. Add entries corresponding to the
"example.org" and "www.example.org" lines in your
squid file. In the following example, "example.org"
maps to the root Zope folder, and "subdomain.example.org"
maps to a subfolder called "subdomain_folder" :

example.org/
www.example.org/
subdomain.example.org/subdomain_folder
www.subdomain.example.org/subdomain_folder

5. Add an "Accelerated HTTP Cache Manager" object
to your Zope root, if you don't have one already.
Click on the [Properties] tab, and edit the field
labeled "Notify URLs (via PURGE)" as follows:

http://1.2.3.5:3128/http://example.org
http://1.2.3.5:3128/http://www.example.org

Again, replace "1.2.3.5" with the real IP address
of your Squid server.

6. Click on the [Associate] tab of your Cache manager
object. Click the [Locate] button. Scroll to the
bottom of the list. Click [Select All] and then
click [Save]

7. Add another "Accelerated HTTP Cache Manager" object
to your "/subdomain_folder" folder. Click on the [Properties] tab, and
edit the field labeled "Notify URLs (via PURGE)" as follows:

http://1.2.3.5:3128/http://subdomain.example.org
http://1.2.3.5:3128/http://www.subdomain.example.org

Again, replace "1.2.3.5" with the real IP address
of your Squid server.

8. Click on the [Associate] tab of your subdomain's
Cache manager object. Click the [Locate] button.
Scroll to the bottom of the list. Click [Select All] and
then click [Save]

There. Now Squid will cache your objects until and unless they
change, at which time it will request new copies.


Comments:

Edited to work with virtual domains by bobvin - 2004-10-14
The previously posted recipe didn't work with virtual domains (Squid wasn't 
receiving the proper PURGE request) so I patched Zope and rewrote the recipe. 


by peterbe - 2004-10-14
What if squid is running on :80 and Zope on :8083 ?
Both servers on the same machine.

I'm happy to use VHM Mappings but would prefer VirtualHost redirecting, but I haven't figured out how to do that with pyredir (favorite so far), squidGuard or squirm.
 
Re: Virtual Hosts by mtb - 2004-10-14
I'm using a Python-based redirector I wrote for my virtual domains with Squid and Zope 2.7.0 quite happily.  Watch my page for details; I'll post the details soon (http://mikebabcock.ca or my Zope home directory; user mtb).