Apache Access Logs Find Spiders by Counting Requests to IP Addresses

If you would like a quick summary of the IP addresses that are beating the **** out of your server by firing lots of requests for quite possibly malicious or otherwise nefarious reasons then try this little bash script: #!/bin/bash LOG_FILE=/var/www/vhosts/DOMAIN.co.uk/statistics/logs/access_log OUT_FILE=/tmp/spider_analysis #This generates a file with the top 20 IP addresses by number of requests cat $LOG_FILE | awk '{print $1}' | sort | uniq -c | sort -nr | head -n 20 > $OUT_FILE echo "Top 20 IP addresses by number of request" cat $OUT_FILE #allow for loop to split on new line IFS_BAK=$IFS IFS=" " for i in `cat $OUT_FILE` do COUNT=`echo $i | awk '{print $1}'` IP_ADD=`echo $i | awk '{print $2}'` echo "" echo "---------------------------------" echo "" echo "$IP_ADD has made $COUNT requests" echo "Whois Information" whois $IP_ADD #lynx -dump http://who.cc/$IP_ADD # whois was blocked on the server i was using for some reason, use lynx as a work around echo "" echo "---------------------------------" echo "" done # set that back IFS=$IFS_BAK IFS_BAK= You would use this to give you some idea of which IPs are hitting the server a lot. Read On →

Magento Security Update Testimonials

If you run a Magento store or are otherwise familiar with the world of Magento then you have no doubt seen the urgent security patch that live stores needed to apply quickly. We opted to offer the security patching service for free to our existing clients and then also to the rest of the world. The process of applying the patch is quick and painless thanks to a nice Bash script we put together so there was really no problem in offering this service. Read On →

Magento Profiler Improved - Aoe_Profiler by Fabrizio Branca

The Magento profiler is useful when you are trying to discover what potential bottlenecks are in place in a specific Magento implementation that are slowing down the performance of the site. Often when doing this kind of analysis you might find that a particular block or extension is responsible for more than 70% of the total page load time. At this point you would choose to either optimise, cache or disable the block altogether to remove this performance bottleneck and improve page speed significantly. Read On →

Format searched log files

When you are monitoring a log file you may want to narrow it down, and format the results. This simple one line command will break up the output from a log file to make it easier to quickly read tail -f /path/to/log | grep “search term” | sed ’s/(.*)/———-\n\1\n———-/’

Get all product attributes for a magento object using SQL

Sometimes you may want to quickly compare all of the attributes that object (category / product) has in magento. The EAV structure of the database makes this difficult to do this simply, so this is a query that will do it for you SELECT * FROM ( SELECT ce.sku, ea.attribute_id, ea.attribute_code, CASE ea.backend_type WHEN 'varchar' THEN ce_varchar.value WHEN 'int' THEN ce_int.value WHEN 'text' THEN ce_text.value WHEN 'decimal' THEN ce_decimal.value WHEN 'datetime' THEN ce_datetime.value ELSE ea.backend_type END AS value, ea.is_required AS required FROM catalog_product_entity AS ce LEFT JOIN eav_attribute AS ea ON ce.entity_type_id = ea.entity_type_id LEFT JOIN catalog_product_entity_varchar AS ce_varchar ON ce.entity_id = ce_varchar.entity_id AND ea.attribute_id = ce_varchar.attribute_id AND ea.backend_type = 'varchar' LEFT JOIN catalog_product_entity_int AS ce_int ON ce.entity_id = ce_int.entity_id AND ea.attribute_id = ce_int.attribute_id AND ea.backend_type = 'int' LEFT JOIN catalog_product_entity_text AS ce_text ON ce.entity_id = ce_text.entity_id AND ea.attribute_id = ce_text.attribute_id AND ea.backend_type = 'text' LEFT JOIN catalog_product_entity_decimal AS ce_decimal ON ce.entity_id = ce_decimal.entity_id AND ea.attribute_id = ce_decimal.attribute_id AND ea.backend_type = 'decimal' LEFT JOIN catalog_product_entity_datetime AS ce_datetime ON ce.entity_id = ce_datetime.entity_id AND ea.attribute_id = ce_datetime.attribute_id AND ea.backend_type = 'datetime' WHERE ce.sku = YOUR_SKU ) AS tab WHERE tab.value != ""; This version of the query will only return values that are not blank, but if you only run the sub query it will give you every value. Read On →

Comply with the cookie law with very little effort

A lot of concern has been caused by the “Cookie Law” which says you must tell people that you use cookies if you do, and 99% of sites do! Well here’s a very simple jQuery bolt-on that deals with that, and although visitors need javascript enabled to see it, you have made “best efforts” to inform them, and on any eCommerce site you’re going to have a “this site requires javascript message” anyway. Read On →