IBBoard

Well, not so much MDX but the fact that it's an entirely separate download, even for people who have the latest DirectX installed?

I'm trying to update the Dawn of War Texture Tool to be able to convert TGA images to DDS images. The application is written in C#, so I downloaded the DirectX SDK for .Net. After a bit of fighting to get it to flip a loaded texture, not only do I find out that DirectX doesn't have a direct way of doing it (manual byte swapping is the easiest way) but I also find that the documentation has less substance than a political manifesto and that the Managed DirectX isn't a normal part of most people's DirectX installs. That means my 75KB application has to come with a 4MB+ DirectX installer to add in Managed DirectX support!

You'd have thought that for something like DirectX, Microsoft would have done a better job of making C#/.Net applications that use more accessible, but apparently not. Now I'm trying to work out how to include unmanaged DirectX in my application, and not getting very far.

Normally I like C# for its cleaness, power and ease of use, but at the moment I'm beginning to realise why some people don't like it! Oh well, time to blame it on Microsoft for not making things as easy as they could do again.

It was so easy and in the last two weeks since I installed the extra modification, I've had only two spammers successfully get through with trackbacks. The tactic? Block all trackbacks whose titles don't follow one of the simplest rules of English that most, if not all, bloggers I know follow. Capitalisation. And if that rule catches real trackbacks, well then they don't deserve to track back to me if they can't use capitals! Luckily, it allows through trackbacks like this one will be, as well.

The two that got through had actually used some capitalisation, so now I've noticed another pattern that never occurs in real trackbacks. It involves the blog's name and the supposed blog post's title, and it'd have caught the two that got through. Hopefully this'll be the last of the anti-spam coding until I eventually find something better to do with the domain.

Many many years ago (sometime around September 2004 according to the time stamp on the zip of the source code, but I suspect it was earlier) I took over development of the Rollcall Army Builder. It was first written in 1997 by someone called John L. Martin, and then a guy I knew only as MudiPaws looked after it for a little while and updated it to be compatible with Warhammer 6th Edition.

Anyway, following some recent attempts to get the various sites of the Warvault.Net Toplist to contribute to new files, it turned out that Games Workshop have added extra complications to some armies since MudiPaws' updates. Time for more work from me! Progress is being logged in a thread on the HWT Forums.

The application itself is great - writing it from scratch would be much bigger project than I'd normally undertake on my own (or at least bigger than I'd undertake and complete) but modifying it isn't so bad. The worst part is what it's written in - Visual Basic! Why? What on earth convinced anyone that Visual Basic was a good language to program in?

OK, so it isn't too difficult, and I can cope with not having to put semicolons at the end of lines etc. but I try doing an IF with "a or b" and it complains when I put in "||" for the OR symbol. An equivalent to a 'continue' statement in loops seems out of the question (one exists, but it doesn't want to recognise it for me in VB6), so I'm botching it with GoTo statements. ElseIf has to be one word, while End If is two. Concatenation using "string"&var&"string" errors, but "string" & var & "string" is fine because the & isn't trying to do anything strange to the variable.

There are only two good things I've found so far. First is that it is more direct about syntax errors - you have a syntax error on a line (like an IF without a comparison) and it won't let you do anything until you fix it. The counter side to that is if you start an IF and want to copy the statement from elsewhere, you've got to comment the line out, copy, paste and uncomment. There's a lot errors it won't catch, though. Loose typing means you can get away with a lot in valid syntax but invalid types, and it won't say anything until you run that exact line of code or enter the function that contains it.

The other good thing, something that I'm not sure whether it was re-introduced in .Net 2.0 or not, is editing on-the-fly. Hit an error, it tells you, you can edit the code and continue without having to restart. It's a great help if you're part way through a long-ish set of actions when you hit some mismatch that a C# compiler would have picked up, but that's one of the minus sides - a C# compiler would normally have picked it up before then!

Oh well, I guess updating the code is easier than writing the thing from scratch. I just don't have a clue who would ever want to call themselves a programmer and voluntarily work in VB. Maybe it was something about the 90s?

Although closing comments on all old posts has cut down on a lot of the spam, the odd bit is still getting through. Our neighbourhood repeat-IP spammer (instead of the usual spam-bot net spammers who come from a range of IPs with related spams) has tried again, and the usual range of trackback spam has continued.

I also changed the feedback emails that it sends me when someone's trackback gets stopped, and I've noticed something rather obvious to block a good proportion of the trackbacks dead - HTML character encoded vowels. OK, so people may need to use character encoding for some characters in their blog titles, that's not a problem, but only vowels? Not likely, unless it's a spammer hiding his "credit card" title from direct detection by disguising the vowels. The anti-spam system had already been converting them back to their real characters anyway, but now it stops trackbacks dead in their tracks if the encode vowels and only vowels.

Now to wait for the next set of trackbacks and see how it goes...

Recently there seems to have been a minor swarm of a new style of spam - insurance spam. An extra little bit to the regex and all should be good: insurance for your car, travel, motor, life or medical should all now be blocked.

Also, I've noticed before that spammers were encoding the comment title on a trackback, using the "&#" number code for the vowels. Recently, they've started doing it on the 'blog name' as well, so an update has been made:

$blog_name = preg_replace("~&(#?)(48|49|5[0-7]|6[4-9]|[78][0-9]|9[07-9]|1[01][0-9]|12[0-2]);~e",
    "chr('\\2')", $blog_name);

Now all blog names that encode any standard english letter will be swapped back to their real letter.

As a final note, it's rather strange, but I've actually received hits for searches on "anti-spam regex". As much as I like the idea of being some important spam-fighter, the complete regex is staying private so that it doesn't become common and the spammers don't learn it.

Edit: Typical - my anti-spam filter blocked my trackback to the last anti-spam post. Oh well, at least I know it's working!

Edit 2: ten hours later and four comments got through. Turns out that "\b" for word boundary counts a hyphen as a word boundary, but not an underscore! That minor oversight is now fixed and insurance-scam spam should now be fairly well blocked (until they decide to advertise some other form of insurance, on they try Pratchett's Inn-Sewer-Ants!)

It seems like it's the most effective measure out there - create a custom RegEx within the Comment code, Trackback code and hit logging and alter it as I get things that pass through. It might be less effective on busier sites, where it'll get more false positives, but for this size and popularity of blog it's fine.

I've been keeping my "Trackback blocked" emails since the 24th January (I also used to send Comment blocked emails, but stopped them when comment spam came in a flood and I started getting too many emails). I've just updated my RegEx with the latest bout of words - about 20 in total - and checked the success ratio.

Having not updated the RegEx for a couple of weeks, and not blocking anything, I had 28 comments or trackbacks that had managed to sneak through in the three weeks, and 191 trackbacks stopped. Yes, that's just trackbacks, not including comments (which don't email me any more) or hit logs (which never emailed me). Not a bad ratio

Now to see how long this update lasts before someone gets through it!

Firstly, this is a coding update, but preceeded by a commentary on the stupidity of some people.

Dawn of War skins aren't packaged as self-installing EXEs. If it is only a single skin, it doesn't make sense (zipped they are only about 600KB). If it is a full pack, then mine don't come with self-installers because I also make the Pattern Switcher, which wouldn't work with directly installed patterns. Besides, how difficult can it be to open a zip, enter a location and hit "extract"?

Apparently, for some people, too difficult. Now that isn't much of a problem on its own, I understand that not everyone will be as computer savvy, which is why I'm willing to help people who have problems.

What is bad is when people then go to the download and rate it as 1 with a comment 'Doesn't work!!' instead of contacting me. If it doesn't work, then how did everyone else install it? How did I get the in-game screenshots? and why didn't you read the bit on the comment form that says Please note: this form is for rating the download only. If you have any questions about it or are experiencing and problems, then please read the FAQs, contact the Administrator or ask at the Hive World Terra Forums?

Even worse is the arrogant and obnoxious American AOL user who instead of ignorance and lack of knowledge, showed stupidity by being abusive in his comment and rating it as 1 just because the creator got the name of one of the Chaos Legions wrong! Lets just say that the comment has been censored and nulled, and the less seen of people like that the better.

On to the actual news - the Download rating system has now been updated. As well as fixing a bug where it showed my name as the author on the View Ratings page, it now also allows for comments without ratings, as well as a method for me to null ratings. Hopefully it won't have to be used too much. It will now also send me an email when a comment is added, so I can catch people like this Virginian AOL user (as in the state in America) before the comment has been sat there too long.

Relic's Dawn of War game (and the Winter Assault expansion pack) use an archive format called SGA. These SGAs contain all of the files that the game needs in the way of models, textures, sounds and stats/abilities. Extracting these files means you can edit just about all them, and some of them need only minor hex-editing to be converted for use in an entire new race (e.g. the WHM and WHE files for models and their movements and the RSH and WTP files for their textures).

Having already created a .Net library that can be used by any application to allow it to open, examine and extract from SGA files and then used it in specific Halved Marine Mod and Quartered Marine Mod installers, the command-line extractor makes a generic tool that can be used by anyone and can especially be used by the different Mod Teams out there to keep the download size of their mods smaller (for those with .Net installed).

Ladies and gentlemen, I present, for your viewing pleasure, the Command-line SGA Extractor!

For the past few months, I've been working on a new website and a new use for my CMS.

The latest website to run on the system is FloridaHomesDirectory.co.uk, the new Florida holiday villa listing website that is being set up and run by my girlfriend and her mother. The website aims to link holiday makers with the villas they want, all through a small set of search filters. Each villa will be listed with a photograph and a small description, ensuring that visitors can see what they are getting before they click through to the villa website.

The site isn't live quite yet, we're waiting until the New Year for the actual launch, but it should be fully functional now.The site design has also been checked in a variety of browsers. Mozilla 0.9 chokes on the styling used for forms (but then it is quite a few years old now!), Opera 6 wasn't too happy with some parts of the styling either, and Netscape 4 manages to get confused by the hitcounter (stupid thing) but other than that all versions of IE, the latest Firefox and v7 and above of Opera handle it fine.

Since the original "Junk Removed From Logs" post, I've still had referers getting through and I've had some comment spam as well.

Although it's always going to be an uphill battle, I'm willing to keep blocking different keywords if it means less junk in the logs and comments.

Some of the words I've blacklisted so far include (but are not limited to):

• realty
• prescription
• mortgage
• medications
• pharmacy
• texas-holdem and texas-hold-em
• brand names I know and things that are probably brand names but I've never heard of e.g. cialis, viagra, amoxil, meridia, prilosec, propecia, prozac and phentermine

This list might ignore some valid referers, but I'd prefer that over referer spam in my logs. The blacklist has also been applied to the URL field of the comment box, so any spam in there will get a "sorry, the URL is potential spam" message if they try to submit spam comments. That one's the safer one, as although someone might blog "how to stop texas-holdem spam" and have it in a referer address, they're less likely to have it in their real URL that they enter for making a comment.

Now to leave it a while and see how the extra keywords go, adding more when I find what gets through

Update (the next day): Last night's bombardment of spam (including some comment spam) appears to have been fought off without a single casualty! Not even a single comment or referer got through that shouldn't have Hopefully the trend will continue with the comparative lack of successful spammage.

Update (11th December): Two nights and only two bits of spam got through One was from a completely impossible to detect domain with no give away page (hoodia.op-clan.com) and the other has what appears to be another brand name that is now getting blocked. Approximately a dozen referer spams have been blocked, though, possibly a little more.

Update (12th December): One more comment got through last night, although it had no URL in it anywhere, just lots of variations on "buy phentramine" or something. One referrer got through again, but again it was with a brand name of drug so it won't get through again. Soon enough I'll know all of the drug brand names and have them all blacklisted!

I should have post about this earlier, but I forgot, so the post is here now.

On Sunday, I added a new version of the Dawn of War Pattern Switcher. This version adds Winter Assault functionality, allowing the importing of Imperial Guard textures and the launching of Dawn of War: Winter Assault.

The latest Pattern Switcher still works with plain Dawn of War, but if you don't have Winter Assault installed then the "Launch" button will throw an error.

*has a sudden realisation* Hmmmmmmm, maybe I should have handled that better. Never mind, hopefully people aren't that dumb!

Anyway, the latest version of the Pattern Switcher is available at the normal location: http://skins.hiveworldterra.co.uk/PatternSwitcher/.

Following a survey of the beta readers at Perfect Imagination Harry Potter Beta Reader Directory, some new features have been planned for the Beta Reader Control Panel.

The most important, and just coded but not released, feature is more detailed logging of assignment progress. This means that beta readers can track their assigned stories chapter by chapter and report what step each chapter is at.

Also planned is the inclusion of an automated "have you abandonned this story?" email and the ability for beta readers to add their own assignments - useful for people who want to track all of their assignments in one place, even if the author didn't contact them through Perfect Imagination.

Hopefully these features will prove useful when they are released.

As well as very recent work on the Beta Search module for Magic, I've been working on performance improvements for Grots, the CMS over at Skins@HWT. Although the speed improvements won't be restricted to Skins@HWT, they are currently being tested or about to be tested there.

Current performance enhancements of Grots include caching language and skinning into PHP files instead of reading them from the database every time (originally implemented because of the annoying habit of webservers not to have the correct permissions to edit flat files without setting them to 777 - writable by everyone). Also being added is caching of CSS (which can already be done by proxy servers anyway, because of the headers that I pass) and caching of the template into a PHP file.

Add to this a few code changes (such as taking advantage of SQL functions and streamlining code where I can now see I took the long route to the solution) I should hopefully be able to optimise the code even further. Add to this the fact that no-one has been very helpful in my Optimisation of Loops thread on the DevShed forums and that I've not been able to recreate the php.lt benchmark stats and it looks like that'll be the end of most of my optimisations. There would be something to gain from flattening some functions, but just about all of them are for improved reusability and maintenance and so are staying.

In other news, it looks like there could be a new spider that has bypassed the old isSpider() function that I created and use. It hit a lot of my download pages (just about all of them) within a couple of minutes and failed on all of them (because it didn't have a session). It's name is "DTAAgent" and all I can find on it is a post or two on some forums (WebMasterWorld.com I think it was) which say that it might be an IBM development or it resolves to an IP Address owned by British Telephone Directory company 192.com. I'll keep an eye on it and see if any more information emerges.

An additional feature has now been added to the Beta Reader Directory module over at Perfect Imagination (which runs on Magic). It isn't visible to the majority of users, but registered Beta Readers now get to see a hit counter of visitors to their profile. This will help give a better idea of whether they are not receiving many assignments because people are not viewing their profile after finding it in a search (meaning that their profile is too focused) or whether people are viewing but not requesting them (meaning that their profile isn't advertising their skills to the full).

Hopefully this will be a useful improvement for the Harry Potter Beta Readers over at the Perfect Imagination Beta Reader Directory.

Over today and yesterday I've added a new module to Skins@HWT (an FAQ module), updated the Encyclopedia module to include a "source" for the information and even had time to fix the paging and other issues that the Encyclopedia was having.

I've also just updated my development server (Linux Fedora Core 3) to run MySQL 4.0 instead of the 3.2 it came with. It could have been easier -- the Yum repositories only go up to v4.0 if you're running Fedora Core 4, and trying to update from them confuses Yum with the amount of associated updates it has to try to do -- but having found a few references on the Internet, I went to the MySQL Download Site and grabbed the x86 RPM Downloads (including "Dynamic client libraries") and all is now good I just need to set up the database again and I can use things like FOUND_ROWS() and SQL_CALC_FOUND_ROWS to get better performance, and I'll be able to test them on my development machine instead of just the live server.

Or "How I Got Speed Increases In My CMS"

I've just updated the Skins@HWT CMS (known as Grots) with some speed increases that I've been working on. Although timings can still be variable (due to other things running on the server) the front page generation time is now down from averaging between 0.0075 and 0.0085 to averaging between 0.0065 and 0.0075. This doesn't sound like much, but in reality it's one thousandth of a second on a load time of around eight thousandths of a second - so it's about an eighth!

The main improvement is to use static files for the HTML skinning functions and language strings instead of pulling them out of the database each time. I can now generate the static files through my admin control panel (as long as I manually temporarily set the permissions on the parent folder to 777 when I first create the files!). I've also made some minor tweaks to stop it performing excessive checks such as strpos() in some situations, caching the result instead.

One very useful feature that I found for the skinning (since they are functions of a class) is that PHP classes can actually be "extended" like real object-oriented classes. All this time I've been using PHP4 classes, and I just thought that they were ways of grouping functions and variables while not being fully object orientated!

Now, I wonder if I can squeeze any more out of it...

Some new features have just been added to Magic, the CMS that runs the Perfect Imagination website.

The changes add features required for the pending affiliation with Fiction Alley. Also added is extra security in the Beta Reader Accreditation section that will make it even more difficult for someone to get away with impersonating an accredited beta reader.

In the pipeline are some speed improvements. Following recent problems on the server caused by spammers abusing resources, I gained a sudden interest in improving the performance and reducing the load of Grots and Magic. To do this, I have added XDebug into the PHP installation on my Linux FC3 development machine. Hopefully this will help me find out the slowest parts of my code and improve my coding standards and conventions.

After trying to get some server load/uptime scripts running for my sites, I had to submit a support ticket to the techs to find a way of getting the average loads without fread()ing the /proc/uptime 'file'. Apparently we don't get read access to /proc/ because of security/hacking issues. The solution? use PHP's backticks (` - not to be confused with apostrophe's) and run the uptime command through PHP's shell execution!

Now it seems slightly counter-intuitive to me that /proc/ can't have read access, but we can execute shell commands, but such is life. Here is my simple server load and apache uptime script (note: you'll probably need to highlight the text and scroll it sideways):

<?php
$uptime = @exec('uptime');
//echo $uptime;
preg_match("/averages?: ([0-9\.]+),[\s]+([0-9\.]+),[\s]+([0-9\.]+)/",$uptime,$avgs);
$uptime = explode(' up ', $uptime);

$uptime = explode(',', $uptime[1]);

if (strpos($uptime[0],"day")===false){
if (strpos($uptime[0],"min")===false){
$days = 0;
list($hours, $minutes) = explode(':',trim($uptime[0]));
}
else{
$days = $hours = 0;
$minutes = (int)$uptime[0];
}
}
else{
$days = (int)$uptime[0];
list($hours, $minutes) = explode(':',trim($uptime[1]));
}

Header('Content-type: text/plain');
echo <<<EOF
Average Load (1 min): {$avgs[1]}
Average Load (5 min): {$avgs[2]}
Average Load (15min): {$avgs[3]}
Uptime: $days days $hours hours and $minutes minutes
EOF;
?>

And if you're interested in the current IBBoard server stats, it's running at /uptime.php

I've noticed recently (especially on my girlfriend's Perfect Imagination site where the counter is at the top) that PPHLogger (the hitcounter I use that is installed on my server) was giving very bad performance. It wasn't due to the large tables, but because of the slowness of the PHP getHostByAddr() function that converts, for example, 172.0.0.1 into localhost (a reverse DNS lookup).

So what was the speed improvement? Stop it looking up the host every time and just store the IP (instead of host and IP). I've now got a separate process that I'll be putting into a Cron job to run regularly to keep my stats updated with the hosts. The only possible downside is that my country stats might not work, but then I never paid much attention to those anyway!

It isn't work for my site, but it is work on my CMS system. I've now added an extra two fields to the Beta Search module for my girlfriend's Perfect Imagination Beta Reader Search module. The two fields were a request that she thought might keep me busy for a while, but they were done this morning before the Formula One began

I've also got more functionality added into the Downloads module (or made more accessible and complete anyway), but I've yet to upload it - possibly when I've got a few more additions done.

P&111;k&101;r poker has been showing up in trackbacks a lot recently, so test the new banning

Update: OK, so I tested the new filtering and then forgot to remove the test! This post can now remain, since Craig spotted it and commented on it before I deleted it.

Grots (my custom CMS) has now been updated. The most obvious difference is the change from folder structure addresses to static-looking addresses. As with the previous incarnation, this is all done using Mod-Rewrite to make things easier on the end user, and hopefully make the URLs easier to remember. Other features using Mod-Rewrite (such as PermaLink style links where a download can be accessed at /Downloads/DownloadName.html) may be added later.

Most other changes are generally back-end and admin control panel related, but some of the smaller user-visible changes include decimal places on the file size of Downloads of 1-10KB or 1-10MB and more accurate counting (although still not perfect counting) of the words in the additional text of news reports. I can also now leave replies to people's comments on Downloads, which can be useful on occasion (especially when people use comments to ask questions!)

Finally, for some reason Google's Mediapartner spider keeps spidering my site recently - I don't have any ideas why, since it's supposed to spider for AdSense adverts and I don't use them. I've now updated my list of spiders to include it:

  //a simple attempt to identify if the 'user' is a real user or a spider
  function isSpider(){
    $haystack = strtolower($_SERVER['HTTP_USER_AGENT']);
    //the following needles cover Googlebot, MSNbot, Yahoo's Slurp Cat and more, using the common names, plus special catching for the Polish Szukacz and Google AdSense's Mediapartners bots
    $needles = array("bot", "slurp", "grub", "crawl", "scan", "szukacz", "mediapartners");
    foreach($needles as $n)
      {
        if (strpos($haystack, $n) !== FALSE )
	  {
	    return true;
	  }
      }
    //if we got this far then we didn't find any of the substrings so we think it isn't a spider
    return false;
  }

Please feel free to use it if you want in your PHP code (the original basis is taken from a php.net submission, I've just made it into a function and made it exit as soon as it has an answer).

OK, so I haven't done any website programming so far - I've been working on the Dawn of War Pattern Switcher instead. One person has been emailing me in the past couple of days about a problem with it. It ended up being a mistake in the folder structure on his part and a minor flaw in the way the code decided whether it had anything to import for a given race, but it has now been fixed. I also wanted to add the capability to switch RSH files, since Sandwichwarrior released his amazing Fixed Colour Aspect Warrior skins, which use RSH files.

As well as all of that, I've also made use of some event-driven code and a new progress window - things I'd only consider doing because of some of the code I've seen on my Industrial Experience placement of the past year.

I'm going to be working on updates for the CMS today. I've managed to get a different format for the URLs working, it just involves some back end work as well to make it fully compatible. Once I'm done, everything will look like a nice normal HTML site although in some cases that'll be a nice normal HTML site with a query string as well

Also, I've got a list of improvements and extra features to add/check (mostly, but not all, user visible) that covers almost a side of A4. Some of them are comparatively low priority and will be done after some more skinning, but others will be worked on today.

More news once I've actually started doing some of the work!