Apache: Blocking “Bad Robots”

In some cases, robots can go berserk and cause performance issues with an Apache server. To some extent, robots can be helpful as they are used by search engines to index a world-visible site.

But “bad robots” can be blocked.


Robots that are less bad, will pay attention to a page named “robots.txt” which is a plain text file in the root of each virtual web server (i.e. the URL for it would be https://your.server.name/robots.txt). The contents of which look something like :-

User-agent: BadBot Disallow: /
User-agent: NotQuiteSoBadBot Disallow: /infinite-depth

The “User-Agent” parameter is used to specify what robots the following directive(s) should apply to. A “User-Agent: *” would apply to all robots (but not web browsers piloted by people); you can also limit access to certain parts of the web-site – as shown by “/infinite-depth” in the second example.

Some automatically created sub-sites will actually be of infinite depth which is why robots.txt was invented in the first place.

But we’re probably more interested in the really bad robots that ignore robots.txt.

Blocking Robots

This method involves changing the Apache configuration files (at least the way I’ll show you). The first step is to check the file /etc/sysconfig/apache2 to verify that it contains :-

APACHE_MODULES="actions alias auth_basic authn_file authz_host authz_groupfile authz_core authz_user autoindex cgi dir env expires include log_config mime negotiation setenvif ssl socache_shmcb userdir reqtimeout authn_core headers" 

And that it contains “setenvif” so that module gets enabled. If it doesn’t you will have to add it, and restart the Apache service (and not just apachectl graceful) :-

systemctl restart apache2.service

The next step is to add configuration for the setenvif module to add a “tag” to requests with the relevant User-Agent value. Within the file server-tuning.conf is a section guarded by “<IfModule mod_setenvif.c>”. Add to that section something like :-

        BrowserMatchNoCase "Bytespider" bad_bot
        BrowserMatchNoCase "Bytedance" bad_bot
        BrowserMatchNoCase "AhrefsBot" bad_bot
        BrowserMatchNoCase "PetalBot" bad_bot
        BrowserMatchNoCase "SemrushBot" bad_bot

This list can be modified of course, all you need is a unique string found within the “User-Agent” string that can replace “Bytespider”.

The next step is to add the following to each virtual server. Because it applies to the whole virtual server, it is quite likely there is already a section like this, so you may have to adapt the configuration example :-

        <Location "/">
                Order deny,allow
                Deny from env=bad_bot

The final step is to restart apache, but it is helpful to check that the configuration is likely to be working :-

# apachectl configtest
Syntax OK

The command will return nothing at all if the configuration hasn’t changed, and will return some useful information if there’s an error :-

# apachectl graceful
AH00526: Syntax error on line 232 of /etc/apache2/apache2.conf:
Invalid command 'BlahBlah', perhaps misspelled or defined by a module not included in the server configuration
Action 'graceful' failed.
The Apache error log may have more information.

One finished, you can restart Apache with apachectl graceful (which is intentionally non-disruptive as existing sessions remain running with the old configuration), but if you have been messing with which modules are loaded, you may need to restart using systemctl :-

systemctl restart apache2.service


There is no point in changing configuration if you are not going to test the changes – it certainly took me an attempt or six to get the right stuff in the right place! The key is to have a browser that will change the “User-Agent” string, which is apparently something often available in ordinary browsers with the “Development Tools”.

However, for better or worse, I did my testing with curl :-

» curl -H "User-Agent: Bytespider" http://brucellosis-ice.nss.eps.is.port.ac.uk/ 
<title>403 Forbidden</title>
<p>You don't have permission to access this resource.</p>
Posted in Uncategorized | Comments Off on Apache: Blocking “Bad Robots”

Security and Using Artificial Intelligence

Over the past year or so, we have had a bit of an upsurge in the use of “artificial intelligence” with things like ChatGPT, etc. An interesting development – particularly amusing when I started my career with a summer job working at a company doing AI.

So far there have been no big well publicised security incidents regarding AI, so in general it is being allowed through the firewall.

On the other hand, AIs thrive on information – if you upload data into an AI for it. to answer a question, there is a significant risk that the information will be used to enhance the AI’s ‘knowledge base’. And in turn, that information may be ‘leaked’ to third parties when they use the same AI instance.

So it would be advisable to be careful of what information you make available to an AI. Certainly uploading any personal information on say students would be strongly discouraged. Research data? It depends on how public you want it to be.

In some cases, contracts may also restrict making data ‘public’.

Posted in General | Tagged , , , | Comments Off on Security and Using Artificial Intelligence

Cyber security and universities: Managing the risk

Drafted jointly by sector experts from UniversitiesUK, Jisc and the NCSC, with support from UCISA, this guidance outlines the main threats facing the sector and the impact of recent attacks against individual organisations across the UK research and education sector It sets out leaders’ responsibilities to understand and mitigate these risks and provides action points to consider and more detailed advice around our suggested approach to cyber security.


Posted in Uncategorized | Comments Off on Cyber security and universities: Managing the risk

Does That Suspicious Email Contain A QR Code?

In some cases, suspicious emails might contain QR codes to take you to a web site for further action :-

For example :-

From: UoP Helpdesk <helpdesk.port.ac.uk@gmail.com>
Subject: Mail Quota

Dear User,

Your email quota is close to being used up. To enable additional quote, please fill out the form found on the link below :-

In general, QR codes can contain web site addresses, but because they are encoded, it makes it harder for you to read them (so you can’t think “Hey! That looks odd”) and harder for security software to process them.

Anywhere where they appear where an ordinary link would serve just as useful service, should add some suspicion. Painted on the side of the building is another matter.

For example, in the email above :-

  1. There is no “To” and your address doesn’t appear in it.
  2. The “From” address is wrong both in terms of the “name” (we don’t have a “Helpdesk”, we have a “Servicedesk”) and in the form of the address (helpdesk.port.ac.uk@gmail.com – look where the “@” is, and the presence of “port.ac.uk” as part of the bit before the “@”).
  3. The salutation (“Dear User,”) is generic and not specific. Legitimate emails can be generic but it’s still a suspicion point to add to the overall score.
  4. The “Your email quota is close to being used up” adds a sense of urgency to take the action before bad consequences.
  5. And lastly using a QR code instead of a web site address so you can’t inspect the address adds more suspicion.

Posted in Active Attacks, Email | Tagged , | Comments Off on Does That Suspicious Email Contain A QR Code?

Encrypting Apple Devices

So to begin with, why encrypt?

To keep your data private in the event of other people obtaining access to your device. In some cases that is in your own interest (such as for personal devices not used for work) and in some cases the university wants you to!

And in the future, you may well find your access to university services becomes limited if you do not.

And it is so easy to do.

Encrypting iPhones and iPads

These instructions were prepared using an iPad, but they should apply well to an iPhone as well.

First of all, select the Settings app :-

One that is running, select “Touch ID and Passcode” which is found on the left-hand side :-

And then select “Turn Passcode On” (circled in some sort of pink-like colour).

This will take you through the process of setting a six-digit passcode (which you will have to enter twice), and just as a precaution it also requires you to login to iCloud.

Once it is set, that should be all that needs doing. Everything on your iThingie is now encrypted.

On macOS Laptops

Turning on encryption for a laptop with macOS running on it is only marginally more complicated. First of all, visit the Settings app (which has much the same cog-like icon as above), and select “Privacy and Security” down the left-hand side.

Once there, you may have to scroll down to see it, but the “FileVault” section is what you need. In my case, it is already turned on (and I’m not turning it off for this), but click on the “Turn On” button and it will take you through the process.

Oh! And if you have a red circled number next to your “Software Update Available” menu item, do an update too. I’m only waiting until the end of the working day before I kick mine off.

Posted in General | Tagged , , , , , | Comments Off on Encrypting Apple Devices

Do You Know How Many Cyber Attacks We See?

One of the things that most people are probably not aware of is just how many attacks the university sees on a daily basis. For example, yesterday (a middling day in September) the firewall identified and blocked 100,839 attacks. Now most of those attacks were not especially serious, but many were.

One of the many things that the LIS Cyber Operations team does, is to identify the most serious of those attacks, and block them for 3 months, a year, or permanently depending on whether this is the first attack, the second, or the third.

As you can see, every month we block a very rough average of 1,200 attackers. Actually we can add to those figures a few hundred more attacker addresses that are made known to use as “threat intelligence” – attacks that may not have attacked us, but have attacked others.

In case anyone is worrying about blocking legitimate sites, that very rarely happens – not only do legitimate sites rarely perform attacks, but our block list is currently 7,357 entries long. This is approximately 0.00017% of the Internet (or to be more precise the technical maximum of IPv4 addresses on the Internet; many of which are reserved).

Not much more to say about it – criminals are going to crime. And no these events aren’t organised by spotty teenagers in basements; it’s organised crime.

Posted in Active Attacks, Malware | Tagged , , | Comments Off on Do You Know How Many Cyber Attacks We See?

I’m gonna stop you little phishie….

Some organisations put a lot of effort into training their staff to detect and evade phishing attacks. Some even punish them if they slip up. Here’s some thinking from the National Cyber Security Centre…


Posted in Uncategorized | Comments Off on I’m gonna stop you little phishie….

LastPass Leak: What You Need to Do to Protect Your Passwords (CNET article)

For anyone using LastPass…
In late December, LastPass announced that a security incident had allowed an unauthorised party to steal customer account information and vault data.

What should LastPass subscribers do?
Unfortunately, LastPass have not revealed how many users were affected by the breach, and LastPass didn’t respond to CNET’s request for additional comment on the breach.
If you’re a LastPass subscriber, you need to operate under the assumption that your user and vault data are in the hands of an unauthorised party with malicious intentions. Though the most sensitive data is encrypted, the problem is that the threat actor can run “brute force” attacks on those stolen local files. LastPass estimates it would take “millions of years” to guess your master password — if you’ve followed its best practices.

If you just want total peace of mind — you’ll need to spend time and effort changing your individual passwords.

Here’s what you need to do right now if you’re a LastPass subscriber:

1. Find a new password manager. Given LastPass’ history with security incidents and considering the severity of this latest breach, now’s a better time than ever to seek an alternative.

2. Change your most important site-level passwords immediately. This includes passwords for anything like online banking, financial records, internal company logins and medical information. Make sure these new passwords are strong and unique.

3. Change every single one of your other online passwords. It’s a good idea to change your passwords in order of importance here too. Start with changing the passwords to accounts like email and social media profiles, then you can start moving backward to other accounts that may not be as critical.

4. Enable two-factor authentication wherever possible. Once you’ve changed your passwords, make sure to enable 2FA on any online account that offers it. This will give you an added layer of protection by alerting you and requiring you to authorize each login attempt. That means even if someone ends up obtaining your new password, they shouldn’t be able to gain access to a given site without your secondary authenticating device (typically your phone).

5. Change your master password. Though this doesn’t change the threat level to the stolen vaults, it’s still prudent to help mitigate the threats of any potential future attack — that is, if you decide you want to stay with LastPass.

Ref: https://www.cnet.com/tech/services-and-software/lastpass-leak-what-you-need-to-do-to-protect-your-passwords/

Posted in Uncategorized | Comments Off on LastPass Leak: What You Need to Do to Protect Your Passwords (CNET article)

Locating Java Installs

In some cases vulnerability scanners will tell you that there is a vulnerable version of Java installed but not tell you where it is. This is a short post on solving that problem – for Linux machines.

Run the following code :-

for j in $(find / -type f -executable -name java)
echo $j, $($j -version 2>&1 |head -n 1)
done 2> /dev/null

And you will get a list of pathnames to java binaries with the version shown after the comma :-

/opt/java_splunk/jdk1.8.0_212/bin/java, java version "1.8.0_212"
/opt/java_splunk/jdk1.8.0_212/jre/bin/java, java version "1.8.0_212"
Posted in General, Technical | Comments Off on Locating Java Installs

Phishing Attacks Against Academics with an Interest in Russia/Ukraine

We have been alerted to the activities of a politically-motivated phishing “crew” targeting (amongst others) the Higher Education sector with particular reference to academics with interests in Russia and Ukraine.

The attacks look to be targeted to specific individuals with reconnaissance being carried out in advance using social media (specifically LinkedIn) or other public information (OSINT). The attacker will then create email accounts at consumer email providers with email addresses configured to resemble known contacts.

The attacker will then contact the target very often with an initially benign email before mentioning a missing attachment (with a topic of interest). A reply will result in a “weaponized” email being sent which may consist of the following forms :-

  1. A website link to malicious content.
  2. An attached PDF with a website link to malicious content.
  3. A link to a Microsoft OneDrive share containing a PDF with a website link to malicious content.

The website link is usually a link to a credentials acquisition site – i.e. it will capture usernames and passwords. And then will show some innocuous (and relevant) information.

To defend against such attacks :-

  1. If you are working, turn on the GlobalProtect VPN. There are some additional defences against phishing when you go through the University firewall (which includes the VPN).
  2. Be suspicious of new contacts – does the email address match previously published email addresses? Does it look like a personal address rather than an academic address?
  3. Be suspicious of old contacts who exhibit a change – are they using their usual email address? Has the tone of their language changed?

Posted in Active Attacks, Email | Tagged , , | Comments Off on Phishing Attacks Against Academics with an Interest in Russia/Ukraine