On Tuesday 8th January 2009, Adam Dabrowski stood watching the trams go by in his home town of Lodz, Poland. Adam had long been fascinated by the trams and how Lodz’ tram system worked. This ran deeper than just an interest in the sight, sounds and mechanics of Lodz’ ageing rolling stock, however. What Adam was really fascinated by was how the whole thing was controlled.
As another tram began to pass over the junction in front of him, Adam looked down at his hand. In it he held what, from the outside, appeared to be a standard universal TV remote. In reality, though, it was much more than that.
The idea for it had first come to Adam whilst he was in electronics class at school, but it had taken him months to actually create. Indeed building it had involved careful research online, as well as some late night trespassing in tram depots to get more data. Finally, though, it had all come together and now Adam was pretty certain that it would work.
As the wheels of the tram’s first carriage passed over the junction and it began turning to the right, Adam raised his hand and pointed his jury-rigged infra-red remote control at the points. With a solid ‘thunk’, they suddenly switch back to the left. A split second later, the tram’s second bogie hit the reversed points and, with a screech of metal, the tram derailed.
With a mixture of joy and surprise, Adam turned and began to walk away.
Piłsudskiego Avenue, Lodz. Courtesy Zorro2212.
Everything is vulnerable
By the time he was apprehended by Polish police, Adam Dabrowski had caused several more incidents on the Lodz network, luckily resulting in only four injuries. The subsequent investigation would reveal that in the course of his fascination for all things light rail and electronic, Adam had discovered that Lodz’ infra-red based signalling system had a critical flaw. In the right circumstances, it was possible to record the signal sent in one place to a set of points and play the signal back to get the same result in another location.
It was a crude, but effective, cyber attack. Adam was no ‘cyber terrorist’ in the traditional sense (despite being branded as such in the press at the time), but for Lodz’ light rail network the end result was the same. Even before Lodz, transport operators had become increasingly aware of the impact that a concerted cyber attack could have on the services they provided. Lodz served as a reminder that it wasn’t just laptops and desktops they needed to worry about. In an increasingly digital world, anything could be compromised.
The lock out
On 25th November 2016, bargain hunters across America woke early. It was ‘Black Friday’, the day after Thanksgiving, when many shops across the United States open earlier than usual to begin a traditional sales period. In San Francisco, many of those shoppers soon began to descend on the city’s public transport system, the MUNI. There these bargain hunters were greeted by an unexpected surprise – the MUNI was free.
It soon became clear to those travellers – and the world – that this wasn’t due to the beneficence of San Francisco’s transit authority. It was because the entire network’s ticketing system had been hacked, along with a significant percentage of its regular computing systems.
That hack was not something that the transit authority could try and quietly deal with and control. That they were subject to a ‘ransomware’ attack was writ large across the screens of their ticketing machines network-wide, which proclaimed, in broken English:
‘You Hacked. ALL data encrypted.’
Behind the scenes they had also received notification from a hacker group operating under the alias of ‘Andy Saolis’ (a common pseudonym used for this style of attack) as to what the price for returning control of the system would be – 100 bitcoin, or approximately $73,000. It was also accompanied by a further threat which the operator had no way of verifying – to release over 30GB of customer and employee personal and financial data.
Luckily, for the Transit Authority, their backups were unaffected. Nonetheless, it would not be until the following Sunday that they would be able to collect fares again. In fare-box terms alone this represented a $1.5m hit. The overall financial impact and the reputation damage, however, was much greater.
Information and Operational Technology
The nature of the attacks on the Lodz and San Francisco systems were very different. In this, they highlight two of the differing types of cyber attack that rail operators can face. On the one hand, as was the case in San Francisco, operators can be subjected to the type of attack with which most people would associate ‘hacking’. That is, a direct attack on the IT systems that provide the supporting communications and information management infrastructure on which a rail network relies. Lodz, meanwhile, was an attack on what might be regarded as the operational technology (OT) layer – the hardware and software that controls the physical systems which keep the network moving.
In many ways, the possibility of an attack on its OT layer is something that the rail industry has had to be aware of, and deal with, for several decades. Indeed the Lodz attack itself actually highlights just how long this attack vector has existed – it was an infra-red attack, targeted at a very old signalling integration. Nonetheless, it is a growing issue. Just as the ‘internet of things’ is beginning to have its impact on people’s homes – through everything from WIFI kettles to heating systems controlled by iPhone apps – so digital technology continues to penetrate and permeate the control layer of rail’s signalling systems and more.
On the one hand, that brings an enormous range of speed, safety and efficiency benefits. On the other, it drastically changes the way rail’s operators and infrastructure managers have to think about the control systems they run and maintain. As the industry has increasingly discovered, however, this is not as simple as an operator transferring its standard IT approach to its OT as well.
“For a while now, there’s been an appreciation, more than anything else, that OT systems and the challenges that critical national infrastructure operators such as rail face when managing them, are not necessarily the same as those for IT systems.” Says Dr. Alex Tarter, Head of Cyber Consulting at Thales.
“One can glibly say: ‘OT is not IT’. But what that really means is that IT people don’t necessarily understand how OT operates. It’s just a different environment. It’s populated by people in boots, hard-hats and high-vis, not people sitting around in offices installing CAT-5 cables or doing cyber risk assessments and vulnerability investigations.”
Getting secure
How to successfully meld IT security policies and techniques with OT systems is one of the biggest cyber-security issues facing the rail industry. One element of the path forward seems clear – ‘security by design’. This is something that all the major signalling and system manufacturers are all increasingly factoring in by default – and which operators themselves are increasingly mandating up front. It means that the systems being implemented today must include the need for strong cyber-security at the design stage. It’s a simple concept, but one that is not always proving easy to implement, partly because rail operators, and governance groups, are still in the process of defining the universal frameworks within which all systems should operate. Nonetheless, progress is being made.
Security by design, however, only addresses one element of the wider OT cyber-security problem. For whilst it ensures that a system is secure at launch, it doesn’t address what are arguably the two bigger issues – the constant risk of future threats (whether due to existing unknown software flaws, or those introduced through later patches) and the need to secure older, existing systems (which were not designed for security) from attack.
Dealing with the first problem means operators accepting that threats evolve and therefore protection must do too. That’s made ‘cyber as a service’ something increasingly important to the industry. For smaller operators, this means using dedicated outside cyber monitoring services to keep their infrastructure secure. For the larger operators (such as Network Rail or TfL) it often means using the same external services as a way of supporting and educating internal cyber-security teams when new threats arise. Often those services also provide a route to tackling the second problem as well – building the procedures and mitigation measures required to secure ageing infrastructure that is now exposed to new threats.
Crossing disciplines
Trying to secure existing systems and keeping new ones secure, however, brings with it a major challenge – one which again stems from the traditional differences between the way IT and OT have been approached. The techniques for dealing with the issues in both now overlap, but this doesn’t mean that skills in one area are instantly transferable to the other. This is something that both operators and individuals often underestimate.
“A cyber-security consultant, can’t just wander in and say: ‘Well, we understand cyber-security and here’s how they’ll attack you.’” Says Dr. Tarter.
“Because they’ll be told: ‘Yes, but you don’t understand the physical environment we’re working in.’ And they’ll be right. Those people will be unable to tie the genuine cyber risks into the real-world impact.”
“It can be as simple as understanding what is actually running on a system. You can’t just point at an issue and say: ‘Well there’s a flaw there. Someone could introduce malware.’ Because if nothing on that system is running windows or another protocol that the malware is going to care about, then it’s not going to run.”
“That’s one of the biggest challenges rail operators, actually all critical infrastructure companies, have. Indeed our policy, and recommendation, is always that you have to match up a cyber-security expert with a domain expert.”
The concept of meshing cyber-security and domain expertise is not new. Perhaps unsurprisingly it can trace its origins, as an approach, to the world of defence. Indeed whilst they may not have invented it, the British Ministry of Defence (MOD) were certainly an early pioneer. As it re-launched and upgraded Britain’s surface and subsurface naval capacity, the MOD recognised that it was now essentially managing a number of floating industrial control systems.
Those control systems were well designed to be protected against kinetic attacks, and their commanders and crews were well trained in the limits of that protection in combat. The same level of protection and understanding was not being factored into the digital aspect of those control systems, however. Without that, they had no way of ensuring that their digital systems weren’t under (or over) designed, and were robbing their commanders of a critical piece of knowledge in combat. Put bluntly, a captain needed to know which computer systems he had to protect most in battle as much as which physical areas of his ship.
Indeed the adoption of this mentality is one reason why both the MOD and the cyber-security industry didn’t react to ‘revelations’ in the mainstream press that Britain’s nuclear submarine fleet was still using Windows XP. It didn’t matter. Both the MOD and the Navy were confident that they had the combined cyber and domain knowledge to know where that system was used, if it was on a critical attack path and, if so, that on a design – and command – level they had taken the necessary steps to mitigate it.
Taking that same concept of cross-domain expertise into rail is now something that it is critical for operators to do themselves, or get help to do.
“This is actually one area where the nature of Thales as a business, and the range of what we do, means we’re quite lucky.” Admits Dr. Tarter. “But also because we’ve embedded that culture. You need people who are cyber-security and domain experts. Don’t just hire IT people. You have to hire OT people and then we make them cognisant of both disciplines.”
“That’s important because it’s exactly the same in critical infrastructure.” He explains. “Here, the operators get told: ‘We’ve done a risk assessment to ISO 27000. We’ve followed the process and here are the outstanding risks.’ But the business owners may still not understand whether a cyber attack is still going to stop them running the railway.”
“This is because, fundamentally, they don’t understand what is an unacceptable impact and what isn’t, as well as what their operational staff can work around and what they can’t. For that, you have to have domain experts and cyber experts. Because the domain people will understand the effect for each cause. They’ll know what will actually impact the system and what will not. They understand the whole system overall and have that operational knowledge.”
The human factor
The need for expertise and understanding extends beyond the design and management layer as well. This is one area in which the rail industry’s IT and OT security issues overlap. Indeed nothing highlights the risks here better than the San Francisco hack. It would be easy to imagine that MUNI’s systems were penetrated by a dedicated team of Hollywood-style attackers working away in a monitor-lit darkened room. The reality is very different. In fact, the evidence suggests that it was a ‘spray and pray’ attack, in which a transport authority worker unwittingly uploaded the ransomware virus to MUNI systems via a zip file or USB device.
“This is why the human factor has to be at the heart of an operator’s approach.” Dr. Tarter says. “Speaking with my consultant hat on, good cyber-security practice means sitting down with people and saying: ‘Okay, what do you actually need to do?’ We then work out how we can design a system that helps them operate in a more secure manner, but which also makes it really hard for them to act in an insecure manner.”
“When it comes to critical infrastructure, we have to build a ‘secure by default’ mentality into people as well as systems.” He continues. “Apple is a great example of this, where the iPhone, by default, says ‘Hey! Use your fingerprint!’. They make security straight forward and easy to do. Now, when it comes to rail, you can’t just adopt the same practices. There are real world operational aspects to running railways that we can’t ignore. Not least, we can’t expect that people who have had 50 years of operating in one manner in the rail industry will operate in a different manner after a one half-hour training course in anti-phishing. That’s ludicrous.”
“But that’s fine!” He says. “We just have to embrace that and actually make sure that standard operating procedures are designed, by default, to be secure.”
With people as much as systems, however, secure by default is only half the battle. Proactive monitoring services are again key.
“People will deviate from those operating procedures. That’s just human nature.” Dr. Tarter says. “So it’s important that you try to make it so they have to deviate in an insecure manner that you can easily detect and mitigate.”
“We had one system, for example, where the operators and the maintainers were from two different companies.” He says. “Which is not unusual in rail, but from a cyber-security perspective is always problematic. By policy, the maintaining engineers shouldn’t have been modifying the system without letting the operators – and their cyber-security team – know. But because the computer ports were easily accessible, they could just wander up and plug in to do configurations. When challenged, their response would be: ‘Well I did send someone an email to say I was going to do it.’”
“So we put that system, with its easily accessible ports, inside a locked box with an audible alarm which was also tied in, over the network, back to the security operations centre.”
“Now if an engineer wanted to go in and make modifications he had to open the box. And if he opened the box and didn’t tell anybody, then an alarm would go off and someone in the control room would by definition hear about it. Straight forward, but effective.”
As an example, it is one that actually highlights nicely many of the issues that getting cyber-security right in rail means facing – a mix of physical and digital systems, processes that need to cross corporate boundaries and, beneath all that, the overall need to keep people moving.
At root this is, of course, the core cyber-security challenge facing the railway – in London, the UK and ultimately the world. Because for rail, like all elements of critical infrastructure, not providing a service during, or because of, a cyber attack often simply isn’t an option.
Falling back to ‘wetware’
The need to operate a service during an incident is something that is particularly pertinent to cities like London. One of the risks that political proponents of ‘driverless trains’ and reduced staffing levels on the Underground (and Britain’s railways in general) often overlook is the impact that this can have if the network is under attack or potentially compromised digitally.
A ransomware attack on the ticketing or information layer is relatively easy to operate around. Something that targets control systems (the OT layer) though can have a much greater impact. Here, there naturally has to be a higher burden of proof when it comes to demonstrating that a problem has been fully assessed and remediated. And until that burden of proof is met, those digital control systems can’t be used.
As both TfL and the mainline operators push towards more efficient (and thus less human-resource-intensive) systems, this leads to a new problem. The systems that allow an operator to be more robust and efficient in normal operation not only become a barrier to operation when a cyber incident occurs, but potentially magnify its consequences.
“Right now, the rail industry has the ability to flip to manual – to proceed at the slowest speed, with visual inspection and do it that way.” Dr Tarter explains. “So at least you can continue to operate. With a lot less capacity and efficiency, yes, but it’s not a case of fundamentally stopping moving.”
“One of the things we see in disaster recovery exercises is that all critical infrastructure relies increasingly heavily on machines and computers though. And if you’ve moved to a model where you’ve reduced staffing levels thanks to technical efficiencies, then you suddenly discover that going manual requires more resources than you actually have available to you – or at least more trained resources.”
For rail operators, whether they realise it or not, disaster recovery is thus increasingly about making sure their ‘wetware backups’ are as good as their software backups. People, it seems, have become something of the modern equivalent of the strategic steam reserve. They are the old, trustworthy failsafe on which the railways may one day need to depend.
All of this highlights the complexity and importance of understanding cyber-security in the railway context. Because as technology becomes increasingly embedded in railway systems, both the risk of, and scale of impact from, a cyber incident grows.
“In a regular enterprise, the IT systems support the business.” Agrees Dr. Tarter. “These days in critical infrastructure like rail, they are the business.”
“If those systems aren’t working, then as an operator you’re either not providing a service or can only do so by not making money. That’s really not a position in which you want to be.”
If you enjoy what you read, then why not support our writing through buying our magazine.
With thanks to Thales for their assistance and sponsorship of this look at cyber-security on the railways.
Interesting stuff.
A deliberate attack by an “unfriendly power” seems to be a lot more scary than a hacker doing it for the lolz.
The documentary “Zero Days” may still be available on watch-again services but it covers the cyber-attack exploiting industrial programmable logic controllers used in Iranian nuclear program.
So not only did the centrifuges spin themselves to destruction, they kept on reporting that everything was fine…
A deliberate attack by an “unfriendly power” seems to be a lot more scary than a hacker doing it for the lolz.
You’d think, but ultimately they have the same consequences. They also require the same clean up and shut things down just as much.
One thing I cut from this though (because it didn’t seem to add much overall) was a couple of paragraphs on the Seoul metro hack by North Korea. If you want an example of a foreign power intervention then that’s a good one to Google. Outside of rail, there was also a major Russian hack of Ukraine’s power network.
Seconding the Zero Day documentary as well. Definitely worth a watch.
Fascinating article.
Excuse my ignorance, but presumably the risks arise because the systems are connected to the internet. Can someone explain what the benefits are of doing this when the railways have historically had their own standalone communications networks?
@RogerB – the system doesn’t necessarily need to be connected to the internet. Hacking the Iranian Programmable Logic Controllers was rumoured to use an infected USB Flash drive. This was left somewhere, someone then plugged it into their PC out of curiosity, then the virus spread between PCs on the internal private network until it found a PC which could be use to infect the PLCs.
The Polish tram incident in the article didn’t involve the internet and there are rumoured to be devices on the market that can capture, block and retransmit car security codes so a thief can steal it.
@ROGERB: Even if systems are built to be completely separated, it’s remarkably easy for them to become accidentally connected to the internet….
As Reynolds953 has already noted, the Iranian hack involved a USB drive…. But a laptop running some diagnostic software would do just as easily! Who knows what it might have been connected to before?
Yes, no internet required. USB is a great attack vector and you can play on the natural curiosity of people to infect systems.
As per the Register: In a 2016 experiment, 48% of people plugged in USB sticks that had been deliberately scattered around a number of car parks.
The Chicago experiment, as reported in The Register and referenced by John Bull, is really scary. You don’t have to be clever to be a hacker when you can rely on human greed and curiosity.
And as if on cue:
Hospitals across England hit by large-scale cyber-attack
https://www.theguardian.com/society/2017/may/12/hospitals-across-england-hit-by-large-scale-cyber-attack?CMP=Share_AndroidApp_Slack
Good grief! Makes me even more determined not to send anything up to this cloud thingy. Probably ought to back up more frequently too.
And as if on cue:
Hospitals across England hit by large-scale cyber-attack
Wasn’t me! I was in the gym!
It’s exactly the same type of attack that took down San Fran though (albeit a different Ransomware variant).
In more security conscious offices, IT will disable the USB ports of company issued PCs to prevent people plugging in any old USB stick (or people using USB drives to steal data…)
However another trend is for “bring your own device” as often people’s personal computer or mobile is better than the crappy old thing the company provides so many want to use their own devices for work purposes.
The majority of hackers will go after the easiest targets, which means your main effort should be to protect your most vulnerable systems. If your systems take a bit more effort to hack, the hacker will move onto easier targets. The reason why ticketing systems are attacked is because they are an easier target, not because they are less secure, but because they use standard, off-the-shelf solutions, which can be attacked by adapting off-the-shelf malware. The vast majority of attacks are for financial gain so the attacker doesn’t want to make a huge effort on the attack. Attacking a bespoke railway signalling system is a completely different game, fair beyond the average hackers ability or interest.
This doesn’t mean that you shouldn’t secure OT systems where you can, just that the chances of an attack against them, particularly legacy ones, are very very low. The Lodz example if very unusual in that the perpetrator spent a lot of time on it, even though it didn’t result in any financial gain. These sorts of attacks are very unusual. There is a cost-benefit decision to be made on protecting legacy OT systems and in most case, the risk is so low, the cost isn’t worth it.
This is a fascinating article, thank you, and not a little scary. But having an interest in transport architecture, the illustration of that tram station in Lodz was very welcome. I’m off to find out more about it.
For us “non geeks” what is “wet ware”? People?
Interesting article overall. I am no expert in any of this but I do recall a common failing that people would get terribly excited about new bits of IT or new systems and then not be terribly bothered about what it meant for processes or back ups etc. This was more the corporate side than the pure operational railway where rules and regs are an engrained mindset. If you didn’t sort this stuff out the auditors had great fun coming along and writing an enormous list of recommendations for improvement.
I expect things are vastly more complex now given the spread of wifi, tablets, smartphones and “flexible” working. IIRC TfL disabled just about every single usb socket on normal desktop sites. To get one to work required all sorts of authorisations and protections being put in place.
WW
Yes
Should be standard procedure ..
NOT to ban USB’s, but to ensure all imported USB’a are put through a screening process, using a dedicated stand-alone machine.
Which can be wiped & reprogrammed, if necessary, of course.
You can plug a memory stick into our computers at work, but the computers have been programmed not to read them unless they have a certain level of encryption on them. This of course makes them useless on any other computer
@Greg Tingey
It might be worth pointing out that anti-virus screening is constantly playing catch-up with the ingenuity of malware writers, therefore not infallible. When I first started in IT Security, I think there were around 50 known viruses. They are almost uncountable now.
It’s just as important to run on a ‘supported’ operating system (not Windows XP for example), and keep it patched up to date.
Recognising when emails received or websites visited ‘look wrong’ is very useful, also having backups, including older ones in case the latest contains the encrypted versions.
One of the commonest problems I and others have found with industries which say “we have full backups of all our systems and keep them regularly updated” is that, when push comes to shove, they find they don’t work or are incomplete because they don’t actually *test* them regularly too. And, like the NHS ransomware today, it isn’t just the data you need to recover but also the ‘clean’ machines you need, at least temporarily, to get back up and running asap while you deal with the attacked equipment.
I heard about the MUNI experience very shortly after it started, from friends in the Bay area. Given that, arguably, that area has the highest density of computer systems experts yet they were _still_ vulnerable should give everyone cause for careful thought. Yes, you can design-out a lot of ways into systems (and yes, using old versions of Windows – as the NHS was also doing – is one of those red flags) but the human axis will always be the one which matters. Just like idiots throwing iron bars onto railway tracks, it is difficult to realise just how stupid people can be.
Back when I started using computers (1971) they were big things, kept in secure air-conditioned rooms where only the authorised few (the priests and priestesses!) could access them. Now, people take their own laptops, tablets, mobile phones into their workplace and expect to be able to use them on the company network. Or the train’s or plane’s wifi. We’re all asking for trouble, basically.
@AlisonW: Backups are useless unless you can restore them.
I know of an example where (on Unix) somebody forgot to set the backup device corectly before putting a system live.
As far as the operators were concerned they could see backups taking place. The problem was the backup device was /dev/null (a.k.a. The bit bucket).
Nobody noticed untill it was too late….
Testing backups is not quite as simple as you might think. To do it properly, you’ve got to aggressively delete all your data. You will only do that if you are fully confident that the backups are in place and can be used. But whether that is the case is exactly what you are trying to find out!
Malcolm,
Nonsense.
You create your new backup as you would if you had a genuine cyber-attack or other problem (or just running a routine backup) then flip to make that the current database and make the current database the backup. That way you know for certain your backup works without destroying anything. This is really, really basic stuff but it is a sad fact that few people even consider doing this – it is not always appropriate.
A better system is the similar grandfather, father and son which has the advantage that one backup is left untouched when you overwrite the two-generations-ago backup with the current one. The benefits were seen at Golden Rail in York when by mistake I overwrote son with grandfather instead of grandfather with son. At least we had father as our last final backup. Two weeks frantic re-keying in by the team of data clerks and then dealing with the accumulated backlog whilst this was done meant that we had restored the situation.
Southern Heights,
One boss I had used to back-up to a disk called “temp” because the firm that hosted our small IT operation didn’t charge us storage for anything placed there. Needless to say one day he discovered why it was called “temp”. He then in all seriousness asked the hosting company’s IT department for the backup and was taken aback to learnt that this was the one disk that wasn’t backed up.
It does raise the serious point that you have to test against human stupidity.
Trying to be a bit more on topic…
British Rail had a data centre at Darlington. To cater for potential powercuts there was a backup generator. This was occasionally tested by switching off the power supply to the data room and ensuring the generator, located nearby, kicked in.
Come the day when the first genuine power-cut was experienced it was discovered that the back-up generator had been wired with an electric start from the mains electricity supply.
Malicious attacked are, of course, a danger and need to be taken seriously but it is surprising and worrying how often the critical back-up plan can fail for the silliest of reasons. A classic problem is that the operators don’t have the necessary passwords or phone numbers or that, wait for it, they do have these but have them stored on the main computer with no hardcopy available.
A prescient article – as the picture in this German newspaper article on the Deutsche Bahn last Friday shows. It was the same hack that affected our NHS systems.
http://www.sueddeutsche.de/digital/wannacry-nsa-und-bnd-globaler-cyber-angriff-zeigt-wie-riskant-die-arbeit-der-geheimdienste-ist-1.3504126
The key to good security is good education (of the users). Black and white “no USB” are no longer are sufficient; all it takes is one user who can use a USB port to bring a virus in. USB sticks for transferring that presentation from your corporate laptop to a colleague from another organisation, or for plugging in that smart card reader for authenticating yourself, or any other one of a myriad of causes all introduce a need to let a potential attack vector be used for legitimate purposes.
Adhering to effective standards (ie. your user account does not run as admin), making passwords single-sign on (ie. so that users have one complex password to remember and don’t write them down to remember a dozen different ones), making sure backups are routine – and restoring data tested, having effective antivirus/firewall processes and teaching users to virus check data / encrypt before sending is crucial.
The need for humans to interact with a computer system is always it’s greatest weakness.
Pedantic of Purley 13 May 2017 at 21:59
I am told that Albert Einstein sais that only two things were infinite;
1) the Universe
2) human stupidity
and that he wasn’t sure about the Universe
I know this sounds Luddite,but at some point one needs to ask whether the payoff from more extensive computerisation is worth it. This is a question that is *never* asked when faced with the demand to upgrade systems – the trouble is that once one has embarked on the slope, getting off it is tricky. Today’s total shut down of the manchester Metrolink is a case in point. Not a hack, it seems, but because the control room has lost contact with the vehicles, they cannot be allowed to operate. Now – it has been for the previous century or more quite possible to operate a tramway with no continuous contact with the vehicles. Sure, there are some gains from continuous supervision – but – are they worth it?
@Graham H
Surely the point you make is not about over-reliance on computers but an inherent dumbing down in the assumptions of what an unaided human can do. From what I have read – and this may well be not the entire truth, or even far from the truth – Manchester Metrolink shut down because the rules (either coded in or merely in the rule book) now say you can’t operate without continuous contact with vehicles. If the rules had allowed for a failure and then set out arrangements for driving on sight that would have been better.
basic errors being made by those who:
– persist with XP or other out-of-support operating systems
– don’t maintain antivirus updates with efficacy plenty of reputable providers
– don’t test backup processes: it’s a lot easier to recover if you’ve done a rehearsal or have a swap-over system, as explained earlier
– don’t have any kind of emergency procedures
– don’t educate their users on data security
@ Graham H / Quinlet – somewhat astounded to hear about that Metrolink shut down and the reason. As you say it’s a tram system and they’ve only been running on line of sight and, in emergencies, manually operated points, for oh decades and decades. Someone needs to ask *very* basic questions about that rule book change. I assume it’s something to do with the (much delayed) introduction of TMS (tram management system).
Interesting article, and a difficult problem for the industry. Very kind of Thales to sponsor this – does this mean that we will only hear positive news about them henceforth?
I have very little understanding of IT systems – I suppose my position is somewhat like that of someone who can drive safely enough without fully understanding how the engine works – but I know enough not to open attachments to emails unless I am expecting them or they are from a trusted source. Respondents to LR commonly include a link to a topic under discussion (I have done it myself) and I click on the link without thinking, even when the poster is not a regular contributor. Is it safe for me to do so or am I in danger of opening the gate to a Trojan horse?
As I’m sure some are aware here, the most common “hack” is a socially engineered one “Hi, I’m Dave, the Password Inspector, please tell me your Password” (I mean, some are little less crude than that, but you’d be surprised how easy it is to get a password out of someone).
After than it’s phishing (“click here to win a billion pounds”). The common flaw, people. There’s notably insufficient training in IT security. This is the sort of stuff that should get taught in schools. It’s pretty easy to detect phishing and social engineering, passwords should only ever go from the input to the database and nowhere else (pretty much all modern security systems use hashing and salting which in layman’s terms means that the database is never asked for the password, just the garbled mess of a password to compare the input to), that includes scraps of “reminder” paper and conversations. When you open an email, check the sender (and the sender’s email address – specifically the domain name – the bit after the @).
Those two bits of information could stop 90% of successful hacks alone. That’s before you get to the stuff that the newspapers love writing about (like rainbow attacks – rapidly trying a list of common passwords using bits of extraneous data to see what would be most likely – really hard to do these days because most systems kick you out if you try too many passwords at once).
To people who are worried about “too much computerisation”, I would suggest the problem is “too little training”. In terms of the NHS hack, given the level of security I assume these records have (doctor-patient confidentiality etc), I’m amazed that the NHS doesn’t have a soft-closed network for its computers (ie: software prevents all internet requests to everything but stuff at a specific address), but then I’m expecting government contractors to do anything but the cheapest solution………..
NB: I write computer programs for a living and I can say with confidence that as a profession, we’ve made it really hard for “hacks” to work without user input.
@ glbotu – No argument that the “stupidity” of users is a key problem. I’m generally pretty cautious but I’ve got caught out once by an E Mail with a nasty attached to it. There’s always a way to trigger someone’s curiosity / interest / greed / compassion and the scammers know that all too well.
I tend to take the view that no matter how hard the programmers try to make things difficult for the hackers it is simply raising the level of the challenge. “Oh look, a taller wall to get over, A new challenge for us to show that company x is not invulnerable”. That may be disproved by the facts but it’s my perception as an outsider.
Walthamstow Writer (and Graham H),
As you say it’s a tram system and they’ve only been running on line of sight and, in emergencies, manually operated points, for oh decades and decades.
There is the very fundamental question of what makes a system a tram system as opposed to a rail system. If you only have a single tram (or train) then it is really hard to differentiate.
Something I read years ago was that a tram system did not have central control but a railway did. Basically the question comes down to who has the power to switch the points – the driver or the signaller?
The more I have read about Manchester Tramlink, the more I have come to the conclusion it isn’t really a tram system at all but a rail system masquerading as a tram system.
Questions are asked as to whether central control and knowing where vehicles are at all times is going too far. I think the fatal accident at Sandilands shows why those in power are wary of having an uncontrolled system.
FYI, but a “rainbow” attack is where you know the probable password length, or maximum length, and it is a simple task to create every possible password as a list which you then ‘secure’ by the simple hash algorithm you believe the site has used. Then you just compare the hashes against the database you are trying to crack. This is why you hear the word “salt” applied to password storage; if you add something random (the salt) to each individual password string it becomes far more difficult to hack. Without one it is very easy with current equipment.
On the matter of backups, I worked for a while on the Metropolitan Police’s METCAD system. There was a ‘live-live’ system, which did the actual live command and control work. a ‘live-test’ on separate hardware under their control. Then a ‘test-live’ and ‘test-test’ at their outsourced software development supplier. Yes, an ‘urgent’ patch would take some hours (at least) to get through the system, but they never risked the important live-live system getting attacked or misused.
The stupidity of users? To a point. That’s like blaming that Spanish train driver (the overturned train a couple of years ago) as the sole person to blame and not thinking about the design of a new high speed rail that went from running at nearly 200kph then suddenly arrive at a 50kph corner with no protection.
The hacking is partly about inadequately maintained IT infrastructure. But also stupid design, computer systems that aren’t easy to use, mean thousands of organisations only function because staff use “work arounds” to actually complete daily tasks. Management turn a collective blind eye because thinking about the bigger picture doesn’t earn plaudits/promotion/bonus (delete according to sector).
The most courageous manager I ever worked with refused to sign off new systems if they required staff to learn yet another new password, because she recognised that once you got beyond a few passwords people have to write them down. The project teams hated her because she made them work harder to solve the security conundrum.
PoP tram systems have a variety of control systems. I can’t speak for Manchester but in Croydon, the principle you describe (line of sight, tram operated points) is the method of operation. However the trams need a link back to the central control system to get a download of the route for each journey ie the control system tells the trams which points to switch. Without the download, its a matter of the degraded mode. If the tram driver can stop before the points and operate them for the cab, then I imagine some sort of service could be operated. If the degraded mode involves leaving the tram, I think a busy network such as Manchester would become impossibly congested at least in the centre.
(I realise in writing this that I don’t know enough about tram systems, for example whether the signals prevent setting conflicting routes at points.)
@ALISONW – Yes, I know how rainbow attacks work, I was trying to explain them to laymen, which you’ve probably done better than me.
@ISLANDDWELLER – I sort of agree that some solutions are poorly thought through. That’s to do with a different problem that exists in the software development industry. We call it management………..
Basically, because a lot of people don’t seem to understand that software development is difficult and time consuming while at the same time, pretty much everyone can use a computer, many of those people who find themselves in managerial positions tend to specify things which are unnecessary or downright pointless, but nonetheless add a level of complexity to the user – simply because they think they “get it”. I can almost guarantee that if you’ve ever been frustrated at a piece of software, the developer was equally frustrated that they had to write it in the first place.
Your manager’s request to not sign off new systems is partly right, provided that was at the point of design and not at the point of delivery. ie: if no-one’s programmed the program yet, then it’s fine, you can integrate security systems effectively. It’s called security by design. However you can’t do that if the system’s already been written, that’s just a waste of everyone’s money.
@ PoP – I’m concsious that we may be wandering off topic but I think your point leads us to a discussion about defining “central control”. We don’t know for certain what *all* the root causes are in the Croydon accident but I would be surprised if the control room and its role was a key factor in any of it. I’ve been lucky enough to visit the control centre at Therapia Lane and seen it in operation.
I don’t know enough about the vastly expanded Manchester Metrolink as to how it now works. I do know the “TMS” has proved very problematic and has been much delayed in its introduction and affected the level of service that could be run through the central area. However whether its existence and role makes Metrolink a railway I just don’t know.
I think many tram systems have a way of relaying vehicle location to a control point and the tram itself may well trigger junctions and priority in conjunction with traffic signals. In terms of busy systems then I have certainly seen manual activation of points in Amsterdam, Zurich and Vienna and they have huge and complex networks. I believe they all have some sort of tracking and simple control system that can see how the network is performing and even give schedule advice to the tram drivers. I recall an old TV programme that explained how Zurich “controlled” all of its trams, trolleys and buses and ensured on time running. I think my mind blew a gasket when I saw what they’d done and even more so when I used the network for myself and experienced the co-ordinated modal interchange.
Yes it may cause momentary delay if some automatic element of the system has failed but I can’t conceive of the entire tram networks in those cities being closed because some central control facility doesn’t know where the trams are. Trams are just vehicles on tracks that run on roads or separate alignments. They get through junctions under controlled signals just like the rest of the traffic does. In theory Manchester *should* be the same but maybe someone decided it was “special” and needed to be different to places with decades of operational experience?
@WW: I think the trams in Brussels and Antwerp are still using driver initiated point control (a switch on the dashboard, or failing that a metal bar behind the chair). They sometimes have to wait a little while before approaching the point to allow the preceding tram to clear the junction. In the tunnel sections they do have signalling, but then that’s to be expected…
I too find it odd that the trams in Manchester cannot run (perhaps at a slightly reduced service level) without the central control. Sounds to me more like a cock-up! I would be very interested in finding out why this is…
Do the tram signals on line of sight routes do anything more than tell the driver whether they have right of way? Having the points set for you being a necessary condition for a clear signal, but not a sufficient one (there may be conflicting road traffic for example)
@timbeau: Do the tram signals on line of sight routes do anything more than tell the driver whether they have right of way
Generally, no. In traditional tram operation it is the driver’s responsibility to check that the points are set correctly (whether by visually checking the point blades or by looking at a points indicator) before proceeding. Conflicts can also be designed out of the system by, for example, using trailing crossovers not facing crossovers.
The complicating factor in somewhere like Manchester is that much of the route of the original system was originally a railway line with railway block signalling and this approach was retained when it opened. Part of the intention behind the switch to the TMS was to enable the signalling on the ex-railway bits to be abandoned. I assume that getting this past the (railway) safety regulator involved adding extra safeguards which may have complicated the system and made it more fragile.
There are different levels of control to consider – are we talking about control in the sense of the strict control of vehicle movement that railway signalling provides, or control in the sense of a control centre which issues instructions to drivers, controls passenger information systems, deals with disruptions etc?
In principle in the latter case if the central control centre’s systems go down, a tram system should be able to keep working so long as there is some kind of agreed set of procedures (eg. drivers have printed route cards that they stick to until the end of their shift; and having the ability to override the automatic system that sets points). These procedures need to be well defined if they are to be safe (eg. compulsory stop before every set of facing points to check they are set correctly). Having adequate fallbacks in place is a management problem not a technology one.
The comments towards the end of the article about the difficulty of implementing manual systems when there is a problem, with fewer staff available were brought out in the last 2 weeks when there was some sort of signalling communication failure on the Great Western Main Line between (I think) Didcot and Swindon. To implement eh manual system and start running trains needed all the points inspecting and clipping manually. It seems that there just weren’t enough people around who were qualified and available to do the job. As a result all trains were stopped for several hours and only started running again when a part had been brought from Doncaster to get the system running again.
I went to the trolleybus museum at Sandtoft recently. The “frogs” (essentially upside-down points in the trolley wires) are operated by the conductors, pulling a lever attached to the nearest standard (post) supporting the wires, and I understand this was the usual practice in London when we had such thing (tramways had pointsmen). Not quite as arduous as it sounds as each frog were biased to the more commonly-used route, but still requires a lot of jumping on and off.
There was an automatic system which seemed to work by switching the frog from one route to the other depending on whether power was applied to the bus (e.g. coast for right, power for left), but I don’t think it was in widespread use.
How did it work on trolleybuses with doors? One-person operation?
Trolleybuses in the UK didn’t have doors, nor one-person operation. I think neither was permitted. (There were a few one-person trams, but not in London of course). And an occasional bit of jumping off would not have seemed much of an extra burden compared to collecting the fares from 80 or so passengers – including standees – going typical distances of 3 miles – say at least 3 new passengers per minute at the busiest times.
London trolleybus frogs were almost always manually pulled for the ‘diversionary’ route but were generally of a type where the conductor could pull the lever on the traction pole at the side of the road and jump back on, whilst the lever and frog would stay where they were until the trolley poles had passed through, whereby the pull lever was automatically released to reset the route for the ‘mainline’. However, many towns, such as Bournemouth, had auto-frogs over much of the system, using the power-and-coast system controlled by the driver, which required no involvement by the conductor.
As for Malcolm’s theory that “an occasional bit of jumping off would not have seemed much of an extra burden compared to collecting the fares” wouldn’t have gone down well with trolleybus conductors in certain parts of London at complex junctions like the Nag’s Head, Holloway, with all the traffic and approaching some 300 trolleybus movements per hour! The fact is that LT seemed never really to have had the will to modernise its trolleybus kit, even long after driver-operated auto-frogs had become the norm elsewhere in the UK.
Even with modern trolleybus systems with remote-controlled frogs (usually permitting ‘high-speed’ travel through them rather than the 5mph limit in London), it’s still possible in the event of failure of a frog mechanism to get out and transfer the trolley poles from one set of wires to the other. They can carry on. Ditto the tramways as regards the points, which can still be changed manually.
I’ll ask a pal about Manchester and its apparent control centre problems/restrictions as described above. It wouldn’t surprise me, because as PoP suggests, they always did think of the system as a railway and indeed for well over two decades insisted upon refusing to use the word “tram” in any of its literature.
Some British trolleybuses had doors, like this one at the Sandtoft museum.
http://2.bp.blogspot.com/-Eq7NoD-OGfM/TiblFLAGKwI/AAAAAAAAEPo/aQLsRJ–Gcg/s1600/50693.JPG
When I rode on it, the conductor left the doors open.
Southern Heights (Light Railway) 16 May 2017 at 15:55 – “They get through junctions under controlled signals just like the rest of the traffic does. In theory Manchester *should* be the same but maybe someone decided it was “special” and needed to be different to places with decades of operational experience?”
Manchester Metrolink was conceived in the early ‘80s by one or two brilliant engineers with an acute awareness of tramway pedigree. The system was then implemented, at the Treasury’s insistence, by a consortium which conspicuously scorned any respect for trams and tramways and disregarded lessons from the past and overseas. In that way the disturbing character of “trains in the street” was allowed to be imposed upon Metrolink, with plenty of ingrained confusion and ambiguity and self-delusion about its true nature. The starkest aberration is the odd audible warning of approach used on-street, based on an LMS railway steam hooter and chosen in a pub. Metrolink obstinately persists with that dismal sound, demonstrating its unfitness for purpose every time the off-street horn is blasted impolitely at pedestrians in the city centre, afflicting Manchester with what must be the largest fleet of improper-sounding trams in the world. All the system’s other quirks and deficiencies, some of which have already been referred to, stem from the same birth defect – a tramway system born in scorn for tramways.
Why was Metrolink Closed down after the hacking , just because apparently no one knew where the vehicles were. They would have known by simply asking the power supply controllers where traction currant was being drawn:eg, When I was a tram driver in Melbourne in the 90s I blew a motor, resulting in supply circuitbreakers opening due to sudden excessive power drawn. Before I had stopped to carry out my proceedures control was on the radio to confirm my location(which they knew) & whether my Vehicle was mobile (which it was). Who needs computers to know location of Vehicles. As for the drivers doesnot management trust the drivers to operate themselves without computers telling them what to do. Common problem with modern management is they trust and rely on computers more than real people, a dangerous view just to save a few cents. Computers are supposed to be a tool not a conquering takeover machine
An article in Rail Technology Magazine by Ken Munro, a security entrepreneur at Pen Test Partners, describes how customer communication networks can be used to take over trains and have catastrophic consequences – easy access to passenger credit card information, CCTV footage, and even the messages shared by crew across the network. He has a handy checklist of how to make sure the wi-fi infrastructure is safe.
LBM
That is deeply scary
( And, I think, applies to a lot of other “Free” WiFi networks, too ( e.g. In pubs? )
@LBM: Greg T: I’m surprised this is even remotely news. Badly configured networks cause a lot of problems. This is nothing new, it has become very obvious since Wifi was invented but was quite common even before then.
Every single IP address out there (except non-routed private IP’s) is scanned port by port several times a day by nasties looking for vulnerabilities. Similarly if you have a Wifi network at home somebody will try and break into it.
Why AI is a growing cyber threat to public transport. Public transit is facing increasing levels of hacking from artificial intelligence (AI) and automated attacks, says a leading cybersecurity expert.
I think the above is a Monty Python series 2, episode 12 issue.
Signed an actual cybersecurity expert of actual long standing.