Two companies can each sell you a “penetration test” and deliver completely different things. One is an expert spending days trying to break into your systems. The other is a scanner’s output with a logo on the front. Both produce a PDF, and from the outside they can look much the same.
Yes, we sell penetration testing, so read this with that in mind. But everything below applies just as much when you’re assessing us, and we’d rather you asked these questions than not.
Scan or test? Know what you’re buying
A vulnerability scan is automated. It compares your systems against known problems and reports what might be wrong. It’s cheap and fast, and worth running all the time.
A penetration test is carried out by people mimicking a real attacker. NIST’s guide to security testing draws the line clearly: scanners only check whether a vulnerability may exist, while a penetration test goes on to exploit it and confirm that it does.
That difference matters most in the things machines miss. A scanner rarely knows that one customer shouldn’t be able to see another’s invoices (we wrote about that here), or that three low-risk issues combine into one serious problem.
You may not need one yet
A good tester should tell you this. If you haven’t patched your servers, don’t use multi-factor authentication and have never run a scanner, you’re paying a specialist to find things a free tool would find.
Do the basics first. A test is much better value once the easy problems are gone, because the time goes on the difficult ones.
The other useful timing rule is to test before a big change goes live, or after one. A test only describes your systems as they were on those days. If you ship every week, it starts ageing right away.
Decide what you want to learn
“We need a pentest” isn’t a goal. These are:
- Can one customer reach another customer’s data?
- Could a stolen employee laptop lead to our production database?
- What can an outsider reach from the internet with no account at all?
- We need evidence for a customer or an audit. What will they accept?
Your answer shapes the scope. Be careful with a scope written mainly to produce a clean report: an attacker won’t limit themselves to the parts you agreed to.
How much should the tester know?
- Black box: the tester starts with almost nothing, like an outsider. It sounds the most realistic, but a lot of your budget goes on discovering things you could have just told them.
- Grey box: the tester gets ordinary user accounts and some documentation. For most web applications this is the best value, because it allows real testing of what each user can reach.
- White box: the tester also gets source code and architecture documents. It finds the most per day, and is hardest for a real attacker to match.
Real attackers have plenty of time, and you’re paying for a few days. Sharing information is how you make up for that difference. It isn’t cheating.
Eight questions to ask before you sign
- Can we see a sample report, with client details removed? This is the most useful thing you can ask for. If it reads like raw scanner output, that’s what you’ll get.
- Who will do the testing, and can we speak to them? Ask about the actual person, as the one who sold you the project may not be the one doing it.
- How much of the work is manual? Tools are normal and sensible. What you want to hear is what a person does after they run.
- How do you handle production systems and our data? You want clear rules on what’s off limits, who to call if something breaks, and how findings are stored, encrypted and deleted.
- What happens when you find something critical? The answer should be “we call you that day”, not “it’ll be in the report”.
- Is a retest included? Otherwise you won’t have proof that your fixes work.
- Will you explain the findings to our developers? A live walkthrough can be worth more than the document.
- Is there a signed authorisation and contract? Written permission, scope and confidentiality terms protect both of you. Testing without them can be illegal, so a tester should insist on them.
Red flags
- A guarantee that you’ll “pass”, or that you’ll be secure afterwards. Nobody can honestly promise either.
- A fixed price quoted before anyone has asked what your system does.
- A very short turnaround for a large application.
- No questions about your business, your users or what would actually hurt you.
- A sample report full of generic text and very little evidence.
- Credentials with no people behind them. Certifications are a useful signal, but you’re hiring a person’s judgment. Ask how they would approach your system.
What a good report contains
- A plain-language summary that a CEO can read and act on.
- Findings ranked by real risk to your business, beyond the raw CVSS number.
- Proof for each finding, with steps your developers can reproduce.
- Specific fixes rather than “implement proper input validation”.
- What was tested and what wasn’t, so nobody assumes more coverage than there was.
- What held up. Knowing which defences worked is useful too.
Get ready, then get your money’s worth
A little preparation saves a lot of paid time. Have test accounts ready for each role, with realistic data. Tell your hosting provider if its terms require it, and warn your own monitoring team, unless testing their response is part of the goal. Name one person who can answer questions quickly.
Then fix what you find, and retest. A report with no follow-up has cost you money, and now also documents exactly how to break in.
A good penetration test won’t prove you are secure. It tells you how a capable person would attack you today, and what to do about it before anyone less friendly tries the same thing.