TLDR; Too many scope and reporting restrictions on testing attenuates both noise and signal.
I’ve tried a few crowdsourced testing environments - as a tester.
And I’ve tried a few crowdsourced security testing bug bounty environments - as a security researcher.
Unfortunately both of these environments create a management system that limits the defects that can be raised.
i.e.
- when taking part in a BugBounty - functional defects are out of scope.
- When taking part in a crowdsource testing project - so many defect categories are out of scope
If I was a company outsourcing to these programs, I would view that as a risk.
Example 1 - Crowdsourced Testing Security Defects
I was taking part in a Crowdsourced testing project, and only functional defects in a particular domain range were accepted.
I took the opportunity to perform some exploratory testing, looked under the covers, saw some api calls, amended them a little and was able to see the user details of all the users in the test environment. That is a security defect. I amended the URL to the production system and was able to see the user details of the registered users in the live environment. That is a fairly serious security defect.
I raised it as a bug. And it was rejected.
I actually had to ‘fight’ to have it acknowledged as an issue, so that the crowdsourced testing company would pass it through to the client company. I did this by marking it as a ’technical’ defect.
The company however, was only interested in functional defects. And they rejected the defect.
I think the company thought I was advocating for an out of scope defect for the money. At this point I should explain that defects were being awarded the princely sum of $5 per defect. I wasn’t doing this for the money.
I was amazed that both the client company and the outsourced company could possibly ignore this
I think they may have eventually awarded me $5. But since that is lower than the minimum payout from the crowdsourcing testing company I don’t think I ever received money for this.
The point is the system in place: for dictating the scope of testing, and managing the reporting of the defects; limited the defects raised and some of them were clearly important.
Crowdsourced testing sites normally have ratings for the testers. If you continually raise out of scope defects or defects that get rejected then your rating goes down, and your access to higher paying testing projects goes down. This incentivizes the testers to stay rigidly within scope, and they may then not explore the system hard.
Examples 2,3 and 4 - Functional Defects in Bug Bounties
I’m brushing up on my security testing skills and diving into BugBounty programs. And I face the same issue, but this time. I find functional defects, that I have no scope for passing on to the company.
Example 2 - a web application
- A 500 response on a particular system request.
- This doesn’t expose any information about the system. Can’t really be classed as an exploitable problem.
- I did try to raise it by categorising it as a potential denial of service target because responses take 2.5 - 3 seconds to return, instead of the normal 200 - 300 ms. (I was scraping the barrel trying to squeeze a defect in through the process).
- And as expected the defect was rejected.
- If I was working on this project as a tester then I would be pushing to have this fixed because it should be returning a 404 and as a developer I think that 500 errors are something to avoid.
I did try and raise this separately with the company. But had to contact someone working at the company via LinkedIn as there seemed no other route.
I assume the defect is not a big deal but one never knows.
Example 3 - another web application
Another minor defect, which I did not raise, in a different web app:
- Terms and conditions acceptance can be bypassed from the GUI
- When registering on a site I was able to bypass the terms and conditions acceptance box
- This might become an issue with GDPR.
- Since the login form doesn’t even pass that information through to the server anyway (regardless of it being ticked or not), I didn’t think it was a defect the company would care about.
- I didn’t raise it with the company or go outside of the bug bounty reporting system.
I don’t think this could be fudged as a ‘security’ issue. I do think it is a compliance issue, but not one that could be reported through a Security Bug Bounty programme.
Example 4 - a mobile application
Since mobile apps don’t always get a lot of traction on bug bounty programmes they can make good targets. So I tried a mobile app.
- Mobile app email system integration was fixed to default email app, but I use a different application so had to go through hoops to continue the registration process. This is a functional flow defect that would prevent some users from registering on the app.
- Mobile application crashes after triggering an SMS verification code. I tried to push my registration through by sending API requests from ZAP, but I couldn’t proceed any further.
With the mobile app. I saw that review on the app stores, from people in the real world, were experiencing both these issues, so I didn’t think it was worth me raising them.
More Incentive
The Bug Bounty programs however have a system that actively discourages raising functional defects. Security researchers are given a rating. If you raise incorrect defects then your rating goes down, thereby guaranteeing that functional defects will not be reported, if they are noticed.
Back in the Olden Days
When I worked on waterfall projects with some of the big heavy weight test management tools with projects using big heavy weight processes. We often found ourselves in the situation where we would spot something ‘off’ in the application under test. We’d have to go off script to investigate it. Find a problem. Then be unable to raise the defect because the defect had to be cross referenced to a step in a script.
Sometimes we would go through the hassle of creating new test conditions, cross referenced to a requirement, and write a test case and then create a script that would cover this. In order to raise the defect.
Sometimes we would squeeze it into the script we deviated from at the step we deviated and hope that the person who was going to re-test the defect when it was marked as fixed would follow the steps written in the defect report and not the script it was cross referenced to.
With either of the ‘sometimes’ scenarios. The management and reporting ‘system’ in place was making us jump through hoops to report valid issues.
In the Real World
We really don’t want process hoops getting in the way of useful information.
I understand the need for controls when using crowdsourcing sites.
Companies using these, perfectly viable avenues for additional testing and information, need to be aware that the crowdsourcing system is preventing them from learning all they could. The system which is designed to prevent noise interfering in the system, is also losing valid signals.
From a personal perspective, I find this annoying because at the moment I only seem to find security defects during functional testing (or for sites not in a bug bounty program), and my security testing mainly seems to identify technically based functional defects. But that’s life. I’ll survive and adapt.
Crowdsource testing can add value, but it also offers the risk of a narrow scope, and if you are unaware of this then it might bite you in the future.
