Disclaimer out of the way: No sense of entitlement here, every time I see a challenge like this I expect to fail. I'm really, really happy that people (Stripe, here) offer their time and resources to entertain us.
I'm a proud owner of the CTF2 shirt.
This time I couldn't finish level4. My takeaway is: I invested too much time and shouldn't have done it. I guess I'll pass next time when a challenge w/ a deadline starts.
Level0 was a nice warm-up.
Level1 was immensely satisfying and interesting, but the pvp challenge was .. well.. dead. By the time real life let me join others were throwing GPU based miners at the problem and traffic was constantly heavy anyway. So amazing level, the arena type thing was imho not a good idea.
Level2: Here the problems began. Local tests and remote tests diverged, plus this level was pushing you to use node.js. Will come back to this later. The actual challenge was interesting, so my issues here were infrastructure and .. language
Level3: This was absurd. Again, the challenge sounded interesting. The sample solution you're supposed to fix came in Scala and you had some constraints (time/memory). I had a couple of (locally) working solutions that never made it remotely. After hours and hours I gave up and solved this challenge - by executing grep for each request. Yes, I called grep and parsed the output, didn't index a thing. That felt .. bad.
Level4: Didn't solve it. Utterly frustrating. The consensus (hah.. hah.. get it?) seemed to be that raft (goraft) would be the way to go, but I invested again hours (8? Probably a lot more) and didn't make it. Problems with the deployment were mentioned on the channel again and again, goraft has even open pull requests and recent tickets from the stripe guys, as far as I can tell. Dependencies were a mess. Someone built the reference solution to this level and found bugs in goraft (see Github, goraft issues)? I .. don't think this was a good idea. Plus, forcing unix-sockets into the game (where everything around it doesn't support those and hey - if you want to have a 'curl for unix sockets' you end up finding another go project that just didn't work for me) was really messy.
So, putting aside the fact that I wasn't smart enough to clear level4 (it was obviously possible!): The templates came in nodejs/Scala/go. I knew a little about nodejs, nothing about the other two. Ripping the provided framework out was kinda possible, but a pain in the ass to debug (you could run a script that installed stuff remotely - I tried solving level3 in Clojure and it was a miserable experience). So challenging tasks plus new languages, plus deadline, plus 'servers are overloaded and cannot answer': This was just way too much for me. I still tried to solve level4 until the server fell over again and the CTF was finally closed.
My take away: All the ideas, the problems, were cool and interesting. But I guess even the Stripe guys were surprised to see the load/spikes/outages - probably a consequence of the 'distributed'/'load handling' type of problems, versus the 'try to break in' challenges last time. It hooked me with the leaderboard, nice website, good problems. But it was an exercise in frustration for me (and Scala/Go are _really_ not for me..).
That aside: Thanks a lot, Stripe. I cannot even begin to imagine how much work you guys invested in the preparation and administration/maintenance. I do consider the extension fair - and didn't gain a thing from that (was already at l4 at that point).
This is good feedback. Honest feedback like this helps us really understand how successful we're being with CTF.
One of our design goals was to make it hard to have something that worked locally but not remotely. It seems like we didn't quite make it, and that caused a lot of frustration. Sorry for that — it gives us something to work towards next time. In any case, I hope you walked away knowing some things you didn't before :).
If anyone else has feedback, I'd love to hear — feel free to ping me at gdb@stripe.com.
Level1 was immensely satisfying and interesting, but the pvp challenge was .. well.. dead. By the time real life let me join others were throwing GPU based miners at the problem and traffic was constantly heavy anyway. So amazing level, the arena type thing was imho not a good idea.
PvP challange dead .. but heavy traffic? Sounds like a contradiction. I think it's a flawed deduction that the arena was a bad idea because it took more than a casual glance at the challange to be competetive. For me, level 1 was definitely the highlight of the CTF. I was one of the few who wrote a GPU miner, and the constant competition of improving my miner vs. the other active participators was extremely exciting.
Had zero experience with programming for the GPU, or with OpenCL. Took this as an opportunity to learn both. I was surprised both with how amazingly fast the GPU is, and how amazingly buggy the AMD OpenCL compiler optimizer is. (Found multiple bugs where it generates memory corrupting code.)
> Ripping the provided framework out was kinda possible, but a pain in the ass to debug
This is very different from my experience. They give you a build.sh script that does anything you want and a container with open access to the internet. Anything you print to stderr is shown to you. I didn't use the provided code for any of the challenges, and thought it was awesome that they didn't force you into a particular implementation. It's just a reference to help you get started.
And hackers staying up all night to write openCL gitcoin miners? That's just awesome. I don't know how you concluded that pvp "wasn't a good idea". I thought that challenge was just fantastic.
- Ripping out the framework caused lots of issues for me. Maybe my choice was wrong? I actually (in spite of my complaints) used two 'interesting' languages for myself. I polished my ocaml for the miner and switched to Clojure for level3. The latter was hard, because the infrastructure was down a lot and you already had a hard time getting past 'Kicking of your trial' messages - only to fail because leiningen complains about the inability to verify ssl certs for example. Try to fix it (doesn't happen locally...), push, fail to kick of the execution of the build script.. Loop.
- The miner: As I said, I loved the idea of that challenge. The problem was highly interesting. My problem with pvp was mostly "There's no way to compete". When I joined it felt already like installing a bitcoin miner on my home machine. openCL based miners? That's changing the field of course and the only way to participate at that point is to follow - and I'm running a laptop here with an integrated Intel chipset for everything else. That totally ignores the fact that I don't know a thing about openCL or CUDA and would've started from scratch (as w/ Scala, as w/ go).
I certainly understand the fascination, but .. how many people _could_ compete against each other? The implementation (gpu based) and the network (git pull was sooo slow, same for push) made this inaccessible. At least for me.
I personally enjoyed learning more about the different languages that were used at each level. Granted I'd be meaning to try out node/scala/go in a more meaningful manner and the ctf problems were the perfect setting. The downside to this is that it can be difficult to learn a new language and solve the problem at hand.
I agree level4 was a bit undercooked and not as didactic as it could be. One could simply clear the level by using raft as a black box. Personally I enjoyed level 3 the most given the trade-offs that were present in solving it.
Disclaimer out of the way: No sense of entitlement here, every time I see a challenge like this I expect to fail. I'm really, really happy that people (Stripe, here) offer their time and resources to entertain us.
I'm a proud owner of the CTF2 shirt.
This time I couldn't finish level4. My takeaway is: I invested too much time and shouldn't have done it. I guess I'll pass next time when a challenge w/ a deadline starts.
Level0 was a nice warm-up.
Level1 was immensely satisfying and interesting, but the pvp challenge was .. well.. dead. By the time real life let me join others were throwing GPU based miners at the problem and traffic was constantly heavy anyway. So amazing level, the arena type thing was imho not a good idea.
Level2: Here the problems began. Local tests and remote tests diverged, plus this level was pushing you to use node.js. Will come back to this later. The actual challenge was interesting, so my issues here were infrastructure and .. language
Level3: This was absurd. Again, the challenge sounded interesting. The sample solution you're supposed to fix came in Scala and you had some constraints (time/memory). I had a couple of (locally) working solutions that never made it remotely. After hours and hours I gave up and solved this challenge - by executing grep for each request. Yes, I called grep and parsed the output, didn't index a thing. That felt .. bad.
Level4: Didn't solve it. Utterly frustrating. The consensus (hah.. hah.. get it?) seemed to be that raft (goraft) would be the way to go, but I invested again hours (8? Probably a lot more) and didn't make it. Problems with the deployment were mentioned on the channel again and again, goraft has even open pull requests and recent tickets from the stripe guys, as far as I can tell. Dependencies were a mess. Someone built the reference solution to this level and found bugs in goraft (see Github, goraft issues)? I .. don't think this was a good idea. Plus, forcing unix-sockets into the game (where everything around it doesn't support those and hey - if you want to have a 'curl for unix sockets' you end up finding another go project that just didn't work for me) was really messy.
So, putting aside the fact that I wasn't smart enough to clear level4 (it was obviously possible!): The templates came in nodejs/Scala/go. I knew a little about nodejs, nothing about the other two. Ripping the provided framework out was kinda possible, but a pain in the ass to debug (you could run a script that installed stuff remotely - I tried solving level3 in Clojure and it was a miserable experience). So challenging tasks plus new languages, plus deadline, plus 'servers are overloaded and cannot answer': This was just way too much for me. I still tried to solve level4 until the server fell over again and the CTF was finally closed.
My take away: All the ideas, the problems, were cool and interesting. But I guess even the Stripe guys were surprised to see the load/spikes/outages - probably a consequence of the 'distributed'/'load handling' type of problems, versus the 'try to break in' challenges last time. It hooked me with the leaderboard, nice website, good problems. But it was an exercise in frustration for me (and Scala/Go are _really_ not for me..).
That aside: Thanks a lot, Stripe. I cannot even begin to imagine how much work you guys invested in the preparation and administration/maintenance. I do consider the extension fair - and didn't gain a thing from that (was already at l4 at that point).