I like work; it fascinates me; I can sit and look at it for hours
Ramblings of
Alex Lovett
RSS
Twitter
Tumblr
Youtube
LinkedIn

Navigation:

Up a Level - Store - Experiments

Documents:

Lumen - Website - WheelReview - Painting_with_Light - Drawing - Food - GKN - Inspiration - House - Valideus - RoundTree - Reality_2.0 - Math_Art - Lilly - Unity5 - GameDesign - WishList - Fridge - XFactor
Show comments

Warning serious NERD article ahoy!

Some major changes to the 'blog' fixing a few dozen problems, all very exciting stuff. Most confusing aspect was getting my lazily loaded images ( only the visible images load till you scroll the page ) working with read it later services like Pocket and Readability. I found a work around in the end for Pocket using PHP's $_SERVER['HTTP_USER_AGENT'] and detecting if it is PHP Pear grabbing my site, nasty I know, but the only way to do it for now. Pocket appears to use PHP behind the scenes to parse and so It's identity is returned as 'HTTP_Request2/2.2.0 (pear.php.net/package/http_request2) PHP/5.3.28' Readability is a bit nicer in that it reports as 'Readability/6534b2 - readability.com/about/' and Readability has the handy tag to ignore content 'class="entry-unrelated'

And I just added a comment system finally! I keep seeing people use Disqus so gave it a try. Turns out it cannot show more than one comment box per page which is no good, but found a work around... after a few hours

:-/

Nothing is ever simple.

In any case here is a zoomed out view of the code that runs the blog. The White code is the pre processor that is made of very squirrely looking AppleScript code and ran on my Mac. The grey code is a mix of PHP HTML and CSS which runs on the server side and reads in the XML produced by AppleScript. As you can see it is all a horribly ultra nested sprawling mess, though in my defense I did start this a decade ago and hadn't ever read a single thing on programming.

The whole blog is generated automagically out of folders with TextEdit documents in, just some RTF files that can hold Images and Videos. So I simple write an entry in a normal text editor, copy paste an image or video in. Done. And my AppleScript massages this into XML and export all the images, and videos, upload the videos to youtube and sync the whole site to the web host and so on.

Ah what kind of crazy bastard sets out and writes his own blog platform and website from scratch. Well Wordpress did'nt exist when I started, and I like images a lot on a blog, so copy pasting and ease of authorship was a must. Otherwise I wouldn't bother. But admittedly scary amounts of hours have probably been sunk into this over the years, bit by bit adding things.

I just made the whole applescript side multi threaded, take that crappy performing applescript code! It will now spin off a separate thread for each rtf file it is processing. Speeding the whole thing up by 100's of times. So that's nice. So no longer such a monolithic single block of code as I broke it up into several independent pieces. Go team me.



Below, See, makes perfect sense, The joy of escaping escape characters that are escaped. This abomination is what helps clean up an RTF file into something more legible, It somehow works



I also added caching for PHP page rendering using :

Link: www.phpfastcache.com --- www


As my site does a stupid amount of heavy string searching, replacing and other guff that is scary and abhorrent. So if say, I become really popular and everyone liked me so much as to visit my site 100 times a second, well the server would explode, and then my adoring fans would be deprived of access to my wisdom.
So typical processing of a page takes 0.2 seconds on a bad day. With caching... 0.001.

I wrote an AppleScript to spawn curl processes to assault my site as a test and total up the page rendering times with and without caching:

So for example with 5 users hitting the site every 0.1 seconds for a duration of 5 seconds you get:

39 seconds taken to run 5 seconds of access
5 concurrent accesses every 0.1 sec for 5 sec
556.525 seconds for sum of all load times for 250 page loads
2.2261 seconds avg per download

Without caching it takes 2.2261 seconds to serve up a page that took 0.2 seconds to render when only 1 render was being done at once, this will be due to contentions for disk access that are only revealed when multiple access happen simultaneously

So it basically took 40 seconds to serve up 550 seconds worth of page renders so it appears to be effectively achieving 13x at once ( on a 8 core Mac )

Now lets try with caching on:

7 seconds taken to run 5 seconds of access
5 concurrent accesses every 0.1 sec for 5 sec
17.886 seconds for sum of all load times for 250 page loads
0.071544 seconds avg per download

Nice, so now it took 7 seconds to run 5 seconds of access, bare in mind I am using the same machine to download the pages as serve them, it probably means it is coping 100% now
A difference of total page rendering time of 18 seconds versus the uncached 556 seconds of cpu time
It takes 0.07 seconds to serve up a page that took 0.001 seconds to render when only 1 render was being done at once now

Lets see how far I can push the cached load, given the over head of downloading from this many threads it likely the biggest issue now

20 seconds taken to run 60 seconds of access
5 concurrent accesses every 0.05 sec for 60 sec
8.178 seconds for sum of all load times for 6000.0 page loads
0.001363 seconds avg per download

CPU Load during serving up the cached pages on the left half, and uncached right half, plenty of room to spare for the cached, I suspect the red is the server and green will be me downloading the results. And my machine pretty much locks up while serving the uncached so.
Imagine how the poor typical 1 x 2ghz server would handle the uncached instead of a Mac Pro with 8 x 3ghz would not be pretty.


It would appear in any case that the majority of the cpu use is due to the overhead of downloading not the serving of the page. But it was a 'fun' exercise to stress test.

Now all of the above was tested on my local web server, lets try the remote Amazon EC2

With it uncached it hung the server really badly with some requests taking 40 seconds, and many just getting a blank return, only 300 out of 2000 came back! and those averaged 13 seconds per load

462 seconds taken to run 60 seconds of access
10 concurrent accesses every 0.3 sec for 60 sec
4818.08 seconds for sum of all load times for 373 out of 2000.0 page loads
12.917104557641 seconds avg per download

Lets run it again as I think my machine is struggling a bit with max proc limits

360 seconds taken to run 60 seconds of access
10 concurrent accesses every 0.5 sec for 60 sec
5007.7 seconds for sum of all load times for 370 out of 1200 page loads
13.534324324324 seconds avg per download

Interesting it craps out at the same 370 downloads, maybe some kind of spam protection on the server as I am loading all this from one IP

After thrashing the Amazon server, trying to access my site from a mobile phone ( so different IP ) fails completely. It is basically shutdown. Nice one amazon. And the whole time the Amazon status of the server shows as ok and not under much use.

Definitely odd as if I run again even a few small requests it takes 30 seconds per request ( I timeout curl after 30 seconds ):
88 seconds taken to run 1 seconds of access
10 concurrent accesses every 0.5 sec for 1 sec
560.322 seconds for sum of all load times for 18 out of 20 page loads
31.129 seconds avg per download

Below are a some cached runs thru, the first run thru is still showing hurt from the prior uncached stress test, after that they are fine again

11 seconds taken to run 1 seconds of access
10 concurrent accesses every 0.5 sec for 1 sec
31.288 seconds for sum of all load times for 20 out of 20 page loads
1.5644 seconds avg per download

8 seconds taken to run 1 seconds of access
10 concurrent accesses every 0.5 sec for 1 sec
0.288 seconds for sum of all load times for 20 out of 20 page loads
0.0144 seconds avg per download

9 seconds taken to run 1 seconds of access
10 concurrent accesses every 0.5 sec for 1 sec
0.389 seconds for sum of all load times for 20 out of 20 page loads
0.01945 seconds avg per download

9 seconds taken to run 1 seconds of access
10 concurrent accesses every 0.5 sec for 1 sec
0.336 seconds for sum of all load times for 20 out of 20 page loads
0.0168 seconds avg per download

And now for fun lets try a free web host ( opened host ) using the cached copy

11 seconds taken to run 1 seconds of access
10 concurrent accesses every 0.5 sec for 1 sec
0.104 seconds for sum of all load times for 20 out of 20 page loads
0.0052 seconds avg per download

Ok lets ramp it up for 60 seconds
152 seconds taken to run 60 seconds of access
10 concurrent accesses every 0.5 sec for 60 sec
3.476 seconds for sum of all load times for 1117 out of 1200 page loads
0.003111906893 seconds avg per download

ok so that's actually better than Amazon.... bastards
Lets try uncached now:

Webhost refuses to respond now

:-P

ok maybe not better than Amazon.. wait that does appear to be a DOS prevention as I can access site from my phone still but not the computer I was using to thrash the server

Anyway

I could just have used a service like load impact

:-P

which lets you test for free unto 250 users per second ( not connections but users with some kind of average page download per minute ) and pay for more based on use:

Link: loadimpact.com



But it doesn't hit the server as hard as my own script does

When using LoadImpact at 250 users graph looks like this, resulting in 1478 page loads over 2 minutes ( When not caching page )




Ouch over 7 seconds load time, though my internet is running off crappy ADSL at the moment, waiting for fibre to be installed, and this doesn't include serving up the actual HTML and Images, that's just to process the page and return a number

And now with it cached:





Ok lets test my Amazon EC2 hosting



huh it is still loading stuff from my domain... *tries again*



Ah that is better

And with it uncached:



Identical, but I suspect the test was still cached

CPU graph on Amazon:



And for future references, here is what the site currently looks like:




Show comments for 'Blog Engine'
Tags: - Programming - Games - Shadowood - Unity
Show comments

Ah so simple, yet so not so simple.

I have a lovely fluid solution after many hours, and it renders all onto a flat 2D plane. Now I hadn't really thought much about how the rest of the 3D scene will sit with this fluid. I can't just have it all float over the top of it nor hide underneath it, I need it to merge with the fluid, and as It's a 2D plane by default objects just stick thru it with hard edges ruining the illusion of the 3D for example:



It is quite jarring where the cube suddenly enters the fluid. So I need to mask the fluid away based on the proximity of the cubes or the depth they sit at... simple.. I thought. Nope requires editing shaders which is never simple. But alas I finally prevailed and figured it out, the effect is almost unnoticeable, which is good because it is meant to be something you do not notice, versus clearly noticing how bad the above images is. Also the particles will fade out gently around the cubes not just suddenly get clipped by them.

So it looks better like this


Done with help from this site here:

Link: chrismflynn.wordpress.com --- fun-with-shaders-and-the-depth-buffer



Or even betterer like this:




Wonderful, job done ( aside from figuring out how to vertex color particles within the shader now )

Except none of it runs on the actual iOS device DOES IT!!!!!!

So back to the drawing board, and to find a way to capture camera depth that works in iOS and PRAYING all the shader jiggery I am doing will continue to work.

And how nice do a few cubes look ( only after 200 hours in Unity too! )



Phew redid pretty much everything from scratch and got it working on iPad

:-P



And Just had the idea to have the fluid depth take into account the brightness of the fluid, so brighter fluid is closer to the camera, to help break up the linear fog, works quite well

Now to just got get particle depth working again, and tidy things up ( remove all the fudge and hack )



Just added Velocity Texture support

Can now do kickass never seen beforeness such as but not limited to: suck in fluid like a crazy blackhole upon hitting a cube, spurt out red jets of fury if you miss a box... and other such nonsense

As it just instances / splats a texture directly into the velocity field such as:



The colours represent the angle to direct the fluid. Hurt my brain working out how to draw such an image in Photoshop but tadda



The above is basically pushing fluid out in allow directions uniformly

FINALLY got the fluid simulation working with none square area... took 17 hours of docking about poking in the dark and fumbling about banging my head

>_<



FluidAspectWorking



The way I see it you have normal programming like JavaScript lets say that's difficulty 1, then twice as hard as that is doing it really well with proper abstraction and organisation with good performant code being twice as difficult at level 2, then you have writing graphics shaders at level 3, implementing a fluid sim with a shader level 4, writing the formulas used for the fluid sim level 5, and understanding that fucking picture at level 10

But I do keep surprising myself, after sticking with a few 'impossible' tasks for a while, they become... understandable. Most of what makes things look so scary or beyond your capacity is not understanding the language something is written in. When new things use a different vocabulary it creates the impression that it is forever beyond comprehension. Time and time again I have felt this way despite knowing full well I can learn new languages and then suddenly the impossible becomes simple.

I am probably the only human being that has both the persistence and ability to get down and nerdy with a simulation like this, and still possess enough artistic ability and eye for detail to make it look good too... hell I might even be sensible enough to make something people would enjoy playing on top of that... but lets not get too carried away

Most normal people are scared off by how complicated adding feature to a fluid sim are, I am clearly immune to the normal healthy desire to run away screaming

even though this was a side embellishment to a boring whack a mole game, I'm thinking I need to make something that uses this fluid sim as the key gameplay mechanic

hubris aside though... I spent 17 hours just trying to fix the aspect ratio / render in a none square area

:-/



Oh and then I checked the performance increase of rendering less pixels / cells in the fluid sim with this fantastic optimisation:



Yup, around 1ms at best..... FFS

On the iPhone the increase is probably a bit higher, but even still.... premature optimisation strikes again

Testing with higher resolution simulation and the result is:



Ooh 2 more frames per second

:-/



Well at least it is making a difference... glass half full.. and it is saving some memory... *sigh*






Show comments for 'Foggy Fluid'
Tags: - Programming - Games - Shadowood - Unity
Show comments

Finally gave in after being driven up the wall by MonoDevelop for the last time... Moved to Microsoft VisualStudio on Windows, ah now everything just works-ish

Took forever to get communication working between VisualStudio on Windows and Unity on the Mac, so that you can double click debug logs entries and files in Unity and have it jump to the right Project / File and Line Number in Visual Studio running in a Virtual machine on the Mac. Had to write my own PC EXE acting as a bridge and all kinds of funky stuff... But I just couldn't stand MonoDevelop or Xamarin anymore, they are such junk.

Between that, and upgrading to the new Unity... then upgrading to another new Unity, then upgrading the heart of the FluidSim code from JavaScript to C# version, and moving to iOS 7, and a new iPad.... eaten all my time! No game work done!

And I'm really fretting and struggling around the concept / design / look and feel. I need to nail it down now, but it is impossible to draw or even photoshop mockup any of the themes I have in my head.

Maybe better approach is to have a few visual ideas / concepts and start playing around in Unity bringing them to life, then in a serendipitous manner discover and stumble upon something I like that I may never have thought of otherwise. Just get the wheels moving instead of being stuck in planning limbo.





Show comments for 'Visual Studio Huzzah'
Tags: - Programming - Games - Shadowood - Unity
Show comments

Toying around with various visual concepts in this mockup, I have since abandoned this concept:


Playing around with the idea of using DOF / Spherical Aberrations or 'Bokeh' effects




Show comments for 'Squishy Concepts'
Tags: - C4D - Vray
Show comments

Continued toying with Vray here are the results, first with post effects and second without:







Show comments for 'Bathrooms'
You have reached the end of this page - But there's more! Click Older for more
Subscribe to my News Feed.. or screw you then!
I don't know why I did it, I don't know why I enjoyed it, and I don't know why I'll do it again.
Copyright © 2006 - 2017 - Alex Lovett
Site and content designed, built and massaged by
Alex Lovett
( HD6 / HeliosDoubleSix )
contact me by email:
Page Rendered in: 0.071 seconds, like a boss