• October 20, 2021, 03:39:06 AM

Author Topic:  Preferred implementation of a distributed computing fractal program  (Read 224 times)

0 Members and 1 Guest are viewing this topic.

Offline LorenDB

  • Fractal Freshman
  • *
  • Posts: 7
    My fractal program, fracture (
https://github.com/LorenDB/fracture) is hopefully going to be a distributed computing application. I'm trying to decide how to implement this. I've looked at MPI, but I'm not sure if it's a good fit for what I want. I'm envisioning something like this:

  • Ad-hoc networking; nodes connect to each other automatically and the network can handle nodes dropping offline.
  • Support for multiple clients connecting to the server network at one time; jobs would be spread out among servers more or less equally.
  • Possibly communicating via HTTP with JSON as the body (e.g. REST API).

This is a flyover at 30,000 feet, if you will, but I think it covers what I've been thinking about.

The problem with MPI is that, from what I've found in my research, it doesn't really do what I had envisioned: it requires pre-configuration of a hosts file, which precludes ad-hoc networking; it is designed to act like a single program running on multiple nodes, which would make the process of having multiple clients more difficult, although not impossible; and it has an API that is decidedly C oriented (I use C++). However, it's definitely well-supported, robust, and very well optimized, having been in existence for decades.

If I implement my own API/communication interface, I can make it more object-oriented if I want, add ad-hoc support, and potentially make it extensible and even turn it into a protocol that is application-independent, meaning that some of the apps I've seen mentioned here and elsewhere (e.g. Kalles Fractaler) could have support added.

Does anybody have any opinions on which path I should take? I'm currently leaning towards homebrewing something but I'll consider any suggestions.

P.S. I have attached a few images of renders done by fracture, plus a screenshot of the window in action. It's still very much a work in progress, but it's coming![/list]

Linkback: https://fractalforums.org/index.php?topic=4436.0

Offline quaz0r

  • Fractal Feline
  • **
  • Posts: 185
Re: Preferred implementation of a distributed computing fractal program
« Reply #1 on: October 01, 2021, 06:35:37 AM »
ive thought about this before...i think it really only makes sense for the granularity of the work sharing to be on a per-image basis at least, if not even perhaps ranges of images, within the context of a zoom video or other animation / sequence.  which would be way easier to implement as well

Offline Alef

  • Fractal Flamingo
  • ****
  • Posts: 327
  • a catalisator / z=z*sinh(z)-c^2
    • My deviant art page
Re: Preferred implementation of a distributed computing fractal program
« Reply #2 on: October 01, 2021, 07:23:32 AM »
I have no idea but I fully aprove this.
a catalisator / z=z*sinh(z)-c2

Offline LorenDB

  • Fractal Freshman
  • *
  • Posts: 7
Re: Preferred implementation of a distributed computing fractal program
« Reply #3 on: October 02, 2021, 08:06:46 PM »
ive thought about this before...i think it really only makes sense for the granularity of the work sharing to be on a per-image basis at least, if not even perhaps ranges of images, within the context of a zoom video or other animation / sequence.  which would be way easier to implement as well

Do you mind explaining further what you mean? I'm not sure what the problem is here you're pointing out. My current algorithm hasn't been optimized to use perturbation or anything, so it just calculates each point individually. I've been splitting the area of the fractal that the user wants calculated into several areas and calculating them using QtConcurrent::blockingMap (which calculates each segment of the final image in a different thread). In my mind, I can just distribute the fractal areas over the network to different nodes.

In other news, I have run across another candidate: HPX. It looks very promising. It's object-oriented and written in C++. I was leaning towards homebrewing my own library but I may end up going with HPX if I think it is close enough to what I want.

Offline quaz0r

  • Fractal Feline
  • **
  • Posts: 185
Re: Preferred implementation of a distributed computing fractal program
« Reply #4 on: October 02, 2021, 08:18:44 PM »
hpx is awesome!  :thumbs:  i was going to transition my own program from std::thread to hpx just for fun, even though the main initial benefit would just be to gain a threadpool.

if you are going to do deep zooms iterating normally with arbitrary precision then you would stand to benefit from splitting the work out onto different nodes on a network, sure, since that work takes literally forever.  in any other case (hardware precision / perturbation), either the cost of sending all that data over the network will outweigh the envisioned benefit of splitting up the work across nodes, and/or the time it takes to render a single image is so minimal it makes the whole thing kinda moot.

Offline LorenDB

  • Fractal Freshman
  • *
  • Posts: 7
Re: Preferred implementation of a distributed computing fractal program
« Reply #5 on: October 02, 2021, 10:11:08 PM »
Yeah, I totally get your point there. Probably I could do something like "get width and height of the part of the fractal, and if it is greater than [some predefined number or something], then don't spread it out over all available nodes, although some could be used."

Even with perturbation, I figure that deep zooms (yes, I do want to get arbitrary precision working, although I haven't figured out how to get that working right with boost::multiprecision::mpfr_float) will eventually take minutes to complete even on my 8-logical-core laptop, so I do want to add some sort of distributed computing no matter what optimizations I make; after all, arbitrary precision zooms take a long time once you get in far enough! My math teacher wrote his own fractal program, and while I don't know what type of optimizations he used (I do have the source code, but haven't looked at it in depth yet), I do know that (a) it's optimized beyond simple normal iteration and (b) once you start zooming in, you can easily get up to 10 minute zooms on modest hardware (e.g. early-gen Core processors) (he had a fractal that ran for hours before it completed, I think).

By the way, do you know if HPX supports dynamic networking? I haven't figured that out yet.

Offline LorenDB

  • Fractal Freshman
  • *
  • Posts: 7
Re: Preferred implementation of a distributed computing fractal program
« Reply #6 on: October 03, 2021, 12:09:39 AM »
OK, let me take back what I said about perturbation.

I looked up SuperFractalThing on the Internet Archive and got a link to the SourceForge page. I downloaded the jar and ran it and wow! That program renders fast! I had no idea that perturbation could operate so quickly at such high precision! I am going to try to implement that this evening and see if I can get similar results.

Offline LorenDB

  • Fractal Freshman
  • *
  • Posts: 7
Re: Preferred implementation of a distributed computing fractal program
« Reply #7 on: October 03, 2021, 03:02:29 AM »
Let me take that back. I don't really understand what I need to do to implement the perturbation theory, so that's off for now.

Offline pauldelbrot

  • 3f
  • ******
  • Posts: 3173
Re: Preferred implementation of a distributed computing fractal program
« Reply #8 on: October 03, 2021, 05:06:40 AM »
he had a fractal that ran for hours before it completed, I think).

Hours, you say?

<smh>

Kids these days ...

<mutters something about a two-month render job>

:)

If you're willing to explore beyond the confines of the C family of languages, you might want to take a look at Go and Erlang. Those languages are purpose-designed for implementing distributed processing systems.

(Currently active render: a Mandelbrot deep zoom, magnification 2x1053, 1080p HD resolution, minimum iterations around 16k, using perturbation and series expansion. ETA 10 minutes, just passing the 2/3 mark now, so half an hour or so total. On a nearly-new 12-core desktop tower, not some craptop. It might be a combination of the iterations and the 100-samples-per-pixel render quality making it take this long. :))

Offline gerrit

  • 3f
  • ******
  • Posts: 2469
Re: Preferred implementation of a distributed computing fractal program
« Reply #9 on: October 03, 2021, 07:50:28 PM »
Hours, you say?

<smh>

Kids these days ...

<mutters something about a two-month render job>

:)
I think I once did a 2-3 mo render, a low resolution Mandelbrot picture at depth \( 10^{2000,000} \) or so. Picture should be in my image thread somewhere.

99% of the time was spent on the reference orbit which can't be parallellized with many threads. You could try full (extended) precision rendering on a cluster but even if you have one pixel per node, those will take as long as the reference orbit and that's the best you can do.

Ultra Fractal can do distributed renders but I never tried it.


xx
Distributed computing for Mset area bound

Started by marcm200 on Collaborations & Jobs

0 Replies
471 Views
Last post May 20, 2020, 09:41:33 AM
by marcm200
xx
How would quantum computing affect fractal rendering?

Started by greentexas on Programming

1 Replies
772 Views
Last post December 12, 2017, 01:40:20 AM
by eiffie
clip
Fractal program for Windows

Started by kosalos on Programming

18 Replies
1281 Views
Last post June 03, 2019, 11:05:44 PM
by LionHeart
clip
WebGL Fractal Program

Started by kosalos on Programming

8 Replies
172 Views
Last post September 18, 2021, 01:18:59 AM
by 3DickUlus
xx
Made a basic fractal program some time ago

Started by Fluoroantimonic_Acid on Programming

3 Replies
282 Views
Last post June 12, 2020, 02:59:31 AM
by Fluoroantimonic_Acid