(Question) importing arbitarary channels from EXR files

  • 21 Replies
  • 313 Views

0 Members and 1 Guest are viewing this topic.

Offline claude

  • *
  • 3f
  • ******
  • Posts: 1255
    • mathr.co.uk
« on: September 17, 2019, 03:43:54 AM »
Now that KF can export raw iteration data to EXR, wouldn't it be cool to be able to import it into FragM for colouring?

Something to start bikeshedding from:
Code: [Select]
sampler2D kfNF; texture[example.exr] channel[NF] // FLOAT or HALF or normalized UINT data should be supported for sampler2D channels
usample2D kfN; texture[example.exr] channel[N] // unnormalized UINT should be supported for usampler2D channels
sampler2D kfDE; texture[example.exr] channel[DEX,DEY] // up to four channels could be packed into one texture, but they all need to be the same data type
sampler2D kfRGB; texture[example.exr] channel[R, G, B] // could also load KF's colouring for comparison/sanity-checking
// [R,G,B,A] should probably be the default EXR import channels, with A defaulting to 1.0 if unspecified

I don't particularly care about the syntax, only that I can read arbitrary channels from an EXR file.

It would be fine if the operation failed with an error message if the channels to be packed into a texture were of different types, or there was no such channel.

Linkback: https://fractalforums.org/fragmentarium/17/importing-arbitarary-channels-from-exr-files/3080/

Offline 3DickUlus

  • *
  • 3f
  • ******
  • Posts: 1377
    • Digilantism
« Reply #1 on: September 17, 2019, 04:24:16 AM »
That's an interesting idea  :thumbs:

So, right now we can import only the colour channels R,G,B,A , as the predefined structure, in reality it could be any data in the file, position tracking xyz or what have you.

What you suggest is "named" channel import? and it's up to the coder to address it in a sensible way, as iteration data or position etc...

from https://www.khronos.org/opengl/wiki/Sampler_(GLSL)

GLSL samplerOpenGL texture enumTexture type
sampler1DGL_TEXTURE_1D1D texture
sampler2DGL_TEXTURE_2D2D texture
sampler3DGL_TEXTURE_3D3D texture
samplerCubeGL_TEXTURE_CUBE_MAPCubemap Texture
sampler2DRectGL_TEXTURE_RECTANGLERectangle Texture
sampler1DArrayGL_TEXTURE_1D_ARRAY1D Array Texture
sampler2DArrayGL_TEXTURE_2D_ARRAY2D Array Texture
samplerCubeArrayGL_TEXTURE_CUBE_MAP_ARRAYCubemap Array Texture (requires GL 4.0 or ARB_texture_cube_map_array)
samplerBufferGL_TEXTURE_BUFFERBuffer Texture
sampler2DMSGL_TEXTURE_2D_MULTISAMPLEMultisample Texture
sampler2DMSArrayGL_TEXTURE_2D_MULTISAMPLE_ARRAYMultisample Array Texture

...these are our options for textures, maybe just adding some optional input for 1D as in filename + channelname, or to make it complete add all of these texture types and an option for channel name(s) for all?

Right now FragM only supports sampler 2D|Cube

So I would proceed as: add sampler1D to FragM, figure out loading 1 named channel from any file (image or data), then add channeling to the other existing (relevant) samplers and then make it work for all sampler types :)

Internally the code is already there it just needs to be re-arranged and enhanced a bit ;)

edit: of course we would have to add all of the relevant GL_TEXTURE options too  :-\
Fragmentarium is not a toy, it is a very versatile tool that can be used to make toys ;)

https://en.wikibooks.org/wiki/Fractals/fragmentarium

Offline 3DickUlus

  • *
  • 3f
  • ******
  • Posts: 1377
    • Digilantism
« Reply #2 on: September 17, 2019, 04:50:17 AM »
I confess, I'm not familiar with the KF data in EXRs, start on 2D maybe?
edit: I suppose 1D would be more useful for position tracking data rather than pixelxy data.

Offline claude

  • *
  • 3f
  • ******
  • Posts: 1255
    • mathr.co.uk
« Reply #3 on: September 17, 2019, 04:52:19 AM »
EXR images are all 2D afaik? just with more/fewer channels, each has its own type.  OpenGL GL_TEXTURE_2D can have 1,2,3, or 4 channels, all must be the same type.

Offline 3DickUlus

  • *
  • 3f
  • ******
  • Posts: 1377
    • Digilantism
« Reply #4 on: September 17, 2019, 05:02:18 AM »
 Texture 1D can too, 4 channels? yeah start on 2D, how to solve the sub-type? or range of sub-types as GL needs all the same.

int or double as the 4th channel with the rest float? I don't think GL will like that  :-\

edit: can just load the data to how ever many channels needed, up to max 4, and not refer to the color data, leave that for regular (current) sampler type

ok sounds really promising, not just for KF data but a lot of other stuff too :)

edit2: uhm yeah. it's finally soaked in to my tired brain , named channels, brilliant!
« Last Edit: September 17, 2019, 05:23:01 AM by 3DickUlus »

Offline 3DickUlus

  • *
  • 3f
  • ******
  • Posts: 1377
    • Digilantism
« Reply #5 on: September 17, 2019, 05:32:52 AM »
I think it might work well as...
Code: [Select]
sampler2D kfNF; texture[example.exr, NF] // FLOAT or HALF or normalized UINT data should be supported for sampler2D channels
usample2D kfN; texture[example.exr, N] // unnormalized UINT should be supported for usampler2D channels
sampler2D kfDE; texture[example.exr, DEXY  ] // up to four channels could be packed into one texture, but they all need to be the same data type
sampler2D kfRGB; texture[example.exr, RGB ] // could also load KF's colouring for comparison/sanity-checking

simply
Code: [Select]
sampler2D varName; texture[ filename.exr, channelName... ] where channelName is optional or multiple

are you suggesting specific channels? ie:KF channels that are named NF N DEX DEY R G B ? should be generic and not KF centric.

we'll have to do some hacking and see what comes up, I will have this on my mind all week but won't be able to put in any real keyboard time until the weekend.

Offline claude

  • *
  • 3f
  • ******
  • Posts: 1255
    • mathr.co.uk
« Reply #6 on: September 17, 2019, 07:40:29 AM »
should be generic and not KF centric.
absolutely!

note that EXR channel names are strings, not just single letters

Offline 3DickUlus

  • *
  • 3f
  • ******
  • Posts: 1377
    • Digilantism
« Reply #7 on: September 17, 2019, 05:35:01 PM »
Yep, I did get familiar with EXR internals for depth buffer to alpha channel, but out of format channel names are not really standardized eg depth is often referred to as Z and not DEPTH. Maybe can read the channel names and let user choose which ones they want to use for a sampler?

Offline claude

  • *
  • 3f
  • ******
  • Posts: 1255
    • mathr.co.uk
« Reply #8 on: September 17, 2019, 07:00:51 PM »
that could work, but a way to save it in frags as code too would be good (like a preset, perhaps)

Offline 3DickUlus

  • *
  • 3f
  • ******
  • Posts: 1377
    • Digilantism
« Reply #9 on: September 18, 2019, 01:00:29 AM »
Code: [Select]
sampler2D varName; texture[ filename.exr, channelName... ]
if channelName not found in filename.exr warn with log line that adds channel list read from file? saying "Channel must be one of..." nnn

edit: ...and the setting is saved in presets parameter files and image files along with the rest :) , of course it will fail in older versions of FragM  :-\
« Last Edit: September 18, 2019, 01:15:39 AM by 3DickUlus »

Offline 3DickUlus

  • *
  • 3f
  • ******
  • Posts: 1377
    • Digilantism
« Reply #10 on: September 19, 2019, 05:59:46 AM »
...actually there are a lot of samplers...

   100
         sampler2D
         samplerCube
   110
         sampler1D
         sampler3D
         sampler1DShadow
         sampler2DShadow
   130
         samplerCubeShadow
         sampler1DArray
         sampler2DArray
         sampler1DArrayShadow
         sampler2DArrayShadow
         isampler1D
         isampler2D
         isampler3D
         isamplerCube
         isampler1DArray
         isampler2DArray
         usampler1D
         usampler2D
         usampler3D
         usamplerCube
         usampler1DArray
         usampler2DArray
   140
         sampler2DRect
         sampler2DRectShadow
         isampler2DRect
         usampler2DRect
         samplerBuffer
         isamplerBuffer
         usamplerBuffer
   150
         sampler2DMS
         isampler2DMS
         usampler2DMS
         sampler2DMSArray
         isampler2DMSArray
         usampler2DMSArray

   in 330 but not in 300 es

         samplerBuffer
         isamplerBuffer
         usamplerBuffer
         sampler1D
         sampler1DShadow
         sampler1DArray
         sampler1DArrayShadow
         isampler1D
         usampler1D
         sampler2DMS
         isampler2DMS
         usampler2DMS
         sampler2DMSArray
         isampler2DMSArray
         usampler2DMSArray
   400
         samplerCubeArray
         samplerCubeArrayShadow
         isamplerCubeArray
         usamplerCubeArray

Offline 3DickUlus

  • *
  • 3f
  • ******
  • Posts: 1377
    • Digilantism
« Reply #11 on: September 21, 2019, 07:59:33 AM »
that could work, but a way to save it in frags as code too would be good (like a preset, perhaps)

an excellent regex tester, a must-have for the tool box https://regex101.com/r/D1kVJc/4/

code...
Code: [Select]
sampler2D varName; texture[ filename.exr, channelName(s) ]preset...
Code: [Select]
varName = Images/multichannel.exr RGBZ
I've got the preliminaries working, just about to the point where the GL stuff starts, log reports "...file using channel(s) RGBZ", but it just loads the file normally,  I want to make sure the GUI stuff is working first.

The sampler widget gets split into left and right pulldowns, the left is a comboBox with filenames( as is now) and the right is a comboBox with channel names :D
« Last Edit: September 21, 2019, 08:10:57 AM by 3DickUlus »

Offline claude

  • *
  • 3f
  • ******
  • Posts: 1255
    • mathr.co.uk
« Reply #12 on: September 21, 2019, 01:38:26 PM »
the right is a comboBox with channel names
I guess this should be extended to 4x comboBox for the 4 channels GL textures can have?

Offline 3DickUlus

  • *
  • 3f
  • ******
  • Posts: 1377
    • Digilantism
« Reply #13 on: September 22, 2019, 01:08:49 AM »
just putting together a commit for Development branch...

here's what I've got so far...

added optional channel spec to SamplerWidget
added optional channel spec to Parameters
reads and writes optional channel spec to settings
scans EXR file and creates channels pulldown
if user channel does not exist in file boarder of pulldown goes red

status: GUI only, I haven't coded any of the GL or special channel loading yet, just loads all channels as in default behavior

note: the channel spec can be adjusted to account for multiple channels or subset of All as a comma or semicolon separated list...
Code: [Select]
sampler2D varName; texture[ filename.exr, R;G;B ]At the stage now where this would be trivial to implement.

The only catch of course is that they actually have to exist, not sure yet how different format channels will work, float vs int, uint, half etc, I suppose OpenEXR can tell what format each channel is so will just have to add that and create the GL texture accordingly.

have a look, run from cmdline (no verbose) for some debug, outputting at where I left off...
Code: [Select]
"DisplayWidget.cpp" 728 loadEXRTexture Default:  "myTexture" "texture2.jpg" ""
"DisplayWidget.cpp" 734 loadEXRTexture Requested:  "myTexture" "Ditch-River_2k.exr" "All"
"DisplayWidget.cpp" 1645 setShaderUniforms Setting: "myTexture" "Ditch-River_2k.exr" "All"

Offline 3DickUlus

  • *
  • 3f
  • ******
  • Posts: 1377
    • Digilantism
« Reply #14 on: September 22, 2019, 09:12:06 PM »
Well.. got it working... but there are some hoops to jump through

OpenEXR::RgbaInputFile and OpenEXR::InputFile are both expecting HALF type in the file and convert, on-the-fly, to half as the file is being read (at least that's my impression from looking at struct Rgba) this will certainly play havoc with your KFR files  :-\ at least when using struct Rgba, maybe there is a templated struct RAW or something?

OpenEXR has 11 RgbaChannels defined as...
Code: [Select]
    WRITE_R = 0x01, // Red
    WRITE_G = 0x02, // Green
    WRITE_B = 0x04, // Blue
    WRITE_A = 0x08, // Alpha
    WRITE_Y = 0x10, // Luminance, for black-and-white images, or in combination with chroma
    WRITE_C = 0x20, // Chroma (two subsampled channels, RY and BY, supported only for scanline-based files)
    WRITE_RGB = 0x07, // Red, green, blue
    WRITE_RGBA = 0x0f, // Red, green, blue, alpha
    WRITE_YC = 0x30, // Luminance, chroma
    WRITE_YA = 0x18, // Luminance, alpha
    WRITE_YCA = 0x38 // Luminance, chroma, alpha
...these are handled by OpenEXR, any thing with a special name/type will require some special handling.

edit: ...and looking at images saved by FragM as .exr all have data format HALF even though that format is not specified when saving.
Array2D<Rgba> pixels is HALF type  :-\
edit2: ...so this has revealed something else that needs to be addressed before going any further, going to add an "EXR Output Format" option to preferences for selecting HALF16 or FLOAT32 and some checking to detect and handle format when loading EXR textures.
« Last Edit: September 22, 2019, 10:21:42 PM by 3DickUlus »


xx
"Time Span"

Started by cricke49 on Fractal Image Gallery

0 Replies
338 Views
Last post August 02, 2018, 07:05:21 AM
by cricke49
xx
EXR add save channels to seperate files option

Started by stilikon on Mandelbulber

0 Replies
85 Views
Last post June 03, 2019, 11:18:51 AM
by stilikon
xx
Birdie Style

Started by gannjondal on Fractal Image Gallery

1 Replies
407 Views
Last post May 08, 2018, 02:39:37 PM
by who8mypnuts
clip
Neural Style Transfer with Fractal Art

Started by reallybigname on Other Artforms

1 Replies
179 Views
Last post July 20, 2019, 04:25:41 PM
by reallybigname
xx
Buddhabrot-style Burning Ship [65536x24576]

Started by programagor on Fractal Image Gallery

12 Replies
669 Views
Last post October 10, 2018, 02:12:39 AM
by 3DickUlus