3Dmigoto now open-source...
  62 / 141    
[quote="helifax"][quote="bo3b"]BTW, you could also generate all the .bin files using the command line assembler now, and send them along with the fix as a pre-assembled set of shaders. They'll be loaded now, whereas before the .bin were ignored for ASM files. If you go that route, please still include the .txt files as reference for other ShaderHackers. I really, really don't want to get into a mode where we only ship binaries. As a side note, I used 7zip on maximum compression, and the archive was 13.5M with just text files. As a general idea, I think that .zip is what we should ship, because it's more universal, no one will have trouble with a .zip file, whereas a few people will get snagged by .7z or .rar files. Just thought it was weird that text files didn't compress better for .rar. If you include the presumably not compressible .bin files, that will make the archive 80% bigger, but that's still OK in today's world. Tradeoff of having a slow download versus a slow first launch. [/quote] Yeah! Big thanks bo3b! I was thinking of making the following: - Use cmd_compiler to get the .bin for all the asm shaders. + add the HLSL shaders as text files (which are around 10 I believe in the whole fix) - All the txt and source code I will zip-up and put it next to the shaders. (so we still have the reference and the actual ASM shader code) and place it inside the fix (in a folder called ASMSourceCode or similar) This will make the archive smaller as well, we still have the source code (inside the zip) and the .bins that are actually used. What do you think?:)[/quote] That would certainly be an OK way to do it, although I would say that it is unnecessary. Windows(ntfs) can handle 100K files in a folder without measurable impact, so I'd just go with the easiest of everything in ShaderFixes. [url]http://stackoverflow.com/questions/197162/ntfs-performance-and-large-volumes-of-files-and-directories[/url] The caveat being that this is a folder that changes not at all. For folders that get a lot of rewrites things do get slower. For the normal play mode, we look for a specific file name constructed out of the shader type and hash code, we aren't scanning for a match, we just say "do you have file e706a624b9fe754a-ps.bin?" That leaves the lookup in NTFS which of course uses b-trees, not a file by file lookup.
helifax said:
bo3b said:BTW, you could also generate all the .bin files using the command line assembler now, and send them along with the fix as a pre-assembled set of shaders. They'll be loaded now, whereas before the .bin were ignored for ASM files.

If you go that route, please still include the .txt files as reference for other ShaderHackers. I really, really don't want to get into a mode where we only ship binaries.


As a side note, I used 7zip on maximum compression, and the archive was 13.5M with just text files. As a general idea, I think that .zip is what we should ship, because it's more universal, no one will have trouble with a .zip file, whereas a few people will get snagged by .7z or .rar files. Just thought it was weird that text files didn't compress better for .rar.

If you include the presumably not compressible .bin files, that will make the archive 80% bigger, but that's still OK in today's world. Tradeoff of having a slow download versus a slow first launch.


Yeah! Big thanks bo3b! I was thinking of making the following:

- Use cmd_compiler to get the .bin for all the asm shaders. + add the HLSL shaders as text files (which are around 10 I believe in the whole fix)
- All the txt and source code I will zip-up and put it next to the shaders. (so we still have the reference and the actual ASM shader code) and place it inside the fix (in a folder called ASMSourceCode or similar)

This will make the archive smaller as well, we still have the source code (inside the zip) and the .bins that are actually used.

What do you think?:)

That would certainly be an OK way to do it, although I would say that it is unnecessary. Windows(ntfs) can handle 100K files in a folder without measurable impact, so I'd just go with the easiest of everything in ShaderFixes.

http://stackoverflow.com/questions/197162/ntfs-performance-and-large-volumes-of-files-and-directories

The caveat being that this is a folder that changes not at all. For folders that get a lot of rewrites things do get slower.


For the normal play mode, we look for a specific file name constructed out of the shader type and hash code, we aren't scanning for a match, we just say "do you have file e706a624b9fe754a-ps.bin?" That leaves the lookup in NTFS which of course uses b-trees, not a file by file lookup.

Acer H5360 (1280x720@120Hz) - ASUS VG248QE with GSync mod - 3D Vision 1&2 - Driver 372.54
GTX 970 - i5-4670K@4.2GHz - 12GB RAM - Win7x64+evilKB2670838 - 4 Disk X25 RAID
SAGER NP9870-S - GTX 980 - i7-6700K - Win10 Pro 1607
Latest 3Dmigoto Release
Bo3b's School for ShaderHackers

Posted 03/08/2016 07:00 AM   
Is there any way to increase the depth buffer precision using some rendertarget overrides? In Dirt 3 It's now R24G8 but I wish to force R32. I also didn't see any pixel shaders writing to the depth buffer, so I assume It's done in compute shaders which are not fully implemented yet. Is it possible at all in the current state of 3DMigoto development?
Is there any way to increase the depth buffer precision using some rendertarget overrides? In Dirt 3 It's now R24G8 but I wish to force R32. I also didn't see any pixel shaders writing to the depth buffer, so I assume It's done in compute shaders which are not fully implemented yet. Is it possible at all in the current state of 3DMigoto development?

EVGA GeForce GTX 980 SC
Core i5 2500K
MSI Z77A-G45
8GB DDR3
Windows 10 x64

Posted 03/09/2016 12:06 PM   
The depth buffer is usually written by the [s]rasterizer[/s] output merger stage, not any shaders (it can be written to from pixel shaders, but that's for special cases). You can identify it as the <DepthTarget> on a given pixel shader in the ShaderUsage.txt, or from a frame analysis log/dump. In theory you can force the format to DXGI_FORMAT_D32_FLOAT (R32 is not for depth buffers - that's what they are converted to if they are passed as an input to a shader) with something like: [code] [TextureOverrideDepthBuffer] hash = fill me in format = 40 [/code] Since you noted the game is using R24G8 (which is not a depth buffer format - was that the input to a shader? I assume they were actually using DXGI_FORMAT_D24_UNORM_S8_UINT for the depth buffer), you might be better using DXGI_FORMAT_D32_FLOAT_S8X24_UINT (Format = 20) instead so that you still get a stencil buffer. However I have my doubts this feature will work at all - it's one of the original features but I don't think it has ever been (successfully?) tested. If the game passes NULL for the depth buffer view description it might work, but if the game passes in a description with the D24S8 format it will almost certainly fail.
The depth buffer is usually written by the rasterizer output merger stage, not any shaders (it can be written to from pixel shaders, but that's for special cases). You can identify it as the <DepthTarget> on a given pixel shader in the ShaderUsage.txt, or from a frame analysis log/dump.

In theory you can force the format to DXGI_FORMAT_D32_FLOAT (R32 is not for depth buffers - that's what they are converted to if they are passed as an input to a shader) with something like:
[TextureOverrideDepthBuffer]
hash = fill me in
format = 40

Since you noted the game is using R24G8 (which is not a depth buffer format - was that the input to a shader? I assume they were actually using DXGI_FORMAT_D24_UNORM_S8_UINT for the depth buffer), you might be better using DXGI_FORMAT_D32_FLOAT_S8X24_UINT (Format = 20) instead so that you still get a stencil buffer.

However I have my doubts this feature will work at all - it's one of the original features but I don't think it has ever been (successfully?) tested. If the game passes NULL for the depth buffer view description it might work, but if the game passes in a description with the D24S8 format it will almost certainly fail.

2x Geforce GTX 980 in SLI provided by NVIDIA, i7 6700K 4GHz CPU, Asus 27" VG278HE 144Hz 3D Monitor, BenQ W1070 3D Projector, 120" Elite Screens YardMaster 2, 32GB Corsair DDR4 3200MHz RAM, Samsung 850 EVO 500G SSD, 4x750GB HDD in RAID5, Gigabyte Z170X-Gaming 7 Motherboard, Corsair Obsidian 750D Airflow Edition Case, Corsair RM850i PSU, HTC Vive, Win 10 64bit

Alienware M17x R4 w/ built in 3D, Intel i7 3740QM, GTX 680m 2GB, 16GB DDR3 1600MHz RAM, Win7 64bit, 1TB SSD, 1TB HDD, 750GB HDD

Pre-release 3D fixes, shadertool.py and other goodies: http://github.com/DarkStarSword/3d-fixes
Support me on Patreon: https://www.patreon.com/DarkStarSword or PayPal: https://www.paypal.me/DarkStarSword

Posted 03/09/2016 03:12 PM   
3DMigoto 1.2.35 is out. The only change in this version is to allow #include directives in shaders to refer to files in the same directory as those shaders. That means instead of doing: #include <ShaderFixes/crosshair.hlsl> You can now do: #include <crosshair.hlsl> The old paths will still work, but I recommend switching to the new paths since they don't depend on the game's working directory being consistent (in particular, this fixes an issue due to a difference between the Steam and UPlay versions of FC4 & FCPrimal)
3DMigoto 1.2.35 is out. The only change in this version is to allow #include directives in shaders to refer to files in the same directory as those shaders.

That means instead of doing:
#include <ShaderFixes/crosshair.hlsl>

You can now do:
#include <crosshair.hlsl>

The old paths will still work, but I recommend switching to the new paths since they don't depend on the game's working directory being consistent (in particular, this fixes an issue due to a difference between the Steam and UPlay versions of FC4 & FCPrimal)

2x Geforce GTX 980 in SLI provided by NVIDIA, i7 6700K 4GHz CPU, Asus 27" VG278HE 144Hz 3D Monitor, BenQ W1070 3D Projector, 120" Elite Screens YardMaster 2, 32GB Corsair DDR4 3200MHz RAM, Samsung 850 EVO 500G SSD, 4x750GB HDD in RAID5, Gigabyte Z170X-Gaming 7 Motherboard, Corsair Obsidian 750D Airflow Edition Case, Corsair RM850i PSU, HTC Vive, Win 10 64bit

Alienware M17x R4 w/ built in 3D, Intel i7 3740QM, GTX 680m 2GB, 16GB DDR3 1600MHz RAM, Win7 64bit, 1TB SSD, 1TB HDD, 750GB HDD

Pre-release 3D fixes, shadertool.py and other goodies: http://github.com/DarkStarSword/3d-fixes
Support me on Patreon: https://www.patreon.com/DarkStarSword or PayPal: https://www.paypal.me/DarkStarSword

Posted 03/09/2016 03:19 PM   
[quote="DarkStarSword"]The depth buffer is usually written by the rasterizer, not any shaders (it can be written to from pixel shaders, but that's for special cases). You can identify it as the <DepthTarget> on a given pixel shader in the ShaderUsage.txt, or from a frame analysis log/dump. In theory you can force the format to DXGI_FORMAT_D32_FLOAT (R32 is not for depth buffers - that's what they are converted to if they are passed as an input to a shader) with something like: [code] [TextureOverrideDepthBuffer] hash = fill me in format = 40 [/code] Since you noted the game is using R24G8 (which is not a depth buffer format - was that the input to a shader? I assume they were actually using DXGI_FORMAT_D24_UNORM_S8_UINT for the depth buffer), you might be better using DXGI_FORMAT_D32_FLOAT_S8X24_UINT (Format = 20) instead so that you still get a stencil buffer. However I have my doubts this feature will work at all - it's one of the original features but I don't think it has ever been (successfully?) tested. If the game passes NULL for the depth buffer view description it might work, but if the game passes in a description with the D24S8 format it will almost certainly fail.[/quote] This is what I got: [code]<DepthTarget orig_hash=1271d08b type=Texture2D Width=4096 Height=4096 MipLevels=1 ArraySize=1 RawFormat=53 Format="R16_TYPELESS" SampleDesc.Count=1 SampleDesc.Quality=0 Usage=0 BindFlags=0x48 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget> <DepthTarget orig_hash=13081375 type=Texture2D Width=128 Height=128 MipLevels=1 ArraySize=1 RawFormat=45 Format="D24_UNORM_S8_UINT" SampleDesc.Count=4 SampleDesc.Quality=0 Usage=0 BindFlags=0x40 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget> <DepthTarget orig_hash=4fd84737 type=Texture2D Width=2048 Height=2048 MipLevels=1 ArraySize=1 RawFormat=45 Format="D24_UNORM_S8_UINT" SampleDesc.Count=4 SampleDesc.Quality=0 Usage=0 BindFlags=0x40 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget> <DepthTarget orig_hash=85fbc44f type=Texture2D Width=1920 Height=540 MipLevels=1 ArraySize=1 RawFormat=45 Format="D24_UNORM_S8_UINT" SampleDesc.Count=1 SampleDesc.Quality=0 Usage=0 BindFlags=0x40 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget> <DepthTarget orig_hash=8b502ce3 type=Texture2D Width=480 Height=270 MipLevels=1 ArraySize=1 RawFormat=45 Format="D24_UNORM_S8_UINT" SampleDesc.Count=1 SampleDesc.Quality=0 Usage=0 BindFlags=0x40 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget> <DepthTarget orig_hash=a98c980a type=Texture2D Width=1920 Height=1080 MipLevels=1 ArraySize=1 RawFormat=44 Format="R24G8_TYPELESS" SampleDesc.Count=1 SampleDesc.Quality=0 Usage=0 BindFlags=0x48 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget> <DepthTarget orig_hash=dd53e11b type=Texture2D Width=1920 Height=1080 MipLevels=1 ArraySize=1 RawFormat=45 Format="D24_UNORM_S8_UINT" SampleDesc.Count=4 SampleDesc.Quality=0 Usage=0 BindFlags=0x40 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget> [/code] Some are D24_UNORM_S8_UINT as you said but some R24G8_TYPELESS (actually just one). I've also looked which shaders are bound to the depth targets but all I got is a bunch of shaders with the declarations loking like that: [code]<RenderTarget id=0 handle=2F606910>f8ab9b9b</RenderTarget> <DepthTarget handle=3205DE90>a98c980a</DepthTarget> [/code] sometimes the handle matches with the rendertarget: [code]<RenderTarget id=0 handle=2F605190>adefa88c</RenderTarget> <DepthTarget handle=2F605590>dd53e11b</DepthTarget> [/code] How should I understand that? Are those shaders reading or writing to the depth target? I've tried to override all of them, but I't did nothing.
DarkStarSword said:The depth buffer is usually written by the rasterizer, not any shaders (it can be written to from pixel shaders, but that's for special cases). You can identify it as the <DepthTarget> on a given pixel shader in the ShaderUsage.txt, or from a frame analysis log/dump.

In theory you can force the format to DXGI_FORMAT_D32_FLOAT (R32 is not for depth buffers - that's what they are converted to if they are passed as an input to a shader) with something like:
[TextureOverrideDepthBuffer]
hash = fill me in
format = 40

Since you noted the game is using R24G8 (which is not a depth buffer format - was that the input to a shader? I assume they were actually using DXGI_FORMAT_D24_UNORM_S8_UINT for the depth buffer), you might be better using DXGI_FORMAT_D32_FLOAT_S8X24_UINT (Format = 20) instead so that you still get a stencil buffer.

However I have my doubts this feature will work at all - it's one of the original features but I don't think it has ever been (successfully?) tested. If the game passes NULL for the depth buffer view description it might work, but if the game passes in a description with the D24S8 format it will almost certainly fail.


This is what I got:

<DepthTarget orig_hash=1271d08b type=Texture2D Width=4096 Height=4096 MipLevels=1 ArraySize=1 RawFormat=53 Format="R16_TYPELESS" SampleDesc.Count=1 SampleDesc.Quality=0 Usage=0 BindFlags=0x48 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget>
<DepthTarget orig_hash=13081375 type=Texture2D Width=128 Height=128 MipLevels=1 ArraySize=1 RawFormat=45 Format="D24_UNORM_S8_UINT" SampleDesc.Count=4 SampleDesc.Quality=0 Usage=0 BindFlags=0x40 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget>
<DepthTarget orig_hash=4fd84737 type=Texture2D Width=2048 Height=2048 MipLevels=1 ArraySize=1 RawFormat=45 Format="D24_UNORM_S8_UINT" SampleDesc.Count=4 SampleDesc.Quality=0 Usage=0 BindFlags=0x40 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget>
<DepthTarget orig_hash=85fbc44f type=Texture2D Width=1920 Height=540 MipLevels=1 ArraySize=1 RawFormat=45 Format="D24_UNORM_S8_UINT" SampleDesc.Count=1 SampleDesc.Quality=0 Usage=0 BindFlags=0x40 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget>
<DepthTarget orig_hash=8b502ce3 type=Texture2D Width=480 Height=270 MipLevels=1 ArraySize=1 RawFormat=45 Format="D24_UNORM_S8_UINT" SampleDesc.Count=1 SampleDesc.Quality=0 Usage=0 BindFlags=0x40 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget>
<DepthTarget orig_hash=a98c980a type=Texture2D Width=1920 Height=1080 MipLevels=1 ArraySize=1 RawFormat=44 Format="R24G8_TYPELESS" SampleDesc.Count=1 SampleDesc.Quality=0 Usage=0 BindFlags=0x48 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget>
<DepthTarget orig_hash=dd53e11b type=Texture2D Width=1920 Height=1080 MipLevels=1 ArraySize=1 RawFormat=45 Format="D24_UNORM_S8_UINT" SampleDesc.Count=4 SampleDesc.Quality=0 Usage=0 BindFlags=0x40 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget>


Some are D24_UNORM_S8_UINT as you said but some R24G8_TYPELESS (actually just one).

I've also looked which shaders are bound to the depth targets but all I got is a bunch of shaders with the declarations loking like that:

<RenderTarget id=0 handle=2F606910>f8ab9b9b</RenderTarget>
<DepthTarget handle=3205DE90>a98c980a</DepthTarget>


sometimes the handle matches with the rendertarget:

<RenderTarget id=0 handle=2F605190>adefa88c</RenderTarget>
<DepthTarget handle=2F605590>dd53e11b</DepthTarget>


How should I understand that? Are those shaders reading or writing to the depth target?

I've tried to override all of them, but I't did nothing.

EVGA GeForce GTX 980 SC
Core i5 2500K
MSI Z77A-G45
8GB DDR3
Windows 10 x64

Posted 03/09/2016 11:27 PM   
[quote="bo3b"][size="M"][center][color="orange"]3DMigoto 1.2.34 is out:[/color] [url]https://github.com/bo3b/3Dmigoto/releases[/url][/center][/size] This release is to fix bugs with ASM shader handling. This version should properly handle ASM text files, properly generate the corresponding .bin files with cache_shaders=1, and properly reload the .bin first and not reassemble. In particular this removes all references to the *_reasm.txt files. Those were for assembler validation and are no longer necessary. There should only be four files now, with a sensible naming convention, e.g.: [i]e706a624b9fe754a-ps_replace.txt[/i] - HLSL decompiled shader [i]e706a624b9fe754a-ps_replace.bin[/i] - binary cached version of HLSL shader [i]e706a624b9fe754a-ps.txt[/i] - ASM text file shader [i]e706a624b9fe754a-ps.bin[/i] - binary cached version of ASM shader. If you have any fixes with _reasm.txt files, you should delete all of those, or reinstall so that caching regenerates the .bin files. They will cause problems, as I removed the workarounds for those files when hunting. Let me know if you see any new bugs, I refactored some big chunks of code, and it's always possible to have missed something. Tested only in Witcher 3.[/quote] Does it mean I can rename all the *-ps/vs_replace.bin to *.ps/vs.bin delete all the hlsl shaders and they should load? I've tried and they are not loading.
bo3b said:

This release is to fix bugs with ASM shader handling. This version should properly handle ASM text files, properly generate the corresponding .bin files with cache_shaders=1, and properly reload the .bin first and not reassemble.

In particular this removes all references to the *_reasm.txt files. Those were for assembler validation and are no longer necessary.

There should only be four files now, with a sensible naming convention, e.g.:

e706a624b9fe754a-ps_replace.txt - HLSL decompiled shader
e706a624b9fe754a-ps_replace.bin - binary cached version of HLSL shader
e706a624b9fe754a-ps.txt - ASM text file shader
e706a624b9fe754a-ps.bin - binary cached version of ASM shader.

If you have any fixes with _reasm.txt files, you should delete all of those, or reinstall so that caching regenerates the .bin files. They will cause problems, as I removed the workarounds for those files when hunting.

Let me know if you see any new bugs, I refactored some big chunks of code, and it's always possible to have missed something. Tested only in Witcher 3.


Does it mean I can rename all the *-ps/vs_replace.bin to *.ps/vs.bin delete all the hlsl shaders and they should load? I've tried and they are not loading.

EVGA GeForce GTX 980 SC
Core i5 2500K
MSI Z77A-G45
8GB DDR3
Windows 10 x64

Posted 03/10/2016 12:05 AM   
[quote="Oomek"][quote="bo3b"][size="M"][center][color="orange"]3DMigoto 1.2.34 is out:[/color] [url]https://github.com/bo3b/3Dmigoto/releases[/url][/center][/size] This release is to fix bugs with ASM shader handling. This version should properly handle ASM text files, properly generate the corresponding .bin files with cache_shaders=1, and properly reload the .bin first and not reassemble. In particular this removes all references to the *_reasm.txt files. Those were for assembler validation and are no longer necessary. There should only be four files now, with a sensible naming convention, e.g.: [i]e706a624b9fe754a-ps_replace.txt[/i] - HLSL decompiled shader [i]e706a624b9fe754a-ps_replace.bin[/i] - binary cached version of HLSL shader [i]e706a624b9fe754a-ps.txt[/i] - ASM text file shader [i]e706a624b9fe754a-ps.bin[/i] - binary cached version of ASM shader. If you have any fixes with _reasm.txt files, you should delete all of those, or reinstall so that caching regenerates the .bin files. They will cause problems, as I removed the workarounds for those files when hunting. Let me know if you see any new bugs, I refactored some big chunks of code, and it's always possible to have missed something. Tested only in Witcher 3.[/quote] Does it mean I can rename all the *-ps/vs_replace.bin to *.ps/vs.bin delete all the hlsl shaders and they should load? I've tried and they are not loading.[/quote] Should be unnecessary to do anything. If you already have a fix installed, delete all the .bin and any *_reasm files and set cache_shaders=1 and it will regenerate everything it needs as .bin. Don't rename the files, as I'm not sure how that will work, and would be a strange variant that is not interesting to to debug. With these latest versions, only the 4 file names I specify there are supported. It should work if you delete all the .txt files and only ship .bin, but that would be breaking the spirit of the open-source software. None of us will give you a hard time about it, but it would make it impossible for anyone to learn from or extend your work. There is no performance penalty to keep all files.
Oomek said:
bo3b said:

This release is to fix bugs with ASM shader handling. This version should properly handle ASM text files, properly generate the corresponding .bin files with cache_shaders=1, and properly reload the .bin first and not reassemble.

In particular this removes all references to the *_reasm.txt files. Those were for assembler validation and are no longer necessary.

There should only be four files now, with a sensible naming convention, e.g.:

e706a624b9fe754a-ps_replace.txt - HLSL decompiled shader
e706a624b9fe754a-ps_replace.bin - binary cached version of HLSL shader
e706a624b9fe754a-ps.txt - ASM text file shader
e706a624b9fe754a-ps.bin - binary cached version of ASM shader.

If you have any fixes with _reasm.txt files, you should delete all of those, or reinstall so that caching regenerates the .bin files. They will cause problems, as I removed the workarounds for those files when hunting.

Let me know if you see any new bugs, I refactored some big chunks of code, and it's always possible to have missed something. Tested only in Witcher 3.


Does it mean I can rename all the *-ps/vs_replace.bin to *.ps/vs.bin delete all the hlsl shaders and they should load? I've tried and they are not loading.

Should be unnecessary to do anything. If you already have a fix installed, delete all the .bin and any *_reasm files and set cache_shaders=1 and it will regenerate everything it needs as .bin.

Don't rename the files, as I'm not sure how that will work, and would be a strange variant that is not interesting to to debug.

With these latest versions, only the 4 file names I specify there are supported.

It should work if you delete all the .txt files and only ship .bin, but that would be breaking the spirit of the open-source software. None of us will give you a hard time about it, but it would make it impossible for anyone to learn from or extend your work. There is no performance penalty to keep all files.

Acer H5360 (1280x720@120Hz) - ASUS VG248QE with GSync mod - 3D Vision 1&2 - Driver 372.54
GTX 970 - i5-4670K@4.2GHz - 12GB RAM - Win7x64+evilKB2670838 - 4 Disk X25 RAID
SAGER NP9870-S - GTX 980 - i7-6700K - Win10 Pro 1607
Latest 3Dmigoto Release
Bo3b's School for ShaderHackers

Posted 03/10/2016 12:14 AM   
The reason Is a bit different than you would suspect. When It comes to the 3D fixes I agree, giving away sources is a must. But for mods it's a bit different story. I'm just too lazy to clean up all the messy coding in my 200 shaders, all my comments, overrides in #defines which are meaningless for an end user. Leaving just the bins would save me a lot of work.
The reason Is a bit different than you would suspect. When It comes to the 3D fixes I agree, giving away sources is a must. But for mods it's a bit different story. I'm just too lazy to clean up all the messy coding in my 200 shaders, all my comments, overrides in #defines which are meaningless for an end user. Leaving just the bins would save me a lot of work.

EVGA GeForce GTX 980 SC
Core i5 2500K
MSI Z77A-G45
8GB DDR3
Windows 10 x64

Posted 03/10/2016 12:25 AM   
[quote="Oomek"]The reason Is a bit different than you would suspect. When It comes to the 3D fixes I agree, giving away sources is a must. But for mods it's a bit different story. I'm just to lazy to clean up all the messy coding in my 200 shaders, all my comments, overrides in #defines which are meaningless for an end user. Leaving just the bins would save me a lot of work. [/quote] Well, all I can offer is that we [i]all [/i]have that "shame of coding", it's never as clean or good as we'd like. Best suggestion I can offer from extensive experience is to just don't worry about it. There's no need to clean it up, and even 'skanky' code is 100x better than zero code. I say that as a policy you should adopt for [i]all [/i]your coding, not just mods. I would strongly encourage you ship it as is. Crazy defines, bad comments, no changes, no extra work on it. Someone may or may not ever look at it, and if they have questions you can worry about it then. Don't get me wrong, it's hard, I don't like to be embarrassed by my bad code either, but in the long run it's better for everyone. Case in point- Helix. In principle he had no problem open-sourcing HelixMod, but just wanted to "clean-up" a few things first. Net result...
Oomek said:The reason Is a bit different than you would suspect. When It comes to the 3D fixes I agree, giving away sources is a must. But for mods it's a bit different story. I'm just to lazy to clean up all the messy coding in my 200 shaders, all my comments, overrides in #defines which are meaningless for an end user. Leaving just the bins would save me a lot of work.

Well, all I can offer is that we all have that "shame of coding", it's never as clean or good as we'd like. Best suggestion I can offer from extensive experience is to just don't worry about it. There's no need to clean it up, and even 'skanky' code is 100x better than zero code. I say that as a policy you should adopt for all your coding, not just mods.

I would strongly encourage you ship it as is. Crazy defines, bad comments, no changes, no extra work on it. Someone may or may not ever look at it, and if they have questions you can worry about it then.


Don't get me wrong, it's hard, I don't like to be embarrassed by my bad code either, but in the long run it's better for everyone. Case in point- Helix. In principle he had no problem open-sourcing HelixMod, but just wanted to "clean-up" a few things first. Net result...

Acer H5360 (1280x720@120Hz) - ASUS VG248QE with GSync mod - 3D Vision 1&2 - Driver 372.54
GTX 970 - i5-4670K@4.2GHz - 12GB RAM - Win7x64+evilKB2670838 - 4 Disk X25 RAID
SAGER NP9870-S - GTX 980 - i7-6700K - Win10 Pro 1607
Latest 3Dmigoto Release
Bo3b's School for ShaderHackers

Posted 03/10/2016 12:30 AM   
I get what you mean, but mentality is the hardest thing to change :)
I get what you mean, but mentality is the hardest thing to change :)

EVGA GeForce GTX 980 SC
Core i5 2500K
MSI Z77A-G45
8GB DDR3
Windows 10 x64

Posted 03/10/2016 12:50 AM   
Edit: I hadn't refreshed this page in a few hours, so haven't read the latest discussion. If you haven't already done so, read this background material - it's doesn't answer all your questions unfortunately (there might be better background material, but your google search is as good as mine) and is missing some practical detail (the types of things you only learn by trying), but I need to make sure you at least have the fundamentals down: Graphics pipelines: https://msdn.microsoft.com/en-us/library/windows/desktop/ff476882(v=vs.85).aspx Rasterizer Stage: https://msdn.microsoft.com/en-us/library/windows/desktop/bb205125(v=vs.85).aspx Shader Stages (particularly Pixel Shader Stage): [url]https://msdn.microsoft.com/en-us/library/windows/desktop/bb205146(v=vs.85).aspx#Pixel_Shader_Stage[/url] Output-Merger Stage (particularly Depth-Stencil Testing Overview): https://msdn.microsoft.com/en-us/library/windows/desktop/bb205120(v=vs.85).aspx Configuring Depth-Stencil Functionality: https://msdn.microsoft.com/en-us/library/windows/desktop/bb205074(v=vs.85).aspx Semantics (particularly SV_Depth & related semantics, but ignore what it says about them being readable/writable by any shader, which as far as I know is incorrect - they are only writable and only by the pixel shader): https://msdn.microsoft.com/en-us/library/windows/desktop/bb509647(v=vs.85).aspx One big piece of missing background material there (because it wasn't even a part of DirectX until shader model 5, rather it's an optimisation that every card in existence does because it would be insane not to) is how early Z testing works. My google is as good as yours found this on the subject: [url]https://stackoverflow.com/questions/17898738/early-z-test-depth-test-in-directx-11[/url] Shader model 5 did introduce a pixel shader flag to explicitly enable early Z tests: https://msdn.microsoft.com/en-us/library/windows/desktop/ff471439(v=vs.85).aspx [quote="Oomek]This is what I got: [code]<DepthTarget orig_hash=1271d08b type=Texture2D Width=4096 Height=4096 MipLevels=1 ArraySize=1 RawFormat=53 Format="R16_TYPELESS" SampleDesc.Count=1 SampleDesc.Quality=0 Usage=0 BindFlags=0x48 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget> <DepthTarget orig_hash=13081375 type=Texture2D Width=128 Height=128 MipLevels=1 ArraySize=1 RawFormat=45 Format="D24_UNORM_S8_UINT" SampleDesc.Count=4 SampleDesc.Quality=0 Usage=0 BindFlags=0x40 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget> <DepthTarget orig_hash=4fd84737 type=Texture2D Width=2048 Height=2048 MipLevels=1 ArraySize=1 RawFormat=45 Format="D24_UNORM_S8_UINT" SampleDesc.Count=4 SampleDesc.Quality=0 Usage=0 BindFlags=0x40 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget> <DepthTarget orig_hash=85fbc44f type=Texture2D Width=1920 Height=540 MipLevels=1 ArraySize=1 RawFormat=45 Format="D24_UNORM_S8_UINT" SampleDesc.Count=1 SampleDesc.Quality=0 Usage=0 BindFlags=0x40 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget> <DepthTarget orig_hash=8b502ce3 type=Texture2D Width=480 Height=270 MipLevels=1 ArraySize=1 RawFormat=45 Format="D24_UNORM_S8_UINT" SampleDesc.Count=1 SampleDesc.Quality=0 Usage=0 BindFlags=0x40 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget> <DepthTarget orig_hash=a98c980a type=Texture2D Width=1920 Height=1080 MipLevels=1 ArraySize=1 RawFormat=44 Format="R24G8_TYPELESS" SampleDesc.Count=1 SampleDesc.Quality=0 Usage=0 BindFlags=0x48 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget> <DepthTarget orig_hash=dd53e11b type=Texture2D Width=1920 Height=1080 MipLevels=1 ArraySize=1 RawFormat=45 Format="D24_UNORM_S8_UINT" SampleDesc.Count=4 SampleDesc.Quality=0 Usage=0 BindFlags=0x40 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget> [/code] Some are D24_UNORM_S8_UINT as you said but some R24G8_TYPELESS (actually just one).[/quote]Looks right. The fully typeless one is likely one the game uses as both an input and output (but not at the same time) and will be assigned a type in the resource view depending on which it is being used for (either a fully typed D24_UNORM_S8_UINT format for use as a depth/stencil buffer, or a partially typed DXGI_FORMAT_R24_UNORM_X8_TYPELESS or DXGI_FORMAT_X24_TYPELESS_G8_UINT format when it is used as a shader resource (input), depending on whether the shader is to read the depth (R24_UNORM) or stencil (G8_UINT) sides of the resource. It cannot read both from a single resource view, though two resource views with different types can be used simultaneously to allow that). [quote]I've also looked which shaders are bound to the depth targets but all I got is a bunch of shaders with the declarations loking like that: [code]<RenderTarget id=0 handle=2F606910>f8ab9b9b</RenderTarget> <DepthTarget handle=3205DE90>a98c980a</DepthTarget> [/code] [/quote]You're missing the actual shader hashes here. [quote] sometimes the handle matches with the rendertarget: [code]<RenderTarget id=0 handle=2F605190>adefa88c</RenderTarget> <DepthTarget handle=2F605590>dd53e11b</DepthTarget> [/code] How should I understand that?[/quote]I don't see a match here (....691. != ....519.), those are separate resources. If the handle matches then they are either the same resource, or the handle has been freed and reused on a new resource. ShaderUsage.txt collects stats since the game was launched whenever the overlay is enabled, so it does not provide a snapshot of a single point in time and will just grow and occasionally a handle might get reused. The frame analysis log is better for giving a direct account of how something was used in a single frame, but even it is missing logging of when a resource is created (which is elsewhere in the object model, but we could potentially add it) or freed (which we don't track at all and cannot log without significant rework). [quote]Are those shaders reading or writing to the depth target?[/quote]Depends on what you mean exactly and state that we can't know from this file alone. The OM state matters for this, as does certain features that the pixel shader may use. Generally speaking the depth test does not occur inside the shader - either an early Z test will be performed before the pixel shader is run (not part of the spec, but the most common scenario), or a late Z test will be performed in the output merger stage after it has run. In advanced scenarios the pixel shader may write to oDepth (or oDepthGE or oDepthLE) to override the depth value from the rasterizer (and disable early Z testing as a result), or may use the discard instruction to perform it's own arbitrary testing. In no circumstance that I know of will the pixel shader read from the bound depth buffer, though it might read from a shader resource (a "Register" in the ShaderUsage.txt like any other texture) that was previously used as a depth buffer. The depth buffer will be written to by the output merger stage provided that depth buffer writing is enabled, a depth buffer is bound, the depth test passes, and the pixel has not already been culled/discarded for any other reason. [quote]I've tried to override all of them, but I't did nothing.[/quote]I really only expect it will work in some relatively simple cases. We could potentially override the view description to catch any cases where the game binds the buffer to the pipeline with a different format (minding the required format conversions mentioned above) than what we forced when the resource was created, but even if we do that it is still likely to run into issues if the game ever attempts to copy the resource, so we might end up in a rabbit hole figuring out what we need to override, when we can't answer that question until it is too late to do so. This might still be enough for your case, but I really can't say with any certainty. If that is not enough a more general wildcard approach to format overrides might help, with special treatment for the four possible types of views. Changing 3DMigoto to Wrap the resources is another possible alternative approach we could take, which could allow us to swap out the wrapped resource later at a time when we can answer the question of whether we need to override the format or not (there are also some other potential benefits from this, like swapping out a stereo resource with a mono one or vice versa, and getting us one step closer to being able to replace the 3D Vision middleware and it's heuristics which I am now pretty certain does this type of thing). An alternative to forcing the game's depth buffer format that *potentially* would not need any 3DMigoto changes would be to try to use arbitrary resource copying to swap the depth buffer out for your own one. This is also a long shot - the game might well perform a copy of the depth buffer and will copy it's blank depth buffer (though maybe you could copy it yourself). Attempts by the game to clear the depth buffer will clear it's copy, not yours (though maybe you could clear it yourself with a custom shader), and you would need to track everywhere the depth buffer is assigned, both as a depth target and as a shader resource - plus where any copies of the depth buffer are assigned. Overall I'd rate this as theoretically possible but extremely difficult and very much not recommended. I think this is where I use the usual open source cop out of "I don't need this myself and don't want to write that code nor do I have time to do so, but this project is open source so you could always learn C++ and add the feature yourself", or more concisely: "patches welcome!"
Edit: I hadn't refreshed this page in a few hours, so haven't read the latest discussion.

If you haven't already done so, read this background material - it's doesn't answer all your questions unfortunately (there might be better background material, but your google search is as good as mine) and is missing some practical detail (the types of things you only learn by trying), but I need to make sure you at least have the fundamentals down:

Graphics pipelines:
https://msdn.microsoft.com/en-us/library/windows/desktop/ff476882(v=vs.85).aspx

Rasterizer Stage:
https://msdn.microsoft.com/en-us/library/windows/desktop/bb205125(v=vs.85).aspx

Shader Stages (particularly Pixel Shader Stage):
https://msdn.microsoft.com/en-us/library/windows/desktop/bb205146(v=vs.85).aspx#Pixel_Shader_Stage

Output-Merger Stage (particularly Depth-Stencil Testing Overview):
https://msdn.microsoft.com/en-us/library/windows/desktop/bb205120(v=vs.85).aspx

Configuring Depth-Stencil Functionality:
https://msdn.microsoft.com/en-us/library/windows/desktop/bb205074(v=vs.85).aspx

Semantics (particularly SV_Depth & related semantics, but ignore what it says about them being readable/writable by any shader, which as far as I know is incorrect - they are only writable and only by the pixel shader):
https://msdn.microsoft.com/en-us/library/windows/desktop/bb509647(v=vs.85).aspx

One big piece of missing background material there (because it wasn't even a part of DirectX until shader model 5, rather it's an optimisation that every card in existence does because it would be insane not to) is how early Z testing works. My google is as good as yours found this on the subject:
https://stackoverflow.com/questions/17898738/early-z-test-depth-test-in-directx-11

Shader model 5 did introduce a pixel shader flag to explicitly enable early Z tests:
https://msdn.microsoft.com/en-us/library/windows/desktop/ff471439(v=vs.85).aspx


Oomek said:This is what I got:

<DepthTarget orig_hash=1271d08b type=Texture2D Width=4096 Height=4096 MipLevels=1 ArraySize=1 RawFormat=53 Format="R16_TYPELESS" SampleDesc.Count=1 SampleDesc.Quality=0 Usage=0 BindFlags=0x48 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget>
<DepthTarget orig_hash=13081375 type=Texture2D Width=128 Height=128 MipLevels=1 ArraySize=1 RawFormat=45 Format="D24_UNORM_S8_UINT" SampleDesc.Count=4 SampleDesc.Quality=0 Usage=0 BindFlags=0x40 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget>
<DepthTarget orig_hash=4fd84737 type=Texture2D Width=2048 Height=2048 MipLevels=1 ArraySize=1 RawFormat=45 Format="D24_UNORM_S8_UINT" SampleDesc.Count=4 SampleDesc.Quality=0 Usage=0 BindFlags=0x40 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget>
<DepthTarget orig_hash=85fbc44f type=Texture2D Width=1920 Height=540 MipLevels=1 ArraySize=1 RawFormat=45 Format="D24_UNORM_S8_UINT" SampleDesc.Count=1 SampleDesc.Quality=0 Usage=0 BindFlags=0x40 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget>
<DepthTarget orig_hash=8b502ce3 type=Texture2D Width=480 Height=270 MipLevels=1 ArraySize=1 RawFormat=45 Format="D24_UNORM_S8_UINT" SampleDesc.Count=1 SampleDesc.Quality=0 Usage=0 BindFlags=0x40 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget>
<DepthTarget orig_hash=a98c980a type=Texture2D Width=1920 Height=1080 MipLevels=1 ArraySize=1 RawFormat=44 Format="R24G8_TYPELESS" SampleDesc.Count=1 SampleDesc.Quality=0 Usage=0 BindFlags=0x48 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget>
<DepthTarget orig_hash=dd53e11b type=Texture2D Width=1920 Height=1080 MipLevels=1 ArraySize=1 RawFormat=45 Format="D24_UNORM_S8_UINT" SampleDesc.Count=4 SampleDesc.Quality=0 Usage=0 BindFlags=0x40 CPUAccessFlags=0x0 MiscFlags=0x0></DepthTarget>


Some are D24_UNORM_S8_UINT as you said but some R24G8_TYPELESS (actually just one).
Looks right. The fully typeless one is likely one the game uses as both an input and output (but not at the same time) and will be assigned a type in the resource view depending on which it is being used for (either a fully typed D24_UNORM_S8_UINT format for use as a depth/stencil buffer, or a partially typed DXGI_FORMAT_R24_UNORM_X8_TYPELESS or DXGI_FORMAT_X24_TYPELESS_G8_UINT format when it is used as a shader resource (input), depending on whether the shader is to read the depth (R24_UNORM) or stencil (G8_UINT) sides of the resource. It cannot read both from a single resource view, though two resource views with different types can be used simultaneously to allow that).

I've also looked which shaders are bound to the depth targets but all I got is a bunch of shaders with the declarations loking like that:

<RenderTarget id=0 handle=2F606910>f8ab9b9b</RenderTarget>
<DepthTarget handle=3205DE90>a98c980a</DepthTarget>

You're missing the actual shader hashes here.


sometimes the handle matches with the rendertarget:

<RenderTarget id=0 handle=2F605190>adefa88c</RenderTarget>
<DepthTarget handle=2F605590>dd53e11b</DepthTarget>


How should I understand that?
I don't see a match here (....691. != ....519.), those are separate resources. If the handle matches then they are either the same resource, or the handle has been freed and reused on a new resource.

ShaderUsage.txt collects stats since the game was launched whenever the overlay is enabled, so it does not provide a snapshot of a single point in time and will just grow and occasionally a handle might get reused.

The frame analysis log is better for giving a direct account of how something was used in a single frame, but even it is missing logging of when a resource is created (which is elsewhere in the object model, but we could potentially add it) or freed (which we don't track at all and cannot log without significant rework).

Are those shaders reading or writing to the depth target?
Depends on what you mean exactly and state that we can't know from this file alone. The OM state matters for this, as does certain features that the pixel shader may use.

Generally speaking the depth test does not occur inside the shader - either an early Z test will be performed before the pixel shader is run (not part of the spec, but the most common scenario), or a late Z test will be performed in the output merger stage after it has run.

In advanced scenarios the pixel shader may write to oDepth (or oDepthGE or oDepthLE) to override the depth value from the rasterizer (and disable early Z testing as a result), or may use the discard instruction to perform it's own arbitrary testing.

In no circumstance that I know of will the pixel shader read from the bound depth buffer, though it might read from a shader resource (a "Register" in the ShaderUsage.txt like any other texture) that was previously used as a depth buffer.

The depth buffer will be written to by the output merger stage provided that depth buffer writing is enabled, a depth buffer is bound, the depth test passes, and the pixel has not already been culled/discarded for any other reason.

I've tried to override all of them, but I't did nothing.
I really only expect it will work in some relatively simple cases. We could potentially override the view description to catch any cases where the game binds the buffer to the pipeline with a different format (minding the required format conversions mentioned above) than what we forced when the resource was created, but even if we do that it is still likely to run into issues if the game ever attempts to copy the resource, so we might end up in a rabbit hole figuring out what we need to override, when we can't answer that question until it is too late to do so. This might still be enough for your case, but I really can't say with any certainty. If that is not enough a more general wildcard approach to format overrides might help, with special treatment for the four possible types of views.

Changing 3DMigoto to Wrap the resources is another possible alternative approach we could take, which could allow us to swap out the wrapped resource later at a time when we can answer the question of whether we need to override the format or not (there are also some other potential benefits from this, like swapping out a stereo resource with a mono one or vice versa, and getting us one step closer to being able to replace the 3D Vision middleware and it's heuristics which I am now pretty certain does this type of thing).


An alternative to forcing the game's depth buffer format that *potentially* would not need any 3DMigoto changes would be to try to use arbitrary resource copying to swap the depth buffer out for your own one. This is also a long shot - the game might well perform a copy of the depth buffer and will copy it's blank depth buffer (though maybe you could copy it yourself). Attempts by the game to clear the depth buffer will clear it's copy, not yours (though maybe you could clear it yourself with a custom shader), and you would need to track everywhere the depth buffer is assigned, both as a depth target and as a shader resource - plus where any copies of the depth buffer are assigned. Overall I'd rate this as theoretically possible but extremely difficult and very much not recommended.


I think this is where I use the usual open source cop out of "I don't need this myself and don't want to write that code nor do I have time to do so, but this project is open source so you could always learn C++ and add the feature yourself", or more concisely: "patches welcome!"

2x Geforce GTX 980 in SLI provided by NVIDIA, i7 6700K 4GHz CPU, Asus 27" VG278HE 144Hz 3D Monitor, BenQ W1070 3D Projector, 120" Elite Screens YardMaster 2, 32GB Corsair DDR4 3200MHz RAM, Samsung 850 EVO 500G SSD, 4x750GB HDD in RAID5, Gigabyte Z170X-Gaming 7 Motherboard, Corsair Obsidian 750D Airflow Edition Case, Corsair RM850i PSU, HTC Vive, Win 10 64bit

Alienware M17x R4 w/ built in 3D, Intel i7 3740QM, GTX 680m 2GB, 16GB DDR3 1600MHz RAM, Win7 64bit, 1TB SSD, 1TB HDD, 750GB HDD

Pre-release 3D fixes, shadertool.py and other goodies: http://github.com/DarkStarSword/3d-fixes
Support me on Patreon: https://www.patreon.com/DarkStarSword or PayPal: https://www.paypal.me/DarkStarSword

Posted 03/10/2016 04:02 AM   
[quote="bo3b"][quote="Oomek"]The reason Is a bit different than you would suspect. When It comes to the 3D fixes I agree, giving away sources is a must. But for mods it's a bit different story. I'm just to lazy to clean up all the messy coding in my 200 shaders, all my comments, overrides in #defines which are meaningless for an end user. Leaving just the bins would save me a lot of work. [/quote] Well, all I can offer is that we [i]all [/i]have that "shame of coding", it's never as clean or good as we'd like. Best suggestion I can offer from extensive experience is to just don't worry about it. There's no need to clean it up, and even 'skanky' code is 100x better than zero code. I say that as a policy you should adopt for [i]all [/i]your coding, not just mods. I would strongly encourage you ship it as is. Crazy defines, bad comments, no changes, no extra work on it. Someone may or may not ever look at it, and if they have questions you can worry about it then. Don't get me wrong, it's hard, I don't like to be embarrassed by my bad code either, but in the long run it's better for everyone. Case in point- Helix. In principle he had no problem open-sourcing HelixMod, but just wanted to "clean-up" a few things first. Net result...[/quote] +1 I have seen too many projects die because the developer just wanted to clean it up before releasing it. Helix Mod is a good example of this, but unfortunately far from the only one. Nowadays all my code goes out warts and all.
bo3b said:
Oomek said:The reason Is a bit different than you would suspect. When It comes to the 3D fixes I agree, giving away sources is a must. But for mods it's a bit different story. I'm just to lazy to clean up all the messy coding in my 200 shaders, all my comments, overrides in #defines which are meaningless for an end user. Leaving just the bins would save me a lot of work.

Well, all I can offer is that we all have that "shame of coding", it's never as clean or good as we'd like. Best suggestion I can offer from extensive experience is to just don't worry about it. There's no need to clean it up, and even 'skanky' code is 100x better than zero code. I say that as a policy you should adopt for all your coding, not just mods.

I would strongly encourage you ship it as is. Crazy defines, bad comments, no changes, no extra work on it. Someone may or may not ever look at it, and if they have questions you can worry about it then.


Don't get me wrong, it's hard, I don't like to be embarrassed by my bad code either, but in the long run it's better for everyone. Case in point- Helix. In principle he had no problem open-sourcing HelixMod, but just wanted to "clean-up" a few things first. Net result...


+1

I have seen too many projects die because the developer just wanted to clean it up before releasing it. Helix Mod is a good example of this, but unfortunately far from the only one. Nowadays all my code goes out warts and all.

2x Geforce GTX 980 in SLI provided by NVIDIA, i7 6700K 4GHz CPU, Asus 27" VG278HE 144Hz 3D Monitor, BenQ W1070 3D Projector, 120" Elite Screens YardMaster 2, 32GB Corsair DDR4 3200MHz RAM, Samsung 850 EVO 500G SSD, 4x750GB HDD in RAID5, Gigabyte Z170X-Gaming 7 Motherboard, Corsair Obsidian 750D Airflow Edition Case, Corsair RM850i PSU, HTC Vive, Win 10 64bit

Alienware M17x R4 w/ built in 3D, Intel i7 3740QM, GTX 680m 2GB, 16GB DDR3 1600MHz RAM, Win7 64bit, 1TB SSD, 1TB HDD, 750GB HDD

Pre-release 3D fixes, shadertool.py and other goodies: http://github.com/DarkStarSword/3d-fixes
Support me on Patreon: https://www.patreon.com/DarkStarSword or PayPal: https://www.paypal.me/DarkStarSword

Posted 03/10/2016 04:06 AM   
Wow, we're really getting a lot of out DSS these past few days. I look forward to reading through all those resources to further my understanding, so thanks from me as well for that last lengthy post. I ended up getting the resource copying working last night, and it gave me a perfect fix on that shader! Yay. Turns out the shader I was copying the VPM from actually wasn't being used in the scene, so I switched to another one that is always present (as far as I'm aware) and bingo, problem solved. (which I realized to check this from your last response to me, so thanks again for that one too) Any chance of getting something like the matrix inverse functionality built into 3DM, and able to be triggered in the d3dx.ini file (like how it's done in HelixMod), rather than having to include the custom shader? Just would be nice to cut down on the number of additional things needed to be added into each individual shader. I can put up a feature request on the wiki if it's at all feasible, but if there's reason to not do that, then I won't. (even if it's just that it's not worth the extra coding) Main reason I'm making this post is I've found 2 games with different anomalies regarding shader dumping. First is Metro Last Light. This one is kinda weird, in that I've done up a small fix for the few issues that there were in the game, and with 3DM installed, I'm finding some shaders that are being broken, that I'm pretty sure weren't broken without it installed, but here's the kicker: as soon as I dump the broken shader, it gets fixed without me having to do anything with it. Does that mean it's an issue with the shaders being generated in the shader cache? I've been meaning to try deleting everthing in the shader cache folder, and just run without automatically generating anything there to see if it resolves an issue, but have been focusing my time on other games lately. The 2nd one is The Division, and this one is a little more concerning. Basically, any items that are equipped on the character (hats, guns, backpacks, etc) are broken in a weird way (kinda hard to describe). When I dump the shader, I get a low beep, and the object just disappears. Normally when this happens, I just convert the shader to ASM, and then that at least gets it working as it originally was (still broken, of course), but in this game that's not happening, and the object still is missing. Does that mean there's an issue with the ASM decompiler as well? Also, another question/potential feature request: for games with frequent HLSL decompiler issues, is it possible to have 3DM only generate ASM shaders, both in the shader cache and/or when you dump them? Another request'ish type question for DSS: the shaders are not dumping with headers. How likely is it possible to be able to adapt your Unity/Unreal extract shader headers tool to work with this engine?
Wow, we're really getting a lot of out DSS these past few days. I look forward to reading through all those resources to further my understanding, so thanks from me as well for that last lengthy post.

I ended up getting the resource copying working last night, and it gave me a perfect fix on that shader! Yay. Turns out the shader I was copying the VPM from actually wasn't being used in the scene, so I switched to another one that is always present (as far as I'm aware) and bingo, problem solved. (which I realized to check this from your last response to me, so thanks again for that one too)

Any chance of getting something like the matrix inverse functionality built into 3DM, and able to be triggered in the d3dx.ini file (like how it's done in HelixMod), rather than having to include the custom shader? Just would be nice to cut down on the number of additional things needed to be added into each individual shader. I can put up a feature request on the wiki if it's at all feasible, but if there's reason to not do that, then I won't. (even if it's just that it's not worth the extra coding)

Main reason I'm making this post is I've found 2 games with different anomalies regarding shader dumping. First is Metro Last Light. This one is kinda weird, in that I've done up a small fix for the few issues that there were in the game, and with 3DM installed, I'm finding some shaders that are being broken, that I'm pretty sure weren't broken without it installed, but here's the kicker: as soon as I dump the broken shader, it gets fixed without me having to do anything with it. Does that mean it's an issue with the shaders being generated in the shader cache? I've been meaning to try deleting everthing in the shader cache folder, and just run without automatically generating anything there to see if it resolves an issue, but have been focusing my time on other games lately.

The 2nd one is The Division, and this one is a little more concerning. Basically, any items that are equipped on the character (hats, guns, backpacks, etc) are broken in a weird way (kinda hard to describe). When I dump the shader, I get a low beep, and the object just disappears. Normally when this happens, I just convert the shader to ASM, and then that at least gets it working as it originally was (still broken, of course), but in this game that's not happening, and the object still is missing. Does that mean there's an issue with the ASM decompiler as well?

Also, another question/potential feature request: for games with frequent HLSL decompiler issues, is it possible to have 3DM only generate ASM shaders, both in the shader cache and/or when you dump them?

Another request'ish type question for DSS: the shaders are not dumping with headers. How likely is it possible to be able to adapt your Unity/Unreal extract shader headers tool to work with this engine?

3D Gaming Rig: CPU: i7 7700K @ 4.9Ghz | Mobo: Asus Maximus Hero VIII | RAM: Corsair Dominator 16GB | GPU: 2 x GTX 1080 Ti SLI | 3xSSDs for OS and Apps, 2 x HDD's for 11GB storage | PSU: Seasonic X-1250 M2| Case: Corsair C70 | Cooling: Corsair H115i Hydro cooler | Displays: Asus PG278QR, BenQ XL2420TX & BenQ HT1075 | OS: Windows 10 Pro + Windows 7 dual boot

Like my fixes? Dontations can be made to: www.paypal.me/DShanz or rshannonca@gmail.com
Like electronic music? Check out: www.soundcloud.com/dj-ryan-king

Posted 03/10/2016 06:46 AM   
[quote="DJ-RK"]Any chance of getting something like the matrix inverse functionality built into 3DM, and able to be triggered in the d3dx.ini file (like how it's done in HelixMod), rather than having to include the custom shader? Just would be nice to cut down on the number of additional things needed to be added into each individual shader. I can put up a feature request on the wiki if it's at all feasible, but if there's reason to not do that, then I won't. (even if it's just that it's not worth the extra coding)[/quote]This is something that is a lot easier in DX9 than DX11 because of the differences in constant registers vs constant buffers, and that DX9 software has a cached copy of each of those constant registers (unless DX9 was initialised with a "pure" device, in which case AFAIK matrix copying won't work at all, whereas we should never have that problem) - we technically could do it in the same way Helix Mod does, but it would mean stalling the pipeline while we fetch a buffer from the GPU, inverse the matrix on the CPU then write that to another buffer and send it back to the GPU. I have thought about this, and there is a way we could do it by running the matrix inverse on the GPU from a custom shader (probably a custom compute shader) writing the matrix to another buffer that is then set as an input wherever you need it. We should be able to do that today with the arbitrary resource copying + resource creation (assuming there aren't any more bugs waiting for us). Eventually we could build a shortcut into 3DMigoto to take care of the details of this, but it's not a high priority since we have a working solution today. Assembly shaders is where this would probably gain us the most since writing a matrix inversion routine in assembly can be quite tricky as it won't work until the whole thing is correct (especially in light of gotchas like noticing that a .yzw swizzle does not work as expected in DX11). I haven't tried the custom shader solution for this yet, but I think it should work. [quote]Main reason I'm making this post is I've found 2 games with different anomalies regarding shader dumping. First is Metro Last Light. This one is kinda weird, in that I've done up a small fix for the few issues that there were in the game, and with 3DM installed, I'm finding some shaders that are being broken, that I'm pretty sure weren't broken without it installed, but here's the kicker: as soon as I dump the broken shader, it gets fixed without me having to do anything with it. Does that mean it's an issue with the shaders being generated in the shader cache? I've been meaning to try deleting everthing in the shader cache folder, and just run without automatically generating anything there to see if it resolves an issue, but have been focusing my time on other games lately.[/quote]Weird :-/ It's probably worth trying that on 3DMigoto 1.2.33 since some of the shader management code was reworked in 1.2.34 and that always brings the possibility of introducing new bugs. After decompiling it has magically fixed it, does it remain that way if you quit and restart the game? Does it get magically fixed if you use the assembly version instead of the HLSL version? It might also be a driver heuristic issue - we still don't fully understand everything those can effect. [quote]The 2nd one is The Division, and this one is a little more concerning. Basically, any items that are equipped on the character (hats, guns, backpacks, etc) are broken in a weird way (kinda hard to describe). When I dump the shader, I get a low beep, and the object just disappears. Normally when this happens, I just convert the shader to ASM, and then that at least gets it working as it originally was (still broken, of course), but in this game that's not happening, and the object still is missing. Does that mean there's an issue with the ASM decompiler as well?[/quote]Can you enable export_binary=1 (it's an undocumented option) and send me the bin file of one of these shaders? I can quickly do some tests to see if the disassembler or assembler is damaging it at all. Notably I hit a problem in in Arkham Knight where certain effects (I think it was a fog layer) disappeared when reloading the tile lighting shaders (or was it any shader...?), but I could not find any difference in the shader binary before or after doing that, and if the game was restarted with the replaced shader it worked fine. [quote]Also, another question/potential feature request: for games with frequent HLSL decompiler issues, is it possible to have 3DM only generate ASM shaders, both in the shader cache and/or when you dump them?[/quote]Also something I'd like and haven't had time to code since it's not that hard to do by hand and I've got other higher priority things to work on. I'd also like an option to switch copy on mark to use the assembly versions for games where HLSL is a lost cause. [quote]Another request'ish type question for DSS: the shaders are not dumping with headers. How likely is it possible to be able to adapt your Unity/Unreal extract shader headers tool to work with this engine?[/quote]That depends entirely on the engine unfortunately - the file formats are pretty engine specific, and there is no guarantee that the information will be anywhere. In Unity it was possible because the information we needed was in the asset files right next to the shaders, but not in the shaders themselves. The Unreal script currently only extracts the shader names (and only works on a few games at the moment) - I think it might be possible to get more there, but I haven't got back to studying their code / file formats to work out how yet (and I'm still not super clear on all the different places that engine can put things - there's clearly more than the file I'm looking at at the moment). As a first step to investigate this, see if you can work out which file(s) contain the shaders (look for the 'DXBC' signature and the semantic names like 'SV_Position', 'TEXCOORD', etc) and see if there is any other text strings in the same file that might be the headers.
DJ-RK said:Any chance of getting something like the matrix inverse functionality built into 3DM, and able to be triggered in the d3dx.ini file (like how it's done in HelixMod), rather than having to include the custom shader? Just would be nice to cut down on the number of additional things needed to be added into each individual shader. I can put up a feature request on the wiki if it's at all feasible, but if there's reason to not do that, then I won't. (even if it's just that it's not worth the extra coding)
This is something that is a lot easier in DX9 than DX11 because of the differences in constant registers vs constant buffers, and that DX9 software has a cached copy of each of those constant registers (unless DX9 was initialised with a "pure" device, in which case AFAIK matrix copying won't work at all, whereas we should never have that problem) - we technically could do it in the same way Helix Mod does, but it would mean stalling the pipeline while we fetch a buffer from the GPU, inverse the matrix on the CPU then write that to another buffer and send it back to the GPU. I have thought about this, and there is a way we could do it by running the matrix inverse on the GPU from a custom shader (probably a custom compute shader) writing the matrix to another buffer that is then set as an input wherever you need it. We should be able to do that today with the arbitrary resource copying + resource creation (assuming there aren't any more bugs waiting for us). Eventually we could build a shortcut into 3DMigoto to take care of the details of this, but it's not a high priority since we have a working solution today.

Assembly shaders is where this would probably gain us the most since writing a matrix inversion routine in assembly can be quite tricky as it won't work until the whole thing is correct (especially in light of gotchas like noticing that a .yzw swizzle does not work as expected in DX11). I haven't tried the custom shader solution for this yet, but I think it should work.

Main reason I'm making this post is I've found 2 games with different anomalies regarding shader dumping. First is Metro Last Light. This one is kinda weird, in that I've done up a small fix for the few issues that there were in the game, and with 3DM installed, I'm finding some shaders that are being broken, that I'm pretty sure weren't broken without it installed, but here's the kicker: as soon as I dump the broken shader, it gets fixed without me having to do anything with it. Does that mean it's an issue with the shaders being generated in the shader cache? I've been meaning to try deleting everthing in the shader cache folder, and just run without automatically generating anything there to see if it resolves an issue, but have been focusing my time on other games lately.
Weird :-/

It's probably worth trying that on 3DMigoto 1.2.33 since some of the shader management code was reworked in 1.2.34 and that always brings the possibility of introducing new bugs.

After decompiling it has magically fixed it, does it remain that way if you quit and restart the game?

Does it get magically fixed if you use the assembly version instead of the HLSL version?

It might also be a driver heuristic issue - we still don't fully understand everything those can effect.

The 2nd one is The Division, and this one is a little more concerning. Basically, any items that are equipped on the character (hats, guns, backpacks, etc) are broken in a weird way (kinda hard to describe). When I dump the shader, I get a low beep, and the object just disappears. Normally when this happens, I just convert the shader to ASM, and then that at least gets it working as it originally was (still broken, of course), but in this game that's not happening, and the object still is missing. Does that mean there's an issue with the ASM decompiler as well?
Can you enable export_binary=1 (it's an undocumented option) and send me the bin file of one of these shaders? I can quickly do some tests to see if the disassembler or assembler is damaging it at all.

Notably I hit a problem in in Arkham Knight where certain effects (I think it was a fog layer) disappeared when reloading the tile lighting shaders (or was it any shader...?), but I could not find any difference in the shader binary before or after doing that, and if the game was restarted with the replaced shader it worked fine.

Also, another question/potential feature request: for games with frequent HLSL decompiler issues, is it possible to have 3DM only generate ASM shaders, both in the shader cache and/or when you dump them?
Also something I'd like and haven't had time to code since it's not that hard to do by hand and I've got other higher priority things to work on. I'd also like an option to switch copy on mark to use the assembly versions for games where HLSL is a lost cause.

Another request'ish type question for DSS: the shaders are not dumping with headers. How likely is it possible to be able to adapt your Unity/Unreal extract shader headers tool to work with this engine?
That depends entirely on the engine unfortunately - the file formats are pretty engine specific, and there is no guarantee that the information will be anywhere. In Unity it was possible because the information we needed was in the asset files right next to the shaders, but not in the shaders themselves. The Unreal script currently only extracts the shader names (and only works on a few games at the moment) - I think it might be possible to get more there, but I haven't got back to studying their code / file formats to work out how yet (and I'm still not super clear on all the different places that engine can put things - there's clearly more than the file I'm looking at at the moment).

As a first step to investigate this, see if you can work out which file(s) contain the shaders (look for the 'DXBC' signature and the semantic names like 'SV_Position', 'TEXCOORD', etc) and see if there is any other text strings in the same file that might be the headers.

2x Geforce GTX 980 in SLI provided by NVIDIA, i7 6700K 4GHz CPU, Asus 27" VG278HE 144Hz 3D Monitor, BenQ W1070 3D Projector, 120" Elite Screens YardMaster 2, 32GB Corsair DDR4 3200MHz RAM, Samsung 850 EVO 500G SSD, 4x750GB HDD in RAID5, Gigabyte Z170X-Gaming 7 Motherboard, Corsair Obsidian 750D Airflow Edition Case, Corsair RM850i PSU, HTC Vive, Win 10 64bit

Alienware M17x R4 w/ built in 3D, Intel i7 3740QM, GTX 680m 2GB, 16GB DDR3 1600MHz RAM, Win7 64bit, 1TB SSD, 1TB HDD, 750GB HDD

Pre-release 3D fixes, shadertool.py and other goodies: http://github.com/DarkStarSword/3d-fixes
Support me on Patreon: https://www.patreon.com/DarkStarSword or PayPal: https://www.paypal.me/DarkStarSword

Posted 03/10/2016 10:26 AM   
@darkstarsword Wow, you never cease to surprise me. You have such immense knowledge in that subject and it's great that you are willing to share it. I can see now that what I thought I knew about the depth buffers was completely wrong. I'll need a few times to read what you wrote and some time process it. I'm studying now all the links you've shared.
@darkstarsword

Wow, you never cease to surprise me. You have such immense knowledge in that subject and it's great that you are willing to share it. I can see now that what I thought I knew about the depth buffers was completely wrong. I'll need a few times to read what you wrote and some time process it. I'm studying now all the links you've shared.

EVGA GeForce GTX 980 SC
Core i5 2500K
MSI Z77A-G45
8GB DDR3
Windows 10 x64

Posted 03/10/2016 01:50 PM   
  62 / 141    
Scroll To Top