Who here appreciate my work?
  3 / 4    
I think it was mostly Chiri doing the coding if I'm correct but both of them we're definitelly behind it with a lot of time fixing shaders as there was a lot of shaders in that game. HeliX used massive amounts of lua to handle PS shaders. VS shaders I think he made normaly. Back to Chiri I don't think their fix was ever completed.
I think it was mostly Chiri doing the coding if I'm correct but both of them we're definitelly behind it with a lot of time fixing shaders as there was a lot of shaders in that game. HeliX used massive amounts of lua to handle PS shaders. VS shaders I think he made normaly. Back to Chiri I don't think their fix was ever completed.

Thanks to everybody using my assembler it warms my heart.
To have a critical piece of code that everyone can enjoy!
What more can you ask for?

donations: ulfjalmbrant@hotmail.com

#31
Posted 05/04/2018 05:19 PM   
[quote="Flugan"]Can we all agree that it's great to have an assembler and it's up to the 3Dmigoto developers if it needs to be rewritten, replaced or updated. DarkStarSword has done a lot of work on documenting and fixing the assembler. I'm happy it has brought enjoyment to many and being one among many resposible.[/quote]Yes, absolutely we can agree there :) [quote="Flugan"]What seems to have been missing for a while is passing games through my debug procedure just for the assembler. As far as I can tell that part is completely missing from 3Dmigoto.[/quote]We have cmd_Decompiler, which includes a validation mode for the assembler + disassembler. [quote="D-Man11"]elbarterino might also be included, although I'm unsure as to what extent was the role he played.[/quote][quote="Flugan"]I think it was mostly Chiri doing the coding if I'm correct but both of them we're definitelly behind it with a lot of time fixing shaders as there was a lot of shaders in that game. HeliX used massive amounts of lua to handle PS shaders. VS shaders I think he made normaly. Back to Chiri I don't think their fix was ever completed.[/quote]Hmmm, well since I wasn't around the community at the time I don't know the answer to that. I've been under the assumption that the initial open source release of 3DMigoto was Chiri's work alone, but if that turns out not to be the case we may need to update the AUTHORS.txt file to credit elbarterino as well. Your comment prompted me to go and read some of the threads on the origins of 3DMigoto, but I'm still not sure (I do see a statement that Chiri contributed the wrapper, but it's not clear to me if that means 3DMigoto was all Chiri's work, or if that was a starting point and elbarterino began also working on it after that point). I've sent Chiri a message asking for clarification, but it looks like his last post was 5 years ago so I have no idea whether he will see it or not.
Flugan said:Can we all agree that it's great to have an assembler and it's up to the 3Dmigoto developers if it needs to be rewritten, replaced or updated. DarkStarSword has done a lot of work on documenting and fixing the assembler. I'm happy it has brought enjoyment to many and being one among many resposible.
Yes, absolutely we can agree there :)

Flugan said:What seems to have been missing for a while is passing games through my debug procedure just for the assembler.
As far as I can tell that part is completely missing from 3Dmigoto.
We have cmd_Decompiler, which includes a validation mode for the assembler + disassembler.

D-Man11 said:elbarterino might also be included, although I'm unsure as to what extent was the role he played.
Flugan said:I think it was mostly Chiri doing the coding if I'm correct but both of them we're definitelly behind it with a lot of time fixing shaders as there was a lot of shaders in that game. HeliX used massive amounts of lua to handle PS shaders. VS shaders I think he made normaly. Back to Chiri I don't think their fix was ever completed.
Hmmm, well since I wasn't around the community at the time I don't know the answer to that. I've been under the assumption that the initial open source release of 3DMigoto was Chiri's work alone, but if that turns out not to be the case we may need to update the AUTHORS.txt file to credit elbarterino as well. Your comment prompted me to go and read some of the threads on the origins of 3DMigoto, but I'm still not sure (I do see a statement that Chiri contributed the wrapper, but it's not clear to me if that means 3DMigoto was all Chiri's work, or if that was a starting point and elbarterino began also working on it after that point). I've sent Chiri a message asking for clarification, but it looks like his last post was 5 years ago so I have no idea whether he will see it or not.

2x Geforce GTX 980 in SLI provided by NVIDIA, i7 6700K 4GHz CPU, Asus 27" VG278HE 144Hz 3D Monitor, BenQ W1070 3D Projector, 120" Elite Screens YardMaster 2, 32GB Corsair DDR4 3200MHz RAM, Samsung 850 EVO 500G SSD, 4x750GB HDD in RAID5, Gigabyte Z170X-Gaming 7 Motherboard, Corsair Obsidian 750D Airflow Edition Case, Corsair RM850i PSU, HTC Vive, Win 10 64bit

Alienware M17x R4 w/ built in 3D, Intel i7 3740QM, GTX 680m 2GB, 16GB DDR3 1600MHz RAM, Win7 64bit, 1TB SSD, 1TB HDD, 750GB HDD

Pre-release 3D fixes, shadertool.py and other goodies: http://github.com/DarkStarSword/3d-fixes
Support me on Patreon: https://www.patreon.com/DarkStarSword or PayPal: https://www.paypal.me/DarkStarSword

#32
Posted 05/04/2018 06:50 PM   
I might be wrong but I don't think cmd_Decompiler is it. Sounded like a bo3b attempt at cmd line. 3Dmigoto/D3D_Shaders is what I'm talking about. Basically ending up with WriteLUT() in the end. edit: I just realized what HeliX said about the MS decompiler being broken. We found the bug with values not represented well in the resulting ASM. I believe there are many more bugs and as HeliX wrote his own disassembler and assembler he avoided the bugs. Currently we are living with these bugs and they are built into the assembler. The only thing I do is take one line of code and return the original binary when I know how. At no point do I concern myself if the code makes sense or if the binary really matches the cleartext.
I might be wrong but I don't think cmd_Decompiler is it. Sounded like a bo3b attempt at cmd line.

3Dmigoto/D3D_Shaders is what I'm talking about. Basically ending up with WriteLUT() in the end.

edit:
I just realized what HeliX said about the MS decompiler being broken.
We found the bug with values not represented well in the resulting ASM.
I believe there are many more bugs and as HeliX wrote his own disassembler and assembler he avoided the bugs.

Currently we are living with these bugs and they are built into the assembler.
The only thing I do is take one line of code and return the original binary when I know how.

At no point do I concern myself if the code makes sense or if the binary really matches the cleartext.

Thanks to everybody using my assembler it warms my heart.
To have a critical piece of code that everyone can enjoy!
What more can you ask for?

donations: ulfjalmbrant@hotmail.com

#33
Posted 05/04/2018 07:29 PM   
And no one thanks mrorange for 6 years of sticking to the community, posting, taking pictures, testing all those 3D vision games, playing even more and my fix on Resident Evil Revelations... yeah and donations of course... :(
And no one thanks mrorange for 6 years of sticking to the community, posting, taking pictures, testing all those 3D vision games, playing even more and my fix on Resident Evil Revelations... yeah and donations of course... :(

Intel Core i7-3820, 4 X 3,60 GHz overclocked to 4,50 GHz ; EVGA Titan X 12VRAM ; 16 GB Corsair Vengeance DDR-1600 (4x 4 GB) ; Asus VG278H 27-inch incl. 3D vision 2 glasses, integrated transmitter ; Xbox One Elite wireless controller ; Windows 10HTC VIVE 2,5 m2 roomscale3D VISION GAMERS - VISIT ME ON STEAM and feel free to add me: http://steamcommunity.com/profiles/76561198064106555 YOUTUBE: https://www.youtube.com/channel/UC1UE5TPoF0HX0HVpF_E4uPQ STEAM CURATOR: https://store.steampowered.com/curator/33611530-Streaming-Deluxe/ Image

#34
Posted 05/04/2018 07:46 PM   
Thank you for being awesome and being my friend.
Thank you for being awesome and being my friend.

Thanks to everybody using my assembler it warms my heart.
To have a critical piece of code that everyone can enjoy!
What more can you ask for?

donations: ulfjalmbrant@hotmail.com

#35
Posted 05/04/2018 07:55 PM   
[quote="Flugan"]I might be wrong but I don't think cmd_Decompiler is it. Sounded like a bo3b attempt at cmd line.[/quote]Bo3b started it and began hooking up the decompiler to it, but it's mostly my work because I wanted a way to work with shaders that didn't require running 3DMigoto itself. I finished hooking up the decompiler to it as well as your assembler and disassembler, and added a validate pass, e.g. the before and after validation passes from a bug fix affecting WATCH_DOGS2: [code] dss@Gideon ~/WATCH_DOGS2/ShaderCache $ ~/cmd_Decompiler-1.2.42.exe -d -V 09c13461f885fc1c-ps.bin Disassembling (Flugan) 09c13461f885fc1c-ps.bin... *** Assembly verification pass failed! *** At least one error occurred during run *** [/code] [code] dss@Gideon ~/WATCH_DOGS2/ShaderCache $ ~/cmd_Decompiler-1.2.52.exe -d -V 09c13461f885fc1c-ps.bin Disassembling (Flugan) 09c13461f885fc1c-ps.bin... Parsing Input Signature section... Parsing Output Signature section... Manufacturing placeholder SHEX section... Reassembled shader missing RDEF section Reassembled shader missing STAT section Assembly verification pass succeeded -> 09c13461f885fc1c-ps.asm [/code] I also use cmd_Decompiler for the offline Unity fixes - my scripts extract the shader binaries directly from the game files without using 3DMigoto (because that allows them to get more information from the game files that isn't available to 3DMigoto) and use cmd_Decompiler to both decompile & disassemble them, then my scripts patch the shaders and put them in ShaderFixes ready for 3DMigoto to use. [quote]3Dmigoto/D3D_Shaders is what I'm talking about. Basically ending up with WriteLUT() in the end.[/quote]We don't have that specific debug facility hooked up, but being able to use cmd_Decompiler is pretty useful since we can dump a problematic shader and debug the assembler completely outside of the game.
Flugan said:I might be wrong but I don't think cmd_Decompiler is it. Sounded like a bo3b attempt at cmd line.
Bo3b started it and began hooking up the decompiler to it, but it's mostly my work because I wanted a way to work with shaders that didn't require running 3DMigoto itself. I finished hooking up the decompiler to it as well as your assembler and disassembler, and added a validate pass,

e.g. the before and after validation passes from a bug fix affecting WATCH_DOGS2:

dss@Gideon ~/WATCH_DOGS2/ShaderCache
$ ~/cmd_Decompiler-1.2.42.exe -d -V 09c13461f885fc1c-ps.bin
Disassembling (Flugan) 09c13461f885fc1c-ps.bin...

*** Assembly verification pass failed!

*** At least one error occurred during run ***


dss@Gideon ~/WATCH_DOGS2/ShaderCache
$ ~/cmd_Decompiler-1.2.52.exe -d -V 09c13461f885fc1c-ps.bin
Disassembling (Flugan) 09c13461f885fc1c-ps.bin...
Parsing Input Signature section...
Parsing Output Signature section...
Manufacturing placeholder SHEX section...
Reassembled shader missing RDEF section
Reassembled shader missing STAT section
Assembly verification pass succeeded
-> 09c13461f885fc1c-ps.asm


I also use cmd_Decompiler for the offline Unity fixes - my scripts extract the shader binaries directly from the game files without using 3DMigoto (because that allows them to get more information from the game files that isn't available to 3DMigoto) and use cmd_Decompiler to both decompile & disassemble them, then my scripts patch the shaders and put them in ShaderFixes ready for 3DMigoto to use.

3Dmigoto/D3D_Shaders is what I'm talking about. Basically ending up with WriteLUT() in the end.
We don't have that specific debug facility hooked up, but being able to use cmd_Decompiler is pretty useful since we can dump a problematic shader and debug the assembler completely outside of the game.

2x Geforce GTX 980 in SLI provided by NVIDIA, i7 6700K 4GHz CPU, Asus 27" VG278HE 144Hz 3D Monitor, BenQ W1070 3D Projector, 120" Elite Screens YardMaster 2, 32GB Corsair DDR4 3200MHz RAM, Samsung 850 EVO 500G SSD, 4x750GB HDD in RAID5, Gigabyte Z170X-Gaming 7 Motherboard, Corsair Obsidian 750D Airflow Edition Case, Corsair RM850i PSU, HTC Vive, Win 10 64bit

Alienware M17x R4 w/ built in 3D, Intel i7 3740QM, GTX 680m 2GB, 16GB DDR3 1600MHz RAM, Win7 64bit, 1TB SSD, 1TB HDD, 750GB HDD

Pre-release 3D fixes, shadertool.py and other goodies: http://github.com/DarkStarSword/3d-fixes
Support me on Patreon: https://www.patreon.com/DarkStarSword or PayPal: https://www.paypal.me/DarkStarSword

#36
Posted 05/04/2018 08:29 PM   
elbarterino was the original poster of 3Dmigoto, which in itself, leads me to believe that he he had a hand in coding it. https://forums.geforce.com/default/topic/548507/ This is the thread where it all began, for those interested mike_ar69 was in early contact with them and would know, he initially hosted their work on his website 3DSolutionGaming. Bo3b, might also know, he too was also in contact with them and as we all know, was eventually given control of the wrapper. https://forums.geforce.com/default/topic/548507/
elbarterino was the original poster of 3Dmigoto, which in itself, leads me to believe that he he had a hand in coding it.

https://forums.geforce.com/default/topic/548507/
This is the thread where it all began, for those interested


mike_ar69 was in early contact with them and would know, he initially hosted their work on his website 3DSolutionGaming. Bo3b, might also know, he too was also in contact with them and as we all know, was eventually given control of the wrapper.


https://forums.geforce.com/default/topic/548507/

#37
Posted 05/05/2018 12:41 AM   
There are different scenarios, fixing the assembler for a single shader which is very similar to fixing the decompiler for a single shader. This is the route most often taken with 3Dmigoto. Only fix shaders used in a fix. My approach is getting the best coverage with all shaders. An assembler bug should not be allowed to exist. I can't do this alone, shaderdumps are needed for all games possible. By working with all possible shaders. The goal is not to support 3Dmigoto but to make the best possible assembler that can assemble all known shaders. There are various problems like games that you need to playthrough to dump all shaders. Here I need help too. WriteLUT is one of the few robust ways to debug the assembler. You can use it with a single shader but you might break something elsewhere. I assume you are doing a black box binary to binary validation step to make sure everything is perfect. If we work together we can create the 2018 version of the assembler. A lot has happened. Offload the assembler to me while I'm fixing every bug I can find. The least I can do is to fix all the bugs that exists. I might need some help integrating it back into 3Dmigoto to add some desperatelly needed error checking. I have no time plan as some bugs might be tricky but the current assembler is needed in the meantime. Some things I'm considering doing: DX12 EAC assembler From what I can see the assembler currently has bugs which is not really acceptable. I built the assembler I am probably the best person to fix it. Time to update Project Flugan to improve 3Dmigoto with an improved assembler.
There are different scenarios, fixing the assembler for a single shader which is very similar to fixing the decompiler for a single shader.

This is the route most often taken with 3Dmigoto. Only fix shaders used in a fix.
My approach is getting the best coverage with all shaders. An assembler bug should not be allowed to exist.
I can't do this alone, shaderdumps are needed for all games possible. By working with all possible shaders.
The goal is not to support 3Dmigoto but to make the best possible assembler that can assemble all known shaders.
There are various problems like games that you need to playthrough to dump all shaders. Here I need help too.

WriteLUT is one of the few robust ways to debug the assembler. You can use it with a single shader but you might break something elsewhere. I assume you are doing a black box binary to binary validation step to make sure everything is perfect.

If we work together we can create the 2018 version of the assembler. A lot has happened. Offload the assembler to me while I'm fixing every bug I can find. The least I can do is to fix all the bugs that exists.
I might need some help integrating it back into 3Dmigoto to add some desperatelly needed error checking.
I have no time plan as some bugs might be tricky but the current assembler is needed in the meantime.

Some things I'm considering doing:
DX12
EAC
assembler

From what I can see the assembler currently has bugs which is not really acceptable. I built the assembler I am probably the best person to fix it.

Time to update Project Flugan to improve 3Dmigoto with an improved assembler.

Thanks to everybody using my assembler it warms my heart.
To have a critical piece of code that everyone can enjoy!
What more can you ask for?

donations: ulfjalmbrant@hotmail.com

#38
Posted 05/05/2018 02:28 AM   
[quote="Necropants"]I think intelligent people in generally are critical of themselves and think too much. There seems to be a link between happiness and how intelligent someone is or lack of intelligence =) Generalisations yes. I also think that people are too quick to diagnose mental disorders and the global mental health care of humanity is in a pretty bad state. Of course not to trivialise anyone here who does genuinely suffer from mental illness. Personally I was in my youth clinically depressed but I found my own ways to combat it and it doesn't seem to afflict me much now. Flugan, People do of course appreciate your work, I am one of them. But have you considered trying to just get into making game fixes through the existing tech we have? People are more receptive to the final results in a game that can be played than the back end although of course that's of vital importance. I am sure you are more than capable of getting your head around it. And hell knows we are lacking in those sort of people at the moment. [/quote] Re the thread - I appreciate anyone who contributes to the 3d gaming scene, including Flugan. I really feel I should donate more and did like the method in place a little bit back where you pay towards a set amount per fix and, though I know this tended to favour certain games over others, if a fix I wanted was all paid up I'd just pay towards another game I liked on the list. Re intelligence, there is something called the Dunning–Kruger effect that explains how less self aware people do better because they are deluded into thinking more highly of themselves. I have a reasonable IQ and pull things apart to try and understand as much as possible - my wife claims that in doing so I ruin things. I tell her she is just afraid to confront reality and she counters by asking if all of the 'truths' I have learnt have made me happy. I know they have made me misanthropic and that seems logical from my point of view. It would seem ignorance truly IS bliss.
Necropants said:I think intelligent people in generally are critical of themselves and think too much. There seems to be a link between happiness and how intelligent someone is or lack of intelligence =)

Generalisations yes. I also think that people are too quick to diagnose mental disorders and the global mental health care of humanity is in a pretty bad state. Of course not to trivialise anyone here who does genuinely suffer from mental illness. Personally I was in my youth clinically depressed but I found my own ways to combat it and it doesn't seem to afflict me much now.

Flugan, People do of course appreciate your work, I am one of them. But have you considered trying to just get into making game fixes through the existing tech we have? People are more receptive to the final results in a game that can be played than the back end although of course that's of vital importance. I am sure you are more than capable of getting your head around it.

And hell knows we are lacking in those sort of people at the moment.



Re the thread - I appreciate anyone who contributes to the 3d gaming scene, including Flugan. I really feel I should donate more and did like the method in place a little bit back where you pay towards a set amount per fix and, though I know this tended to favour certain games over others, if a fix I wanted was all paid up I'd just pay towards another game I liked on the list.

Re intelligence, there is something called the Dunning–Kruger effect that explains how less self aware people do better because they are deluded into thinking more highly of themselves.

I have a reasonable IQ and pull things apart to try and understand as much as possible - my wife claims that in doing so I ruin things. I tell her she is just afraid to confront reality and she counters by asking if all of the 'truths' I have learnt have made me happy. I know they have made me misanthropic and that seems logical from my point of view.

It would seem ignorance truly IS bliss.

#39
Posted 05/05/2018 07:04 AM   
[quote="Flugan"]There are different scenarios, fixing the assembler for a single shader which is very similar to fixing the decompiler for a single shader. This is the route most often taken with 3Dmigoto. Only fix shaders used in a fix.[/quote]No, that's not what we were doing - WATCH_DOGS2 was one specific example where I hit an issue in the assembler that prompted me to finally get around to thoroughly auditing it, but I did not stop at fixing that specific issue and I completely overhauled the assembler to identify and fix as many issues as I could. I had one bug affecting that game, but I fixed dozens more that we'd never seen in practice. [quote]My approach is getting the best coverage with all shaders. An assembler bug should not be allowed to exist. I can't do this alone, shaderdumps are needed for all games possible. By working with all possible shaders. [/quote]I don't actually think that's a good approach at all, and the approach I take doesn't need any games [i]at all[/i] - those only come into play if we miss something and to make sure that the assembler does cope with real world cases, not just theoretical ones. The problem with dumping shaders from a game and testing against those is you are only testing against the subset of HLSL that the game/engine developers actually used, which will never cover everything and you will need a huge range of games that only marginally increase your test coverage, costs a whole lot of time, effort and money, and still doesn't give you 100% coverage. Instead, I wrote my own test cases - silly things that don't serve any practical purpose other than to generate as many variations of shader assembly as possible. These don't take much time to write, but they can enormously increase the test coverage to 99% or better with minimal time and effort - in terms of value for money, these are a platinum mine, while testing against games is panning for gold in a dried up river bed that has already been panned to death. If the assembler can handle our test cases, it can handle almost anything we would find out in the wild - and since I went through and did that process, we haven't found a single case where the assembler failed to assemble a shader from a game. I know that there are still one or two cases it doesn't handle (e.g. double precision literal strings, DX11.1 minimum precision types, functions, debug layer, one hull shader specific declaration), but we have yet to find any games that use those, and we wouldn't even know about those if we were only dumping shaders from games. But that's not the problem - I already did that work back in 2016 to give the assembler close to 100% coverage of all real-world shaders we are likely to see in a DX11 game and 98% coverage of all shaders we theoretically could see. The problem is that the assembler routinely fails with hand crafted assembly. Our shaderhackers at the moment have to be very careful when writing assembly. If they make a typo in an instruction the assembler silently drops the entire line, if they use too few operands the assembler crashes (which we catch, but there is no indication to the shaderhacker as to what the problem was), if they use too many operands the assembler silently ignores the excess ones, if they use a temporary register without allocating it they destabilise the driver and may cause a driver or game crash (or not - it's dumb luck), if they use a swizzle of the wrong length DirectX treats it as though they used .xyzw. If they reference a texture or constant buffer they forgot to declare it assembles but things don't work properly. If they forget the "r" in a temporary register the assembler produces a corrupt binary (can't even be disassembled), if they make a typo in a load instruction data type the assembler hangs or a typo in a system value leads to a corrupt shader, and I'm sure there's probably more (and would encourage shaderhackers to update issue #36 if you have encountered any others). The problem is not how well the assembler can assemble machine generated assembly - it is how badly it copes with human generated assembly. There used to be a lot more issues that I have since resolved, like an open bracket in a comment causing the assembler to enter an infinite loop and hang the program, or lines being silently discarded because they were indented with a tab instead of a space, or a blank line corrupting the entire shader if DOS style newlines were in use. In 3DMigoto 1.3.11 I added an expression evaluator, and if a human made a typo it would do this: [img]https://forums.geforce.com/cmd/default/download-comment-attachment/75005/[/img] That tells the shaderhacker that something is wrong and points to the exact token where the parser detected the problem and what the problem was. That's what I want to see the assembler do - if a human makes an error, they need immediate feedback to know that [i]something[/i] is wrong, and if at all possible it should point to exactly what is wrong. Ideally, the assembler wouldn't even continue at that point - it would just stop and return an error and 3DMigoto wouldn't try to use the corrupt binary risking a driver crash, but unfortunately we also have to consider backwards compatibility - since we very likely have fixes out in the wild that have corrupt shaders that just happen to work anyway we have to be careful if we change the assembler to error out as that could break them, since that is a backwards incompatible change. For these cases we have to make a bit of a judgement call - either we maintain backwards compatibility, go through and audit every fix to make sure it won't break them, or accept that those fixes will either have to be manually updated by the shaderhackers or cannot be updated to a new version of 3DMigoto. We could go through a deprecation phase, where we allow the fixes to continue working with an updated 3DMigoto for a while, but complain loudly and one day in the future we finally go ahead and pull out the backwards compatibility code, breaking those fixes, but giving them time to solve the issue first. [quote]The goal is not to support 3Dmigoto but to make the best possible assembler that can assemble all known shaders.[/quote]I would say that is only a subset of the goal - from 3DMigoto's POV the assembler has to be robust and be able to cope with and report errors. It does also need to be able to assemble any shader we might realistically encounter, but if all we were doing was disassembling and exactly reassembling shaders from a game then what are we even doing here? [quote]There are various problems like games that you need to playthrough to dump all shaders. Here I need help too.[/quote]Like I said above I do not consider that to be a good approach. Test Driven Development is a much better approach here, and while we do also need to test against real-world shaders from games, we don't need to go out of our way for that. [quote]WriteLUT is one of the few robust ways to debug the assembler.[/quote]I don't doubt that, and I'm sure it helped a lot when you were writing the assembler - if you feel that it is still helpful you might consider hooking it up in cmd_Decompiler, but when you're running outside of the game you also have the luxury of being able to use a debugger. [quote]You can use it with a single shader but you might break something elsewhere.[/quote]If that's a risk then write some test cases to catch regressions. At the moment we have test cases for the assembler (and these don't cover everything the assembler could already handle), but we aren't doing automated testing, so we're only half way there - if we could run these from publish.bat we'd find out if we'd broken anything sooner and be able to fix them before release, and if we wanted we could set up an automated system that runs tests every time we push up code to github. [quote]I assume you are doing a black box binary to binary validation step to make sure everything is perfect.[/quote]Yes - the validation pass compares each section individually and fails if there is any differences in the SHEX, SHDR, ISGN, OSGN, etc. sections. It doesn't consider a missing section to be a failure (since sections like RDEF and STAT are optional), but does print a message to indicate their absence. [quote]If we work together we can create the 2018 version of the assembler. A lot has happened. Offload the assembler to me while I'm fixing every bug I can find. The least I can do is to fix all the bugs that exists. I might need some help integrating it back into 3Dmigoto to add some desperatelly needed error checking.[/quote]You don't have commit access to the repository any more, but you can still send us pull requests that give us an opportunity to review the code before accepting it. [quote]I have no time plan as some bugs might be tricky but the current assembler is needed in the meantime.[/quote]That's up to you - in the past when we've raised these kind of issues you have indicated that you felt they would require a full rewrite to cope with. You probably know the assembler better than I do though, so if you feel that they can be shoehorned on to the existing assembler we can go down that path, or maybe just a partial rewrite of the parser might be enough. One thing though - if you want to do this, it might be a good idea for you to get some first hand experience writing assembly and dealing with the assembler's quirks, which probably means trying to use it to fix a game for real. Without that experience you can still improve the assembler and cross off some of the items we've already identified on issue #36, but we only find out about these kind of issues by trying to use it in practice and making mistakes.
Flugan said:There are different scenarios, fixing the assembler for a single shader which is very similar to fixing the decompiler for a single shader.

This is the route most often taken with 3Dmigoto. Only fix shaders used in a fix.
No, that's not what we were doing - WATCH_DOGS2 was one specific example where I hit an issue in the assembler that prompted me to finally get around to thoroughly auditing it, but I did not stop at fixing that specific issue and I completely overhauled the assembler to identify and fix as many issues as I could. I had one bug affecting that game, but I fixed dozens more that we'd never seen in practice.

My approach is getting the best coverage with all shaders. An assembler bug should not be allowed to exist.
I can't do this alone, shaderdumps are needed for all games possible. By working with all possible shaders.
I don't actually think that's a good approach at all, and the approach I take doesn't need any games at all - those only come into play if we miss something and to make sure that the assembler does cope with real world cases, not just theoretical ones.

The problem with dumping shaders from a game and testing against those is you are only testing against the subset of HLSL that the game/engine developers actually used, which will never cover everything and you will need a huge range of games that only marginally increase your test coverage, costs a whole lot of time, effort and money, and still doesn't give you 100% coverage.

Instead, I wrote my own test cases - silly things that don't serve any practical purpose other than to generate as many variations of shader assembly as possible. These don't take much time to write, but they can enormously increase the test coverage to 99% or better with minimal time and effort - in terms of value for money, these are a platinum mine, while testing against games is panning for gold in a dried up river bed that has already been panned to death. If the assembler can handle our test cases, it can handle almost anything we would find out in the wild - and since I went through and did that process, we haven't found a single case where the assembler failed to assemble a shader from a game. I know that there are still one or two cases it doesn't handle (e.g. double precision literal strings, DX11.1 minimum precision types, functions, debug layer, one hull shader specific declaration), but we have yet to find any games that use those, and we wouldn't even know about those if we were only dumping shaders from games.

But that's not the problem - I already did that work back in 2016 to give the assembler close to 100% coverage of all real-world shaders we are likely to see in a DX11 game and 98% coverage of all shaders we theoretically could see.

The problem is that the assembler routinely fails with hand crafted assembly.

Our shaderhackers at the moment have to be very careful when writing assembly. If they make a typo in an instruction the assembler silently drops the entire line, if they use too few operands the assembler crashes (which we catch, but there is no indication to the shaderhacker as to what the problem was), if they use too many operands the assembler silently ignores the excess ones, if they use a temporary register without allocating it they destabilise the driver and may cause a driver or game crash (or not - it's dumb luck), if they use a swizzle of the wrong length DirectX treats it as though they used .xyzw. If they reference a texture or constant buffer they forgot to declare it assembles but things don't work properly. If they forget the "r" in a temporary register the assembler produces a corrupt binary (can't even be disassembled), if they make a typo in a load instruction data type the assembler hangs or a typo in a system value leads to a corrupt shader, and I'm sure there's probably more (and would encourage shaderhackers to update issue #36 if you have encountered any others).

The problem is not how well the assembler can assemble machine generated assembly - it is how badly it copes with human generated assembly.

There used to be a lot more issues that I have since resolved, like an open bracket in a comment causing the assembler to enter an infinite loop and hang the program, or lines being silently discarded because they were indented with a tab instead of a space, or a blank line corrupting the entire shader if DOS style newlines were in use.

In 3DMigoto 1.3.11 I added an expression evaluator, and if a human made a typo it would do this:

Image

That tells the shaderhacker that something is wrong and points to the exact token where the parser detected the problem and what the problem was. That's what I want to see the assembler do - if a human makes an error, they need immediate feedback to know that something is wrong, and if at all possible it should point to exactly what is wrong.

Ideally, the assembler wouldn't even continue at that point - it would just stop and return an error and 3DMigoto wouldn't try to use the corrupt binary risking a driver crash, but unfortunately we also have to consider backwards compatibility - since we very likely have fixes out in the wild that have corrupt shaders that just happen to work anyway we have to be careful if we change the assembler to error out as that could break them, since that is a backwards incompatible change. For these cases we have to make a bit of a judgement call - either we maintain backwards compatibility, go through and audit every fix to make sure it won't break them, or accept that those fixes will either have to be manually updated by the shaderhackers or cannot be updated to a new version of 3DMigoto. We could go through a deprecation phase, where we allow the fixes to continue working with an updated 3DMigoto for a while, but complain loudly and one day in the future we finally go ahead and pull out the backwards compatibility code, breaking those fixes, but giving them time to solve the issue first.

The goal is not to support 3Dmigoto but to make the best possible assembler that can assemble all known shaders.
I would say that is only a subset of the goal - from 3DMigoto's POV the assembler has to be robust and be able to cope with and report errors. It does also need to be able to assemble any shader we might realistically encounter, but if all we were doing was disassembling and exactly reassembling shaders from a game then what are we even doing here?

There are various problems like games that you need to playthrough to dump all shaders. Here I need help too.
Like I said above I do not consider that to be a good approach. Test Driven Development is a much better approach here, and while we do also need to test against real-world shaders from games, we don't need to go out of our way for that.

WriteLUT is one of the few robust ways to debug the assembler.
I don't doubt that, and I'm sure it helped a lot when you were writing the assembler - if you feel that it is still helpful you might consider hooking it up in cmd_Decompiler, but when you're running outside of the game you also have the luxury of being able to use a debugger.

You can use it with a single shader but you might break something elsewhere.
If that's a risk then write some test cases to catch regressions. At the moment we have test cases for the assembler (and these don't cover everything the assembler could already handle), but we aren't doing automated testing, so we're only half way there - if we could run these from publish.bat we'd find out if we'd broken anything sooner and be able to fix them before release, and if we wanted we could set up an automated system that runs tests every time we push up code to github.

I assume you are doing a black box binary to binary validation step to make sure everything is perfect.
Yes - the validation pass compares each section individually and fails if there is any differences in the SHEX, SHDR, ISGN, OSGN, etc. sections. It doesn't consider a missing section to be a failure (since sections like RDEF and STAT are optional), but does print a message to indicate their absence.

If we work together we can create the 2018 version of the assembler. A lot has happened. Offload the assembler to me while I'm fixing every bug I can find. The least I can do is to fix all the bugs that exists.
I might need some help integrating it back into 3Dmigoto to add some desperatelly needed error checking.
You don't have commit access to the repository any more, but you can still send us pull requests that give us an opportunity to review the code before accepting it.

I have no time plan as some bugs might be tricky but the current assembler is needed in the meantime.
That's up to you - in the past when we've raised these kind of issues you have indicated that you felt they would require a full rewrite to cope with. You probably know the assembler better than I do though, so if you feel that they can be shoehorned on to the existing assembler we can go down that path, or maybe just a partial rewrite of the parser might be enough.

One thing though - if you want to do this, it might be a good idea for you to get some first hand experience writing assembly and dealing with the assembler's quirks, which probably means trying to use it to fix a game for real. Without that experience you can still improve the assembler and cross off some of the items we've already identified on issue #36, but we only find out about these kind of issues by trying to use it in practice and making mistakes.

2x Geforce GTX 980 in SLI provided by NVIDIA, i7 6700K 4GHz CPU, Asus 27" VG278HE 144Hz 3D Monitor, BenQ W1070 3D Projector, 120" Elite Screens YardMaster 2, 32GB Corsair DDR4 3200MHz RAM, Samsung 850 EVO 500G SSD, 4x750GB HDD in RAID5, Gigabyte Z170X-Gaming 7 Motherboard, Corsair Obsidian 750D Airflow Edition Case, Corsair RM850i PSU, HTC Vive, Win 10 64bit

Alienware M17x R4 w/ built in 3D, Intel i7 3740QM, GTX 680m 2GB, 16GB DDR3 1600MHz RAM, Win7 64bit, 1TB SSD, 1TB HDD, 750GB HDD

Pre-release 3D fixes, shadertool.py and other goodies: http://github.com/DarkStarSword/3d-fixes
Support me on Patreon: https://www.patreon.com/DarkStarSword or PayPal: https://www.paypal.me/DarkStarSword

#40
Posted 05/05/2018 04:29 PM   
Well saw I your test cases when I looked through your rewrite of the assembler where some parts was untouched but a lot had been improved there. Dumping games will never reach 100% coverage at all and cost a lot of time, time != money so it's basically free. If you keep the assembler so that it covers everything found in the wild it will have 100% coverage where it matters. The process is simple but time consuming. I'm not saying it's a good way to build an assembler but without knowing a list of instructions even I did manage to make a working assembler without using documentation. You can probably imagine how often the assembler broke when applying it to a new game just by using a instruction I didn't know existed. As you write we need to make it a lot better when things go wrong. Unmodified code always works unless there is an assembler bug. What is needed is to improve the writing shaders experience with error checking etc. The assembler does not use the common lexer/parser style of coding. It's pretty robotic. My thinking is adding a lexer parser to validate if the code is correct and report the line and issue, and 3Dmigoto would supply which file it happened in and recover by using the normal binary code. I really think you can add a error checking layer on top with little change to the assembler itself. It's very understandable that I do not have commit access to 3Dmigoto. Any pull request have to be pretty solid. There is also the helix option of replacing the disassembler to get rid of MS bugs. Think it has problems with swizzle and masks for instance. Because the assembler has to handle buggy code due to MS bugs things probably look strange and behave strange when writing ASM code. As I plan to do this the stupid way like before. I like it. If there is a problem with a game you can quickly see if it actually is an assembler bug. Assembler bug should be fixed obviously. Human error should be detected and reported properly. I don't think the parser will add too much cost but it could be disabled outside hunting mode. I've read through most of issue 36 and it's about time to actually sit down and fix them now when the assembler is well used. I appreciate all input and I appreciate everything you've done for 3Dmigoto you are a very nice person. I wish people could see me for the person I really am. edit: removed the word RAGEdemon is refering to. similarly I would say that the hardest thing to deal with is that I'm crazy. With this I refer to my mental illness. I've been through a lot. Being different is not always nice when it comes to bullying. Using these words on another human being is not very nice but here I'm pretty much stating the ugly truth. I can't really be offended by what I write about myself. I have high functioning autism which means that normal people can seem very strange. I might even be considered an autistic savant in a sense. The fact that I'm autistic remains and I don't know how much that comes across in my writing. I could write a lot on this topic as I find it quite interesting.
Well saw I your test cases when I looked through your rewrite of the assembler where some parts was untouched but a lot had been improved there.

Dumping games will never reach 100% coverage at all and cost a lot of time, time != money so it's basically free. If you keep the assembler so that it covers everything found in the wild it will have 100% coverage where it matters. The process is simple but time consuming.

I'm not saying it's a good way to build an assembler but without knowing a list of instructions even I did manage to make a working assembler without using documentation. You can probably imagine how often the assembler broke when applying it to a new game just by using a instruction I didn't know existed.

As you write we need to make it a lot better when things go wrong. Unmodified code always works unless there is an assembler bug. What is needed is to improve the writing shaders experience with error checking etc.

The assembler does not use the common lexer/parser style of coding. It's pretty robotic.
My thinking is adding a lexer parser to validate if the code is correct and report the line and issue, and 3Dmigoto would supply which file it happened in and recover by using the normal binary code.
I really think you can add a error checking layer on top with little change to the assembler itself.

It's very understandable that I do not have commit access to 3Dmigoto. Any pull request have to be pretty solid.
There is also the helix option of replacing the disassembler to get rid of MS bugs. Think it has problems with swizzle and masks for instance. Because the assembler has to handle buggy code due to MS bugs things probably look strange and behave strange when writing ASM code.

As I plan to do this the stupid way like before. I like it. If there is a problem with a game you can quickly see if it actually is an assembler bug. Assembler bug should be fixed obviously. Human error should be detected and reported properly. I don't think the parser will add too much cost but it could be disabled outside hunting mode.

I've read through most of issue 36 and it's about time to actually sit down and fix them now when the assembler is well used.

I appreciate all input and I appreciate everything you've done for 3Dmigoto you are a very nice person.

I wish people could see me for the person I really am.

edit:
removed the word RAGEdemon is refering to.
similarly I would say that the hardest thing to deal with is that I'm crazy.
With this I refer to my mental illness. I've been through a lot.
Being different is not always nice when it comes to bullying.
Using these words on another human being is not very nice but here I'm pretty much stating the ugly truth.
I can't really be offended by what I write about myself.

I have high functioning autism which means that normal people can seem very strange.
I might even be considered an autistic savant in a sense.
The fact that I'm autistic remains and I don't know how much that comes across in my writing.
I could write a lot on this topic as I find it quite interesting.

Thanks to everybody using my assembler it warms my heart.
To have a critical piece of code that everyone can enjoy!
What more can you ask for?

donations: ulfjalmbrant@hotmail.com

#41
Posted 05/05/2018 09:41 PM   
Flugan, no-one has ever hinted at the R word around here; you will find that most of us abhor that word's usage. In reply to your sentiments: “We judge ourselves by our intentions and others by their behaviour.” ― The SPEED of Trust, Stephen M.R. Covey. Perhaps to rebuild some of that trust, you might consider writing a waiver with your code, that it is irrevocable and perpetually open and that you waive any rights to remove it, come what may. We still love ya man, but having a sword hanging over people's heads, especially when one might be perceived to be 'unstable', isn't the right thing to do. We're all rooting for you!
Flugan, no-one has ever hinted at the R word around here; you will find that most of us abhor that word's usage.

In reply to your sentiments:

“We judge ourselves by our intentions and others by their behaviour.” ― The SPEED of Trust, Stephen M.R. Covey.

Perhaps to rebuild some of that trust, you might consider writing a waiver with your code, that it is irrevocable and perpetually open and that you waive any rights to remove it, come what may.

We still love ya man, but having a sword hanging over people's heads, especially when one might be perceived to be 'unstable', isn't the right thing to do.

We're all rooting for you!

Windows 10 64-bit, Intel 7700K @ 5.1GHz, 16GB 3600MHz CL15 DDR4 RAM, 2x GTX 1080 SLI, Asus Maximus IX Hero, Sound Blaster ZxR, PCIe Quad SSD, Oculus Rift CV1, DLP Link PGD-150 glasses, ViewSonic PJD6531w 3D DLP Projector @ 1280x800 120Hz native / 2560x1600 120Hz DSR 3D Gaming.

#42
Posted 05/05/2018 11:20 PM   
I've mentioned removing the assembler but that is just a bad joke that might be misunderstood. I have zero ability to remove anything at all. I have the chance to improve the assembler which would benefit us all. When it's done it can be brought into 3Dmigoto again but while I work on it it won't touch 3Dmigoto. The code is completely stand alone. I want to thank the person who just donated to me. Due to the way paypal works I have no way of telling who you are.
I've mentioned removing the assembler but that is just a bad joke that might be misunderstood.

I have zero ability to remove anything at all.

I have the chance to improve the assembler which would benefit us all.
When it's done it can be brought into 3Dmigoto again but while I work on it it won't touch 3Dmigoto.
The code is completely stand alone.

I want to thank the person who just donated to me. Due to the way paypal works I have no way of telling who you are.

Thanks to everybody using my assembler it warms my heart.
To have a critical piece of code that everyone can enjoy!
What more can you ask for?

donations: ulfjalmbrant@hotmail.com

#43
Posted 05/06/2018 06:54 AM   
[quote="Flugan"]I've mentioned removing the assembler but that is just a bad joke that might be misunderstood.[/quote] hmmm....
Flugan said:I've mentioned removing the assembler but that is just a bad joke that might be misunderstood.


hmmm....

#44
Posted 05/07/2018 07:28 AM   
@D-Man11: Everyone knows it is completely impossible to remove the assembler without the main developers consent.. Don't type something that make you look stupid. Just a hint.
@D-Man11:
Everyone knows it is completely impossible to remove the assembler without the main developers consent..

Don't type something that make you look stupid. Just a hint.

Thanks to everybody using my assembler it warms my heart.
To have a critical piece of code that everyone can enjoy!
What more can you ask for?

donations: ulfjalmbrant@hotmail.com

#45
Posted 05/07/2018 05:16 PM   
  3 / 4    
Scroll To Top