View Full Version : Demo combiner
Good news everyone
As everyone already know - on 42 - all recorded demos ...are split into parts
This is not without a reason ...
The game is old and buggy ... so when some players get bad time ...and will de-sync - demo will get damaged...
to avoid that - at some point were decided to limit the demo time ...(better have 1 damaged part ...than the whole demo)
At the moment - each recorded demo part - is 3 minutes long.
Well .. recently ...i've created a remedy to the fragmentation of the demos ...
From now on - every downloaded demo file with multiple parts - will have an application inside allowing you to combine the parts
into single demo file (i'm sure - who is recording a video - would appreciate it :)
The zipped file - is all the same - just 2 more items present:
6523
directory /tools
with an application inside - 42DemoFileCombiner.exe
and a windows batch file to run it - combine_demos.bat
When you run the combine_demos.bat it will initiate the combining application ...and all demos inside unzipped directory will be combined into single file
note :
only 42amsterdam name formatted files will be used in process
6524
Resulting file - will be the combination of all demos inside directory - good for longer playback.
If any of the demos are broken - you still get a half broken demo. It will not fix anything ...
My best advise - - check the demos manually - if they are all fine - then you can combine them
If not - remove the broken one ...AND all demos after it ... (well ..move them to other place atleast)
you can combine demos after broken part later
------------------------------------------------------
The program file included - is an open source one - you can download source code here - https://42amsterdam.net/samforum/downloads.php?do=file&id=520
there are both original PHP and eventual C++ transcoded versions available
Would not do any good combining other than 42 demos - but as a part of other projects...
feel free to use it ...
There is a little but:)
Demos generated on 42 prior 30 Nov 2018 ...are a bit out of sync ...each part is missing few frames...
so when combining old demos - you'll get a few messages in the game log - about sync ...
6522
Demos generated after 30 Nov - are perfectly sync .... and should give not errors whatsoever...
"correcton: ...demos generated after 2nd December 15:00 CET.... on SE there were still some missing frames"
Good news everyone
As everyone already know - on 42 - all recorded demos ...are split into parts
This is not without a reason ...
The game is old and buggy ... so when some players get bad time ...and will de-sync - demo will get damaged...
to avoid that - at some point were decided to limit the demo time ...(better have 1 damaged part ...than the whole demo)
At the moment - each recorded demo part - is 3 minutes long.
Well .. recently ...i've created a remedy to the fragmentation of the demos ...
From now on - every downloaded demo file with multiple parts - will have an application inside allowing you to combine the parts
into single demo file (i'm sure - who is recording a video - would appreciate it :)
The zipped file - is all the same - its just there are 2 more items present:
6523
directory /tools
with an application inside - 42DemoFileCombiner.exe
and a windows batch file to run it - combine_demos.bat
When you run the combine_demos.bat it will initiate the combining application ...and all demos inside unzipped directory will be combined into single file
note :
only 42amsterdam name formatted files will be used in process
6524
Resulting file - will be the combination of all demos inside directory - good for longer playback.
If any of the demos are broken - you still get a half broken demo. It will not fix anything ...
My best advise - - check the demos manually - if they are all fine - then you can combine them
If not - remove the broken one ...AND all demos after it ... (well ..move them to other place atleast)
you can combine demos after broken part later
------------------------------------------------------
The program file included - is an open source one - you can download source code here - https://42amsterdam.net/samforum/downloads.php?do=file&id=520
there are both original PHP and eventual C++ transcoded versions available
Would not do any good combining other than 42 demos - but as a part of other projects...
feel free to use it ...
There is a little but:)
Demos generated on 42 prior 30 Nov 2018 ...are a bit out of sync ...each part is missing few frames...
so when combining old demos - you'll get a few messages in the game log - about sync ...
6522
Demos generated after 30 Nov - are perfectly sync .... and should give not errors whatsoever...
"correcton: ...demos generated after 2nd December 15:00 CET.... on SE there were still some missing frames"
Hi Ostap, I have seen this thread a few months ago. There would be other threads I have to respond, but I don't have much time.
Ok, I think that as a Demo Combiner was released... Another tool to work in the opposite way is needed.
It would be a little OT, but there is an issue with demos from some maps. Some ones get de-synced and their variants not.
Little Trouble is indeed a BIG trouble when watching demos.
I have found 2007 demos of 1.05 that get de-synced after 1min 40secs on LT and there is no way to re-sync them!
Paradoxically, demos from Shotty Trouble played on 1.05 can be watched on a 1.07 copy of the game. [FALSE! See below]
I thought it was a dll issue, because there are some demos from another website I couldn't watch on Yodeller. After installing Fix42, they got re-synced!
This also happened to an old demo from 2015 (or 2016) when I played on PRO - Multimap* server with GrandPa from Germany and other casual people. I remember there were several corrupted parts, but I was watching them on another notebook. When I transferred that archive onto my netbook, last year, I was able to re-watch that demo. Same for another demo in Hole.
But this isn't the case of Little Trouble... I think there is an error on the map itself. Yes, it sometimes happens to Fortress (there is an old thread here with a de-synced demo published by Bloodnowsky, back in early 2017), but it is rare.
Then, my guess is that it couldn't be a dll issue for sure. Heh, after 6+ years I play SS multiplayer, I'm still noob and ignorant about this! <_<
Looks like you have to draw another flow chart on how demo recordings work in Sam code! :D
So, a tool for splitting demos (that I assume it already exists) could help. But this tool must be customized, like: I want to split a demo in n parts of 1 min duration, or 30 secs duration, 2 mins, 5 mins, etc... So the de-synced parts could be cut out and then rebuild the entire demo with this Demo Combiner.
What do you think about this?
*That demo was recorded on Little Trouble as well.
Paradoxically, demos from Shotty Trouble played on 1.05 can be watched on a 1.07 copy of the game.
Ok, I wrote a fake news: the 1.05 demo of Shotty Trouble on 1.07 installation gets de-synced after 7 mins 15 secs. On the 1.05 installation it plays flawless, but LT demos are still de-synced.
Demo Split ?
good idea :) - i'll make one later
So, a tool for splitting demos (that I assume it already exists) could help. But this tool must be customized, like: I want to split a demo in n parts of 1 min duration, or 30 secs duration, 2 mins, 5 mins, etc... So the de-synced parts could be cut out and then rebuild the entire demo with this Demo Combiner.
What do you think about this?
Splitting will work as you described .... by 1 min ...30sec ...whatever ...
but you wont be able to glue them together after you'll identify the broken one ... - ANYTHING AFTER the broken part - need to be combined separately ....as separate demo ....it cannot be added to the previous part !
Splitting will work as you described .... by 1 min ...30sec ...whatever ...
but you wont be able to glue them together after you'll identify the broken one ... - ANYTHING AFTER the broken part - need to be combined separately ....as separate demo ....it cannot be added to the previous part !
So after splitting and combining, resulting in a creation of 2 different demos, will I get the popup "Missing frames" you showed in the first screenshot, then?
Sorry, me and my zone have been 18 days without telephone/internet line and I read this just a few days ago.
No problem for having different demo parts: I'd prefer 2-3 working demos with a duration of 7mins or 3min30secs per part rather than a corrupted one of 14 mins.
So after splitting and combining, resulting in a creation of 2 different demos, will I get the popup "Missing frames" you showed in the first screenshot, then?
Sorry, me and my zone have been 18 days without telephone/internet line and I read this just a few days ago.
No problem for having different demo parts: I'd prefer 2-3 working demos with a duration of 7mins or 3min30secs per part rather than a corrupted one of 14 mins.
Inside demo there are recorded frames ....each one has time tick stamp. When just few ticks are missing - you'll get the "missing frame" message (even a single one)
When more than few are missing - demo will simply freeze ...until end.
As you said - after splitting to remove bad part - you'll get 2 demos ...... working ...no missing frames ...no warnings
Inside demo there are recorded frames ....each one has time tick stamp. When just few ticks are missing - you'll get the "missing frame" message (even a single one)
When more than few are missing - demo will simply freeze ...until end.
As you said - after splitting to remove bad part - you'll get 2 demos ...... working ...no missing frames ...no warnings
Aaah, then it is ok. Thanks. :)
Actually - after all ...
You gave me an idea ....
Why not break recording of demo once bad sync is detected ? (and maybe mark it somehow ....that next demo is bad..)
then when all players have good sync - start recording again new demo ....
This way bad syncs will be isolated automatically
I think this idea has some sound ... i'll check into it
Actually - after all ...
You gave me an idea ....
Why not break recording of demo once bad sync is detected ? (and maybe mark it somehow ....that next demo is bad..)
then when all players have good sync - start recording again new demo ....
This way bad syncs will be isolated automatically
I think this idea has some sound ... i'll check into it
I think this is a realtime solution to apply server-side when demos automatically start recording.
If you want to make a try with an already recorded demo, try this one from last night.
6603
Part 5 is corrupted after a few seconds. Basically the tool should locate when players re-sync again at only part 5. But still, you will need to cut at least after 10-15 seconds part 5 has started.
Or how is your tool supposed to locate desyncing and then resyncing inside a whole demo?
Hmm, I think we are now talking about 2 different things. :hmmm:
PS: I don't get why the screen says "Little Trouble", whether Jack and Lendru were playing on Fortress... :wtf:
I think this is a realtime solution to apply server-side when demos automatically start recording.
If you want to make a try with an already recorded demo, try this one from last night.
6603
Part 5 is corrupted after a few seconds. Basically the tool should locate when players re-sync again at only part 5. But still, you will need to cut at least after 10-15 seconds part 5 has started.
Or how is your tool supposed to locate desyncing and then resyncing inside a whole demo?
Hmm, I think we are now talking about 2 different things. :hmmm:
PS: I don't get why the screen says "Little Trouble", whether Jack and Lendru were playing on Fortress... :wtf:
We are :) talking about two different things
I meant - while demo is recording ...... server does know when de-sync occur
Not afterwards
or ...:)
during demo recording simply drop into text file times when de-sync is present ...
when combining - this could be used to cut at those places
Good news everyone
There is "Bad-sync" filter now implemented in demo combiner!
How it works:
During demo recording - server checks if there are any bad-syncs
when detected - it will record them into a CSV file (.bst file with same name as demo)
6612
Recorded data:
Client ID
Sequence Index (time tick - there are 20 ticks in a second)
Bad Sync Counter
as you can see - right at the begin - player with index 1 - had some struggle .... few bad syncs ...then good ...again few bad syncs ...good again .... 14 bad syncs ....eventually it counted up to 1200 ..and he got disconnected
6610
When combining (click the "combine_demos_no_bad_sync.bat") - and when bad sync records are present - demo will be cut at that point
Result of combining - will be few playable files without any bad syncs...
6620
Here are some examples of how bad sync may be distributed across demo parts and how demos are combined:
In case of "single damaged demo file":
6615
In case of "few overlapping damaged demo files - variant 1"
6618
In case of "few overlapping damaged demo files - variant 2"
6617
Good part of this filtering - sequences until the first bad sync will be included in working demo
bad part - all sequences in current demo after bad sync section - cannot be re-combined - so if bad sync happens just 10 sec
after demo part starts - total loss is 2m 50 sec ...problem with re-combining in this case ...is that header of each source demo contains snapshot of the world ...when de-synced sequences are removed - anything behind - becomes invalid relative to the world condition when bad-sync started.
(to fix it a bit ....i'll check the possibility to lower recording time to 1-2 minutes to minimize the loss)
Note: file "combine_demos_no_bad_sync.bat" will be added to the zip archive only if there were bad syncs recorded ...(demos recorded before 21-apr-2019 10:00 - will have none)
Or maybe better solution would be "flush" demo recording after bad-sync are gone for more than 3 seconds for example...
This would allow to harvest good sequences that otherwise would be lost
Or maybe better solution would be "flush" demo recording after bad-sync are gone for more than 3 seconds for example...
This would allow to harvest good sequences that otherwise would be lost
Done :)
Now demo part containing bad-syncs will be flushed (stopped and started again) ...after 2 seconds last bad sequence were recorded.
So .. theoretically - maximal lost time when only one player had bad sync - would be 62 seconds (assuming he will play till disconnected)
here is a first example of one:
6622
Wow, good job Ostap!
I'm quite surprised that, with your last version of Demo Combiner, didn't find any bad sync in that demo I posted earlier at reply #9.
I mean, after I upgraded Fix42Mod to latest version, and then combined the demo, I didn't lose any part. The demo worked 100% with no desyncing.
I have another question now: if ".dem" files are archives where Serious Engine saves thousand frames in (while recording), how have you been able to "open" a demo archive and see that?
I'm quite surprised that, with your last version of Demo Combiner, didn't find any bad sync in that demo I posted earlier at reply #9.
I mean, after I upgraded Fix42Mod to latest version, and then combined the demo, I didn't lose any part. The demo worked 100% with no desyncing.
It shouldn't .... that demo has no "bad-sync" times recorded .... frankly - i'm not sure how it got better :)
I have another question now: if ".dem" files are archives where Serious Engine saves thousand frames in (while recording), how have you been able to "open" a demo archive and see that?
Demo is not an archive ....its a binary file with bunch of data dumped into it ....(world state and all the player/monster actions that were present during the game)
If you know the structure of file - its not a problem to read it :) (structure of the file you can derive from C++ code in distributed 1.10 SDK of the game)
I mean, after I upgraded Fix42Mod to latest version, and then combined the demo, I didn't lose any part. The demo worked 100% with no desyncing.
Besides - if you take a closer look at the .bst file - you'll see there is a client ID - all bad syncs are per player !
Bad syncs present in demo file - not necessary break the whole demo ..... have you checked all players ?
It shouldn't .... that demo has no "bad-sync" times recorded .... frankly - i'm not sure how it got better :)
Demo is not an archive ....its a binary file with bunch of data dumped into it ....(world state and all the player/monster actions that were present during the game)
If you know the structure of file - its not a problem to read it :) (structure of the file you can derive from C++ code in distributed 1.10 SDK of the game)
So: I can learn about ".dem" structure on 1.10 SDK, but there isn't a program that opens them... Do I have to compile it in C++, based on ".dem" structure? Correct?
Besides - if you take a closer look at the .bst file - you'll see there is a client ID - all bad syncs are per player !
Bad syncs present in demo file - not necessary break the whole demo ..... have you checked all players ?
No, it didn't give me any bst for this demo. I didn't combine it previously (2+ weeks ago, when I didn't update Fix42Mod yet), because I manually checked all parts before.
I have to check other demos and see if there are any broken. Eventually, would be Notepad++ enough for opening bst files?
So: I can learn about ".dem" structure on 1.10 SDK, but there isn't a program that opens them... Do I have to compile it in C++, based on ".dem" structure? Correct?
Yes but why ? :) ...unless you planning to contribute to the project ...
Eventually, would be Notepad++ enough for opening bst files?
Yep..
Also once you'll get the .bst file .... - you can control when demo will be cut ...
(open for editing the ".bat" file ....you'll see the the command: "tools\42DemoFileCombiner.exe Combined_Demo 1")
That first parameter is the filtering value for bad-syncs ...(default =1 .... cut at first bad sync)
6610
In the .bst file - you can see that player had bad syncs ...but recovered a few times....1,2...RECOVERED.... 1,2...RECOVERED
So ....you can play with the parameter ....and set it to ...for example 14 ...(after 14 counted bad syncs demo will be cut)
This will allow to get bit more of possibly OK frames into working demo ...or even skip the demo splitting ...if there were just few bad syncs. (Max filtering value is 1200 = 60 sec)
Minor update on tweaking the "cut off" parameter:
After checking some demos - If I correctly understand the way it all works - then amount of bad syncs is VERY relative number
It all depends if player had any interactions during bad sync time
If he was just running around without shooting anyone or being shut by anyone - huge possibility that even at high count of bad syncs - demo will still be OK !
There is a little but:)
Demos generated on 42 prior 30 Nov 2018 ...are a bit out of sync ...each part is missing few frames...
so when combining old demos - you'll get a few messages in the game log - about sync ...
I want to UP this because I did the experiment last september. Well, I saved a lot of 42demos since 2013.
I gave the combiner a try with a demo from september, but 2018 (Sep-29-2018, a high score of BadBoy by 34-0*, in which I was part of; he played under the alias of BUBLIK; Map: The Lost Tomb, on Frag 07).
Basically the combiner does the job, but when playing the demo, it just freezes when it reaches the, let's say, point-break of the first part. For instance: if a random demo's Part #1 ends at 3:01, the combined old demo will freeze at that point.
So, my question is: how did you find out that demos prior Nov-30-2018 don't work, while after that date they do? Is the combiner coded specifically for newest demos? Which are the parameters that caused this?
Note: I took the combiner exe and files needed from a random demo (a recent one, from 2020) and then I started my test.
*Indeed it was actually 33-1, since I managed to throw him into the trap! :D
So, my question is: how did you find out that demos prior Nov-30-2018 don't work, while after that date they do? Is the combiner coded specifically for newest demos? Which are the parameters that caused this?
Coz on Nov 30 i've made the fix ....that will stop and restart demo within 1 tick ....without loosing anything
all demos prior to that date - were restarted using old command ...and would loose about 1 sec of data ....
Coz on Nov 30 i've made the fix ....that will stop and restart demo within 1 tick ....without loosing anything
all demos prior to that date - were restarted using old command ...and would loose about 1 sec of data ....
You mean an internal command in the combiner? :huh:
Because I've tried to unfreeze the demo with "dem" commands in Sam, but demo still remains frozen at its part checkpoint. :(
And it will terminate execution within 1-2 mins...
You mean an internal command in the combiner? :huh:
Because I've tried to unfreeze the demo with "dem" commands in Sam, but demo still remains frozen at its part checkpoint. :(
And it will terminate execution within 1-2 mins...
Nop ... within the server it self
You should play demos apart and then combine recorded videos afterwards...
Powered by vBulletin® Version 4.1.12 Copyright © 2024 vBulletin Solutions, Inc. All rights reserved.